SoylentNews is people
SoylentNews
Scientific literature often mis-names genes and boffins say Microsoft Excel is partly to blame.
"Automatic conversion of gene symbols to dates and floating-point numbers is a problematic feature of Excel software," In a paper titled write Mark Ziemann, Yotam Eren and Assam El-OstaEmai of the Baker IDI Heart & Diabetes Institute in Australia in a paper titled Gene name errors are widespread in the scientific literature .
Among the things Excel does to gene names include changing "SEPT2", the name of a gene thought to have a role in proper formation of cell structure, to the date "2-Sep". The "MARCH1" gene becomes "1-Mar".
The paper notes that this is a problem that's been know for over a decade, but one which remains pervasive. The trio studied 35,175 Excel tables attached to 3,597 scientific papers published between 2005 and 2015 and found errors in "987 supplementary files from 704 published articles. Of the selected journals, the proportion of published articles with Excel files containing gene lists that are affected by gene name errors is 19.6 per cent."
It's not hard to change the default format of Excel cell to avoid changes of this sort: you can get it done in a click or three. Much of the problem in these papers is therefore between scientists' ears, rather than within Excel itself. The paper's silent on why genetic scientists, who The Register will assume are not short of intelligence, have been making Excel errors for years.
This article focuses on errors resulting from auto-correction of gene names; certainly other subject areas have suffered from similarly 'helpful' software. What hilarious and/or cringe-worthy 'corrections' have YOU seen?
(Score: 2) by martyb on Friday August 26 2016, @11:29AM
Comma-Separated Value (CSV) files have other heartaches for those who are not wary.
In my experience, there is no single, definitive format for CSV files — each vendor seems to have their own take on things. This most frequently tortured me when dealing with how different products 'escaped' certain values. From memory: double quote ("), comma (,), and semi-colon (;).
Oh, when you *really* mean that you want a double quote as part of the value in a cell, just enter it twice... or do you precede it with an escape character like slash (/)?
Wikipedia explains CSV [wikipedia.org] it better than I can this early in the morning:
In retrospect, what was needed was an initial record in the file which defined the separator and quoting characters. Analogous to a shell script which starts with a bang-path to differentiate shell, bourne shell, bash, perl, etc. programs.
At this stage, IMHO. possibly the best one could hope for is a standard test file which exhaustively expresses these possible misinterpretations and makes clear what is done in each case.
Wit is intellect, dancing.
(Score: 2) by digitalaudiorock on Friday August 26 2016, @01:53PM
Absolutely. Tab separated values without quotes are 1000 times better. I've never understood why a format like CSV, which depends on separators that are so likely to appear in data became so common.
(Score: 3, Insightful) by bzipitidoo on Friday August 26 2016, @03:19PM
> separators that are so likely to appear in data
That's always the problem. Every time a new way to separate information was created, it became embedded, already used so that another method of separation had to be created to avoid confusion. Sometimes that leads to monstrosities such as HTML. Originally, writing did not have punctuation-- no commas, periods, semicolons and colons, and not even spaces. Yes, the space is considered punctuation.
Heck, much ancient writing didn't even have vowels. Vowels are formally part of the alphabet, have been for centuries, but punctuation still is not, and children are not taught a list of punctuation symbols in the same manner as taught the alphabet and numerical digits. English speakers all know that the English alphabet has 26 letters, but can anyone say how many standard punctuation symbols there are off the top of their heads? Most people would overlook the space, despite that being by far the largest key on the typical keyboard. Or, more like wouldn't think of it as punctuation. Punctuation is just quietly slipped in as something of an afterthought while teaching writing. Much early writing uses a mid level dot ยท to separate most words but not all, as it was only a tool to reduce ambiguity and fairly often there was no ambiguity and so no need for a word separator.
We have an ongoing debate between fixed and variable width fonts. I have yet to see a programming language that is written in a proportional font. Think what a mess that would make of Python especially. For separation schemes that depend on spacing, placing symbols on a grid is essential. Can get away with a proportional font for CSV. But for more sophisticated schemes for separating data elements, it sure is nice to be able to use spacing.
(Score: 2) by digitalaudiorock on Friday August 26 2016, @05:29PM
Wow...I'm not sure I've ever read anything that's confused me more. A delimited text format is intended for data transport purposes and simply must be parsable by whatever uses it. What on earth do fonts have to do with any of that in any way?
(Score: 0) by Anonymous Coward on Friday August 26 2016, @08:52PM
I have no clue what fonts have to do with it but I was waiting for the GP to complain that RPG columns don't line up with variable width fonts. Yes, RPG - I went there because I'm ooooold school.
(Score: 1) by nitehawk214 on Friday August 26 2016, @02:31PM
In my experience, there is no single, definitive format for CSV files.
My current project is importing CSV with the following format.
1. Blindly place double quotes around every string, whether it is needed or not.
2. Do not escape double quotes inside the strings, screwing most parsers over
3. Place random carriage returns in the quoted strings. Failing anything with extra double quotes.
Of course the data inside the CSV is not well structured at all, anyhow.
Opencsv [mvnrepository.com] does a pretty decent job on it, but a lot of lines get rejected.
Fuck CSV.
"Don't you ever miss the days when you used to be nostalgic?" -Loiosh
(Score: 5, Informative) by jdccdevel on Friday August 26 2016, @05:06PM
What bothers me is that ASCII even has characters for exactly this purpose! If people had just used the characters specified by the standards properly in their file format, we could have embedded any kind of readable text in a record (newlines, quotes, tabs, commas, whatever), without having to resort to escape sequences at all!
From the ASCII table:
0x1D => Group Separator (to indicate the end of a row)
0x1E => Record Separator (to separate columns)
Seriously, ASCII has been around for How Long? Why the hell are we always re-inventing the wheel, and with something that doesn't work as well?!
How many man hours would have been saved if someone had put just a little more thought into their file format?
</rant>
Sorry about that, the article seems to have struck a nerve.
(Score: 0) by Anonymous Coward on Friday August 26 2016, @08:58PM
I use 1C (FS), 1D (GS), 1E (RS) and 1F (US) in my data all the time, especially in Ajax data transmissions. Like you said, that's why they're there.