SoylentNews is people
SoylentNews
A study of pull requests made by nearly 1.4 million users of Github has found that code changes made by women were more likely to get accepted, unless their gender was easily identifiable. The study is awaiting peer review, so keep that in mind:
The researchers, from the computer science departments at Caly Poly and North Carolina State University, looked at around four million people who logged on to Github on a single day - 1 April 2015. Github is an enormous developer community which does not request gender information from its 12 million users. However the team was able to identify whether roughly 1.4m were male or female - either because it was clear from the users' profiles or because their email addresses could be matched with the Google+ social network. The researchers accepted that this was a privacy risk but said they did not intend to publish the raw data.
The team found that 78.6% of pull requests made by women were accepted compared with 74.6% of those by men. The researchers considered various factors, such as whether women were more likely to be responding to known issues, whether their contributions were shorter in length and so easier to appraise, and which programming language they were using, but they could not find a correlation.
However among users who were not well known within the community, those whose profiles made clear that they were women had a much lower acceptance rate than those whose gender was not obvious. "For outsiders, we see evidence for gender bias: women's acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong," the paper noted.
"Women have a higher acceptance rate of pull requests overall, but when they're outsiders and their gender is identifiable, they have a lower acceptance rate than men. Our results suggest that although women on Github may be more competent overall, bias against them exists nonetheless," the researchers concluded.
[Continues...]
The excellent Slate Star Codex has analysed this data.
I would highly recommend reading Scott Alexander's full analysis, but here's his summation...
So, let’s review. A non-peer-reviewed paper shows that women get more requests accepted than men. In one subgroup, unblinding gender gives women a bigger advantage; in another subgroup, unblinding gender gives men a bigger advantage. When gender is unblinded, both men and women do worse; it’s unclear if there are statistically significant differences in this regard.Only one of the study’s subgroups showed lower acceptance for women than men, and the size of the difference was 63% vs. 64%, which may or may not be statistically significant. This may or may not be related to the fact, demonstrated in the study, that women propose bigger and less useful changes on average; no attempt was made to control for this. This tiny amount of discrimination against women seems to be mostly from other women, not from men.
The media uses this to conclude that “a vile male hive mind is running an assault mission against women in tech.”
Every time I say I’m nervous about the institutionalized social justice movement, people tell me that I’m crazy, that I’m just sexist and privileged, and that feminism is merely the belief that women are people so any discomfort with it is totally beyond the pale. I would nevertheless like to re-emphasize my concerns at this point.
(Score: 5, Insightful) by TGV on Monday February 15 2016, @06:52AM
Statistical conclusions in the article are faulty: the significance is based on a chi square with enormously high dfs, as high as 3,064,667. Any difference is significant with a df that high, even though the differences are minute (14% vs 14.5%, really?). The researchers should think if their model that the underlying data truly only represents a difference in gender and all other possible variables are identical is true.
A large part of the article focuses on arguments like "they feel dejected" while in reality the numbers hardly differ. Not only that, they are in the women's favor, even on the first request. How can you then speak about feelings of dejection or abandoning because of "an unreasonably aggressive argument style" (as if women are by definition incapable of that)? No, it's just clutching at straws because they have to write an article.
But it's the final graph that is the nail in the coffin of this article: even with their self-chosen statistics, there is no difference in acceptance rate for men and women when gender is known (although "known" is too strong a word), not even in the outsider category. To get to some form of political conclusion, they then phrase it like this: "There is a similar drop for men, but the effect is not as strong" while not having even the cheapest statistical argument to support it. That's the best they can come up.
So the conclusion of this article should be: women have a slight advantage in pull requests on github. The rest is FUD.
(Score: 3, Insightful) by bradley13 on Monday February 15 2016, @07:27AM
I agree with the comments above: the authors clearly start with their desired conclusions, and then torture the data into supporting those conclusions. Even then, they are not actually successful.
I would also like to point out another aspect: This recent practice of publicizing articles that have not yet been accepted for publication - this is despicable science. If the reviewers know anything about statistics, this paper will have to be massively revised before publication. Indeed, it may well have to conclude the opposite of what the authors want, namely, that the analysis shows that there is no gender bias at all.
Everyone is somebody else's weirdo.
(Score: 3, Insightful) by TGV on Monday February 15 2016, @08:07AM
> If the reviewers know anything about statistics
That's a big assumption. Most researchers know just enough about statistics to be dangerous. The problems of Fisher/Neyman significance testing are abundantly clear, the H0 phallacy is well-known since the 1960s, but in social sciences, the researchers just continue using them in order to get positive conclusions. And then we don't even discuss the methodological errors and wrong use of statistical tools (all correlations between 200 items of a questionnaire, etc).
(Score: 3, Insightful) by Vanderhoth on Monday February 15 2016, @12:35PM
Depends on who you mean by "reviewers", if you read the study at the top of each page it says, "not peer-reviewed", in big bold letters. So the only reviewers of this study are the journalists writing about it. Yes the author screwed up, they had a conclusion and went about finding data to support that. To me it doesn't even look like they did a good job. It's like they found the opposite, then stated the conclusion anyway and cut out anything that might have brought that conclusion in to SERIOUS consideration.
The journalist though. They looked at this and saw a beautiful opportunity for clickbait even if there were serious errors. Although, I'm not sure if it's they didn't understand it or if they did, but pushed it anyway. Journalists normally aren't math people. Most of the time they're just writers who write about A LOT of different topics and have at best an entry level understanding of what it is they're reporting on. So they go to the first simplest source that supports what they want to write and use that to spin an article.
I'm a little P.O'ed at takyon for submitting this garbage and at the editors for letting it through because they're giving the journalist exactly what they want, clicks. Positive re-enforcement to keep lying, misrepresenting facts, and pushing agendas by giving attention to obviously shitty studies and producing shitty clickbait. Which in turn leads researchers to continue producing shitty studies because they KNOW they'll get attention if their claims are outlandish enough, which will bring them notoriety and funding to keep "researching" ideologically driven issues. Poorly, I might add.
"Now we know", "And knowing is half the battle". -G.I. Joooooe
(Score: 2) by shrewdsheep on Monday February 15 2016, @12:01PM
TLDR
Statistical conclusions in the article are faulty: the significance is based on a chi square with enormously high dfs, as high as 3,064,667. Any difference is significant with a df that high, even though the differences are minute (14% vs 14.5%, really?).
It is the other way round. High degrees of freedom (df) make it more difficult to reject the null hypothesis (the tail become much more heavy) as compared to a test with lower df. Increasing *sample size* will ultimately allow to prove even the tiniest difference. A reasonable approach that is followed sometimes (and I thought is prevalent in sociology, but maybe not) to test whether differences (frequency differences in this case) exceed a threshold of minimal relevance.
The researchers should think if their model that the underlying data truly only represents a difference in gender and all other possible variables are identical is true.
To me, this would be the critical point. Studies like this are full of confounding (https://en.wikipedia.org/wiki/Confounding) and proper control thereof seems nigh impossible in this case.
(Score: 2) by TGV on Monday February 15 2016, @07:22PM
You're right, it *is* the other way around. df = 1, n is 3M.