Gender Bias in Language Research?

On the WPA-L, someone sent around the link to The Gender Genie, a program that uses an algorithm to determine if the author of a text was male or female. First of all, I think the whole thing is hugely dubious. It is based on statistical analysis of the British National Corpus, so there is nothing remotely innate about the correlation between gender and "the use of pronouns and certain types of noun modifiers" (Argamon, Koppel, Fine, Shimon [.pdf document]). The conclusions are rightly stated as "socialization of gender."

However, even this claim needs to be tempered. I ran my own little test using the Gender Genie. When submitting blocks of text, one must choose the type of text it is from broad genre categories of fiction, non-fiction, or blog. Since I don't really have a great deal of fiction to submit, I first took a piece of this blog and ran it through the Genie. It claimed the author was female. As a comparison, I took a piece of my dissertation. It claimed the author was male. Why the difference?

It maybe that the samples I submitted were too short. The blog was 289 words and the section of my dissertation was 382 words. So, I ran another test, this time with a different blog entry and a different section of my dissertation. They were 816 and 900 words, respectively. I achieved the same results with these samples as with the first test. My blog entry was seen as "female" and my dissertation was seen as "male."

My dissertation discusses theory and empirical data while my blog tends to remain more theory oriented. However, I would think this would actually score a difference the other way around with real experience scoring as more "feminine," assuming the Gender Genie reinforces the stereotype of abstract discourse as a penchant for males and experiential or intuitive discourse as something innately female. Moreover, since both blog entries and my dissertation involve academic thinking about discourse, the fact that any difference was detected strikes me as patently odd.

Now, to be fair, my dissertation involves more conclusive theorizing than the two blog entries I used previously. So, I ran a third test, this time involving only one strongly argued (and firmly believed) blog entry since we know the dissertation is "male." Based on only 441 words, mind you, the Gender Genie found this one to be "male" as it did my dissertation.

Conclusions? Beyond that the Gender Genie is absolute bunk (like you needed me to tell you)? That what is being measured here are stereotypes. The dissertation sections and the lone blog entry that scored as male did so because of the conviction and assured manner in which they were argued, not the more tentative and speculative manner of my blog entries. As such, the algorithm picks out a degree of certainty with which the author uses language to relate the reader to the subject matter being written. That is it. There is another step required to make the leap from that measure to the conclusion that these are indicative of the ways in which British or even Western societies socialize along gendered lines. The authors of the Gender Genie makes that step, equating a self-assured manner of writing with gender characteristics. But, in doing so only reinforces sexist stereotypes about women's and men's ways of writing.

Don't believe me? As a final check, I ran two more texts: Gwen Ifill's op-ed about Don Imus' derogation of the Rutger's women's basketball team and this blog entry. Ifill's op-ed that took Imus to task was 765 words and was seen as male. This entry? 618 words and written by a male.