How Not To Handle Failure

Firebrand attention whore Milo Yiannopoulos has hit on some pretty interesting work at interviewing.io, in which the company attempted to counter interviewer sex bias by using voice masking technology. The results (emboldening, for once, is original equipment):

interviewing.io is a platform where people can practice technical interviewing anonymously and, in the process, find jobs based on their interview performance rather than their resumes. Since we started, we’ve amassed data from thousands of technical interviews, and in this blog, we routinely share some of the surprising stuff we’ve learned. In this post, I’ll talk about what happened when we built real-time voice masking to investigate the magnitude of bias against women in technical interviews. In short, we made men sound like women and women sound like men and looked at how that affected their interview performance. We also looked at what happened when women did poorly in interviews, how drastically that differed from men’s behavior, and why that difference matters for the thorny issue of the gender gap in tech.

One of the big motivators to think about voice masking was the increasingly uncomfortable disparity in interview performance on the platform between men and women¹. At that time, we had amassed over a thousand interviews with enough data to do some comparisons and were surprised to discover that women really were doing worse. Specifically, men were getting advanced to the next round 1.4 times more often than women. Interviewee technical score wasn’t faring that well either — men on the platform had an average technical score of 3 out of 4, as compared to a 2.5 out of 4 for women.

Despite these numbers, it was really difficult for me to believe that women were just somehow worse at computers, so when some of our customers asked us to build voice masking to see if that would make a difference in the conversion rates of female candidates, we didn’t need much convincing.

They ran this experiment on 234 interviews, of which roughly two-thirds were male. Et voilà:

After running the experiment, we ended up with some rather surprising results. Contrary to what we expected (and probably contrary to what you expected as well!), masking gender had no effect on interview performance with respect to any of the scoring criteria (would advance to next round, technical ability, problem solving ability). If anything, we started to notice some trends in the opposite direction of what we expected: for technical ability, it appeared that men who were modulated to sound like women did a bit better than unmodulated men and that women who were modulated to sound like men did a bit worse than unmodulated women. Though these trends weren’t statistically significant, I am mentioning them because they were unexpected and definitely something to watch for as we collect more data. 

Women were leaving interviewing.io at a rate seven times that of men after a poor performance, and an overall retention curve that looks like this (blue is male, red is female):

 Which is to say, women get frustrated easier and quit earlier. What does this mean overall for women in STEM fields?

Now, as I said, this is pretty speculative, but it really got me thinking about what these curves might mean in the broader context of women in computer science. How many “attrition events” does one encounter between primary and secondary education and entering a collegiate program in CS and then starting to embark on a career? So, I don’t know, let’s say there are 8 of these events between getting into programming and looking around for a job. If that’s true, then we need 3 times as many women studying computer science than men to get to the same number in our pipelines. Note that that’s 3 times more than men, not 3 times more than there are now. If we think about how many there are now, which, depending on your source, is between 1/3 and a 1/4 of the number of men, to get to pipeline parity, we actually have to increase the number of women studying computer science by an entire order of magnitude.

That's... kind of daunting. So, recapping, not only do women not interview as well as men, they also give up quicker, and thus to make up for the lack of women capable of these feats,  we need ten times as many women as men at the front of the STEM pipeline to meet parity. "When I told the interviewing.io team about the disparity in attrition between genders," she continues, "the resounding response was along the lines of, 'Well, yeah. Just think about dating from a man’s perspective.'" The ideas that women should never have to perform under stress, should be hired for jobs regardless of qualification or experience, idiotic theories that Star Wars posters keep girls away from STEM fields — all these and many more amount to so much post hoc-ery evading the unfortunate reality that, as a population, women lack male resilience. It does appear there are steps available for those wishing to improve this state of affairs, e.g. girls engaged in team sports develop more confidence in themselves subsequently. But even there, the confidence gap cuts off a lot of girls at the knees (emboldening this time mine):

Studies evaluating the impact of the 1972 Title IX legislation, which made it illegal for public schools to spend more on boys’ athletics than on girls’, have found that girls who play team sports are more likely to graduate from college, find a job, and be employed in male-dominated industries. There’s even a direct link between playing sports in high school and earning a bigger salary as an adult. Learning to own victory and survive defeat in sports is apparently good training for owning triumphs and surviving setbacks at work. And yet, despite Title IX, fewer girls than boys participate in athletics, and many who do quit early. According to the Centers for Disease Control and Prevention, girls are still six times as likely as boys to drop off sports teams, with the steepest decline in participation coming during adolescence. This is probably because girls suffer a larger decrease in self-esteem during that time than do boys.

 All of which is to say, I don't see a potential solution at the scale needed to move the needle significantly overall, successes at Harvey Mudd notwithstanding; these results suggest women will just move from one institution to another, rather than expanding the overall pool.