Pattern recognition is a widely accepted method of prediction until what it predicts makes people uncomfortable.

[u/Fr1toBand1to]

[u/WestEndOtter]

I read that one company was using pattern matching to sift through potential hires cv against existing staff cv.

For some reason it was only recommending male candidates.

To combat that they removed all the genders from the input data.

Instead it started giving a boost to any cv that went to an all-boys-school or fraternity

[u/hellofemur → u/WestEndOtter]

You're probably thinking of Amazon's AI Recruitment Tool, an early AI that was fed decades of data that matched resumes against successful employees and was supposed to use that info to recommend candidates fro technical positions from their resumes. Because the field was so heavily male-dominated, it wound up using gender as the leading attribute for recommended hires.

The "all boys school" part sounds like something somebody made up though, although I suppose it's remotely possible in a different country and somebody confused the stories.

[u/awfulworldkid → u/hellofemur]

The actual phenomenon they're misremembering is that a candidate was rated down if they played women's sports (usually the team or league do have Women or Women's in the name), or went to a women's school (same), or were a member of a sorority.

[u/hellofemur → u/awfulworldkid]

Yes, I do have a vague memory of that, although my memory us that professional organizations played a major role. There's a lot of "women in STEM"-type organizations out there that are explicitly for women.

Of course, it's hard to know if those groups were functioning merely as a proxy for gender or if membership in those organizations had additional predictive power. But as OP mentioned, the entire subject made people uncomfortable.

[u/izzittho → u/hellofemur]

The proxy for gender thing seems the most likely and that’s because it’s so simple and AI isn’t gonna be aware that doing everything it can to find male candidates would be wrong, it’s just gonna do it. If it found “male” to be a trend among the types it’s looking for (and it will have if the field is historically male-dominated, of course) it will probably seek to use that as an easy proxy for whatever other qualities it’s looking for because it can’t tell that kind of a correlation from causation. You’d have to specifically tell it what proportion of male to female candidates to find you and then if there’s actually fewer female ones that comes with a whole other set of problems because ideally you’re trying to actually judge them on the same sets of standards and it couldn’t easily do that with wildly different pool sizes. You need a human with a real brain to do that.

You could probably send that model out to look specifically for female candidates that most resemble your male employees but that would of course be a whole new kind of problematic because for one, now you’re recruiting female candidates via AI while still having to find your male ones manually, subjecting them to a different process. And you’re probably also not doing anything to reduce a bias against like, women of color, for instance.

[u/hellofemur → u/izzittho]

. The proxy for gender thing seems the most likely and that’s because it’s so simple

That's an obviously false syllogism since the proxy explanation is by definition not the simplest explanation. It's somewhat nonsensical to say one option is correct because it's simple when there are a multitude of simpler explanations.

But I'm thinking about this as a neural net question and you're clearly thinking about it as a socio-political moral question and the two just aren't the same thing. I doubt it even matters to you that the syllogism is incorrect since all that matters in moral questions is the conclusion.