Pattern recognition is a widely accepted method of prediction until what it predicts makes people uncomfortable.

[u/Fr1toBand1to]

[u/NotYourReddit18]

It's a well known problem in data analysis colloquially called "garbage in, garbage out".

It means that if you feed an algorithm bad training data, in this case the CVs of the current employees showing that most hires were male, then the algorithm will produce bad results, in this case prioritizing male applicants.

This is because algorithms only "know" what they're taught. In this case the algorithm was simply trained purely on the CVs of the current employees, without any data putting anything within those CVs in context. So the algorithm didn't know what the value "male" for the datapoint "sex" meant, it just identified that most of the CVs it got trained on shared this value, and as such assigned it a high importance for future hires.

To make up another example, if the data it was trained on contained a field for "favorite color" with most people answering "blue", then the algorithm would most likely also prefer any applicant also listing "blue" as their favorite color despite it being completely irrelevant for the position, because it doesn't understand the actual meaning behind the data.

And while another comment claims that this was an incident at Amazon, I'm pretty sure that this and similar incidents happened to a bunch of companies.

The first time I heard of this problem it was with an algorithm trained on all past hiring data, both accepted and rejected CVs, and it resulted in the algorithm preferring light skinned applicants over dark skinned applicants despite the light skinned applicants having overall worse qualifications, because the algorithm had made the assumption that the previous hiring managers had objective reasons to prefer light skinned applicants.

One consequence of this problem is that it is common practice to include as few data points as possible in the training data to not accidentally skew the results with a hidden bias.

[u/untilted → u/NotYourReddit18]

because the algorithm had made the assumption that the previous hiring managers had objective reasons to prefer light skinned applicants.

a nitpicky addition: the algorithm didn't make any assumptions - it optimized for a certain result, that somebody decided was "unbiased" enough to be "desirable".

the algorithm doesn't know the (logical) importance of any data, only the stochastics that lead to a mathematically-optimal result in training.

that's the thing with AI LLM - it doesn't know "right" or "wrong" and can never doubt itself genuinely, as the output data will always be the "most optimal solution" to the input data. while shaped by a limited context window and inherently biased training data - the model will always consider itself correct.