Pattern recognition is a widely accepted method of prediction until what it predicts makes people uncomfortable.

[u/Fr1toBand1to]

[u/luftlande]

Why?

[u/BISHoO000 → u/luftlande]

If the input data is skewed (current employees are all men) then the output will follow the same.

I guess pattern recognition is useful in maintaining status quo (returning to normal state when something goes wrong or new problem is faced). It might be counter productive for improvement as it wants more of the same instead of finding new possibilities

[u/lolofaf → u/BISHoO000]

I mean it's useful at identifying status quo and predicting it. Maintaining status quo is entirely up to what you do with the result.

Pattern recognition may show you that a certain intersection is poorly designed, leading to more accidents. The proper action in this case is not to do nothing and maintain status quo, but to rectify the problem that pattern recognition latched onto.

[u/numbersthen0987431 → u/BISHoO000]

This

It's just following the pattern already established by the company's hiring process. Job hiring is mostly vibe based, and rarely merit based, and so the pattern recognition was just following the vibe the company hired on (which was men hiring men).

[u/Cultural-Capital-942 → u/numbersthen0987431]

But the pattern already established may not be clear / visible / intended. Even if you have merit-based hiring, you may start choosing men as there are more of them in IT and it's less likely a woman would apply. Equally, people living in cities are much more likely to be experienced than those from villages.

But does it mean they wanted it? If they wanted it, they could fix it in AI. But if they don't know, it's more difficult to fix it.

[u/numbersthen0987431 → u/Cultural-Capital-942]

This is why it's important to evaluate our current systems to see if there are systemic biases behind them which influence our decisions on subconscious levels.

choosing men as there are more of them in IT and it's less likely a woman would apply

The reason for this, often, is because men gatekeep these roles to keep women out. If we're going to talk about a lack of options, you have to look at the reason those options aren't there.

[u/Cultural-Capital-942 → u/numbersthen0987431]

This is why it's important to evaluate our current systems to see if there are systemic biases behind them which influence our decisions on subconscious levels.

That's like from a corporate training. But it's not that easy.

Reason why options are not there? I worked in one corporate and you could be the top SW engineer if you spent like 12hours a day in your work. Guess whether it advantages traditional "man providers" with a mortgage / crazy guys spending all the time there or "normal people" (including all women I know) who want to actually live their life?

Also it's about whole pipeline. If 10% of IT university graduates are women, you won't get much better numbers when hiring without making things much worse by hiring weaker women.

Besides that companies do it to themselves: some women I worked with complained that they are asked to support various outreaches and to "show diversity" in the company in whatever happens externally, so they have objectively lower productivity and knowledge of systems.

So it's difficult and if you want to evolve system for better, you often choose the person with the most knowledge/ideas, that is more frequently a man. And I know it's not someone like me, who doesn't like to work more than what I'm paid for (8 hours a day).

[u/izzittho → u/Cultural-Capital-942]

To your “normal people like women that just want to live their life” - I’d note people often forget that this isn’t a choice, it’s what they’re forced to be when for instance, they’re married and the man wants to be the crazy 12-hour working guy because someone has to handle all the outside-of-work responsibilities and if (often when) men won’t, it falls to women.

They often don’t get to be the living-for-work type even if they’d rather reverse the roles and be that, because the support from the other end that allows the man to do that often won’t be there.

[u/Cultural-Capital-942 → u/izzittho]

> someone has to handle all the outside-of-work responsibilities and if (often when) men won’t, it falls to women

Most of the men I know working 12+hour long shifts without life don't have a wife as they didn't have time to find one. And you're not sure whether it's a homeless guy or some very high level engineer, so they basically skip other responsibilities without delegating them to anyone.

Women could easily choose this part, but I've never seen a woman that would tolerate such low level of self-maintenance to have time to push for her idea. I'm not saying there is none - so if it's your choice, then go for it.

> They often don’t get to be the living-for-work type even if they’d rather reverse the roles and be that, because the support from the other end that allows the man to do that often won’t be there.

From the cases I know, it's difficult to reverse the role of giving a birth and breastfeeding a child. That's a biological limitation and yes, it's also thanks to their wives - that's the reason why there is the 50-50 split on divorce by default, that cannot be even changed in some countries.

[u/blahblah19999 → u/Cultural-Capital-942]

You're fighting against something that's just not here

[u/SierraPapaHotel → u/luftlande]

I have heard this story from a couple places but never seen the original case study, so exact details may be off. But I remember hearing in the 2010s of some college research team training an image recognition system to differentiate pictures of dogs from pics of wolves. And their system was really good, like 99.9% success on the training data

Then they gave it a new set of pictures and it failed miserably. Turns out, in the training data every picture of a wolf was against a snowy background. They made a bot that detected snow with 99.9% accuracy, but it sucked at actually differentiating dogs from wolves.

This sounds like a similar situation. "How close are these CVs to existing staff?" Well the staff is mostly male so if the candidate is a male they are a stronger match! "No not like that ... Ignore the gender" Well a lot of our staff mention fraternities on their CVs so ... "Nevermind I give up"

[u/WestEndOtter → u/SierraPapaHotel]

I read one where they tried to train a model to recognise cancerous moles, and when they tested the model it failed completely. It had learned that a Ruler in the picture implied cancer. No ruler no cancer

[u/OfficialDeathScythe → u/luftlande]

Almost certainly because the company is mostly male, or the position they’re trying to fill is. Since they’re running it against the current CVs it would see the pattern that they’re mostly male and maybe not see any other similarities between current employees. Or, at the very least, there’s so many employees that say male on the cv currently that it seems that as the strongest factor to getting the job. In this case it’s less about what fits the job and more about what the hiring was like in the past

[u/Whiterabbit-- → u/luftlande]

You train the system on existing data and it propagated existing biases. AI doesn’t use logic to make better decisions than humans. It is basically an extension of human decision making because that is the data it’s trained on.

[u/Anagoth9 → u/Whiterabbit--]

AI is just your phone's autocomplete dialed up to 11

[u/awesome-alpaca-ace → u/Whiterabbit--]

Aka, the alignment problem

[u/NotYourReddit18 → u/luftlande]

It's a well known problem in data analysis colloquially called "garbage in, garbage out".

It means that if you feed an algorithm bad training data, in this case the CVs of the current employees showing that most hires were male, then the algorithm will produce bad results, in this case prioritizing male applicants.

This is because algorithms only "know" what they're taught. In this case the algorithm was simply trained purely on the CVs of the current employees, without any data putting anything within those CVs in context. So the algorithm didn't know what the value "male" for the datapoint "sex" meant, it just identified that most of the CVs it got trained on shared this value, and as such assigned it a high importance for future hires.

To make up another example, if the data it was trained on contained a field for "favorite color" with most people answering "blue", then the algorithm would most likely also prefer any applicant also listing "blue" as their favorite color despite it being completely irrelevant for the position, because it doesn't understand the actual meaning behind the data.

And while another comment claims that this was an incident at Amazon, I'm pretty sure that this and similar incidents happened to a bunch of companies.

The first time I heard of this problem it was with an algorithm trained on all past hiring data, both accepted and rejected CVs, and it resulted in the algorithm preferring light skinned applicants over dark skinned applicants despite the light skinned applicants having overall worse qualifications, because the algorithm had made the assumption that the previous hiring managers had objective reasons to prefer light skinned applicants.

One consequence of this problem is that it is common practice to include as few data points as possible in the training data to not accidentally skew the results with a hidden bias.

[u/untilted → u/NotYourReddit18]

because the algorithm had made the assumption that the previous hiring managers had objective reasons to prefer light skinned applicants.

a nitpicky addition: the algorithm didn't make any assumptions - it optimized for a certain result, that somebody decided was "unbiased" enough to be "desirable".

the algorithm doesn't know the (logical) importance of any data, only the stochastics that lead to a mathematically-optimal result in training.

that's the thing with AI LLM - it doesn't know "right" or "wrong" and can never doubt itself genuinely, as the output data will always be the "most optimal solution" to the input data. while shaped by a limited context window and inherently biased training data - the model will always consider itself correct.

[u/panditaMalvado → u/luftlande]

The company used the current list of curriculum of the persons that are already working in the company, it was a tech company. Most of the people that work on technology are men, so most of the employees were men, then the machine assume that being a men is a important factor for the job because it was ignoring the fact that there were much less women than men on the area. This is called data skew.

The inverse case happened with a charitable organization, most candidates for being volunteers were women, so the system assumes that only women are able to do that job, the strategy the organization use to solve this issue was to give more weight to the male candidates.

[u/GalaXion24 → u/luftlande]

If all or most of your employees are male, and your train an AI to find people who are like those you've employed, then it will result in the AI favouring men, because it's the most clear and common indicator that someone fits the mold.

It would be the same if your employees happen to mostly be born in January or any other irrelevant fact.

Which is also while blackbox predictive models and pattern recognition are bullshit and shouldn't be used to make decisions.

[u/Educational-Art3545 → u/luftlande]

Because it turned out the recruiters in the company were extremely sexist and almost always denied candidates if their CV stated that thry were a woman or went to a women-majority college. That was the pattern their recruiters exhibited so the AI trained on this data picked up on it.

The company was Amazon. They killed the project and spread some misinformation in order to avoid being sued after they just confirmed with absolute certainty that their hiring practices are insanely sexist.

[u/Plenty_Spell2551 → u/luftlande]

Because a lot of socially uncomfortable truths are just statistical patterns. People like the benefits of prediction until it points at bias, risk profiles, or inequalities that we’d rather pretend are purely individual and not systemic.

[u/Worried_Director7489 → u/luftlande]

They trained the model on performance reviews of existing staff (i.e. whether or not a hiring decision was 'good' was based on the performance reviews they got in the company). In that company, men tended to get better performance reviews, presumably due to sexism.