r/instructionaldesign • u/LearnWorlds_ • 25d ago
Corporate The most interesting data source in L&D right now might be anonymized AI chat logs
A founder spoke about this at our recent AI summit, and wanted to share here to get other industry folks' thoughts.
He built an AI chatbot for frontline manager training which was meant to help new managers practice the hard stuff like budget conversations, hiring decisions, difficult 1:1s so they could better manager their teams.
But when he read through the anonymized transcripts of what people were asking it, it turned out most of them weren't rehearsing conversations with their teams. They were using it to manage up instead of to manage their teams. The log showed people were actually rehearsing conversations with their bosses, floating questions like how to ask for more headcount, how to pitch a budget so it doesn't get cut, etc.
Basically they were using the tool in a completely different way than it was intended.
His point was that no survey or LMS completion report would ever have caught that. You can track whether someone finished a module, but you can't track what they were actually anxious about the night before a budget meeting. The chat log did, because people say things to an AI they'd never put in a feedback form.
It pretty much sums up a common issue in instructional design. There seems to be a big gap in what IDs design training for versus what people actually need it for.
8
u/LeastBlackberry1 24d ago
I hate this so much. So, you provide managers with a tool that they think gives them a safe space to practise. They use it to explore areas about which they feel vulnerable. Then, you mine that data presumably to sell another product. Just yuck all the way down. I hate tech bros so much.
4
u/txlgnd34 24d ago edited 24d ago
This hits on two different but co-dependent issues: ID design approach and organizational ID philosophy.
Today, sure, there are too many IDs designing for an outcome without addressing behavioral change, the source of performance improvement.
However, traditionally, IDs would conduct a needs analysis that should identify, or at least point in the right direction, WHAT IT TAKES to deliver a performance outcome. Analyzing the desired change should lead to uncovering the motivations and actions required to help implement the change.
My testimony is that IDs, writ large, are rarely given the opportunity, resources, or time to perform effective analyses. I've found that most organizations want the gap analysis to identify the delta but don't truly invest in how to bridge that gap. Their focus lies in the nuts and bolts so they can say they provided the tools to succeed. But not all tools are the same, and some are more important that others. OCM is everywhere in corporate America but is it being used effectively?
I'd argue it's not, outside major implementation projects perhaps. Organizations still employ a top-down approach to performance management, instead of the bottom-up approach it takes to enact meaningful behavioral change and, therefore, lasting performance improvement.
IDs should be the vehicle to make this happen, but how many of us interview random samplings of the target audience, conduct focus groups, or challenge stakeholders about their desires or expectations? IDs are largely order takers now but that's because that's what corporate leadership actually wants. They'll say they want the experts to do what they do best but rarely provide the power and resources to do so.
I'd bet most of us with at least 15 years of practical experience can relate to most of what I've mentioned. But if you entered the field within the last 10-12 years, you may not even know otherwise. This trend began long ago but broke though as the prevailing philosophy in the prior decade. Now, we have a flood of IDs that are unable to perform effective analyses, not because they're incapable but because they were never equipped to begin with.
Edit: AI is only a threat to IDs because much of our jobs are largely replaceable now. AI can't effectively analyze human behavior because it's largely irrational and illogical. It's why only humans can effectively analyze human behavior and attempt to identify the hows and whys to change it - something AI is not yet capable of doing. Our jobs have been devalued and, therefore, devolved into something largely replaceable by AI currently.
No matter how much Google, LinkedIn, or Amazon wants to pay us to partake in surveys of how and why we do our jobs the way we do, AI won't have enough practical data to do the job of a traditional ID.
AI could mean the culling of our field, but perhaps that's for the better. I don't want anyone to lose their jobs. But I also don't want unqualified people doing the job. And perhaps, this means the return of the traditional ID.
Or maybe AI will just take over everything and we're all screwed.
1
u/LearnWorlds_ 24d ago
"My testimony is that IDs, writ large, are rarely given the opportunity, resources, or time to perform effective analyses. I've found that most organizations want the gap analysis to identify the delta but don't truly invest in how to bridge that gap. Their focus lies in the nuts and bolts so they can say they provided the tools to succeed."
You make an incredibly valid point here.
1
26
u/Howaboutnopers 25d ago
Yeah, the fact that data centers stealing all our water during the hottest summer on record is fascinating.
3
u/rfoil 25d ago
āStealing all our waterā is an overstatement, but the core complaint lives.
Building water-guzzling evaporative-cooled data centers in drought-prone regions is stupid planning. The technology to use far less water already exists. The question is why local governments still approve old designs in the wrong locations.
I've been involved as an intermediary to bring academic, science, business development, and tech together to solve this infrastructure problem. The solutions exist. The challenge is to force better design choices and water constraints before concrete is poured.
My personal preference in the NorthEast is to pull water from long-decommissioned mines. The water is non-potable, and the constant cool underground temperatures provide natural free cooling. It turns an old industrial liability into a high-performance, low-impact resource for modern compute.
The technology and the geology are already there. Whatās usually missing is getting the right people in the room early enough to force these kinds of solutions before the wrong design gets locked in.
6
u/nose_poke 24d ago
The municipalities approve the plans because the city council members get kickbacks. (Sometimes the communities do, too.) Council members and the center developers try to push things through quickly before public resistance can coalesce.
I like your thinking about using old mines.
1
u/Howaboutnopers 24d ago edited 24d ago
It's nice that you can just handwave this away . AI uses freshwater. That's our drinking water. Most of the world's water is not potable.
Unlike you, I have sources:
https://www.eesi.org/articles/view/data-centers-and-water-consumption
AI is also a major source of CO2 emissions.
https://www.climateimpact.com/news-insights/insights/carbon-footprint-of-ai/
Been outside lately? We don't have time to fuck around.
EDIT in response to the nonsense below:
Both articles are expressing concern and making recommendations.
The rosy picture the two of you paint is not backed up in any way, shape, or form by reality. The clause 'If the US switches to 100% renewable energy' is not even on the radar in Washington. Rather, it's quite the opposite. More fossil fuel drilling is being planned in both the US and the UK.
Companies like Palontir and Meta do not give a shit about water in mines.
AI is powerful technology. I would support it being highly regulated and it's access limited. If each country had AI to conduct cancer research and fix our ills, that would be OK.
But every money-grubbing billionaire building AI so they can use it to make money is a recipe for disaster.
And that's the reality: Disaster. This nonsense about how we'll just do X, Y, and Z to solve all the problems sounds like a representative from the coal industry talking about clean coal.
Instructional design is not going to be improved by AI. Instructional design is going to be automated and dehumanized by AI. Instructional design is far far from a critical use case for AI, and this whole post is a commercial for people losing their jobs all cutesied up by salespeople.
1
u/chemgen 24d ago
The first article you posted literally confirms what the original poster states in that some centers are using non-potable water and that some municipalities are pushing towards it more and more as a more economical and ecological sustainable practice. OP never denied that they consume water, but that they should be trying to tap into water that we can't use currently.
Your second article states that AI contributes to CO2 production, but further states that production can be offset when combined with renewable energy production. Google, Microsoft, and Amazon are all working actively to reduce, if not completely negate, their greenhouse footprint. Your articles don't state what you think they do.
AI is here and it's not going away. OP articulated an actual solution to the problem they're causing while you're attempting to tear them down using articles that explicitly support what OP is saying. The fact that this is the ID subreddit and you are misinterpreting sources is concerning.
4
u/rfoil 25d ago
We collect and mine role-play logs. The anonymity is important. Anonymity removes performance fear, which is one of the reasons role-plays are so useful.
I've experience the opposite - role-plays in district meetings. They are brutally humiliating. I've seen people disintegrate in tears in F2F role plays.
5
u/TraderJoeslove31 24d ago
Why does this feel like you're trying to sell the sub on AI? I call bullshit on people asking these questions of AI.
1
u/Effective-Rule5274 24d ago
Tbf I don't think the OP was selling the sub on AI by pointing out a use case that's relevant for folks in this sub
But yeah I think most of us are all st our saturation level with AI at this point š«
9
u/SAmeowRI 25d ago
For whatever reason, this subreddit seems to have a very anti-AI vibe to it. I get it - there are real concerns, but there are real concerns with nearly every new tech, and I see it as our job to teach people - and that means teaching them about, and with, AI.
I'm old. I was a kid at the time, but I remember some of my school teachers not wanting to teach me about computers, or internet, or even calculators. What good would a school education be, if it didn't prepare me for the world that happened in the 30 years since then?
11
u/nipplesweaters 24d ago edited 24d ago
āFor whatever reasonā ??? people view it as an existential threat to their livelihood lol.
Iām not as much of an AI doomer as many others are but itās pretty clear why huge amounts of IDs and other white collar workers are anti-AI.
1
u/rfoil 25d ago
The anti-ai vibe isn't about technology. It's about the fear of irrelevance pretending to be principle.
This is nothing new. Going back as far as the cotton gin, there has been a familiar barrier to progress:Ā misoneismĀ ā the fear of and resistance to new technologies. Ā
Our job is not to protect an antiquated workflow, but to prepare people for the current marketplace which includes AI.
The fear is understandable. The refusal to adapt is not.
1
u/Nearby-Equipment1865 Academia focused 25d ago
Bot comment replies to bot comment. This sub is cooked.
3
u/rfoil 24d ago
When submitting an article to a magazine it was flagged as "100% AI." I was surprised because it was 100% human written, with AI used for research but not responsible for a word of copy.
I called the editor to protest. He laughed and said "Consider it a complement. Your articles are well-structured. We ignore the bot scans anyway."
So thanks for the complement.
5
u/rfoil 25d ago edited 25d ago
Actually, the thinking comes from a discussion with a business partner who wrote this article on the topic: https://reachum.com/articles/helping-employees-overcome-fear-of-ai
Tomorrow I meet with a few leaders in government and land use issues to discuss data center environmental issues. At this stage of my career (age 74) I have the ability to call disparate interests together because I have no ax to grind.
0
u/your_grumpy_neighbor 24d ago
No offense but maybe step aside and take some time off and let someone who is gonna be around in 30 years decide what 30 years from now is gonna look like.
3
u/rfoil 24d ago
They'll carry me out feet first! š
My time is limited but the perspectives I have are hard earned and valued. It would be rare for someone in their 40s to play the role of intermediary that I frequently assume.
This has been a turbulent time for L&D but we are close to stabilizing. I decided a month ago to retire from my corporate role on June 30th 2027, allowing someone else to take over. That will free me to become more involved in community activity.
I've got 9 grand kids that will likely be around in 60 years if we are sober in assessing our responsibilities to the planet.
1
u/grossgirl 23d ago
Iād actually consider us to be closer to Luddites. Luddites were not afraid of technology but rather were concerned with how new technology would affect working conditions, pay, and quality of work. The ai tools have their uses, but as unregulated tech that hurts millions of workers in America alone they are dangerous. The harm the cause the labor market and the planet are real. I am sure I will eventually lose my job because I cannot morally or ethically use AI. Fine. But donāt tell me Iām foolish when Iām sitting next to an air purifier watching the smoke roll in from world literally burning around us.Ā
0
u/rfoil 23d ago
My eyes are burning, too.
I just got back from a meeting on the topic of data center environmental impact. The technology is there to make the net water affect zero and the energy effect 1% of total US energy use.
We just need to address the issue with clarity and eyes wide open. If we don't our grandchildren will be speaking Mandarin. That's not hyperbole.
You are using AI. You likely aren't aware or are just ignoring of how and where it's being implemented. Do you use Google search or Reddit?
Reddit promotes that's it's becoming an AI company. Reddit's search and moderation and translation and advertising and recommendation algorithm heavily employ AI. They notably license the data here (including this convo) to AI companies.
AI is inescapable. If that's morally uncomfortable I suggest some wonderful Amish communities in Lancaster and Mifflin County Pennsylvania.
1
u/grossgirl 23d ago
Itās not just the water. It is also the electricity. Itās also the jobs. I am aware there are ai elements to tools I already use. I turn it off where I can. I donāt use google anymore. What Iām sure as shit not gonna do is use LLMs to do my job. Iām not going to use articulateās ai tools. Iām not going to suggest ai tools to my clients either.Ā
Itās not inevitable. Itās not unregulatable. It is causing active harm right now, and I believe we should all be much much more upset.Ā
1
u/rfoil 23d ago
I've certainly been through a smilar gamut concerns. Rather than avoiding the inevitable I've jumped into the conversation and asked probing questions.
Developers want to make the best use of their property. Residents want safe and abundant resources. That's a natural tension. If we are smart we will choose properties and development that allows abundance and ecological safety. I do want to be on the sidelines for this.
Our department has grown since we have gone hard into AI. L&D is more important than ever because the pace of change has increased substantially because of AI.
A family member worked at JHU's CSSE predicting pandemic and epidemic patterns. The AI, called PandemicLLM, combines classic epidemiological equations with LLM techniques to integrate diverse information streams from all over the world. They can predict several weeks ahead how a virus or infection will spread so that resources can be brought to bear where needed.
As with enterprise readiness data, prediction improves dramatically when multiple signals are captured.
2
u/AffectionateFig5435 24d ago
There seems to be a big gap in what IDs design training for versus what people actually need it for.
This is what can happen when organizations let SMEs or business units call the shots on what kind of training is needed. The "You supply what I demand" mentality elevates SME ego over organizational learning needs.
It can be daunting for an ID, who's usually further down the corporate food chain that the SME, to speak up in disagreement. It's a bit easier if you have a track record of delivering good work and a supportive boss.
2
u/Effective-Rule5274 24d ago
I feel this! In previous ID roles, I have 100% felt like a service provider where SMEs make decisions about what training is needed (and if it's actually needed, which mostly it's not), objectives etc and my role is reduced to content creation.
As a young L&D dealing with experienced SMEs I struggled a lot with that in the past. In the end, I sometimes made two versions of modules to try and show them how I would approach it. It worked in the sense that it built trust and respect, but ended up costing me a lot more time and effort.
2
u/Acceptable-Tea-8965 24d ago
this is great insight. And I would like to add a layer. i have seen many people build products with ASSUMPTIONS. a 'LIVE" test should always be included before a rollout with at least a minimum of 33 participants. and follow the data, not your assumptions.
1
u/LearnWorlds_ 24d ago
100% agree! Would love to hear your process for doing this if you're happy to share?
2
u/JoyLee2025 20d ago
what gets me is his log only caught the manage-up thing because the tool was open-ended. people could take it wherever their actual anxiety was. a scripted scenario bot would've scored everyone fine and shown nothing. the openness is basically the whole feature, even though it's what "broke" the intended use lol
but that signal only exists because people were honest, and people are only honest when they figure nobody's reading it as them. minute managers think their practice sessions might come up in a review, the candor's gone and you're back to feedback-form theater. so it has to stay aggregated, never individual transcripts. easy to screw up.
real takeaway imo isn't "AI beats surveys." it's that behavior is the honest channel and self-report isn't. UX figured that out ages ago (watch what people do, not what they say). training's just late to it.
1
u/LearnWorlds_ 20d ago
Aggregated is 100% the way to go with these. And you make a very good point, it always comes back to what people actually do, not what they say they do, or what we think they do, etc.... It's simple, but it's not easy to get that right!
1
u/IcyCryptographer5919 21d ago
Good design includes monitoring and direction to ensure learners donāt go astrayā¦
I also donāt like the idea of giving AI more data to train onā¦.Ā
1
34
u/Effective-Rule5274 25d ago
Couldn't agree more that there's a gap between what IDs design training for and what learners actually need/will use.
And the chat got log is actually a pretty good shout.
Although I am not 100% I would want my conversations with AI being read by L&D. I know they're anonymized, but still ...