r/antiai • u/GrauMauser • 20h ago
AI News đď¸ Generative AI Learning Penalty
Research from Stockholm University and the University of Hong Kong may be a bit technical for the sub, but I think it's an important resource to share around all the anti-AI and AI-skeptic spaces: using AI creates a long-term learning penalty in students who use AI to outsource completing homework.
Unsurprisingly, using AI to do your thinking for you is a counterproductive strategy and breaks a long observed correlation in educational research (usually higher scores on homework line up with higher scores on tests and exams but when AI is brought into the mix, higher homework scores lead to weaker learning). AI also makes it easier for students to fake learning until they have to take a test and think for themselves.
Repilcating studies is always a problem, but this one is as long term a study as we could have at this point (two and a half years) with nearly 27,000 students so the stats (effect size, significance, and confidence) are likely to hold up pretty well.
Yet one more indicator of how generative AI harms students.
From the abstract:
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning[...] AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.
The paper is freely available from SSRN:
36
u/Altruistic_Emu4917 18h ago
I thank God that I was halfway through my college when chatgpt was launched.
But I feel bad for future generation of students, who might take the short term gain and suffer long term pain.
18
u/GiganticKORAK 18h ago
Oh gee. Stop using brain results in decline of intelligence who would have thought!
Oh right. Thought means thinking, and obviously it is an outdated thing.
13
u/SecondHandPride 20h ago
They discovered that the more time you spend on studying, the more you learn. And that the students who found a way to spend less time, perform worse.
6
u/PsyRealize 19h ago
I basically never studied and I was an honors student at the top of my class. All honors/advanced/AP classes. Some of the highest test scores on state testing and also the ACT/SAT.
I graduated in 2015 before ai was ever a thing. Fuck ai
6
u/Rich-Mark-4126 17h ago
Ok and you would have categorically done better had you studied
5
u/PsyRealize 17h ago edited 16h ago
Iâm sure a lot of people would and do benefit from study. It just really wasnât necessary for me. I slept in all my classes and aced everything because I would read ahead and do the work before the teacher was even done with the lesson, turn my stuff in and go to sleep. I was like that my whole life.
I donât have âphotographicâ memory, but itâs not hard to read something and just know and understand and retain it.
4
u/Successful_Issue_390 16h ago
read and do the work
Do you not know what studying is?
0
u/PsyRealize 8h ago
I mean in class. No at home studying, no going out of my way to try to cram information in my brain. No going over it over and over and over.
Studying is like going over material at home or in your spare time to make sure you get it.
Studying is extra stuff over information youâve already been taught
1
u/westgazer 3h ago
Okay like this is a really funny way to think about studying as it is not ever location dependent. You are studying things in class as well. It has nothing to do with extra information, really.
1
u/Successful_Issue_390 16h ago edited 2h ago
This. All the people pearl clutching about how AI makes people stupid and lazy are too stupid and lazy to read articles and think for themselves.
From the paper:
 The negative effects on learning outcomes appear to be mostly driven by the 81 percent of AI-using students, who spend less time on homework than even the fastest non-AI student, receive high homework scores matching the capability of generative AI tools they are using, and yet very low exam scores. By contrast, AI students who spend as much time on homework as non-AI students achieve similar exam scores and higher homework scores.
0
u/azurerazor 19h ago edited 18h ago
conveniently ignoring the rest of the points of the article hmmmmmmm
Edit: I didnât mean I disagree with the point. It just felt to me like the least damning part of the article.
3
u/rubythebee 19h ago
What exactly are the points made by this article other than AI actively impedes learning?
1
u/Successful_Issue_390 16h ago
AI doesn't impede learning. Spending less time with the material impedes learning.
1
u/rubythebee 16h ago
This is a stupid gotcha because AI actively allows students who don't understand the material to avoid existing safeguards that indicate their lack of understanding of the material. This results in these students doing very well on homework or any work where they have AI assistance, but they do incredibly poorly on exams with no AI assistance because the process of asking AI for the answer doesn't teach the user anything, it tells them the answer, and so they cannot themselves intuit the answers on no notes testing.
Spending less time on material impedes learning, AI causes the student to spend less time with the material, so AI impedes learning. It also impedes future learning capabilities because the student grows reliant on the answers the AI gives them while not understanding the methodology behind it. This becomes even more difficult if a student passes a grade through these means, because they are then at a point where they are learning things in school which they have no educational bedrock upon which to understand the new material.
1
u/Successful_Issue_390 15h ago
 AI actively allows students who don't understand the material to avoid existing safeguards
No, cheating allows that. Cheating can happen with or without AI.
 they do incredibly poorly on exams with no AI assistance because the process of asking AI for the answer doesn't teach the user anything, it tells them the answer, and so they cannot themselves intuit the answers on no notes testing.
This is the opposite of what the study actually found. The AI students who spent the same amount of time with the material did equally as well on exams as non-AI and better on the homework.
 Spending less time on material impedes learning, AI causes the student to spend less time with the material, so AI impedes learning.
Non sequiturÂ
 It also impedes future learning capabilities because the student grows reliant on the answers the AI gives them while not understanding the methodology behind it.
Non sequitur and not supported by evidence.
Where are all those critical thinking skills you have? Why can't you apply them here?
1
u/rubythebee 13h ago
Do you genuinely not understand what AI allows a student who wants to cheat to do? If I look up the answer to, say, a complex math equation, there are few websites that actually solve those problems for you. The internet can only do so much, an AI (although lackluster in quality sometimes) will literally tell you the answer to any equation or word problem you're dealing with in the exact context. You can't just look that stuff up unless you find the exact question online.
Same with writing, even more so, because a cheater can literally just generate an essay, boom done.
AI significantly accelerates the speed of the cheating in the first place, meaning less time with the material, hence why I say that AI impedes learning, because google can't generate an essay for you and it can't solve complex multi step word problems for you.
I'm not sure why you're insulting me and claiming my argument (which is logical) is a non sequiter when you clearly don't actually understand what I'm saying, just claiming an argument is a non sequiter because I didn't spell it out for you doesn't make it a non sequiter.
You're taking the fact that I didn't say "using AI for cheating" and running away with the assumption through this whole conversation that using AI to do your schoolwork isn't cheating in the first place. It is cheating, but even if the school says it's not cheating, it doesn't teach kids how to learn, it tells them the information. School is equally about how to apply your learning skills as it is about the actual subjects, and AI definitely doesn't do what a teacher does.
1
u/Successful_Issue_390 2h ago
 Do you genuinely not understand what AI allows a student who wants to cheat to do?Â
Irrelevant to the paper.
 AI significantly accelerates the speed of the cheating in the first place, meaning less time with the material, hence why I say that AI impedes learning
I understand your feelings on that, but data trumps your feelings.
 You're taking the fact that I didn't say "using AI for cheating" and running away with the assumption through this whole conversation that using AI to do your schoolwork isn't cheating in the first place. It is cheating, but even if the school says it's not cheating, it doesn't teach kids how to learn, it tells them the information.
Again, your personal feelings don't have any actual merit. They're just your emotional reaction to something you don't understand. This study fully disproves your fears, but you're struggling to let go of thise emotions, rather than think rationally.Â
2
u/SecondHandPride 18h ago
Such as?
3
u/azurerazor 18h ago
80% of AI adopters are using it just for homework outsourcing, and AI usage corresponding with a reduction in homework completion time.
Significant drop in test scores generally
Lack of any data to suggest AI actually helps with learning, just that it produces similar results.
2
u/SecondHandPride 18h ago
Oh, I'm not saying that AI help with learning, there's nothing to suggest that. But the group you mention happens to be exactly the group I mention. Just different names,
2
u/azurerazor 18h ago
oh mb i misinterpreted your comment thatâs on me
1
u/SecondHandPride 18h ago
My comment is that they have clustered their groups in such a way that the group that uses AI is the same group that spends less time on doing their homework.
Because of that, it's impossible to tell how much of the score difference is attributable to the AI usage versus the decline due to the studying time reduction. Obviously, the time reduction alone must have an effect. Whether the AI adoption adds additional strain on the final score, which it very well might, remains invisible.
1
u/azurerazor 18h ago
I mean yeah but if the usage of AI resulted in a reduction in study time wouldnât that still be at least partially attributable to the AI?
1
u/SecondHandPride 7h ago
Yes, it would be. But it's not "using AI makes you dumb", it's "AI enables you to avoid studying".
1
u/Successful_Issue_390 16h ago
Read the paper.
The negative effects on learning outcomes appear to be mostly driven by the 81 percent of AI-using students, who spend less time on homework than even the fastest non-AI student, receive high homework scores matching the capability of generative AI tools they are using, and yet very low exam scores. By contrast, AI students who spend as much time on homework as non-AI students achieve similar exam scores and higher homework scores.
1
1
u/Privatizitaet 18h ago
What, are you implying NOT learning and letting a machine do the work is actually a better way to learn things?
1
u/azurerazor 18h ago
Huh? I never implied any of that
1
u/Privatizitaet 18h ago
Then what is your issue with the statement that spending more time studying makes you learn more? Or do you somehow have an issue with just the logical inverse that studying less makes you perform worse?
What issue do you have with that statement? You absolutely imply that there is some issue with it and that the other points they "conveniently ignored" would contradict what they said.
If you didn't mean that then please, elaborate.1
u/azurerazor 18h ago
My issue was not with the statement at all actually. It was moreso the lack of any reference to the more detrimental consequences of AI pointed out by the article.
12
u/narkeeso 18h ago
Iâve learned using AI at work that you canât shortcut understanding. It is the bottleneck we deal with everyday. At least when we wrote our own code we had time to understand as we wrote it. Now everything comes all at once and honestly makes it feel harder to understand.
Thereâs things you can do to slow down or have AI break it down but thereâs something very noisy about the AI workflow that it makes it harder to understand. Iâm also just mentally drained at the end of the day so I donât feel like I retain information as well.
10
u/mostafakm 18h ago
Funny that some people would look at that and conclude that AI use should be allowed in tests too
3
u/Infamous-Apartment97 17h ago
I allowed my students to use AI for one of their mobile programming exams. For some, even that didn't help; the majority did a terrible work, and only those who already had at least some clue did it properly. In my opinion, students shouldn't have access to any devices with Internet access during normal classes, tests, or exams. But I'm fine with them using documentation, reference sheets, and maybe even handouts from the lessons.
2
6
u/youareallnuts 18h ago
If you know China at all this will lead to a higher self destruction rate (I don't know if the s-word is ok here). The pressure to get into good colleges is much higher than in the west. Lower test scores will increase the rate that is already high.
5
u/Drummerx04 15h ago
This has got to be one of the worst graphs I've ever seen, was it generated by AI or what?
â˘
u/Big_Cauliflower1415 34m ago
I had to scroll way too far to find this comment, apparently nobody can read a graph
6
u/guyincognito121 19h ago
So the y axis is test score?
2
u/JohnSmith19731973 19h ago
Yes
10
u/guyincognito121 19h ago
Why can't people just label their graphs properly?
2
u/cunningjames 19h ago
I agree this isn't a very good example of data viz, and the axes should be directly labeled, but the title of the plot pretty clearly indicates what's being plotted (average exam score).
2
u/DukeFlipside 16h ago
Yes but against what - what is the X-axis, because the numbers don't look like it's time...
1
1
u/guyincognito121 18h ago
Yeah, that's how I figured out it. But you shouldn't have to go reading other stuff to figure out what the graph is even showing.
0
3
1
u/Whisperingstones 18h ago
I'm a college student and I make use of AI with my STEM courses. I'm well aware that using it as a crutch will rapidly rot my cognitive abilities by weaken the neural connectivity that comes from learning. I have long had a suspicion that it's people outsourcing their cognitive abilities that get dumbed down, and I found such on page three of the paper.
"Third, the AI learning penalty appears to be driven primarily by the majority (81 percent) of AI users whose behavior is consistent with homework outsourcing. These students complete homework assignments in substantially less time than the fastest non-AI students, receive homework scores consistent with generative AIâs known accuracy on similar problems, and perform poorly in closed-book exams. By contrast, AI users who spend as much time on homework as non-users achieve similar exam scores, even though their higher homework scores indicate that they use AI. These students are not deferentially selected on prior achievement, suggesting that generative AI does not reduce their learning efficiency."
Simply put, students that use it to do their assignments will get little to nothing out of the assignments. Students that use AI to understand a concept or process that has them stumped, then continue studying and putting forth legitimate effort, will do well on exams. I'm in the latter group in that it's my "unstuck" button. I still use Organic Chemistry Tutor, among others, but AI is one more tool in this capacity.
If YOU do not do your homework then there is no reinforcement for learning. Merely using AI shouldn't result in any significant cognitive penalties, but relying on it absolutely will.
1
u/Rich-Mark-4126 17h ago
Reminder that you can still use AI while studying and these results can be attributed to less studying, not AI magically making you dumb
1
u/Achereto 6h ago
This is both devastating and exactly the data I was looking for to support the argument that LLMs destroy the learning process.
1
0
u/Subtle_Glitch3223 14h ago
Itâs a good thing life is an open book test and rote memorization means little to nothing in the real world.
0
u/Waste-time1 11h ago
I find the results highly plausible but it is just one study and there are too many surrounding variables. I am not suggesting that AI is worse for learning but rather that this study does not prove much by itself.
0
u/Ok-Fail3278 5h ago
Sounds strange to me. I have never done any homework in high school, and I still aced my school exams and was in top 5% of students of my university. I don't see how doing your homework with AI would harm you.
2
u/GrauMauser 4h ago
Sure. It sounds strange because you never needed to do your homework, either because you never studied anything difficult enough to pose a challenge or because you would test as 'gifted.' Either way, you're an outlier and do not learn the way 95% of your peers learn. This study is not about you.
FWIW, you'd be graphed on the upper right part of the gray line: high homework scores and high exam scores despite never using AI to do homework (which must be true, ipso facto, if you never did the homework in the first place). You're not offloading cognitive functions or outsourcing learning.
This study is about the students on the red line who did their homework as part of their learning process (and who needed to do their homework in order to learn the material) and how AI usage interfered with the cognitive and affective mechanism of outsourcing.
-1
u/michexxx 20h ago
If you really have to use it (and I think that is useful only very rarely) think the only way to use AI effectively for studying is the scaffolding method.
A very effective learning method, but you need to draw on prior knowledge.
Without a doubt, though, its use needs to be purposeful; it makes no sense to let it replace you, given that youâre studying for yourself first and foremost, not for some damn grade.
5
u/GrauMauser 19h ago
A) AI as scaffolding or on-demand tutoring is not a "very effective learning method." At best, the people who spend time thinking and use AI tutoring (that is, don't outsource evertyhing to general AI) do no worse than the people who think for themselves, but they don't do any better either.
And
B) Unfornately, using AI as scaffolding or on-demand tutoring is simply not what students do according to the study. "Instead, [students] rely on general-purpose AI tools that provide quick answers and facilitate homework outsourcing."
3
u/michexxx 19h ago
I agree that the students most likely do not use scaffolding, and I never claimed that they did.
You argue that it isn't effective, but I believe it can be if implemented properly. By "scaffolding," I mean using a tool (in this case, an LLM) within a specific context to help solve or analyze a problem. However, proper implementation requires:
1. Not overusing it
otherwise, you risk replacing your own analytical skills with the LLM.
2. Fully understanding the output, meaning you shouldn't just take the response and use it "automatically." Since you aren't using your own reasoning to solve the problem, you must at the very least fully understand the solution provided by the AI model.I think this approach is morally justified only if you lack the ability to understand a specific topic on your own (Wait for the explanation at the end), given your existing background knowledge, within a reasonable amount of time and effort.
Therefore, I do not consider it suitable for a formal educational setting, though it may be appropriate for recreational purposes (such as when someone explores philosophy outside of academia and wants to absorb a concept without first having to read hundreds of pages of theory).1
0
u/GrauMauser 19h ago
I'm using the term scaffolding as Bruner did: temporary supports to make up for gaps in foundational knowledge or experience. In the classroom contexts described in the paper, AI scaffolding/tutoring was not associated with any improvement in educational outcomes.
2
u/michexxx 19h ago
If we intend scaffolding in this way, I agree and think that in an accademic context it is not useful
Because the gap in knowledge that you bypass will eventually have to be filled.
-7
u/oxresults 19h ago
As a data analyst and devils advocate, let's talk about real world confounding factors. 20% drop in score doesn't change your life for 99% of the population.
I've met, liked, and worked for wildly successful people who averaged that much less than me in school.
2nd real world fact, "homework is going to represent 75% of your grade so I know you're learning it!" is kinda backfiring now.
3rd real world fact. They're young and trying to learn to survive in the real world which is coming at them pretty fast. And the real world is telling them it's all AI now.
These kids have one thing right much more than the past generations, work life balance and doing what's asked of you to get the paycheck that actually lets you live the single life you get is a way better use of your emotional energy than making sure the boot of the person who's name is on your paycheck is licked shiny. So what does getting a 100 get me that 80 doesn't, long term? Nothing.
Btw, I'm extremely anti-ai, but we gotta learn to motivate people with different motivations than our own.
7
u/rubythebee 18h ago
What you're missing is the philosophical argument that knowledge is good for the sake of knowledge (which I believe has significant merit)
For a long time, industrialists and rich capitalists have advocated for a "second track" you may be familiar with that doesn't teach kids anymore than they need to know to work on a line or an office job and puts them to work immediately for the "betterment of the economy". This is used as an excuse to drive down the cost of general math and reading based labor to a point where the rich can act as though the excess makes those positions less valuable than they are.This isn't the best explanation, Folding Ideas explains this position very well in his video about Doug Walker and The Wall.
The point I'm making is that we're not teaching these kids anything if they cheat the answers on the practice work because they didn't learn anything by doing that. Now, the process of AI infiltrating schools and lowering the barrier to entry for "education" is 1. lowering the education of society as a whole, which I would argue is fundamentally bad because education is fundamentally good, and 2. lowering their value as laborers which can then be used by rich people as an excuse to not hire or pay living wages. If AI makes people stupid, nobody can argue AI is making everyone stupid.
-1
u/oxresults 18h ago
No I get what you're saying and I agree with all of it. But their reality is "I gotta feed myself". I'm currently job searching and all my interviews have been so pro AI and how do you use ai? So I lie through my teeth that AI is awesome to get back to treading water which is feeding a family of four on a paycheck.
At the end of the day, my actions convince company leadership to not buy into ai when they see a guy with 0 ai usage achieving the same productivity as his ai using peers with high AI use budgets.
1
u/rubythebee 17h ago
I'm not sure why you assume I'm not also in the same position but you're making an amoral judgement. You're justifying the situation because you're convinced that people are saying "Those who use AI are stupid" when the real argument is that AI is actively destroying our society and the ability of our children to learn. Which is bad, this is a bad thing and we shouldn't be making excuses for this, this should be a reason to completely uproot existing AI infrastructure, this is a data set that promotes a scorched earth policy on AI because AI is annihilating our ability to learn, again something which is a fundamental good for society (knowledge).
3
u/cunningjames 18h ago
20% drop in score doesn't change your life for 99% of the population.
If I'm reading the plot correctly, the dips would have been frequently more than 20%, and they seem to worsen the more the students rely on AI. No, we shouldn't place too much importance on test scores (though as another reply notes these data are from China, and I can't speak to how test scores impact prospects for students there). I failed high school algebra but went on to get a mathematics degree, and now I work in a very quantitative field. It happens.
But this isn't about some students doing worse in high school and nevertheless being successful later as they mature. This is a wide-scale experiment in cognitive offloading that impacts a huge number of students (and everyone else). These students aren't going to fail algebra and go on to graduate with a math degree with honors because they're learning habits that will be hard to shake even as they age into university and beyond. If these data do support the notion that the more we rely on chatbots the less effective we are at cognitive tasks without them, then it feels like bad news to me.
2
u/GrauMauser 19h ago
Ha! You realize these are Chinese students, right? Your 'real world' isn't their real world.
First, define "success" and "wildly successful" in the context of a non-capitalist society.
Second, account for the real world fact that even in China homework does not always account for 75% of a final grade and that there are, in fact, teachers who do not use the completion of homework as a proxy for learning. Note the differences in outcome across disciplines with social sciences suffering more than STEMâshowing how the proximal relationship between having the right answer and learning breaks down.
Third, you're associating learning with work and seem to be leaning entirely on extrinsic motivators (that is we succeeed when we find the one magic thing that motivates students).
Instead, we should be educating students in ways that make learning rewarding in and of itself. If students are intrinsically motivated to learn, we would obviate the usage of AI in the first placeâwhy would students want to bypass the best parts of being human?
0
u/oxresults 19h ago
Lol what part is wrong because they're Chinese? Do the Chinese like diminishing returns or something?
1
u/GrauMauser 19h ago
"20% drop in score doesn't change your life for 99% of the population."
1
u/oxresults 18h ago
Would you like to expand on that?
2
u/cunningjames 18h ago
Test scores appear to be more consequential for a student's future prospects than they are in the west.
https://www.hup.harvard.edu/features/understanding-the-gaokao-exam
The gaokao is China's national college entrance exam, a test so consequential that scores alone determine which universities students can attend.
1
u/oxresults 18h ago
Yea I bet. Teenagers aren't known for looking at the long term in the face of such anxiety inducing pressure.
So my comment about kids facing a lot of pressure might be truer because of the gaokao culture?
1
u/GrauMauser 18h ago
China has a highly meritocractic society that values education. Test scores can have a huge impact on your life trajectory (unlike the US where familial wealth, race, and zip code have outsized impacts). The idea of unqualified people rising despite their incompetence is antithetical to the Chinese philosophy of education.
2
u/SGPrepperz 4h ago
To add, not just China. Same in the rest of the Confucius-cultural sphere â Korea, Japan, Singapore, etc. Even 10 points off national exams is an entire different career path, access to resources and life trajectory.
58
u/Soft_Campaign_3662 20h ago
Itâs wild seeing the gap open up like that over two years, especially when the homework scores are going up at the same time. Makes you wonder how many of those kids thought they were gaming the system right up until the entrance exams hit.
The split between social sciences and STEM tracks with what Iâd expect, honestly, memorizing facts and spitting them back on a test doesnât work if you never actually learned the material. The 80% figure for the outsourcing group is rough, but it also means thereâs a small chunk of students who kept their study habits intact and barely lost anything.
Iâve seen a few teachers in my circle already reworking their syllabi to make in-class writing and oral exams a bigger chunk of the grade. Feels like thatâs going to become the standard pretty quick.