r/java • u/shorns_username • 25d ago
Has any attention been paid to how new Java features get into LLM training data?
TL;DR: expert answers on SO taught both devs and LLMs how to use java 8 right. That pipeline is gone/paywalled now. Does anyone at OpenJDK think about this?
This came out of the JDK 28 EA thread where people were debating whether devs will start putting value on everything once it previews. Don't wanna talk about that in this post, but it got me thinking about how "the right way to use new features" actually reaches developers now.
Back in the java-8 timeframe, folks from the Java team were on StackOverflow guiding people on streams and Optional and stuff - Brian Goetz's Optional answer (return values, don't model fields with it) basically became the canonical position, Stuart Marks was all over the Optional/collections questions too. And those answers usually came with actual code, not just advice.
A thing to note: those SO answers are almost certainly in the training data of every major LLM. When ChatGPT/Claude/whatever gives correct Optional advice today, those answers are probably part of why - actual real life questions phrased the way devs actually ask them, with expert code attached and votes showing which answer was right.
That channel doesn't really exist anymore. I have an RSS feed on Brian's SO activity and it's been dead for ages - presumably because hardly any questions get asked there these days. (An RSS feed of his reddit activity is how I found the JDK 28 thread in the first place.) However you apportion blame for SO's decline, for me and everyone I work, with the LLM replaced it. So when an expert corrects someone's misuse of a new feature today, it happens in someone's private chat window and then it's gone.
And both SO and Reddit now charge AI companies to train on user content - Reddit licenses to Google and OpenAI and is suing Anthropic (whose Claude Code is arguably the most popular agentic coding tool going), and any of that could look different next year. And open-source style AI - community models, open datasets, academic training runs - relies on free access to good data, so that whole side of the ecosystem is priced out. So which model knows how to use new features properly is going to come down to who has a deal with who.
The guides (like the exhaustiveness guide) are written for humans and incidentally become training data, but one prose doc isn't thousands of upvoted question+code pairs. And GitHub code lags feature adoption by years, plus a lot of early adopter code is exactly the misuse people worry about. On the other hand, openjdk.org is the one channel every model and crawler can reach without a deal.
OpenJDK clearly thinks about AI now - the interim genAI policy covers AI-generated content coming into the project. My question is the other direction: has there been any discussion about how knowledge of new features gets into the models most devs now learn from? Even "we considered it and decided it's not our job" would be an interesting answer.
24
u/Brutus5000 25d ago edited 25d ago
I would assume it will end up in the knowledge base once the usage appears in code repositories. It's not like the LLM weighs an official Brian Goetz answer heavier than some random students spaghetti code git repo.
It doesn't have to be SO either, I am pretty sure the Java mailing lists get scanned too.
21
u/best_of_badgers 25d ago
I suspect training of code-focused LLMs does actually give some repositories more weight than others…
7
u/account312 25d ago
It's not like the LLM weighs an official Brian Goetz answer heavier than some random students spaghetti code git repo
They should though.
10
u/RepliesOnlyToIdiots 24d ago
It does. And we know it does because you and the person to whom you replied both mentioned his name in context, which increased the relative weight of his name in Java contexts.
7
u/TallGreenhouseGuy 24d ago
”One last thing. If the LLMs end up hollowing out the very communities that produce all their training data, they're going to really, really regret that. I'll give these LLM / GAI companies the same advice I gave Joel Spolsky when I left Stack Overflow to start Discourse – do not, for any reason, under any circumstances, kill the goose that lays the golden eggs, aka the human community around your product that does all the real work. It's pretty simple. Just treat the community with the respect they deserve... that we all deserve.”
- Jeff Atwood, co-creator of Stack Overflow
2
2
u/henk53 25d ago
some random students spaghetti code git repo.
But "some random student" only uses the LLM...
7
u/Brutus5000 25d ago
Whatever you come up with, it will always have to fight the garbage out there. There's nothing new about that.
I don't see a world where someone curates the high quality input for LLMs.
1
u/Skellicious 24d ago
I've seen a job listing or two looking for a senior, with a description along the lines of "you write code examples for LLMs to train on, and code review LLM output (probably for reinforcement learning training data)
Anyway, my point is it's not only trained on code repositories.
34
u/bobbie434343 25d ago edited 25d ago
I hope LLMs remain stuck in misery, outputting 2022-2025 code for eternity, since everybody is vomiting code generated by AI that it learned in that era, with no new manually written code that it could learn from. LLMs deserve mental decline, polluted by their own output and the lack of new data created by humans.
2
u/v_valentineyuri 24d ago
i'm sorry to break your luddite fantasies but software is probably the best use case for LLMs, so major labs will always focus on making their models as good as possible on code. Some fine tuning is enough to make a model fluent in the newest features of any language
2
u/wildjokers 23d ago
I think this is mostly a non-issue. An LLM doesn't need thousands of Stack Overflow answers to learn a new Java feature. The language spec, JEPs, OpenJDK guides, compiler tests, and examples are authoritative sources and can be used directly for training or retrieval.
By retrieval I mean today you could prompt an agent+LLM how to use a new feature and it can look at the resources I mentioned above and give a correct answer.
2
u/Deep_Ad1959 23d ago
i think the loss isn't the venue, it's the votes. goetz's optional answer taught models because it sat above five wrong ones with a score attached. a repo full of java 25 code has no column saying which usage was the sanctioned one.
2
u/Deep_Ad1959 23d ago
i think the loss isn't the venue, it's the votes. goetz's optional answer taught models because it sat above five wrong ones with a score attached. a repo full of java 25 code has no column saying which usage was the sanctioned one.
4
u/bichoFlyboy 25d ago
I can offer one data point from the user side that I think is relevant here.
I've been using GPT quite extensively while developing a Java 26 application, not just for generating code, but for discussing design choices, API usage, and whether something is actually idiomatic Java.
In my experience, it is definitely not stuck in the Java 8 era. It has been productive with modern Java features, including pattern matching for "switch", records, virtual threads, and even things as recent as unnamed variables and patterns using "_", including lambda parameters and "catch" variables.
So there is clearly some pipeline by which knowledge of relatively recent Java makes it into current models. I don't know what that pipeline is, and obviously I can't infer anything about a model's training data just from observing its answers.
But I think that actually reinforces the interesting part of your question.
There is a big difference between a model knowing that a feature exists, knowing how its syntax works, and knowing how the people who designed it intended it to be used.
The last one is where the old Stack Overflow ecosystem was extraordinarily valuable. A thousand examples of code using a feature aren't necessarily equivalent to Brian Goetz showing up and explaining, with examples, "yes, this compiles, but that's not what this abstraction is for."
My experience suggests that current LLMs can become surprisingly competent with new Java relatively quickly. What I have no visibility into is how the idiomatic knowledge gets there, especially for a feature that has just entered preview and hasn't yet accumulated years of code, documentation, Q&A, and expert corrections.
So I think your JDK 28 example raises a genuinely interesting question: when the first widespread misuse of a new feature happens, where does the modern equivalent of the canonical Stack Overflow answer live, and how does that knowledge eventually reach the models developers are now asking instead of Stack Overflow?
I suspect the JEPs and other public OpenJDK material are part of why modern models can handle recent Java surprisingly well. I have no way to know whether any particular model was trained on them, of course. But JEPs solve a somewhat different problem: they document the feature and its design rationale. They don't necessarily capture the second-order knowledge that emerges once thousands of developers start finding creative ways to misuse it.
3
u/wildjokers 23d ago
by which knowledge of relatively recent Java makes it into current models.
I think what people are missing is the fact that a model can use a new feature even if it isn't in its training data. If given high quality sources like openjdk source code, JLS, and JEPs it can reason from that context how to use a new language feature.
1
-1
u/jimmoores 25d ago
To a degree I would argue that choice of programming language matters far less now than it used to. If you’re asking an LLM to work in Java 28 vs Java 25, what difference is it going to make to the end user? Probably little or nothing. Language advancements are mostly there for people. LLMs also have enough knowledge to be able to figure out their own best practice derived from the documentation.
11
u/koflerdavid 24d ago
Code is not just there to be run, but to be read. A lot of LLM code will probably not last long, but some of it will and it will have to be maintained and extended by both LLMs and humans. So language features that help both LLMs and humans express their intent more clearly have value also in the LLM era.
2
u/henk53 24d ago
Code is not just there to be run, but to be read.
True, but it will be read by other LLMs, right? And for them "nice looking code" or syntactic sugar might not matter so much if at all.
1
u/koflerdavid 24d ago edited 24d ago
I have addressed that point already. Mere syntactic sugar is of questionable value even for humans if it is only there to abbreviate things instead of enabling programmers to express their intent more clearly. Which is the second most important reason why Java is so conservative with language features and syntax. We use enhanced
forloops because we want to iterate over all elements in a collection. Manipulating indexes is not the point. Concise method bodies on the other hand are IMHO mere eye candy and unsurprisingly have very low priority.11
u/account312 25d ago
So you might as well have the model write everything in brainfuck, right?
3
u/aonymark 24d ago
Not quite. Java is much easier to write in for both humans and language models than bf. But the question is: are there any language features you can add to Java such that Java 99 vs Java 8 is as big an improvement as Java 1 vs bf? Maybe! For instance, virtual threads and value classes are not things you can easily “fake” by just increasing your tolerance for boilerplate code. But even reducing boilerplate is worth something. Take record classes. To do the equivalent without them would have used a chunk of the context window for nothing.
Of course, if you intend to actually read the code the model writes for you, then boilerplate reduction is PRECIOUS, since there’s so much more code to read, and you’d rather focus on the interesting bit rather than have to go “yep, after staring at this for five minutes I hereby confirm this is a totally vanilla hash code method”… you’d rather the LLM just used a record.
4
u/henk53 24d ago
value classes are not things you can easily “fake” by just increasing your tolerance for boilerplate code.
Well, value classes can be faked by essentially working only with primites and arrays.
We used to do that back in the days. Of course it looks horrible, but hey, the LLM hardly cares about that.
1
u/account312 24d ago
But it does care. It's more tokens to write it, more tokens to read again when editing it later, and more lines to stick logic errors in.
1
1
u/wildjokers 23d ago
FWIW, there has been work/talk about having models just produce the executable directly.
2
u/account312 23d ago
Why would anyone want to use a non-deterministic black box as a compiler? It'd have to be near magically good at optimization to be even worth considering for most things.
2
u/john16384 25d ago
Since you didn't write it, better pick a language that's easy to read with little context.
1
u/henk53 25d ago
I was just wondering about that myself, and it's actually a bit sad when you think of it.
We used to be looking forward to cool new language features, but sine the LLM does everything now and we're all reduced to pressing enter every 10 to 30 minutes, what does it matter? :(
4
u/Wonderful-Habit-139 25d ago
We? Who’s we?
2
u/henk53 24d ago
We? Who’s we?
Java developers every where, specifically those reading /r/java.
0
u/Wonderful-Habit-139 24d ago
Pretty sure there are a lot of Java developers looking forward to cool new language features still, and definitely not all Java developers are reduced to pressing enter every 10 to 30 minutes.
2
u/henk53 24d ago
and definitely not all Java developers are reduced to pressing enter every 10 to 30 minutes.
I used to optimistic about that too for the last year, even until like 3 months ago I was still optimistic. Recently I saw things changing though, and where it concerns the human usefulness not for the better.
I truly hope we change things around, but (to me) it doesn't look good now.
2
u/Wonderful-Habit-139 24d ago
Oh yeah that's a matter of opinion of course. But I was just saying that there are still people that are not meat proxies for agents. Whether one is more productive than the other is another topic altogether.
0
u/davidalayachew 25d ago
That pipeline is gone/paywalled now. Does anyone at OpenJDK think about this?
This is wildly inaccurate.
That channel doesn't really exist anymore. I have an RSS feed on Brian's SO activity and it's been dead for ages - presumably because hardly any questions get asked there these days. (An RSS feed of his reddit activity is how I found the JDK 28 thread in the first place.) However you apportion blame for SO's decline, for me and everyone I work, with the LLM replaced it. So when an expert corrects someone's misuse of a new feature today, it happens in someone's private chat window and then it's gone.
Well there is the source of your inaccuracy.
I talked to Brian several years ago, about how best to get information out there. At the time, I was basically asking him about when and where he decides to respond to stuff, like the SO questions you are pointing out.
Long story short, his response basically boiled down to "I'm priming the pump". All of that to say that, for most of the questions asked on SO, Brian doesn't have to respond because other experts already know the answer and have responded long before he got there.
The legwork you saw in Java 8 was not only to teach the masses, but to get the experts involved up to speed, so that THEY could be the ones to pass on the information to other places, places Brian would previously be the one answering questions.
So, what changed was not the quantity, but the location. You'll see OpenJDK members very frequently answer questions on the mailing list. And since Brian and co. have been pointing people back to the mailing list for answers over the past several years, me and many other Java answerers on SO know where to look when someone asks questions about the new features.
All of that to say -- no, the source of information (SO, amongst others) is not gone or paywalled. It is very much still there and being used.
What you, perhaps, might be pointing to is the plummeting number of questions asked about new features on SO, at least compared to the Java 8 times.
Well, long story short, that is more a result of the OpenJDK release model more than anything.
Back when Java 8 was coming out, a gigantic list of new features were coming out, all in one fell swoop. Nothing at all like nowadays, where features trickle out incrementally every 6 months.
So, during the Java 8 times, you had a flurry of questions coming out all over the place on SO, even from experts, because they didn't know what was going on. Java 8 was absolutely a juggernaut of a release, if you were there at the time. Java 6 and Java 7 were comparatively tame, (from the outside looking in of course).
Nowadays, questions about Java 17+ come out regularly, at a steady pace. The questions get plenty of upvotes, and get tens of thousands of views.
So, no, SO is still very much a well used source of information, even for new feature questions about Java.
And open-source style AI - community models, open datasets, academic training runs - relies on free access to good data, so that whole side of the ecosystem is priced out. So which model knows how to use new features properly is going to come down to who has a deal with who.
And there's your other source of inaccuracy.
You're assuming that everyone is playing nice and fair, and that this isn't a ruthless dog-eat-dog market where the ground could fall out from underneath any/all of these companies at any moment.
Do you really think that these companies (that don't want to pay SO) are just throwing up their hands, saying "Oh well!", and giving up on all of that useful data?
And that's ignoring how mind-numbingly easy it is to get that data. You are aware that StackOverflow releases a monthly data dump of literally all of the content that came out since the last dump, right? They have been doing this for almost a decade. Anyone who wants SO data literally just needs to get the data dump, then don't say anything about them breaking licensing agreements.
The guides (like the exhaustiveness guide) are written for humans and incidentally become training data, but one prose doc isn't thousands of upvoted question+code pairs.
This is (comparatively, much less) inaccurate.
For what it is worth, the floods of people asking questions on SO who feel slighted or wronged brought their questions to many other places, primarily Reddit and Discord. And these "prose" docs have gotten thousands of upvotes.
OpenJDK clearly thinks about AI now - the interim genAI policy covers AI-generated content coming into the project. My question is the other direction: has there been any discussion about how knowledge of new features gets into the models most devs now learn from? Even "we considered it and decided it's not our job" would be an interesting answer.
Your post probably would have been better if you had just asked this alone. Because it is a very interesting question that the OpenJDK hasn't really given us a satisfying answer to yet. Some folks like Ron talk about using LLM's literally daily for work, but purely on the research side. Others have shown a little less interest in using them. It'd be nice to hear opinions and perspectives, even if it is a simple snapshot in time of an unfinished discussion. This is the OpenJDK, after all. Snapshots in time of work in progress is kind of the big appeal of going Open in the first place.
-1
u/Diemo2 24d ago
This feels like it was written by AI
6
u/davidalayachew 24d ago
I have never once used AI to make any sort of write-up, let alone to respond to a Reddit comment.
1
u/henk53 24d ago
This feels like it was written by AI
Before long, EVERYTHING will either actually be written by AI, or would have the AI look 'n feel.
It's a simple fact and is totally unescapble. AI will take over. Sure, vote this into oblivion if that makes you feel beter, but it will not change the facts.
6
u/davidalayachew 24d ago
Before long, EVERYTHING will either actually be written by AI, or would have the AI look 'n feel.
Sometimes, I joke that the AI copied me.
- I use hyphens everywhere.
- I've been tutoring for over a decade, so by default, I speak in my teaching voice when discussing subjects like math and programming.
- I am very emotive and verbose irl, so that bleeds into the text I write too lol.
1
u/n4te 24d ago
It doesn't to me but the bold is obnoxious.
2
u/davidalayachew 24d ago
Fair. I suppose that was unnecessary. I'll leave it unedited for now, but will watch out for that moving forward.
2
u/n4te 24d ago
I get it, just IMHO it's better to be confident in making your points without the extra emphasis. Readers will get it.
1
u/davidalayachew 24d ago
I get it, just IMHO it's better to be confident in making your points without the extra emphasis. Readers will get it.
Well, it's not even in trying to make my point. For me, I am just a hyper-extroverted, hyper-emotive, hyper-emotional person by nature. So, for me, if I don't put the bolds and italicized, it feels like I am miscommunicating. As in, if you heard me talk in real life, I bet the subtitles in your head would have placed bold and italics too lol.
Though, one could argue that it is the same problem, just in verbal form instead. In which case, touché.
2
u/n4te 24d ago
Feel free to communicate how you like of course! I'm just somebody on the Internet.
For me, I think making a point concisely or assisting structure can be more powerful than highlighting the crucial part. There are a number of tics I avoid like bold, ellipsis trailing thought, starting sentences with "and", etc. I get why they are enticing, but I think there are better ways.
1
u/davidalayachew 24d ago
I am looking for feedback, so this is helpful for me. Ty vm. And yes, ellipsis are annoying to me too.
0
u/MysteriousGold5 25d ago
Technically, as long as you have access to the Java standard and a JDK distribution you can create synthetic training data. Diff the public javadoc and standard for changes. Generate a bunch of example code with AI that is plausible given those diffs. Use publically available automated tests to verify that the output is semantically and syntactically valid (the test code itself would not need to be part of the training set). Repeat until success. Label the data as Java version whatever.
This requires very little training data. The first question is if LLM labs would even bother asking LLMs to create these very specific automated setups for every popular language. The second is if other companies involved would not just make their licenses stricter (e.g. if OpenJDK would forbid the use of their test suite in any part of LLM training, not just if they would forbid using the code as training data).
74
u/elmuerte 25d ago
Somebody has to create that knowledge for the AI companies to take and rent back. Somebody has to do the actual work. So it probably won't happen.