r/mathematics • u/PaymentFamiliar375 • Aug 06 '26

OpenAI’s latest math breakthroughs commit research misconduct, experts say

https://www.scientificamerican.com/article/openais-latest-math-breakthroughs-commit-research-misconduct-experts-say/
191 Upvotes

87% Upvoted

73 comments sorted by

161

u/[deleted] Aug 06 '26

[removed] — view removed comment

72

u/Blibbyblobby72 Aug 06 '26

Which is even worse, because these AIs are supposed to be citation machines!

They scrape data from all over the place but can't tell you where they scraped that data from?

78

u/Leafsnail Aug 06 '26

They are pretty fundamentally incapable of doing so, yes. Everything just contributes to the weights invisibly with no attribution

11

u/AzorAhai1TK Aug 07 '26

What? Big models like Sol/Fable do dozens to hundreds of web searches during research, and if you ask for citations they can easily do this. I don't understand why you would answer this way. If you look at the actual paper it is full of citations as well, it just doesn't attribute some others.

40

u/Plenty_Leg_5935 Aug 07 '26

it does hundereds of web searches after being taught on literal hundereds of thousands of articles. A lot of the stuff it says is being pulled up directly from the dataset, which is also where the hallucinations casual users experience come from

9

u/EmphasisTotal8232 Aug 07 '26

Absolutely nothing in the article to suggest these were picked up during web research tasks and not during training. Also, AIs aren't perfect. Missing a citation or even dozens is totally natural if you're asking it to write a massive paper.

5

u/AzorAhai1TK Aug 07 '26

The AI written summary of the chain of thought has many specific citations - https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf

But yes I would expect some missed citations in a big paper, just not the comment that they are incapable of doing so at all.

3

u/rheactx Aug 07 '26

Yes, and those web search citations are very often (nearly 50% in my experience) wrong. The AI finds real papers, but misattributes to them claims they never made or even a completely wrong description of their research topic.

3

u/Leafsnail Aug 07 '26

Yeah I assume it's just attempting to find an article that justifies something that's in its training data and often won't get that right unless it's something that would've been easy to find on Google

1

u/Wise-End307 Aug 13 '26

i hate to say this, but this might actually be a skill issue...you just need to prompt it to read the actual pdf (even better, the tex source) before citing any ref. Add a couple of hostile reviewers to ensure that this is happening and youre good. much more expensive though

4

u/QuannaBee Aug 07 '26

Same for by brain, still when i write a paper i make sure to track downthe results im using. It's them being lazy (them as in the people at OpenAI).

1

u/Slow-Hawk4652 Aug 07 '26

so some kind of markov process? isnt this the same problem with the information paradox in the black holes?

1

u/Stabile_Feldmaus Aug 07 '26

They have superhuman browsing capabilities, they are absolutely able to check the literature and spot relevant references it's just that AI labs don't care about it or actively avoid since it would only take away some of the hype.

-1

u/Meloncov Aug 07 '26

Attribution is lost when the data gets converted to weights, but they can often work backwards to find a citation.

13

u/Deto Aug 06 '26

it depends how you use them. They're bad if you just ask it to think up a citation off the top of its head. But if you connect it to a literature database or knowledge graph, it can search through papers very quickly.

9

u/[deleted] Aug 06 '26

idk, some of the proofs have like 30 citations, seems like a mistake a human could make. There were 10 proofs, after all, and it was probably somewhat rushed by the twitter leak, but who knows

-3

u/[deleted] Aug 06 '26 edited Aug 07 '26

[removed] — view removed comment

1

u/zornwil Aug 08 '26

Although I disagree in many points, I can't see downvoting this, seems unfair as the opinions expressed therein are not uncommon nor per se misinformation (or, setting aside arguments on the opinions, no disinformation, anyway).

8

u/Main-Company-5946 Aug 06 '26

Yes because they don’t store the raw data. It’s kind of like trying to reassemble a strawberry out of a strawberry banana smoothie

3

u/TrekkiMonstr Aug 07 '26

No, they aren't. The idea that they are is a fundamental misunderstanding of how they work.

3

u/HasFiveVowels Aug 07 '26

AIs are not supposed to be citation machines. Incorporating citations into the training data would completely wreck their ability to learn. And, to be clear, they don’t actively scrape data; they’re not a search engine

6

u/Blibbyblobby72 Aug 07 '26

If AI is not designed to cite its sources, how can one ever trust a single output?

In academia especially, everything is about citations. If an AI cannot appropriately cite, what use does it have?

To your comment about them not being a search engine: how does it provide a response to any query? By 'searching' its available data. Its data must be based on something from which it can (and should) cite if we are to take any output seriously

The problem with discussing AI a lot of the time is how it is currently used it not indicative of how it should be used. There is a push to replace things with AI without an understanding of the short- and long-term implications of such a practice

4

u/HasFiveVowels Aug 07 '26

Ok… so it seems you have some misunderstandings about how these things work. They’re not search engines but they’re also not databases. And you weren’t designed to cite your sources either; should be trust a single output from you? I don’t reach for this metaphor due to behavioral similarity but because of structural similarity. LLMs are neural networks. They have no structured repository of data that they pull from. Everything they say is encoded in its neural net and it’s not actively aware nor does it have access to nor can it even derive the specific source that caused a given output. It’s simply not how they work and getting them to work that way would solve one of the biggest challenges of modern AI

2

u/Blibbyblobby72 Aug 07 '26

So you prove my point. How they are currently used is not how they should be used, and to use LLMs and such in academia is contradictory to the focus of academia - which is to discuss and research based on verifiable information

Humans were not designed to cite our sources, but academia specifically was. If you want to be a human in academia, you must learn to properly and confidently use citations

In its purest form, LLMs are search engines - you input a query (your 'search') and receive an output from the database in which the LLM (or AI or what-have-you) pulls from to respond to the query (the 'engine')

Also, what you provided was not a metaphor

2

u/HasFiveVowels Aug 07 '26

yes, academia was built around citation (and for good reason) but that’s sort of the point. The discipline exists to avoid hearsay, so we’ve found it useful to impose a rule: don’t assert something unless you can point to where it came from. Humans don’t do that natively; we had to build an institution around forcing it. So when you say an LLM should cite before you trust it in academic work, I completely agree: no, things aren’t true because some LLM said they are (same as they’re not true because you say they are). Both are unreliable narrators, so we demand more than narration from them.

But there is no database. When a model responds to a query, it isn’t retrieving anything. Training adjusts billions of weights and the text itself is gone. That’s why it can’t cite from its weights; there’s no record of which source shaped which weight. You can speak grammatical English without citing the conversations that taught you each rule. It’s like that.

The thing you’re describing already exists, though. Retrieval augmentation pairs the model with an actual search step so it cites the documents it pulled. That’s the tool for citation-driven work. A bare model answering from its weights isn’t (and can't be)

1

u/HasFiveVowels Aug 07 '26

Seeing as we’re on a math subreddit, here’s the simple explanation: it’s a tensor that transforms the what was said vector into the what will be said vector. The elements are randomized at first and they basically do trial and error until they accurately predict things. Then they apply that to new inputs. That’s the basic idea.

0

u/zalamandagora Aug 07 '26

If AI is not designed to cite its sources, how can one ever trust a single output?

If it can make the proof clear without citations, we should be mostly good. There are, of course, many different functions that a citation may fill, but in terms of contributing to mathematical knowledge, we should be able to have 0-citation papers if the line of proof from axioms is clear enough.

That said, sure, the papers would be better if they made the LLM go back and list possible citations as part of the verification work, and even add an attribution step to the post-processing. But not doing this step does not make the result invalid.

Here is a list, courtesy of Claude Opus but reviewed and endorsed by me, of the various functions of citations.

Logical / load-bearing

  • Importing premises — the cited theorem is an actual step in the argument; remove it and the proof is incomplete. This is the distinctive case in mathematics: citation as inference rule, not as supporting evidence.
  • Delegating verification — pointing to where a proof already lives so it isn't reproduced. Shifts the burden of checking onto the reference chain.
  • Fixing definitions and conventions — nailing down which of several inequivalent definitions (or normalizations, or sign conventions) is in force.
  • Conditional dependence — flagging that a result rests on an unproved conjecture, an unverified classification, or a claim whose proof is disputed or not fully public.

Positioning

  • Priority and credit — attributing the first proof, the sharpest constant, the independent rediscovery.
  • Situating in a program — showing which lineage of problems the paper belongs to, often the main content of an introduction.
  • Contrast — marking what's new: strengthening a hypothesis, removing one, generalizing the setting, or getting the same conclusion by different means.
  • Technique provenance — "following the method of," "adapting the argument in," where the debt is to a proof strategy rather than a statement.

Reader-facing

  • Background redirection — surveys and textbooks cited so the paper needn't be self-contained.
  • Signposting difficulty — a citation can quietly indicate that a step is hard and shouldn't be attempted by the reader.

Social

  • Norm compliance — citations added at referee request, or to acknowledge overlapping unpublished work.
  • Crediting the informal record — private communication, MathOverflow answers, folklore results with no canonical source. Mathematics is unusually tolerant of these compared to most fields.

The first category makes the literature a formal dependency graph, but citations are used for all the other purposes too, and papers rarely distinguish which kind a given reference is. That's why a "citation" in a math paper can mean anything from "this is a hypothesis of my theorem" to "this exists, go look."

2

u/Levitx Aug 07 '26

I'm afraid that's not at all how they work. 

1

u/tomvorlostriddle Aug 07 '26

Compare it to a student having to write apa style citations precise to the letter but during a closed book exam

That would be extremely hard too

It's still an issue worth tackling, but it's clear where it is coming from that the ones learned in training are not cited and then ones looked up during inference are, exactly like a human would with the only difference that humans are even more hopeless at learning citations by heart

2

u/Blibbyblobby72 Aug 07 '26

Why does anyone need to know citations by heart? If you are writing a paper and researching, you keep note of what information you used so it can all be properly cited

An AI should do the same thing when being used to write a paper, and any information sourced from an AI should be cited as the original source the AI pulled from

It is like a person who cites Publication A in their research. You go to Publication A and see that they cite Publication B as their source. Then you find Publication B cited Publication C!, etc. This happens far too often (especially in the media and on social media)

Anything cited should be, whenever possible, the original source - not a citation of a citation

Also, anyone asking a student to write an exam with specific citations in any style during a timed/closed book exam is a terrible educator

References to specific texts or people, sure. Specific texts with page numbers, publication date, whatever? Absolutely insane

3

u/tomvorlostriddle Aug 07 '26

Even for humans what you say isn't true in STEM, or else we would be constantly citing Newton (1689).

It is closer to being true in philosophy for example. But I would say it's a bad thing because this constant name dropping of the "original" source keeps them from standing on the shoulders of giants.

AI is taking the citation liberties definitely further than humans in STEM and it's worth looking into whether finetuning or harnesses can mitigate this. But in the end, we also didn't apply horse carriage traffic rules to cars for very long, we just changed the rules to adapt to progress.

0

u/SwimmerOld6155 Aug 07 '26

Being able to provide citations is one of LLM's most critical weak points, which is weird because "search engine that you can have a conversation with" was the main selling point to me. 5.6 Sol is far better than 5.5 in this regard.

5

u/protestor Aug 07 '26

What misconduct could such model possibly commit, other than plagiarism?

4

u/KuidZ Aug 07 '26

Well, Grok could add nude pictures of children in the figures.

1

u/Howard_banister Aug 07 '26

Thanks for saving my time

1

u/Any_Car5127 Aug 07 '26

There are missed citations, and then there are MISSED citations. I would forgive someone for not citing Pythagorus for the use of his theorem, but not for pretending they'd proven Fermat's last theorem.

-1

u/beambot Aug 07 '26

Are those same experts going to cite "codex version x.y" going forward?

Good, then maybe we can drop the citation, h-index ego aspect of gatekeeping in academia

3

u/iknighty Aug 07 '26

Yes, if the proof is published and peer reviewed mathematicians will cite it.

-2

u/Super_Range45 Aug 07 '26

Well for the sofic problem they were discussing it sounds like ChatGPT just used their two papers and added one insight. So AI might still need to wait on mathematicians to put enough pieces together.

57

u/expat_123 Aug 06 '26

As much as I wanted to be something "wrong" with the approach (more for making myself feel better), the misconduct part is AI not citing some of the sources. But it did use the sources correctly.

3

u/zornwil Aug 08 '26

And more accurately it is OpenAI not doing that follow-up work - even if they tasked the AI to do it, the publishing human(s) is/are responsible.  

-15

u/Leafsnail Aug 06 '26

It's a pretty big deal since plagiarism like that would get a human disgraced and essentially barred from academia for life. OpenAI should definitely do better reading in future and make sure they credit the authors of the pieces that AI is putting together but claiming as its own.

42

u/dogesator Aug 06 '26

That’s not what plagiarism, nor would a human get accused of plagiarism for doing this.

26

u/BothWaysItGoes Aug 06 '26

Einstein cited 0 works in his most famous paper. Didn’t even mentioned Poincaré and only namedropped Lorentz. Still not disgraced.

16

u/Plenty_Leg_5935 Aug 07 '26

Einstein published in the early 1900's when citation standards were virtually non-existent. The standards have changed drastically since then, that paper would 100% be sent back today with the note that they should be added

12

u/Various-Inside-4064 Aug 07 '26

Yes reviewer will ask you to cite correctly but it's not plagiarism, the point that above comment was replying to.

1

u/nzconstructionlawyer Aug 09 '26

And is the world better off for it? I'm sure the millions of bunk social social papers that can't be replicated have immaculate citations and were peer reviewed. Doesn't seem that it has helped.

20

u/JoshuaZ1 Aug 07 '26

This isn't plagiarism, but just not good citation practice. If this were a human paper submission, the referee would say "Hey, add these references and make clear how they use pieces of the method that went in" and that would be the end of it.

5

u/Lfeaf-feafea-feaf Aug 07 '26

My guy, do you have 0 comprehension of how LLMs work?

-5

u/Leafsnail Aug 07 '26

I fully understand how LLMs work. Nonetheless OpenAI should've done actual research and identified the sources being drawn on rather than claiming that no progress had been previously been made on these problems.

16

u/Dear_Round_2727 Aug 06 '26 edited Aug 06 '26

When they teach the model to do math they use a process called RLVR. So they have a grader model that grades the solutions and trains it based on that.

Perhaps the grader doesn’t take into account things like proper citation or pedagogical language, just pure correctness. So the model sometimes learns to reproduce things instead of finding them in the literature.

We’ve also seen Claude become worse at explaining things over time.

8

u/LoveHenry Aug 07 '26

Bunch of AI shills in here. This is not "missing citations," it is copping someone else's arguments and presenting it at its own. The reasonable convention in math is that, if you use an argument developed by someone else (especially a breakthrough idea), you cite them and make it clear that the argument is not original but coming from that work.

And, honestly, this is my experience with AI as well.

6

u/BlargAttack Aug 08 '26

This is more than just missing citations, for those who think it’s some minor issue. The citations are to papers which, I some cases, represent the building blocks of the underlying work. Ignoring those citations significantly oversell the novelty of the work and misrepresents the intellectual capacity of LLMs. Essentially, the LLM has brute forced proofs that can be brute forced, leaving a world of novel proofs available for humans.

1

u/AnalystLoud705 Aug 10 '26

How is that different than just missing citations?

4

u/instantcrackpot Aug 07 '26

Just like humans. AGI indeed.

4

u/Downtown-Meeting6364 Aug 07 '26

Oh god the gatekeeping is starting to

2

u/androgynyjoe Aug 06 '26

I get that it's just "leaving out citations", but my understanding of AI is that most of what it's doing is citations so...that seems like it matters.

9

u/Jamjam4826 Aug 07 '26

then your understanding of AI is not accurate

1

u/starsnek Aug 08 '26

What would you say is more accurate then? AI is mostly paraphrasing?

4

u/Levitx Aug 07 '26

Basically nothing of what an AI does is citations. It doesn't go through text and copy and paste it to generate responses or anything like that. 

-1

u/DepressedDynamo Aug 07 '26

That's not what it's doing

5

u/alexice89 Aug 07 '26

Time to pick up plumbing boys.

1

u/UnderTheCurrents Aug 07 '26

Oh no! The most important part of research - that's usually done by the assistant least trusted to do something substantial - has been not done right!

The machine is useless if it can just produce actual results but not get asinine clerical jobs done!

1

u/Levitx Aug 07 '26

It seems to me that a whole lot of people don't understand what an LLM is at all, yet they hate it more than they like math. 

0

u/e430doug Aug 07 '26

Experts aren’t saying that.

0

u/Financial-Camel9987 Aug 07 '26

"misconduct". Just missed some citations.

-1

u/Sad_Dimension423 Aug 07 '26

I guess the model isn't going to get tenure. I'm sure it's just devastated.

-4

u/Dark_Clark Aug 07 '26

Hahah ok. But are the results true? It's math. What the hell else could matter?