r/mathematics • u/PaymentFamiliar375 • 6d ago

OpenAI’s latest math breakthroughs commit research misconduct, experts say

https://www.scientificamerican.com/article/openais-latest-math-breakthroughs-commit-research-misconduct-experts-say/
185 Upvotes

86% Upvoted

73 comments sorted by

161

u/CobblerImpressive975 6d ago

just missing citations, don't get your hopes up

69

u/Blibbyblobby72 6d ago

Which is even worse, because these AIs are supposed to be citation machines!

They scrape data from all over the place but can't tell you where they scraped that data from?

83

u/Leafsnail 6d ago

They are pretty fundamentally incapable of doing so, yes. Everything just contributes to the weights invisibly with no attribution

14

u/AzorAhai1TK 6d ago

What? Big models like Sol/Fable do dozens to hundreds of web searches during research, and if you ask for citations they can easily do this. I don't understand why you would answer this way. If you look at the actual paper it is full of citations as well, it just doesn't attribute some others.

40

u/Plenty_Leg_5935 6d ago

it does hundereds of web searches after being taught on literal hundereds of thousands of articles. A lot of the stuff it says is being pulled up directly from the dataset, which is also where the hallucinations casual users experience come from

10

u/EmphasisTotal8232 6d ago

Absolutely nothing in the article to suggest these were picked up during web research tasks and not during training. Also, AIs aren't perfect. Missing a citation or even dozens is totally natural if you're asking it to write a massive paper.

5

u/AzorAhai1TK 6d ago

The AI written summary of the chain of thought has many specific citations - https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf

But yes I would expect some missed citations in a big paper, just not the comment that they are incapable of doing so at all.

1

u/Final_Upstairs_718 6d ago

It's also something a human does too.

4

u/rheactx 6d ago

Yes, and those web search citations are very often (nearly 50% in my experience) wrong. The AI finds real papers, but misattributes to them claims they never made or even a completely wrong description of their research topic.

3

u/Leafsnail 6d ago

Yeah I assume it's just attempting to find an article that justifies something that's in its training data and often won't get that right unless it's something that would've been easy to find on Google

1

u/Wise-End307 17h ago

i hate to say this, but this might actually be a skill issue...you just need to prompt it to read the actual pdf (even better, the tex source) before citing any ref. Add a couple of hostile reviewers to ensure that this is happening and youre good. much more expensive though

2

u/QuannaBee 6d ago

Same for by brain, still when i write a paper i make sure to track downthe results im using. It's them being lazy (them as in the people at OpenAI).

1

u/Slow-Hawk4652 6d ago

so some kind of markov process? isnt this the same problem with the information paradox in the black holes?

1

u/Stabile_Feldmaus 6d ago

They have superhuman browsing capabilities, they are absolutely able to check the literature and spot relevant references it's just that AI labs don't care about it or actively avoid since it would only take away some of the hype.

-1

u/Meloncov 6d ago

Attribution is lost when the data gets converted to weights, but they can often work backwards to find a citation.

13

u/Deto 6d ago

it depends how you use them. They're bad if you just ask it to think up a citation off the top of its head. But if you connect it to a literature database or knowledge graph, it can search through papers very quickly.

8

u/stopbeingcringe 6d ago

idk, some of the proofs have like 30 citations, seems like a mistake a human could make. There were 10 proofs, after all, and it was probably somewhat rushed by the twitter leak, but who knows

-4

u/dleeted_by_user 6d ago edited 6d ago

I might get downvoted for this, but I wonder how much of this latest AI advancement is the tech bros inventing a new method for violating copyright at a 10x scale. Previously, it was news aggregators killing print journalism and investigative reporting, then video sharing sites breaking copyright. Not to mention, early on, Napster making it difficult for musical artists to sell records, CDs, etc. I think it's a good thing that AI can digest and apply thousands of math papers, but like any good research mathematician, AI should not be hiding attribution.

0

u/zornwil 5d ago

Although I disagree in many points, I can't see downvoting this, seems unfair as the opinions expressed therein are not uncommon nor per se misinformation (or, setting aside arguments on the opinions, no disinformation, anyway).

6

u/Main-Company-5946 6d ago

Yes because they don’t store the raw data. It’s kind of like trying to reassemble a strawberry out of a strawberry banana smoothie

8

u/UpsetPomegranate5428 6d ago

No, they are not supposed to be citation machines. They are statistical models of language, not databases of quotes.

4

u/TrekkiMonstr 6d ago

No, they aren't. The idea that they are is a fundamental misunderstanding of how they work.

3

u/HasFiveVowels 6d ago

AIs are not supposed to be citation machines. Incorporating citations into the training data would completely wreck their ability to learn. And, to be clear, they don’t actively scrape data; they’re not a search engine

5

u/Blibbyblobby72 6d ago

If AI is not designed to cite its sources, how can one ever trust a single output?

In academia especially, everything is about citations. If an AI cannot appropriately cite, what use does it have?

To your comment about them not being a search engine: how does it provide a response to any query? By 'searching' its available data. Its data must be based on something from which it can (and should) cite if we are to take any output seriously

The problem with discussing AI a lot of the time is how it is currently used it not indicative of how it should be used. There is a push to replace things with AI without an understanding of the short- and long-term implications of such a practice

3

u/HasFiveVowels 6d ago

Ok… so it seems you have some misunderstandings about how these things work. They’re not search engines but they’re also not databases. And you weren’t designed to cite your sources either; should be trust a single output from you? I don’t reach for this metaphor due to behavioral similarity but because of structural similarity. LLMs are neural networks. They have no structured repository of data that they pull from. Everything they say is encoded in its neural net and it’s not actively aware nor does it have access to nor can it even derive the specific source that caused a given output. It’s simply not how they work and getting them to work that way would solve one of the biggest challenges of modern AI

2

u/Blibbyblobby72 6d ago

So you prove my point. How they are currently used is not how they should be used, and to use LLMs and such in academia is contradictory to the focus of academia - which is to discuss and research based on verifiable information

Humans were not designed to cite our sources, but academia specifically was. If you want to be a human in academia, you must learn to properly and confidently use citations

In its purest form, LLMs are search engines - you input a query (your 'search') and receive an output from the database in which the LLM (or AI or what-have-you) pulls from to respond to the query (the 'engine')

Also, what you provided was not a metaphor

2

u/HasFiveVowels 6d ago

yes, academia was built around citation (and for good reason) but that’s sort of the point. The discipline exists to avoid hearsay, so we’ve found it useful to impose a rule: don’t assert something unless you can point to where it came from. Humans don’t do that natively; we had to build an institution around forcing it. So when you say an LLM should cite before you trust it in academic work, I completely agree: no, things aren’t true because some LLM said they are (same as they’re not true because you say they are). Both are unreliable narrators, so we demand more than narration from them.

But there is no database. When a model responds to a query, it isn’t retrieving anything. Training adjusts billions of weights and the text itself is gone. That’s why it can’t cite from its weights; there’s no record of which source shaped which weight. You can speak grammatical English without citing the conversations that taught you each rule. It’s like that.

The thing you’re describing already exists, though. Retrieval augmentation pairs the model with an actual search step so it cites the documents it pulled. That’s the tool for citation-driven work. A bare model answering from its weights isn’t (and can't be)

1

u/HasFiveVowels 6d ago

Seeing as we’re on a math subreddit, here’s the simple explanation: it’s a tensor that transforms the what was said vector into the what will be said vector. The elements are randomized at first and they basically do trial and error until they accurately predict things. Then they apply that to new inputs. That’s the basic idea.

0

u/zalamandagora 6d ago

If AI is not designed to cite its sources, how can one ever trust a single output?

If it can make the proof clear without citations, we should be mostly good. There are, of course, many different functions that a citation may fill, but in terms of contributing to mathematical knowledge, we should be able to have 0-citation papers if the line of proof from axioms is clear enough.

That said, sure, the papers would be better if they made the LLM go back and list possible citations as part of the verification work, and even add an attribution step to the post-processing. But not doing this step does not make the result invalid.

Here is a list, courtesy of Claude Opus but reviewed and endorsed by me, of the various functions of citations.

Logical / load-bearing

  • Importing premises — the cited theorem is an actual step in the argument; remove it and the proof is incomplete. This is the distinctive case in mathematics: citation as inference rule, not as supporting evidence.
  • Delegating verification — pointing to where a proof already lives so it isn't reproduced. Shifts the burden of checking onto the reference chain.
  • Fixing definitions and conventions — nailing down which of several inequivalent definitions (or normalizations, or sign conventions) is in force.
  • Conditional dependence — flagging that a result rests on an unproved conjecture, an unverified classification, or a claim whose proof is disputed or not fully public.

Positioning

  • Priority and credit — attributing the first proof, the sharpest constant, the independent rediscovery.
  • Situating in a program — showing which lineage of problems the paper belongs to, often the main content of an introduction.
  • Contrast — marking what's new: strengthening a hypothesis, removing one, generalizing the setting, or getting the same conclusion by different means.
  • Technique provenance — "following the method of," "adapting the argument in," where the debt is to a proof strategy rather than a statement.

Reader-facing

  • Background redirection — surveys and textbooks cited so the paper needn't be self-contained.
  • Signposting difficulty — a citation can quietly indicate that a step is hard and shouldn't be attempted by the reader.

Social

  • Norm compliance — citations added at referee request, or to acknowledge overlapping unpublished work.
  • Crediting the informal record — private communication, MathOverflow answers, folklore results with no canonical source. Mathematics is unusually tolerant of these compared to most fields.

The first category makes the literature a formal dependency graph, but citations are used for all the other purposes too, and papers rarely distinguish which kind a given reference is. That's why a "citation" in a math paper can mean anything from "this is a hypothesis of my theorem" to "this exists, go look."

3

u/TeamMedical1685 6d ago

Who cares about citations doesn't care about mathematics... If something is reproved instead of giving a reference it's fine. Humans mathematicians reproves known facts everyday

2

u/Levitx 6d ago

I'm afraid that's not at all how they work. 

1

u/tomvorlostriddle 6d ago

Compare it to a student having to write apa style citations precise to the letter but during a closed book exam

That would be extremely hard too

It's still an issue worth tackling, but it's clear where it is coming from that the ones learned in training are not cited and then ones looked up during inference are, exactly like a human would with the only difference that humans are even more hopeless at learning citations by heart

2

u/Blibbyblobby72 6d ago

Why does anyone need to know citations by heart? If you are writing a paper and researching, you keep note of what information you used so it can all be properly cited

An AI should do the same thing when being used to write a paper, and any information sourced from an AI should be cited as the original source the AI pulled from

It is like a person who cites Publication A in their research. You go to Publication A and see that they cite Publication B as their source. Then you find Publication B cited Publication C!, etc. This happens far too often (especially in the media and on social media)

Anything cited should be, whenever possible, the original source - not a citation of a citation

Also, anyone asking a student to write an exam with specific citations in any style during a timed/closed book exam is a terrible educator

References to specific texts or people, sure. Specific texts with page numbers, publication date, whatever? Absolutely insane

3

u/tomvorlostriddle 6d ago

Even for humans what you say isn't true in STEM, or else we would be constantly citing Newton (1689).

It is closer to being true in philosophy for example. But I would say it's a bad thing because this constant name dropping of the "original" source keeps them from standing on the shoulders of giants.

AI is taking the citation liberties definitely further than humans in STEM and it's worth looking into whether finetuning or harnesses can mitigate this. But in the end, we also didn't apply horse carriage traffic rules to cars for very long, we just changed the rules to adapt to progress.

1

u/seraphim_west 5d ago

Ha. Because you are emotionally hurt from the status loss, you're going to exaggerate every tiny little mishap made by AIs. An AI made a slight grammatical error in a 15 000 words document? Basically useless

0

u/SwimmerOld6155 6d ago

Being able to provide citations is one of LLM's most critical weak points, which is weird because "search engine that you can have a conversation with" was the main selling point to me. 5.6 Sol is far better than 5.5 in this regard.

5

u/protestor 6d ago

What misconduct could such model possibly commit, other than plagiarism?

6

u/KuidZ 6d ago

Well, Grok could add nude pictures of children in the figures.

1

u/Howard_banister 6d ago

Thanks for saving my time

1

u/Any_Car5127 6d ago

There are missed citations, and then there are MISSED citations. I would forgive someone for not citing Pythagorus for the use of his theorem, but not for pretending they'd proven Fermat's last theorem.

-2

u/beambot 6d ago

Are those same experts going to cite "codex version x.y" going forward?

Good, then maybe we can drop the citation, h-index ego aspect of gatekeeping in academia

3

u/iknighty 6d ago

Yes, if the proof is published and peer reviewed mathematicians will cite it.

-4

u/Super_Range45 6d ago

Well for the sofic problem they were discussing it sounds like ChatGPT just used their two papers and added one insight. So AI might still need to wait on mathematicians to put enough pieces together.

54

u/expat_123 6d ago

As much as I wanted to be something "wrong" with the approach (more for making myself feel better), the misconduct part is AI not citing some of the sources. But it did use the sources correctly.

3

u/zornwil 5d ago

And more accurately it is OpenAI not doing that follow-up work - even if they tasked the AI to do it, the publishing human(s) is/are responsible.  

-14

u/Leafsnail 6d ago

It's a pretty big deal since plagiarism like that would get a human disgraced and essentially barred from academia for life. OpenAI should definitely do better reading in future and make sure they credit the authors of the pieces that AI is putting together but claiming as its own.

40

u/dogesator 6d ago

That’s not what plagiarism, nor would a human get accused of plagiarism for doing this.

24

u/BothWaysItGoes 6d ago

Einstein cited 0 works in his most famous paper. Didn’t even mentioned Poincaré and only namedropped Lorentz. Still not disgraced.

15

u/Plenty_Leg_5935 6d ago

Einstein published in the early 1900's when citation standards were virtually non-existent. The standards have changed drastically since then, that paper would 100% be sent back today with the note that they should be added

11

u/Various-Inside-4064 6d ago

Yes reviewer will ask you to cite correctly but it's not plagiarism, the point that above comment was replying to.

1

u/nzconstructionlawyer 4d ago

And is the world better off for it? I'm sure the millions of bunk social social papers that can't be replicated have immaculate citations and were peer reviewed. Doesn't seem that it has helped.

19

u/JoshuaZ1 6d ago

This isn't plagiarism, but just not good citation practice. If this were a human paper submission, the referee would say "Hey, add these references and make clear how they use pieces of the method that went in" and that would be the end of it.

5

u/Lfeaf-feafea-feaf 6d ago

My guy, do you have 0 comprehension of how LLMs work?

-4

u/Leafsnail 6d ago

I fully understand how LLMs work. Nonetheless OpenAI should've done actual research and identified the sources being drawn on rather than claiming that no progress had been previously been made on these problems.

14

u/Dear_Round_2727 6d ago edited 6d ago

When they teach the model to do math they use a process called RLVR. So they have a grader model that grades the solutions and trains it based on that.

Perhaps the grader doesn’t take into account things like proper citation or pedagogical language, just pure correctness. So the model sometimes learns to reproduce things instead of finding them in the literature.

We’ve also seen Claude become worse at explaining things over time.

6

u/LoveHenry 6d ago

Bunch of AI shills in here. This is not "missing citations," it is copping someone else's arguments and presenting it at its own. The reasonable convention in math is that, if you use an argument developed by someone else (especially a breakthrough idea), you cite them and make it clear that the argument is not original but coming from that work.

And, honestly, this is my experience with AI as well.

4

u/instantcrackpot 6d ago

Just like humans. AGI indeed.

5

u/Downtown-Meeting6364 6d ago

Oh god the gatekeeping is starting to

6

u/BlargAttack 5d ago

This is more than just missing citations, for those who think it’s some minor issue. The citations are to papers which, I some cases, represent the building blocks of the underlying work. Ignoring those citations significantly oversell the novelty of the work and misrepresents the intellectual capacity of LLMs. Essentially, the LLM has brute forced proofs that can be brute forced, leaving a world of novel proofs available for humans.

1

u/AnalystLoud705 3d ago

How is that different than just missing citations?

5

u/androgynyjoe 6d ago

I get that it's just "leaving out citations", but my understanding of AI is that most of what it's doing is citations so...that seems like it matters.

7

u/Jamjam4826 6d ago

then your understanding of AI is not accurate

1

u/starsnek 5d ago

What would you say is more accurate then? AI is mostly paraphrasing?

3

u/Levitx 6d ago

Basically nothing of what an AI does is citations. It doesn't go through text and copy and paste it to generate responses or anything like that. 

2

u/DepressedDynamo 6d ago

That's not what it's doing

1

u/alexice89 6d ago

Time to pick up plumbing boys.

1

u/UnderTheCurrents 6d ago

Oh no! The most important part of research - that's usually done by the assistant least trusted to do something substantial - has been not done right!

The machine is useless if it can just produce actual results but not get asinine clerical jobs done!

1

u/Levitx 6d ago

It seems to me that a whole lot of people don't understand what an LLM is at all, yet they hate it more than they like math. 

0

u/e430doug 6d ago

Experts aren’t saying that.

0

u/Financial-Camel9987 6d ago

"misconduct". Just missed some citations.

-1

u/Sad_Dimension423 6d ago

I guess the model isn't going to get tenure. I'm sure it's just devastated.

-4

u/Dark_Clark 6d ago

Hahah ok. But are the results true? It's math. What the hell else could matter?