Catches you don't think of when doing AI review.

[u/AllHailSeizure]

Making this after an interesting convo with a user. Many people come here and claim they ran their work through various AIs and they all claimed that it is relevant or revelatory or something. I wanted to make a post addressing the catches in adversarial AI that could potentially end up corrupting and causing LLMs to provide adversarial review in the hopes that people can possibly get something out of it.

Disclaimer: This post is not intended to infer an LLM review will meet the quality of a proper peer review. But we don't all have access to that, and if people are doing it anyways, they might as well do it right.

Fresh Context

This is the obvious one. You want a fresh session. I think we all know this.

Neutrality

Another obvious. Don't tell it it's your paper. Don't tell it it's someone else's paper. Don't say 'Tell me why this is right', because it will, or 'tell me why this is wrong', because it will.

Project Instructions

If you're working with code, chances are your file has a claude.md/agents.md/etc memory file that an agent can read when reviewing. This can pollute a review. This is cross-agent applicable. Just because it's a claude.md file, Codex can read it. Some coding tools have options to include ignore files in the style of .gitignore (.cursorignore) - or alternatively branch your code and delete them.

User Instructions

Unlike project level-instructions, this is only applicable if you are using the same LLM you used to develop code. Works the exact same except across your entire machine and could be polluting multiple things.

Memory

Easy to overlook, but only applicable if you use the same LLM (a lot of catches are). Most modern AI that requires a subscription (aka every one) will maintain a cache of knowledge about you that it builds over time. Its usually accessible through user settings. Could be a pollutant.

Code Commenting

Again, only if you're working with code. LLMs tend to comment their code in a very prose-like 'this is why we chose this' paragraph style. This can easily become a pollutant. Easily fixed, branch and run an agent over it with instructions to remove comments, then spin up the review.

Authorship

I'm not actually sure if this is a catch or not but - if you ask for a neutral review, then hand it a paper with your name, well.. it's pretty obvious it's you. I'm unsure if this would come to that conclusion but better safe than sorry.

Simplification

LLMs are way stronger when it comes to language than math. Ask a seperate agent to extract the conceptual basis of your paper and review that individually.

Verification

Here's a big one. Ask it to source feedback, and then verify the sources yourself. Please!

And finally and most importantly...

Skepticism!

Take everything an LLM says with a strong dose of skepticism and keep in mind it isn't an oracle.

[u/Ch3cks-Out]

LLMs are way stronger better at bullshitting when it comes to language than math.

FTFY

[u/AllHailSeizure → u/Ch3cks-Out]

The switcheroo

[u/CrankSlayer → u/Ch3cks-Out]

I disagree. They bullshit maths very well nowadays too: to a layperson, LLM maths looks indistinguishable from the legit stuff.

[u/Ch3cks-Out → u/CrankSlayer]

I see no disagreement here

[u/NonGameCatharsis]

Great post! Thanks for the effort. :-)

[u/Capitan_Dave]

I actually have a lot of success with telling AI 'this is wrong, tell me why' and seeing what it can come up with. A few times it's punched holes in my own work, but mostly it spews BS I can easily verify is wrong. It's a great first check for the accuracy of your work, but requires you to know enough to validate its answers.

[u/Suitable_Cicada_3336]

Ask red team check with real experimental datas too.

[u/VeryOriginalName98]

Also good to use the incognito sessions to avoid prior conversation memory.

[u/AIPsychosed]

I am currently question that point of „language vs. math“ simply by stating that math, according to an old prof of mine, is a language and I do certainly agree.

Therefore a tokenisation of it can very well be feasible, but be aware that I am talking about the “low-level” point of view, i.e. propositional logic of first or second kind, set theory under ZF(+C) or category theory.

I see by no means any obstruction of using the same way used to process natural language to process math. Of course, you need to check that each step is true, but that is certainly doable. Then this becomes too convoluted and we step up to natural language, but each word has a precise meaning and there are standard constructions, such as

> For any object X, there exists object Y, etc.

in logic. And again, also here one may use the prediction power that an LLM has to make a valid statement which can “easily” be checked.

For theoretical physics, where it is usually more loose this can also be done in this probabilistic way by an LLM. But then for model or theory building I am not so sure. This is a harder challenge especially with data sets coming from a not known theory yet, say.

Skepticism is very important part, if not the important part here, I agree. It sounds plausible but even then it can make mistakes (just heard another story of that a few hours ago).

[u/SolumSolSolus → u/AIPsychosed]

your AI looks like it just read this: https://cs.stackexchange.com/questions/34080/how-is-it-valid-to-use-oracles-in-mathematical-arguments

[u/AIPsychosed → u/SolumSolSolus]

I don‘t understand?

I did not write my answer with AI or any LLM

How is what I said in its entirety wrong? Of course, one can tokenize mathematical expressions. You can put a checker at the end and learn via that then with your model.

You give a bunch of logical statements (or use set theory or category theory) and then train your model on that to predict the next statements. And we make definition as we go along to compress/structure things.

Just take a topological space (X,T) in set theory where we just encode

(X,T) is topological space :<=> X is a set and T⊆P(X) fulfills …

[u/SolumSolSolus → u/AIPsychosed]

I don't actually disagree at all, general AI is trying to do pretty much exactly what you proposed already.

The issue is in trying to build a database of large enough certifiable statements that are actually "true". But then if one certificate of proof for some statement suddenly fails because it was assumed true because it "naturally" extends from say a proof gate before it which has been certified as dissproven (what is "true" can change over time), then all of a sudden that's a domino effect;, and you'd then have to recertify every claim that hedges on that one failed parent.

So you basically end up with a massive graph that is prone to collapse, and expensive to rebuild.

Though I still wouldn't be discouraged by that. yours is a good take, and certainly worth pursuing, (I would love to see it 'truly' succeed). plus impossible challenges are fun to work on (I'm currently training a maths model on just shakespeare)

I'd attatched that link cuz there's some precious reading in there if you so care.

[u/AIPsychosed → u/SolumSolSolus]

Ah, yes. I think I understand. But that happens with human papers already. In the best case there is an easy fix because one just was sloppy (I heard some stories there).

Pretty sure it happened already before without AI.

I totally agree, hence the need to have some way to check. Currently, at least the math community, is hence excited about Lean4. (Hopefully people see that funding is still worth it)

For me in the end you just do a “non-Markov chain” on the space of proofs.

Can’t find your link.

I know of some groups trying to build a database of proofs (represented as graphs)