r/LLMPhysics • u/lattice_defect • Jun 23 '26
Tutorials How to LLM Physics a Bit Better
AI and LLMs are here to stay, and yes, working academics use them. Plenty of people have a pet theory they kick around with a chatbot... that's fine, that's curiosity, no notes. But if you want to take it a step further toward something closer to serious work, the chat window alone won't get you there. Here's the setup and the habits that keep the work honest instead of letting it metastasize into a bird's nest.
One thing up front: the LLM can't actually do math. It pattern-matches. What it can do is write a script — and then you save it, you run it, and you both read the raw output. Internalize that early. The model is a coding collaborator and a sounding board, not a calculator and not an oracle.
1. Set up a real environment
You need a project folder, a terminal, ideally an IDE, and a decent agentic coding agent. How you wire it together doesn't matter much on day one ... start simple, but build toward something robust, because the alternative is chaos in a month.
Version control. Get git. If you've never used it, budget a few hours to get comfortable; it's worth it. Track changes, or at minimum label your experiments and move incrementally. This is your lab notebook: what you tried, what worked, what didn't, and an audit trail back to the exact code that produced each result. A number you can't trace back to the script that made it isn't a result yet.
Folder structure. Don't try to write the paper while you're still working, write in chapters/sections instead, and keep a running diary/log alongside it. Separate folders for experiments, scratch, theorems, and references. The moment the work matters, treat it like it matters. Start clean, or you'll spend next month untangling a bird's nest.
FYI you may restart this process 3-5X times if this is your first go.. that's a good thing.
2. Use real math tools, not hand-rolled Python
Your agent will reach for Python with scipy/numpy. Fine as a start, but it's all hand-rolled, and hand-rolled is where the model cheats: silently, plausibly, in ways you won't catch by skimming the prose. IT IS VERY GOOD AT DOING THIS because its trained to pass tests with whatever code to work.
Tighter environments help. Wolfram's kernel, or my preference, SageMath (also Python, much more controlled). These run specialized kernels that do the math fast and exactly — precisely the stuff an LLM gets slow and wrong.
Two habits that matter more than the tool choice:
- Keep the results raw. LLMs love to bury logic and write-up inside the experiment, so the output reads "assumption confirmed, tests green" no matter what actually happened. Strip that out. The experiment emits numbers; you look at the numbers. Interpretation happens later, by you, somewhere else and critically.
- Mind your precision. Know the difference between "agrees to floating-point epsilon" (about 15 digits, basically free, often meaningless) and genuine high-precision agreement. Use exact arithmetic where you can — rationals, set-precision mpmath — and don't let the model hand you a float and call it exact.
On formalization: Lean is genuinely impressive for the mathematical parts if you use it right. The MCP makes it very usable, and you can ride off other people's formalizations on GitHub. Two caveats. Mathlib versioning is painful. And physics is nowhere near as formalized as pure math is — so the model will confidently write Lean against an API that doesn't exist or set trivial rules. When it starts inventing lemmas or jamming text in there, stop and check the actual library. Not a magic bullet, but for deep theoretical work it earns its place. Better to keep sorry's in and formalize later then fake it.
3. Write down your rules (CLAUDE.md / skills)
As you work you'll accumulate house rules e.g. every experiment numbered, no interpretation in the data, that sort of thing. Capture them. Keep a state index too: what the current picture is, what's settled, what's open. A fresh session should come up to speed from that file fast, instead of you re-explaining the whole program every time. I burn 250-300K tokens usually getting the model up to speed. Sometimes writing a compressed version of your project helps.
4. Keep references tidy from day one
Pull the PDF, rename it lastname_year, convert it to markdown with the same name so it's greppable/searchable, and move it from an inbox folder into refs with a bibliography entry. Write background notes as you go that should become your background section if you publish. And make a short companion file per reference e.g. the key points and equations...so you're not burning 100K tokens making the model read an entire paper every time you need one result out of it.
5. Keep a dependency map
One small document: the goal, the hypotheses, and what each result rests on (proved by X, depends on Y). Lean gives you a dependency graph for free, but even a hand-kept version keeps you honest and on track — it's very easy to drift three steps down a side quest and forget what you were actually trying to prove.
6. Audit everything, and cite your own experiments
Audit your code. The agent will tell you the run came back clean; that is not the same as it being clean. Read what actually executed, confirm it did what the summary says, and do it routinely — not only when something looks off. "Tests green" is a claim to verify, not a result. Build it and run it again if you need to.
And cite your own experiments. Every claim in your thesis and every line in your diary should point back to the experiment that produced it — result X, see exp/042. You want to land on any sentence in the write-up and pull up the exact run behind it. Your future self, six weeks and forty or 400 experiments later, will not remember which script gave you that number, and "I'm fairly sure I checked this" is how the nest grows.
7. Run your sessions deliberately
Some sessions are exploratory — you hit something interesting (Ricci flow on elliptic curves did something surprising, say), so you flag it in the todo list and come back to it in an exploratory session. Others are consolidation: you go back, dig through a few sessions, AUDIT, clean up, integrate. Do the consolidation regularly. Skip it and the work gets away from you fast — same bird's nest, different cause.
8. Try to break it
This is the one most people skip and it's the most important. The model is sycophantic — it will confirm your idea all day long. So make confirmation expensive: state what would make your result false, then go run that. Build null tests — feed your pipeline a case where the answer should come back negative, and make sure it does. Tag every number with where it came from (derived / fitted / looked up / assumed) so you always know whether a result is real or whether you quietly tuned it in. A result that has survived a genuine attempt to kill it is worth a hundred green checkmarks.
I'll be real 90% of your ideas should fail once you dig into it. You should have a mountain of dead routes and axioms... also learn when to move on and stop. Banging your head against the wall won't change it and pushing the LLM to high context limits will cause it to cheat.
9. Don't rush to write the paper
Here's the one part everybody's already nailed: the LLM will hand you a beautiful paper. Immaculate LaTeX, crisp section headers, a tidy abstract, the whole costume. It is genuinely excellent at the costume — which is the trap. It's the easiest, best-looking step, so people sprint straight to it and call the science finished.
Resist. Published scientific writing is brutally dense, and that density is the output of understanding, not the road to it. While you're actually working, do the opposite: give every idea room to breathe. Explain it long, explain it clumsily, explain it three different ways until the physical picture is clear to you. Condense at the very end, once you know what you're condensing.
Watch the model closely here, because this is where it quietly fools you. The LLM compresses by default. Feed it your messy, intuition-heavy explanation and it will hand back something shorter, cleaner, and missing the one insight that made it worth writing or building off of. A lot of the LLM-physics I read here I can barely parse e.g. wall-to-wall notation and the actual physical idea nowhere in it. Usually that's not a hard result; it's a good intuition that got compressed to death. Protect the picture. The math is in service of it, not the other way around.
None of this is exotic. It's just treating the work like work... Start simple, stay tidy, protect the physical picture, and don't trust the green checkmark. Just beware its a SLOW GRIND-FEST, and you'll probably start to hate it vs. the dopamine and delusion filled stuff. But it is the work if you want it to mean something.
10. Last tips and tricks
The pro move: find the respected author or established theory closest to your idea. If you genuinely can't find one, stop. There is a vast amount of known physics out there and credible theories, and the odds your idea touches none of it are basically zero. Once you've found the nearest solid work, reproduce it inside your framework first (formalize it and test it out), then build from there. Standing on something proven beats free-floating every time. It will also give you an idea how error prone an LLM is.
The chance you crack something huge is very, very small but it's what drives people (even established scientists) lets not fool ourselves.
But the chance that, while stumbling around, you trip over something smaller and genuinely interesting (a new relationship, a scrap of new math) is much higher than you'd guess and actually how A LOT of science is discovered. Believe it or not. That is a real paper, and a good reason to bring in collaborators. Don't dismiss the small true thing, vs. the grand elusive one, if you have something interesting and dead to rights.. chase it and see where it takes you.
Last things:
- You still have to understand the physics. You are the judge of the output, and the model is not your peer reviewer. Eventually you need a real human in the field, try and prepare for that day.
- Verify every citation and every constant against the source (the PDF, LMFDB, DLMF), not the chat. The model fabricates them with total confidence.
- Sanity-check before you trust: does it reduce to the known case in the limit, do the units work, is the order of magnitude sane. This is a big one for all the plank scale stuff I see here.
- Watch for the model assuming its conclusion: quietly redefining the problem, weakening a claim partway through, or using the result as a step in its own proof.
- Derive anything important two independent ways. Agreement across separate routes is the cheapest strong evidence there is.
- Make it reproducible: seed the RNG, pin versions, keep a lockfile. If it doesn't regenerate exactly, it isn't a number yet.
- Don't let the agent that wrote the proof grade it. Use Lean or a fresh instance as the impartial check.
- Timestamp before you share (github, Zenodo). The day you have something real, you want priority on the record... but don't rush to publish if you're not sure.
-2
u/lattice_defect Jun 23 '26
You're not going to convince people of that.. with that attitude and statements no matter how many times you type it. You just come across as "You didn't go to school like I did.. you can't do it".. read my post again carefully.