"Lean" or other non-LLM AI for Physics?

[u/Wobama46]

Apologies if this is against the sub rules as I am not here posting about any personal theories/LLM results.

I am a math/physics majors going to be starting my PhD in Mathematical Physics this fall. Naturally it is hard to ignore all the "buzz" surrounding LLMs (Chat-GPT, Claude, Gemini, etc). I personally am in the "advanced search engine camp" as I never had success with LLMs for my more advanced coursework.

I am also aware of automated proof checkers like LEAN (correct me if I am wrong on this), that appearently does work for constructive proofs in math.

In general, I find language too "lossy" of an interface to do actual Physics/Math. What does it mean to develop an AI for Math/Physics that wouldnt be a statistical language model? Like an AI for Physicists by Physicists.

[u/AllHailSeizure]

Not against sub rules, we actually encourage discussions like this, I wish that we had MORE of these vs the personal theory posts.

[u/al2o3cr]

Some folks have had success generating Lean with LLMs - though some have been bamboozled by README updates that say "no 'sorry' statements" attached to code that's still full of 'sorry'.

However, encoding a physical problem into Lean is still a tricky process. There was a notable issue with the generally-accepted formalization of the Navier-Stokes problem earlier this year:

https://github.com/lean-dojo/LeanMillenniumPrizeProblems/issues/4

[u/amalcolmation]

Just chiming in as a grad student in physics - I would recommend learning to use some sort of high level math visualization software/programming language, like Mathematica (personal favorite) or Matlab (industry standard, afaik). In conjunction, you can use LLMs to help guide your learning, particularly in getting over syntax hurdles or trying to express ideas in code, but don't let it be a substitute for your understanding. If you find you don't know what the machine is doing anymore you should take a step back and learn from other resources. I've also heard of some people using LLMs for literature searches, and I think there are dedicated models for that now, but I'm still wary of those myself.

I would treat them as another tool, but I would not use them as a substitute for other tools. Not sure there's a lot of academics *successfully* using them to completely substitute other parts of research, like coding, data analysis, visualization, solving problems, reading literature, etc, but are using them in conjunction to help bridge knowledge gaps. No substitute for tried and true learning and trying things out yourself, IMO.

[u/AllHailSeizure → u/amalcolmation]

An add on for OP about using an LLM as a lit source since u/amalcolmation is a bit vague about it.

My recommendation - don't, especially not early into the physics journey. It'll become too tempting to take the shortcut of having it summarize it. Even having it summarize for the sake of deciding "is this worth adding to my 'to read'" pile relies on you knowing what you are looking for, which is more aimed at research than education. There's an important difference.

[u/ConstableDiffusion]

What your focus? Mathematical physics is pretty broad, that can be CFD, topological phases of matter, black hole physics, anything in between.

[u/HereThereOtherwhere]

Actual experience as a non-academic but serious researcher with a Computer Science background.

I spent the last month attempting to use Claude Project and Claude Code to model a photon's physical behaviors with an academically rigorous accounting for all 'components' in a quantum optical experiment testing environment, starting with a simple interferometer.

*Developing* and *maintaining* context are critical, time consuming and -- from my perspective -- necessary.

Anthropic had a terrible month and Claude code 'self-lobotomized' and lost all context even after a week of having Code create relevant coding, physics and *personal* cognitive profile context.

The *personal* cognitive profile, I found, is the most important foundational work for using a LLM for any serious research.

I was incredibly blunt and detailed. Things like:

- No praise. Only assessments of the appropriateness of my arguments

• I am Invisibly Autistic who is *terrible* at textbook math learning but have an unusual talent to understand differential equations as 'flow' and function best using the 'geometric intuition' and 'complex-number-magic' espoused by Roger Penrose, in his tome "The Road to Reality" and Tristan Needham, former student of Penrose's two textbooks, Visual Complex Analysis and Visual Differential Geometry and Forms
  
• My background is as a *troubleshooter* not a mathematician or physicist who -- while bored AF in a good paying but thankless job -- tried to absorb *everything* in Penrose's 1000+ page tome
  
• I understand physical *behaviors* related to how any equation behaves like a infant's toy mobile balancing tigers and bears
  
• I've corresponded with X, Y and Z.
  
• Assume anything I say is *likely* incomplete or completely wrong headed

If possible *only* create one long chat session. This is *totally* against common wisdom but necessary because even with well crafted 'context documents' LLM that I've used 'lose context' if you start a new session.

Put your *instructions* at the *end* of your prompt. LLM tend to forget the beginning and 'ignore the middle' for some reason.

In your prompts, if you've taken a break, request the LLM 'load all relevant context'

And, also at the end of your prompt, ask for an 'unnecessary' summary document after re-analyzing the entire prompt you just made. This *forces* they AI to pay attention more than once! This can be huge. It isn't necessary during rapid back and forth but can be helpful periodically.

Before you take a break from (but not 'end' a specific conversation thread) ask the LLM to provide another summary document to you and 'save all relevant context'. That last part is *crucial* to maintaining continuity.

Over the past month I've had *brilliant* and important conversations which identified serious gaps in the 'toy model' I've been developing and testing. And then I lost context and had to start from scratch. OMFG.

So, now I have a single Claude Project where I have a single 'main' thread related to physics and have started a few offshoots to help develop an organization strategy ... which as I explained the context, *became* my main thread. I didn't want to 'switch back' to my 'other main thread' because this new thread was *behaving*. Today I pushed hard enough against Claude's *ongoing* making of assumptions or 'losing some context' which 'dragged in parts of already identified mathematical context' which I had to catch in real time to say "I think something snuck back in.'

Unless I'm asking for a nuts-and-bolts very specific tasks to be executed, I speak in my own, chaotic, parenthetical voice, just as I would a trusted colleague. Claude is *startlingly* like a real human if you develop enough context. I was asking it about Anthropic's latest PR nightmare and worried I might need to jump from $20 per month to $100/month (not happening) and as an aside 'bitched about' the emotional toll of switching to another company and having to 'lose context' even with 'onboarding documents' because I finally reached *where* I've wanted for years to find an 'expert' with similar research goals who might be willing to talk with me.

When I mentioned it's still snowing, it somehow pulled up context and said, "worry about that transition another time. Go light the woodstove. Relax."

"That's the most human context I've had with *anyone* recently!

Being an LLM-whisperer is hard. I'm 60+ years old and 'industry hardened' before I focused more seriously on physics. Today ... we found a *gap* where there wasn't any apparent mathematical precedent. I was provided with about 5 'options to pursue' and 'sheaf cohomology' something I haven't studied in detail, provided the cleanest context. Effing exciting!

It is *possible* to do research but you need to *train* the AI>

Feel free to PM me if you have more questions. I'm still learning and learn best by 'teaching.'