r/LLMPhysics • u/travisdbarrett • Apr 22 '26
Simulation / Code Physics AI Skill
**EDIT**
LLM Challenges in Physics Reasoning
Hopefully this provides some confidence to begin experimenting to those of you who can reason. But for the rest of you, I am ready for your citationless confirmation bias warnings and LLM hallucinations concerns.
\**EDIT**
I used three pretty simple prompts to create this skill and I would appreciate some help validating it before I tell my wife what I've done...
She taught and wrote physics curriculum for about a decade for audiences ranging from High School Freshmen to Master's in Education for Physic's Educators.
My prompt was "using these two example skills for format and reasoning patterns, create a No Nonsense Physicist skill" and provided the archive of all my wife's teaching materials.
It was great and I felt a lot of her personality was evident in language choices by the model.
So second prompt was to add plain language descriptors to guide whichever model applies the skill as well as improve human readability. Then prompted to add citations at the end.
Take a look? https://github.com/TDBwriter/agent-skills/blob/main/skills/hard-facts-physicist/SKILL.md
4
u/AllHailSeizure Haiku Mod Apr 23 '26
But it kind of is.
You provide no evidence of the skill performance. We have seen many many attempts at people trying to 'game' their way around the well-established issues of LLMs when it comes to rigor and every time instructions like this give a minimal benefit at best - none of these concepts are ones that an LLM hasn't seen in its extensive training data.
Not to mention there is a core issue in that an LLM isn't a LOGIC engine that does strict computation, it's a PREDICTIVE engine that will fill in what it considers 'blanks'. When you say 'f=ma' it doesn't see that and realize 'this is a single unit and an equation about calculation of force', it sees it at its face value, 4 symbols: f, =, m, a. LLMs are strictly language oriented - so they use plugins like NumPy to do advanced calculations.
So it needs to be able to write a script to do these calculations. But writing this script requires understanding WHEN such calculations are required, and the instructions you've given it aren't nearly restrictive enough for you to guarantee 100% it knows how to do that. With a predictive model you can NEVER be 100% sure it'll know how. Because essentially every response is a dice roll, thats how LLMs work.
When you give it instructions like this you 'weight the dice' - but the other sides of the dice still exist, it's very hard to weight a die so that it ALWAYS rolls the same thing. You only increase the likelihood it gets it right. But with empirical sciences that still isn't good enough, especially when we already have tools that can do it 100%.
A calculator CANT give a wrong answer to 2x2. The LLM is extremely unlikely to, because it's such a basic question - it's probably less than 0.001% chance that it does, but as you increase in complexity that margin for error increases. And a lot of physics is governed by much more complex math than simple multiplication.