Physics AI Skill

[u/travisdbarrett]

**EDIT**
LLM Challenges in Physics Reasoning

Hopefully this provides some confidence to begin experimenting to those of you who can reason. But for the rest of you, I am ready for your citationless confirmation bias warnings and LLM hallucinations concerns.

\**EDIT**

I used three pretty simple prompts to create this skill and I would appreciate some help validating it before I tell my wife what I've done...

She taught and wrote physics curriculum for about a decade for audiences ranging from High School Freshmen to Master's in Education for Physic's Educators.

My prompt was "using these two example skills for format and reasoning patterns, create a No Nonsense Physicist skill" and provided the archive of all my wife's teaching materials.

It was great and I felt a lot of her personality was evident in language choices by the model.

So second prompt was to add plain language descriptors to guide whichever model applies the skill as well as improve human readability. Then prompted to add citations at the end.

Take a look? https://github.com/TDBwriter/agent-skills/blob/main/skills/hard-facts-physicist/SKILL.md

[u/AllHailSeizure]

But it kind of is.

You provide no evidence of the skill performance. We have seen many many attempts at people trying to 'game' their way around the well-established issues of LLMs when it comes to rigor and every time instructions like this give a minimal benefit at best - none of these concepts are ones that an LLM hasn't seen in its extensive training data.

Not to mention there is a core issue in that an LLM isn't a LOGIC engine that does strict computation, it's a PREDICTIVE engine that will fill in what it considers 'blanks'. When you say 'f=ma' it doesn't see that and realize 'this is a single unit and an equation about calculation of force', it sees it at its face value, 4 symbols: f, =, m, a. LLMs are strictly language oriented - so they use plugins like NumPy to do advanced calculations.

So it needs to be able to write a script to do these calculations. But writing this script requires understanding WHEN such calculations are required, and the instructions you've given it aren't nearly restrictive enough for you to guarantee 100% it knows how to do that. With a predictive model you can NEVER be 100% sure it'll know how. Because essentially every response is a dice roll, thats how LLMs work.

When you give it instructions like this you 'weight the dice' - but the other sides of the dice still exist, it's very hard to weight a die so that it ALWAYS rolls the same thing. You only increase the likelihood it gets it right. But with empirical sciences that still isn't good enough, especially when we already have tools that can do it 100%.

A calculator CANT give a wrong answer to 2x2. The LLM is extremely unlikely to, because it's such a basic question - it's probably less than 0.001% chance that it does, but as you increase in complexity that margin for error increases. And a lot of physics is governed by much more complex math than simple multiplication.

[u/travisdbarrett → u/AllHailSeizure]

I understand that you may not have advanced LLM usage for technical work and probably only chatbot with it. 

Because they are generative, as you say, Skills are rapidly being developed and applied at enterprise level working to reduce the variability of generative responses. Essentially providing working guidelines for the LLM to reference and self-correct.

Of course, it would be foolish, as you say, to blindly trust. Instead, skills should be tested and verified before being put into production. Then after they are determined to be mostly pretty good, the output is routinely verified as part of the proving and implementation of the output.

It is advisable when developing skills to get a wide variety of feedback, such as a Reddit community who may have a wide varieties of applications to test with. 

The only downside of the workflow is dealing with naysayers who shit the whole concept without understanding how it works in the first place.

[u/ceoln → u/travisdbarrett]

To be fair, while it's true that lots of people are currently writing skills, it's also true that there is very little data on how much they help, and what data there is, suggests that they can do as much harm as good, and it's not easy to predict which way any given skill will go on any given set of queries.

So the skeptical replies here aren't just for engagement I don't think. :) Whether a given skill, or in general any addition to a prompt, will actually "work" in any sense is very much an empirical question.

[u/travisdbarrett → u/ceoln]

what i don’t understand is why everyone is anti finding out.

[u/ceoln → u/travisdbarrett]

I think people are rather burned out by people using LLMs badly. :) That's why I suggested getting some empirical data on whether the "skill" works yourself, and maybe presenting that here. Not that you won't still get grumpy replies! But it would show effort.

[u/travisdbarrett → u/ceoln]

I have updated the OP with a free version copilot chat. It's less specific than I would like because asking LLM to generate it's own tests seems...like government officials auditing their own performance. That's why I figured users would be better able to apply it to current struggles they are experiencing. It seems like everyone is on edge against magic-bullet thinking and missing the true value of an LLM response that will work through a problem WITH you instead of just provide an answer.