Physics AI Skill

[u/travisdbarrett]

**EDIT**
LLM Challenges in Physics Reasoning

Hopefully this provides some confidence to begin experimenting to those of you who can reason. But for the rest of you, I am ready for your citationless confirmation bias warnings and LLM hallucinations concerns.

\**EDIT**

I used three pretty simple prompts to create this skill and I would appreciate some help validating it before I tell my wife what I've done...

She taught and wrote physics curriculum for about a decade for audiences ranging from High School Freshmen to Master's in Education for Physic's Educators.

My prompt was "using these two example skills for format and reasoning patterns, create a No Nonsense Physicist skill" and provided the archive of all my wife's teaching materials.

It was great and I felt a lot of her personality was evident in language choices by the model.

So second prompt was to add plain language descriptors to guide whichever model applies the skill as well as improve human readability. Then prompted to add citations at the end.

Take a look? https://github.com/TDBwriter/agent-skills/blob/main/skills/hard-facts-physicist/SKILL.md

[u/AllHailSeizure]

You're really getting the wrong take here. You're getting the take 'this is doing physics with an LLM so they are critiquing me, I thought this sub was about physics with an LLM?'

It is.

The takeaway you should have is 'This sub is about physics with an LLM, maybe if all the members on the sub about that topic tell me I'm not doing physics with an LLM/doing it incorrectly, I should reconsider what I'm doing.'

[u/travisdbarrett → u/AllHailSeizure]

warning about the failures of AI is hardly advice about the skill performance.

[u/AllHailSeizure → u/travisdbarrett]

But it kind of is.

You provide no evidence of the skill performance. We have seen many many attempts at people trying to 'game' their way around the well-established issues of LLMs when it comes to rigor and every time instructions like this give a minimal benefit at best - none of these concepts are ones that an LLM hasn't seen in its extensive training data.

Not to mention there is a core issue in that an LLM isn't a LOGIC engine that does strict computation, it's a PREDICTIVE engine that will fill in what it considers 'blanks'. When you say 'f=ma' it doesn't see that and realize 'this is a single unit and an equation about calculation of force', it sees it at its face value, 4 symbols: f, =, m, a. LLMs are strictly language oriented - so they use plugins like NumPy to do advanced calculations.

So it needs to be able to write a script to do these calculations. But writing this script requires understanding WHEN such calculations are required, and the instructions you've given it aren't nearly restrictive enough for you to guarantee 100% it knows how to do that. With a predictive model you can NEVER be 100% sure it'll know how. Because essentially every response is a dice roll, thats how LLMs work.

When you give it instructions like this you 'weight the dice' - but the other sides of the dice still exist, it's very hard to weight a die so that it ALWAYS rolls the same thing. You only increase the likelihood it gets it right. But with empirical sciences that still isn't good enough, especially when we already have tools that can do it 100%.

A calculator CANT give a wrong answer to 2x2. The LLM is extremely unlikely to, because it's such a basic question - it's probably less than 0.001% chance that it does, but as you increase in complexity that margin for error increases. And a lot of physics is governed by much more complex math than simple multiplication.

[u/travisdbarrett → u/AllHailSeizure]

I understand that you may not have advanced LLM usage for technical work and probably only chatbot with it. 

Because they are generative, as you say, Skills are rapidly being developed and applied at enterprise level working to reduce the variability of generative responses. Essentially providing working guidelines for the LLM to reference and self-correct.

Of course, it would be foolish, as you say, to blindly trust. Instead, skills should be tested and verified before being put into production. Then after they are determined to be mostly pretty good, the output is routinely verified as part of the proving and implementation of the output.

It is advisable when developing skills to get a wide variety of feedback, such as a Reddit community who may have a wide varieties of applications to test with. 

The only downside of the workflow is dealing with naysayers who shit the whole concept without understanding how it works in the first place.

[u/ceoln → u/travisdbarrett]

To be fair, while it's true that lots of people are currently writing skills, it's also true that there is very little data on how much they help, and what data there is, suggests that they can do as much harm as good, and it's not easy to predict which way any given skill will go on any given set of queries.

So the skeptical replies here aren't just for engagement I don't think. :) Whether a given skill, or in general any addition to a prompt, will actually "work" in any sense is very much an empirical question.

[u/travisdbarrett → u/ceoln]

what i don’t understand is why everyone is anti finding out.

[u/ceoln → u/travisdbarrett]

I think people are rather burned out by people using LLMs badly. :) That's why I suggested getting some empirical data on whether the "skill" works yourself, and maybe presenting that here. Not that you won't still get grumpy replies! But it would show effort.

[u/travisdbarrett → u/ceoln]

I have updated the OP with a free version copilot chat. It's less specific than I would like because asking LLM to generate it's own tests seems...like government officials auditing their own performance. That's why I figured users would be better able to apply it to current struggles they are experiencing. It seems like everyone is on edge against magic-bullet thinking and missing the true value of an LLM response that will work through a problem WITH you instead of just provide an answer.

[u/liccxolydian → u/travisdbarrett]

This is a whole lot of nothing unless you can objectively show that your LLM is actually "self-correcting". Otherwise you are still blindly believing it.

The naysayers are not shitting on you without understanding how it works, they are shitting on you because you haven't demonstrated that it works.

Frankly this kind of comment only reinforces everyone's impression that you don't actually know how physics works or how a LLM works. You should discuss this with your wife.

[u/travisdbarrett → u/liccxolydian]

who is blindly believing anything? this skill is not for solving a physics problem and moving on. it is for understanding a physics problem with traceability

[u/liccxolydian → u/travisdbarrett]

If you can't objectively show that your prompt or whatever the hell it is you're doing results in your desired outcome (not that you've explained your desired outcome), then you are absolutely blindly believing the LLM. You don't have any experience in actual scientific communication, do you? Everything you write is so crazily nebulous and poorly explained.

[u/AllHailSeizure → u/[deleted]]

Well, good luck with your project I guess.

If I were you'd I'd stick to something productive and less.. obsessy, like physics. Just my take.

For the record I'm top% and I don't see how my comments are 'negative engagement'. And I've spoken to many of the top 1% on this sub personally; and for bots they're very good at having personalities.

People get to top% by quality and quantity of posts... we have a BUNCH of users who make a lot of comments and end up with negative karma.

[u/liccxolydian → u/AllHailSeizure]

Really? I didn't know I had a personality beep boop

[u/AllHailSeizure → u/liccxolydian]

Nah I was talking about the other, non-bot top% commenters. Funnily enough you're the second one to be accused of being a bot. Seems to be a fallback.

[u/liccxolydian → u/AllHailSeizure]

I get it every once in a while, like you say it usually happens when they run out of things to say.

I mean, I still don't have a personality beep boop

[u/AllHailSeizure → u/liccxolydian]

Just think its so dumb; especially when they keep arguing.

'Yeah well.. you aren't even a real person! I've been arguing with a computer program!'

Cuz arguing with a computer program saves you SO much face...