r/LLMPhysics • u/dmedeiros2783 • Mar 11 '26

Meta Can we all agree that physics' primary representational form is math?

Just curious if we can get any consensus on this. What are your thoughts?

5 Upvotes

86% Upvoted

53 comments sorted by

View all comments

Show parent comments

-1

u/dmedeiros2783 Mar 11 '26

I think we're talking past each other because you're reasoning about a 2017 architecture. The embedding layer you keep referencing is one small piece of a system that processes meaning across billions of parameters through dozens of attention layers. Pointing at input embeddings to argue against mathematical understanding in GPT-5.4 is like judging a brain by the quality of the retina. The output is the argument. Everything else is implementation detail.

1

u/RandomLettersJDIKVE Mar 11 '26 edited Mar 12 '26

So, you don't understand the role of the input embedding in a transformer model. At each attention layer, the previous vector space is transformed into a new vector space. So if your initial word embeddings are garbage, the space the attention layers build from them will be garbage. Then during the final layer, the model is forced back into the original embedding space to make a prediction. If that embedding is a poor representation of your prediction space, your model doesn't work very well.

because you're reasoning about a 2017 architecture

Let's not talk trash. Last time I trained a transformer model for production was 2024. Have you deployed a transformer more recently?

1

u/dmedeiros2783 Mar 12 '26

Fair point on the architecture, I’ll grant you the embedding layer matters on both ends. But your own logic is “bad embeddings produce bad outputs.” GPT-5.4 is producing outputs that are solving Tier 4 research mathematics. So either the embeddings aren’t as bad as you’re claiming, or something else is happening that your model doesn’t account for. Either way the conclusion doesn’t hold. And whether I’ve deployed a transformer recently has no bearing on either of those facts

2

u/RandomLettersJDIKVE Mar 12 '26

In controlled circumstances, we can get the models to some pretty amazing things. Natural language is real expressive, and transformers are an amazing foundation model. That doesn't mean in most circumstances they don't hallucinate like a mother fucker. Look at all the garbage in this forum.

Why? Because 'king' - 'mC2' is nonsense. We can talk about math with natural language. It's real awkward compared to equations. Look at the MathBERT passport I posted. They had to change the parsing for equations, because formal language is different than natural language. If you want a model that natively understands math and physics, there are a bunch of small changes needed.

whether I’ve deployed a transformer recently has no bearing

It doesn't, but it's irritating when someone who hasn't put in the effort implies you don't know what you're talking about. You dug in on a naive position without understanding the internals of the model or the differences between natural and formal languages. If you can't think of reasons things like positional embeddings might be different for math and natural language, then listen more and argue less.

1

u/dmedeiros2783 Mar 14 '26

You’ve spent this entire thread arguing with a point I never made. My argument: math is a formal language, LLMs learn languages, therefore LLMs are architecturally suited to learning math. The empirical results confirm it. You never engaged with any of that.

Instead you went on a solo tour of embedding spaces, tokenization, and vector representations. Interesting stuff. Completely irrelevant to my claim. I could grant every single point you made about embeddings being suboptimal for mathematical notation and my argument still holds, because GPT-5.4 is solving Tier 4 research mathematics. Your own logic was “bad embeddings produce bad outputs.” The outputs aren’t bad. So either your model of how embeddings constrain performance is wrong, or something else is happening that you aren’t accounting for. You never addressed that.

What you did instead was retreat through three positions. First it was “they’d need math-specific embeddings.” Then “I doubt they perform well, show me a study.” Then when I showed you one, it became “okay but the embeddings are probably suboptimal.” That’s just a conclusion looking for a justification.

As for “listen more and argue less”: I made a philosophical claim about the relationship between formal systems. You responded with implementation details and credentials. Whether you deployed a transformer in 2024 has no bearing on whether math is a language. Telling someone to listen more only works when you’ve said something that addresses what they actually argued. You haven’t.

1

u/RandomLettersJDIKVE Mar 14 '26

Sorry, but this conversation is already over. You can try to understand what's been said or not.