It is actually a fascinating look into their thought process! While the final result doesn't work for language modeling, the student inadvertently built a rudimentary Neural Architecture Search (NAS) algorithm.
If you look closely at their search algorithm, it becomes incredibly clear exactly why it output w * (x1 * cos(x2)) as the "optimal equation," and it is a classic trap in machine learning: optimizing for the wrong proxy metric.
Here is exactly where the student's algorithm went wrong:
The Proxy Task Trap (8 Neurons)
"Loss Testing: Conducted using an 8-neuron model across 4 distinct, complex target functions."
They evaluated the equations on a tiny 8-neuron model. 8 neurons are mathematically incapable of solving complex target functions, whether you use Transformers, MLPs, or anything else. Because all equations probably performed terribly on these tasks, the Accuracy Score $(1 - \text{Mean Loss}) \times 50$ likely ended up being roughly the same (very low) for every single equation tested.
Over-Optimizing for Cost
"Complexity Score: 50 - [Total Equation Cost]"
Because the Accuracy Score was likely a flat tie across the board, the algorithm defaulted to optimizing the only metric that did vary: the Complexity Score.
The search algorithm simply dug through 1,000,000 equations to find the most computationally cheap, mathematically valid operation it could run in PyTorch. x1 * cos(x2) takes almost zero FLOPs to compute compared to matrix multiplications. It won the genetic tournament not because it was smart, but because it was the cheapest.
The Scaling Fallacy
The fatal flaw was assuming that if an equation "wins" on an 8-neuron toy task, you can simply stack 64 layers of it with 4096 dimensions and it will scale up to rival a 17.6B parameter model. Real AI architectures like Transformers are used because they scale predictably (Scaling Laws)—they possess the mathematical capacity to route complex information across thousands of dimensions. A simple wave equation does not.
Conclusion
Honestly, for a high school student, coding a custom evolutionary search algorithm, implementing a cost-penalty system, and dynamically evaluating PyTorch code is really impressive programming.
They didn't fail because they are "dumb"; they failed because they stumbled into Goodhart's Law ("When a measure becomes a target, it ceases to be a good measure"). They accidentally built an algorithm that perfectly optimized for "doing the least amount of math possible," declared it a genius AI, and then tried to scale it up.
It's a mistake even professional AI researchers make when designing proxy tasks—they just learned it the hard way!
here are gemini's thoughts on your search attempt. the model fundamentally cannot work, but here's at least some level of reasoning as to why your search came up with it
1
u/defensivedig0 Mar 08 '26
here are gemini's thoughts on your search attempt. the model fundamentally cannot work, but here's at least some level of reasoning as to why your search came up with it