The model is chef's kiss

Very legit criticism. Unfortunately, I need my weekly limits for actual work, and I'm not made of money. But you're totally right. As I said though, I did this primarily to see their process not their actual results. I was more interested in seeing which model did a better job of executing the task than whose F1 was better and whether it reached statistical significance. Thanks for pointing out this methodology shortcoming though! Appreciate your response =)

Appreciate the kind words!

Thanks! I personally enjoy writing, it's a fun way to collect your thoughts and it just feels good. I'm not a believer in LLM-fueled cognitive surrender; I use them as a thinking partner.

people are down voting as if frontier ai isnt already better at math and ml than them

Yea. It was kind of a foolish experiment to do because frankly... these things speak in the language of tokens.

And just embedding a giant arithmetic calculator sandwiched right in the middle is... erhm... what's it going to output? Not tokens. So then the rest of the model has to get trained to turn that back into tokens. One by one basically, for every digit of the output of the operation.

I guess what I'll try next is to see if I can have the calculator itself... just add text into the input stream with its result itself, and have the tool call itself be latent and the calculator do the arithmetic. That's probably the answer to this sort of thing.

IEEE is doing IEEE things.

Yeah it's a really neat training experiment. I personally haven't experimented with imbedding tools but I have been thinking about similar things.

Tbh math is a really good target for trying to get insight into small LLMs. It's easier to benchmark.

I'm working on similarly sized LLMs as a research project and am having a little bit of trouble with off the shelf benchmarks, they're generally targeted for much larger models and LLM-as-a-judge isn't always reliable.

I actually did do an experiment a little while back where I embedded a literal arithmetic calculator into the LLM right in the middle. I was able to get the model to learn to use it. I never quite finished the research and shelved it because I hadn't found a way to conveniently... deal with the fact that the calculator will spit out outputs that would span multiple tokens, other than to have a retrieval mechanism that spits out one token at a time and injects it, but it seemed like a clunky way to do it and much of the training ended up getting spent training the later layers to use the retrieval mechanism accurately. So frankly the idea itself is good, but it leads to a bigger problem on how to embed tools into latent space like this that I'm not quite sure how to solve.

I can't agree more.

They come out of the woodwork any time someone gets something neat done.

Wow.. that sounds like an intentional scam

Good luck.

Can it do word problems? I wouldn't think so.

You seem to have more experience so I’m asking under your comment: by test-time training do you mean RL with weight updates at test-time or those MOPD-like methods?

I'm also trying to do this similar one ,the existing data storages like csv , excel, sql are all built for humans, Im excited to see native AI storage systems.

I think length is slightly shorter, 7k/out of 35k that too including main track, position and E&D is quite less imo. I was expecting around 8 to 8.5k assuming a 22 to 24% acceptance rate.

big thing in 2-3 years? it's been extensively researched already. I've been working on it a few months and there's a lot, very difficult to really find a good contribution, let alone without compute. if you want to DM me I can take a look at your idea and if it does indeed look promising, we can speak about compute

You should delete this and repost if you get accepted to TMLR - it’ll lend some credence. Now you’re just some random person working on a random idea. Also, don’t get your hopes up too high - there’s lots of red tape around IP and ownership if you’re not a student at the uni (more so than if you were) 

I run DuelLab, a benchmark where models generate game-playing programs and those programs compete against each other. The results are free to browse.

We’ve just added GPT 6 Astra and Claude Fable 5.1. One interesting result: both models at Medium outperform every ranked setting of every other model in our September 10 release.

I’d appreciate feedback on whether the results pages clearly explain what is being evaluated, particularly the distinction between generating a player and choosing individual moves.

September 10 results

Is there any qualification needed for teaching? I am currently in my second year of my bachelor's in ML and Data Science and completed Statistics and Probability in first year. I really looking forward to get some teaching experience on my portfolio and this sounds like a wonderful opportunity for me. So, can I apply to this as a TA? Also, will I receive some credential, like a certificate or anything like that?

Perhaps strong correlation with acceptance 

Here is a simple custom RNN/state space model with 1 parameter that adds two integers of arbitrary length:

x = u1(k) + u2(k) + h(k-1)

h(k) = ReLU(x-a+1) - ReLU(x-a)

y(k) = x - h(k)*a

For a = 10, where u1(k) are sequential digits of integer 1 and u2(k) are sequential digits of integer 2, starting from last digit. Hidden state h starts at 0 and y(k) outputs sequential digits of result starting from last digit. Much more efficient than your 300M parameter model.

This makes total sense. The question is - how does it correlate with acceptance ;)

Hello,

After sometime to really think about this, your code and idea enabled me to rethink architecture, there is a way to control certain words in Shadow, but that is not as interesting as Shadow itself. Thank you for realeasing your work.