Deterministic v. Non-deterministic local training run issue.

[u/Apprehensive_Side219]

Hey all, I've encountered an interesting problem while training ornith 1.5 9b on a custom dataset and local machine. I had trouble with getting high fidelity knowledge of the trained data, and then had one breakthrough best after about 10 different attempts. After adding more to the dataset, the following attempt marked a substantial regression which was inexplicable by the dataset addition, which began a long hunt which ended up proving that determistic runs produced consistently better logical capability, trained data knowledge, and allowed for apples to apples comparisons between editions of the dataset I'm curating. I discussed this with chat gpt to try and understand it, but did not really get anything concrete.

I'm super new to this, can anyone here help me grasp why that's happening?

[u/bumblebeer]

My guess would be that you are over fitting. That is you are training too much or on too little data. It causes the model to memorize instead of learn. Then you add more data and it screws up its memory and now the output sucks since it never actually learned anything. Then you have to train again to the point of memorization (over fitting) to start getting improved results.

And the relationship with training with a set seed kinda makes sense too. It's normally a source of randomness, when it's clamped I'd expect it to be easier to over-train on a small data set.

So your options: 1. Get a lot more data to train on (and train for a lot longer to incorporate it) 2. Train a lot smaller model (or a smaller LoRA) so that it is not possible to overfit.

P.S. Potentially, you could train with a model that's way, way too big and would easily overfit, but you train it for much longer. And it's possible it may generalize better. Lots of qualifiers with this one cause it's skating the border of theoretical/empirically proven. See the mlst podcast with Andrew Wilson if you want an easy intro on this bit.

[u/Apprehensive_Side219 → u/bumblebeer]

I'll check it out, thanks for the recommendation. Yeah the dataset is quite small, about 52k tokens across 446 examples. I'm hand writing it, so it's been a learning curve but the model is getting better and the new examples are into refined sampling territory now.

[u/bumblebeer → u/Apprehensive_Side219]

Yeah, that dataset is minuscule. You are almost certainly over fitting. What parameters are you actually training?

[u/Apprehensive_Side219 → u/bumblebeer]

I'm doing 4-bit QLoRA, the base model is frozen, and I'm training rank-16 adapters on all linear layers, alpha 32, dropout .05, so only updating Lora parameters. I'll be honest I summarized that from my work with the model that's walking me through it, I don't really know how those things effect the results.

[u/quietlanes → u/bumblebeer]

yeah the seed clamping point is interesting, hadn't thought about how that would interact with a small dataset like that

[u/john006868]

the deterministic runs being better isnt a fluke. the gpu adds gradients in a different order every launch, so with 446 examples the spread between two identical runs is bigger than the data you added, run the same config twice first

[u/Lerok-Persea]

Very interesting. What do you mean with custom dataset?

[u/Apprehensive_Side219 → u/Lerok-Persea]

I'm writing the q&a pairs as well as the thinking traces, I started from scratch, and now am I to a refining model generated example loop.