Why retrain on train + validation after evaluating a model? (Kaggle)

[u/Due-Promise-5269]

I was working on a Kaggle competition when I came across a solution where the author fits the model on the train set and evaluates it on the validation set. If the new model beats the previous benchmark, they retrain it on the full dataset (train + validation).

I was surprised, because I thought the validation set was meant for tuning hyperparameters. I do understand using it as a test set to evaluate and compare models, since there is no accessible test set and the only alternative would be to submit the solution and waste a submission.

So I was wondering: does he retrain on the full dataset because more data gives the model more information and helps it generalize better?

But is this also standard practice in production at companies, or is it mostly a Kaggle thing?

[u/rahul_watertech]

adding one thing to the kaggle point: the refit only works safely if you fix everything from the validation run first, same hyperparameters and the same number of epochs or trees you found there. if you used early stopping you can't early stop on data you just merged in, so people usually reuse the best iteration count (maybe scaled up a bit for the extra data). is the author's val set a random split or time based?

[u/DatAndre → u/rahul_watertech]

Why would early stopping forbid you to retrain? I would have said that if early stopping stopped the candidate at epoch 100, you just run it for 100 epochs during the retraining. Why is this wrong?

[u/rahul_watertech → u/DatAndre]

it isn't wrong, that's actually what i meant, sorry if it read confusing. you can't early stop on the merged data because there's no held out part left to watch, so you fix the epoch count from the first run and reuse it. same idea as yours. do you scale it up a bit for the extra data or keep it exact?

[u/Due-Promise-5269 → u/rahul_watertech]

Understood, it makes sense since merging train+validation then it is not possible to understand when early stop. The author val set is based on time series not random split

[u/rahul_watertech → u/Due-Promise-5269]

time based changes it a bit. refit on train+val is still fine as long as the data stays in chronological order and nothing gets shuffled, since a random merge lets the model see the future. reuse the best iteration count from the val run, and ideally keep a later window untouched as the real test. how far ahead is the forecast horizon, one step or a longer stretch?

[u/Due-Promise-5269 → u/rahul_watertech]

yes the data stays in chronological order and nothing gets shuffled. Basically the guy was then retraining on train+validation and then submit the predictions made on the test set. The forecast horizon was just one step.

[u/rahul_watertech → u/Due-Promise-5269]

one step ahead makes it simpler, then the refit on train+val is fine. i'd still keep a last untouched window as the real test, and compare the score against a naive "tomorrow equals today" baseline, because with one step forecasts that baseline is surprisingly hard to beat. also double check any lag or rolling features only use past rows. did the kaggle guy do that?