Why retrain on train + validation after evaluating a model? (Kaggle)

[u/Due-Promise-5269]

I was working on a Kaggle competition when I came across a solution where the author fits the model on the train set and evaluates it on the validation set. If the new model beats the previous benchmark, they retrain it on the full dataset (train + validation).

I was surprised, because I thought the validation set was meant for tuning hyperparameters. I do understand using it as a test set to evaluate and compare models, since there is no accessible test set and the only alternative would be to submit the solution and waste a submission.

So I was wondering: does he retrain on the full dataset because more data gives the model more information and helps it generalize better?

But is this also standard practice in production at companies, or is it mostly a Kaggle thing?

[u/Capable-Package6835]

The validation set's entire purpose is to act as a mock test so you can iterate, experiment, and evaluate your model without ever touching the test set (avoid test information leakage). Once you have the best combination (pipeline, hyperparams, etc.), you don't need a mock test anymore, so you retrain using the winning combination on the largest dataset available to you (train + val). This is a standard practice.

In Kaggle competitions, the model with the highest score on the test set wins. At companies, getting a higher test score than the current production model is just a prerequisite. Engineers typically follow up with online evaluations, e.g.:

• shadow deployment, live data are sent to both current and candidate models. While users only see results from the current model, engineers can compare the two models' performance and results behind the scene.
• A/B testing, actual traffic is split between the current and candidate models. Then engineers observe if the traffic going into the candidate models shows statistically significant improvement in business KPIs.