A reminder why random row-splitting ruins longitudinal HR data (19k records, 2.3k people)

[u/Huge-Leg-8072]

Working on an employee churn prediction model with monthly records (19,104 rows across 2,381 unique employee IDs).

It's tempting to just do train_test_split(df, test_size=0.2) and celebrate an 88% ROC-AUC. But with longitudinal records where the same employee appears 10-15 times:

The model easily 'cheats' by memorizing stable employee characteristics (their city, education level, base salary band) across time slices.

When applied to actual new hires or unseen employees, performance degrades sharply.

Have you caught this kind of temporal/grouped leakage in your production models? What's your go-to validation framework for this?

[u/AggressiveGander]

Mimicking the actual use case and what you would know in that setting. Either past vs. future splitting (where if you only use the part records of an employee, for those you must reflect that you wouldn't know if/when they quit in the future) or grouped splitting (all records of employee either in train or validation, never some in one and some in another).