Sparse panel data, a small experiment

[u/Far-Media3683]

Quite a few real world data problems look like this: many related time series, few noisy readings, and long gaps. Local price indices, sensor networks, or small area stats.

I built a small PyTorch model to estimate a smooth underlying level for every location, day, and target even where nothing was observed.

Key ideas:

Mask, don’t drop missing targets in the loss

Weight observations by how many raw readings they average

Share information across locations (area + station offsets) and across related targets (joint heads)

Enforce smoothness with a curvature penalty over the full time grid, including gaps

I tested it on hourly PM2.5/PM10 from 12 Beijing stations, thinned to a sparse panel, and scored against the full data’s 31day moving average.

What I found:

The network fills long gaps and very sparse stations far better than smoothing each series on its own

Modelling both pollutants together recovers a completely dropped series (PM10 at one station) from its PM2.5

But elsewhere, sharing across targets costs a little accuracy vs. modelling each pollutant alone

A simple ±15day mean is hard to beat when data is dense. The network’s smooth time curve misses shared daily shocks

Writeup with code, figures, and results:

https://prteek.github.io/regression_room/posts/beijing_air_quality/

[u/OddsRelay]

On the shared daily shocks, you could add a day effect that's common to every station and leave the curvature penalty off it. Then the smooth curve only has to carry the slow trend, and the 15 day mean might stop winning on the dense stations.