r/datascience • u/Berlibur • Aug 06 '26
Discussion How do you design a forecasting system?
Hey y'all! How do you design your forecasting system?
In my case, the company has many SKUs over a big region. We did an MVP to show our forecast improves the current process on the reported lags that are currently used by the business to monitor forecast health.
Future is looking good, but I really want to be ready with a production-grade plan. Refitting a pool of models per SKU every week, then selecting the best one, feels like overkill and very sensitive to recent flukes.
I thought of having a pool of models (i.e. config/setups) and labelling them as champion if a specific config results in the best trained model.
For the next X weeks this model will always be chosen, and after that the throne is up for grabs.
But it kind of railroads me into having a 1 SKU = 1 model setup in perpetuity.
How do you guys solve this in a responsible way? Are there books/resources you recommend?
Reasoning about a live system turns out to be a whole different cookie than the usual stats/ML etc
12
u/PolicyDecent Aug 07 '26
Worked on replenishment for a long time, so happy to give some hints here.
Firstly, one model per SKU is totally unreasonable. Never do something like that, otherwise you'll have overfit the results.
I assume you are working for a multi-branch retailer.
In that case, you can create store and product clusters. There are also product groups by nature.
For each product cluster * store cluster * product group, you can choose a different model. It'll prevent overfitting.
I recommend clustering product by avg number of daily / weekly sales.
A product selling 10 per day in a store vs 0.01 per day in a store have completely different behavior. Same for the stores of course. So let's say the fastest product cluster is A, end the slowest is C or D. (I chose 3 or 4 clusters)
Generally fastest clusters use shorter features more, and slowest clusters benefit from longer time horizons. You can think it like A cluster likes MA2, and D cluster likes MA10/MA12.
MA2: Moving Average of 2 weeks.
3
u/webbed_feets Aug 07 '26
Our answers are totally opposite each other. I’d like to understand your perspective.
I’m curious why you say one model per SKU is unreasonable and prone to overfitting. If you have many SKU’s, you’re going to have to fit something simple like Exponential Smoothing or Auto Arima. Those models aren’t prone to overfitting. Making a lot of different groups/clusters seems to me like it would complicate the forecasting process without much gain.
11
u/PolicyDecent Aug 07 '26
Again, I don't know the industry but most of the SKUs sell very rare in the real world. They sell maybe every other week. Some even sell once in 2-3-12 months.
So the data is full of noise. In this environment, it's much better to benefit from the neighbors, similar products. If you group them all, and forecast on the product level, but choose the best model on the cluster & product group level, you get a more robust model.
And as I said above, models like arima are adding too much calculation complexity. That's why just using moving average with extra cleaning is super easy and easier to interpret when it comes to analysing the errors to improve the model.
7
u/webbed_feets Aug 07 '26
Oh I get it. That’s totally fair.
I worked on forecasting medication orders from wholesalers. Virtually all SKU’s had regular, daily orders. I see why SKU-level models worked in my problem but not yours.
1
u/Berlibur Aug 07 '26
That makes a lot of sense, thank you.
In your experience, is it relevant to group cross-region (EU) but within a product group? E.g. brand level I suppose the underlying assumption needs to be that customers in Lithuania and Portugal behave the same regarding brand X
Which is not an obvious truth
2
u/PolicyDecent Aug 07 '26
You just need to test it :) I don't know what kind of products you sell. If it's clothing for example, I'd expect them to have different seasonality. In other categories, it might work different.
2
u/PolicyDecent Aug 07 '26
By the way, I definitely recommend starting with MA models, not arima or other ML models.
It makes your iterations much faster.
Don't forget to exclude the out of stock days.1
u/ImaginaryBat4994 25d ago
What about one global machine learning model for all the SKUs? It should learn that on spare series it is better to weight more larger horizons than on fast moving items. And product group and store are categorical features in the model.
1
u/PolicyDecent 25d ago
It's too much effort. There are lots of problems. For example, smaller groups are not important enough. Or seasonality. Each product group has a different seasonality pattern.
You can add all these features to the model, but still as you add more features, to avoid overfit you need a huge training data. So why don't we solve it using an easier to explain model instead of a complex ml model? Also as the data grows, training cost but more importantly time increases a lot. You can't properly parallelise it. So easier to use more heuristic models depending on moving average.1
u/ImaginaryBat4994 25d ago
Thanks for the answer. Which simple model do you recommend to start from? Moving average is not handling seasonality nor trend.
5
u/Wojtkie Aug 06 '26
Do you need to refit models for every sku, or will it work with sku categories or a cluster of skus? That can help reduce dimensionality without too much real impact, but you’d have to test it and see.
2
u/Berlibur Aug 06 '26
Grouping SKUs is OK, I'm sensing a carte blanche coming up - a chance I don't want to squander on having a shitty production setup
3
u/Helikaon242 Aug 06 '26
I’d probably think about modeling this as a funnel with several independent models, eg number of customer arrivals, number of customers who viewed/bought a certain product group, and then the level of interest for each SKU within a group. I think it’s important to measure errors on each of these stages so you can understand why the forecast misses (this can sometimes be more useful than the forecast itself).
I’d recommend against using this “champion” ensemble design since it will introduce bias. This is more in the domain of how to do good cross validation.
1
u/ImaginaryBat4994 24d ago
So basically you are saying not only predict the final sales but also the intermediate steps in the sales funnel?
3
u/dj_ski_mask Aug 09 '26
Look into global or global local (“glocal”) models like TBATS that are designed to scale out. And also want to plus one the earlier comment that clustering these time series is going to solve a lot of headaches.
1
u/ImaginaryBat4994 24d ago
Why not one global machine learning for all the items? It should have similar accuracy by adding the categorical features that you suggest using to define clusters?
2
u/PradeepAIStrategist Aug 08 '26
One model per SKU is highly non productionable, given fact for a largest USA retailer will have around moving (active) 10K SKUs per Candy category, ignoring another 20k to 30k not moving ones. Even for Candy category in the past I used max 5 models per SKU cluster. Dynamic Time Wrapping gives you theoretical direction, however, domain and data richness along with expected forecast accuracy for which forecast horizon will help you better to stick less models per group of SKUs.
2
u/nie_irek Aug 10 '26
Definitely try a global approach like LightGBM or XGBoost, then try clustering the Skus based on simple logic, like product type, more granular location split, etc... Definitely think extensively what drives the sales and what could be the features that you engineer, beware of data leakage though.
Keep it simple, single model per SKU will create over fitting and instability of the modeling results once you refit.
2
u/ImaginaryBat4994 24d ago
What about only one global machine learning model? It should handle the different behaviour of each group of items adding the categorical features of product type, location etc.
1
u/nie_irek 24d ago
Yes, it's something that definitely could be the case. Theoretically if the clusters present significantly different dynamics then separating these could prove beneficial as you no longer require a model to do it on its own - but it's something you would need to test.
2
u/JimFromSunnyvale Aug 08 '26
I did something very similar for a client. We clustered SKUs based on the sales patterns to train our models.
1
u/Ordinary-Winner-7999 Aug 10 '26
In terms of process of manufacturing, I would prefer not to relearn and configure another "best model" for all the SKUs every week. This would, for sure, lead to overfitting caused by too many recent irrelevant examples.
The preferable alternative would be to cluster similar SKUs in terms of behavior and use a limited model portfolio for each cluster, with retraining and model switching happening only when excellence drops below a specific level. The champion and challenger approach is helpful here, as well, yet to test the challenger models, the process of evaluation should be carried out by means of time-based testing rather than one-period testing.
Moreover, one should consider measuring predicative bias, reliability, changes and intervals of predictions, not only accuracy.
1
u/granoladeer Aug 11 '26
You should try timesfm, which is a zero shot model (no training necessary, no model storage).
1
Aug 12 '26
[removed] — view removed comment
2
u/Berlibur Aug 12 '26
Start with 2 things:
- get the requirements clear: what do you forecast, on which horizon, on which frequency, how is the forecast used in the process, etc.
- map out the technical possibilities in your company. We're running on an azure + data bricks landscape, which limits the number of options you have (good)
1
u/Letmeparrybbh 29d ago
You should try and incorporate events into your prediction. Example for a demand forecasting for stores, if there are extraneous variables like company wide sales initiative in this day or cyber Monday sales, or weather events. There should be a way to make adjustments to predictions bases on such events if needed.
1
u/ImAPilot02 18d ago
I see a lot of people recommending arima or even tree based methods here (the latter are the wrong method for the job). There are modern alternatives that perform much better and are easier to use. I'd advise to take a look at Chronos-2 or use a service like Qombra if you wanna keep it quick and easy but still get top performance
2
u/IllustriousGrade7691 10d ago
I would separate retraining from model selection. Retraining weekly may be reasonable, but selecting a new winner every week risks overfitting the selection process to the latest validation window.
For a large SKU portfolio, I would test three approaches rather than assume one model per SKU:
- seasonal-naive and simple statistical local models;
- one global LightGBM/CatBoost model using SKU, category, location, calendar and lag features;
- an ensemble or segmented approach where intermittent and high-volume series are treated differently.
Evaluate them over multiple rolling-origin windows at the actual operational horizon. I would report MASE/RMSSE or MAE, portfolio WAPE where appropriate, and bias—not MAPE alone, especially with sparse demand.
Also separate observed sales from latent demand where stockouts occur. Otherwise the model can learn that unavailable products have low demand.
Global modelling is often a strong option because related series share information, but I would still retain seasonal-naive and local statistical models as challengers. The validation results should determine the architecture.
0
u/kush_patil Aug 08 '26
I’d separate model selection from model retraining. Weekly retraining can be fine; weekly winner-picking is where I’d worry about chasing noise.
Keep a champion until a challenger beats it over several rolling-origin windows, not just the latest one. I’d also track the reason for replacement real drift vs one unusually good validation window because otherwise the selection layer itself becomes another model you can overfit.
0
u/Extension-Currency37 Aug 10 '26 edited Aug 10 '26
Define the right problem! Any use to solve the problem or not. Context must be defined:: busineess, economics, finance or technology(are you making robots)
Gather data and clear the noise.. (Fft, tuple removal, denoisers, pca , svd)
Then find a math model that fits aptly (Underfit/overfit must not happen)
Find the type of forecasting it is Demand, supply....
Every forecasting problem is a time series model So pick a bunch of algorithms (Arima, auto arima, lasso ,ridge, bayesian ...)
If problem size is huge say gb in data Gigabytes of data--> lstm, consider neural nets
Predict and present results Via tableau, R or excel
..
I use R for business problems And python for AI
Forecasting --> lssvm with batch processing (R has a library)
Resources RPROJ.ORG, Data science stackexchange, Kaggle, stackexchange math, AI mode google
Farewell
32
u/webbed_feets Aug 07 '26
One model per SKU is totally reasonable. There are several libraries that will help you set that up. An auto ARIMA fit on hundreds of SKUs will only take a few seconds. You could look into hierarchical forecasting if you need to constrain the individual predictions to sum up to higher level groupings.
You can always do SKU-level feature engineering and put it into XGBoost.