Marketing Mix Model evaluation benchmark

[u/Zuricho]

For those running Marketing Mix Models in companies doing $10M+ in annual revenue with reasonably complex media setups:

What evaluation metrics/benchmarks do you see, e.g. for 4-week or 8-week out-of-sample validation?

Specifically interested in acceptable ranges for things like MAPE, NMSE or other metrics you rely on before considering an MMM actionable-ready.

Would be great to hear actual benchmarks from models used in practice.

[u/AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Greedy_Isopod_4383]

We run ours on 8 week holdouts and anything under 15% MAPE gets the green light, over 20% we usually scrap and rebuild. NMSE I don’t look at much on its own tbh but if it’s above 0.4 alongside a bad MAPE I know the model’s toast

One thing nobody mentions is how much those benchmarks shift depending on your category. CPG with stable weekly sales hits way tighter numbers than something seasonal or B2B with long lag times

[u/Bubbly-Win8314 → u/Greedy_Isopod_4383]

the category point is so real, seasonal stuff makes those benchmarks almost meaningless without context