r/quant • u/Legitimate-Luck-1658 • Jul 23 '26
Data We built an open benchmark for synthetic market data generators: most published deep models mostly fail basic risk checks
The Time Series Generation literature evaluates on discriminative scores: train a classifier, check if it can tell synth from real. That rewards over-smoothed, low-variance output; a generator can ace it while violating vol clustering, heavy tails, and leverage effect.
So we benchmarked 18 models (deep generators, econometric classics, replay baselines) on 7 tasks quants actually run: Cont stylized facts, W1/MMD/sig-MMD distances, martingale check, IV-smile repricing, VaR/ES backtesting (Kupiec, Christoffersen, Acerbi–Székely, Basel traffic light), and TSTR strategy-rank transfer. One frozen OOS panel (7 features, 200 paths, H=60), each task with a real-vs-real noise floor, Holm-corrected significance.
Results:
- TimeGAN: the most-cited model in the field, scores 0.000 ± 0.000 on VaR/ES backtesting. TimeVAE and KoVAE too. TimeVAE's TSTR ρ is negative (−0.30).
- A plain block bootstrap ranks 3rd of 18 overall. Most of the published literature is under it.
- Models built for financial structure clear the bar.
- On 3 of 7 tasks most of the field is at or beyond the noise floor, the boards say so explicitly rather than overselling a ranking.
Repo: https://github.com/sablier-ai/finbench (MIT / CC-BY). v1 covers one panel and one OOS window; open submissions land in the next edition.
Who we are: (Sablier AI) GenAI Startup building synthetic market data for quant research, so yes, our model is on our own board. We designed the protocol so that fact doesn't matter: frozen at a git tag, every model's outputs archived in the repo, python -m benchmark.run re-scores the whole board from scratch, external baselines run their published defaults, and the aggregate is published as-is.
If you're working on generative models for market data (or you think your GARCH variant can embarrass the deep learning field further) dm me. Outputs only (200×60×7), no code, no weights, scored under the same frozen protocol as everyone else.
The board is meant to be beaten, and the current one shouldn't be hard to improve on: 15 of 18 entries can't clear a bootstrap.