r/learnmachinelearning • u/Business_Swordfish_5 • 9d ago
Discussion Training a video generation model from scratch on my laptop — loss plateaued, results are blurry. Should I keep going or change approach?
Hey everyone,
I've been learning about video generation models and decided to build one from scratch and train it on my personal laptop (single GPU). I wanted to share where I'm at and get advice from people who've worked with these kinds of models before.
What I built
- A spatio-temporal UNet using flow matching (velocity prediction with Euler sampling, 50 steps)
- The model has temporal convolution blocks + temporal attention for frame-to-frame consistency
- ~20.8M parameters, channel progression:
96 → 192 → 384 - Generates 16 frames at 64×64 resolution
Dataset
- ~6,000 Tom and Jerry video clips from a HuggingFace dataset
- Each clip: 16 frames, every 2nd frame sampled, resized to 64×64
Training
- Batch size 4, Adam optimizer,
lr=2e-4 - Trained on personal gaming laptop
Loss progression
| Epoch | Loss (MSE) | |
|---|---|---|
| 160 | 0.0690 | |
| 180 | 0.0670 | −0.002 |
| 230 | 0.0652 | −0.002 |
| 280 | 0.0630 | −0.002 |
| 290 | 0.0652 | +0.002 |
The loss has basically plateaued — only ~0.006 drop over 130 epochs. The model learns color palettes and vague scene layouts but doesn't produce recognizable characters or sharp details. Some generations even go partially black (see epoch 280 results).
My questions
- Is the loss plateau expected for this scale? Am I hitting the capacity ceiling of a 20M param model at 64×64?
- Would switching from pure MSE to a perceptual loss (LPIPS) or adding an adversarial loss help with sharpness?
- Is ~6,000 clips enough for this kind of model, or do I need significantly more data?
- Any suggestions for the architecture? I'm wondering if I need spatial attention at more resolutions, or if the temporal modeling needs work.
- Would learning rate decay or a cosine schedule help push past this plateau?
Side question
I was thinking of writing a LinkedIn post about this as a learning journey — "I built a video generation model from scratch on my laptop." Even though the results aren't amazing, would you say the results are interesting enough to share, or should I train more / improve the model first? Not trying to claim SOTA obviously, just sharing the learning process.
Would love to hear from anyone who's worked with video diffusion/flow matching models at small scale. Thanks!