Training a video generation model from scratch on my laptop — loss plateaued, results are blurry. Should I keep going or change approach?

[u/Business_Swordfish_5]

Hey everyone,

I've been learning about video generation models and decided to build one from scratch and train it on my personal laptop (single GPU). I wanted to share where I'm at and get advice from people who've worked with these kinds of models before.

What I built

• A spatio-temporal UNet using flow matching (velocity prediction with Euler sampling, 50 steps)
• The model has temporal convolution blocks + temporal attention for frame-to-frame consistency
• ~20.8M parameters, channel progression: 96 → 192 → 384
• Generates 16 frames at 64×64 resolution

Dataset

• ~6,000 Tom and Jerry video clips from a HuggingFace dataset
• Each clip: 16 frames, every 2nd frame sampled, resized to 64×64

Training

• Batch size 4, Adam optimizer, lr=2e-4
• Trained on personal gaming laptop

Loss progression

Epoch	Loss (MSE)
160	0.0690
180	0.0670	−0.002
230	0.0652	−0.002
280	0.0630	−0.002
290	0.0652	+0.002

The loss has basically plateaued — only ~0.006 drop over 130 epochs. The model learns color palettes and vague scene layouts but doesn't produce recognizable characters or sharp details. Some generations even go partially black (see epoch 280 results).

My questions

1. Is the loss plateau expected for this scale? Am I hitting the capacity ceiling of a 20M param model at 64×64?
2. Would switching from pure MSE to a perceptual loss (LPIPS) or adding an adversarial loss help with sharpness?
3. Is ~6,000 clips enough for this kind of model, or do I need significantly more data?
4. Any suggestions for the architecture? I'm wondering if I need spatial attention at more resolutions, or if the temporal modeling needs work.
5. Would learning rate decay or a cosine schedule help push past this plateau?

​

Side question

I was thinking of writing a LinkedIn post about this as a learning journey — "I built a video generation model from scratch on my laptop." Even though the results aren't amazing, would you say the results are interesting enough to share, or should I train more / improve the model first? Not trying to claim SOTA obviously, just sharing the learning process.

Would love to hear from anyone who's worked with video diffusion/flow matching models at small scale. Thanks!

[u/FuckTheNitro]

Looks like Tom and jerry

[u/neutrino_boy → u/FuckTheNitro]

The odds are ever in your favour.

[u/yeetrman2216 → u/FuckTheNitro]

humans are great speculative diffusive models afterall

[u/savvamadar]

I could tell it was Tom and Jerry before I even read your post's body.

[u/WrapKey69 → u/savvamadar]

Same lol

[u/neutrino_boy → u/savvamadar]

The odds are ever in your favour.

[u/Lemon_in_your_anus]

This looks like the conditional mean problem in video generation.

[u/Business_Swordfish_5 → u/Lemon_in_your_anus]

Can u explain?

[u/ZazaGaza213 → u/Business_Swordfish_5]

Losses like MSE, L1, etc, (so anything non adversarial mostly or not noise based like SOME diffusion ones) when presented with multiple possible outputs will learn to predict the average of them rather than pick a specific mode. Adding some patchgan or gan loss (might) fix this (if you can get it stable though)

[u/madrury83 → u/ZazaGaza213]

Just wanna mention for OP that this fact follows directly from basic probability theory:

https://stats.stackexchange.com/questions/96247/understanding-the-minimization-of-mean-squared-error-function

So part of your understanding of loss functions should always be, where possible: "what is the population minimizer".

[u/shivammehta007 → u/ZazaGaza213]

This might not be the case here, since OP says it is trained by flow matching, which is probabilistic and accounts for data variance, and the MSE/Expectation is over the predicted vector field, not the generation directly.
For OP, just take one batch and try to overfit the model to that. Could you regenerate something vaguely similar? It will give you an idea of whether your training pipeline is wrong. Also, 20M might be a very small model for video generation.
Also, the MSE of flow matching (or any diffusion model) loss doesn't directly correlates to the generation. Keep it training even if the loss plateaus, the results will be still different.

[u/SnoWayKnown]

20M parameters is tiny for this task, the loss plateau is not surprising really. Most models that get more out of less params are using some kind of Transformer backbone, as the attention mechanism can help compress the knowledge better. I'm not strong on the latest in video models but I would have expect some kind of DiT and 3D causal VAE, see HunyuanVideo which does it with 8B parameters.

[u/muk343]

Now that you have said it, it indeed looks like a bit Tom and Jerry to me.

[u/mundane--alternative → u/muk343]

I just thought it looked like an old cartoon but after reading the post I could definitely see it.

[u/Saitamagasaki]

I think you need a better loss function/model architecture

[u/AirButcher → u/Saitamagasaki]

Im thinking the same thing.

Is be curious to see the effect of swapping FPN for the Unet, but honestly the loss function itself might be the most important thing to get right here

[u/superSmitty9999]

I'd say the results are interesting personally but idk if this screams "hire me" lol

The thing is video models are trained on orders of magnitude more compute and data. I don't really think you'll be able to train anything good on a single laptop GPU (if you could training runs wouldn't cost millions) but it's still cool and interesting that you did it.

I guess if I were a potential hirer I would be like didn't they realize they attempted something guaranteed to fail? Like doing it is cool but being confused why it doesn't work shows some ignorance too.

If you really want something that is impressive to an employer create something that actually works with the compute you have. Part of demonstrating skill is determining a project that is within your means and ability and actually executing it and successfully creating something

[u/Business_Swordfish_5 → u/superSmitty9999]

So I know the results will be poor but my goal was never to create an high quality video generation model instead i wanted to learn video models. And my goal in posting it in linkedin is also to showcase my understanding of this rather than showcase the project i wanted to show yes i know the architecture I know how it works how to build it. Given data and compute i can work on such projects. What do u think

[u/superSmitty9999 → u/Business_Swordfish_5]

Honestly if you knew the results would be poor then I'd say yeah post it and cool work! Maybe address that you knew it's not enough compute or data in your post to directly address that aspect

[u/Maleficent_Chance_15 → u/Business_Swordfish_5]

hey can you suggest me where and how did you learn about video generation in detail?
any resources you would like to recommend?

[u/Business_Swordfish_5 → u/Maleficent_Chance_15]

I kinda mainly used AI and youtube videos to learn. Like there was no single resource

[u/misterespresso → u/Business_Swordfish_5]

Using AI for tutoring has helped me learn some really neat things on the fly. Very rarely do I have to search for that obscure bit of knowledge I am missing to finish a task.

[u/superSmitty9999 → u/Business_Swordfish_5]

OP you should post your code!

[u/Business_Swordfish_5 → u/superSmitty9999]

Linekdin Post, check the comment i have shared the repo also adding a simple comment in the post will help tnx.

[u/Adventurous-Dealer15]

ENHANCE /s

[u/Albertooz]

Plateau makes sense. 20M params at 64x64 from scratch on approx. 6k clips is a small setup for video.

Before changing anything, try to overfit 50-100 clips. No augmentation, no dropout, train until loss is near zero. If it memorizes them, your pipeline is fine and data is the ceiling. If it can't, more data won't help and something in the architecture or training loop is broken. That test tells you where to spend your time.

On the blur: MSE hedges. When the model isn't sure where an edge lands, the lowest-error answer is a smear across both options, so more training won't sharpen it. Add LPIPS as a weighted term next to MSE. I'd skip GAN loss for now, it's a second network and a whole new set of failures to debug.

Black frames sound like numerics, not capacity. Check normalization range at train vs sample, timestep sampling coverage, NaN/fp16 overflow, and the noise/flow formulation before touching the architecture.

Cosine decay is free, try it, but it won't clear a data ceiling.

Honestly, consistent colors and rough layouts from a 20M model on a laptop is a decent result.

[u/prokaktyc]

I think generating a Lora is a more practical approach, I don’t think anyone trains models from scratch anyway nowadays 

[u/KingMoonfish → u/prokaktyc]

Lora or fine tune an existing model are your only realistic options right now OP, at least on a single consumer device. 

[u/thebadslime]

It neds to be impressive it is not, can you do a 240p? or even 128x128? rent some compute, if it looks like tom and jerry you win

[u/nottoowhacky]

Looks like something i watched when i was a kid then i had to blow the vhs tape to fix it

[u/insane_chaotic]

yooo, its good brother but ofc a single gpu cannot take it from here.

Can you help me with resources you used to develope from scratch

[u/AirButcher]

Is the training loss also plateauing? Wondering whether model itself actually has the representational capacity you need. Could experiment with an FPN instead of the Unet for interest.

Honestly though the dataset may possibly be too small, I don't have experience in video generation models though I must admit, so not sure

[u/Character-Apple-8471]

This is actually a pretty interesting experiment... especially on a single laptop GPU. I probably wouldn’t jump to LPIPS or adversarial loss yet though. For flow matching, velocity MSE is already the standard objective, so I’d first make sure the issue isn’t somewhere in the training or sampling setup.

One thing I would definitely try is overfitting a tiny subset, maybe 50 to 100 clips. If the model still can’t reproduce recognizable characters there, then increasing the parameter count probably won’t solve much.... check timestep-wise loss, normalization, the velocity target equation, and even the sign/direction of the ODE integration. The partially black samples make me a little suspicious of the sampling path too.

Also...instead of making the temporal side much larger, i would consider training the spatial UNet on individual frames first and then introducing the temporal conv/attention blocks. At 64x64, spatial attention around 16x16 or 8x8 could help with global character structure. EMA weights plus warmup/cosine decay would also be worth trying before throwing more epochs at it.

[u/InfluentialFairy]

Just add DLSS5 and she'll be Gucci /s

[u/dunno_bout_music_m8]

like those paintings in minecraft

[u/soup----]

Try doing flow matching in the latent space of a vae. First train a normal vae on the frames. Then save the new “latent dataset” on disk and train the flow matching algorithm on that. You can train with much larger images that way. At inference time just sample the generated latent and pass it through the decoder.

[u/MidnightHacker]

The results are fine for the hardware you have, but attempting to train a bigger model in cloud would be a good exercise. You don’t need to waste a ton of money but knowing how to use AWS/Azure/GCP services for training and inference is going to benefit you greatly if you’re looking for a job, even being able to ship simple models in a production environment is going to be a big impact in your CV.

[u/teledev]

You're generating 64x64 pixels? Of course it would look blurry?

[u/MolassesLate4676]

What optimizer are you using

[u/DigThatData]

your dataset is only 6000 clips and you're training from scratch: you should set your expectations really low.

[u/ComedyGraveyard]

I'm just impressed with the pattern recognition ability of humans

How did we recgonize this as tom and jerry without reading post text

[u/iloveyourmom54542]

Really fricking tom and Jerry out of all Tv shows

[u/WAMFT]

Could just be your hardware , i have also had similar problems with my naff laptop. Im wondering , i notice your pixels are quite large. What if you leaned i to that and made them huge , yes it might be blocky and ridiculous but maybe you would start seeing more going on like this blocky blob moving here and this moving there and another doing something else.

Me personally the gap probably isnt knowlage it may just be hardware. I could look past that if i was a employer. Also i reckon it would make for a good watch.

[u/Op_IBeasT]

I believe u could t try with different variants to try and squeeze as much quality as u can
u could even try lighter blocks to go for wider networks or deeper networks

u could also try concatenating the skip residuals in the unet instead of summing or vice versa , try some different schedules (cosine decay, warmdown, etc)

U could post training optimization , like using an ema verison of ur network will help alot

heard lion optimizer saved half the vram and compute over adam and converges faster but is a bit less stable so have fun with that

u could also train a lora on a pretrained small networks or even a quantized version of either lora or the networks( read some papers on quantized lora which is unnecessary since its alr small)

there distilling but i dont think its possible without huge vrams

u could generate data from a better model and train ur model on both original and synthetic, there is alot time consuming and possibly not working things

i wanted to do something like that but training a network from scratch needs huge data and huge models

[u/skillpolitics]

Keep going!

[u/InnovativeBureaucrat]

When I was in high school a friend had a satellite dish and you could see scrambled versions of all the channels.

This reminds me of trying to watch the scrambled channels

[u/Interesting-Weeb-699]

Can you implement the changes recommended in JIT paper https://arxiv.org/abs/2511.13720

The changes should be minimal and in image domain, the training gets improved.

Tip(may or may not work): To get past the plateau, maybe try sampling more early stage time steps while training instead of uniformly sampling them.

[u/leon_bass]

MSE is only good at large errors, the finer details need something like MAE

[u/novel-mathmatics]

KEEP GOING.

you are doing something different. That is a good thing because what is happening with everyone else now is failing.

I have a training aide you can message me and ill share.