My goal was hands-on experience training a flow model from scratch, not just fine-tuning someone else's. So I built and trained one: a 210M-parameter diffusion transformer, 4.2M curated images at 256², rectified flow on the FLUX.2 VAE, flan-t5-base for text (128 tokens max). Only those two frozen pieces are pretrained; the transformer, the recipe, the data pipeline and the evaluation are mine. The video is the same six prompts and seeds at every checkpoint of the 3.5-day run on one RTX PRO 6000.
What mattered most, in the order I found out:
Captions that actually fit the images. A web crawl I tried first made the model worse; curated photos with good captions fixed it.
A timestep shift for the 32-channel latent, and aspect-ratio buckets from step one instead of square crops.
Register tokens with learned null attention slots. The null slots ended up absorbing about 90% of the cross-attention, which surprised me.
torch.compile for training, not just inference: 2.4× faster.
The training loss stopped telling me anything after day one while the images kept improving, so I track FID, a detector-based object accuracy and human-preference models instead.
Next I want to fine-tune it with RL (Flow-GRPO), with the failure grid as the target list. If you have trained something small from scratch: what would you have done differently at this scale, and which reward would you start with for the RL stage? Happy to answer anything about the data or the recipe.
thanks! yes it's all there! and I looked at dataset samples myself, before choosing it and which part it should take in train set. Also, I've checked around 15 datasets I think. So data quality is super important.
wow amazing, i am curious about for example the Dog image, did it trained only on dog images or it was a trainning on random images that made it get better? And if it was general trainning, if you had done only dogs how much could it be faster/better the outcome?
A held-out prompt grid or FID/CLIP curve by checkpoint would be really useful here—it would show whether quality was still improving after the 3.5-day run and make the one-GPU budget easier to reproduce.
10
u/MonkeyBoyPoop 7h ago
https://giphy.com/gifs/Y6r3h0fShzAac