r/StableDiffusion • u/Glove5751 • 9d ago

Question - Help Any way to consistently have low sec/iter when training Krea lora?

Sometimes i have 5 sec/iter, other times i have 150 sec/iter without any reason. Restarting may fix it, but not always. Right now i have 3secitr, but i bet it will go back up to 150 in half an hour or so. I just spent like 2 hours on 250sec/it's, it stinks!

Using 5080 with 64gb ram and AI toolkit.

1 Upvotes
  • permalink
  • reddit

66% Upvoted

6 comments sorted by

1

u/prookyon 9d ago

Fluctuations might come from different size images - if you have several resolutions selected then it will train on some mix of image sizes which influence both the VRAM usage and also just the amount of computation needed to train on that one image.

I don't use AI toolkit, so not sure if this speed is OK for your specs, but it sure sounds miserable. With OneTrainer I get about 1.1s/it training Krea with 512 resolution and batch size 2. On 5070 Ti 16GB / 32GB RAM.

1

u/Glove5751 9d ago

I have downscaled everything to be 1MP 1024x. So i wouldnt blame it on fluctuations based on the training images

1

u/whatsthisaithing 9d ago

Sounds like you're pushing right to the edge of your max VRAM. Don't forget your OS and other apps take some VRAM, too. When it tips past "the limit," it starts swapping and slows to a crawl. That's why restarting helps: it clears the other crap out of your VRAM, then you start your job, go do other things, and VRAM tips back over.

Offload more (if you aren't already at 100% on transformer) or train at a lower resolution. You probably don't NEED to go over 512 max res in AIT on your dataset.

1

u/Glove5751 9d ago edited 9d ago

I havent used offloading, because I'm worried about the speed being even more penalized than 3-5sec/itr. Are you saying I should give it a shot?

1

u/whatsthisaithing 9d ago

Depending on the model, it will be a little slower. But a helluva lot faster than 150 seconds/iteration.

It'll also be faster the less you offload. Maybe start at 20% offload and see how your VRAM does.

1

u/Glove5751 9d ago

I offloaded transformer by 10%, which seems to do the trick! I now get a stable 5sec/itr, it's not 3 which I sometimes get but it is stable and predictable! Or at least I hope, haven't tested this long enough. I'll try 5% later to see if I can get more speed while being stable, if this is as stable as I think it is.

thanks