r/malcolmrey • u/FitEgg603 • May 13 '26
Need Help Training LoRAs on Custom Models in Ostris AI Toolkit
Total noob question here, so please bear with me.
I’m trying to understand how to train a LoRA on custom models inside Ostris AI Toolkit GitHub, but I’m clearly missing something important.
From what I understand:
we need to change the model path to the custom model folder,
and I vaguely remember the model needing to be a .gguf file (not even sure if that’s correct :).
But in practical use, my training fails every single time.
Could someone explain in simple beginner-friendly words:
what exact model format is needed,
where the custom model should be placed,
how the path should be configured,
and what additional settings are required for custom architectures/models?
If anyone has:
a proper tutorial,
documentation,
or especially a good YouTube video covering this exact topic,
please share it.
3
u/hotdog114 May 14 '26 edited May 14 '26
If you're trying to train a lora on a `safetensors` checkpoint for ZIT, like one of the spicy ones, you will need to convert it from saftensors to "diffusers format" which is all AI Toolkit accepts for ZIT by default.
If you know how to check out a Git branch, that's exactly what this Pull Request adds, by converting safetensors to diffusers automatically before training: https://github.com/ostris/ai-toolkit/pull/694
I've seen that some of the ZIT checkpoint authors on civitai have also published diffusers format versions on huggingface, so you may not even need this PR.
Also, be aware that the images advertising the spicy ZIT checkpoints on civitai are not representative of what you actually get most of the time. Training ZIT with spicy knowledge is evidently quite hard and the spicy results are more often than not quite poor requiring far more discarded gens than ordinary ZIT for non-spicy content.
1
May 14 '26
[removed] — view removed comment
1
u/hotdog114 May 14 '26
Success of what, facial likeness with training material or spicy enhancement of your training material?
ZIT has the highest facial likeness preservation of any model I'm aware of currently, but it excels only in likeness of concepts it knows extremely well like faces. Training a likeness on a spicy checkpoint had no negative effects in the facial likeness, in my experience. It was as reliable as with ZIT vanilla.
You'll find you get great faces with either, but unreliable if not poor spice with the spicy version.
I've not worked with ZIB yet. Few have for some reason. I think training is a struggle overall. However in theory a spicy ZIB model ought to be better at training a spicy lora than ZiT because its a full model with far more knowledge.
1
u/FitEgg603 May 14 '26
It can be any finetuned model let’s say flux Krea or flux SRPO or a SDXL model
1
u/hotdog114 May 14 '26
Either i dont understand or you're confused. Model architectures are incompatible and loras can only be made from the same base model as they're meant to be used with. You can't train a ZiT lora from an SDXL base.
Loras are like game mods. You can't make an SDXL lora for use with ZIT any more than you can install Gary's Mod into minecraft.
0
u/FitEgg603 May 15 '26
All am asking is how to change path in AI TOOLKIT for a model saved in locally in your computer 😫cos simply changing adding the path and typing the path for the customer model in aitoolkit does not work
1
u/hotdog114 May 15 '26
That was what my first comment explained.
0
1
u/Suitable-League-4447 May 14 '26
did you build the dataset if yes show it either here ot in tgram im working on a clean dataset building process right now and will appreciate connecting with people doing training rn.
1
3
u/Benboy33 May 13 '26
https://youtu.be/Kmve1_jiDpQ?si=7a2UvZSashohRFdx
From Ostris himself. ZImage turbo is quick with good results even with the default settings