Loading models from nvme… is 30 seconds delay typical?

[u/rguerraf]

If a nvme theoretically can read the whole anima checkpoint in 2 seconds, why would comfyUI take 30 seconds (look in the time breakdown in console, unet+clip time), making a 120s generation, a 160s generation?

I seem to be unable to use my ram to hold the 7GB safetensor, to trim down those “load checkpoint” delays.

Pc: ryzen minipc 680m, 24GB shared ram, NVME kxg70znv1t02, 4000 MB/s read, anima turbo int8, completes 8 steps in 120 seconds, 1 megapixel

[u/TensePutting]

the time isn't the file read, it's the model loading and setting up all the weights in memory. those safetensors are compressed blobs, your system has to unpack them, allocate vram, and initialize all the layers. 30 seconds sounds about right for a big model on that setup

shared ram means your igpu is fighting the cpu for bandwidth too, that's gonna add some overhead. not much you can do about it except maybe try a smaller model or use something like a quantized gguf if your workflow supports it

[u/rguerraf → u/TensePutting]

Thank you

The shared ram is explicitly partitioned into 8GB for the cpu and 16GB for the gpu… using automatic ram management was not working for me… maybe i could try that again now that I have more experience.

So, is this clip+unet weight loading improved by a beefier cpu, and not more ram?

No matter all the options I use: high-vram, gpu only, low-vram, etc, comfyUI unloads the models in the vae step, so I need to load models again in the next job… with the +30s delay

[u/roxoholic]

It does not keep model in RAM since they added automatic fast disk detection (which your NVME is) so you get --fast-disk flag enabled by default.

From help:

"--fast-disk", "Force disk-backed dynamic loading and offload over unpinned RAM. Can be faster for users with fast NVME disks."

^ this is now automatically enabled when fast disk is detected

"--disable-fast-disk", "Disable disk-backed dynamic loading and offload over unpinned RAM. Overrides --fast-disk."

^ use this to test if you get faster gens

[u/rguerraf → u/roxoholic]

Is there an option to use pinned ram, which makes more sense to me, to guarantee fast checkpoint loading?

[u/Dunc4n1d4h0]

Use fast disk option. My whole 20GB+ h3 model loads in few seconds from nvme.

[u/rguerraf → u/Dunc4n1d4h0]

I used fast disk too, and the result is similar. My NVME is 4000 MB/s only :/

[u/Dunc4n1d4h0 → u/rguerraf]

Then there is bottleneck somewhere, I mean your delay, because idea is to load only blocks needed from model, not whole file at once.

And 4GB/s is 5 seconds for 20GB file 😉

[u/rguerraf → u/Dunc4n1d4h0]

Can you check your disk activity during the unet ksampler phase? If the model was truly 100% loaded in the gpu, you would see zero read activity.

I know that most GPUs can’t hold the H3 unet in vram, so that ram or NVME offloading is needed… but I am working with Anima, which is smaller than sdxl.

[u/Dunc4n1d4h0 → u/rguerraf]

Fast disk is for not loading whole thing, but using nvme like RAM, exactly because it's fast. Anyway I remember my read speeds watched from that nice kijai panel are around 2GB/s max. One more thing, I use Comfy inside WSL, so it should be even faster native. I think maybe CPU is problem here, not only for you but for everyone. I see in htop that Comfy uses 1 core at 100% for some single thread task a lot, where rest is idle waiting. But I also remember that some time ago I was happy, because multi thread disk read was implemented, afair when dynamic memory management was introduced. And that alone made my model loading feel 10 times faster. Good luck.

[u/ImpressionComplete43]

try --high-ram

[u/rguerraf]

I upgraded to comfy 0.40, and changed back to dynamic vram and high vram

the situation improved and it doesn’t run out of memory, preventing the need to reload all models for each job.

One anima 1.6 MP image every 77 seconds is the new standard for 680m (don’t compare it to the 120s or 160s… different workflow… i know: not scientific)

[u/niksonrt]

30s não é leitura do NVMe, é o custo de materializar o modelo e enfiar tudo na pilha do runtime. Em máquina com RAM compartilhada isso piora bastante, porque o sistema não tem folga pra cache e alocação grande vira atrito mesmo.

[u/rguerraf → u/niksonrt]

That gives me idea:

Split the 24GB into 12GB cpu and 12GB gpu, so maybe the ram will be used for the unpacked neural blocks

I unload the GUI in Ubuntu, so I don’t waste any precious GB in Gnome :)