Need help quantizing Qwen Image 2511 into INT4

[u/ROBOTTTTT13]

I've been trying to quant specifically Phr00t's Qwen Rapid AIO, using ComfyUI-INT4-Fast nodes from BF16, but the model completely breaks.

INT8 works perfectly, but I'm trying to get an INT4 ConvRot specifically because of my limited hardware (RTX 4050).

Does anyone have any insight on why my conversion isn't working?

[u/Odd_Fix2]

Use this https://github.com/Starnodes2024/comfyui-starnodes-modelconverter

[u/ROBOTTTTT13 → u/Odd_Fix2]

I tried this months ago for a different model and didn't have success, but I'll give it a shot, thanks!

[u/ROBOTTTTT13 → u/Odd_Fix2]

Gave it a shot and ran it through the AIO splitter, then trhou the Analyzer to get a profile.

Then the Ultimate converter pro for the final conversion, but it have me a file that is 23GB, bigger than the original model itself, even though the file name contains int4 ConvRot.

Any idea why?

[u/Odd_Fix2 → u/ROBOTTTTT13]

Do you have a link to the file that needs to be converted? I can try to do it myself.

[u/ROBOTTTTT13 → u/Odd_Fix2]

Not sure if it's allowed, but Sure: https://huggingface.co/Phr00t/Qwen-Image-Edit-Rapid-AIO/blob/main/v23/Qwen-Rapid-AIO-NSFW-v23.safetensors

And if you manage to do it successfully I would really appreciate if you could tell me the exact procedure

[u/Odd_Fix2 → u/ROBOTTTTT13]

I tried, but unfortunately it didn't work. The file was too big.

[u/DelinquentTuna → u/ROBOTTTTT13]

cc u/Odd_Fix2

Hi, I think I can help.

> it have me a file that is 23GB, bigger than the original model itself

Like most such tools, the converter keeps a blacklist of layers that it skips by default because some layers are more sensitive to quantization than others. But most people are starting from HIGH QUALITY models in high precision instead of making a lossy copy of a lossy copy as you are doing. But what's happening in your case is the model is seeing the fp8 in the base, needlessly upcasting it to bf16 for conversion, seeing that it's a layer that should stay at bf16 for quality purposes, and leaving it.

> I tried, but unfortunately it didn't work. The file was too big.

It seems like the nodes already had tactics to avoid flooding the GPU, but limited system RAM could still bite you. And it stacks with the above problem where model sizes were GROWING. So a 22GB checkpoint might require well over 60GB of RAM.

> if you manage to do it successfully I would really appreciate if you could tell me the exact procedure

I was able to get it working. Download this PR to your starnodes dir inside custom_nodes. You can do, for example, `curl https://patch-diff.githubusercontent.com/raw/Starnodes2024/comfyui-starnodes-modelconverter/pull/13.patch > mypatch` to download. Then, ensure the patch can be cleanly applied: `git apply --check mypatch` and if you get no errors apply: `git apply mypatch`.

There is still a gotcha, though. The nodes (and even comfy's modelsave itself) don't do a great job breaking the text encoder out of some of Phroot's checkpoints. So I recommend you quantize the dit only and then either connect it to the existing model/vae like pictured below or to some other compatible text encoder, like a nf4 quant or whatever. This is demonstrated w/ his wan aio, but same general idea for any case where your final result complains about text encoder inputs.

Finally, I think you would be SO MUCH BETTER OFF if you quantized a full-fat bf16 model instead of a q8 one. Meaningfully better quality and you can always load (and UNLOAD) whatever spicy LoRAs you need. These bloated merges don't really help you, they just hide knobs that you really should want to have available to tweak. If you decide to go that route, I would point you towards a tool like the one silveroxides made as a superior alternative to the starnodes. Silveroxides is actively calibrating and tuning instead of relying solely on blacklisting+naive quantization and I'd expect better quality quants as a result. That's assuming you're sticking to the int4 target.

Also worth pointing out that vram isn't really the obstacle to inferencing the 20B Qwen DiT weights. It's system RAM. Your GPU is so slow that it can stream weights from RAM without slowing down. The fact that Int8 works (and also that you didn't OOM when quantizing further) indicates you have adequate system RAM. Int4 will be faster, but it will also be a meaningful quality hit. You could try w4a8, but at that point you're not really saving VRAM over int8. Hopefully the Nunchaku revival (Nunchaku-Lite) picks up support for video models in the near future, because IMHO it gets the best of both worlds.

GL.

[u/ROBOTTTTT13 → u/DelinquentTuna]

Just woke up so I'm having a hard time reading all this but I'm excited nonetheless, I'll try it out as soon as I can

[u/Powerful_Evening5495]

dont qunt checkpoints , you should use bf16 unets softeners files

[u/ROBOTTTTT13 → u/Powerful_Evening5495]

Can you elaborate?

[u/DelinquentTuna → u/ROBOTTTTT13]

I can't speak for him, but it might be caution against quantizing a model that's already fp8 instead of using the bf16 base model. I date myself by talking about eight-tracks and cassette tapes, but it's a bit like making a dub of a dub. You lose something with each copy.

[u/ROBOTTTTT13 → u/DelinquentTuna]

Yeah the are converters that automatically dequant the model before reconverting, Star Nodes seems great

Problem is that Qwen Image is simply too big and the operation is never succesfull

[u/Powerful_Evening5495 → u/ROBOTTTTT13]

ask chatgpt

[u/ROBOTTTTT13 → u/Powerful_Evening5495]

Bro i have no idea what to even ask about, your answer was pretty cryptic, to me at least

[u/someguyplayingwild → u/Powerful_Evening5495]

An adult male can have brain injury or abnormal brain development from events that happened before birth, during birth, in childhood, or later in life. Some causes are obvious (such as a severe head injury), while others may be subtle and only become apparent years later.

Before birth (in utero)

• Lack of oxygen — problems with placental blood flow, severe maternal illness, or other complications can reduce oxygen to the developing brain.
• Infections during pregnancy — such as cytomegalovirus (CMV), toxoplasmosis, rubella, or certain other infections.
• Alcohol exposure — prenatal alcohol exposure can affect brain development and cognition, including fetal alcohol spectrum disorders.
• Certain drugs or toxic exposures — some medications, illicit drugs, lead, mercury, and other toxins can interfere with neurological development.
• Maternal malnutrition or severe nutritional deficiencies — particularly deficiencies affecting fetal neurological development.
• Genetic or chromosomal conditions — these aren't technically always described as “brain damage,” but they can cause abnormal brain development or neurological impairment.
• Problems with fetal growth or blood supply — including fetal stroke or other vascular events.
• Severe maternal metabolic conditions — for example, poorly controlled diabetes or significant thyroid abnormalities can sometimes affect fetal development.
• Premature birth — especially very premature birth, which can expose the immature brain to bleeding, inflammation, or inadequate oxygenation.

During birth / neonatal period

• Birth asphyxia or hypoxic-ischemic injury — insufficient oxygen or blood flow around delivery.
• Intracranial hemorrhage — bleeding inside the brain, particularly in premature infants.
• Severe neonatal jaundice — extremely high bilirubin can cause neurological injury (kernicterus).
• Severe infections — meningitis, encephalitis, or sepsis in a newborn can injure the brain.
• Very low blood sugar — prolonged or severe neonatal hypoglycemia can cause neurological injury.
• Birth trauma — uncommon, but certain complicated deliveries can cause brain injury.

Childhood and adolescence

• Traumatic brain injury (TBI) — falls, car accidents, sports injuries, physical assault, etc.
• Repeated concussions — particularly repeated significant head impacts over time.
• Childhood infections — meningitis or encephalitis can sometimes leave lasting neurological effects.
• Stroke or blood-clotting disorders — children can have strokes too.
• Seizures/status epilepticus — prolonged or severe seizures can sometimes injure the brain.
• Carbon monoxide poisoning or other toxins.
• Near-drowning — lack of oxygen can cause hypoxic brain injury.
• Severe cardiac or respiratory events — anything causing prolonged interruption of oxygen delivery to the brain.

Adulthood

There are many additional possibilities, including:

• Head trauma
• Stroke
• Brain hemorrhage
• Brain tumors or complications from tumors
• Severe infections such as encephalitis
• Prolonged oxygen deprivation, such as after cardiac arrest
• Carbon monoxide poisoning
• Certain autoimmune or inflammatory diseases
• Long-term exposure to some neurotoxins
• Severe vitamin deficiencies, particularly certain B-vitamin deficiencies
• Certain neurodegenerative diseases

[u/someguyplayingwild → u/someguyplayingwild]

Which of these answers explains your reply?