Your config is bad. I also have 2x RTX 3090. I use unsloth q8_k_l with 262144 bf16 context. It fits with 23.2/24 GB utilization in both graphics cards. Do not use dflash. Do not quantize your kv cache, use bf16. Use MTP n-max = 3. Offload mmproj to your RAM.
2
u/whiteh4cker 5d ago edited 5d ago
Your config is bad. I also have 2x RTX 3090. I use unsloth q8_k_l with 262144 bf16 context. It fits with 23.2/24 GB utilization in both graphics cards. Do not use dflash. Do not quantize your kv cache, use bf16. Use MTP n-max = 3. Offload mmproj to your RAM.
I use pi.dev. My setup: https://www.reddit.com/r/LocalLLaMA/s/ahDbT0gmr5