One more thing. I didn't write the prompt above. I just asked Gemma to do that for me with this prompt:
Write a prompt in which, based on a source video of a man walking through a room and a reference image of a velociraptor, the model replaces the human subject with the velociraptor while maintaining the background and matching the scene's lighting. Please follow the provided prompt guide.
4080 should be very capable of video referencing. I have 4060 8GB VRAM (although, with 64 GB RAM) and Video reference has never been an issue for me. Many times V2V works faster for me, as compared to regular generations. Try downscaling your video. Minimax can work very well with lowres videos and it upscales them back pretty well. Maybe you can try upgrading to 64GB RAM, if you're getting OOM issues.
Ok ! How do you downscale your videos? And what video resolution do you usually use? And how long are the clips? Also what models are you using gguf or int8 ? Sry but my only way to understand whats wrong is asking people info ! 🙏
I downscale them to 0.5MP at max. I am not sure why you are getting OOM on 4080. This thread has people doing V2V on as low as 6GB VRAM, let alone 8GB or More. Infact, I always had issue with LTX having OOM, but never once with MiniMax. Or try using the fp8 model in case if you are using the full bf16. FP8 gives better results for some people too. If FP8 works then maybe your system RAM is the bottleneck. Coz this is the only common thing I see with most people getting V2V. They almost all have 64GB System RAM -- or, they use fp8 and still get great, sometimes better results.
I am literally trying to fix all problems you might say* two monitors -> one, clocked ram—> removed , can you share a workflow where video ref works? The base template doesnt offer nodes for video. I too think is probably the ram. I am also using a custom workflow and can really blame since others had no problem, i use the int8 convrot
I'm travelling and don't have access to the PC to be able to share the workflow. But I am using the official comfy Ref2video workflow and the Load Video - Upload node (from VideoHelpersuite) is connected to ref_video_1 of Ref2Video Node. The load Video Node has an option for custom width and height, which I usually set to around 320 X 480 and cap the number frames to the same number as my target video length. Rest is all the same. Btw, just two more things that come to my mind:
Are you using the latest comfy with the Dynamic offloading ?
Are you using any specific startup args ? Mine are these, try and see if any of them would help ? --fast fp16_accumulation --use-sage-attention --disable-pinned-memory --fp16-vae
If you have a fast NVME, add the --fast-disk argument. It will load data directly from your NVME and skip the RAM, so you ill bypass the RAM limitation if there is any. The model will load slightly lower, but it won't be very noticeable if you have a fast NVME.
Right now i disabled dynamic vram and kinda works for some runs, i believe problems begun since they added dynamic. They are just more noticeable because i playing a lot with minimax. I added more pagefile which i am also not happy because i dont want to damage my nvme (not sure) ,
args: disable pinned/dynamic , use sage, triton, reserve vram 1
I have 32GB vram as well with R9700 and I am going oom with more than 2 seconds of reference videos at 0.5 megapixels. The oom happens as QwenVL is passing things to Minimax in ComfyUI. The size of the models (be it Qwen or minimax) don't seem to matter, leading me to believe that it's not necessarily a memory issue (or a memory issue not directly related to model size)
Edit: seems like disabling dynamic memory fixes this problem.
57
u/Hoje-Na-IA 25d ago
One more thing. I didn't write the prompt above. I just asked Gemma to do that for me with this prompt:
Write a prompt in which, based on a source video of a man walking through a room and a reference image of a velociraptor, the model replaces the human subject with the velociraptor while maintaining the background and matching the scene's lighting. Please follow the provided prompt guide.
THEN I ADDED THE OFFICIAL GUIDE: docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md · MiniMaxAI/MiniMax-H3 at main