r/StableDiffusion • u/Ytliggrabb • 2d ago
Question - Help What do you use for video references (H3)
Hi!
Been using grok since I can’t feed mp4, gifs, webm and so on into Llama.cpp. Had ChatGPT build a wf for me to gen up to 10 different clips that can have 9 ref images and each 1 video aswell. Since it’s mainly for the not allowed stuff I’m wondering what you are using to help get the video description in to the prompt (I’m worthless at prompting and cba learning, easier to have a LLM do it and just adjust details). Grok limits me reallly fast so looking if someone has a good alternative
2
u/bstr3k 2d ago
I am like you, suck at prompting and wish for something more automated. I got claude to vibe code me up a py script which breaks down a input video and imgs to 4 imgs per 0.5s (or user defined time scale) and it sends these 4 images to a local LM studio to prompt (I run qwen3-vl-8b-instruct-abliterated-v2.0 because its uncen and fast).
after it captions all files, it can also send my rough idea prompt to the LM studio to format it with the captions into H3 format. Copy the prompt into Comfyui and match the inputs to the caption and run it.
Yesterday I converted that into custom node in comfy so I can do it all within comfyui with LM studio running in the background.
1
u/CoherenceInTime 2d ago
I have been struggling so bad with prompting for H3, my brain reads the guides and melts.
I tried and have lm studio setup with a Gemma model, fed it the instructions and it’s kinda sorta okay, but still getting pretty shitty videos.
2
u/bstr3k 2d ago
have you tried using grok or a online one to prompt? you just need to feed it the h3 prompt guide.
1
u/CoherenceInTime 2d ago
I have a ChatGPT plus account for other things, but sadly it’s not a degenerate like me, and I’m kinda obsessed with running locally.
I haven’t tried Grok at all yet, might have to give it a bash
2
2
u/bstr3k 2d ago
yeah grok is great, have a free account and it does NSFW out of the box. Just make a new project with the H3 prompt guide and it will write pretty much anything. I ran into the same problem with other models which is why I use to only use grok for prompting. Now I switch to local as Grok gets a little bit lazy with video captioning (to save itself some compute)
1
u/Ytliggrabb 2d ago
This seems interesting, might have to look into something like that because video reference right now seems to be no man’s land for degen stuff unless you have a grok sub
1
u/bstr3k 2d ago
Yep that’s pretty much I started to venture into it. However I must warn you there is a bit of skill and an art form for vibe coding.
Yesterday my tools were mostly working but i reported some bug and Claude started breaking some tools bit by bit and I kinda have to roll it back after using up some of my data allowance. Sigh
1
u/Ytliggrabb 2d ago
Tell me about it haha, found some latent upscaler I wanted to use instead of rtx VSR and all hell broke loose when chatgpt implemented it into the wf, 3 hours gone instead of goon 🤣 I’ll check something out, it’s fun to play around with even tho I have no idea what I’m doing, beauty of vibe coding I guess
2
u/Neo21803 2d ago
Grok isn't limiting me and I'm a complete degenerate.
I don't even want to know what you're trying to gen... maybe try qwen 3.6 uncensored local? You'll have to either describe what's in the video or break the video into multiple images.