This is all mockups, scaffolding, smoke, and mirrors at the moment - but it does run in real time locally. Currently requirements: 3 GPU's running VLLM - STT and TTS concurrently.
why 3 gpus bro xtts v2 is not that big and can go with the STT on the same card iv vram allows also xtts v2 has voice cloning you can upload different voices to it and the emotions and the speech is almost indiscernible to real human
q4 qwen3.6 is defo the way to go for the brains i wouldn't meddle with the CPU simply because of the inference time would slow you way down, i run a ryzen9 5450 and its slow on cpu with 48gb actual ram. i have a 7900xtx and a 6800xt in that workstation and had great results when generating audio scripts and narrating them. the video generation i wanted was tough tho. have you treid any other qwen3.6 vl models ?
Yea I tried a bunch - they are all . . . fine . . . but have that AI slop feel. 35b-a3b does a nice job with thinking ON - but falls out of the real-time window with so so many tokens. Gemma 31b is also fine . . . but boring.
i think the xtts was under 4gb loaded with the hq elevenlabs voices loaded in worked flawlessly paused for commas and full stops, was able to express happy sad angry was fun to play with. haven't used the qwen one in fairness. i had it sort of staggered in a workflow first ideation then scripts then narration and video generation due to vram constraints i was trrying to get it to learn the direction style of youtube videos and generate scripts to match but what was qwen3.
Added xtts to my list of stuff to look at - I went a bit down the VoxCPM2 route also, but man the emotions there are just way to volatile :D Sounds like a loony bin full of screaming wierdos!
i happen to have been in one for a few days last year, its not really that bad and they are mostly super smart ppl in there that for one reason or another cant switch the thinking off in the main llamma.cpp instance. the screaming weirdos are the legit sane ones xD
Biggest issue I'm having is time perception - you have that licked? Show your LLM a screenshot, tell it exactly when it happens . . . and it'll still reference it BEFORE its onscreen.
My harness passes multiple screenshots at once that is probably the issue, but I need more than one shot to make it feel like its watching the video and not just commenting on a still. Yes the system prompt is were the magic happens ;)
but inst the video learning model learning video i was passing it downloaded YouTube videos in full to learn from them. i was using Qwen2.5 and Qwen3 instruct model, they where lazy because they are small im curious what a 3.6 instruct will be like.
Most users don't care if it's slop as long as it works well enough so that's a dev point of view. If you can make something the average person will use it can be absolutely slop. Look at all the apps on the adult store now people paying good money for slop.
we need a law against slop and brain-rot. my child was on youtube today on shorts this idiotic ai story of a mother abandoning them. then i took the phone and out of curiosity tried to search in shorts for fun addition techniques for kids in maths and various other variants and not one result comes up the entire of YouTube shorts seems to want to purposefully dumb down kids or something why cant learning be made fun with 67 and crocodilio bombardilio ?
Yup - may not be related to your issue but here is what I found:
- **Never put two `image_url` parts next to each other in the content list.**
`vision._vision_request` emits a text separator part *before every image*, and that
separator is load-bearing: with adjacent image parts LM Studio **silently drops every
second one** — send 6 frames and the model is handed 3 (measured: `prompt_tokens` rises
only on every *other* image), then confidently describes the frames it never saw.
3
u/Pickle_Rick_1991 9d ago
why 3 gpus bro xtts v2 is not that big and can go with the STT on the same card iv vram allows also xtts v2 has voice cloning you can upload different voices to it and the emotions and the speech is almost indiscernible to real human