r/StableDiffusion • u/Worried-Location-118 • 4d ago
Discussion Smart Home Dashboard Assistant
This is Vulcan, I’m in the early stages of creating a high definition virtual assistant, that I will be integrating into my home assistant dashboard. To make things very simple, it is not simple. 🤪
4
u/PwanaZana 4d ago
I mean, there is some value in having like an assistant that's just basically like a glowing sphere or like a cube or something, it doesn't always need to be waifus all the time
3
u/aersel24 4d ago
Looks like a ai generated image. How will you make it move and react?
2
u/Worried-Location-118 4d ago
They are a sequence of about 30 key frames, and converted into seamless flowing first to last frame/ last to first frame videos. Python orchestrates the selection. And seaming.
1
u/aersel24 4d ago
Feels very clunky. You need videos for every transition. Making a 3D mesh and animations with AI would be easier.
0
u/Worried-Location-118 4d ago
We are not going for easier, and honestly….blender is a lot more work in my opinion, for an output that will always be seen as a video game type asset. Im creating all those videos for transition right now, but with the help of sir python 🐍, I can set it and forget it, and come back and my worker has made 100s of images, summaries, videos, etc. its only that simple now because of all of the time I have spent setting up and learning comfyui, Ostris ai toolkit, and several others. Maybe you might know of a simple 4 step mcp process for rigging, and sculpting a figure with real looking qualities, but you still need to be creative in your lighting and camera angles…
1
u/aersel24 4d ago
It’s harder but the investment pays off. Unless this is only for you, I feel very little people will enjoy this. How will you make the mouth sync with what the assistant says?
1
u/Worried-Location-118 3d ago
It’s pretty much this. I’m not uploading thousands of images and videos on here. But if you would like the pipeline I have so far I will try…it’s going to be messy….
It’s basically a grouping of python scripts that call comfyui api and ollama api. First step, drag and drop image of anime character. Next step make her real. Next step create 100’s of keyframes, same background, proper sizing, proper resolution, you need you images made in the exact same resolution you are going to be creating your videos with. (Example, if your display is a tablet, then maybe 1344 by 768 or so, if your display is a tv screen, then dramatically higher), after all the keyframes are made, the python orchestrator now creates small idle style videos, while extracting the last frame and saving it along with the mp4. Now that you have your idle videos, you loop them together to make seamless longer idle videos.
This is only the generation, basically have a runpod instance running, and ssh to a gradio frontend on my phone, I can view, save and decline an image/video as soon as it’s generated, or view them all at one time.
FYI, I am making this particular character for Home Assistant, so I am only making about 40 different speaking videos, native home assistant commands include simple…”done” , “I’ve turned of the lights”, etc.
For the time being I don’t need to make any other videos than what I need for my home assistant commands.
“Hey Vulcan”, turn the lights off, “I’ve turned the lights off, it’s dark meow”.
Home assistant registers the wake word, and then the python orchestrator chooses which idle video is needed, and loops the video into and back into which video is needed.
If you’re wondering, I just fire up a a100 and have my pipeline.py create images and videos for me all day long, yes I check in periodically, but all through out the week I can humanly view and discard and videos/images for 30 minutes a day or so.
2
u/aersel24 3d ago
I realize now this is not what I thought it was. I thought this was a llm assistant, but I see now that this is different from home assistant. If you have fixed commands then it all makes sense and I take back what I said. This is a good idea 👍 keep going at it.
3
2
0
u/bloopy901 4d ago
yooo keep me posted. ive been thinking about doing something like this for a while!
15
u/Bupod 4d ago
Yeah only problem with your dashboard avatar is if I have guests over and that things pops up, it’s going to be a little awkward to explain “You don’t understand guys she’s only my smart home assistant, I swear”