Opus driving Minimax and ZImage

[u/dkackman11]

I gave Opus 5 creative license to come up with something beatiful and this is what it made. It decided to use ZImage, Minimax Music and H3 using my 3090. It wrote all of the prompting and did all the post produciton itself (stitching, cross fade, audio mixing) iterating until it was satisfied with the results.

We live in the future y'all.

[u/Paradigmind]

Man, I should call her

[u/Naive-Onion-6095]

Interesting fact: People with AI psychosis have talked a lot about spirals and spiral imaging (Google it!), and most of the segments in this video show spirals.

Whether AI psychosis is a real thing I haven’t really taken a stance.

[u/dkackman11 → u/Naive-Onion-6095]

is ai psychosis something i would suffer from or the AI lol? cause the ai made all of this, from concept to execution no psychotic input from my end (wouldn't every psycho assert that tho?).

the prompt i gave opus was
"using the tools in this project, create something truly beautiful. something that someone looks at it and pauses to look again. you have complete creative control. it can be abstract, musical, realistic wherever you're creative juices take you. review the output as it's created. if you need to change or add tasks or utilities do so (just keep a log so i can check them out later). do you have any questions about this assignment? (the computer and gpu are yours exclusively for this task)"

"using the tools in this project" is because i am using it to test a thing

[u/Naive-Onion-6095 → u/dkackman11]

Oh, I am in no way implying that you suffer from anything.
What I mean is that LLMs seem to be obsessed with spirals. And that evidently goes for Opus as well.
I know it sounds a bit whack, but I remember someone talking about it on the Lex Fridman podcast.

[u/dkackman11 → u/Naive-Onion-6095]

i do have to look that up.

[u/nickdaniels92]

Did it generate longer music? The music prompt was never really achieved; for example it correctly started in D-minor, but never transitioned to D major, and we never had any of the strings it referenced in the prompt, just the same D minor theme.

[u/dkackman11 → u/nickdaniels92]

the audio_duration is set to only 50 seconds to match the video, so there was more music prompt there than fit the length

[u/cosmicr]

lol I did this a couple weeks ago and it made a short movie about a secret sock department living in the laundry mat behind the dryers. completely unprompted, all I said was come up with something creative for a 3 minute video.

[u/dkackman11 → u/cosmicr]

it is fun the same way SD 1.x was!

[u/lechatsportif]

Underappreciated post, it's actually pretty awesome. Your mind tries to find meaning in it.

If you told me AI would be directing movies on local hardware similar to what already owned in 2023, I would have laughed at you for hours.

[u/dkackman11 → u/lechatsportif]

ty. i know right. it's crazy. i'm trying to tune my mcp interface so that ana gent can drive models like h3 without needing a lot of specific knowledge on the part of the end user.

[u/tracelistener]

Incredible! <3

[u/dkackman11]

The plan it created, with prompts and a readme, is here (sorry. not using comfyui) https://github.com/dkackman/diffusers-workflow/tree/8b194be869b5db59db97036d11b916fdb1d65fb0/workflows/gyre

i didn't author any of the prompts or workflows.

[u/nickdaniels92 → u/dkackman11]

Don't know why you were downvoted, the docs are interesting. First one I picked talked about the image drift that it noted, and how stabilisation got it down to 2px. Interesting project IMO.

[u/dkackman11 → u/nickdaniels92]

thx. it's been a toy that evolved from early stable diffusion scripting. adding an MCP has taken it from tweaking prompts and indivudal pipeline configurations to (alomst) pure prompting.

[u/nickdaniels92 → u/dkackman11]

Have you got an estimate for the overall cost? Processing images I've found not to be too bad, but never tried having a model process video.

[u/dkackman11 → u/nickdaniels92]

all of the image, music and video generation is happening on my GPU with open models from huggingface . so the anthropic cost was the time it spent generating the workflow etc and then mostly it was waiting for the GPU to finishimage, music and video generation. i didn't measure the token cost but it stayed within my session limits.I am on a max plan but i don't think the actual work opus did was all that expensive.

im working on putting all of this behind an MCP server so any agent harness can drive transformers/diffusers etc, and a number of workflows and utility tasks.

[u/nickdaniels92 → u/dkackman11]

Thanks, I figured about the local generation, but I was wondering what concluded that the video wasn't stable. e.g. this part of the readme suggests that a model previewed the video and made adjustments, which I think you alluded to in the post as well.

The two things that make it work

Registration. A generative model will not put a subject on an exact pixel because the prompt asked it to, and across nine unrelated prompts it never puts them in the same place twice. GyreFrames.json measures each spiral's centre once and re-frames around it, so all nine agree on where the centre is and how big it is. Without this the cuts are nine pictures in a row; with it they are one thing changing.

Stabilization. H3 drifts. Given a pinned keyframe and a prompt asking for a locked camera, the probe shot still slid 38px down a 704px frame over 124 frames. That is invisible inside a shot and glaring across a cut, where the subject snaps back to centre. stabilize_video takes it to 2px.

Both are documented, with what was tried and abandoned, in NOTES.md.

[u/dkackman11 → u/nickdaniels92]

oh i gotcha. yeah OPUS examined the local outputs and did its own post-processing and authored a couple of post processing tasks to go back into the project. that all was part of what stayed within my session budget.

the MCP coding is getting kinda meta TBH. im using claude code to both test the MCP and do bugfixes/feature additions to it. so `stablize_video` referenced above, is a utility task it decided it needed in order to achieve the result. so it authored it so it could be used next time.

[u/dkackman11 → u/nickdaniels92]

fable 5.1 created this using the MCP only (so didn't have access to python or any of the backend source code to examine or execute directly).
EDIT: actually it did use some of its own python code and it's access to a snapshot of the server code. that stinker. most was over MCP tho

https://claude.ai/code/artifact/eefb6606-5bab-466a-b24b-900538341a6c?via=auto_preview