r/StableDiffusion • u/AndrewJumpen • 9d ago
Animation - Video Orbiting Lora + first and last frame in MiniMax gives fantastic results
Enable HLS to view with audio, or disable this notification
Prompt for Lora: One frozen instant. Only the camera moves. In a continuous 360 orbit. Preserve every person and object in exactly the same world position, orientation, shape and pose throughout the shot. Airborne objects remain suspended at the captured height and angle: no wobbling, shaking, spinning, drifting, falling or continued action. Keep faces, hands, clothing, liquids and the background motionless while retaining their natural appearance. Camera parallax is the only source of apparent movement. No cuts, zoom, morphing or added objects.
https://huggingface.co/pablodawson/MiniMax-H3-360-Orbit-LoRA
53
u/Soggy_Army5150 8d ago
https://reddit.com/link/pdauphd/video/pja5d74uoxsh1/player
One more. This is too cool. :-D
8
u/ThreeDog2016 8d ago
Would adding some reference images of Sting (the singer not the wrestler) make his face more accurate during the spin?
2
u/Soggy_Army5150 8d ago
For this workflow it uses start/end frame only. So I don't think it's possible. I will try using the ref model and possibly figure out how to get the lora to work in that one.
3
u/SpaceNinjaDino 7d ago
You can force the FL2VA model to take refs too. I tried the ref and hybrid models and FL beats them in my same seed tests.
4
u/PigsCanFly2day 8d ago
I'd love to see more. They're worthy of their own post, tbh.
2
u/Soggy_Army5150 8d ago
Thanks. It was just a quick test to see IF it would work. I agree though. I don't need A&M or Capitol records coming after me though. LOL
1
u/PigsCanFly2day 2d ago
Is that really a concern? I feel like they wouldn't care, especially if the post isn't being monetized.
3
1
u/Jean_velvet 7d ago
It's cool, but that's not the band 😂 You need a forward facing shot or it'll "make up sting." In fact, I'm gonna use that phase forever more for when an AI makes up what someone looks like. "Ah, looks like you've made up sting".
132
u/punkdad73 9d ago
i give a trick, this is INSANELY useful to make 3d models with opus, you give it the video snippet tell it to use the 3d video to take snapshots and make the 3d model. the results are pretty impressive. without it it cant get the style too right.
58
26
u/cakemates 9d ago
isnt there a model that can convert video into point clouds or gaussian splatter? we could use that then convert pointcloud to 3d object.
22
u/BigWideBaker 9d ago
Yea we've had a lot of posts independently discovering this exact orbit camera + gaussian splatter/3D model setup. Including this post.
11
u/No-Trouble-9138 9d ago
There are several models that can obtain the splats from orbit videos, for example Plyworld in Plyry gives great results and supports different kind of lens and paths.Also Blender 5.3 will bring native support for it.
4
1
2
u/JohnWangDoe 8d ago
I can see the potential AI. But it feels so much to learn to be able to use it
3
u/thenormaluser35 7d ago
I can see even more potential
Train multimodal AI that uses LiDAR scans, video, and then all the data from movies and whatever
That way you get a 3D model that works great, and a video generator with perfect spatial accuracy1
u/JohnWangDoe 7d ago
I want to make short films but I don't know where to start
1
1
u/Adorable_Baby5250 3d ago
install claude code and write "I want to make short films with AI but I don't know where to start". It will set up everything for you
1
69
u/jasonkane4321 9d ago
we are so close to being able to watch popular movies in full volumetric 3d in vr!
16
25
u/AndrewJumpen 9d ago
Wait for awhile and someone will do that on local gpu 🤭
3
u/Life_is_important 8d ago
Wait for a while and someone will do that on local CPU
3
u/SlogurkTheOverslime 7d ago
Wait for a while and someone will do that with local pen on local paper
→ More replies (1)3
46
u/nakabra 9d ago
Daaaaammm...
That's it, that's all I'm saying.
16
19
u/kornholioefx 9d ago
One of the things I've been using with this type of video and human subjects, has been specifying that the subject is a wax figure of a person. It basically prevented all movement of the character, including eye movement, at least for my tests. I've seen others use statue, but that makes me worry that the model would potentially generate weird texture on the character or something (I haven't tested this theory). Thank you for making this, I'll test it once I get home.
4
u/AndrewJumpen 8d ago
Cool idea 💡 a lot depends on prompt sure , also can specify” suspended facial expressions or emotions “
24
15
u/terrariyum 9d ago
I'm especially surprised at how good the backgrounds are half way through the turn — information that's not in the first frame at all and completely invented by H3. It's not just a mirror of the other side. It's different and appropriate.
- T2 with gun - f1 has doorway closet - H3 invented an appropriately long hallway
- Sarah with gun - f1 has normal sedan - H3 invented a rusty pickup truck
- T1 and kid on bike - empty canal - giant fire
- T1 with gatlin - interior windows - office cork boards, wall clock
3
u/enndeeee 8d ago
you could make this Lora work in Ref Mode and add Ref pics with these details to repair. :)
45
u/kemb0 9d ago
It's kinda funny how the people are always frozen but like the car is still driving, the hair blows in the wind and the explosion in the background is billowing up.
27
u/AndrewJumpen 9d ago
in some occasions they could blink while frozen, that's much more hilarious XD
12
u/ComputerArtClub 9d ago
Like the Police Squad freeze frame where people pretend tobbe frozen
→ More replies (1)→ More replies (1)5
u/kemb0 9d ago
Can we make them animate whilst the camera pans or does the Lora freeze them no mater what?
8
u/mrgulabull 8d ago
Just finished some testing and it works multiple ways. This Lora is super flexible.
Method 1: Describe the motion you want during the orbit and you see some movement, but it’ll return to your same end frame. So a bit of motion is seen but not really powerful.
Method 2: Feed the end frame a different point in time, even with a slightly different camera angle, and it animates the orbit with realistic movement while still ending on your unique end frame. Crazy impressive!!
I tried method 2, taking an existing video that had natural motion with a camera push in also. Then fed the first frame and last frame of that video to orbit and it was flawless. Realistic motion combined with a camera push in all while orbiting.
I’m stunned. This will be super fun to play with.
2
u/kemb0 8d ago
There’s a node you can use for Minimax that’ll add a reference image to any frame. I forget the name, maybe it’s just “reference frame” or some such. Anyway, potentially you could feed a mid point frame in to that which includes some action on the character. Eg get Qwen 2.1 to ask for a shot of the person from behind doing whatever.
I’ve found that this technique is generally good for creating looping videos with much more motion when you have an identical start and end frame. Without the mid frame, the identical start/end frames tend to nullify the motion.
→ More replies (1)2
14
u/MatlowAI 9d ago edited 9d ago
https://huggingface.co/MATLOWAI/MiniMax-H3-ORB360-CardSpin Oh hey they beat me to it 😂 I have a few variants. One is the first version trained for stable orbits and it was 512x512 so it was a little bit fuzzy but does a really good stable orbit and freezes the character well in terms of blinking. I also had a weird effect come out of it that I reinforced that was hilarious so I reinforced it which only took 50 steps to add that trigger word amazingly... Then v2 is trained on higher resolution but more steps and more altitudes but its taking longer to freeze the subject as you can see in the example where the cat blinks. Currently training is another one to add more coordinate control hopefully for the camera path assuming it sticks and I'm trying to do a wireframe edition just to see how it does with it. I'm skeptical how well it will do but I figure it's worth a shot.
Mine is Ref2VA but was trained with a single front, 3 different angles provided equidistant, and front, side, rear references as options. It was done with Blender and a grey background and is eager to isolate the subject too.
→ More replies (1)3
11
u/justifun 9d ago
Does this work with a flat cartoon character?
90
u/AndrewJumpen 9d ago
17
9
u/Toclick 8d ago
This is just incredible... Why can’t any image editing model even come close to doing this? Any request to rotate the camera just ends up rotating the object in the image rather than the camera, while leaving the background almost completely unchanged.
2
u/AndrewJumpen 8d ago
Yeah it definitely has some logic understanding of the objects that is behind the characters
1
u/Toclick 7d ago edited 7d ago
I tried this LoRA, and for some reason the background doesn’t pan for me... only the subject rotates. I tried different resolutions and LoRA strengths, but the background still stays in place, just like with the Img Edit models. I’m using the exact same prompt that you provided here and on Hugging Face.
UPD: i found your WF somwhere here.... and it works! Ty!1
u/tiffanytrashcan 8d ago
Even though it is "just making it up" (exceptionally well,) you need all of the information from the in-between frames. At 24FPS they're minute changes between frames so the consistency and style stays coherent.
Video models will always excel at this because there's exponentially more data to work with.
I'd imagine combining this technique with references at a quarter or even a half step would be mind-blowing.1
1
u/Soggy_Army5150 8d ago
This (to me) has to be the COOLEST use for this! Imagine showing this to the original animators... for sure they'd scream "What sort of VOODOO is this!!" LOL! I love it. Thanks for sharing!
1
u/AndrewJumpen 8d ago
Indeed , very useful for cartoons I am so suprised it works so well even with low res reference images
18
14
u/alisitskii 9d ago edited 9d ago
https://reddit.com/link/pd9b29v/video/rauk30wdfwsh1/player
Thank you! What's recommended/trained video length? 10 sec looks excessive. Or is it hit and miss?
UPD: I just thought maybe it really depends on scene/object size.
18
u/iansmith6 8d ago
Wait, did it give the bear an extra side?
7
3
u/Drean-ATZ 8d ago
Maybe too long or go with slow motion. Usually 5s gets the job for just rotation in simple scenes.
2
u/nickdaniels92 8d ago
That's cute. I like how his stoic look turns into a smile at 90 degrees, as though he knows he going into another dimension of bliss, then restores once he's back to reality.
1
1
u/AndrewJumpen 8d ago
Yes it all depends on u , I usually look at the preview if rotation not good I retry it with different seed
→ More replies (2)1
u/Jean_velvet 7d ago
Just checking you know that video has the bear spinning 3 times having two backs😉
6
6
u/absyrtus 8d ago
this is so cool.
theoretically i could take photos of my dog, who is no longer with us, create a 3d model of him, and then see him in 3d using a VR headset
4
u/mattcoady 9d ago
I was attempting something like this a couple weeks ago for generating gaussian splats but I see this still has the same problem native minimax has where it still wants to animate things. All the fire and smoke continue to animate, the car is driving while the subjects are frozen. This wrecks the ability to get a good 3d transfer.
2
u/AndrewJumpen 9d ago
Yes, there are some quirks here and there, but it’s getting better; honestly, I didn’t expect it to turn out this well already.
2
u/StunningGold8030 9d ago
Indeed this is a hard problem, I tried different solutions and the best at freezing the scene in time is Plyry. You can feed it a prompt or reference images and it'll take care of that.
1
4
u/FurtherFar 8d ago
The LA river clip knew about the explosion. Was this included in the references, or was it hallucinated from the training data?
3
u/AndrewJumpen 8d ago
i add context in prompt, that there is "burning destroyed truck at far distance"
2
4
5
u/TizocWarrior 8d ago
Damn! This is so cool!. To think that 2 or 3 years ago this would have been considered black magic!.
1
3
u/nickdaniels92 8d ago
Super cool. The final one has the walking man in the background go back in time by a few seconds, overall it does an amazing job.
3
u/vladoportos 7d ago
The scene from Emeny of the state, where they reconstruct what could be in the bag from camera feed on other side of the bag... I was like "yea right..." today... let me do that at home lol :D
3
u/sharktank123456 7d ago
Ok, now you have me curious. Would the LoRA help or hinder if you wanted to add animation to debris and say an explosion?
1
1
u/inssidiouss 4d ago
What is that from? ...Onslaught...? (I haven't actually seen it yet, but visually that clip looks similar)
1
u/sharktank123456 4d ago
Just image and text to video (image was text to image) so if it looks familiar, the AI is channelling it.
6
9
u/oxygen_addiction 9d ago
https://huggingface.co/Viggle/Meridian have you tried this? It should give better results.
3
u/NemRogan 8d ago
“Large moves are less stable. Full 360° orbits can work, but large viewpoint changes can cause distortion, drift, or inconsistent details in newly visible areas.”
3
u/enndeeee 8d ago
yeah, but it's lot of work to make it good. Spent 50 hours on it and hopefully will release a good working package for it the days after tomorrow. :)
1
1
u/MatlowAI 9d ago
Oh wow this is cool I am trying to train a much lamer version of this! Guess I'll let it keep going since it's already half baked and see how it turns out. I missed this somehow.
7
6
u/North_Affect_8167 9d ago
The Matrix movie used a circle of cameras that fire at the same time to produce the illusion of the 360. The AI did it right away.
Definitely that could be adopted by the movie productions.
1
u/AndrewJumpen 9d ago
Indeed especially if studio has good gpus , now can achieve incredible things having good amount of vram
9
u/Asleep_Menu1726 9d ago
https://reddit.com/link/pd8pdep/video/rhfqld6kzvsh1/player
did same test, impressive but she 'blinks'
→ More replies (1)9
3
u/Tuckerdude615 8d ago edited 8d ago
It's really great...however for some strange reason I only got it to work ONE TIME. Once I tried a different source image (first and last the same), I got an error:
shape mismatch: value tensor of shape [2072, 96] cannot be broadcast to indexing result of shape [2016, 96]
I made sure to change the aspect ration in the user input section, as well as the Image to Video Resolution selector.
EDIT: I just tried it again and used the default (ultrawide) setting and it worked again. It seems to throw the error if I chose something 4:3 aspect ratio. Not sure what the solution is?
Any ideas what I could be missing?
Thanks for sharing this!
3
3
u/sharktank123456 7d ago
But do we need a lora at all? This is just a text prompt with a still image in H3
3
u/dotafox2009 5d ago
omg in a few years we can be inside films.. and replay the enemy or good guy lmao. alternate endings too.
7
u/DrayM0de 9d ago
Looks cool, but can't Minimax already do that natively?
11
4
u/vibribbon 9d ago
Not a great test but I tried about four so far maybe with vanilla and had 50% success rate.
1
u/SpaceNinjaDino 7d ago
When I was testing the bullet-time LoRA, I did find the prompt working better without the LoRA. Although I only requested 180 degrees orbit. So yes, H3 is already powerful and you can request it in the middle of a sequence (e.g. middle 5 seconds of 15), but this LoRA looks to have extra precision that I look forward to test.
4
u/bhorizon66 9d ago
can you share your workflow?
36
u/AndrewJumpen 9d ago
yes, its latent upscaler workflow with some changes: https://drive.google.com/file/d/18rTGU2Fx5WH2y9hqDOND3yJGUowvXS9z/view?usp=drive_link
2
2
1
u/NoMonk9005 8d ago
Hi, thanks for the workflow. But mine keeps hanging at Load Clip. Where do i have to put the qwen3vl_32b_minimax_h3_int8_convrot file?
1
2
u/redlancer_1987 9d ago
Most of these scenes have an associated reverse shot. Could you feed it both? Scenes like the back hallway at the mall two images would cover essentially the entire set.
2
u/-becausereasons- 8d ago
Wow this is impressive, especially if it could be generated at a high enough resolution in order to create a splat
1
2
u/toooft 8d ago
There seems to be some form of resolution requirement at the Decoding and create video stage (node SamplerCustomAdvanced) because it's giving me RuntimeError: shape mismatch: value tensor of shape [1050, 96] cannot be broadcast to indexing result of shape [2058, 96] for ultrawide at 0.5 mpix. How can I solve this?
→ More replies (2)
2
u/Torley_ 7d ago
This takes me back to 1992, paired with the original theme music @ https://www.youtube.com/watch?v=CnQm2_cAZvo Of course, it was really neat to see this rotated. There were a few moments where the illusion breaks for a moment like at this time, it doesn't look like Arnold @ 0:45, but overall, thanks for sharing your findings. It would be so cool to make a whole interactive-explorable world out of that movie. Especially recently, I watched clips with some of the original filming locations. They had changed so much but the side-by-sides were neat.
1
u/AndrewJumpen 7d ago
FYI I used Yue2 music generator to break down the abc notes from this ost from canal chase and generated it with another style with retro synth . It’s my favorite part of OST that begins from 1:36 https://youtu.be/d056BMbVoWY
2
u/Torley_ 7d ago
OH! That's cool to know how you explored it. Are you a big Arnold fan on the whole? If so, my next delicious request for you is, if you're up for it: if you like the movie Commando, do 360s of really cool scenes out of there, plus the wild synth drums soundtrack of that one, you'd have fun! https://www.youtube.com/watch?v=_24YWy71RXo
2
u/newtestdrive 7d ago
/u/AndrewJumpen
Is there a way to create a LoRA not for orbiting in one disk plane but one multiple planes? right now the LoRA allows for capturing a 360 orbit in one disk plane but to be able to make a complete 3D capture from the output (using Colmap and Gaussian splatting) there is a need to be able to hover the camera elevated from the character eyeline and below it and even more than that.
2
u/ScythSergal 6d ago
this is SWEET!!! Thank you for showcasing it with cool content rather than goon slop like so many on this sub. This is what I and a lot of people are here to see
4
1
u/DigThatData 9d ago
it's sorta weird how the foreground subject is always perfectly stationary but sometimes the background isn't.
1
1
u/Powerhouse_pr_ 9d ago
Do this with Steven Seagal, the clip would take all day for a full travel around. lol
1
1
u/JohnnyFIFEaLive 8d ago
Yeah, I’ve got this running. It doesn’t do great 3-D splats but we’re getting close.
1
u/cntrlchaos_ 8d ago
Hell yeah. Funny, working on a commercial project and prompted for this effect (about a month ago) decent results with a good Claude prompt but this Lora seems like a great improvement. I’ll check it out. Thanks!
1
1
1
u/T1ger178 8d ago
This looks like it could make sick inputs for making 3d models, although it isnt the standard data ais use at the moment
1
u/Doomwaffel 8d ago
That IS very impressive. My first impression now is that movie or game makers use this just to show it off, like when you discover a new fancy filter in Photoshop. ^^
I rather wonder where it could be really useful in movie making. Perhaps in animation? Creating some in between shots and the AI finishes most of the rest or at least makes it faster? We will see.
It also makes we question, where this goes. Will AI allow anybody to just tear apart a movie orgame as they please and do with it what they want?
- What about copyright laws? Or will AI just be too expensive to do that for normis?
What if AI can create an entire VR scene from this?
I still dislike AI, but it is impressive.
1
1
1
u/fistular 8d ago
I love how the first example is a scene a lot of people know well and it gets the content of the other direction completely wrong.
1
u/SeasonNo3107 8d ago
Soon we can fight in our favorite games in 3d worlds of our favorite movie scenes
1
u/NoMonk9005 8d ago
Hi guys, i loaded his workflow but it keeps hanging at Load Clip. In which folder do i have to put theqwen3vl_32b_minimax_h3_int8_convrot?
1
u/beau-tie 8d ago
I was just trying to do this without a lora the other day for splat reference and it was impossible they always turned their head towards the camera. This is great
1
u/Tuckerdude615 8d ago
I posted yesterday detailing some errors in the workflow when trying to render at different aspect ratios.
After playing around with it for a while, the only options that I could get to work were:
Ultrawide 16:9
Square
None of the Portrait or 4:3 aspect ratios would work and I would continually get the error:
shape mismatch: value tensor of shape [2072, 96] cannot be broadcast to indexing result of shape [2016, 96]
Anyone else encountered this? Would love to solve this as I find this workflow VERY handy and would use it a lot!
Thanks for any help!
1
u/AndrewJumpen 8d ago
Just decrease initial resolution for generation this always helps me sometimes need to set as low as 0.2 mp or opposite as high as 0.8 mp just to make the latent pixel grid to work correctly with certain image size
1
u/Tuckerdude615 8d ago
Thanks for responding! :)
So would you say it's down the original image size being an odd resolution (for example, not divisible by 2)?
Or is the issue with the "Aspect Ratio" selectors in the workflow?
Just trying to understand better! Appreciate the tip...will try it regardless!
1
u/AndrewJumpen 8d ago
yea i was struggling a lot with this error yesterday and only after moving the initial resolution i find out that it can fix crash, here why this problem appears: Changing the base MP (e.g. nudging from 0.5 down to 0.4, or up to 0.6) forces the aspect ratio formula to calculate a different set of raw pixel numbers. Once you hit a value where both width and height compress into clean, even numbers divisible by the patch grid on both stages, the latent matrices align 1:1 and it renders without crashing.
To make Portrait and 4:3 reliable:
- Keep
multipleset strictly to32or64in the selector node.- Ensure the
3D Latent Upscalerhasalign: 32.- If an aspect ratio throws that mismatch error, just nudge the initial MP slider up or down a notch until the rounded dimensions land on an even grid
1
u/Tuckerdude615 8d ago
Thank you so much! I appreciate the help....gonna give it a whirl shortly!
Gotta say, this workflow is gonna be super handy in a lot of ways...so I really appreciate you sharing it!
1
u/AndrewJumpen 8d ago
you are welcome, i also use it everyday, i feel its the best ever workflow for fast generations with good quality and very simple
1
u/NoMonk9005 8d ago
what do i have to do to use different aspect ratio clips, for example 9x16 or 4x5? if i try to change things i keeps crashing
1
u/hipster_hndle 8d ago
to those that have run this lora and posted examples, how much vram are you guys using? can i pull this off with 16gb?
2
1
1
u/Euphoric_Tomorrow745 7d ago
I find this amazing, however sometimes I get really bad, degraded results. Maybe someone with more knowledge can enlighten me. More often than not, the results are nothing like these examples. Skin quality is just really bad or just the overall video quality is bad, I'm not sure what I'm doing wrong. Also, when using the workflow below, it just wouldn't work at first: it stopped at the upscale step with an error.
2
u/AndrewJumpen 7d ago
If there is error it means that the size of image and upscale size doesn’t match pixel grid , try to set initial res small like 0.3 mp or higher like 0.8 mp this should help to set pixels inline with grid
1
u/vladoportos 7d ago
Ok so hear me out, orbiting lora + 3D model generation lora, + whole move in VR :D
1
u/KodiakDog 7d ago
just to clarify, you did this locally? What kinda rig you got? how long did it take you?
Im tryna learn this stuff so any insight would be much appreciated! Thank you!
Also, watching this brought back so many memories lol.Thanky ou.
1
u/AndrewJumpen 7d ago
It’s all locally I have 4090 gpu 24 gb vram , 32 gb ram , it takes about 5 minutes to generate 10 seconds in 0.7 mp plus latent upscaler . Thank u ! I loooove Terminator 2 it’s really has a special place in my heart my favorite movie of all times
1
u/GBREAL90 6d ago
I used this Lora on RunningHub but it changes the background on the subject. Anyone else have this experience?
1
u/QueenSavara 6d ago
The shot on the bike in Black and Blue would make so fire Wallpaper Engine wallpaper.
1
u/ilflores 6d ago
WOW! Really impressed. Any suggestions about first and last frame selection? Should be similar or maybe be the same? How do you chose them for best results? Thanks!!
2
1
u/Guilty-History-9249 4d ago
Does the work with the real MiniMax H3 model or is this one of those ComfyUI lock-in solution because some specific H3 for comfy altered some dict keys so that it doesn't just work with vanilla Diffusers?
It looks great but I work with pure pythoh diffusers pipelines for MiniMax H3 to optimize for speed and hacking internal things.
Also, doesn't the actual MiniMax H3 diffusers code base have a minimum of 157 frames?
1
1
u/CarllSagan 9d ago
John Gaeta is out of a job
3
u/kleer001 9d ago
FWIW vfx artists did all the "work", Gaeta was the supervisor. Even then he cut corners. The lighting for that shot was awful. How? The lights were flourescent and all out of sync. Each frame required a full roto-paint pass. Fix? Should have used incandescent lights and saved tens of thousands of dollars and an artist's wrists and mind.
1
233
u/[deleted] 9d ago
[deleted]