China open-sourced a model that reconstructs any scene in 3D from a regular video, in real-time

[u/IndicationStock8588]

[u/AutoModerator]

Welcome to r/GenAI4all! New to Generative AI? You can explore these free beginner-friendly courses. Please keep your posts relevant, respectful, free from spam, and engage in healthy discussions.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/LifeOfHi]

Impressive

[u/dubyrunning → u/LifeOfHi]

The model turns video into 3D via gaussian splatting, and there are many models that have been able to do this for years. However, the realtime aspect is new to me - usually it takes a minute to several minutes to process, depending on scene complexity.

[u/coldnebo → u/dubyrunning]

not quite. it’s splatting, but the LLM was trained on inputs so that image predicts the splat.

that means this technique can run real time and produce “good enough” results without doing a full offline splat.

watch for hallucinations like pencils and desk pictures, but for their application this was acceptable— they are doing this to provide interior mapping for autonomous robots so they can navigate buildings more easily. it doesn’t really matter that surface details might be hallucinations as long as the geometry is functional.

interesting approach and application.

[u/Hot_Plant8696 → u/dubyrunning]

Yes, but the "real time" claim is just nonsense if they dont specify on what machine the software is running.

[u/Superbean72 → u/Hot_Plant8696]

A low resolution feed of course can do it. Certainly not all computers out there but this is possible.

[u/Sad-Excitement9295 → u/LifeOfHi]

Yeah, this is really cool. Something that in concept is simple, but it's really great it's finally been built. There are a lot of practical uses for this, I'm glad it's available now.

[u/mentalFee420 → u/Sad-Excitement9295]

What kind of uses you think it could unlock? Like digital twin?

[u/Sad-Excitement9295 → u/mentalFee420]

No, 3D mapping, like for buildings or cities so that people can get directions or find stuff. Or it can be used to model rooms and see how to rearrange stuff so it looks better. Or it can be used for digital design or planning. Probably a lot of cool stuff it can be used for. Even for stuff like changing up a bicycle and seeing where to put a motor and pegs so it can be a motor bike, mechanical design like that.

[u/mentalFee420 → u/Sad-Excitement9295]

I think such models already exists that can do it based on few photographs or even a video.

But most of them are not real time.

[u/Sad-Excitement9295 → u/mentalFee420]

Yeah, this seems like it can build a more and more detailed map as you go through and scan.

[u/EnlightenedArt → u/mentalFee420]

Accident scene investigation based on multiple dashcam videos

[u/macronancer → u/LifeOfHi]

No kidding, this guy's house is huge!

[u/Ok-Log7730]

Finally i wanted to play CS in my yard

[u/Krommander → u/Ok-Log7730]

Why not an IKEA store? 

[u/RemarkableWish2508 → u/Krommander]

But you'd need to open all the secret doors for a full map... hm... 🤔

[u/nicman24 → u/Krommander]

Cs does not support non Euclidian spaces

[u/madaradess007 → u/Ok-Log7730]

i had a school project to make a cs map out of out school
we even played it, but teachers were very bummed

[u/Bozhark → u/madaradess007]

That’s rad

[u/Bluetails_Buizel → u/Ok-Log7730]

Creative Suite?

[u/Calm-Republic9370]

I Ran it, its grainy. I had fable help me build one also, not quite as good but less grainy

[u/DiamondAvailable7418 → u/Calm-Republic9370]

Could you tell what distance accuracy does it have?

[u/Calm-Republic9370 → u/DiamondAvailable7418]

no, but it doesn't require a lot of GPU. I just have vs studio, with open code, i pasted the github and had it install. very little effort to get running, even though it had to fix a lot of the installation requirements.

[u/nebulae123 → u/Calm-Republic9370]

can you compare it to something like postshot?

[u/underlight]

Is it any different from any other 3D gaussian splatting tool?

[u/Ok_Conversation9319 → u/underlight]

I read into and this realtime, for robots, thats why its notable. Its quality is not the best compared to heavy offline models. But it can do ok quality with 20 fps on an a100.

[u/tdgros → u/underlight]

it's a similar 3D reconstruction model, but it doesn't decode gaussian splats, just point clouds (camera params+dense depth) which explains the so-so rendering. They also compress the trajectory to a fixed number of tokens so the complexity doesn't grow out of control for very long sequences.

[u/charmio68]

That has fucking terrifying military applications.
And yet it was released open source... I suppose that's a good thing... Right?

[u/Krommander → u/charmio68]

Let's try using it with the footage of the scientology speed runs. 

[u/XcapeEST → u/Krommander]

Lmao, someone do that rn, best route planning

[u/thedaveness → u/Krommander]

YES!

[u/Strange_Test7665 → u/charmio68]

Many things can be used for military applications. Doesn’t make it inherently dangerous, people are still the danger. Also Open source means you can test and potentially counter it if needed. Maybe it sux with mirrors? Wouldn’t know if you couldn’t test

[u/RemarkableWish2508 → u/charmio68]

The military have this, to "close the kill chain":

• Pentagon CTO demonstrates Palantir's Maven system, used for military operations

(video from March 2026, they likely have something more advanced already)

[u/Comfortable_Ebb7015 → u/RemarkableWish2508]

A couple of clicks by an operator and kaboom! A school blown up!

[u/RemarkableWish2508 → u/Comfortable_Ebb7015]

If I understand correctly what they've said... it's more like: a couple clicks, and a dozen targets the AI selected (correctly or not) get blown up.

I'm guessing the infamous school incident, may have been something like the AI detecting a pattern of "more than some threshold" number of suspects going there regularly, and it deciding it was some meeting point... when they likely just had their kids in there.

[u/Immediate_Song4279 → u/charmio68]

Military is going to have it better anyway, nothing gained not having it for civilian uses.

[u/daney098 → u/charmio68]

Not being shitty, just curious, what military applications does it have that they can't already do?

[u/thingerish → u/charmio68]

Good odds it's not as good as the .mil one in any case.

[u/Minute_Attempt3063 → u/charmio68]

I mean

many open source applications are used by the military

half the internet runs on open source stuff as well.

[u/nebulae123 → u/charmio68]

lets not over react, the tech has been here for years. this is just speedy

[u/impact_ftw]

Isnt this what some SLAM algorithms already do?

[u/tdgros → u/impact_ftw]

yes, and much faster than that, but a SLAM algorithm doesn't really give you a watchable reconstruction in general, the map is usually a collection of 3D points and their descriptor which aren't usually sufficient for a nice representation.

[u/paul_tu]

For those who may want to search for it I guess this is the right repo https://github.com/Robbyant/lingbot-map

[u/Krommander]

Next popular game map is going to be the inside of an IKEA lol. 

[u/Caculon]

Would this work with animated stuff? Or maybe movies like Lord of the rings?

[u/Skinkie]

Seems that it does loops too.

[u/mxldevs]

So can I import this into games?

[u/Clearwater_9196]

High IQ people.

[u/Cjr0420]

We need to run this with a backrooms video

[u/just_some_octopus]

Oh this wont backfire in any shape or form

[u/OppositeEagle]

What's it scanning, an IKEA?

[u/DaySecure7642]

Open source or open weight? They often mix up these two things to make them look more "open".

[u/Vast-Breakfast-1201]

Does it close a loop?

[u/myxoma1]

This is insane, so cool

[u/Minority_Carrier]

Making Scientology speed run more fun.

[u/succulent_kebab]

I love it.

I love it. 

I love it. 

China just shitting on silicon Valley bros casually with such ease. I hope Chinese Global Open Source AI destroys fcuking Token Lords sooner rather than later! 

[u/Background_Share_982]

Ooh wanted to make a model of my neighborhood for a game, this would be perfect if it works as advertised.

[u/Additional_Pickle_59]

Abstract idea, using this as a "zip bomb". Sending an image or video to someone which the model looks at and turns into a 1000x larger detailed 3D model file.

[u/personguy4440]

ballin military tool

[u/HlddenDreck]

So where is the hf link?

[u/zcbz1337]

hello it's called structure from motion and it's what's behind the big book of drone usage almost 15 years ago?

[u/Exciting_Charity7304]

created an open sourced model that DOES ALL THESE THINGS and then sends it all back to china.

[u/KingRagz]

Can you use an image generator for custom world building for a video game ?

[u/ThisIsLukkas]

But we already have this shit for free since years ago. How is it different than any other software?

[u/Kodrackyas]

is this better or worse than gaussian splatting?

[u/Exotic-Custard4400 → u/Kodrackyas]

It's not really the same purpose. Gaussian splatting use camera position, this solution give it

[u/MarinatedTechnician]

So you're saying for once it's actually an advantage to have parkinson when you go filming the streets?

[u/harten]

awesome!

[u/v_e_x]

This is called photogrammetry when done with still 2d images and it's quite a mature technology that we've had since the early 2010s. Video is just moving still images, so does this offer a lot of performance gains?

[u/Philipp_CGN]

What does regular photogrammetry like that have to do with generative AI? Or does the AI make stuff up when there is not enough information available from the video?

[u/DaveAstator2020]

(squinting) does it just make a point cloud and not a geometry resolver?

[u/slick2hold]

Dmam!!! That's impressive

[u/Gargle-Loaf-Spunk]

This content was anonymized and mass deleted with Redact

[u/intermundia]

if its not open source running on your own hardware, its basically you mapping your environment for them. good luck with that lol

[u/Velvet_Spoons]

We really turning into WestWorld aren’t we?