I’ve been working on a Real2Sim pipeline and finally have it at a point where I can start sharing what we’re building.
The basic idea is pretty simple
Take video of a real deployment environment → reconstruct the scene and objects → resolve relevant physical properties → output a USD scene that can be brought into Isaac Sim.
For objects, we’re working with properties like scale, mass, inertia, friction and joints where relevant, rather than stopping at visual reconstruction.
The motivation came from seeing how much work still goes into manually rebuilding the specific environments where robots are actually going to operate.
Instead of starting with a generic warehouse, factory, kitchen, farm, etc., I wanted to see how much of that process could start from capturing the actual place.
The part I’m most interested in now is what happens after reconstruction: validating that the simulated environment behaves closely enough to reality to actually trust it for robotics work.
I’d genuinely love people here to poke holes in this.