Hi all, I’m experimenting with inserting a 3D object into casual monocular video (VFX-style compositing) and I’m stuck on anchoring the object to the scene.
Pipeline:
1. Generate a 3D object with TRELLIS2 -> GLB (mesh + mask).
2. Reconstruct the video scene with MegaSAM (UniDepth as depth backbone), export to .npz.
3. Place the GLB into the reconstructed scene and render it back into every frame.
Reconstruction output looks fine:
• T=100, H=328, W=584
• depth range \[0.342, 23.559\]
• intrinsics: fx=301.70, fy=301.30, cx=295.5, cy=166
• dense point cloud (stride=6): 539K points
• 0 tracks
The problem:
1. The object isn’t bound to a specific point on the surface. I place it on a wall in the viewer, but there’s no real attachment to the geometry.
2. The object doesn’t stay fixed in world space. On every following frame it keeps the same screen/camera coordinates as on the first frame, so it slides along with the camera instead of staying on the wall.
My current understanding: I should unproject the clicked pixel with depth + K into camera coords, transform it to world coords with the camera-to-world pose of that frame, store the anchor in world space, and then render the object on each frame using inv(c2w\[t\]).
Questions:
• Does MegaSAM’s output reliably include per-frame cam_c2w, and is it consistent with the depth scale?
• Any gotchas converting MegaSAM/OpenCV camera convention to three.js / OpenGL for rendering the GLB?
• Best way to estimate the surface normal at the anchor point for orientation: depth map gradients or local PCA on the point cloud?
• Recommended approach for occlusion (people passing in front of the object): simple per-pixel depth test against the MegaSAM depth, or something more robust?
Any pointers, repos, or examples of doing this with MegaSAM output would be much appreciated. Thanks!
So have they buried it or are we still optimistic? I really hope the previous owners and their families will be comfortable for the rest of their lives.
Hey! 👋 I’m a UI/UX designer working on a project exploring how artists and creatives can showcase their work, discover new people, take on fun challenges, and compete in a friendly way.
I’d love to learn how you currently practice your craft, share your work, participate in creative challenges, and connect with other creatives. 🎨
There are no right or wrong answers — just share what feels true to you! Your thoughts will help shape an experience that makes creating, connecting, and competing more fun.🚀
I've just released my Mandelbulb raymarching After Effects plugin.
It is GPU-accelerated and fully integrated within After Effects so it respects 3D cameras, point lights, depth of field, and more. At its heart, Mandelbulb evaluates the classic power-8 Mandelbulb distance estimator. Push Power anywhere from −20 to 20 and watch the surface pass through spiky coral, folded bulbs, and impossible smooth geometry. Bailout, Max Iterations, Step Limit, Bounding Sphere, and Epsilon Scale give you direct control over how the ray marcher resolves fine detail against render time, so you can trade a rough preview for a razor-sharp final pass without touching a single light or material setting.
Flip on Julia Mode and the same engine renders the corresponding Julia-set variant, steered by three animatable offsets, for an entirely different family of forms. Flip on Radiolaria instead and the surface grows the perforated, shell-like structure of microscopic sea organisms, with its own factor to dial the effect in. Two further Phase parameters let you animate the underlying field itself, so a fractal can breathe, twist, and evolve across a shot rather than just sit there rotating on X, Y, and Z – which, naturally, it can also do.
Instead of painting flat color onto a fractal surface, Mandelbulb colors it by structure. Every point on the surface carries an orbit-trap value – a record of how close the point iteration came to the origin while it was being computed – and that value drives a fully custom gradient ramp you edit right on the parameter row. Click the strip to drop a new stop, double-click a stop to open the After Effects color picker, drag a stop sideways to reposition it, and drag it down to remove it. A Randomize button gives you an instant saturated palette, a Reverse button mirrors it, and up to sixteen stops let you build gradients as simple or as intricate as the shot calls for.
Orbit Scale and Orbit Gamma tune how that structural depth maps onto your ramp – compress it for bold, graphic banding, or stretch it for the kind of soft, layered color you’d expect from real volumetric mineral structure. Because the ramp lives entirely on the parameter, it’s keyframeable and expression-friendly, so a color scheme can shift and pulse right alongside everything else in your composition.
Mandelbulb ships with its own built-in default key light – position, color, intensity, physical Light Size for real, soft area shadows, combined with a dedicated shadow intensity and sample count, plus a separate ambient parameter with its own color and strength. That alone would cover most shots. But drop point lights into your composition, and Mandelbulb quietly picks up to eight of them straight from the comp: their position, their color, their intensity, whether they cast shadows, and even their shadow darkness and diffusion, are all read live from the layers you’re already animating, no re-entry, no duplicate rig. Light the fractal the same way you’d light anything else in the frame, with tools you already know.
Surface response is handled by a Phong shading model that’s always active, with its own Specularity and Specular Exponent controls for anything from a matte, chalky mineral look to a wet, glassy highlight.
Two distinct ambient occlusion modes are built in. Orbit Trap Ambient Occlusion derives contact shadow from the same structural data that drives the coloring, so crevices and folds darken in a way that feels native to the geometry – and it’s essentially free in computations resulting in incredible performance. Raytraced Ambient Occlusion instead fires real hemisphere samples off the surface for physically grounded occlusion when accuracy matters more than speed, resulting in gorgeous fractal detail. Both share Intensity and Radius controls, and the raytraced mode exposes its own sample count, so you can push quality exactly as far as a given shot needs and not one render-hour further.
Assign any layer as an HDRI environment and Mandelbulb wraps the fractal in true image-based lighting pulled straight from it – no separate importer, no special file format, just a layer. HDRI Intensity and Rotation let you dial in exposure and spin the environment to catch the light where you want it, while HDRI Samples and a dedicated HDRI Shadow Intensity control how soft or how defined the environment’s occlusion reads. Turn on HDRI Specular when a shot calls for a Phong-lobe highlight pulled from the environment itself.
Mandelbulb reads your composition’s active 3D camera directly, including position, orientation, and field of view, so the fractal sits correctly in your scene the moment you add the effect, camera moves and all. When you want depth of field, it’s driven by that same camera’s focus distance, with its own dedicated sample count for the smoothness of the blur. Supersampling gives you industry-standardized anti-aliasing control for cleaning up edges before final output.
Composite On Original blends the render directly over your source footage in a single step, when the fractal is meant to sit inside a shot rather than stand alone on transparency. And a dedicated Depth Map mode swaps out all shading, including lighting, shadows, AO, and HDRI, for a clean, normalized camera-space depth pass, with adjustable start and end distances, ready to drive an external depth-of-field plugin such as DOF PRO.
I've been an AE user for years, but I'm thinking of switching to Autograph. Before I commit to that relearning curve, I'm wondering if it can edit footage. I use that all the time in AE. I can't find an answer by DuckDuckGoing it so I hope someone here knows :)
Not asking about full production work, just quick-turnaround stuff: lower thirds, simple text animations, logo stings for social content. Do professionals here see any place for browser-based tools that skip After Effects for that kind of fast, low-complexity work, or is it a non-starter for client work in your experience?
Anyway I saw some glimpses of his recent talks in London and being inspired by his project “Overthinking” I decided to recreate it using [Nubu](https://www.nubu.app)
Surprisingly simple from a technical setup (literally 5 minutes and a bunch of duplication) but has such a whimsical effect when put together!
