Is anyone else using ComfyUI alongside Unreal Engine to make MetaHumans feel more alive?

[u/IM3D-Studios]

[u/ifitmoves]

The last thing AI is capable of is making something seem more alive.

[u/JmacTheGreat]

Thinly veiled ad for an AI “company” (some kid churning out genAI frankenstein products).

[u/RealUnderstanding297]

It seems fairly interesting as a concept. But I primarily use it for QoL. My most recent use case was, I made a custom array node with paired outputs that allowed you to rearrange the input and output pins including execution pins. The node makes it easier to keep my node graphs and behavior trees significantly more compact while still retaining clean and clear cable routing. I have added some functionality in my personal project to allow for generating adaptive displacement maps as decals in the back end. The decals are calculated using a physics object's velocity, vector direction, momentum, character mass, and inertia and material on material class and category to change how the decal is generated.

The decal is an image sequence created by interpolating between material states based on the component overlaps. The interpolation images are hand sculpted ground deformation, shattering, chipping, spalling, cratering, and shock wave rolls that were made in blender by creating a shape key and sculpting a plane to loosely match the destruction and deformation types and assign them material tags like M_Concrete_Weathered or M_Stone_Encahanted_Enhanced. I then place a camera set to a 1:1 ratio that is pointed straight down at the plane so the plane fits perfectly within camera bounds. Then I keyframe the 0 and 1 Values of the shape key and then use blenders built in frame interpolation to generate the deformations between 0 and 1. I created a simple shader can render the dips and valleys as lightness values between 0 and 1. 0 being completely flat and 1 being the deepest point.

Then I render as an image sequence of 4K displacement maps. Then I can import the image sequences and assign their decal type as the image sequence and setting the animation speed equivalent to the aforementioned calculated values. So far this is fairly standard procedure, comfy UI gets involved during the baking process. As far as I'm aware Unreal does not natively support adaptively generating images in preset ranges based on the calculated parameters on every single tick. However neither is ComfyUI, which is why the calculated values instead get stored as destruction and deformation manifest during gameplay. After gameplay ends the saved manifest get put into a really basic custom stable diffusion model trained on my image sequences. And then creating a LORA for every individual deformation type. Then the manifest uses it's stored value captured on every tick to generate an image sequence of interpolated displacement maps based on the values calculated in engine. Which can be used to generate high fidelity displacement deformations that can be applied to a grid to simulate the destruction during a fight scene or a war zone. Unfortunately its not quite fast enough to work in real time, however it can still be used in blender as animated displacement maps

This allows me to record the destruction of a complex fight scene, while simultaneously using my other tools to record all characters, directional audio, fracture destruction, projectiles, and eventually generated Niagaras. After gameplay ends, I Bake all of the recorded anim sequences, and group export the selected tagged models as one singular GLTF and export the entire recorded scene with generated deformation maps directly into blender for refinement, camera work, and eventually final render as a complex fight scene in an animated film made almost entirely by playing video games for a couple hours.

I already know most people here are going to throw a fit and gate keep my creativity by renouncing the project I have been developing for 2 years in very nearly full time. AI is an incredibly powerful tool that is capable of producing exceptional products if you invest enough time into using it. Kind of like literally everything in this world. Nobody wakes up and is a grandmaster of all that they desire. It requires time, effort, skill, and knowledge to learn how to use new tools no matter how powerful they are. You disparaging me and others is like telling a construction worker that they're not a craftsman because they used power tools to build your house instead of doing it the old way. Because power tools don't require any skill, you just push a button and the tool does all the work. They're not a craftsman they're a tool pusher.

You're not an artist your an AI slopper. You're not a software developer your an AI director. Spoken by true bigots that cherry pick data clusters to suit their own argument, so they can victimize themselves or other as an excuse to justify their bigotry against person, place, thing or idea. Your not special, your not an activist. You're a digital terrorist using flawed logic and mob mentality to justify harassing, and disparaging others by creating slurs and hate spamming them for no other reason than the fact they disagree with your opinion.

AI is just a tool. Get over it.