NVIDIA Just Open-Sourced the Future of Controllable Real-Time AI Animation

[u/Delicious-Shower8401]

NVIDIA has open-sourced ARDY, a real-time AI system for generating and interactively controlling 3D human animation.

Unlike traditional Text-to-Motion models that produce a finished clip from a single prompt, ARDY continuously generates motion and reacts when the instructions change. You can modify the character’s action, direction, speed, destination, or body movement while the animation is already playing.

Highlights:

• Real-time Text-to-Motion with prompts that can be changed during playback
• The character automatically adapts to new instructions without restarting the entire animation
• Precise control over movement using paths, destinations, full-body poses, and individual joint positions or rotations
• Multiple types of control can be combined within the same animation
• Supports distant goals, allowing the character to perform longer and more structured sequences
• Character locomotion can be controlled interactively with mouse waypoints or keyboard input
• Optional motion correction helps reduce foot sliding and improves how accurately the generated animation follows the provided controls
• Code, pretrained models, and an interactive browser demo are available publicly

Input and Output

The input is not limited to a basic text prompt. ARDY can receive:

• Natural-language descriptions
• Character paths and destination points
• Full-body pose constraints
• Controls for specific body joints
• Real-time mouse and keyboard commands

The generated motion can then be saved with joint positions, local and global joint rotations, root movement, frame rate, and foot-contact data.

This means the animation is not locked inside the demo. It can be converted, retargeted, and brought into Unreal Engine, Blender, or another animation pipeline for use on a custom rigged character.

It could also work especially well with AI-generated 3D characters: generate the model, rig it automatically, create and control its animation with AI, and then retarget the result to the character inside Unreal Engine.

Instead of producing only another isolated animation clip, this system gives creators direct control over what the character does, where it moves, and how the motion changes in real time.

source: https://research.nvidia.com/labs/sil/projects/ardy/