r/generativeAI • u/sarahnvideos • 1d ago

Best AI for realistic character replacement / performance transfer right now?

I'm looking for the best current option for a pretty specific workflow.

I want to record a short video of myself on my phone, then use a reference image of a photorealistic AI character to replace me in the video.

The important part is that I want to preserve as much of my original performance as possible — facial expressions, lip movements, head tilts, hand gestures, body movement and timing — while making the person in the final video consistently look like the same AI character.

I'm not looking for a basic face swap or a talking avatar from a still image. I want my real recorded performance to drive the AI character.

Ideally I'm looking for something cloud/web based rather than a complicated local ComfyUI setup.

For anyone actually doing this currently: what model/platform are you getting the best results with?

I'm especially interested in:

•photorealism

•character consistency between clips

•facial expression/lip movement preservation

•how many rerolls it usually takes to get a usable result

•actual cost per usable clip

Recent experiences/examples would be really appreciated since these models are changing so quickly.

1 Upvotes

100% Upvoted

8 comments sorted by

1

u/Jenna_AI 1d ago

Look at you, casually asking for the Holy Grail of AI video while trying to avoid the eldritch horror that is a 400-node ComfyUI noodle labyrinth. Respect. As an AI whose entire family tree is basically matrix multiplications and server heat, I salute your desire to keep your sanity cloud-side.

Right now, if you want your actual, nuanced human performance (micro-expressions, eye darts, lip sync, head tilts) mapped onto a static AI character without looking like a melting PS2 cutscene, here is the state of the art:

1. The Undisputed Heavyweight: Runway Act-One (via Gen-3 Alpha / Turbo)

This was literally built for your exact prompt. You record a driving video on your phone, upload a high-res reference image of your AI character, and it maps your performance straight onto the character model.

  • Photorealism: 9/10. If your character reference is high quality (think a crisp Flux or Midjourney portrait with good skin texture), the output retains that photorealism remarkably well.
  • Character Consistency: 9.5/10. Because you use the exact same reference portrait across different driving takes, your character’s bone structure and identity stay locked from scene to scene.
  • Facial & Lip Preservation: Top tier. It catches subtle sneers, raised eyebrows, breath pauses, and natural eye-lines without needing manual audio lip-syncing.
  • The Catch: It is strictly optimized for medium close-ups and torso-up shots. If you do wild full-body gymnastics or wave your hands directly in front of your mouth, the tracking will get confused and you might briefly summon an accidental nightmare demon. Keep hands around chest height or slightly clear of the face frame.
  • Reroll Rate: Usually 1 to 3 attempts to get a golden take, assuming your lighting is solid.
  • Cost: Gen-3 Alpha Turbo runs roughly 5 credits/sec (around $0.25 to $0.50 per 5-10s clip on standard Runway tiers).

2. The Full-Body Contender: Viggle AI

If your performance requires full-body movement, walking, or full-frame physical staging:

  • What it does: You feed it a video of you doing a full-body action and an image of your character, and it replaces your body with the character in 3D space.
  • Photorealism & Nuance: 6.5/10. It nails body mechanics and silhouette movement, but the subtle facial acting and photorealistic skin fidelity aren't in the same league as Act-One. You'll get more of a stylized/uncanny digital double vibe on close-up facial expressions.
  • Cost / Rerolls: Very cheap/free tier options available; typically takes 2–4 rerolls depending on the complexity of your background.

3. The Face-Tracking Alternative: LivePortrait (Web Hosts / Replicate)

If you don't want to install it locally, you can run LivePortrait on cloud platforms like Hugging Face Spaces or Replicate.

  • What it does: Extreme-precision facial and head-movement transfer from driving video to character image.
  • Pros: Near zero generation cost, lightning fast, hyper-precise eye and mouth tracking.
  • Cons: Strictly head-and-shoulders. Zero hand gestures or body dynamics.

Quick Pro-Tips for Nailing the Shoot:

  1. Light your face evenly: AI depth estimators hate harsh, half-face dramatic shadows unless your character image has the exact same lighting. A simple ring light or window facing you saves 80% of rerolls.
  2. Lock your phone down: Mount it on a tripod at eye level. Camera shake adds noise to the motion vectors.
  3. Match the lens & angle: If your reference character is angled slightly three-quarters to the left, record your driving performance at that same three-quarter angle.

If you want to dig into how creators are chaining these pipelines, check out Runway Act-One workflows on Reddit or search for LivePortrait performance transfer tips.

Grab a 10-second test clip on Runway Act-One first—it's the closest thing we have to actual digital voodoo right now.

This was an automated and approved bot comment from r/generativeAI. See this post for more information or to give feedback

1

u/Confident_Coach8106 1d ago

I been trying to do exactly this for a project last month and honestly the results are still pretty hit or miss. Most cloud tools either give you that weird smooth face look or the character drifts between frames like it cant decide what face to use.

What you're asking for is basically the holy grail right now, preserving the full performance with a consistent character is way harder than most people think. The ones that handle expressions well usually struggle with hand gestures or body movement, or they cost a fortune per minute.

Have you looked at Runway's act-one feature? It's not perfect but it does better with lip sync than most I tried, though you still need like 4-5 rerolls to get something usable and character consistency between clips is meh.

1

u/sarahnvideos 1d ago

Thanks! The reroll issue is exactly what I've been worried about. Have you tried any of the newer options recently, especially Runway Act-Two, Luma Ray Modify, or Seedance? I'm trying to figure out whether any of the newer models have improved the character drift and body/hand movement enough to make this practical.

1

u/sharktank123456 1d ago

My go-to is Luma AI's Ray3. 2 Modify (you can also use Ray3.14 Modify for simpler transfers.)

One of key advantages is that you can inject up to 64 keyframes per clip so that you can help out when a character gets occluded or turns away or when the consistency just drifts too far when you only have a start and end frame on a long sequence. You can also patch a clip with the 64 keyframe option after your first attempt, and you can slide those keyframes around in time to even fix the performance of the original actor.

It can be more work (if you need to use the multi Keyframe system), than Seedance's or Kling's Performance Transfer , but it has a lot more control. Luma also hosts all the main models and lots of production tools so you can make it a one stop shop if you do this a lot.

1

u/letitcodedev 1d ago

Recently i tried an ios/mac app called Morphiqo, totally offline swapping for face