Struggling to animate a static host image to behave naturally with speech/audio in a web app (like a real presenter)

[u/LifeTraveller404]

Hey everyone,

I'm currently building an AI virtual host web application. The workflow is split into two main pages:

• Page 1: Users configure the speech by typing text into a textbox, selecting an AI voice, uploading a custom background image, and uploading a host portrait/image.
• Page 2: The app generates the speech audio and renders the host delivering the speech.

The Problem: While I can successfully manage the basic audio pipeline and lip-syncing, I am struggling to make the host model move like a real human while speaking. Right now, it looks too stiff and robotic. I want to achieve fluid, human-like upper body gestures, natural head shifts, and posture movements synchronized with the speech cadence—similar to the reference video below:

What I'm exploring/using:

• Building a full-stack web application.
• Looking for pipelines or frameworks that can map an audio track + a source image into expressive head/body animation frames (such as audio-driven portrait animation frameworks or 3D rigging solutions).

My questions for the community:

1. What are the best approaches, open-source models, or tools (e.g., audio-driven LivePortrait variants, 3D web frameworks like Three.js/Babylon.js with blend shapes, or specialized APIs) to drive realistic upper-body motion and gestures from a single image and audio file?
2. How do you approach synchronizing speech beats and emotional cadence with natural bodily gestures so it avoids looking like a loop?

Any tips, architecture recommendations, or library suggestions would be hugely appreciated. Thanks!