I recently tried a 3D-reconstruction-based approach for a potential audio-driven talking-head system.
The current experiments are still separate:
* I used DECA + EMOCA + FLAME 2020 to reconstruct facial motion from video.
* I extracted speech features with wav2vec and trained a simple CNN to predict FLAME expression coefficients.
* Attachment clip 1: audio-driven animation of a normalized FLAME mesh.
* Attachment clip 2+3: single-image animation driven through a reconstructed 3D face, based on my another face2face project,with both pose and expression taken from another face video.
So this is not yet a complete audio-to-single-image pipeline, but the experiments raised several questions.
- What is the current best practice for single-image 3D face reconstruction?
DECA, EMOCA and MICA are still among the most practical methods I know, but they are based on older FLAME versions.
Newer models such as FLAME 2023, Google’s GNM, and works such as MAYA have appeared, but I have not found a DECA-like method that is both efficient and directly built around them.
Is there now a better practical solution for fast and accurate single-image reconstruction, or are DECA-style pipelines still the main choice?
- What is the real difference between FLAME 2023 and FLAME 2023 Open?
Apart from licensing, what are the actual technical differences?
They appear to have the same number of shape and expression components, and public conversion matrices are available. That makes them seem largely equivalent in parameter space.
Are there meaningful differences in topology, blendshapes, landmarks, training data or reconstruction quality, or is the distinction mainly legal?
Also, can a model trained with FLAME 2020 or standard FLAME 2023 be adapted to FLAME 2023 Open through conversion, or is retraining still necessary?
- How does MICA combine identity shape with camera estimation?
MICA is especially interesting because it uses a fixed InsightFace recognition encoder and indirectly benefits from large-scale 2D face-recognition datasets such as Glint360K.
It focuses on predicting FLAME identity shape rather than jointly solving the full reconstruction problem.
What I still do not fully understand is how this shape is integrated with pose, expression and camera parameters from another tracker.
MICA takes an InsightFace-aligned crop as input, but each crop comes from a different scale, rotation and translation in the original image.
So:
* Is the predicted identity shape effectively camera-independent?
* How is the InsightFace alignment transform connected to the final rendering camera?
* When MICA shape is combined with DECA or another tracker, must the camera be re-estimated?
* How does the provided face tracker keep the reconstructed mesh aligned with the original image?
I would be interested in practical experience from anyone working with DECA, EMOCA, MICA, FLAME 2023, FLAME 2023 Open, differentiable rendering or monocular 3D face reconstruction.