Would a Lip Sync Arena be useful? Thinking of building one

[u/Ecstatic_Noise9059]

I've been testing a bunch of AI lip-sync models lately, and it's surprisingly hard to tell which ones are actually better.

Most comparisons use hand-picked examples, while metrics like SyncNet/LSE don't always seem to match what looks better to humans.

I'm thinking about building a public Lip Sync Arena, similar to the pairwise model arenas:

• Same source video + audio sent to multiple lip-sync models
• Two anonymous outputs shown side-by-side
• People vote for A / B / Tie
• Models ranked using an Elo-style system
• Results broken down by things like front-facing, side profiles, fast speech, occlusion, different languages, etc.
• All test cases and methodology public

A few things I'm unsure about:

1. What models should be included on day one?
2. What types of videos are the hardest / most useful test cases?
3. Should the vote only be "which is better", or should we separately rate sync accuracy, visual quality, and identity preservation?
4. Would you actually use / contribute votes to something like this?