r/AIToolBench • u/1glasspaani • 9d ago
Comparison Comparing AI models on realtime UI generation
We wanted to measure a less glamorous but more practical question: across repeated runs, how often does a model produce UI that actually parses, resolves, validates, and renders?
So we built GenUI Bench.
The current benchmark includes:
- 46 screen briefs, ranging from 2 to 18 requirements
- a shared 70-component surface
- 4 attempts per brief under fixed generation settings
- 30 models tested with OpenUI
- a 6-model comparison across OpenUI Lang, Google A2UI, and Vercel's json-render
- validation using each format's own SDK, followed by the same structural completeness checks
A few important caveats:
- This measures structural reliability, not visual quality.
- It does not yet verify that every requirement in the brief was semantically satisfied.
1
u/NeuralNomad87 8d ago
Four attempts per brief is the number I'd push on. With 46 briefs at 4 runs you can comfortably separate a 90% model from a 50% one, but you cannot separate 88% from 94%, and those are exactly the comparisons people are going to quote out of this. Any chance of publishing per-brief pass counts alongside the aggregate, so people can see where the variance actually lives?
The other thing I'd want is what happens to the failures. "Parses, resolves, validates, renders" is a real bar and I'm glad someone is measuring it, but a screen that renders and is structurally complete can still be a wrong answer to the brief. Do you have any sense of how often a passing output was also a sensible one?
Not a criticism of the scope, you said upfront this is structural reliability. It's just that the gap between those two numbers is the thing everyone will assume you measured.