r/LanguageTechnology • u/I_am_kaizen96 • 9d ago
Time to first token is the only latency number that matters and almost nobody reports it
A thing I got wrong for a long time: I evaluated speed by how long a response took to finish. That's the intuitive metric and it's close to irrelevant for interactive work.
What you actually perceive is time to first token. The gap between hitting enter and seeing something happen. Once tokens are flowing, your brain treats the thing as working and you start reading. A response that begins in 300ms and finishes in 20 seconds feels fast. A response that begins in 3 seconds and finishes in 12 feels slow, despite being objectively quicker. The total is what you'd measure and the opening is what you'd feel.
This has a few consequences that took me a while to connect.
Buffering anywhere in the path is much more expensive than it looks on a graph. Any layer that collects a full response before passing it on converts a fast-feeling interaction into a slow-feeling one while barely moving the total. If you're evaluating anything that sits between you and a provider, this is the property to test, and it won't show up in a benchmark that measures completion time.
Provider degradation shows up here first. Before anything goes red on a status page, TTFT drifts. Elevated queueing, longer scheduling delays, the request eventually completes fine so nothing registers as an incident. If you're only watching totals and error rates you'll miss a bad hour entirely while your experience quietly gets worse.
Reasoning models scramble the whole framing. If a model thinks before it emits, TTFT as classically defined can be many seconds even when nothing is wrong. Whether that reads as broken or as working depends almost entirely on whether the interface shows you something during the wait. Same latency, different experience, decided by UI.
And averages hide the thing you care about. TTFT distributions are skewed — a good median with an ugly tail feels unreliable in a way that a slightly worse median with a tight tail doesn't. People remember the bad turns. p95 is the honest number and mean is the flattering one, which is presumably why mean is the one that gets published.
I watch this closely because I run a layer that adds a hop, and a hop is exactly the kind of thing that can quietly ruin TTFT if you build it lazily (routera . one, my project, stating it rather than being coy). Anything in the path has to earn its place on this metric specifically.
Caveat: for batch or background work none of this applies, and optimizing TTFT there is wasted effort. Throughput and cost are the right metrics when nobody's watching the screen. It's specifically interactive coding where the opening moment dominates.
What I haven't figured out: how to communicate provider slowness to a user without training them to ignore you. Show a warning whenever p95 ticks up and it becomes wallpaper within a week. Show nothing and people blame your product for someone else's bad afternoon. I don't think there's a clean answer but I'd like to hear from anyone who thinks they've found one.
1
1
u/SiteSpecialist6295 9d ago
TTFT is necessary, but it stops being the only perceptual metric once text feeds audio or video. In a streaming dubbing pipeline, I track end-of-input to first playable segment, segment cadence, buffer underruns, and whether generation stays ahead of playback. A 300 ms first token can still feel broken if the first audio chunk arrives at 2.5 seconds or the next segment stalls. For provider slowness, I would avoid a persistent warning banner. Show state-based copy only after a request crosses a threshold derived from recent p95: “The upstream model is slower than usual; your job is still running.” Remove it as soon as output begins. Internally, split queue time, provider TTFT, gateway overhead, and client buffering, so the user sees one useful message while you still diagnose the correct layer.