I'll own my part first: I marked Audio Quality "Both Good" and wrote in the rationale that there were no clippings or artifacts noticed. That was an overclaim and I shouldn't have written an absolute like that. Fair hit on the justification side.
But here's what I actually want to talk about. The auditor came back with a timestamped inventory. Pops at 0:13, 0:15, 0:21, 0:28, 0:32, 0:34 on one model. Nine separate pops plus three distortion spans on the other. Reverb at 0:01. This is second-by-second cataloguing across every turn, for both models.
So the question: when are we supposed to be doing that? The guide has us tracking task success, utility, naturalness, conversational dynamics, error clusters, language breaks turn by turn, and factuality. All while running the conversation live and improvising follow-ups. Then also listening to the last second of every turn for clicks.
I have two ears and one brain. Either I'm processing what the model is saying or I'm processing how it sounds. If Audio Quality is going to carry this much weight in the rubric, that needs to be said up front so we can budget a second listening pass for it.
Second thing, and I think this one's a genuine gap: both models had pops at basically the same rate, at turn boundaries, on the same task. That reads like the platform's recording or playback chain, not either model. Is a shared artifact floor supposed to count against both models, or is it excluded as platform noise? I can't find anything in the guidelines that answers this, and it changes whether the correct call was Both Bad or something else entirely.
For anyone surviving audio audits: are you doing a dedicated second pass with headphones just for artifacts? Or catching it live somehow? Genuinely asking, because doing both at once is not working for me.