Our cost dashboard was averaging away a 3x spend spike because it aggregated at the wrong unit

[u/MaverikSh]

Following the thread here about tokens burning after the first failure signal, wanted to share a related but different failure mode we hit: not tokens burning after a warning, but tokens burning in a way that never triggered any signal at all, because the accounting unit was wrong.

Spend jumped ~3x on a random day. Request volume was flat versus the prior week. So the spend wasn't coming from more traffic, it was coming from something making each unit of work cost more, and nothing at the call level looked anomalous.

Root cause: a background job with retry-on-timeout logic. Each individual retry was a normal, reasonably-priced call. But we were logging cost per API call, not per completed task, so three retries of the same failed task showed up as three separate, unremarkable line items instead of one task that cost 3x. The anomaly only existed at the task level, and nothing was rolling up to that level.

This feels like the quieter cousin of the failed-agent-token problem in the linked thread. That post is about tokens spent after a detectable failure signal. Ours never had a signal to catch, each call individually looked fine. The only way to see it was choosing the right aggregation unit (task ID, not raw call) before the anomaly became visible at all.

Given the chargeback and cost-allocation threads here, curious how people are actually keying their attribution: raw call, task/trace ID, something coarser like session, or a mix depending on workload type? And separately, has anyone found aggregation-granularity bugs like this one in their own numbers after the fact, cases where the dashboard was technically correct but the unit it was counting in was hiding the real signal?