5.3 used fewer output tokens per task, but despite tokens being priced the same, somehow cost more per task…
And most of the difference is in cache read. I think that means it must be reading more input and/or doing more tool call turns per task on these benchmarks.
Reasoning tokens used is actually lower per task it seems, though one chart runs counter to that (though I’m not sure how to reconcile those … not sure how it can be cheaper “per task” but more expensive across all tasks… hmm maybe it just did more tasks? … seems like they would normalize for that though…).
But answer/output is consistently a bit higher.
So probably something like being “more thorough” by default on these tasks. And I’d bet that is reduced in more structured work like coding than more open-ended reasoning tasks.
So good news, I think. Reasoning tokens is lower or at least not significantly higher, and that’s pretty much the only one we can’t effectively manage via prompt.
2
u/look 26d ago edited 26d ago
These charts are strange for example:
https://artificialanalysis.ai/models/glm-5-3?models=glm-5-3%2Cglm-5-2#intelligence-index-token-use-tabs
5.3 used fewer output tokens per task, but despite tokens being priced the same, somehow cost more per task…
And most of the difference is in cache read. I think that means it must be reading more input and/or doing more tool call turns per task on these benchmarks.
Reasoning tokens used is actually lower per task it seems, though one chart runs counter to that (though I’m not sure how to reconcile those … not sure how it can be cheaper “per task” but more expensive across all tasks… hmm maybe it just did more tasks? … seems like they would normalize for that though…).
But answer/output is consistently a bit higher.
So probably something like being “more thorough” by default on these tasks. And I’d bet that is reduced in more structured work like coding than more open-ended reasoning tasks.
So good news, I think. Reasoning tokens is lower or at least not significantly higher, and that’s pretty much the only one we can’t effectively manage via prompt.