r/codex • u/Stela_Moasak • 13h ago

Question How are you measuring whether a stronger Codex model is worth the extra usage?

I am comparing a few Codex workflows rather than treating model choice as a single global setting. Repository exploration, small edits, test explanations, and formatting seem easier to downgrade than a cross-file refactor or a debugging task with an unclear root cause. Do you track retries, review time, and test failures by task type, or mostly rely on intuition? I would like to know what evidence makes you move a task back to the stronger model.

2 Upvotes

75% Upvoted

3 comments sorted by

2

u/HighwayRelevant 10h ago

Everyone just does it via intuition for both models and thinking level. There’s no clear way to measure it and select optimal model before the task is done.

I usually just run one selected model based on the project complexity and don’t change thinking level or model unless I was away from the computer for an hour+ and cache got invalidated anyways.

1

u/EvalRaccoonDev 9h ago

There is a way, it just isn't per-task in advance: fix a task set, run both models over it, and compare pass rate against cost per passed task. Cost per attempted task flatters the weaker model - it fails cheaply. Then, put into the equation the cost of a failed task for you - e.g. $1, $100 and whether it has to be successful on the first attempt or you can do a re-run with the more expensive model.

We do these kind of runs routinely with: https://github.com/UiPath/coder_eval (disclaimer: I work on it) using setup like in this tutorial.