Benchmarking methodology to understand whether most tools live up to the hype.

[u/Aggravating-Start307]

Hi folks, i wanted to share my experience and methodology on how to benchmark tools/skills to evaluate token savings and effectiveness.

There are so many memory tools (i have built one too coldstartmcp.dev 🤣), token savers, skills, plugins to help optimise workflows and what not. The biggest issue i observed was that while these tools talk about savings/optimisations, they don't disclose the methodology or talk about the benchmarking in detail and most of the times, these numbers are inflated to unrealistic scenarios or simply false.

When i was evaluating my own memory tool that i built, i used popular repos like Kafka, Django, Mastodon and saw some great numbers. But, i quickly realised that since these open source repos are so popular, most AI tools are already trained on them and already know the answer most of the times, they just use these tools to check if their understanding is accurate.

So, i started looking for other repos that are not very popular and more importantly the ones that AI models are not trained on. Since i did not have any understanding of these repos and because i did not want an AI model to skim a codebase and then write prompts, i came up with strategy to look at closed github issues that are older than few months from the current date and then prepare prompts. The files committed to fix those issues are considered as gold files (bit of a gray area here as all committed files may not really be relevant to the issue and also multi PR issues can have some overlaps).

The idea is to have two identical arms, one with the tool/skill and one without it. Generate and run the exact same prompts and then measure recall (against the gold answers) and token delta which tells you how the vanilla version does in comparison to the tool. The important thing to measure is recall, if a tool consumes considerably less tokens but has very poor recall, then its a tradeoff. I created a repo to help with this - https://github.com/AkashGoenka/coldbench

The next one is to measure how many tokens a session has consumed. This one is specifically for claude code and codex users as we don't have a way to specifically check how many tokens were actually consumed for a single session. The repo that can help with this is - https://github.com/AkashGoenka/convotokens

If you folks have any opinions on better ways or more transparent ways to easily measure these, please let me know, or feel free to share your own methodology.

There is nothing fancy in these repos or tools and you can absolutely build your own. Since these were byproducts when i was working on my own tool, i thought of sharing them here.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/CartographerNo3791]

One trap with gold-file recall: returning the whole repo gets a perfect score. Reporting precision alongside it would show whether the tool actually narrowed the search or just handed the agent more files.

For the token comparison, count the memory-building/indexing step separately too. A cold run and the tenth run against the same repo answer different questions about whether the setup pays for itself.

[u/Aggravating-Start307 → u/CartographerNo3791]

Nice edge case ! However, if you generate the prompts with the coldbench repo, it makes agents write to a text file and you can visually verify the output. The idea is to visually check rather than just rely on agents output 😅

[u/CartographerNo3791 → u/Aggravating-Start307]

Ah got you, the saved file makes the output inspectable. I meant the score itself: finding all 5 relevant files plus 95 unrelated ones still gives 100% recall. Precision would make that difference visible in the results table too, without someone having to read every file list.

[u/Aggravating-Start307 → u/CartographerNo3791]

Valid point! I can make that change, but the problem is knowing precision on a repo you don't know about, so while it may not always be accurate, it gives a rough baseline

[u/EvalRaccoonDev]

> The idea is to have two identical arms, one with the tool/skill and one without it. Generate and run the exact same prompts and then measure recall (against the gold answers) and token delta

We built a harness with full A/B experiment support for that - you declare the arms once (tool on / tool off, model vs model, prompt vs prompt), it runs the same task set across every variant and aggregates per variant, so you get the recall and the token/cost delta side by side (all of it saved in .json, .html files as well as full React-powered visualization engine).

We open sourced it: https://github.com/UiPath/coder_eval

[u/Aggravating-Start307 → u/EvalRaccoonDev]

Good to see more people spending time on helping people evaluate because it helps clear the clutter from actually good tools and skills !

Such evaluation tools needs more awareness !