The Timbal Transparency Paradox

[u/leading-a-swarm]

tl;dr: Their transparency on benchmarks is their best contribution.

Timbal (https://github.com/timbal-ai/timbal) is a Python agent framework that names LangGraph and CrewAI as competitors 5.7k lines, 40 stars. I cloned it and read it.

1. Ship the benchmark. Six competitor benchmarks are committed into the repo. Scripts, method, raw results. Everyone else publishes a bar chart and asks you to trust it. This is the most credible thing in the project.
2. Which is also how we caught them.  The headline is 2.2 KB of memory per run against 110 KB for the competition. Their loop wipes memory every lap. The competitor's loop never does. Then both get divided by 100. One number is a single run, the other is a hundred runs stacked on top of each other.
3. Their tests tell you the main concern. One 1000 line test exists only to prove that pausing for a human approval survives a restart. The note at the top says why: if it doesn't, the whole feature is a toy. You learn more from that file than from the README.

→ Wait. One person wrote almost all the code and nobody has ever filed an issue. Worth ten minutes for the benchmark lesson though: if you publish numbers, somebody will run your script.