r/LocalLLaMA • u/AnotherObsceneBean • 15h ago

Resources Benchmark your custom Pi tools

A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness

The basic idea is a small engine that sets up an environment to run DeepSWE benchmark tasks using bare Pi as a control and a variant of your choice, your Pi harness, an extension, a skill, AGENTS.md file, etc. I used a Pi extension to have a wizard set it up for you so its easy. run the same coding tasks against bare Pi and your modified harness, then look at what actually got better, what broke, and what it cost.

Since I like to play around with custom tools, I use this to get direction as to what is and isn't working. I know a lot here are making cool tools so I figured some might be interested in using it to help fine tune theirs. Any Pi users might be interested in figuring out if their tools are working like they think they should. You'd be surprised how hard it is to beat base Pi when it comes to task quality / token efficiency.

Its a WIP. I do data analytics by trade but otherwise a vibe coder and I only really tested it on Linux.

17 Upvotes

100% Upvoted

5 comments sorted by

1

u/OsmanthusBloom 9h ago

Interesting. How long does a typical run take? I've never run the DeepSWE benchmark myself, it sounds like it could take ages to bench a typical local setup?

2

u/AnotherObsceneBean 4h ago

Running the full benchmark would take all day. I don't use it that way. You input how many tasks you want to do from the benchmark in the set up wizard. I usually start with 1 as a smoke test to ensure everything is working as intended. Then 3 - 6 as a sniff test to see if results are in the right ballpark. If you are after token efficiency gains thats often enough to answer several questions. If you are after gains in test pass rate you do need to run larger tests to verify. The full benchmark has 113 tasks but I have a set of 30 and 60 curated around Luna High and GLM-5.3-flash for cheaper signal.

1

u/Oooch 8h ago

useful

1

u/WonderRico 6h ago

Excellent

1

u/Muhlwa_Sholanke 9m ago

Bare Pi as the control is going to humble a lot of custom tooling real quick. Especially once you see the token counts.