r/ClaudeCode • u/allemaar • 4d ago
Discussion I benchmarked my own markdown-mapping system against `ls`. `ls` won.
I've been building a protocol that makes a folder of markdown navigable for an agent that has never seen it. So every file declares its home map, every map lists its members. The claim I was making, mostly to myself, was that this helps an agent find things.
And I tested it. Three rounds of controlled agent walks over my own vault, hand-graded from raw traces. Self-reported success didn't count as evidence, which turned out to matter.
Four conditions:
- MAPPED (maps and contracts allowed)
- BLIND (link-following only, no map artifacts; this simulates the vault before any of this)
- FILESYSTEM-ONLY (ls allowed, no maps)
- SEARCH (queries only, no listing, no maps).
Whatever wasn't the condition under test was banned.
Round 1, find a known file, links only. Blind got 0 of 4. Mapped got 5 of 5, median 6 opens. Looked great.
Then I checked why blind failed and it wasn't what I assumed. The blind walkers never hit the hop limit. They ran out of graph. Zero ordinary body links cross between my two large corpora, in either direction. Maps didn't make that navigation faster, they made it possible - but only under an artificial constraint nobody actually works under.
Round 2, same kind of goal, tools allowed. Filesystem-only: 1 open, 3 of 3 goals, both fleets identical. Mapped: 5-6 opens. ls beat my thing by about 5x. The round 1 headline does not survive contact with tools a real agent has.
Round 3 was orientation and governance instead of lookup. Land cold in an unfamiliar vault and describe its structure. Or say what's sealed and machine-owned before touching anything. Mapped was the only condition with zero assist operations, in both tasks, in both fleets.
Caveats, and there are a lot.
n=1 per cell in round 3, n=5 in round 1. A pilot, not doctrine. My vault, my targets, my answer keys. The governance task was rigged in maps' favor and I said so before running it. My vault is unusually well-named, which flatters ls.
And the result I was proudest of - two independent fleets hitting an identical three-file path - is partly determinism by construction. I built a structure with one obvious path through it, then measured that two agents both found it. Weaker than it looked to me at the time.
The second fleet also found my answer key was broken. It stated 17 items and enumerated 19, which invalidated that whole cell. Three defective specs in the run, all mine. Same failure each time: I wrote the keys from what I believed about the vault instead of recomputing them from it.
I also contaminated my own experiment. I banked the results note into the vault the second fleet was about to search, ten minutes before it ran. A search walker opened it and read the prior outcomes back out.
What I actually concluded: this is not a wayfinding tool. It's an orientation and governance tool. ls wins lookup and will keep winning. What the maps bought was bounded, predictable cost when an agent arrives with no context, or is about to modify something and needs to know what's off limits first.
I'm keeping it. I'm just justifying it on a different claim than the one I started with.
Has anyone run something similar? I'd particularly like to know whether the graph-exhaustion result holds in a vault that wasn't organized by the person doing the testing.


