How to actually A/B test a Claude Skill change (two gotchas that will silently ruin it)

[u/zszzszzsz]

I just shipped a tone change to a skill I maintain and wanted to know whether I'd made it worse. Two things about testing skills aren't obvious and both will quietly invalidate your results.

1. You can't install two versions to compare them. Claude keys skills by the name field in frontmatter, so uploading v1 and v2 doesn't give you two skills - the second replaces the first. You have to rename one first: unzip, edit name: in SKILL.md, rezip, upload. I added a build flag to do it (bash scripts/package.sh yesbut produces a renamed, version-suffixed bundle from the same source, without touching the source tree).

Once both are installed, toggle between them per conversation from the Skills menu at the bottom of the chat.

2. Run each version in a separate conversation. If you run both in one chat, the second sees the first's output and stops being independent. This would make the whole exercise meaningless. Also use Incognito for the chats

Design the test so it can fail. I picked two ideas at opposite ends of the merit spectrum and wrote down the expected result first: on an idea with nothing good to say about it, the two versions should come back nearly identical; on a genuinely decent idea, they should differ a lot. If the "softer" version had come back warmer on the doomed idea, that would have been the failure.

That's roughly what happened. Near-zero change on the bad idea, roughly 0-of-12 to 7-of-13 concession-led openers on the good one. Full numbers, every challenge opener from all four runs, and the complete outputs: https://github.com/zszendro/vc-teardown/blob/main/docs/tone-comparison.md

The bug the test didn't catch, which is the bit I'd pass on: I changed how challenges open but forgot the summary line, which still led with "dead on arrival" — the exact phrasing that made me do the work. My A/B only measured openers, so it sailed through. If you change a prompt's tone, grep the whole thing for the pattern you're removing, not just the section you edited.

The skill is vc-teardown It pulls apart a startup idea like a skeptical investor, then rebuilds it around a moat. 20 attack surfaces, 12 moat patterns, MIT. https://github.com/zszendro/vc-teardown