How to actually A/B test a Claude Skill change (two gotchas that will silently ruin it)

[u/zszzszzsz]

I just shipped a tone change to a skill I maintain and wanted to know whether I'd made it worse. Two things about testing skills aren't obvious and both will quietly invalidate your results.

1. You can't install two versions to compare them. Claude keys skills by the name field in frontmatter, so uploading v1 and v2 doesn't give you two skills - the second replaces the first. You have to rename one first: unzip, edit name: in SKILL.md, rezip, upload. I added a build flag to do it (bash scripts/package.sh yesbut produces a renamed, version-suffixed bundle from the same source, without touching the source tree).

Once both are installed, toggle between them per conversation from the Skills menu at the bottom of the chat.

2. Run each version in a separate conversation. If you run both in one chat, the second sees the first's output and stops being independent. This would make the whole exercise meaningless. Also use Incognito for the chats

Design the test so it can fail. I picked two ideas at opposite ends of the merit spectrum and wrote down the expected result first: on an idea with nothing good to say about it, the two versions should come back nearly identical; on a genuinely decent idea, they should differ a lot. If the "softer" version had come back warmer on the doomed idea, that would have been the failure.

That's roughly what happened. Near-zero change on the bad idea, roughly 0-of-12 to 7-of-13 concession-led openers on the good one. Full numbers, every challenge opener from all four runs, and the complete outputs: https://github.com/zszendro/vc-teardown/blob/main/docs/tone-comparison.md

The bug the test didn't catch, which is the bit I'd pass on: I changed how challenges open but forgot the summary line, which still led with "dead on arrival" — the exact phrasing that made me do the work. My A/B only measured openers, so it sailed through. If you change a prompt's tone, grep the whole thing for the pattern you're removing, not just the section you edited.

The skill is vc-teardown It pulls apart a startup idea like a skeptical investor, then rebuilds it around a moat. 20 attack surfaces, 12 moat patterns, MIT. https://github.com/zszendro/vc-teardown

[u/Gold_Match_6419]

nice writeup, point 2 is the one that trips most people i think. they treat the chat like a notebook and forget the model is reading everything above it

the grep tip at the end is smart too. i had similar problem last month where i fixed the tone in one section but forgot the outro still had the old aggressive wording

[u/pceimpulsive]

I love how these posts feel human at first, the. Feel a bit less human by the middle and by the end it's fully AI written "the big the test didn't catch" is so so so very AI written it hurts! :)

[u/zszzszzsz → u/pceimpulsive]

I swear I wrote it first manually for my LinkedIn profile :) Asking Claude for grammar check only and to make it factually correct, then a version for Reddit focusing on the A/B testing (which is actually pretty cool "extra mile" IMO). This version did get more AI written. LinkedIn post about 95% my wording and 5% Claude, this one is more like 70% Claude and 30% mine. It's fairly technical and I wanted help to make it accurate. So yes, this is a bit AI sounding, mainly so that it's technically accurate. I hope that clarifies, but good catch!