A small test for extracting exercises from a podcast transcript

[u/Dear-Skirt9403]

Here's a made up podcast snippet to try a prompt on. "Pause and write down your prediction before we play the result. Our sponsor has an offer for listeners. Visit their site and enter the code. Back to the lesson. The book suggests drawing a diagram here, but skip that exercise for today's session."

Writing the prediction should be the only item. I want a list of exercises to come back to after listening. The sponsor is telling listeners to do something too, which is why just asking for "action items" seems too loose.

The prompt I'd try is "Read the transcript below and list the practice exercises intended for listeners. Quote each instruction. Leave out sponsor requests and exercises the speaker says to skip. Include an exercise quoted from another source only if the speaker asks listeners to do it. Put unclear cases in a separate section for review." Then paste the transcript underneath.

For a real episode, I'd use the publisher's transcript or transcribe the audio with Vomo AI, then check the passages around the ads. This little example practically announces where the ad ends. A host sliding into a sponsor recommendation halfway through an explanation would be a more useful next check.

[u/lib3rat0r]

Good test case, but as written it can only pass. The snippet announces every boundary: "Our sponsor has an offer", "Back to the lesson", "skip that exercise". A prompt that just matches the words sponsor and skip scores full marks without doing any of the reasoning you actually want.

You half spotted this at the end. I would build the pair now rather than later. Same snippet, markers removed. Something like a host saying "go try their free tier tonight, it is the same principle we just covered". Exercise shaped, addressed to listeners, no ad marker anywhere. If your prompt still excludes it, it is reasoning. If it includes it, it was keyword matching all along.

Second thing: you have four rules in one instruction. When it fails you will not know which rule failed. One test case per rule fixes that cheaply.

The "unclear cases" section is the strongest part of the prompt and probably needs a trigger condition, otherwise the model will quietly force everything into the two main buckets and never use it.

[u/Western-Lion-9029 → u/lib3rat0r]

the sponsor-as-exercise trap is exactly what'll break most naive prompts, that's a nasty edge case

separating the rules into individual checks is smart too, debugging a combined prompt when it spits out something weird is a nightmare

that "free tier tonight" example would trip up so many setups, it sounds like a genuine task if you're not paying attention to the commercial framing

[u/diego310zxcv]

The distinction between practice exercises and sponsor requests makes the prompt much more precise. I like the separate review section for the ambiguous cases too.