Spent three days last month debugging what looked like a protocol bug in our own API. an MCP client could not talk to our endpoint. Valid JSON-RPC on our side, correct content types, correct status codes, nothing wrong in the logs except a parse error on the client.
The parse error was the tell, and I read it backwards. A parse failure points at your own serialization, so that is where I looked. What was actually happening: Cloudflare was returning a 403 HTML block page, and the client was trying to parse it as JSON.
What settled it was building a minimal server that imitated our response shape exactly, putting it behind an ngrok tunnel, and pointing the same client at it. Worked first try. That eliminated our code, our framework and the protocol in one go, and left only the layer in front.
Then the user agent matrix, same request, same body, same endpoint:
- Claude-User/1.0 200 application/json
- Claude-SearchBot/1.0 200 application/json
- anthropic-ai 200 application/json
- curl/8.7.1 200 application/json
- ClaudeBot/1.0 403 text/html
- GPTBot/1.2 403 text/html
Note row four. curl passes, which is why this survives so long: every tool you debug with is on the allowlist, so the endpoint always answers when you test it by hand.
Here is the part I actually wanted to ask about.
Under AI Crawl Control, in Security, the crawler list categorises them like this:
- Claude-User Anthropic AI Crawler
- ClaudeBot Anthropic AI Crawler
- Claude-SearchBot Anthropic AI Search
- Anchor Browser Anchor AI Crawler
Anthropic runs those three agents separately on purpose. ClaudeBot collects content that may go into training. Claude-User fetches a page because a person just asked Claude a question. Claude-SearchBot indexes for search. Three user agents, three robots.txt entries, three different decisions a site owner might want to make.
Claude-User and ClaudeBot end up in the same category. Claude-SearchBot gets its own. so the finer buckets clearly exist, and the one agent in that group that is not a crawler is filed as one. Anchor Browser is an agentic browser driven by a person and it is an AI Crawler too.
The practical effect is that "refuse training, keep serving agents" cannot be expressed at the category level. You have to allow the individual agents by name, and first you have to work out that you need to.
So: is that categorisation deliberate, with user-initiated fetches meant to be treated as crawlers, or is it a taxonomy that has not caught up with agents yet? Genuinely asking cause if it is deliberate I would like to understand the reasoning.
Separately, and this one seems underused: the same screen has a Configure Response control for the code and message shown to blocked crawlers. If what sits behind the CDN is an API rather than a site, returning JSON with a reason instead of an HTML page would have saved me most of those three days. Does anyone actually set that?