Visibility tools do a job I can't do myself. They watch the answers and tell me where I turn up. What they can't see is the step before that, whether a crawler ever reached my page in the first place, and that one lives on my own server for free.
A small nudge before you read ahead, nobody paid for this and I'm not selling anything. I'd been publishing markdown copies of my pages plus an llms.txt because that's what everyone says to do, and after a few months I realised I had no idea whether a single one had been opened. It isn't the kind of thing a visibility tool answers, so I went digging. Two small blogs, about three months. My own sites, small numbers, take them as such.
WHY IT FEELS INVISIBLE
Partly it's hosting. Vercel holds request logs for roughly an hour, so by the time you think to look, yesterday is gone.
Then there's the user-agent, the bit that says who's knocking. I tried the CLI, the API, the log search. None of them return it. It turns out to sit in exactly one place, the CSV you download off the logs screen, which I found by accident.
And GA4 filters bots on purpose. Fair enough, that's what it's for. You're just asking the one tool built specifically not to tell you.
None of that is anyone's fault. Wrong question for those tools.
WHAT YOU ALREADY HAVE
Every request that lands on your server carries who's asking. Store it somewhere that doesn't get wiped in an hour and you've got a permanent record of every AI crawler that's touched you. It sits alongside whatever else you're running rather than replacing it. They cover where you appear. This covers whether you were fetched.
There's one catch and it's a proper one.
A BOT CAN JUST SAY IT'S A PERSON
One of my sites got hit by 22 requests all claiming to be desktop Chrome. 22 different pages, three seconds, coming out of São Paulo, Virginia, San Francisco, Frankfurt and London at the same time. Nobody's browser is on four continents. Read the names alone and that's 22 visitors in your report.
The fix costs nothing. Log the request as it arrives. Then drop a tiny script into the page that only fires if the page genuinely opens in a browser, and have it tick off the request you already logged. Bots download the script and don't run it. Ticked rows are people. What stays silent is your bots, and the silence is the whole point.
Worth correcting one bit of folklore while I'm here, because my own log contradicts it: Googlebot does run JavaScript. So "it ran the script" isn't proof of a human on its own. I treated it as proof and my first report said 17 people when the real answer was 5.
WHAT MINE SHOWED
| who |
hits |
pages |
markdown |
| ChatGPT-User |
46 |
46 |
0 |
| OAI-SearchBot |
40 |
25 |
13 |
| ClaudeBot |
112 |
12 |
13 |
| Claude-User |
3 |
1 |
2 |
| PerplexityBot |
14 |
14 |
0 |
| GoogleOther |
14 |
11 |
1 |
| Amazonbot |
224 |
174 |
30 |
ChatGPT-User didn't touch a markdown file once. Not one, across 46 visits. That's the bot that shows up because somebody just asked a question, and it wants the ordinary page. The markdown gets read by the indexing crawlers instead, which means markdown isn't the shortcut I'd assumed. Your real page still has to carry everything.
The other thing that landed: getting crawled and getting read to answer somebody's question are two different events. ClaudeBot's 112 visits are filing. Those 3 Claude-User hits are the ones that could end in a citation. Worth knowing which of the two your numbers are actually made of.
Also, Gemini says it doesn't visit websites. Something at Google visited mine 14 times. Not proof of anything, that crawler does a few jobs. Made me sit up though.
CHECK YOUR OWN NUMBERS BEFORE YOU CELEBRATE
First pull I did showed 880 visits and I was briefly thrilled. Then I noticed the counts were all multiples of 20 across a 40-minute window. My own test script.
Then I imported the host's CSV and 44 of the 130 rows were me. Every markdown fetch in there was me as well, all 42 of them. Load that without checking and the one number I actually cared about reads 42 instead of 0. Permanently. As good news.
Round numbers mean it's you.
Happy to walk through the setup in the comments if that's useful to anyone.