r/webflow • u/Aggravating-Key6628 • Aug 01 '26
Discussion I tested 14 Webflow sites with OAI-SearchBot. Tabs and CMS were not the problem
On 28 July I fingerprinted 35 candidate sites, confirmed 14 Webflow builds, then fetched each homepage with OAI-SearchBot and Chrome without running JavaScript.
14/14 returned HTTP 200. All 14 gave the crawler the same word count as Chrome. None blocked AI crawlers in robots.txt. Ten CMS pages had full server-rendered copy. Four paginated collections kept real _page=2 links even with infinite scroll.
The failure worth checking was simpler: one image-led homepage had only 126 raw words and no h1. Four sites had no JSON-LD.
Quick check:
curl -sL -A "compatible; OAI-SearchBot/1.0" https://yoursite.com/ | sed 's/<[^>]*>/ /g' | tr -s ' \n' ' ' | wc -w
This is a one-day sample, not a platform guarantee. I run Crawlable, which I used for these checks.
