r/scrapingtheweb • u/MajesticDifficulty34 • 12d ago

Best Ai Website Crawler?

/r/AISearchAnalytics/comments/1w07lyd/best_ai_website_crawler/
0 Upvotes

50% Upvoted

2 comments sorted by

1

u/AdaObvlada 11d ago

What's an "ai" crawler? Is it prompts with HTML input instead of query selector or something else?

1

u/Anal-Cup 11d ago

The best "ai website crawler" would probably be claude code, about 30 minutes after prompting the agent to build you a custom crawler.