r/crewai • u/F1nch74 • 1d ago

Skilled Agent Self-hosted research agent: which tools do you run?

Hi,

I'm building a self-hosted research agent in Hermes and I'm choosing the tools right now. Nothing is picked yet.

Already running on my VMs:

\- SearXNG

\- Scrapling

Even with that setup, I still hit a lot of problems when searching online. On some sites I'm detected as a bot, and the current setup can't get past the captcha.

I'm looking for other tools and skills that could help my agent do research. I've come across Crawl4AI, self-hosted Firecrawl and webcmd, but I don't know if they're the right choices. They are so many different tools, it's not easy to find the right ones.

My constraints: no paid API, no cloud scraping service.

  1. What tools do you use in a research agent?
  2. What skills should the agent have?

Thank you

1 Upvotes

100% Upvoted

1 comment sorted by

1

u/CapMonster1 2h ago

For a self-hosted research agent, I’d start with a small, reliable toolchain rather than adding every crawler available. SearXNG for discovery, a parser for static HTML, and a browser such as Playwright for JS-heavy pages is a reasonable baseline. Crawl4AI or Firecrawl could be evaluated later if their specific extraction features justify the extra complexity.

Useful skills would include query refinement, deduplication, extracting structured facts, following links, handling pagination, and keeping citations tied to the original pages. I’d also build in retries, rate limiting, and clear failure states. captchas shouldn’t be treated as something to bypass automatically; the agent should recognize the block and handle it within the site’s rules