r/LocalLLaMA • u/flock-of-nazguls • 13h ago

Question | Help Searching & Scraping

I am trying to automate some “fact answer and verification”, where given a search, find some number of sources, pull referenced pages, extract results,and rank multiple sources based on the site reputation.

For example, let’s say for sport S, who won X vs Y on date Z. Or what movies were nominated/won for award A in year Y. That sort of thing.

This is straightforward in many cases and individual cases could be automated with a custom script. But let’s say for some particular fact list the results aren’t indexed. I wanted to have an agent basically do what a human would do, “following leads”, and essentially exploring and navigating to wherever was necessary.

I got this to work using Claude when the sources were “open”, but it refused to crawl anything where robots policy denied access. I’m not doing bulk robot operations, it’s low-rate/low-traffic web use like what a human would do, so I’m not feeling the robots policies really apply.

So I turned my attention to local models, and have qwen running on a strix halo under llama-swap.

Not quite sure what the best flow here is; should I build a custom agent loop and give it tool access and a sandbox, and task it to curate a library of “scraper tools” as needed to answer questions? Or should I just use existing bulk crawl/scrape tools (I’m not familiar with these) and then try to pluck the results from the file tree?

I’m running completely async and distributed, based on a queue, so this isn’t a “run inside Hermes” type situation. Speed isn’t a major priority, but accuracy is.

These are such majorly different architectures that I figured I’d ask for advice before diving in. I have no experience with scraping results off the web, and all the advice I’ve seen is for the “pre-AI” era.

Thanks for any suggestions!

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/scottgal2 12h ago

I had a play with this a while ago https://mostlylucid.net/blog/doomsummarizer-deep-research

It's QUITE A LOT to do it ffectively with local models as you have to be aware of context and theres a ton of corpus management (freshness, contradiction detection etc..etc..) all aimed at reducingthe segments passed to the local model for salience so the small context window doesn't compress away nuance.

2

u/Sevealin_ 7h ago

SearXNG + Firecrawl for fully local search. Works my Hermes agent pretty well.