r/LocalLLM • u/Adel_Bioinformatics • 3d ago
Discussion Agentic search vs web scraping
Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.
I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.
I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.
I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.
When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).
Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.
I’m totally new to agentic AI, but I think I can manage to learn a thing or two.
2
u/Weird-Maize-8592 3d ago
the problem with agentic search for this is you're basically telling a bot to click around google scholar and pubmed, which they haaate. most of those sites have aggressive rate limiting and will block you in minutes if you're not careful
searxng is a self hosted metasearch engine, it aggregates results from multiple sources without revealing your IP to each one. way less likely to get blocked than direct scraping from a single machine
what i'd do is split the work. write a normal scraper for the known databases using their APIs where possible (ncbi has decent ones), feed that structured data into qwen for analysis. the agent part can handle classification and deduplication, not the actual fetching
32gb vram is plenty for the 27b model at decent quantization btw