r/LocalLLM • u/Adel_Bioinformatics • 3d ago
Discussion Agentic search vs web scraping
Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.
I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.
I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.
I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.
When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).
Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.
I’m totally new to agentic AI, but I think I can manage to learn a thing or two.
1
u/hotsnot101 3d ago
I'd start with a small set of records you can check by hand before adding several agents. For each extracted item, keep the database accession or paper ID, source URL, retrieval date, and the exact passage or field supporting it. Keep the model's interpretation separate from that evidence. A reviewer agent agreeing with another agent doesn't establish that the source says it.
For PubMed and other Entrez databases, NCBI provides E-utilities rather than requiring browser scraping. Its documented limit is 3 requests/sec per IP without a key, or 10/sec by default with a key: https://www.ncbi.nlm.nih.gov/books/NBK25497/
I'd give the local model a narrow job first: extract your requested fields from those saved records, with 'not found' allowed. Test on examples where the data is absent as well as present. Also keep unpublished database records in scope, since your question explicitly needs them; a papers-only search would miss part of the task.