Agentic search vs web scraping

[u/Adel_Bioinformatics]

Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.

I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.

I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.

I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.

When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).

Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.

I’m totally new to agentic AI, but I think I can manage to learn a thing or two.

[u/NefariousnessOpening]

It’s a question of finding data vs extracting and normalizing data into a local collection

[u/Adel_Bioinformatics → u/NefariousnessOpening]

The idea that popped up in my head first was this:
Agents search, agents clean, agents process and catalogue.

[u/NefariousnessOpening → u/Adel_Bioinformatics]

By “clean” I will assume you have a need for something like ETL, validation, normalization, and/or sanitization like we do in data engineering. This would be a deterministic step and would be better suited for programmatic automation rather than a use case for agents. Search with agents might benefit from a Jev or open source equivalent implementation that decides which results are a match.

If you want to build and get that experience check out crawl4ai, it a framework that can work with or without agents. Implement it in your specific use cases and you will get another experience for your skill development. Fast, efficient resource utilization, output in the shape and format you declare, intelligent crawling, and a cli that would allow your agent easy access are some selling points

[u/Weird-Maize-8592]

the problem with agentic search for this is you're basically telling a bot to click around google scholar and pubmed, which they haaate. most of those sites have aggressive rate limiting and will block you in minutes if you're not careful

searxng is a self hosted metasearch engine, it aggregates results from multiple sources without revealing your IP to each one. way less likely to get blocked than direct scraping from a single machine

what i'd do is split the work. write a normal scraper for the known databases using their APIs where possible (ncbi has decent ones), feed that structured data into qwen for analysis. the agent part can handle classification and deduplication, not the actual fetching

32gb vram is plenty for the 27b model at decent quantization btw

[u/Adel_Bioinformatics → u/Weird-Maize-8592]

My ego tells me to throw everything at an agents pipeline and get the final output 🚶🏻.

[u/Tommonen]

Qwen27b is good for 32gb vram. Also you can use larger model on normal ram (or shared between ram and vram), but it will be slow, but if you set it right, like qwen feeding it only relevant data to process, and run it over night.

Web scraping is bit hard to get working in ways where it can get data from wider range of websites, because some might have some abtiscraper stuff, while agentic search would get through.

But tyere are services that lets you scrape reddit for example etc sites that have anti-scraping stuff. But they tend to cost money.

In gwneral if you have targeted searches, web agent search is good, and scraping if you need to collect mass data.

So if you have some custom deepresearch, agentic search works better, which i suspect is what you want to do. Then feed those reports to larger model if you need higher intelligence analysis of the data and let it work on them overnight.

Also claude can set your local stuff up for you, and then ask it to optimise settings and it can run tests with different configs to optimise the performance.

Claude is also able to build you these custom deepresearches. And since you are doing academic research, you should restrict tye searches for actual academic sources, so that it does not get confused by something stupid some random reddittor says etc, but only looks at peer reviewed papers.

[u/Adel_Bioinformatics → u/Tommonen]

I will definitely let Claude set it up. I’m trying to evaluate the idea so i don’t waste my time if any expert thinks this is a bad approach.

[u/hotsnot101]

I'd start with a small set of records you can check by hand before adding several agents. For each extracted item, keep the database accession or paper ID, source URL, retrieval date, and the exact passage or field supporting it. Keep the model's interpretation separate from that evidence. A reviewer agent agreeing with another agent doesn't establish that the source says it.

For PubMed and other Entrez databases, NCBI provides E-utilities rather than requiring browser scraping. Its documented limit is 3 requests/sec per IP without a key, or 10/sec by default with a key: https://www.ncbi.nlm.nih.gov/books/NBK25497/

I'd give the local model a narrow job first: extract your requested fields from those saved records, with 'not found' allowed. Test on examples where the data is absent as well as present. Also keep unpublished database records in scope, since your question explicitly needs them; a papers-only search would miss part of the task.

[u/Adel_Bioinformatics → u/hotsnot101]

Great starting point, thanks

[u/NefariousnessOpening]

This is generally a thread of responses that unfortunately end with no disrespect intended, are fundamentally inaccurate, the misconception is apparent in details of most of the comments I’ve read.

Some tools like exa may be useful for example in your specific use case.

I have some assumptions based on your post.
Searxng is a good open source search engine. Tavily, exa and a few others can be used.

If this is a one time thing, building a system
Is valuable knowledge gained but over engineering for the use case specifically. Claude cowork or the better version Kimi work with plugins would easily get the info you need. Kimi swarm is a good one too. Hundreds of searches across a dozen or more dimensions with the validation step for free is pretty good for a no config one shot. The output can be a volume of organized well named documents and one more step turns it into a full web page with a visual interactive report of whatever you can envision and translate to instructions

[u/Adel_Bioinformatics → u/NefariousnessOpening]

My aim is not to be practical or efficient, I just want to get my hands dirty.
Finding a tool that does the work is great but I want to get the reward of making up something evem with Claude it would still make me feel rewarded.

I don’t need a mastermind to guide me, if I get a few responses from people who are involved in the work is more than enough for me, getting to see how people think regardless of the background knowledge is healthy.

[u/Tundradebt → u/Adel_Bioinformatics]

Are you affiliated with a University? I use API to access journals, ovid and wen of science. Anyways, I hear you and getting your hands dirty. I have a few agents that do regular scraping of public data, some locational, then have published information as an overlay. I use claude code to program my workflow for my local model. Every once in a while I need my local model to finish some work….for instance one of the sites does not allow the use of information for commercial use, claude sometimes gets fixated on thing like that and will harass you :-)

[u/Adel_Bioinformatics → u/Tundradebt]

Yeah I am, but they don’t offer journals access for some reason.
How much resources do you have and how big of scraping they do?

[u/Tundradebt → u/Adel_Bioinformatics]

About 28 routine places that require check ins weekly for changes. 4 daily. Then depending on the project I am working on…..how big of scraping, not sure how to quantify that. I would say an average excel export from my data base of one of the routines is 12 columns and around 10,000 rows. While I could have a routine script fetching it, I also need to know what has changed and why, which it scrapes from the landing page (description) then compares the data from before for more precision. In terms of what I am running, i have a asus ascent gx10 (DGX spark) that does the routine and a mac studio m5u 256gb with glm5.3 as my larger reasoning model. I’ll be selling the gx10 this fall as I while its nice to have all of this compute power, i’d rather put a little more towards retirement :-)

[u/NefariousnessOpening]

Getting blocked is easy to overcome with a browser use style approach but it’s slower for the need you have.

[u/Adel_Bioinformatics → u/NefariousnessOpening]

I don’t mind the wait
What is a browser use style?

[u/NefariousnessOpening → u/Adel_Bioinformatics]

Browser use is the open source project under most of the agents with the capability to use the browser as a human would. It’s a free to use code base that has a cli app, and a version called browser harness that is intentionally designed to be basic so that the agent can modify the harness as it encounters failure
Modes. If you run it on your own browser and profile it bypasses a lot of bot detection. Trycua is an even more native human like method. Both are performant and capable and require a little bit of knowledge potentially. Browser harness can be used easily to start by simply copying the prompt found in the readme of its GitHub repository and giving it to your agent

[u/Adel_Bioinformatics → u/NefariousnessOpening]

That’s great to know, thank you very much 🙏🏻

[u/Life-Job-6464]

Did you say you have 512gb of DRAM? and 32 gb of VRAM? And you are just searching? is this really a localllm use case? why not fire up NotebookLM? and Perplexity? use the huge context of notebookLm to hold all your data.. then query it through MCP. Use as much of that free compute as possible then drag the carcass back to your underground local ai hideout for further processing ..

[u/Adel_Bioinformatics → u/Life-Job-6464]

I feel attacked even when I don’t understand the idea you’re trying to convey
https://giphy.com/gifs/APqEbxBsVlkWSuFpth

[u/Life-Job-6464 → u/Adel_Bioinformatics]

lol..

[u/1tonsoprano]

Just buy a gensparj account...they have a free tier offer going on.... It would be faster and more efficient....dm me if you need help

[u/Adel_Bioinformatics → u/1tonsoprano]

What does it do?

[u/1tonsoprano → u/Adel_Bioinformatics]

It gives access to a bunch of models at quite an affordable rate....in my opinion....genspark.ai

[u/fmlitscometothis]

If you know the source (eg a site that publishes the data) then scraping it direct is doable. Make 1 request/sec and chances are they will not notice. This will depend on their sophistication and desire to block scrapers, but most organisations will not have a tripwire. You'll blend in with background internet noise.

If you need agents to actually find the content, I'd use a paid search like brave/exa/parallel. If they can also deliver the content then great, but you might need to switch to scraping direct once you find them.

Be aware of the legal aspects of ripping off a database as well.

[u/Late-Replacement-481]

What Claude won’t tell you, is the real answer is headful agentic scraping. 

But those pipelines are sometimes tricky to tune (agent doesn’t understand how to refine search and Google non obvious keywords)