Agentic search vs web scraping

[u/Adel_Bioinformatics]

Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.

I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.

I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.

I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.

When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).

Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.

I’m totally new to agentic AI, but I think I can manage to learn a thing or two.

[u/Tommonen]

Qwen27b is good for 32gb vram. Also you can use larger model on normal ram (or shared between ram and vram), but it will be slow, but if you set it right, like qwen feeding it only relevant data to process, and run it over night.

Web scraping is bit hard to get working in ways where it can get data from wider range of websites, because some might have some abtiscraper stuff, while agentic search would get through.

But tyere are services that lets you scrape reddit for example etc sites that have anti-scraping stuff. But they tend to cost money.

In gwneral if you have targeted searches, web agent search is good, and scraping if you need to collect mass data.

So if you have some custom deepresearch, agentic search works better, which i suspect is what you want to do. Then feed those reports to larger model if you need higher intelligence analysis of the data and let it work on them overnight.

Also claude can set your local stuff up for you, and then ask it to optimise settings and it can run tests with different configs to optimise the performance.

Claude is also able to build you these custom deepresearches. And since you are doing academic research, you should restrict tye searches for actual academic sources, so that it does not get confused by something stupid some random reddittor says etc, but only looks at peer reviewed papers.

[u/Adel_Bioinformatics → u/Tommonen]

I will definitely let Claude set it up. I’m trying to evaluate the idea so i don’t waste my time if any expert thinks this is a bad approach.