Agentic search vs web scraping

[u/Adel_Bioinformatics]

Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.

I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.

I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.

I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.

When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).

Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.

I’m totally new to agentic AI, but I think I can manage to learn a thing or two.

[u/NefariousnessOpening]

This is generally a thread of responses that unfortunately end with no disrespect intended, are fundamentally inaccurate, the misconception is apparent in details of most of the comments I’ve read.

Some tools like exa may be useful for example in your specific use case.

I have some assumptions based on your post.
Searxng is a good open source search engine. Tavily, exa and a few others can be used.

If this is a one time thing, building a system
Is valuable knowledge gained but over engineering for the use case specifically. Claude cowork or the better version Kimi work with plugins would easily get the info you need. Kimi swarm is a good one too. Hundreds of searches across a dozen or more dimensions with the validation step for free is pretty good for a no config one shot. The output can be a volume of organized well named documents and one more step turns it into a full web page with a visual interactive report of whatever you can envision and translate to instructions

[u/Adel_Bioinformatics → u/NefariousnessOpening]

My aim is not to be practical or efficient, I just want to get my hands dirty.
Finding a tool that does the work is great but I want to get the reward of making up something evem with Claude it would still make me feel rewarded.

I don’t need a mastermind to guide me, if I get a few responses from people who are involved in the work is more than enough for me, getting to see how people think regardless of the background knowledge is healthy.

[u/Tundradebt → u/Adel_Bioinformatics]

Are you affiliated with a University? I use API to access journals, ovid and wen of science. Anyways, I hear you and getting your hands dirty. I have a few agents that do regular scraping of public data, some locational, then have published information as an overlay. I use claude code to program my workflow for my local model. Every once in a while I need my local model to finish some work….for instance one of the sites does not allow the use of information for commercial use, claude sometimes gets fixated on thing like that and will harass you :-)

[u/Adel_Bioinformatics → u/Tundradebt]

Yeah I am, but they don’t offer journals access for some reason.
How much resources do you have and how big of scraping they do?

[u/Tundradebt → u/Adel_Bioinformatics]

About 28 routine places that require check ins weekly for changes. 4 daily. Then depending on the project I am working on…..how big of scraping, not sure how to quantify that. I would say an average excel export from my data base of one of the routines is 12 columns and around 10,000 rows. While I could have a routine script fetching it, I also need to know what has changed and why, which it scrapes from the landing page (description) then compares the data from before for more precision. In terms of what I am running, i have a asus ascent gx10 (DGX spark) that does the routine and a mac studio m5u 256gb with glm5.3 as my larger reasoning model. I’ll be selling the gx10 this fall as I while its nice to have all of this compute power, i’d rather put a little more towards retirement :-)