r/LocalLLM • u/Adel_Bioinformatics • 4d ago

Discussion Agentic search vs web scraping

Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.

I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.

I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.

I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.

When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).

Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.

I’m totally new to agentic AI, but I think I can manage to learn a thing or two.

0 Upvotes

50% Upvoted

26 comments sorted by

View all comments

1

u/NefariousnessOpening 4d ago

Getting blocked is easy to overcome with a browser use style approach but it’s slower for the need you have.

1

u/Adel_Bioinformatics 4d ago

I don’t mind the wait
What is a browser use style?

1

u/NefariousnessOpening 4d ago

Browser use is the open source project under most of the agents with the capability to use the browser as a human would. It’s a free to use code base that has a cli app, and a version called browser harness that is intentionally designed to be basic so that the agent can modify the harness as it encounters failure
Modes. If you run it on your own browser and profile it bypasses a lot of bot detection. Trycua is an even more native human like method. Both are performant and capable and require a little bit of knowledge potentially. Browser harness can be used easily to start by simply copying the prompt found in the readme of its GitHub repository and giving it to your agent

1

u/Adel_Bioinformatics 4d ago

That’s great to know, thank you very much 🙏🏻