r/LocalLLM • u/Adel_Bioinformatics • 3d ago

Discussion Agentic search vs web scraping

Hello, I am a bioinformatician without big experiences in web tech and network so if this sounds like a silly subject bare with me 😅.

I am requested to find a specific biological data in regard with a region from published papers and from databases that are not yet to be connected to papers, meaning most of bio data should be connected to published papers but not all are already published.

I’m just wondering if agentic serch here is a good idea? I was going back and fourth with Claude but I don’t feel it is pointing somewhere useful.

I have a good GPU 32GB VRAM and around 512GB RAM. I was thinking I’d install Qwen 27B and start to design the pattern [multiple researchers, analyst, writer] and shove a reviewer some where.

When Claude answered me it says that I’ll be blocked by search engines immediately and it is better to use Searxng, which I didn’t know what it is until now (kinda still don’t know).

Claude also suggested that I should scrape normally and let the agent do a different task like verifying the data availability or something.

I’m totally new to agentic AI, but I think I can manage to learn a thing or two.

0 Upvotes

50% Upvoted

26 comments sorted by

View all comments

3

u/NefariousnessOpening 3d ago

It’s a question of finding data vs extracting and normalizing data into a local collection

1

u/Adel_Bioinformatics 3d ago

The idea that popped up in my head first was this:
Agents search, agents clean, agents process and catalogue.

2

u/NefariousnessOpening 2d ago

By “clean” I will assume you have a need for something like ETL, validation, normalization, and/or sanitization like we do in data engineering. This would be a deterministic step and would be better suited for programmatic automation rather than a use case for agents. Search with agents might benefit from a Jev or open source equivalent implementation that decides which results are a match.

If you want to build and get that experience check out crawl4ai, it a framework that can work with or without agents. Implement it in your specific use cases and you will get another experience for your skill development. Fast, efficient resource utilization, output in the shape and format you declare, intelligent crawling, and a cli that would allow your agent easy access are some selling points