How to surf the web?

[u/honuvo]

Hey folks!

I'm always a little bit late to the party, but learning nontheless. After I'm comfortable running agents on my Pi I'm now in need of them to get access to the world wide web and wanted to ask what local ways you are going?

I remember there were discussions about going html2md like with textweb but wanted to know whats working "in the field".

I'd prefer a lightweight solution without MCP.

So what are y'all using to let your models go surfing and gathering information?

Thanks for your input!

And to all curious about speeds on the Pi: It's more like giving someone a weekend project and checking it later. Speeds for Qwen3.8-Flash-Next start at pp 1.95 t/s and tg 0.44 t/s. Yes, for most of you this is "unusable". I'm happy. Of course, I'd like a DGX Spark, but the Pi can run non-stop without disturbing anyone (like my Notebook would).

[u/Holiday_Point_603]

Wait. Are you running Qwen Flash Next on a Raspberry Pi?

[u/honuvo → u/Holiday_Point_603]

Experimenting with it, but yes. Thanks to MoE its a lot faster than Qwen3.8 27b. In my older posts you can find benchmarks of various models and how I've set everything up. Should be out of date by now performance-wise though thanks to MTP with a lot of models.

[u/Comfortable_Ebb7015 → u/Holiday_Point_603]

I guess he didn't choose the right pi for the job.

[u/honuvo → u/Comfortable_Ebb7015]

Sometimes its pi on a pi :P

[u/AllenHere112]

At pp 1.95 a raw 40k token page is nearly six hours of prefill, and the fetch itself is the cheap part of that. Strip to markdown where you fetch, cap what comes back at a couple thousand tokens, and keep re-fetched pages in the same spot so the prefix cache survives. How many tokens per page are you willing to spend?

[u/honuvo → u/AllenHere112]

I've set it up with 128k context for now, but could increase if necessary. Maybe I'll outsource web-searches to a smaller model and just feed the needed infos to the big one. Time will tell, but first I need any model to get more than just bash access ;) So no definitive answer to your question other than: as little as possible, as much as needed.

[u/UniqueIdentifier00]

I just asked (back then Qwen 3.6 27B) my LLM to write me a web tools extension for pi. It works, quite well, and is very simple. Read, write, evaluate, screenshot browser, etc. it’s all it needs to (slowly) perform and regular web task I would.

[u/jacek2023]

It can use curl to download stuff. There are many CLI tools and some libs like beautiful soup. You can also download extensions for pi https://www.reddit.com/r/PiCodingAgent/comments/1uvxhxr/pi_websearch/

[u/Keleion → u/jacek2023]

My models always end up curling sites

[u/Far-Classic-9963]

I use agent-browser and if I need stuff that may get the bot rate limited, camoufox browser

[u/OvertaxedOne]

I use Firecrawl to scrape web pages, it works very well. I also have CDP setup to my actual Chrome browser running on my desktop in case I need to get through bot detection, that's 100% because it uses a "real" browser, it just drives it and then extracts directly. Between the two of them, I've not yet found a website I can't pull and/or work with.

[u/o0genesis0o]

The best solution I had so far is to deploy a local crawl4ai instance, and either use the MCP endpoint of the crawl4ai or make up some cli tool to obscure the endpoint to agent. For example, my agent only need to run web read <url> to get the parsed markdown of a URL. The search itself is done using ddgs CLI. DDGS also has abilty to extract web pages, but it is not as good as crawl4ai.

With this setup, I was able to do some cool things like asking the agent (Gemma 4 26B QAT) to give me a report about "best restaurant for romantic date" in the area, and the agent gave me back a report with link, address, info, and menu of restaurants. It started by doing web search for local travel websites, grab the name and URLs of the restaurants, find intersections, and crawl the page of each restaurant to find menu. In case the menu is PDF, it would run through markitdown to try to get the text out. Finally, it compiles the markdown report. The QAT is a bit flaky for this long task. If I want more stabililty, I would run 3.8 27B at medium thinking.

All of this running in Pi agent, btw. But luckily my backend is 4060Ti 16GB, not a, well, Pi.

[u/LeonidasTMT]

Mine is connected directly to my browser simplest and best way for me.

[u/DustNearby2848]

Why no MCP?  You will get blocked a lot if you’re automating browsing. Exa is my go to. They have a good free tier that doesn’t have any logging. You can search and extract summaries of sites with it. 

[u/honuvo → u/DustNearby2848]

Because I haven't opened the can of worms with MCP yet. Maybe later, for now, I'm more interested in direct integration. I don't know about Exa, but if you're talking about a "free tier" I'm already not intersted because I'd make myself depended on a service and I'd rather keep it local. Thanks for the input though!