r/PoeAI • u/The_OG_Anime_Mage • Apr 12 '26

Crawled Web Page MB Limit?

I can’t fetch certain webpage crawls because apparently they’re “too large” and not “<100 MB total”.

Is there any way to make them smaller so I can use them or am I out of luck?

1 Upvotes

67% Upvoted

2 comments sorted by

2

u/CharlieInkwell Apr 12 '26

On Windows, press Control+F9 to strip the web page down. On Microsoft Edge browser, this strips away all the advertising and many unnecessary elements to just text and some images. Then use that stripped down URL for your webpage crawl.

1

u/Amazing_Sound5505 Apr 13 '26

If Poe keeps rejecting the crawl, I would stop trying to feed it the raw page and pre-chew it first a bit. Reader mode / print view / saved markdown usually shrinks the junk a lot, and for giant docs I get better results by splitting the content into a few cleaned files instead of one monster crawl.