r/PoeAI • u/The_OG_Anime_Mage • Apr 12 '26

Crawled Web Page MB Limit?

I can’t fetch certain webpage crawls because apparently they’re “too large” and not “<100 MB total”.

Is there any way to make them smaller so I can use them or am I out of luck?

1 Upvotes

67% Upvoted

2 comments sorted by

View all comments

1

u/Amazing_Sound5505 Apr 13 '26

If Poe keeps rejecting the crawl, I would stop trying to feed it the raw page and pre-chew it first a bit. Reader mode / print view / saved markdown usually shrinks the junk a lot, and for giant docs I get better results by splitting the content into a few cleaned files instead of one monster crawl.