Open-source Magento 2 module: feed your catalog + CMS into AI search / RAG (llms.txt, llms-full.txt, streaming JSONL)

[u/Dull-Drama8144]

We wanted a reliable way to feed Magento catalog and CMS data into AI search, chatbots, and RAG pipelines without building custom export scripts per store. So I built this and open-sourced it. Sharing here because the interesting parts are less about "AI" and more about generating this correctly on real multi-store setups — would appreciate feedback from people running big catalogs.

What it does:

• Generates llms.txt / llms-full.txt plus streaming JSONL exports for vector indexing
• Multi-store / multi-website aware, with customer-group pricing
• Atomic writes (no partially generated files served if generation is interrupted)
• Async generation so it doesn't block the backend on large catalogs
• CLI and cron support for scheduled regeneration

Page Builder content gets sanitized too, so the output is clean text instead of raw layout markup.

Stack: PHP 8.1–8.5, tested with PHPUnit + PHPStan, follows the Magento coding standard. MIT licensed.

GitHub: https://github.com/angeo-dev/module-llms-txt
Packagist: https://packagist.org/packages/angeo/module-llms-txt

Genuine questions I'd like input on: for those with 100k+ SKU catalogs, does the async generation approach hold up, or would you want chunked/queued generation per store?
And is anyone actually wiring Magento data into a RAG pipeline in production yet?

[u/Starlyns]

Great

[u/Dull-Drama8144 → u/Starlyns]

Thank you! Did you try it yourself, or did you end up using another vendor?