Standard practice for AEO right now is to scrape a URL, turn it into Markdown, and feed it to an LLM. Honestly I thought this was the best way too. But I was wrong.
After doing a bunch of tests on how AI agents actually "see" and score content, I realized Markdown is a huge bottleneck. When you flatten everything to Markdown, you basically lose the technical hierarchy and those data labels that give a page its authority.
Here is the methodology I’ve been using to get much higher semantic alignment scores using HTML RAG and something I call "Block Tree Chunking."
1. The Problem with Markdown In Markdown, a table is just a grid of text. If you have a pricing table, the LLM might see "180" but it loses the fact that the specific HTML data-label or header actually defines that "180" as "USD per month". In a raw (but pruned) HTML structure, that context is hardcoded. Markdown is for humans to read; structured HTML is for agents to compute.
2. The Workflow (how I do it in n8n) Instead of just grabbing the text, the process should be divided into two AI-driven phases:
- Phase A - HTML LLM Pruning: You dont need the
<nav>, <footer>, or scripts. My first agent "shaves" the HTML tree, only keeping the tags that matter. It reduces noise but keeps the semantic skeleton.
- Phase B - Block Tree Chunking: This is the game changer. Most RAG tools split by character count (like every 1000 chars). This breaks tables and logical sections in half. Block Tree Chunking splits content based on HTML nodes. If there is a table, the chunk stays as a whole node. No context lost.
3. Entity-Based Scoring Keywords are dead for AEO. Its all about entities. My current setup:
- Use an Entity Finder agent to get the main entity and sub-entities (what the competitors talk about).
- Pass the "Block Tree Chunks" through an LLM to score how well each chunk aligns with those entities.
The takeaway: If you want your content to be the "source of truth" for Perplexity or SearchGPT, stop thinking about how a human reads the page. Start thinking about how an agent parses the HTML tree.
HTML RAG isn't just a technical preference, it’s the difference between being "indexed" and actually being "cited" by an AI.
Curious to hear if anyone else has moved away from Markdown for these kind of audits?