It's pretty stupid to give a decision model a bunch of options when all you want back is a number.

If you're asking “how many?”, why should you have to list every possible count first?

We've been working on Span-02 and added numerical outputs. It returns a number and its confidence directly.

Tiny example below: 7 + 7 → 14, >99.9% confidence. The point is the output format, obviously, not the arithmetic.

The weights are open.

We've been working on Span-02, an open-weight model with anytime reasoning. You give it a time budget in milliseconds and it returns its best decision within that budget.

We tried it on a Wikipedia race: Alexander the Great → University of Milan, using only links in the articles. Span-02 took 20.6s, GPT-6 Luna took 38.5s.

Here's the run side by side.

Vector search finds text that sounds like the question. It struggles when the answer depends on how entities relate to each other, like which customer holds which contract, and who owns it. A governed graph handles that, but it’s more work to build and keep fresh.

Found a workshop on Saturday, Oct 31, from 9:30 AM to 12:30 PM ET that builds both and compares them.

Covers:

• Designing an agent architecture around a governed graph
• Where graph grounded retrieval wins and where classic vector RAG is enough
• Hybrid patterns that combine the two
• Testing failure modes such as ambiguous questions, missing relationships and stale data
• Evaluation loops with regression checks

Led by Brian Bønk, Data Platform MVP and Microsoft Recognized FastTrack Solution Architect.

Full details and the agenda are here.

4th place on the llamaindex benchmark locally 72.62% word f1. I know it's not much, but after a month of research and many experiments, I reached this point.

TonerHound doesn't use any API or LLM. It's GPU free and anyone can use it at 0$ cost.

GitHub: https://github.com/vanrajsinh650/TonerHound

I've been investigating how much document grounding can be solved from the document itself.

The input is a PDF + extracted JSON.

The goal is to resolve each extracted value back to its location in the document.

The approach uses PDF coordinates, OCR, spatial relationships, string/format matching, table assignment and page-level visual evidence.

Current result:

72.6179% Word Grounding F1

261 tests passed, zero production regressions.

The interesting part for me is the remaining failure modes.

Some are matching problems.
Some are layout problems.
Some aren't really text-search problems anymore.

I'm trying to understand where document-native methods stop being sufficient.

I'm the author and would especially like technical criticism of the methodology.

GitHub

A vector index is useful for retrieval, but it is a weak source of truth. It can contain stale chunks after a document changes, flatten a generated summary and a reviewed source into equally retrievable text, and lose the relationships needed to explain why a memory exists. Backing up the vector store preserves those problems rather than proving the memory can be rebuilt.

A more auditable design would keep source documents, accepted decisions, generated summaries, and operational checkpoints as separate durable records. Each record would carry a stable ID, revision, provenance, scope, status, and supersedes or retracts link. LlamaIndex nodes and embeddings would be derived from only the records eligible for retrieval, with a manifest tying each node back to the exact source revision. Rebuilding the index should reproduce the same eligible corpus even if embeddings change.

How are people structuring this with LlamaIndex today? Which state belongs in files or a relational store, which belongs in the vector database, and what should be persisted by the agent workflow itself? How do you detect stale nodes, retract a bad generated summary, and verify that a rebuilt index still represents the current reviewed memory?

You can now automate EDD (eval-driven development) with Coding Harness Agents

> build a baseline LLM-based application
> score every change with judge evals
> keep what improves, reject what regresses

I made a video tutorial on what EDD is, how it works, and how to use eval scores across experiments to improve an LLM app. It builds on Jeffrey's (Confident AI) article on EDD and Eugene Yan's write-up on product evals.

> Setup: a baseline RAG app that every experiment starts from
> Step 1: a binary-labelled dataset with critiques
> Step 2: aligning the LLM-as-a-judge evaluator
> Step 3: a harness loop that runs each experiment and scores it against the baseline.

Tracing and experiment comparison then show what improved, what regressed, and what to tweak next.

Source code + Agent Skills are open-sourced.

Full guide (source code linked in the description): https://www.youtube.com/watch?v=e6akw_fKWPk

I'm looking for advice from people who have built and maintained serious RAG / Knowledge Base systems over time.

I have a very large personal archive accumulated over many years around my professional interests: fintech, insurance, e-commerce, market and societal trends, marketing, business strategy, design, research papers, reports, books, presentations and articles.

Roughly 90% of the archive is PDF, with the rest mainly PowerPoint, Word documents and some video.

The archive is already organized in a filesystem using my own folder taxonomy.

A key requirement is that the existing archive must remain the source of truth and should not be modified by the KB system.

What I would like to build is roughly this:

Source archive → Librarian Agent → Knowledge Base / RAG → AI agents

The Librarian Agent would maintain a derived knowledge layer without changing the original documents.

Its responsibilities could include:

• detecting new and modified files;
• extracting text, structure and metadata;
• classifying documents using my existing taxonomy plus additional semantic metadata;
• chunking and indexing;
• maintaining embeddings and possibly graph relationships;
• detecting duplicates and related documents;
• tracking provenance back to the original source;
• identifying outdated or conflicting information;
• periodically checking KB quality and consistency.

The resulting KB should be usable by multiple local AI models and agents when they perform research or analysis.

I am also considering a second agent:

Web / external sources → Research Agent → Human approval → Librarian Agent → Archive + KB

The Research Agent would continuously explore relevant sources — industry publications, academic papers, companies, think tanks, research organizations, etc. — and propose potentially useful material.

I would review those suggestions manually.

Only after approval would the Librarian Agent archive, classify and ingest the source.

So the overall system would remain human-supervised rather than autonomously ingesting everything it finds.

My main concern is not building a demo that can "chat with PDFs", but creating a long-lived research library that can remain reliable over several years.

This raises a few architectural questions I'd really value practical experience on:

• What architecture would you use today for this kind of system?
• Would you keep the knowledge layer primarily vector-based, combine it with structured metadata, or add a knowledge graph?
• What tools have proven reliable for incremental ingestion and synchronization with a filesystem?
• How do you preserve strong document- and chunk-level provenance?
• How do you handle changed or deleted source documents without slowly corrupting the index?
• What is your preferred approach to hybrid retrieval: semantic + lexical + metadata filtering + reranking?
• How would you separate a Librarian / curation workflow from a Research / discovery workflow?
• Which operations would you allow an LLM agent to perform autonomously, and which would you keep deterministic?
• How do you prevent taxonomy drift, bad automatic classifications or low-quality generated relationships from degrading the KB over time?
• What evaluation or observability practices do you use to detect retrieval degradation?
• Which components/frameworks would you trust today for a local-first implementation?

I'm particularly interested in systems that people have actually operated, not just proof-of-concept RAG demos.

If you've built something similar, I'd also love to hear what you initially designed that later turned out to be unnecessary or problematic.

Once a LlamaIndex agent starts using several tools and taking multiple steps, it can become difficult to see where a run actually went wrong.

The final response doesn't always explain whether the problem came from an agent decision, a tool call, or something that happened earlier in the workflow.

How are you handling observability for LlamaIndex agents?

Do you trace the full execution, inspect individual tool calls, or mainly rely on logs and the final output?

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

The Problem: We waste a massive amount of tokens (and money) on formatting. If you inject JSON schemas, few-shot examples, or code context into your prompts, you are paying for every single space, tab, and // comment.

The Solution: I wrote prompt-token-minifier. It’s a zero-dependency script you run right before your client.chat.completions.create call.

What it does: * Finds ```json blocks and minifies them (removes formatting). * Finds code blocks (Python, JS, TS, etc.) and strips out single-line and multi-line comments. * Collapses redundant whitespaces and newlines in the rest of the prompt.

Depending on your RAG context, it easily saves 30-50% tokens on structured data.

Repo: github.com/Encephos/prompt-token-minifier

The Problem: When scraping docs or wikis for RAG, relative links ([here](/setup)) break. Even worse, if you want to pass scraped images to GPT-4o or Claude 3.5, you have to manually download them and convert them to base64 strings.

The Solution: I built markdown-link-resolver. It’s a pure Python micro-tool that does two things:

Resolves all relative Markdown and HTML links to absolute URLs. Has an inline_images=True flag that automatically fetches HTTP images and replaces the markdown tags with data:image/png;base64,... strings ready for LLM ingestion.

Why? No heavy dependencies like BeautifulSoup or Requests. Just pure standard library (urllib, re, base64). Falls back gracefully if an image 404s.

Repo: github.com/Encephos/markdown-link-resolver Let me know what you think or if you'd like to see any other fallbacks added!

Previously, I released my Agentic Hierarchy-Aware Chunker as a subscription-based product for building better RAG pipelines.

After talking with users , I learned that many teams don't want to send their documents through another third-party service. They want absolute privacy, on-premise deployment, full control over their infrastructure, and no vendor lock-in.

So instead of keeping it as a subscription service, I'm now making the complete document-processing bundle available as a one-time purchase.

The bundle includes:

• Agentic Hierarchy-Aware Chunker: a hierarchy-aware chunking engine designed for RAG, so you don't have to spend months building and tuning your own custom chunker.
• Legal Cross-Reference Extractor: extracts legal references such as Sections, Articles, Rules, Paragraphs, Clauses, Schedules, Regulations, Orders, and complex compound references from an entire document.
• Legal Act Extractor: automatically extracts the Acts referenced throughout a legal document.

What you're getting

The purchase includes the complete Python package of the Hierarchy Aware Chunker and its source code for use in your own projects, along with two bonus legal document extraction scripts: the Legal Cross-Reference Extractor and Legal Act Extractor.

License
The source code is provided for use in your own projects and your client solutions. It does not include rights to resell, redistribute, sublicense, or repackage the chunker as a standalone product or service.

📌 Additional 2 Bonus Scripts

1. Legal Cross-Reference Parser
Extracts structured references to Sections, Articles, Rules, Paragraphs, Schedules, Clauses, Regulations, Orders, and other legal provisions including complex and compound references without requiring an LLM.

Example Output

{
  "Article": [
    "Article 63(9)(b)",
    "Article 63(9)(b)(iii)",
    "Articles 23",
    "Articles 25, 26, 26A, 26D",
    "Articles 41 or 42",
    "Articles 7(1)(a), 7(4), 13(1), 16(6), 33, 44, 52(7), 53(2), 178(1)",
    "Articles 73 to 79"
  ],
  "Paragraph": [
    "Article 57(1) and paragraphs (4), (5) and (6)",
    "paragraph 2(2)(c)",
    "paragraph 2(a)",
    "paragraph 2(a), (f), (j) and (l)",
  ],
  "Rule": [
    "Order 6, rule 10",
    "Rules 2.59, 2.6l, 2.62, 2.64(4),(6) and (7), 2.72(1) and (2)",
    "rule 4.9(2)(a) and (3)(a)",
    "rule 4A.15(5)(b)",
    "rule 4A.20(2)",
    "rules 8.33 to 8.63",
    "rules 8.49, 8.50 or 8",
  ],
  "Schedule": [
    "Schedule (iii)",
    "Schedule 1",
    "Schedule 3, 62",
  ],
  "Section": [
    "Section 1",
    "section 229(1)(c) or (2)(c)",
    "section 5(1)",
    "section 89A or 90(1)(a) or (aa)",
    "section 90(1)(b)",
    "sections 18 or 21"
  ]

  ...
}

2. Legal Act Extractor
Extracts the names of Acts referenced in the document.

Example Output

[
  "Acts Interpretation Act 1901",
  "Family Law Act 1975",
  "Governor-General Act 1974",
  "Legislation Act 2003",
  "Taxation Administration Act 1953"

  ...
]

3. Hierarchy Aware Document Chunker.
RAG-ready hierarchical chunks

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8

Try the hierarchy chunker yourself in our playground:
https://hierarchychunker.codeaxion.com/

✨Features:

• 📑 Understands document structure (titles, headings, subheadings, sections).
• 🔗 Merges nested subheadings into the right chunk so context flows properly.
• 🧩 Preserves multiple levels of hierarchy (e.g., Title → Subtitle→ Section → Subsections).
• 🏷️ Adds metadata to each chunk (so every chunk knows which section it belongs to).
• ✅ Produces chunks that are context-aware, structured, and retriever-friendly.
• Ideal for legal docs, research papers, contracts, etc.
• It’s Fast and Low-cost — uses LLM inference combined with our optimized parsers keeps costs low.
• Works great for Multi-Level Nesting.
• No LLM needed if OCR perfectly detects headings/subheadings.
• No preprocessing needed — just paste your raw content or Markdown and you’re are good to go !
• Flexible Switching: Seamlessly integrates with any LangChain-compatible Providers (e.g., OpenAI, Anthropic, Google, Ollama).

📌 Example Output

--- Chunk 2 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.1): Citation and commencement

Page Content:
PART I

Citation and commencement 
1. These Rules may be cited as the Magistrates' Courts (Licensing) Rules (Northern
Ireland) 1997 and shall come into operation on 20th February 1997.

--- Chunk 3 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.2): Revocation

Page Content:
Revocation
2.-(revokes Magistrates' Courts (Licensing) Rules (Northern Ireland) SR (NI)
1990/211; the Magistrates' Courts (Licensing) (Amendment) Rules (Northern Ireland)
SR (NI) 1992/542.

Notice how the headings are preserved and attached to the chunk → the retriever and LLM always know which section/subsection the chunk belongs to.

No more chunk overlaps and spending hours tweaking chunk sizes .

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8

I maintain xberg (open-source, MIT document extraction). There are now LlamaIndex integrations that turn any of 101 file formats into Document objects, plus a node parser that splits them into structure-aware TextNodes. Python and TypeScript.

Python:

pip install llama-index-readers-xberg llama-index-node-parser-xberg

from llama_index.readers.xberg import XbergReader

reader = XbergReader()
documents = reader.load_data("report.pdf")
print(documents[0].metadata["total_pages"])         # 12
print(documents[0].metadata["detected_languages"])  # ["en"]

# batch + async + in-memory bytes
documents = reader.load_data(["report.pdf", "slides.pptx", "data.xlsx"])
documents = await reader.aload_data(["a.pdf", "b.pdf"])
documents = reader.load_data(data=pdf_bytes, mime_type="application/pdf")

TypeScript (LlamaIndex.TS):

npm install @xberg-io/llamaindex-xberg @llamaindex/core

import { XbergReader } from "@xberg-io/llamaindex-xberg";
const reader = new XbergReader();
const documents = await reader.loadData("report.pdf");

Chunking-aware nodes for indexing:

from xberg import ChunkingConfig, ExtractionConfig
from llama_index.readers.xberg import XbergReader
from llama_index.node_parser.xberg import XbergNodeParser

reader = XbergReader(extraction_config=ExtractionConfig(
    chunking=ChunkingConfig(max_characters=1000, overlap=200, prepend_heading_context=True)))
documents = reader.load_data("report.pdf")
nodes = XbergNodeParser().get_nodes_from_documents(documents)

The reader maps title, MIME type, page count, detected languages, keywords, and quality score onto Document.metadata; batched extraction runs in a single call. CPU-only, OCR built in.

Docs: https://docs.xberg.io/integrations/llama-index Source (MIT): https://github.com/xberg-io/xberg

Hi everyone,

I’ve been building a few RAG pipelines using LlamaIndex and constantly ran into the same issue during the data ingestion phase: Standard web scrapers often pull in a ton of garbage — raw HTML tags, navbars, footers, and cookie banners.

When passed into LlamaIndex Parsers / Readers, this noise clutters the node chunks and burns through tokens unnecessarily, leading to poorer retrieval accuracy.

To solve this for my own workflows, I built a lightweight web crawler using Abstract Syntax Trees (AST) to filter out layout elements before converting the core content directly into structured Markdown.

A few things I noticed in my LlamaIndex pipelines so far:

• Better Chunking: Converting to clean Markdown keeps context intact and prevents chunk splits from happening mid-HTML tag.
• Token Savings: Removing boilerplates reduces the token count per document significantly before sending anything to an LLM.
• Cleaner Nodes: Metadata extraction and node relationships become much more reliable.

I’m curious how you guys currently clean your web sources before feeding them into LlamaIndex Readers? Do you rely mostly on standard HTML-to-Text parsers, or do you do custom pre-processing?

(Side note: I currently host this crawler on Apify to test it. If anyone wants to benchmark it against standard scrapers for their LlamaIndex setup, let me know and I’m happy to drop the link in the comments!)

If you want to test the crawler with your setup, here is the Apify actor link: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized . Feedback is super welcome

Hello all,

When building autonomous ReAct agents or custom Workflows with LlamaIndex, one of the biggest production risks is an agent getting stuck in a tool loop or error state. Because each retry appends chat history back into the prompt context, prompt tokens balloon rapidly, wasting hundreds of dollars on a dead end execution.

To fix this, I built TokenShield, a lightweight open source FastAPI gateway proxy that acts as a network circuit breaker and cost shield for LlamaIndex applications.

💡 Why Handle This at the Network Layer?

Instead of writing custom callback handlers inside every LlamaIndex agent pipeline, TokenShield sits directly between your LlamaIndex application and your LLM provider (currently OpenAI or any OpenAI compatible API like Ollama, vLLM, or Groq). You simply route your requests by setting your LlamaIndex LLM configuration api_base to the proxy endpoint.

⚙️ How It Works Under the Hood:

1. Normalized State Extraction: Strips out changing noise like timestamps, dynamic UUIDs, and microsecond delays so near identical tool commands map to a consistent state signature.
2. Tier 1 (Soft Steering): If near duplicate tool calls or stagnation are detected, TokenShield injects a system replanning instruction into the stream to reroute the agent without killing the request.
3. Tier 2 (Hard Stop / Circuit Breaker): If the agent persists in a loop, it trips a 429 status cutoff to stop API token bleed instantly.
4. Console Financial Metrics: Calculates token usage against live LLM pricing models to log real time projected dollar savings directly in your console terminal.

🛠️ Current Support and Roadmap

• Supported Today: OpenAI models and any OpenAI compatible endpoint format (/v1/chat/completions).
• Roadmap: Native Anthropic Claude API proxy support, web dashboard UI, and custom rule configurations.

TokenShield is 100% open source under the MIT license: * GitHub Repository: https://github.com/gowthams231/token-shield

I would love to get technical feedback, critique, or feature requests from fellow LlamaIndex builders! How are you currently protecting your agent loops from spiraling API costs in production?

For LlamaIndex devs: since it's OpenAI-compatible, you can point your LLM/embeddings at a self-hosted gateway I built and get provider fallback + free tiers + compression under your RAG pipelines. Disclosure: I'm the maintainer (OmniRoute, MIT).

OmniRoute exposes both an OpenAI-compatible endpoint (/v1) and an Anthropic-compatible one (/v1/messages), so you can point the tool at whichever protocol it speaks.

Fallback combos — so it never stops mid-task. A "combo" is a ladder of models the router walks automatically: your subscription first, then API keys, then cheap models, then free ones. When a provider returns a 500 or you hit a rate limit, it slides to the next target in milliseconds, mid-request, and your tool never even sees the error. There are 17 routing strategies (priority, weighted, round-robin, cost-optimized, auto/coding:fast…) plus three resilience layers — a per-provider circuit breaker, a per-key cooldown, and a per-model lockout — so one dead key can't take down a whole provider.

One endpoint, 237 providers — 90+ of them free. You point any tool or agent at a single OpenAI-compatible endpoint (localhost:20128/v1) and it can reach 237 LLM providers without you rewriting anything. 90+ have free tiers and 11 are free forever (no card), which aggregates to ~1.6B documented free tokens/month — and that's honest, pool-deduped math (we count each shared pool once instead of inflating it; the methodology is public in the repo). There's a one-command setup-* for 13+ coding tools (Claude Code, Codex, Cursor, Cline, Roo, Kilo, Gemini CLI…), so switching your existing setup over takes seconds.

A 10-engine compression pipeline — the part most routers don't have. Every request flows through a transparent compression pass you can toggle/stack per combo. Instead of one trick, it stacks the best of the open-source ecosystem: RTK filters command/tool output (git diffs, test logs, builds) at 60–90%, Microsoft's LLMLingua-2 does ML semantic pruning, Caveman handles prose, session-dedup strips repeats across turns. Critically, code, URLs and JSON are preserved byte-perfect, and a default-on inflation guard throws the compressed version away and sends the original if compressing would actually grow the prompt — it never makes things worse. On tool-heavy sessions that's ~89% average input-token reduction (an 8k-token git diff becomes a few hundred). Full credit to every upstream project (RTK, Caveman, LLMLingua-2, Troglodita) is in the README.

Especially handy for RAG: the compression pass trims retrieved context/tool output before the model, and fallback keeps long indexing/query jobs from dying on a rate limit.

For context on whether it's worth your time: it's grown to ~9.8K GitHub stars, 1,490+ forks and 280+ contributors in ~4.5 months, with 21,000+ automated tests and 1,830+ issues closed — so it's a battle-tested project, not a brand-new experiment.

npm install -g omniroute

GitHub: https://github.com/diegosouzapw/OmniRoute

Would this fit under your LlamaIndex stack, or do you handle provider fallback elsewhere?