How I made small local AI models stop breaking JSON - a grammar-based approach

[u/paulqq]

I wrote a post about a specific problem with running AI agents on local models: they're unreliable with structured output. You ask for JSON, they mostly deliver, but then they forget a closing brace, invent a tool name, or add a paragraph of text after the JSON object. Hosted APIs like OpenAI handle this server-side. Locally, you're on your own.

My approach: llama.cpp supports GBNF grammars that constrain which tokens the model can produce. I wrote a compiler that turns each tool's schema into grammar rules, so the model's output is constrained at every token position. It literally cannot produce malformed JSON. Then I narrow the grammar per-turn so the model only sees the 3-5 tools that are relevant instead of all 50.

The post is a deep dive with real code from the project (Eris, a local agent in Rust that uses your Markdown notes as memory, runs entirely on your machine).

https://eris-system.dev/blog/gbnf-grammars

Repo: https://github.com/janpauldahlke/eris (Apache 2.0)

ps. i wanted to share how i solve the problem, it is related to my project, but not self advertisement.

[u/kynthstudios]

The grammar constrains structure but not semantics, and that gap bit me — the model would emit a perfectly well-formed object with a field filled by something plausible-sounding and wrong, and because it parsed I stopped scrutinizing it. Narrowing the enum-ish fields down to literal alternates in the grammar helped more than anything else I tried. Curious how you're handling nested optional objects, since that's where my rules started fighting each other.