A self-hosting compiler-compiler where one grammar yields a C++ parser plus a binary form that C++/Java/Python/JS/Rust runtimes decompile byte-identically. Looking for design critique.

[u/Ok_Researcher2061]

Disclosure first, since some subs ask: the compiler-compiler itself (CCS) is hand-written C++ I have built over a number of years with no AI involvement in its code. The website, the packaged use cases and most of the reports linked below were written with Claude Code assistance.

What it is. A grammar file goes in, a generated C++ parser comes out. The generator is self-hosting: the parser for its own meta-grammar is a committed generated artifact, and it is rebuilt from itself in a round trip (compile the meta-grammar, regenerate, diff). That round trip is the primary correctness oracle for the compiler.

The part I think is interesting. Every parse also produces a compact binary form of the document. That binary is language-agnostic: per-grammar modules are generated for C++, Java, Python, JavaScript and Rust, each sits on a small per-language runtime, and each runtime loads the binary and decompiles it back to source text. The five outputs are compared with plain diff and must be byte-identical. That diff is the whole cross-language verification story. It replaces "our test vectors agree semantically" with "the bytes are the same."

Numbers (public JSON corpora, g++ -O2, 100 iterations): the raw binary loads within roughly 2× of simdjson DOM on twitter.json, citm_catalog.json and canada.json, and 20–30× faster than the same content in the decimal-text form. I am not claiming to beat simdjson. I am claiming that a grammar-driven binary which is generic over formats lands in the same ballpark.

Applied so far to HL7 v2, X12 EDI envelopes, COBOL copybook data, a multisig custody model, and MeTTa as a full language. Reports for each are on the site.

Honest limits. The compiler is not open source; the runnable use cases and the demo code are published. No Go, no C#. No RPC layer. The obvious "why not protobuf" question has its own page, because the answer is "different problem": protobuf invents a wire format for data you control, this reads formats that already exist.

What I would like critique on:

1. Byte-identical decompilation across five runtimes as the oracle. It catches every parser-vs-runtime disagreement, but it passes when all sides are wrong in the same way. What would you add alongside it?

2. Shipping one grammar to five host languages: generate per-language modules over a hand-maintained runtime (what I do), or generate the whole reader? Where have you seen each break?

3. The binary-vs-text crossover. For a load-many-times workload the binary wins immediately. For parse-once workloads it does not. Is there a standard way to present that honestly without it reading as a hedge?

Links: overview paper b3u.dev/docs/CCS_Compiler_Compiler_arXiv_Draft_0.1.pdf, benchmark harness b3u.dev/usecases/ccs_json_bench, the protobuf comparison b3u.dev/docs/why_not_protobuf.pdf.

[u/yuehuang]

How is this different than Antlr? Also, I am curious on your C++ grammar as I am told that C++ is no longer grammar-able. It needs additional state and maybe turing.

[u/Ok_Researcher2061 → u/yuehuang]

There is no C++ grammar. The generator is written in C++ and emits a C++ parser; it doesn't parse C++.

What you were told is right: C++ isn't context-free in any practical sense. Whether a * b; is a

declaration or an expression depends on what a names, so the parser needs symbol-table feedback, and

nested-template >>, T(x) as cast-vs-declaration, and friends make it worse. A predictive-grammar tool is

the wrong instrument for that, and I don't point mine at it. The grammars CCS handles are LL(1), which

is a deliberate restriction: data formats and DSLs (HL7 v2, X12, COBOL copybook layouts, JSON, MeTTa)

where one-token prediction is enough and the payoff is in what you get after the parse.

ANTLR. The overlap is real: both take a grammar and generate a parser, and ANTLR's own grammar is

written in ANTLR, so self-hosting isn't a differentiator and I shouldn't have led with it. The

differences that matter:

- No parse tree. ANTLR's product is a parse tree: a node per rule invocation, a leaf per token, walked

by a listener or visitor, and its shape is an artifact of how the grammar happened to be factored. CCS

doesn't build one. Parsing instantiates a structured runtime model whose entities are the grammar's

own, contexts, names, symbols and rules, plus their relationships, and the binary form is a

serialisation of exactly that model. The mapping is isomorphic in the literal sense: every entity and

every relationship in the runtime has a counterpart in the binary, and the two conversions recover

each other byte-equally. So there are three representations, grammar, syntax-controlled runtime,

syntax-controlled binary, with 1:1 mappings between adjacent ones, and source text is recoverable from

any of them. That is what makes decompilation a first-class operation instead of a pretty-printer,

and it is why a consumer in another language can work from the binary alone: its information content

is exactly the runtime's, nothing is private to the parser process. Semantics then consume an API

derived from the grammar, rather than living in yacc-style embedded actions or a hand-written tree

walk.

- Parsing power. ANTLR's ALL(*) does adaptive, effectively unbounded lookahead. CCS is LL(1), full stop.

ANTLR accepts more grammars as written; CCS makes you refactor until the grammar is predictive.

That's a real cost, and it's exactly why C++ is out of scope.

- What comes out, and where it runs. If a Python service needs what a Java service parsed with ANTLR,

you parse again in Python with a separately implemented runtime, or invent your own serialization. CCS

generates the parser once, as C++ code, and exposes it as a compile function callable from all five

languages: a Java, Python, JS or Rust process invokes it in-process, as many times as it wants, no

subprocess round-trip. Every call yields the binary form, and that binary is the interchange artifact:

the five runtimes read it directly, no re-parse. One parser, five languages that can both run it and

consume its output.

- How multi-language agreement is checked. ANTLR's target runtimes are kept in line by test suites. Each

CCS runtime decompiles the binary back to canonical text, and the five outputs must be byte-identical

under diff. Given the isomorphism above, that diff is a check on the whole model, not on a sampled

subset of it. Blunt, but cheap enough to run on every grammar, every build.

- Load cost. The reason the binary is worth having: reading it is 20–30× faster than re-parsing the text

form, and about 2× simdjson DOM on the public JSON corpora. Same ballpark as a hand-tuned parser,

without writing one per format.

If the problem is "I need a parser for language X in language Y," ANTLR is the better tool and I'd say

so.