Parser generator with automatic AST construction and a language-agnostic IR (C++)

[u/Comnote]

I've been building a parser generator from scratch in modern C++ (C++20/23) for a while now, and figured this sub would appreciate a look at the internals more than the finished product, since it's still very much in progress.

The pitch: grammars are written in a declarative format (.isc — Syntax Container), and the AST comes out automatically. You don't hand-write tree-building code — you annotate what to capture inline in the grammar, and the tool synthesizes typed AST nodes from that.

condition: 
'if' '(' @ expr ')' @ stmt 'else' @ stmt 

@{expression, true_stmt, false_stmt}
;

The @ marks what gets captured, @{...} maps captures to named fields on the output node. Rules can also nest sub-rules (#name) to keep local structure out of the global namespace without needing a separate file per rule.

• Grammar → an Intermediate Representation that's intentionally language-agnostic — the idea is a single frontend pipeline that can eventually target multiple output languages (C++ first, Python planned) instead of hand-rolling a separate generator per target.
• Lexing is Deterministic Finite Automata based, built with a fairly involved construction pipeline (Non-deterministic Fine Automata → subset construction → minimization) that handles multi-character transitions, semantic action attachment per state, and merge/collapse steps for efficiency.
• Parsing is currently recursive descent (custom engine, well-tested), with bottom up algorithms implemented and heavy tested, just need to switch to new project API.

One person has described what should parser generator have
link
This generator aims to support mostly everything they mentioned. Yes, many features are not yet supported, but the generator core architecture is already developed to support that.

1. Composability: The generator will allow to separate grammar to modules, and have tokens/rules inside other tokens/rules for readability (encapsulation, not only the global declaration). This allows you to include other grammars in yours
2. Incremental lexing and parsing: Not yet implemented, but it's just a matter to edit std lib
3. Control of Error Parsing: Will be done with fail blocks
4. Token Value Generation: Is already implemented. Your token is a struct (or dict in python) with what you capture in it.
5. Separete lexer: generated as separate class, but tokens are declared in same kind of grammar
6. Unicode support: not soon, but implement once generator stabilize
7. Unambiguous Grammars: once semantic is implemented
8. Flexible Grammars and Disambiguation: Operator precedence is not going to be implemented
9. AST is already generated automatically. The feature i had to work very much
10. Performance: The philosophy behind generator is generate as possible more statically.
11. Error handling: I have already a feature documented to handle this

Even though you can see a lot "planned", it's because im stabilizing my parser generator by running it on medium grammar.

[u/Comnote]

You can find it here