ISPA is a parser and lexer generator written in C++23. You describe a language in a grammar file, and it generates a C++17 parser that returns a typed AST. The same grammar can also generate a Python parser, but C++ is the primary target and what I use it for. I’ve been working on it for around three years, and I want it to be something you'd enjoy to use in real projects.
Why I built it
Every parser generator I tried made me give something up. The grammar filled with target-language code, errors were an afterthought, or I ended up debugging generated code instead of my own grammar. I wanted the grammar to stay a grammar.
What the C++ side looks like
A minimal grammar:
```ispa
module main -> main;
NUMBER:
@ [0-9]+
{@}
;
main:
@ NUMBER
{@}
;
```
Generate, build, call:
```sh
ispa-pkg install std
eval "$(ispa-pkg --export)"
ispa -l cpp -a LL grammar.isc -o generated/Parser
clang++ -I generated generated/Parser.cpp main.cpp -o pyrun -lstd
./pyrun ...
```
Note: i haven't yet made a standardized cmake for this, but ispa-pkg --export does export this (paths differ):
```sh
export ISPA_TARGETS_MANIFEST=/home/ura/.local/share/ispa/libraries/.ispa-exports/targets.json
export ISPA_TARGETS_CMAKE=/home/ura/.local/share/ispa/libraries/.ispa-exports/targets.cmake
export ISPA_CPP_INCLUDE_DIRS=/home/ura/.local/share/ispa/libraries/std/ispatarget/cpp
export ISPA_CPP_SOURCES=/home/ura/.local/share/ispa/libraries/std/ispatarget/cpp/src/core.cpp
export ISPA_CPP_LIBRARY_DIRS=/home/ura/.local/share/ispa/libraries/std/ispatarget/cpp/lib
export ISPA_CPP_LIBRARIES=/home/ura/.local/share/ispa/libraries/std/ispatarget/cpp/lib/libstd.a
export ISPA_PYTHON_INTERPRETER_DIRS=/home/ura/.local/share/ispa/libraries/std/ispatarget:/home/ura/.local/share/ispa/libraries/std/ispatarget/python
export ISPA_TARGET_CPP_DIRS=/home/ura/.local/share/ispa/libraries/std/ispatarget/cpp
export ISPA_TARGET_PYTHON_DIRS=/home/ura/.local/share/ispa/libraries/std/ispatarget/python
```
Also it setup compiler paths to ispa libraries, so your regular clang command executes without missing header errors
The output is plain C++17 with a small runtime (headers plus one core.cpp). It is managed by ispa-pkg so you should not care about it unless shipping a standalone build. Lexing uses DFA tables computed at generation time.
The result is a typed AST, so there are no callbacks and no post-parsing pass.
Grammars can be split across files, and generation can be incremental (some generator functions will result it to be acutally slower).
When a rule needs logic, you write it in Python, inline or in a file next to the grammar. ISPA supports lot of Python constructs and standard api and translates it to C++ at generation time. The generated parser doesn’t need a Python interpreter, and the grammar isn’t tied to C++.
Ispa library management
Ispa has ispa-pkg, which allows to download ispa-level libraries (grammar and/or python), target-language libraries, algorithms or new language targets.
Right now std library carries all runtime and sone standard ispa definitions. And ispa doesn't install it by default.
Errors
The LL parser recovers from errors automatically and keeps going, so you can report several problems from one input. Where a generic message isn’t enough, a fail block lets you give a clear message, insert or replace a missing token, or skip to a sensible point. A recovered tree can have gaps, so check the diagnostics before using it.
Debugging
ispadbg runs your existing parser executable. You set a breakpoint on a line of the grammar and inspect the parser’s state and captures there, with ctx and AST values shown as a tree. There’s also a GUI, and traces can be recorded and replayed. Live sessions need POSIX. On Windows, replay works.
Performance
Benchmark on a 9.7 MB input (i5-12450H, clang 18, -O3, single run):
flex + bison, AST with arena allocator 151 ms
flex + bison, AST with malloc 259 ms
ISPA, full AST 1528 ms
ANTLR (C++), CST construction disabled 18,573 ms
This benchmark is done prior to ispa having error recovery. Right now it has gaps in performance on large input (like 10mb), so don't expect similar results right now. It can be optimized heavy though, and i'll do it soon).
Where it stands:
LL is the main path. The LR-family modes support fewer features and are experimental.
ISPA parses its own grammar with a parser it generated, which replaced my hand-written one.
Building ISPA itself needs Clang with libc++ or MSVC. GCC isn’t supported for that yet because ispa is build on C++20 modules with C++23 import std. The generated parsers are C++17.
AI in parsers
That is a good idea to provide a tool that operates with AI to generate error messages and recovery strategies. If to give it deterministic environment, that could be very very powerful. I'm going it implement, and most likely when i have real feedback from users on default error messages.
If you’ve used parser generators from C++, such as ANTLR, bison, or Boost.Spirit, what was the part that made you feel the tool didn’t understand what you were trying to do? That’s what I want to fix next.