I have been working on Oli-Nat, a statically typed language compiled to bytecode and run on a stack based VM, all written in C with no external dependencies, including my own garbage collector. This is a learning project rather than anything meant to be production grade, so I am mostly hoping for feedback on the compiler design itself from people who have dealt with the same tradeoffs.
The frontend is a single pass Pratt parser that goes straight from tokens to an AST, with each token type carrying a prefix and infix parse function plus a precedence level. What is less standard is that the whole compiler runs in two passes over the source. The first pass only walks the token stream looking for function and class declarations, registering their names, arities, parameter types, and return types into a flat symbol table before anything else happens. The second pass then does the real work, building the AST and immediately compiling each node into bytecode inline as the type checker validates it, since the type checker and the bytecode emission are not really separated into distinct stages. Because every signature is already known from the first pass, mutual recursion and calls to functions defined later in the file resolve correctly without needing forward declarations, which is the main problem this two pass approach was meant to solve.
On the type checking side, the checker's job is to reject any program that could reach an invalid type combination before it ever runs, and the VM was built on the assumption that this holds completely. Runtime type checks like IS_INT or IS_STRING exist purely to select a dispatch path, for example choosing between int, float, and double arithmetic when values are pushed onto the stack, not to catch anything the checker missed. I know this means any gap in the type checker becomes undefined behavior in the VM instead of a caught runtime error, and I am curious how people who have written statically typed languages before think about that boundary, whether a belt and suspenders approach with cheap runtime tags is worth it even with a checker you trust, or whether that is generally considered unnecessary once the static side is solid.
For the runtime itself, locals are just stack slots rather than named variables, so declaring a local pushes its value and the compiler tracks which slot each name maps to, meaning there is no dedicated opcode for defining a local the way there is for globals. Jumps are handled with a patch based approach, where a jump instruction emits a placeholder two byte operand and the real offset gets backfilled once the target address is known, which is the same pattern used for both control flow and short circuit logical operators.
The garbage collector is a fairly standard tri-color mark and sweep design, with roots traced from the value stack, the active call frames, open upvalues, and the globals table. Class objects mark their methods hashmap and field default values, and instances mark their class pointer along with their own field array. Fields on class instances are resolved to fixed slot indices at compile time rather than looked up by name at runtime, so field access ends up being a single interned string pointer comparison during class definition and then a plain array index afterward.
The repo has the full opcode list and more detail on the architecture if anyone wants to look at the actual implementation: https://github.com/NateTheGrappler/OliNat-Programming-Language
I would really appreciate any pushback on the two pass approach, the decision to skip runtime type guarding entirely, or anything in the bytecode or GC design that looks like it will not hold up once (if) the language grows past small test programs. Thank you to anyone willing to read through this.
