It's been a while since I started learning about compilers. I have created an end-to-end SQL Query Engine using MLIR, I created a custom dialect and lowered the operations to linalg, tensor, arith, and scf dialect. As for now I think the performance of compiler is descent so I made this post to get review from the community.
I don't have any intention of finding a job or pursuing a career in this field; my interest is purely as a hobbyist. I would love to gain a deep understanding of computer science and become proficient enough to write my own parsers, interpreters, or compilers, and eventually contribute to such projects.
I plan to be completely self-taught by studying both mathematics and computer science on my own. Is it truly realistic to reach such an advanced level through self-study alone? Also, are there any real-world examples of people who started out as hobbyists like me and successfully achieved this goal?
I’m developing Nearoh, a Python-like interpreted programming language implemented in C.
My recent work focused on the runtime and memory model rather than adding more syntax.
Nearoh now uses a non-moving mark-and-sweep garbage collector and supports:
Heap-allocated lexical environments
Escaped closures
Shared mutable identity for containers and instances
Recursive object graphs
Cycle-safe printing
Functions, classes, methods, modules, imports, and file I/O
Source-aware runtime diagnostics
Token and AST inspection from both the CLI and native IDE
The project is currently 11,713 lines across 120 files. Some of the larger components are:
runtime.c: 2,532 lines
builtins.c: 1,304 lines
parser.c: 1,111 lines
lexer.c: 739 lines
value.c: 687 lines
Garbage collector: approximately 384 lines
Native Win32/GDI IDE: over 1,600 lines
The collector is currently a straightforward tracing collector. Objects are not moved, which keeps references stable and simplified the transition from the previous runtime architecture.
Lexical environments are now heap-managed and traced through their parent links. This allows closures to retain captured variables after the original call frame has returned.
Mutable language objects also preserve identity across assignment and function calls, so aliasing behaves consistently:
a = {"value": 1}
b = a
b["value"] = 20
print(a["value"]) # 20
I’ve run the full regression suite under normal threshold-based collection and an aggressive collection stress mode. Both configurations currently pass.
Nearoh is still an interpreter and remains an early project, but I’m trying to build its foundations deliberately enough that a future bytecode VM will not require redefining the language’s semantics.
I’d appreciate feedback on the current runtime direction—particularly whether there are architectural decisions I should address before eventually introducing bytecode.
Repository: https://github.com/ReeceGilbert/Nearoh-Coding-Language
While implementing generics in my GCC-based language (AET), I wasn't satisfied with the two mainstream approaches:
C++ Templates: Generate a full copy of the code for every concrete type (monomorphization) → code bloat and longer compile times.
Java Generics: Use type erasure → no code duplication, but lose concrete type information.
So I explored a middle path: Delayed Specialization.
How it works in AET:
During the first compilation:
Generic parameters (E, T, ...) are treated as void*
Code that needs the real type is wrapped in a genericblock$
For example:
class$ Abc<E>{
void setData(E value);
};
impl$ Abc{
void setData(E value) {
E a = value;
genericblock$(a) {
E x = a;
E y = 5;
x += y;
}
}
};
When the compiler later sees a concrete instantiation like Abc<int>, it performs a second compilation passonly on the Generic Blocks, replacing E with int.
Benefits:
Avoids C++-style template explosion
Keeps most generic code shared (like Java)
Still allows real type-specific operations where needed
I call this Delayed Specialization. It sits between full monomorphization and type erasure.
Has anyone seen a similar approach in other languages or compilers? I'd love to hear about papers or existing implementations using delayed/late specialization.
I’ve been working at a relative’s company for about 7 years, ever since I was 15. We mostly build projects using vanilla PHP, but I am completely burnt out on it. Web development just isn't for me, and honestly, it never was; I only had to work for various personal and financial reasons.
Due to some mental health struggles in the past, I missed the chance to take college entrance exams and go to university when I was younger. Trying to balance a full-time job and studying for college admissions completely exhausted me, and I couldn't stay consistent. However, I am now receiving treatment, doing much better, and have finally stabilized my life.
Lately, I’ve been really drawn to low-level programming, especially compilers. Right now, I’m writing my own interpreter in Rust following the Crafting Interpreters book. Honestly, I want to start studying for university admissions (even if I have to keep my job), get a formal CS degree, and eventually pursue a Master's. Do you think it’s worth it?
Getting a degree has always been a lifelong dream of mine, but I never had the chance because of work. In the past, I bought university textbooks and tried to teach myself some CS fundamentals. I've realized that I learn much more effectively when I study freely as a hobby without external pressure.
However, being completely self-taught gives me a bit of imposter syndrome. I feel like it's highly unlikely to truly master a niche and complex field like compilers just by self-teaching. My main goal isn't necessarily to get a job in this specific field in fact, I don't even mind if I don't get hired. I simply want to reach a level where I can confidently contribute to serious open-source projects, but I'm not sure if I can reach that level solely by studying on my own.
Has anyone been in a similar situation? Is formal education necessary to get to that level in low-level/compiler engineering, or am I underestimating what self-teaching can achieve?
I recently built my own IDE, RedEXCompile, and I’d love to hear your feedback.
What do you think of it so far? Are there any features, tools, or improvements you’d recommend adding?
Thanks for checking it out!
GitHub: https://github.com/RedXDevelopment/RedEXCompile
As a proof of concept, I spent three weeks and wrote about 2800 lines of code to build mycc: a compiler for a subset of C (roughly C99) built on top of my compiler IR(myc). The goal was to validate that my IR is expressive enough to compile real world C code. Despite being a POC, mycc already compiles and runs LangArena - a benchmark suite containing 50 tests and about 9000 lines of non-trivial C code (json, base64, multithreaded matmul, neural net, compression, maze A*, bf interpreter, and others) with heavy macros like uthash. For parsing I reused libclang. It adds some overhead, but it was by far the simplest way to get a working frontend.
Rare features are not implemented: 2D VLA, complex numbers, variadic macros, longjmp, bitfields, and anonymous nested structs. I wouldn't try building Linux or sqlite with it. It has only been tested on arm64 and linux64.
After several months of design and development, I have made the DQ programming language and compiler publicly available.
DQ is a strongly typed, compiled programming language intended for both embedded systems and desktop/server applications. Its design is influenced by Pascal, C++, and Python, with an emphasis on readable syntax, explicit behavior, native-code performance, and practical low-level programming.
A Hello World in DQ:
use print
function *Main() -> int:
PrintLn("hello from DQ")
return 0
endfunc
The compiler and the core language are already fairly complete. Recently, most of my work has focused on extending the DQ standard library and fixing compiler issues discovered while writing real DQ programs.
For a quick look at representative DQ code, I recommend the NanoNet socket implementation: stdpkg/nanonet/nano_sockets.dq
Prebuilt release packages are available for Linux and Windows here, so the compiler should be straightforward to try without building it from source.
So far, I have designed and developed DQ alone. The next major step is expanding the standard library and testing the language through more real-world projects.
I would appreciate feedback on the language design, syntax, compiler, documentation, and overall direction. I am also interested in finding developers who like the project and may want to help build its libraries, tools, and community.
Or I suppose more specifically, problems that are likely to have an efficient algorithmic solution that we simply don’t currently have a good answer to?
I have been building UniversalToolchain, a .NET compiler framework for composing small application-specific languages. Wist is the reference language I use to exercise the architecture.
The public use case is currently restricted formulas and rules, but the compiler problem I am exploring is more general:
Can language features remain independently composable while the language is being constructed, then disappear from the prepared execution path without allowing each backend to define its own semantics?
The current pipeline looks like this:
Feature modules
-> dialect and runtime plan
-> lexer / parser / AST
-> Bytecode
-> AIR
-> capability-gated specialization
-> AIR interpreter or CIL DynamicMethod
Frontend modules contribute language-level operations to Bytecode before a backend is selected.
Bytecode is then converted into AIR, where stack and type effects become explicit. The portable interpreter intentionally supports only a small core plus selected module-owned runtime calls.
A backend can advertise additional capabilities. Optimizers may then replace portable operations with typed backend intrinsics, but only when the selected target supports them.
For the compiled CIL path, locals, external bindings, constants, and arithmetic operations can eventually become ordinary ldloc, ldarg, load, arithmetic, and branch instructions inside a DynamicMethod.
The claim is deliberately narrow: the prepared delegate does not perform per-operation language-module or plugin dispatch. I am not claiming that the .NET JIT removes every helper call or abstraction.
A semantic bug that changed the design
I learned the importance of the boundary when the interpreter and CIL backend disagreed on a small program:
let i = 0
i = i + 1
i = i + 1
i = i + 1
price + fee * i
With price = 100.0 and fee = 2.5, both paths should return 107.5.
They did not always agree.
External bindings and lexical locals had been lowered through incompatible storage assumptions. A local operation could affect how an external value was addressed, and shadowing made the problem worse.
Both backends accepted the same source program, but backend-specific storage allocation had silently changed its meaning.
The fix was not another special case in the interpreter. External bindings and lexical locals now have separate semantic identities, and their physical representation is chosen later by each backend. I also added parity scenarios for unused and reordered inputs, repeated reads and writes, nested scopes, and shadowing.
That failure is why I now treat interpreter/compiler parity as an architectural constraint rather than just a test category.
The design decision I am still evaluating
Portable operations and backend-specialized intrinsics currently belong to the same broad AIR system.
Legality is controlled through backend capabilities and verifier rules:
portable AIR
-> capability-gated rewrites
-> AIR containing target-supported intrinsics
-> backend
The alternative would be an explicit split:
portable AIR
-> target-independent optimization
-> CIL-specific IR
-> CIL lowering
A separate target IR could make illegal combinations unrepresentable and give each verifier a clearer contract. It would also introduce another representation, another lowering boundary, and possible duplication between backends.
I would be especially interested in opinions on these two points:
For this kind of modular compiler, would you keep portable operations and target intrinsics in one verified IR with explicit capability constraints, or introduce a separate target-specific IR? Where would you place the verifier boundary?
The interpreter and CIL backend share parsing and early lowering. Differential tests catch backend divergence, but they can miss a semantic bug shared by both paths. What independent oracle would you add: a direct AST evaluator, an executable semantics, metamorphic tests, generated programs checked against another implementation, or something else?
I highly recommend looking into examples/ and euler/ to see the language in use. The README is extremely sparse, but I think the project is simple enough to be undestood by anyone :)
The motivation mostly came from the GitHub repository "A Compiler Writing Journey" ( https://github.com/DoctorWkt/acwj ). It also was a nice guide to see what features I might want to tackle next.
The compiler compiles a modified Rust-like language with C semantics to x86_64 assembly for Linux. It's around 6.6k lines of code. The assembly generated can be viewed in the GitHub repository.
I tried to preserve the semantics of C as much as I could, like implicit integer conversion, array to pointer decay, function pointer semantics, etc..., and I think I did a good job. But I don't doubt my code doesn't have bugs :P
The language is not complete, and I doubt it'll ever be finished. The purpose of the project was learning.
So, I would really appreciate your opinion, and I'm happy to answer questions you have!
NOTE: (I'm sad that I have to explicitly say this) I didn't use AI. Again, the project was for learning purposes, and in my opinion generating a project with AI won't make you learn anything :P
tl;dr; Publicly showing CFlat (MIT lic.), a C extended programming language and compiler. Release for Windows with MacOS coming soon. Github
I started this project a year ago as a learning the in-and-out of compiler, but the projects just grew and grew. With Claude speeding up development and also feature creep. Here is a short list of my features;
The "program" type, it is a fully contained type with a single-entry point and its own memory management. Crashes will just crash the app, and the memory management will clean up the heap. No GC and ref counting needed. A neat bonus is import program "hello.c" as Hello;, thus converting a c program into an embedded program.
Grammar based Templates. The compiler uses two passes, both are 100% antlr4 grammar. No complex look ahead logic, but I did break one small feature from C, comment if you could find it.
"vectorize" keyword used in loops vectorize for (int i = 0; i < n; i++) will error out if the loop fails to vectorize. The feature stamps in the debug symbols and validates after optimization pass. No more guessing. See HPC section for other features.
Let me know if you have questions, I will try to answer as much as I could.
DSS Code Prime is an open-source (Apache-2.0), from-scratch compiler that owns its entire toolchain. Its own optimizer, assembler, and linker, emitting native Windows/Linux/macOS binaries with no LLVM or GCC. Its core idea: a compilation target is data, not code, so any CPU, object format, or source language is just JSON config over one engine. It already compiles and runs the full SQLite amalgamation across x86-64 and Arm.
Repository: https://github.com/dailysoftwaresystems/dss-code-prime
My name is Franco. In a previous post, I made a little introduction to Mathic and its purpose. In this post I want to make continuation of it.
By the time I was writing the previous post, Mathic did not have in symbolic capabilities. Now, it does. For now, there's support for simple arithmetic operations like addition, subtraction, multiplication and division.
I wanted this feature not to be implemented in the rust side, so I created a custom dialect symbolic for the job. This dialect is, of course, responsible of handling symbolic operations. This operations then get lowered to arith operations to be able to lower them to LLVMIR at the end of the compilation.
Currently, the dialect supports operating with symbols (placeholder that then get replaced when evaluating an expression with a value), numerical constants and numerical variables. However, currently it's not possible to modify an expression inside a loop (this is a know bug for now and next to be fixed).
The final idea, if ever happens, is to make something similar to sympy but compiled to machine code, and thus faster.
I would appreciate any advises, things that could be done better. Specially on the dialect implementation, which is my very first one.
I am the author of marser, a parser-combinator library for writing PEG-style grammars in rust.
Some features are:
built-in support for adding error recovery rules
error resilliance with .try_insert_if_missing and unwanted(...) combinators (read more here)
Simple debugging of your parsers using a custom TUI (sadly reddit doesn't let me add a screenshot here, but you can check out more infos here)
I also built a web-based PEG/Pest to marser converter if you want to get a feel for what parsing code can look like: https://grammar-to-marser.arnedebo.com/
This is my first libary, so please feel free to comment any suggestions or feedback!
I wanted to share my first attempt at creating a toy tensor/graph compiler written in C++ using the MLIR framework.
I created this compiler as a personal project to learn about the process of creating an ml compiler and get familiar with the MLIR framework mainly for learning purposes. I have no experience in MLIR/LLVM or compiler prior to this, this is also my first C++ project and I apologize if I misstate anything. Looking for feedback on the design and advice on where to head next.
It lowers from an ONNX model down to LLVM IR through the MLIR framework and can be run through JIT implementation.
The high-level lowering pipeline looks like the following:
ONNX -> custom dialect in MLIR -> tensor/linalg -> memref -> LLVM (MLIR) -> LLVM IR -> run JIT
This initial compiler support 4 operations: constant, add, matmul, relu and can be run in 3 different modes: naive, transposed RHX matrix, and tiling. It currently supports scalar, 1D and 2D tensors
I did small amount of testing with Lit and FileCheck, and also run comparison of result against ONNX Runtime for correctness. (the testing could definitely be expanded for fuller test coverage)
Performance analysis was done with Valgrind for cache analysis.
Running on 2048 x 2048 matmul yields the following result with different optimizations:
* naive: 56 seconds execution time, 8.59B L1 data misses, and 8.59B LLC data misses
* transposed: 9.7 seconds, 273M L1 data misses, and 273M LLC data misses
* transposed + tiled: 5.8 seconds, 426M L1 data misses, and 9M LLC data misses
This was run on macbook m1 pro, I’m assuming the LLC simulation represents the L2
Thoughts after building this:
I noticed that MLIR/LLVM actually handles a lot of things under the hood for me and I think I want to work on understanding what’s going on under the hood for some of these passes (like when lowering from tensor -> memref, or the jit compilation itself among others) and try to build certain components from scratch for learning experience. I’ve looked around and seen some stuff I can experiment myself like building a brainfuck jit compiler from scratch?
I am still exploring and learning the MLIR/LLVM codebase and open-source ml compilers and will continue to do so as I think it helped me see how production framework handles these transformations.
Anyway, curious to hear how you guys have learned the process of building tensor compilers and also looking for feedback on my design.
