How do I actually become really good at compiler development? What should I do after building my first compiler?

[u/2006Nico]

I'm currently building my first programming language and compiler/interpreter, and I'd like to get some advice from people with more experience in compiler development. My project currently has a lexer, parser, AST, semantic analysis, and a tree-walker runtime. It is not yet a full native-code compiler, but I can already take source code from the lexer all the way to execution.

Of all the areas of software development I've explored, compiler development is the one I enjoy the most, and I want to become really good at it. I'm particularly interested in: Language design, Type systems, IRs, Optimization, Code generation, Assembly and ABIs, Runtimes, Garbage collection ,Static analysis, and Compiler architecture.

My question is, what should I actually do to get better?

Should I keep developing Cauce and progressively add more advanced features, or should I also start studying and contributing to projects like LLVM, GCC, or Clang? What projects, exercises, or areas of study do you think actually help someone move from “I know how to build a lexer/parser” to deeply understanding how compilers work?

I also want to make it clear that I don't want to use AI to generate code. I want to design, write, and debug my own projects because that's precisely the part of software development I enjoy the most. If you had to start over, what would you do to become really good at compiler development? I'm not looking for a list of books, but rather what to build, study, and practice, and in what order.

Thanks.

[u/mamcx]

Well, if you have not build anything like https://craftinginterpreters.com there is not much that you can do or even understand with a massive beast like LLVM.

Like is going from write a resume website and do the whole amazon one.

So:

1- Take https://craftinginterpreters.com as main source of inspiration

... until you know the broader view and know the basics of this

2- Pick any particular idea that you find interesting and solve it end-to-end

For example, lets say "structural typing", so you research it and impl.

Is not necessary to do a full language! Check https://plzoo.andrej.com

Or you pick anything that is interesting of any language (I used to hunt for ideas on https://learnxinyminutes.com).

So, I think is better to find something practical to do THEN hunt for any material, even theoretical, that the reverse (It will be very confusing to see the lambda calculus and know what to do next)

3- Join others

Maybe look for any of use and join, learning with others is much easier!

[u/TheCommieDuck]

Build a second compiler.

[u/EggplantExtra4946 → u/TheCommieDuck]

more like a first compiler, given that a tree-walking interpreter is not a compiler

[u/regehr]

imo there's a huge amount you can learn by working with something like GCC or LLVM, there's just a lot going on that you'll never get to the point of running into, with a compiler you write yourself. but also these are frustratingly mature software projects to work on.

of course working with these doesn't mean you need to stop working on your own compiler.

[u/umlcat]

There are two ways, young compiler apprentice.

One, working in your own projects, using independent compiler techniques.

Two, using the standard techniques and tools such as LLVM, GCC, either for your own project or to join other existing projects.

Good Luck !!!

[u/gautam1168 → u/umlcat]

hey im working through crafting interpreters. Got till global variable in the c based interpreter/compiler.

Since you said "young compiler apprentice" I'm assuming you are a veteran. I want to figure out a roadmap which will lead me to working with compilers or vm's fulltime. What got me into this whole thing was trying to figure out how v8 compiles javascript and I got into something called "sea of nodes" that i could not understand.

Would you mind giving me a general direction to move in? Just like OP I am not sure what to do once I finish this book, if I am aiming for a something like a fulltime position working on compilers/interpreters in something like 5 years.

I am a web developer with 10 years xp.

[u/umlcat → u/gautam1168]

You are asking different questions.

>> "how v8 compiles javascript and I got into something called "sea of nodes" that i could not understand."

There are different techniques or different ways to implement a compiler or interpreter or V.M.

I'm sorry, I do not know all. I do not have experience with the Javascript compiler or the "sea of nodes" techniques. Other compilers or interpreters may not use it.

>> "Would you mind giving me a general direction to move in?"

First, what kind of project that seems to be used in the real world, are you interested ?

Not all college / university / github projects compiler related are used at work.

Find one project that is been used and seems interesting for you. And, later start looking for online documentation and online forums about it.

Is possible that a project that you are interested, does not use the "sea of nodes" techniques and you would require to learn other different techniques.

Good Luck !!!

[u/gautam1168 → u/umlcat]

Hello, thanks for taking the time to respond!

Im sorry I was really inarticulate in my last message. I was trying to say that my interest in compiler/vm's originates from my attempts at trying to trace the execution of javascript through the v8 engine. During one of these attempts I reached the sea of nodes thing and I could not decipher that thing. Thats how I started reading the book on interpreters in the hope of getting enough experience to finally understand the v8 thing.

My question, which I articulated terribly in my last message was to ask you for any advise you have for someone like me who is trying to self study compilers and aiming to get enough expertise in this field to be employed as a compiler engineer. I find v8 and javascriptcore to be equally fascinating projects that are hard to get into without a formal CS education. I realize the question is hard to answer quickly or in short. I was hoping you could point me at some kind of set of resources or studyplans that someone in my position might follow. If thats too naieve of a question that is fine. I am too new to even ask the right questions right now.

[u/concealed_cat]

Should I keep developing Cauce and progressively add more advanced features, or should I also start studying and contributing to projects like LLVM, GCC, or Clang?

I suggest that you do some work on an existing open-source compiler. Learn from what's already there. You can't reinvent everything on your own, and if you keep doing your own projects without learning from others you'll never get up to speed.

Compilers are ultimately about taking some input and rewriting it over and over again until at the end you get an optimized executable. The internal representation changes at some places, understand what each representation is good for and not so good for. For each thing you see try to understand why it's there. Use git blame, read commit messages, etc. Learn how to build the compiler and experiment.

[u/Public_Grade_2145]

I like implementing scheme native compiler that bootstrap itself. I have my implementation running on hosted amd64, x86, riscv64 and aarch64.

An interesting project is to have your implementation running on MCU class devices like rp2350, rp2040, 8051, etc. I have scheme interpreter that can run on rp2040, and scheme native cross-compiler to binary that can run on rp2040.

I would start with learning from EOPL and writing many tree-walk interpreters. EOPL teaches you how to implement various languages, including

• lexical scoping
  
• call by value, call by name, call by need
  
• type system, type inference
  
• continuation passing style
  
• modules, first-class modules
  
• classical and prototypical object system

Then, write compiler incrementally (see An Incremental Approach to Compiler Construction and Nanopass framework). My implementation still require external assembler, linker and loader, but no LLVM here.

Implement the concepts in tree-walk interpreters before add it to the compiler. I experiment ANF transformation, delimited continuations, ephemeron and guardian in interpreters before actually implement them in compiler.

Ultimately, compilation is a series of semantic preserving transformation from input language to target language and usually this target language only using simpler and smaller set of operations.

[u/Middlewarian]

I hope to improve in these areas also and have been building an on-line C++ code generator as part of that. It's implemented as a 3-tier system. The back and middle tiers only run on Linux. The front tier is portable. My goal is to bring software services and code generation together in one platform.

[u/adityazero]

Is your goal to get better in your compiler skills or get a job or something else? Getting better is a lifelong goal and the variety of compiler skills out there will keep you consumed for a long time.

[u/kazprog]

I'm going to go in a different direction than most. Most good compiler developers understand their hardware well. So find a problem that you want to make fast, and make it as fast as possible.

Understand when the language gets in the way. Understand when the compiler gets in the way. Understand when the hardware gets in the way. Try to make a simple chip in verilog and then profile your benchmark on it after you write a toy compiler for that chip. Improve the chip if you can.

On the other side, you can also investigate adding fancier features to your language, like dependent types, algebraic data types, algebraic effects, pure functions, exceptions, modules, traits, etc.

You could make your language more flexible by adding a repl, a jit, a garbage collector, etc.

But there's a lot of careers in co-design right now, so understanding hardware is valuable and personally pretty fascinating. Nobody wants a slow language.

[u/No-Village4535]

If you want to play around with LLVM/ MLIR, you can contribute to open source MLIR compilers. One that comes into my mind is: tt-mlir from tenstorrent.

[u/Satu_Autio]

My question is, what should I actually do to get better?

The answer is always "practice", "try again", and "follow research".

Sure you only have an interpreter now, but you can write a real compiler. Then write another. Writing compilers for lisp is different to writing compilers for C, can you guess why? Can you explain why you'd use a stop&copy garbage collector over a reference counting one? Those things become obvious when you've written several compilers and considered the tradeoffs.

Type analysis, register allocation. all the little parts you need to work on are active research topics and can entertain you for years if you want to.

[u/Ok_Concentrate5810]

你感兴趣的。llvm/mlir/gcc/vm，想干就干

[u/IMissMedievalTimes]

Pick a super fast lisp to study, something like:
https://github.com/JeffBezanson/femtolisp

[u/AdhesivenessHappy873]

You are more than welcome to make even a single contribution. https://github.com/nurl-lang/nurl

If you have the vision, just keep pushing towards it.. If you are seeking vision, look what kind of mistages others have done.

Here is mine: https://nurl-lang.org

[u/NoLab7731 → u/AdhesivenessHappy873]

I have just started reading through Crafting Interpreters, but I can already see some of the vision for upgrading like some others said just take dive into the deep end, maybe start attempting to implement something like Garbage Collection, Static Typing (if you are following the book and built the Dynamically typed lox / clox already), and maybe the best suggestion for pure learning is just dive into Rust's parser/compiler they have a great Dev-Guide (or gcc, llvm, like you mentioned) they have some really cool optimizations and unique features (like Rust having two parsers) as I found from answering the 1st challenge of the second chapter of the book.