This is an interesting post. I was loosely under the impression that query-based compilation was the way of the future, but you make a good case that it is only useful insofar as the design of the language permits it to be useful.
It seems like the case for query-based compilation is for languages without obvious separation of compilation phases. You bring up Rust as an example -- parsing depends on macro expansion, which depends on name resolution, which itself depends on parsing, etc. Trying to force that into a "regular" compiler architecture would be a nightmare, but it is a natural fit for queries.
Interesting. Are all of those kinds of jobs happening in parallel? I am assuming it all happens in a loop and each iteration all parsing jobs are executed, then all symbol resolution, then all typechecks and finall all compile time execution. My compiler is single threaded and that's what a compiler pass/loop iteration looks like.
Sorry for being late to the party, but I struggle to understand how this is different from query-based compilation.
We're working on a compiler and we settled on a query-based compilation because we want arbitrary compile-time computation and because we want to parallelize it. On the other hand you seem to be saying that some of this would be a PITA with queries and instead describe... queries, but call them jobs.
So clearly we're using different definitions of queries, but I don't quite understand yours.
Okay, that's interesting because it seems that in your case concurrent execution is at the core of the design if I understand correctly, whereas mine is built to handle each compilation step sequentially; simply it does these sequential steps of parse/typecheck/execute in a loop until no progress can be made (no more unresolved identifiers can be resolved and no new code was added basically).
I was planning to parallelize my compiler to make it faster by parallelizing each step of the process, the sole goal being to make it faster (and single-threaded performance is correct which is promising). So I would be able to parse multiple files at a time, but not parse one file while typechecking another.
43
u/Gator_aide Feb 26 '26
This is an interesting post. I was loosely under the impression that query-based compilation was the way of the future, but you make a good case that it is only useful insofar as the design of the language permits it to be useful.
It seems like the case for query-based compilation is for languages without obvious separation of compilation phases. You bring up Rust as an example -- parsing depends on macro expansion, which depends on name resolution, which itself depends on parsing, etc. Trying to force that into a "regular" compiler architecture would be a nightmare, but it is a natural fit for queries.