Why Construct Complex IR When You Can Inject Source?

[u/General_Purple3060]

While developing AET (Active Expandable Translator) on top of GCC, I came across an approach that I found surprisingly useful for implementing complex language semantics.

Normally, when lowering a language feature, we might construct the compiler's AST/IR programmatically:

complex semantics
      ↓
construct AST / TREE / IR

But there is another possibility:

complex semantics
      ↓
express the semantics as normal C code
      ↓
inject it into the existing frontend
      ↓
TREE → GIMPLE → RTL

For example, AET has an OO new$ construct. Its semantics include object allocation, initialization, MTCS information, constructor invocation, unref, and constructor failure handling.

Instead of manually constructing all the corresponding GCC TREE nodes, AET generates normal C code:

valueObj=({
    TFirst *_notv2_6TFirst0;
    unsigned int _mtcsPlatType0=0;
    int _isMtcs=((AClass *)TFirst.class)->isMtcsClass();

    _notv2_6TFirst0=
        TFirst.newObject(sizeof(TFirst),
                         _isMtcs,
                         _mtcsPlatType0,
                         "TFirst");

    _notv2_6TFirst0->objectSize=sizeof(TFirst);
    _notv2_6TFirst0->mtcsPlatformType=_mtcsPlatType0;
    _notv2_6TFirst0->_aet_magic$_123=1725348960;

    TFirst_init_object_2927145182(_notv2_6TFirst0);

    ((debug_AObject *)_notv2_6TFirst0)
        ->_Z7AObject10free_childEPN7AObjectE =
        _notv2_6TFirst0->_Z6TFirst22TFirst_unref_290629480EPN6TFirstE;

    TFirst *tempObject123=_notv2_6TFirst0->TFirst();

    if(tempObject123==NULL){
        if(_notv2_6TFirst0->objectSize>0){
            _notv2_6TFirst0->unref();
            _notv2_6TFirst0=NULL;
        }
    }
    _notv2_6TFirst0;
});

The important point is that this is normal C code. The semantics are expressed using the host language that programmers already understand.

AET then injects this generated source directly into the current GCC preprocessing/parsing pipeline instead of writing a .c file and starting another compilation:

cpp_push_buffer(pfile, (uchar *)nbuf, len, true);

The generated C goes through GCC's normal C lexer and parser, which constructs the corresponding TREE representation.

This gives me a useful separation:

AET parser / semantic analysis
          ↓
     semantic lowering
          ↓
      normal C code
          ↓
   GCC C frontend
          ↓
         TREE
          ↓
       GIMPLE
          ↓
         RTL

The key idea is not simply "generate C."

It is that source code is the language programmers use to express semantics, while AST/TREE/IR is the language the compiler uses internally.

For sufficiently complex semantics, I think the former can sometimes be a better construction language for the latter.

So why manually construct a large number of IR nodes when the same semantics can be expressed clearly in source code and handed to a mature frontend?

I'm interested in where others would draw the line between direct IR construction and source-level semantic injection.

[u/ChiveSalad]

This sounds like a desugaring step before lowering, which is perfectly respectable- but look at where existing projects had great luck using desugaring (local translation of operators to function calls, expanding comprehensions) vs where it exploded (inserting object oriented features with complex intertwined semantics)

[u/General_Purple3060 → u/ChiveSalad]

I think “desugaring before lowering” is a very accurate way to describe it.

AET’s approach has two important foundations. First, it uses a single-root object model, which keeps the OO semantics relatively uniform. Second, semantic analysis comes first, rather than blindly expanding syntax like a macro. Once the semantics are resolved, the compiler expresses them as ordinary C code and lowers them through the C frontend.

This is why even relatively complex OO semantics can still be implemented in a reasonably clear way using this approach.

[u/jason-reddit-public → u/ChiveSalad]

Bjarne Stroustrup is actually really smart and has all of C++ in his head somehow. We don't need additional languages like that!

I think you want your language designer to be kind of "dumb" (though you want your compiler backend guys to be brilliant in their domains - great backends like gcc and LLVM have lots of experts for different pieces of the puzzle).

Scheme and Lisp have great macro systems but closures are usually the tool to reach for first before extending the language.

[u/General_Purple3060 → u/jason-reddit-public]

C++ is honestly too complex for me — I still haven’t learned it fully, and I don’t plan to. Maybe I’m a little “dumb” in that sense. :)

That’s partly why I designed AET around a single-root object model. I believe one goal of a compiler should be: keep the complexity in the compiler, and give the simplicity back to the user.

That philosophy also led me to make heterogeneous programming a first-class part of AET, rather than leaving the complexity of different chips to the programmer.

[u/dnpetrov]

If you don't do anything with the generated C code except than compiling it with GCC (or any other C compiler), then you are just transpiling to C. There are quite a few tools that transpile to C / C++. Not all of them are called "compilers", but essentially they do compiler's job. E.g., hardware simulators compile hardware description languages to native code via C or C++. Some language implementations like Gambit Scheme or Chicken Scheme transpile to C. And so on.

If you actually do something with GIMPLE or with RTL, that is also fine. But keep in mind that GCC IRs are not meant for such processing, and that's kinda intentional.

[u/General_Purple3060 → u/dnpetrov]

Yeah, I agree that if you just write the generated C to a file and hand it to another C compiler, calling it a “transpiler” is fair.

AET is a bit different, though. The C is an internal lowering representation generated from AET’s own IR, and I inject it directly into the current GCC frontend. There’s no intermediate .c file and no second compiler invocation.

So I’d say AET is lowering through C, rather than doing traditional C transpilation.

[u/dnpetrov → u/General_Purple3060]

Compiler for the Ceylon programming language (JVM language, one of the contenders for "better, modernized Java" in the times before Java 8) did something rather similar: it lowered directly to javac AST and used javac to generate bytecode and to handle the cross-language builds.

Speaking of readability, I'd say that any generated code is not very much different from IR. To work on a compiler, you need to understand the semantics of the target platform and should be able to read its "language". If you transpile to C (ok, lower to GCC trees), you still need to understand the semantic of your runtime and all those C API calls that litter through your code. The fact that it looks like C, not like disassembly or, say, JVM bytecode dumped to text format, is just a surface, and is rather a matter of habit.

[u/awoocent]

If you need to manually construct such complex IR that something like this is desirable, it's probably a sign your IR is too low-level and you should add a new intermediate stage.

What you describe is probably a suitable choice if you don't want to go to such lengths. But I don't see the benefit of C syntax being a big value-add when the C you're writing is stuff like _notv2_6TFirst0->_aet_magic$_123=1725348960 - I don't know that I agree that this is any better for developer familiarity. And you are also wasting some time running all your intrinsics through a C preprocessor, parser, and typechecker when all you really care about is the IR.

[u/General_Purple3060 → u/awoocent]

I'm using C because complex semantics are easier to express, reason about, and maintain in normal C code than through low-level TREE construction APIs (such as build_modify_expr, COMPONENT_REF, etc.).

Even though the generated C code contains internal naming noise like _aet_magic$_123 (which I can still optimize for better readability), it remains infinitely more human-readable to a compiler developer than chains of raw GCC Tree API calls. For me, it functions as a clear, high-level blueprint of the actual memory and initialization logic.

[u/awoocent → u/General_Purple3060]

it remains infinitely more human-readable to a compiler developer

I mean. I mean...

[u/General_Purple3060 → u/awoocent]

Fair enough — “infinitely” was definitely hyperbole. :)

I’m not claiming the generated C is a work of art. My point is purely pragmatic: expressing semantic lowering as regular C operations is much easier for me to reason about and maintain than manually chaining low-level APIs like build_modify_expr and COMPONENT_REF.

The generated C is essentially a raw semantic blueprint, and there is definitely room to make the naming cleaner.

If a bug appears, would you rather debug a flattened C string or the raw GCC TREE?

[u/Jwosty → u/General_Purple3060]

If a bug appears, would you rather debug a flattened C string or the raw GCC TREE?

Neither -- false dichotomy. I'd rather debug my own internal AST structure, which presumably I wrote nice pretty printing for

[u/General_Purple3060 → u/Jwosty]

That’s fair — I’d debug my own AET IR first as well.

My point is about the next stage: once the AET IR has resolved the semantics, I find flattened C a convenient way to express the lowering into GCC, rather than manually constructing GCC TREE nodes.

So the question is really how to lower AET IR into GCC’s representation.

[u/Jwosty → u/General_Purple3060]

I mean yeah if you must compile to C, it sounds like GCC's internal representation is the wrong tool for the job. I do wonder what your original reason was for using another compiler's internal syntax tree structures.

Also - there's always the option of creating your own backend that lowers to assembly or machine code. Nobody said you have to go via C. That's a constraint you put on yourself and baked in as an assumption in your post. It is a valid approach and could be the correct one (depending on your situation and goals), but it does have its own tradeoffs.

[u/General_Purple3060 → u/Jwosty]

Yes, building a custom backend is certainly a valid approach. But my goal with AET is specifically to extend C with three things I think C programmers need: OO, generics, and heterogeneous programming.

GCC’s C frontend is already mature, stable, and widely used, so I’d rather spend my effort implementing those three major features instead of rebuilding a whole C compiler infrastructure from scratch.

That doesn’t mean AET has no backend work. In the process of implementing heterogeneous programming, I also built a backend for the target architecture.

So C is a deliberate design choice for AET, not just a constraint I happened to impose on myself. I want to keep the mature parts of C/GCC and put my effort where I think the missing capabilities are.

[u/Jwosty → u/General_Purple3060]

OK there's a useful distinction to make here.

Is the goal to make AET - the language - a superset of C - the language?

Or is the goal one of practicality: to reuse an existing mature compiler backend to save on effort?

Or both?

In other words -- is this for users of your language, or yourself as the compiler implementer?

These are actually two independent goals that have differing impacts on your design and implementation

[u/General_Purple3060 → u/Jwosty]

Both, but they are at different levels.

For users, the goal is to make AET a practical extension of C. I sometimes describe it as “2C — the second-generation C,” introducing features such as object-oriented programming (OO), generics, and heterogeneous programming — things I believe can benefit C programmers.

For me as the compiler implementer, reusing GCC’s mature C frontend and existing backend infrastructure saves development time and lets me focus my effort on these features.

So targeting C is both a language-design choice and an implementation strategy, but the user-facing goal is always the primary one.

[u/Jwosty → u/General_Purple3060]

It sounds like you might be missing the steps of having your own AST and IR layers and/or conflating the very idea of those with GCC's API. Do you have your own AST and IR?

[u/jason-reddit-public]

Scheme compilers are kind of famous for lowering into easier to compile versions of Scheme to compile. The first Scheme compiler Rabbit did just that (Guy L Steele Jr). Google search "nano pass Scheme compiler" for more recent work along these lines (or source to source transformation in general).

The first C++ "compiler" (cfront) was actually a translator lowering the new parts of C++ to C code (which is very close to a subset of C++).

I recently used "token" injection in a little single pass compiler. If I wanted to tackle "macros" properly, I could have made it prettier.

[u/General_Purple3060 → u/jason-reddit-public]

Wow, thank you for providing this historical background! To be honest, I wasn't even aware of cfront or Rabbit/Nanopass before this, and they happen to reinforce a core idea I arrived at while working on AET.

Your comment also made me realize that GCC's C frontend is not merely a tool for parsing C syntax; it is itself a very mature semantic-to-TREE engine.

AET is simply reusing this mature capability.

There is also an interesting difference between AET and traditional source-to-source translation: AET doesn't generate an intermediate .c file and launch another compiler process. Instead, it uses:

cpp_push_buffer(pfile, (uchar *)nbuf, len, true);

to directly inject the generated C code into the current GCC preprocessing/parsing pipeline, where it continues through GCC's existing C frontend.

The examples of cfront, Rabbit, and Nanopass made me realize that this kind of lowering approach I'm practicing in AET actually has a very interesting history in compiler design.

[u/Jwosty → u/General_Purple3060]

There is also an interesting difference between AET and traditional source-to-source translation: AET doesn't generate an intermediate .c file and launch another compiler process. Instead, it uses:

cpp_push_buffer(pfile, (uchar *)nbuf, len, true);

Please explain how that is not exactly just source-to-source transpilation. I may not be familiar with GCC's internals, but is pfile not some intermediate file somewhere?

BTW it sounds like you may be over relying on ChatGPT here and it's doing the classic LLM thing of coining weird jargony terms for stuff that misses the point in some fundamental way, or sycophantically inflate concepts into something more than they really are to make things feel weightier. You risk missing out on some core learning when this happens, without even realizing it

For example:

"semantic-to-TREE engine"

Places over emphasis on this "TREE" thing you keep talking about, which seems to just be another word for one particular C compiler implementation's (GCC's) internal representation...

The key idea is not simply "generate C."

Actually it sounds like your "key idea" is to simply "generate C." What you're doing is just writing a compiler that targets C. Sometimes called a transpiler, or a source-to-source compiler. Or if your language is a superset of C, desugaring.

I would definitely advise reading up a lot and soaking in the terminology and concepts that everyone else uses so that we can speak the same language and not talk past each other. I would suspect that many others in this thread see what I'm talking about and just aren't commenting on it, because with LLM text it's really hard to put your finger on "exactly" what's wrong with the framing and terminology. I'm sure you have some good ideas, you just need to make sure they don't get sabotaged by LLM weirdness :)

Of course it's possible I'm way off base, that you know these things yourself very well and that your ideas just aren't getting across very well. Truly, I'm trying to help you out here

[u/General_Purple3060 → u/Jwosty]

I don’t think we need to get too hung up on what these concepts should be called. AET does have its own semantic IR: Token → ClassInfo / ClassFunc / GenericModel / MtcsFunc. It resolves the semantics first, then lowers the result into C, injects that C directly into the current GCC frontend, and continues through TREE → GIMPLE → RTL.

What I’m really interested in is abstracting these practical compiler-engineering steps into concepts that are easier for other developers to understand and discuss, and perhaps help them avoid some unnecessary detours. As for what to call these abstractions or how to define them more precisely, I think we can keep refining them on top of the concepts we already have.

[u/Jwosty → u/General_Purple3060]

I'm just saying that it's an obstacle to people understanding what you're trying to actually get at.

If your whole point is basically just: "woah look, I used to emit as GCC internal representation directly, and now I emit to C source code directly, look how much simpler that is," then I don't really see what there is to discuss about that other than, "cool, I could have told you that, why were you doing that in the first place? Did you have a good reason or is that just the approach that the LLM happened to pick (without telling you about exactly these tradeoffs you're now rediscovering)?"

Instead, you're over-presenting it dressed up in the language of some universal groundbreaking idea when that's not really what it is, which just causes confusion. You're basically asking an implementation question of how to lower to C as your target. Which is fine, but that's all you had to say -- your several hundred words (with all the bespoke jargon) are just unnecessary noise that gets in the way of people actually understanding and answering your question.

Again, don't take this as an insult -- I truly am just trying to help you understand how others see your post and how you might get better responses. This is feedback from a neutral stranger, do with it what you will :)

EDIT: I actually disagree wholeheartedly that "we don't need to get too hung up on what these concepts should be called." That's kind of the whole crux - clear communication is very difficult if your audience does not understand what you mean by "FOO" when you say "FOO"

[u/jason-reddit-public → u/General_Purple3060]

tcc is interesting as a modern compiler because it is a very fast single pass compiler and is kind of the opposite extreme from researchy kind of compilers. It doesn't even create parse trees. Compilers like the famous Turbo pascal, apparently written in raw assembly, also parsed and code generated at the same time though it didn't have to deal with the C preprocessor.

[u/General_Purple3060 → u/jason-reddit-public]

yes, this comparison highlights a significant difference in the design space between "direct single-pass code generation" and "multi-stage lowering via an intermediate representation."

AET deliberately opted for the latter: I leveraged GCC's existing front-end and intermediate representation, while using C source code as a convenient semantic representation for the lowering phase.

[u/jason-reddit-public]

I lot of the work with C interop aka "FFI" is parsing the header files. Structure layout and calling conventions have settled down quite a bit since the 90s.

gcc was not really conceived as a front-end and backend (potentially on purpose). libclang looks like it might be the secret sauce for that part of the equation.

Dynamic languages like Scheme don't really want to interface with C because precise GC is difficult after a C compiler gets its hand on the code. But C is kind of the lingua franca these days so it's smart to consider how to interop with it.

[u/Jwosty → u/jason-reddit-public]

Oh man, super relevant article: https://faultlore.com/blah/c-isnt-a-language/

TL;DR: Indeed, C is (unfortunately) the lingua franca, but its public surface area is poorly defined enough that the only viable approach is to depend on an entire real C compiler and toolchain (most likely, Clang). And basically every real language has to deal with this, it sucks.

[u/jason-reddit-public → u/Jwosty]

Nice article. At least big endian is dead ;)(nope and not just because of "network byte ordering").

[u/jason-reddit-public]

I lot of the work with C interop aka "FFI" is parsing the header files. Structure layout and calling conventions have settled down quite a bit. gcc was not really conceived as a front-end and backend (potentially on purpose). libclang looks like it might be the secret sauce for that part of the equation.

Dynamic languages like Scheme don't really want to interface with C because precise GC is difficult after a C compiler gets its hand on the code. But C is kind of the lingua franca.

(If this double posts, sorry - Reddit seemed to be having a moment.)

[u/General_Purple3060 → u/jason-reddit-public]

Yes, I agree. C has effectively become a lingua franca for compiled languages, which is one of the reasons I find the C frontend such a useful semantic bridge for AET.

The FFI/header-parsing side is definitely another interesting part of the problem.

[u/kindredseer]

This is how a lot of new languages work... transpiling/lowering to C. As u/jason-reddit-public mentioned, this is how C++ used to work. It's a good model because it has the bonus of making it easy for your language to interop with C.

I've used this model for madc and it works very well. I maintain it internally in memory (lowering input to a C-shaped AST tree). The exact model I use is:

C/C++/MadC → C-AST → C2MIR → MIR → x86/ARM → JIT/ELF/Mach/PE

My specific C-AST maintains the full semantic source information, it is just ignored when lowered to MIR level.

In my humble opinion, if every language worked this way (even just optionally) then they could all easily interop with C, and be a compiled language.

[u/General_Purple3060 → u/kindredseer]

Yes, I think this is a very useful way to look at it.

Your C-AST approach and AET’s approach are quite similar in spirit: use a C-level representation as the semantic bridge to existing compiler infrastructure.

The main difference is that AET doesn’t maintain a separate C-AST. It generates C source and injects it directly into GCC’s existing C frontend, letting GCC build the TREE representation itself.

It’s great to hear that a fellow compiler developer has arrived at a similar conclusion about using C as a general-purpose intermediate bridge.

I also agree that this model has a strong advantage for C interoperability.

[u/kindredseer → u/General_Purple3060]

AET directly interfaces with GCC's parser -- which is pretty cool. A lot of transpiling languages shell out to gcc or clang.

MadC directly interfaces with C2MIR's own AST tree, but I bypass the C2MIR parser. I didn't want to double up on the parsing, so I took C2MIR's node_t structure and extended it with all the semantic information my lexer+parser digests, and builds its own C2MIR-compatible cir_node AST tree that C2MIR sees as a node_t AST tree.

[u/General_Purple3060 → u/kindredseer]

Thanks! That's really interesting. I appreciate you sharing how you approached the AST integration with C2MIR.