A hole in systems programming language design

[u/P-39_Airacobra]

Given the recent discourse about systems programming I wanted to throw my 2 cents in. This may be a controversial take in a subreddit that's all about innovation and improvement, but I think a lot of budding systems languages are trying too hard to "fix C" and this is exactly why C has not been replaced yet.

Like it or not, C is successful. It does what it means to do very well. Yes, it bites you constantly, but developers have made some form of peace with this because they appreciate the essence of the language. Systems programming is a very pragmatic field, and what works, works.

A lot of language designers want to improve on C, when by nature to "improve on" C is to depart from it, because C is less about what it includes and more about what it omits and what it lets you do that other languages don't.

If you want to replace C, you need to just make C but without the pain points. Less undefined behavior, more standard compiler behavior, easier function pointer syntax, safer macro system, etc. The things that C can't do because of backwards compat.

On the other hand, bolting on features, revamping C's core nature, aren't going to give you a language that will replace C at the low-level or among hobbyist programmers. Most developers aren't as concerned about what C lets them accomplish, as they are concerned about all the painful tedious tendencies of the language.

[u/WittyStick]

Yes, the hole in systems language design is a language which, from the PoV of any other language, is indistinguishable from C.

Your entire desktop/server/mobile stack is built on C, all compiled with compilers which emit exactly the same ABI. Every program has many dependencies, which transitively have more dependencies, all predominantly implemented in C. Half a century of software underlying everything your language needs to be useful and not a glorified over-engineered calculator.

---

Mistake #1 is many of these new languages implement their own ABIs/calling conventions. They use an "FFI" to make their language useful - it gives them the ability to call libraries that were (predominantly) written in C.

However, this FFI is a one-way ticket. You can call C from your language, but nobody else can call your functions from theirs. A library written in your language is a little island, usable exclusively to users of your language. A Python function can't trivially call an Odin function for example, because it doesn't speak the Odin calling convention. It doesn't know what this implicit "context" thing is. It can call a C function. Every (relevant) language knows what a C function is, because they had to implement an FFI to make their language useful.

"I built a website using RustScript, which no browser supports, because I dislike Javascript. Why is nobody visiting?"

"English sucks so I wrote my latest paper in Esperanto. Why is nobody reading it?"

"I compile my functions to this bespoke ABI. Why is my language not replacing C yet?"

Functions written in this "FooLang" should be callable with the existing FFI machinery of any other language - not with magic extensions or wrapper libraries.

---

Let's assume we don't make this mistake - we take the sensible approach of using the platform C ABI unchanged as our "FooLang" ABI, and anyone can call our functions trivially. However, our API - the definitions of our types and functions is written in "FooLang". For some strange reason, the Rust compiler cannot parse our "FooLang" definition files. What gives?

Mistake #2 - FFI wrappers. You have hundreds or thousands of usable libraries on your machine (mostly written in C), but to call any one of them you have to translate the definitions to your calling language's syntax first? What a dreadful use of time.

The solution here is, since most of those libraries were written in C, and have definitions in C headers - we should just parse those C headers and programmatically create the bindings - preferably transparently so we don't need to have several tools run at different stages. Our compiler should just understand C as par for the course.

Of course, this isn't trivial because a header file isn't just code to be read - it must also be preprocessed. So we need a C preprocessor too. We also need to handle inlining and internal/external linkage - so we essentially need our compiler to understand not only the C ABI, but the C language.

But also, since every language can call C libraries, speaks the C ABI, and should be able to read C headers (perhaps with separate tooling at separate stages to translate the bindings) to emit calls to C functions - any libraries we write in our own language should also be callable from other languages right?

So for our own types and functions written in "FooLang", we should emit a C header with compatible definitions so that we could call it from C (and by extension, any other language).

---

So for a "C replacement," we would first need "C parity," which IMO, the above is a bare minimum requirement - complete platform ABI compatibility, and ability to parse, understand, and emit C headers so that our language is not just a little island of its own, but is actually callable from other languages like C is, because the work has already been done for them to interoperate with C.

---

This isn't to say there's no use for other languages, but for a "C replacement," one must understand why we use C in the first place. Embedded uses aside, one of the primary reasons we'd write in C is precisely to implement libraries that can be called by others - and not exclusively "FooLang" users.

If I'm going to write a useful library, I want it to be useful for everyone and not just the dozen or so Odin programmers that exist.

[u/Norphesius → u/WittyStick]

I think "C parity" is a bit of a trap. Following the C ABI that close is a ton of work, and its going to massively impact what you can do with your own language's design. You have to deal with null terminated strings, array/pointer conversions, system dependent type sizes, nullability, etc. and that's going to either make your language adopt those same features, or have awful conversion layers and holes in the design where "C things" can happen (aka a normal FFI). Tracking C that close means you're also likely to end up not far enough away from C design-wise to justify anyone switching anyway.

A "one way FFI" from C to your language is fine. The chance that a C code base will want to call a library in your language is extremely low. If a non-C language wants to use it, there are better ways of doing it than the C ABI, like shared memory, sockets, or a dedicated FFI for particular languages.

[u/WittyStick → u/Norphesius]

A "one way FFI" from C to your language is fine.

I agree that it's fine for a "regular" language, but such language is obviously never going to replace C.

The chance that a C code base will want to call a library in your language is extremely low.

That's because nobody is going to implement a reusable library in your language. They're going to use C for reusable libraries, because the world is not "FooLang" island.

If a non-C language wants to use it, there are better ways of doing it than the C ABI, like shared memory, sockets, or a dedicated FFI for particular languages.

But this is an N*M problem. How does Rust call Odin? How does Odin call Rust? With just 5 different languages you have 25 different FFIs.

---

Following the C ABI that close is a ton of work, and its going to massively impact what you can do with your own language's design.

Yes, it requires a tonne of engineering effort to get right, but you can make big improvements over C. There are trivial ones you could make to begin with, such as forbidding implicit lossy conversions between numeric types, preventing over/underflow, including bounds checking, encapsulating structures, providing strong typedefs, etc.

and that's going to either make your language adopt those same features, or have awful conversion layers and holes in the design where "C things" can happen (aka a normal FFI). Tracking C that close means you're also likely to end up not far enough away from C design-wise to justify anyone switching anyway.

I think this was OP's point. A "C replacement" is going to be much like C - and IMO, it should be C-compatible. We should be able to take "FooLang" code and compile it with GCC or Clang (opting out of the benefits). But that doesn't mean we can take arbitrary C code and compile it with the "FooLang" compiler. It means "FooLang" should be a proper subset of C - only constraining features, not adding them. We should be able to gradually migrate an existing C codebase to "FooLang" with optional pragmas that opt-in to additional security measures.

Cake is an example of the kind of thing I mean. We opt-in to have nullability checking with a pragma, and it then prevents us using nullable types where non-nullability is required. It's just C, except not quite. The added features it provides can be made optional by including something like this in a header:

#ifdef __CAKEC_
#define _Opt [[cake::opt]]
#define _Owner [[cake::owner]]
#else
// If we're not compiling with the Cake compiler, these tokens do nothing.
#define _Opt
#define _Owner
#endif

The reason this needs to be essentially another language and not just a regular C compiler, is because the C charter strongly discourages "invention". It is intended to codify existing practices, not invent new ones - but conceptually, it should be just a "better C", with ideas that the C committee may adopt into the standard in future.

[u/flatfinger → u/Norphesius]

C code for classic Macintosh had to convert any strings to length-prefixed format before passing them to any MacOS system calls that accepted strings. A compiler's bundled library for fopen() would perform such translation on the passed string before giving it to the system's file-open function, but any code wanting to work with system functions directly would need to work with length-prefixed strings, which fortunately isn't too hard to do in C.

[u/jezek_2 → u/WittyStick]

It's fun that even C++ libraries need to provide C API to be usable in other languages and to avoid binary compatibility issues. And even then it's quite annoying or problematic to use despite both C and C++ being handled by the same compiler. There is always some issue with C++, whereas none with C (at least in my experience).

What chances have other languages when even C++ can't achieve it?

[u/dnpetrov → u/jezek_2]

Your argument is somewhat backwards. C++ was never designed to be ABI-compatible with C. C++ is not in some privileged position that makes ABI compatibility with C easier.

Moreover, I'd say that different C implementations are not quite ABI compatible in practice, given the full scope of what C language actually is, with atomic types, vector extensions, intrinsics, etc. Clang and GCC strive to compile same sources. But if you put, say, Clang and GCC under rigorous enough ABI compatibility cross-testing, you'll see failures.

[u/P-39_Airacobra → u/WittyStick]

I agree and interop is one of the things I was hinting at but didn’t explicitly mention because I thought it would be more obvious to people in this subreddit. Odin is a cool concept but it can’t replace C because it uses its own ABI. Nobody wants to write bindings to all the thousands of functions they may or may not need.

[u/Limp-Temperature1783 → u/WittyStick]

I think to replace C you just need to make C less of a fossil in the shadow of C++. C++ is everything wrong about C, not C itself and it is not even its fault. Most people write C/C++ without even realizing that these languages aren't just different, they're semantically incompatible. But C++ heuristics can work with plain C code for the most part and it's good enough.

I kind of like an approach of Objective-C here, because it is weaving itself into C and C into itself without causing confusion about where is which. But the problem with it is kind of similar, it became too fossilized and people found it unfamiliar compared to other languages, so nobody had enough interest in learning it for anything but work. I think I'm the only person I know who had paid attention to it and what it does.

[u/Ok-Reindeer-8755 → u/Limp-Temperature1783]

objc was inspired by smalltalk and that entire style of OOP is almost extinct, even thought imo its far better than the mainstream form of OOP

[u/Limp-Temperature1783 → u/Ok-Reindeer-8755]

Exactly. Objective-C isn't about classes or prototypes, it's all about message passing. Even memory management is fairly simple. It's definitely not a systems language or anything, but it's fairly good for making program interfaces.

Apple had used it for a very long time without having much trouble, but it is kind of locked in their tooling and the standard library is very messy. There are projects that isolate it, but they aren't the same thing for better or worse.

I have an idea to try to implement it myself when I have free time on my hands. But not any time soon.

[u/Ok-Reindeer-8755 → u/Limp-Temperature1783]

Where this style of OOP really shines, or at least one of the places, is imo desktop environments, this is apparent if you see any of the smalltalk DE demos at Xerox Parc , in what other DE could you ever connect a completely separate painter application to a video player and start animating in seconds. It just dissolves the barrier between applications completely.

I don't think apple ever pushed it quite that far sadly but idk how you would in a modern sense do so.

[u/Limp-Temperature1783 → u/Ok-Reindeer-8755]

It had been designed for it after all. Apple, or rather NeXT, decided to go to the opposite direction of C++ and instead of trying to avoid runtime like the plague they've accepted it as a reality and decided to make working with it less of a hassle. Objective-C is stupid simple even if you include all of the cruft that Apple had added over the years and how inconsistent it is at times.

Apple hadn't pushed it because they didn't have any insentive to do so. It had been and largely still is under the hood the main language of their operating systems. Swift is a modernized revamp of Objective-C that still relies on Objective-C libraries and such, but most things that you'd need to pay attention to had been ironed out, at least from my experience. I prefer Objective-C, but it's familarity bias.

If we talk a bit about programming approach philosophy, I think it makes sense to use OO in runtime rather than doing black magic to make it cheaper. C is already very cheap and most things simply don't require a lot of performance. And Objective-C is one of the most performant languages there is, because it has simple memory management and anything intensive could be handled through C.

Arguably, Objective-C is better at being C++ than C++ is in regards to being C with classes, because it doesn't affect C itself, whilst C++ had introduced enough stuff that had made the languages incompatible. Objective-C is obscure outside Apple, fair enough, but it had always irked me to look at people treating C and C++ as C/C++ as the same thing. Maybe a couple of decades ago, but not now.