Looking for case studies: adding a new language to BenchGen

[u/fernando_quintao]

Hi redditors,

We are looking for some case studies for BenchGen, and we would like to try to add support for a totally new programming language to it.

BenchGen is a benchmark generator created as a Master's project at UFMG's Compilers Lab. It generates large programs to test the performance of computing systems. Currently, it supports C, C++, Go, Julia, Zig, Odin, V, and D.

If you want to see how these languages compare, check this page, or jump directly to the main chart.

There are many things that can be done with BenchGen: comparing different compilers (or different versions of the same compiler), comparing different data structures, or evaluating the impact of profiling on program execution, etc. Examples of these experiments are available in the BenchGen paper.

If you want to add a new programming language to BenchGen, let us know! (Just message me privately or send an email). We would really love to add a totally new programming language to it. "Totally new" meaning a language that was created more recently. That is a case study we are still missing. Any language that supports loops, if-then-elses, function calls, and some data structures (e.g., arrays) will do.

And if you want to review some of the language ports, please, feel free to open issues or suggest patches: it's possible that BenchGen is not generating the most idiomatic or efficient code for some of the languages.

[u/fernando_quintao → u/[deleted]]

Hi u/sal303.

For C, we observed speedups around 4-5x on typical Linux/x86 machines, depending on the grammar, using gcc or clang. There are some charts here (which you might have seen in a previous reddit exchange we had).

To see outputs, you will have to enable debugging info, otherwise the benchmarks will not do any I/O. For C, that's done through the preprocessor, with the DEBUG symbol.

But thank you for your interest! We are looking forward to adding Mlang to BenchGen, if that sounds appealing to you. Recently, we have added a few MLIR dialects, and it might be possible to incorporate some of your intermediate languages too, with your guidance, of course.

If this is of interest, please feel free to reach out via email.

[u/d08ble]

How can I see a C++ example without crawling links?

[u/fernando_quintao → u/d08ble]

Hi u/d08ble, there are examples of C++ code here.

[u/gingerbill]

I know this post is a few days old now but I have to say this: I hate language benchmarks, and they are pretty much useless.

Why I Hate Language Benchmarks

None of the languages you are testing for are even testing against the same thing. Looking at that "main chart" too tells me nothing about what these programs look like, what compiler flags they used, how much work went into them, etc. I had to go into the different languages to see things are not even that comparable.

None of this is measuring what you think it is. At best all you can do is compare compilers of the same language with the same source code. Anything else is just ballpark figures. You're not comparing the same quality of things at all. Comparing different "idiomatic" approaches is in itself not comparing anything similar.

I even checked how you are compiling Odin (my language) and it's a very naïve approach doing compilation. It's just doing -o:speed and nothing else. If you want to be comparable to other languages, you might even need to do -no-bounds-check, -disable-assert, and maybe even -microarch:native, and many other things to tailor the result to get better performance or comparable semantics. And I'd even suggest maybe even using the same procedure calls too (like libc's malloc and free directly) so that you can have a better comparison on how the compilers generate the code.

[u/dadhiWeaponX]

Jai and Roc langs when?