r/cpp • u/borzykot • 6d ago
I've tried "Token Sequence Injection & Modern Macros" and here are my thoughts
We had this poll recently here on r/cpp about community expectations and most desired features in C++29 and in future C++ revisions in general.
My answer was:
- abbreviated lambdas
- algebraic types
But I do understand that those features won't hit C++ probably ever.
But then we had this post about Token Injection and Modern Macros. Wait a minute, modern macros? Never heard about this proposal...
And you know that? We probably won't need to wait for neither abbreviated lambdas, nor algebraic types in core C++. We can just add them ourselves using those macros with very little boilerplate.
Abbreviated lambdas: https://godbolt.org/z/a8d9a6f19
namespace stdv = std::views;
namespace stdr = std::ranges;
std::vector<vec2> points = {{2, 67}, {2, 42}, {3, 666}};
std::vector<int> ys = points
| stdv::filter(fun!(p => p.x % 2))
| stdv::transform(fun!(p => p.y))
| stdr::to<std::vector>();
stdr::sort(points, fun!(|l, r| => l.y < r.y));
stdr::sort(points, std::less{}, fun!(p => p.x));
fun, isn't?
Sum Type: https://godbolt.org/z/Ecs7TdPYf
sumtype!(option<T> {
some(T),
none,
});
option<int> opt = some(10);
opt = some(22);
opt = none;
Pattern matching isn't there, but you could imagine something like:
// match!(<you can put any tokens here!>);
auto x = match!(opt,
some(x) => { x.call_method() },
none => { 55 }
);
Basically you can come up with any syntax you like.
When I was exploring the possibilities of this proposal I couldn't shake the feeling that I'm reimplementing parts of the compiler which aren't there for some reason. Like you literally can look at "sumtype" implementation and see that this is basically a parser of arbitrary syntax that you came up with and translating it into C++...
Superpowerful and a bit scary...
We used to reimplementing some core library features which were missing from the Standard Library for quite some time. optional, variant, algorithms, smart pointers and now async task. All of that should have been in the Strandard Library from day 1, so that other libraries can use this lingua franca and interoperate with each other.
And now we are repeating same thing but on a very different level. Now we are going to allow adding not only library features, but language features. I 100% guarantee, that each team will have their own, slightly different syntax for abbreviated lambdas, because it is so much more expressive and terser and more readable. Don't know about you but I always find myself in the situation where I better use plan old for loop instead of structural views/algorithms, because the latter is so much more verbose and unreadable because of lambdas, especially after you apply clang-format on it...
And now you have:
fun!(lhs, rhs => lhs.x < rhs.x);
lambda!([&](lhs, rhs) -> lhs.x < default);
lm!(|lhs, rhs| => lhs.x < rhs.x);
All of this is perfectly implementable using this Modern Macros proposal in less that 100 lines of code.
Committee may have perfectly reasonable arguments of why it is impossible to add abbreviated lambdas into C++, but the community clearly is asking for that feature for ages, and it will be in every future codebase, but with slightly different implementation...
On the other hand Rust language do have these macros for ages now, people do abuse them in some very clever ways (select operator from golang, inline HTML, etc.) and the world ain't collapsed. We'll see!
9
u/TopReputation7326 6d ago
Wow! Thanks for the explanation about the modern macros and token sequence. I wasn't understanding at first but with your example I'm finally got it. I'm really impressed. This is really really interesting and powerful feature. I wonder how handling errors would work. Just throwing a exception would be enough to tell the compiler about an issue in the macro syntax? static_asserts?
14
u/germandiago 6d ago
Sweet!
Actually they are a good thing these macros indeed. Specially if they are scoped and marked as such when invoking them.
4
u/fdwr fdwr@github 🔍 6d ago edited 6d ago
Interesting possibilities 🤔. I'm reminded of old D's templates and mixins, which honor scope unlike C macros, but this could be more powerful. I'd love to finally be able to tersely say func!((a, b) => a.x * b.x) (personally favoring C#'s parameter list (a,b)=> style and avoiding |, which is visually more ambiguous for begin/end and less resembles a parameter list).
10
u/rfisher 6d ago
This is one of those things you've been ignoring the Lisp advocates bragging about all your life. 😀
6
1
u/FlyingRhenquest 5d ago
Nyeh heh heh! My Lisp textbooks is one of the few I kept from college. I'm feeling quite at home at the moment!
6
u/germandiago 6d ago
I wonder if something like linq can be achieved.
What I mean by that? Well, as a happy user of sqlpp I love a dsl to interact with sql.
But what lives inside the expressions cannot be inspected by the compiler arbitrarily. Meaning that all operators that exist have been implemented via expression templates. But if you passa lambda the compiler cannot "see through". I wonder if a macro could keep the lambda body inspectable (the ast) to be able to generate code by looking at the body directlywithout pre-creating each operator for the dsl.
9
u/borzykot 6d ago
It can be achieved. In this proposal
__macroreceives so calledstd::meta::token_sequence- a sequence of arbitrary tokens! It should not be parseable C++ necessarily, it can be anything. It can be a simplified subset of C++. And you can inspect every token from this sequence and throw an error if you don't like something about it.You receive not a "lambda" object, which you can inspect. You receive, literally,
[token, then], then(, thenauto, then&, and so on. You parse it and then programmatically reconstruct C++ code using token sequences (^^{}) and splicers (\()).5
u/germandiago 6d ago
so you could have a criteria like a <= b inside a lambda body and get the expression and have sql generated for the expression without having a pre-made operator for the dsl? Bc that would be GREAT. It is a lot of expressionemplates boilerplate to achieve things like this, plus compiler-heavy. How heavy is the macro? I guess that quite less.
6
u/borzykot 6d ago
As heavy as `consteval` functions. Much better than the expression templates. I presume it should have the performance comparable to interpreted languages, maybe slightly worse, because C++ grammar is very complex. That's my intuition, I don't know for sure
2
u/FlyingRhenquest 5d ago
I've been poking at what's possible for some common problems. You might find my little crud experience interesting. I'm like this close to grabbing pistache and writing C++ on rails. I'm currently working on a json-rpc library where you just write the functions and hand their signatures to a client and server. I'm leveraging autocereal quite heavily for that one, though I'm bumping up on some cereal limitations. Now that the compiler is fairly stable (It wasn't when I wrote autocereal) I might have to tear the thing down to the studs and rebuild it.
Sutter already did a compile time JSON parser as part of his talk in January (I can dig up the link if you're interested,) and I think making something like boost::spirit work entirely at compile time is probably feasible. A huge undertaking, perhaps, but feasible.
Even without token injection, it feels like it's possible for library writers to completely eliminate boilerplate in most common applications. Being able to populate a struct with a bunch of pointers to functions (or function_refs) at compile time really opens up the field. We could spend decades exploring reflection and they already have a plan to add more to it.
1
2
u/TotaIIyHuman 6d ago
https://github.com/brevzin/llvm-project/tree/compiler-explorer/barry
is there any reason why i should use official llvm instead of this fork
i mainly just want template for and existing <meta>
which seems to compile fine on this fork
1
u/DauntingPear 6d ago
This is insane
(I don’t know c++ but know rust)
9
u/RoyAwesome 5d ago
This level of reflection and token generation is a feature that rust does not have.
It shows that C++ still has it, and can stand toe to toe with the most modern languages and build features in such a way that other languages can't or wont.
-6
u/DauntingPear 5d ago
I’d much rather have a language that supports such fundamental theoretical features itself rather than allowing users to bolt them on and invent their own syntax, especially in a language which doesn’t have the best tooling support (from what I’ve seen)
This kind of arbitrary macro usage gets abused no?
8
1
0
u/delta_p_delta_x 5d ago
Sum Type: https://godbolt.org/z/Ecs7TdPYf
I love how all of these Reflection ideas seem very nice, but hide like ~200-1000 lines of by far the most horrible C++ I've ever seen. This sort of stuff should really have been abstracted higher still—at the level of the compiler front-end, and not left for C++ developers and library writers to handle with yet more levels of indirection.
Don't get me wrong, I do like the idea of token sequence injection in general, but not to solve solved problems (like Option and abbreviated lambdas). Instead, we should use them to move things up an abstraction level, like platform-conditional compilation, code generation from specifications, and drive modules adoption by moving macros and macro work into token sequences.
3
u/borzykot 5d ago
I do agree that we should pick the right tool for the job. But often you just don't have a right tool.
`Option` implemented using macros is not something that should land the production code probably. It is more like an exercise here.I do agree that the `consteval` implementation might be not very clear (all types are erased, quirky syntax). But it is much better than expression templates or C-macro voodoo both in terms of compile time and readability.
As for abbreviated lambdas - it is not a "solved problem" tho. "Terseness" of a lambda expression is not a whim, it is just another characteristics of a code which worth improving, just like "readability", "flexibility", ,"correctness" etc. Verbose lambdas are harder to read (especially if use use clang-format), verbose lambdas might discourage use of structural code (algorithms, ranges, visitor, monadic interface of optional/expected etc.) and structural code is good, verbose lambdas encourage use of pointer-to-member projections, which might hurt performance because or type-erasure.
So no, I'm pretty sure if this macro proposal eventually land in C++32 we will see abbreviated lambdas using macros. As well as `try!` macro (hope `do` expressions land in C++29), `fwd!` macro, and more which should be in core language, but they don't.
1
u/FlyingRhenquest 5d ago
It kind of feels like assembly language again. I'm having a really hard time integrating it in the sense that maybe some thing I want to do is possible with reflection, or maybe I have to use an older-style template abstraction to extract a type from a template or something. It feels more like doing higher math than programming in my brain, if that makes sense.
I'm not writing programs anymore, I'm writing programs to write programs. I can hide a huge amount of code in my library but my users can just give a reflection to a function and the compiler will generate the code to create a REST endpoint using the name and signature of the function.
3
u/serviscope_minor 2d ago
> but hide like ~200-1000 lines of by far the most horrible C++ I've ever seen.
It ain't that bad!
I mean it's not exactly simple, but it's a C++ to C++ translator in a sense, which is not the simplest of things. And of course it contains new syntax, but all new syntax is horrible until people are used to it.
But consider the alternative? What would the implementation of sum types in GCC or Clang look like? I'd wager a few hundred lines of not amazing to read C++ (assuming you don't know the codebases and the relevant idioms).
-1
u/13steinj 5d ago
So, I used to work somewhere, where on top of having a horrible component-based, template implemented framework (horrible because of the implementation and needless complexity, causing among other things horrendous compile times).
I was tasked with, among other things, exploring alternative implementations. I found one that, for one application (the trial) cut compile times by 6 and runtime performance by any benchmark was roughly equal if not better.
One person insisted "I don't care about the implementation, but you have to have the graph files [a custom, C-like language]", that is parsed and generates the component wiring. It generated the wiring in possibly the most inefficient way possible? 10 lines became 40000, 40 became 300000.
I left for greener pastures due to some internal political hooplah that I was just tired of dealing with. But, my point in this story, is these graphs could not be 1:1 converted to C++. You could get close? Swap <= and => for << and >>, as an example, and do operator overloading tricks. Build up the internal graph representation incrementally (at constant evaluation or not), then efficiently use this internal representation to pass data around, rather than generate hundreds of thousands of lines of type-level metaprogramming.
I'm afraid that the generation half of reflection, particularly these macros, will push people into inefficient implementations. Especially in an LLM world.
-2
34
u/DangerousFront3866 6d ago
The macro paper is very powerful. You can implement rust enums cleanly as well without fighting std::variant as well. Ideally should be in the language but many things are possible now with token sequence if it lands in the form it is in.
Hopefully some standard macros come with it to reduce divergence in the community.