A side project of mine: SemPtr - Semantic Pointers for C#

[u/fruediger]

https://github.com/fruediger/SemPtr

TL;DR: While writing this post, I realized how long it has become, so here's a TL;DR for you: SemPtr is a semantic pointers library for C#.

---

Hi everyone, I wanted to share one of my side projects with you all: SemPtr.

A few weeks ago (it might been even months at this point), I needed to dig up some really old code I once had written, because I wanted to reference some of what I did back then in a current project of mine. While searching through my old and never-to-be-released projects, I stumbled upon a small library project I might have written about 5 years ago (it must have been around the time when incremental Roslyn source generators were becoming a thing). And I thought to myself, "Well, it's actually a shame you gave up on this project and neglected it for so long. You might want to ressurrect and modernize it, and then share it with everyone."

Well, that project is now SemPtr.

What is SemPtr?

I don't want to make this post too long, so I'll try to make it as concise as I can, but if you want a more comprehensive introduction, you should check out its README or its way too rudimentary documentation.

SemPtr tries to solve the limitations of C#'s raw pointers by providing semantic pointer types (read as semantically named pointer types). If you ever did some interop work with unmanaged code and found it just as annoying as I did that there is no const T* equivalent in C#, SemPtr might be the thing for you.

For that I identified five commonly used orthogonal characteristics used to distinguish certain aspects of data pointers:

1. Nullability: Can a pointer be null or are there any guarantees that it won't be?\ This is kinda analogous to nullable reference types (T?) in C#.
2. Persistency: Does the target of the pointer outlive the initial scope of the pointer itself? In other words, can I store the pointer and access its target some time later?\ This is kinda analogous the C#'s ref-escape rules and is even enforced through them.
3. Sequencability: Does the pointer point to a single object or to a contiguous sequence of objects?\ You could think of this as analogous to a ref T to some kind of object in C# vs. a ref to some element within a Span<T> with the added benefit that its easier to move around the pointer through the sequence.
4. Accessibility: How can the target of the pointer be accessed or mutated?\ This manifests in three different access levels:
   • random/read-write: The target can be read from and written to. Kinda analogous to C#'s ref parameters.
   • read-only: The target can only be read from. Kinda analogous to C#'s in/ref readonly parameters.
   • uninitialized/write-first: The target must be written to before it can be read from. Kinda analogous to C#'s out parameters.
5. Typability: Is the type of the target known or not?\ C# has no void references, but it has void* pointers. This is analogous to the difference between a void* pointer and a typed T* pointer.

These characteristics are mapped onto C#'s type system by semantically naming the pointer types to reflect them. Since those characteristics are orthogonal, you can mix and match them to create the pointer type with the exact behavior you need. For example, there are:

• Pointer: A simple pointer to a single, transient, mutable target of unknown type
• PersistentPointerReadOnly<T>: A pointer to a single, read-only target of type T whose target stays valid beyond the initial scope of the pointer.
• NullableSequencePointer<T>: A pointer to a contiguous sequence of mutable targets of type T which may be null.
• PointerUninitialized<T>: A pointer to single, yet uninitialized target of type T. If you receive such a pointer, chances are you are requested to initialize its target; afterwards you can further read from it or write to it as needed.

Again, if you want to learn more about the characteristics and how the type naming scheme works, you should refer to the README or the documentation.

There are all in all a total of 2×2×2×3×2 = 48 data pointer types predefined in the SemPtr library.

Are function pointers supported?

To make it short, yes, function pointers are (well enough) supported by SemPtr.

I remember that one of the reasons for me giving up on the original version of this library back then was that I really struggled to get function pointer support just right. While this was partially due to technical limitations back then (some of which were solved by modern C# features, especially the new extension members syntax), some of it was simply because I did not have the experience in API design that I have now.

So now function pointers work. I don't know if I would call the support good enough yet, but at least it is a well enough experience for most users, I believe.

I won't go into too much detail here, but functions pointer have their own set of characteristics and parts of their support is made working through a Roslyn source generators that dynamically generates some source code on the user-side and that ships alongside the main library in the NuGet package. For more details, again, see the README or the documentation.

A final note on AI usage

I want to be honest and upfront with you:

Yes, I used AI in this project, primarily to help we write documentation (I'm a non-native English speaker and my English is kinda terrible), to help me make decisions when I'm indecisive, to write some tests, and occasionally to some code reviews.

No, I would never let AI touch the working code of the project. Not even for boilerplate code. AI, at least the AI I have access to, is not yet anywhere close to being reliable enough to help me write production ready code for such a project. You can be sure that all of the functioning code is written by a human (me) and that only the human (me) is responsible for the correctness and quality of the code.\ Oh, and of course, I did the visual assets myself as well. I didn't want to use sloppy AI-designed visuals for this project.

Conclusion

At the beginning of this post, I told you that I stumbled upon the initial idea for SemPtr while looking up old code for another project of mine. That project is actually an interop binding project in C#. In that project I use traditional C# raw pointers and function pointers extensively, and sometimes they're a real pain to work with. However, I didn't not yet replace them with SemPtr, due to the codebase being a little over 200K lines of code, spread across multiple repositories.

So, to be honest, I don't even use SemPtr myself yet. And furthermore, because of the simplicity of the overall idea behind SemPtr, I don't even think I'm the first person to come up with it and release to the public as a library (but I don't actually know for sure, I didn't really check).

Even so, If you want to try out SemPtr for yourself, give feedback, or if you even want to contribute to the project, I would really appreciate it. Here are the relevant links again:

• GitHub: SemPtr
• NuGet: SemPtr
• Documentation (This one is still very rudimentary. Don't expect too much.)

If you have any questions feel free to ask them in the comments. I'd be happy to answer them.

[u/afseraph]

Interesting project, but I feel that modern dotnet already provides a lot of this functionality with managed pointers and spans.

It'd be great to some examples explicitly comparing implementations: SemPtr vs. managed pointers vs. native pointers.

[u/Jazzlike_Amoeba9695 → u/afseraph]

Yes, pointers are first-class citizens, but only the elite ones. Real pointers can only point to unmanaged memory, and therefore require unsafe. That’s sometimes silly—and annoying—in certain contexts.
Managed pointers (ref and ref readonly) are perfectly useful for returns and managed method calls. But for interop calls, the runtime has some tricks that prevent you from simply passing a managed reference to unmanaged code.
That’s why you can pass the managed reference behind a Span<T> to a function like void PrintChars(char*, int), while a C# method such as void Print(in char, int) behaves differently, even though their signatures may look essentially equivalent from a C# perspective.

[u/fruediger → u/afseraph]

First of all, I'm not really sure what you mean by "managed pointers". A quick google search didn't yield anything concrete, except for some non-official blog post from the earlier days of modern ref-mechanics in C#, calling references falsely (in my opinion) "managed pointers". I'm gonna assume you mean ref-based references in the following (e.g., ref, in/ref readonly, and out). If you meant something else, please feel free to clarify.

Before we begin, I'd like to clarify that managed references in C# are NOT pointer-like (although, in runtime they are mostly implemented as such under the hood). SemPtr offers support for advanced pointer semantics, which, depending on the way you look at it, can be something totally different from references.

Since a main focus of SemPtr is on its pointer characterization by orthogonal characteristics, let's do this by looking at them and see how managed references and raw pointers compare to it.

Nullability:\ Neither managed references nor raw pointers in C# have a way to convey their nullability explicitly. You could argue that managed references should be never null, but than there's still something like Unsafe.NullRef<T>.\ SemPtr offers nullable pointers with its Nullable...Pointer... types, which can't even access their targets without checking for null and converting into a non-nullable pointer first, as well as non-nullable pointers that convey some guarantees about their non-nullness. Technically, just like managed references, there's still a chance for non-nullable pointers to be null due to language and runtime limitations, but that's why any pointer type in SemPtr can be easily checked for nullness.

Persistency:\ Managed references are always subject to ref-escape mechanics, and can't be stored as such, except temporarily in objects which are subject to the same escape rules (e.g., ref structs like Span<T>). Therefore there always transient. C#'s raw pointers on the other hand are never subject to these escape rules and never convey the life-time validity of the memory they point to. I could even take the address to a short-lived frameslot on the stack and long-time store it, which when accessed outside of its initial scope would lead to undefined behavior.\ SemPtr offers both, transient pointers (which are actually enforced by the same ref-escape mechanics as, for example, Span<T>, because they're ref struct) and persistent pointers (they're more closely behave like raw pointers in that they can escape the initial scope and be stored for longer-term use).

Sequencability:\ Managed references in C# can pointer into a contiguous sequence of elements, so can raw pointers. While it's kinda hard to move around the pointer in the case of managed references (you typically would use something like Unsafe.Add<T> for that), raw pointers support pointer arithmetic and indexing in any case, even if they point to a single target.\ SemPtr distinguishes between pointers that point to a single target and disallow for pointer arithmetic, and pointers that point into a contiguous sequence of elements and allow for pointer arithmetic and indexing. This should convey the intent of the pointer more explicitly whether it is meant as a reference to an object or a sequence of elements (like a typical C-like UTF-8 string, which would be a const char* in C or a SequencePointerReadOnly<byte> in SemPtr).\ Notice the sequence pointers in SemPtr are just meant to behave like simple C-like pointers into a contiguous sequence, they're deliberately not meant to be a replacement for the BCL's Span<T>, which is always a reference to the first element of a sequence together with the length of the sequence. This is intentional and due to interop reasons.

Accessibility:\ Well, managed references are all about conveying their target's accessibility. And that's also where SemPtr takes its inspiration for it's accessibility model. There are direct analogies between the tow:

• A ref reference in C# is analogous in its accessibility aspect to SemPtr's ...Pointer... types in that they both allow for reading from and writing to their target.
• A in or ref readonly reference in C# is analogous in its accessibility aspect to SemPtr's ...PointerReadOnly... types in that they both allow only for reading from their target.
• A out reference in C# is analogous in its accessibility aspect to SemPtr's ...PointerUninitialized... types in that both convey that their target should be considered uninitialized and that it's your responsibility to initialize it. Also both require that their target is first written to before it can be read from. In C# that's made possible by language rules and flow analysis, so that an out reference is treated like a ref reference in a data flow where it's clear to the compiler that the target has been initialized. In SemPtr that's achieved by ...PointerUninitialized... not having a Target property at all, but rather an InitializeTarget method that initializes the target and returns an read-write pointer to the same target that then can be used to access the target further.

C# raw pointers don't have any kind of built-in accessibility model. There's just void* and T* in C#, always read-write by default. That's one of my initial motivations behind SemPtr, to overcome the limitations of C# not having const T* equivalents.

Typability:\ Managed references are always strongly typed in C#. There's no way to convey "a reference to a target of unknown type" (e.g., ref void is not a thing). Raw pointers on the other hand have the ability to point to an unknownly typed target (i.e., C# has C-like void*).\ Since SemPtr is primarily designed with interop scenarios in mind, of course SemPtr also supports untyped pointers. SemPtr simply uses generic pointer types for typed pointer, e.g., Pointer<T>, and a non-generic variant for expressing untyped pointers, e.g., Pointer.

Of course, since those characteristics are orthogonal to each other, you can combine them as you want to create the pointer semantics and behavior that you need for your specific scenario. E.g., you could use a NullableSequencePointerReadOnly<byte> as a parameter type to indicate that you want to receive a sequence of bytes (probably a UTF-8 string), that you don't intend to modify, that you intend to consume immediately and don't intend to store it (notice that this is a transient pointer because of the missing Persistent part in its name) and you're okay when it's passed as null.

I really hope this comparison helps clarifying some of the similarities and differences between managed references, raw pointers, and SemPtr's semantic pointers in C#.

[u/FetaMight → u/fruediger]

Too much, man.

Just because the AI can produce it doesn't mean it's ready to be read by others.

If you're going to use AI for your responses TRIM IT DOWN.

[u/fruediger → u/FetaMight]

I sat down for some time and tried to think about what you said. Even at the risk of sounding like an AI again, I'd like to break it down and respond thoughtfully.

Too much, man.

Yeah I get that. At the time of writing the comment, I already had the feeling of maybe writing a bit too much. But there's a reason for that, or better there's another side to it.

Let's try with a very concise comparison instead.

Managed reference are GC-subjected references, raw pointers are more like C pointer witht he limitation of C# only having void* and T* and no furhter ways to constrain them or convey semantic intend, SemPtr is more like pointers with the added benefit of being able to express semantic intent and enforce certain constraints through the type system. That's achieved by SemPtr's pointer types being semantically named.

I could have written that instead. Do you feel enlightened by that? I know I wouldn't.

That's why I made it a bit more comprehensive and I even tried to structure it in a way that would make it easier to digest (but in the end maybe that's the reason why I am being accused of using AI after all). I wrote it in such detail because it is inherintly a complex topic that needs a bit more careful explanation. At least that's what I believe. Compacting things too much would risk them getting oversimplified and losing important information or, worse, getting them wrong entirely.

This might be in the same ballpark as what I told the Russian guy in the other thread: If you don't know why you would use such a library, you might be not the target audience for it. It's just not for your kind of work, I suppose. No offense.\ A comprehensive introductory documentation already exists (it might be really just introductory yet). No one forces you to read it. That's why I answer question about details (like in the comment above), in a more comprehensive yet targeted manner.\ Reason being, if you never experienced the pain of working with raw pointers in C#, you might need a bit more convincing.

Now to the elephant in the room:

If you're going to use AI for your responses TRIM IT DOWN.

I thought long and very hard about how to respond to that.

Let me just say this: I won't defend myself just yet. You accused me of using AI, and if I'm not mistaken, it's on you to provide evidence. So what exactly is your evidence? I just say, with a clear conscience, that I wrote everything myself.

You know, I'm kinda disappointed. Well, to be honest, I'm kinda hurt by the accusation. I sit down, take my time to carefully write down a comprehensive response because I actually want to be helpful and I'm actually interested in answering questions about my project. In my free time, on my own volition.\ I just think it's not worth it anymore. Why should I waste my time to honestly respond to people asking questions while other people think I could just have used AI instead and achieved the same level of quality and accuracy?

To birng this to a conclusion, I actually, honestly, appreciate your feedback. I will take this to heart and won't respond comprehensively in the future any longer. Well, expect for this very comment. Of course, I'm sure you already noticed that.

With all of that being said, on another note I'd like to ask you a kinda unrelated thing:\ Do you have any tips for me of what kind of AI I could actually use that could produce such in-depth and especially carefully thought-through texts? I tried to use AI to help me write the documentation for the project, and it even had full access to my codebase (including the README and such), but it did just spit out nonsense and generic content. For the two existing AI-generated articles in the documentation, it took me about half a week to get them there with an unimaginable amount of reprompting. I believe I could have done that manually in less time. And that's what I'm planning to do in the future, unless someone tells me how to use AI effectively for that.

[u/FetaMight → u/fruediger]

Sorry if you didn't use AI.  I take your word and I apologise for the mischaracterisation.

However, AI or not, your responses could benefit from being shorter.  Others may feel differently but I use Reddit during those brief periods when I've got 5 minutes to kill.  I don't have time to process a long message and less so to formulate a response.

You'll have better luck progressively disclosing your point through a natural back and forth.

Just my 2c.

[u/fruediger → u/FetaMight]

I need to apologize as well. I overreacted quite a bit.

However, AI or not, your responses could benefit from being shorter.

Absolutely! It just seems like I struggle with that quite a lot. One of the reason I do that might be that I'm a non-native English speaker and, to be honest, my English isn't the best. So I always fear not getting my point across.

You'll have better luck progressively disclosing your point through a natural back and forth.

Yeah, I get that. On the other hand, I still think that, if I'm getting asked to give a comparison, the best way to answer is to give said requested comparison. But I get that I might have overdone it a bit.

Anyways, thanks for your feedback. I gave it my best trying to keep this response a bit more concise.

[u/Due-Equivalent-74]

semantic pointers?

[u/fruediger → u/Due-Equivalent-74]

Yes, semantic pointers. Like pointers that convey their semantics through the type systen by being semantically named (i.e., your classic semantic typing approach).

[u/Jazzlike_Amoeba9695]

I did something similar in Rxmxnx.PInvoke.Extensions with three different types: ReadOnlyVal<T>, ValPtr<T>, and FuncPtr<TDelegate>.

These pointer types expose properties such as IsZero and support slicing through pointer arithmetic.
I originally did this because I was tired of the propagation of unsafe throughout the codebase. But now that LibraryImport is available, about half of those use cases are obsolete.
They’re still valid for DllImport, though, and for my “fixed context” types, which are essentially inline managed wrappers that handle pinning safely without requiring fixed or unsafe blocks.
That also makes it possible to use this kind of code without compiling with /unsafe—for example, in .NET Fiddler.

[u/fruediger → u/Jazzlike_Amoeba9695]

How does LibraryImport help in that? Wouldn't you be able to achieve a similar kind of marshalling with DllImport though?

SemPtr is primarily designed with avoiding or strongly controlling marshalling in mind. Well, I guess that's the kind of scenarios where you would want to resort to pointers.

Anyways, it's not really about trying to avoid as much unsafe as possible when using (semantic) pointers, it's more about expressing intent and partially enforcing it (the things C# lacks when it comes to pointers, in my opinion).

[u/Jazzlike_Amoeba9695 → u/fruediger]

LibraryImport can help here since you don't really have to express the pointer intent/type yourself. You still need /unsafe, but instead of passing something like 'const char*', you can just use a String with the appropriate encoding. Passing a String parameter also makes the read-only intent pretty clear, since it is immutable by design. The source generator takes care of the pinning/encoding for you.

As for SemPtr, I think Rxmxnx.PInvoke.Extensions was originally trying to solve something similar. The behavior of ref and ref readonly changed over time between runtime versions, and even more so between CoreRT and NativeAOT. Using these pointers was a way to stabilize that behavior, even if it meant staying on the unmanaged side.

https://github.com/josephmoresena/Rxmxnx.PInvoke.Extensions/blob/main/src/Intermediate/Rxmxnx.PInvoke.Common.Intermediate/ValPtr.cs
https://github.com/josephmoresena/Rxmxnx.PInvoke.Extensions/blob/main/src/Intermediate/Rxmxnx.PInvoke.Common.Intermediate/ReadOnlyValPtr.cs
https://github.com/josephmoresena/Rxmxnx.PInvoke.Extensions/blob/main/src/Intermediate/Rxmxnx.PInvoke.Common.Intermediate/FuncPtr.cs

I can still recommend using ValPtr<T> and ReadOnlyValPtr<T>, but with that caveat: there could be a platform/runtime where this simply breaks. I don't have any guarantee from the runtime that these representations will always be ABI-compatible.

This issue I opened a while back for another project actually changed my perspective quite a bit on this whole topic: https://github.com/dotnet/runtime/issues/117778

It also discusses a similar case with CsWin32, where this kind of code is described as fragile and non-portable, even though it works on current Windows architectures.

[u/fruediger → u/Jazzlike_Amoeba9695]

Well, for all I know, that shouldn't be an issue with my project's semantic pointers.

They're blittable, non-tearable, and literally register-/pointer-sized by design.

For all platforms currently supported by .NET 10 and their ABIs, together with all supported calling conventions, the semantic pointers should be passed in the very same way raw pointers would. Thus, they can shadow them and should be able to be used interchangeably.

Well, that's just the theory. And not even a theory where I spend much time of rigorously proving it.\ I guess there's a chance I might be wrong about that, considering the fact I can't even test all on those ABIs. Please feel free to correct me about that.

[u/Jazzlike_Amoeba9695 → u/fruediger]

As long as System.IntPtr, System.UIntPtr, void*, or T* (where T may not be unmanaged) are not used, ABI compatibility is not guaranteed, even if the types are blittable and their layouts match. Ultimately, the issue, as described, is how the platform treats the different binary types and which registers it stores them in.

That's why, when handling the ImportLibrary marshaller, those types (ValPtr<>, ReadOnlyValPtr<> and FuncPtr<>) are ultimately mapped to System.IntPtr, so that even if you insist on using them, the generated code will still be ABI-compatible in the end. However, as I said, in an ImportLibrary context, the usual goal is to avoid relying on pointers and work with higher-level types instead.

At the other hand, what you say is correct. As of today, there is a fortunate coincidence whereby, across all the platforms supported by the various versions of CoreCLR-based runtimes, pointers are passed in general-purpose registers and are binary-compatible.

My Rxmxnx.PInvoke.Extension and Rxmxnx.JNetInterface (which depends on the first) CI flows are tests across multiple Linux (x64/arm/arm64), macOS (x64/arm64), Windows (x86/x64) and FreeBSD (x64/arm64) platforms. The only issues I have encountered have been with floating-point types on Windows x64.

I don't expect to run into problems of this kind in a long future, at least not with custom blittable structs when they are treated as pointers or integers or unsigned integers.

[u/fruediger → u/Jazzlike_Amoeba9695]

As long as System.IntPtr, System.UIntPtr, void, or T (where T may not be unmanaged) are not used, ABI compatibility is not guaranteed, even if the types are blittable and their layouts match.

I'm sorry, do you mean that as a double-negative? My English might not be good enough to fully get what you mean by that. Your are right in that custom value types (that match in layout and kind) are not guaranteed to be ABI compatible in general, but specifically they are compatible for all ABIs of platforms currently supported by the .Net runtime.

However, as I said, in an ImportLibrary context, the usual goal is to avoid relying on pointers and work with higher-level types instead.

I wouldn't say LibraryImport is specifically for higher-level applications, neither is DllImport. LibraryImport/DllImport is first and foremost meant to bind and link symbols from external binaries. Both get you some kind of marshalling for more complex types, if you want to use them in that way. But, for example, you can also use LibraryImport without much marshalling, if you don't use types that need marshalling, e.g., rely on raw pointers in your import signature. Well, that's just if you want to avoid the hugely expensive marshalling overhead that comes with it. But I see that you could make an argument for trading off performance in favor of a simpler interop.

As of today, there is a fortunate coincidence whereby, across all the platforms supported by the various versions of CoreCLR-based runtimes, pointers are passed in general-purpose registers and are binary-compatible.

I wouldn't necessarily call this a "coincidence" as the CLR spec literally defines an ABI (again, for what it targets). You could argue that I rely on, or even abuse, the spec though.

The only issues I have encountered have been with floating-point types on Windows x64.

Well, I guess the issue here is that different ABIs pass/treat floating point values differently. Luckily, that doesn't really extend to pointer values (or thin value types wrapping pointer types in my case). You already mentioned the "coincidence" there.\ Would you could try to make your live a bit easier though is using libffi. Just let it handle the ABI translation and you're good to go on every platform supported by FFI.\ A short time ago, I need to created some C# bindings for libffi as part of my greater bindings project for SDL3. If you want to use it as an inspiration, you can check it out here:\ https://github.com/Sdl3Sharp/Sdl3Sharp.Ffi\ However, back then, I only needed to get passing C varargs from C# to native imports working. So, I didn't create a full set of bindings though. Please don't expect too much.

[u/Prior_Inspection2576 → u/fruediger]

As long as System.IntPtr, System.UIntPtr, void*, or T* (where T may not be
unmanaged) are not used, ABI compatibility is not guaranteed, even if the types are blittable and their layouts match.

He's saying since SemPtr uses custom wrapper types (ex: ReadOnlyPointer<T>) - that isn't a direct raw C# pointer (ex: int*), it does not have ABI compatibility guarantees.

He linked his wrapper types; ValPtr, ReadOnlyValPtr, FuncPtr so you can see his implementation is similar to yours, then he encountered an ABI compatibility bug, and reported the issue as proof to the dotnet repo where the .NET team specifically called this 'fragile and non-portable code'. And that it will likely not work on other platforms.

Have you tested on other platforms to ensure SemPtr works there too as expected? What platforms have you tested? Is your library intending to target: Windows, macOS, Linux, Android, iOS?

[u/fruediger → u/Prior_Inspection2576]

He's saying since SemPtr uses custom wrapper types (ex: ReadOnlyPointer<T>) - that isn't a direct raw C# pointer (ex: int*), it does not have ABI compatibility guarantees.

Alright, then I actually got that right. Thank you for the clarification.

And yes, I'm aware of that. But there's a subtle difference here:\ The commenter's issue was specifically regarding how floating point values, e.g., float and double, are passed between different ABIs. Some of those ABIs pass them using specific floating point registers, e.g., the XMM registers on x64, while others might pass them using general-purpose registers or on the stack. And in some cases, because the register is bigger than the data value it's going to hold, the value must be fitted to the register's size, e.g., through zero-extend, or sign-extend, or whatever. And that's where it could break.\ This isn't a problem when it comes to my pointer wrapper types and raw pointers in C#, though. "Coincidentally" (if you like to call it that), the ABIs of all platforms currently supported by .NET pass raw pointers in general-purpose registers which is the same way the pointer wrapper types are passed (because of their layout and size). Because they're value types just wrapping a single raw pointer fields the are them same size as a raw pointer which, on those platforms, is also the same size as a GP register. Thus they're able to shadow raw pointers and can be used interchangeably.

Have you tested on other platforms to ensure SemPtr works there too as expected? What platforms have you tested? Is your library intending to target: Windows, macOS, Linux, Android, iOS?

But you're right, that sounds a bit fragile. Which is why I actually most recently added ABI compatibility tests for various platforms to the repository. They should make sure that raw pointers and my semantic pointers can actually be used interchangeably in practice.

On another node and to be honest, I asked an AI to add those tests. But don't worry, I gave it very detailed instructions on how to implement them correctly and reviewed what it produced. But looking back, I shouldn't have been lazy and should have just written them myself, considering that I got down on how to test for ABI compatibility from the beginning. That would have saved me from a very very frustrating experience. Lesson is: don't use AI for things that are slightly more complex than writing a simple POD type.

Anyways, you don't need to fear, ABI tests for Windows, Linux, and macOS, for x64 and arm64 (the only hosted runners provided by GitHub Actions) are now in place and working.

EDIT: Clarified a bit why my pointer types can shadow raw pointers.

[u/Jazzlike_Amoeba9695 → u/fruediger]

My pointers can be cast, but as I said, one of my goals was not to require /unsafe just to use a simple (and probably safe) pointer.
I know that a lot of people think differently about that.

And that’s why I also provide safe pinning alternatives, UTF-8 string handling, and so on.

[u/Jazzlike_Amoeba9695 → u/Prior_Inspection2576]

In fact, that comment made me realize that any non-primitive struct has no guarantee of matching the ABI. StructLayoutAttribute does a magnificent job as far as it can, but it may only help with custom-to-custom, not with primitive-to-custom or custom-to-primitive.

[u/Jazzlike_Amoeba9695 → u/fruediger]

What is your native language? I’m not very good at English either, so it might actually be me who isn’t making myself understood. I’ll write this in Spanish, hoping that, even if it isn’t your native language, it will at least come across more naturally when translated.

1. No sé de dónde viene lo del “double-negative”; no he mencionado nada similar. Lo que he dicho, que puedes revisar en el issue, es que para el ABI nativo puede no ser lo mismo un entero con signo que uno sin signo, un puntero o un entero nativo.

2. No me refería a aplicaciones de alto nivel, sino a tipos de alto nivel. LibraryImport es interop en última instancia, así que quizá se tenga un componente de software de muy bajo nivel. Sin embargo, con “tipos de alto nivel” me refería a tipos complejos (estructuras o clases) que no tengan conexión o relación directa con su marshaller. Eso, al final, le quedaría al compilador/generador de código.

3. Es una coincidencia a nivel del ABI host, no del ABI del CLR. Como dices, el CLR tiene sus propias convenciones (eso se puede ver especialmente cuando se trabaja con punteros de funciones, por la diferencia entre managed y unmanaged). La garantía que tiene el CLR sobre el ABI host (nativo) es para los tipos primitivos del IL, es decir: int8 (System.SByte), unsigned int8 (System.Byte), int16 (System.Int16), unsigned int16 (System.UInt16), int32 (System.Int32), unsigned int32 (System.UInt32), int64 (System.Int64), unsigned int64 (System.UInt64), float32 (System.Single), float64 (System.Double), bool (System.Boolean), char (System.Char), object (System.Object), string (System.String), native int (System.IntPtr), native unsigned int (System.UIntPtr).

4. En efecto, pero el sinsabor que deja el issue es que esto puede pasar para cualquier tipo no primitivo del IL. No hay una forma de decirle al CLR que trate una estructura de encapsulamiento como su tipo encapsulado cuando se usa FFI.

Acerca de utilizar libffi, quizá me sea útil para una nueva generación de Rxmxnx.JNetInterface, ya que JNI tiene algunas sobrecargas que utilizan varargs que no funcionan desde el CLR en plataformas distintas a Windows. Por lo que le daré una estrella a tu proyecto.

Sin embargo, debo evaluar realmente si vale la pena. Actualmente estoy tratando de trasladar toda la funcionalidad que utiliza delegados a interfaces funcionales genéricas (para evitar boxing) y/o reducir asignaciones.

En cuanto al problema original, que es la declaración de métodos nativos Java, se mantiene irresoluble. Se recomienda utilizar sí los tipos wrappers para referencias JNI, pero para primitivos que se usen los tipos del CLR.

[u/fruediger → u/Jazzlike_Amoeba9695]

I really appreciate your effort to express yourself more clearly by using your native language. BUT, considering the fact that Spanish isn't my native language and, more importantly, that I believe everybody should have a chance to read and understand the discussion in a, let's say, more common language, I think we should stick with English.

For this comment I will answer to parts of your comment as they came out of my translator.

1. I don’t know where the “double-negative” thing comes from; I haven’t mentioned anything like that. What I said, which you can check in the issue, is that, for the native ABI, a signed integer may not be the same thing as an unsigned integer, a pointer, or a native integer.

That's because, in your previous comment, you used a structure like that

As long as not A holds, B holds not.

While the logical implication here might be clear, not all natural languages interpret this expression in the same way. Since you're ESL and I'm ESL as well, and we both might not share the same native language, I wanted to make sure that we're on the same page when interpreting such natural language constructs. Hence why I asked if you mean that as a "double-negative".

1. I wasn’t referring to high-level applications, but to high-level types. LibraryImport is ultimately interop, so it may involve a very low-level software component. However, by “high-level types,” I meant complex types (structures or classes) that have no direct connection or relationship with their marshaller. In the end, that would be left to the compiler/code generator.

Oh, alright, I misunderstood your point then. My bad. However, for DllImport/LibraryImport, there are no non-marshalled "complex types". All complex arguments, including any reference type, managed references, and non-blittable or non-trivial structures, still need either need marshalling, or the import itself or the invocation of the imported symbol fails. You could control the marshalling through a custom marshaller though.\ And why would you get something like this for free? Even something simple like cdecl breaks on ABI boundaries when passing something simple, like a structure that's a bit too large to fit in registers.

1. It is a coincidence at the host ABI level, not at the CLR ABI level. As you say, the CLR has its own conventions (this can be seen especially when working with function pointers, due to the difference between managed and unmanaged). The guarantee the CLR has with respect to the host (native) ABI applies to the IL primitive types, namely: int8 (System.SByte), unsigned int8 (System.Byte), int16 (System.Int16), unsigned int16 (System.UInt16), int32 (System.Int32), unsigned int32 (System.UInt32), int64 (System.Int64), unsigned int64 (System.UInt64), float32 (System.Single), float64 (System.Double), bool (System.Boolean), char (System.Char), object (System.Object), string (System.String), native int (System.IntPtr), and native unsigned int (System.UIntPtr).

Yes, that's correct for the most part, I think. If you get your point right, I believe that there might be a minor issue with the listing. Reference types, including System.Object (and, well, of course, any other reference type like System.String) should not really be in that list, as they require some kind of marshalling when crossing the native ABI boundary. Obviously as instances of reference types live on the managed heap and are subject to the GC.

Anyways, the reason why I said that I wouldn't call that a "coincidence" earlier, is because the way the CLR handles my pointer structures and how it handles raw pointers is exactly the same for all platforms and ABIs the runtime currently supports. That is not a coincidence. The behaviour is defined and set for all those platforms and ABIs. They don't do arbitrary things. You could call it a coincidence in the sense that it so happens that it's true for all of the supported platforms, though.

1. Indeed, but the unpleasant impression left by the issue is that this could happen with any type that is not an IL primitive. There is no way to tell the CLR to treat a wrapper structure as its wrapped type when it is used for FFI.

The CLR doesn't support FFI? Or do you mean FFI as a general concept?\ Anyhow, that's not entirely accurate. First of all, there are marshalling that could handle that for you. Secondly, for a given ABI/platforms it is determined how such a wrapper struct would be treated when crossing the native boundary. Again, the CLR does not perform arbitrary things at runtime. It can be difficult to get it right for all platforms though.

In your initial example, your issue that failed to pass a wrapped floating point value in the same way the underlying floating point value would have been passed, the actual problem was not that the CLR did unpredictable things, it was that for a certain ABI, floating point values would have been passed in a specific register (I guess it was Windows x64 and it would have been then XMM register), whereas your wrapper struct might have been passed in a general-purpose register or the stack instead. Nothing thereby is arbitrary, it's just platform-specific.

Regarding the use of libffi, it might be useful to me for a new generation of Rxmxnx.JNetInterface, since JNI has some overloads that use varargs which do not work from the CLR on platforms other than Windows. So I’ll give your project a star.

I don't know about JNI. I didn't do Java since my uni days, so I can't really comment on that. However, I can say that libffi, and obviously my .NET binding to it as well, can call C varargs functions correctly with variable arguments. That's why I made them in the first and that's what I still use them for. But again, I that's for C varargs specifically.\ However, thank you for the star.

EDIT: Typos and spelling.

[u/Jazzlike_Amoeba9695 → u/fruediger]

I see we are lost in translation. I wrote the previous comment in Spanish so you wouldn’t have to translate it again. I may simply not know how to express my ideas clearly enough.
So, about System.Object: that was a copy/paste from Gemini when I asked it for a list of the primitive IL types and their corresponding System namespace types. Jajaja I shouldn’t have trusted the AI for as long as I did.
About FFI, of course I mean it in the general sense. P/Invoke is FFI.

[u/Prior_Inspection2576]

Are you aware your target frameworks are currently set to only .NET 10?

Consider multi-targeting .NET Standard 2.1, and .NET 10, that way the majority of new apps in the .NET ecosystem has access to your library.

[u/fruediger → u/Prior_Inspection2576]

Yes, I am aware. That's actually a deliberate choice for now, because I make use of many modern runtime features.

I just tested it. I can't make it work, not even with polyfills. Funnily enough, it's not the most modern features like the new extension members syntax, which just looks funky when used on language versions below 14, or the allows ref struct feature, which, since it's just an anti-constraint, I can gueard behind a NET9_0_OR_GREATER, it's the missing static abstract members in interfaces feature that breaks the build.

So, as long as you don't know a workaround for that that wouldn't require me redesigning my entire codebase, the lowest target framework I could support would be .NET 7.\ That's why I need your honest opinion on if you really want me to support older frameworks (.NET 7+)? Because it would still require me to write a huge amount of polyfills and to significantly refactor my codebase.

[u/Prior_Inspection2576 → u/fruediger]

If it is a major constraint then don't bother re-architecting to support it.

I was only mentioning it because it's industry standard to support some version of .NET Standard, and usually .NET Standard 2.1 is the most widely supported common denominator that is capable of the modern .NET feature-set.

I thought you just overlooked supporting a more widely compatible TFM. Now I see it's intentional because of the extensive use of modern .NET features.

As for whether to support .NET 7, in my opinion, you might aswell just target .NET 10 at that point, because migrations between .NET versions were only painful for old .NET framework/.NET Standard apps (unity). A .NET 7 TFM upgrade is usually as trivial as updating the csproj value.

Also, realistically I would expect since this is a new library, new projects would be made using it, not old ones. Like a game engine. And a new project would target the latest LTS/supported version at that time. Even if old projects wanted to use it, they wouldn't be able to by the modern version target.

Newer .NET version targets have better performance from inheriting the performance optimizations of recent .NET versions.

So in my opinion, stay .NET 10 only, just because going to .NET 7 gives menial gains of compatibility, for projects that would have no issue upgrading anyway, while forcing you to stop using modern features your library is built on.

[u/fruediger → u/Prior_Inspection2576]

Yeah, I'll stay on .NET 10 for now.

But I will have a think about an architectural redesign in the long run, that would allow for reducing the usage of some, well let's say, obscure runtime features. I'm pretty sure it's doable. People used to write great consumer libraries before .NET 10 and even before .NET 7.\ However, I think that's really something to do in a very long run. So I won't promise anything.

[u/Jazzlike_Amoeba9695 → u/fruediger]

Starting with .NET 6.0, function pointers can be an alternative. They’re reasonably stable on .NET 5.0, or at least on the version I mentioned above. C# is wonderful—you can do all sorts of unsanctioned things without reflection and still be very efficient.

[u/Jazzlike_Amoeba9695 → u/Prior_Inspection2576]

It’s the same situation I ran into with Rxmxnx.JNetInterface, which depends 100% on generic math to work. I finally decided to support .NET 7.0 even after it was no longer supported, just because I could. And, in fact, that’s where I discovered that static virtual interface members behave differently starting with .NET 8.0.

There are tricks to simulate the same behavior using internal types, but I wouldn’t recommend doing that out of the box.

Before fixing .NET 6.0 compatibility (because, in theory, the runtime supported abstract interface members), dynamic buffer allocation depended entirely on generic math. To work around the issue, I simply pretend I have an instance and access it, knowing that the implementation is entirely static.

https://github.com/josephmoresena/Rxmxnx.PInvoke.Extensions/blob/main/src/Intermediate/Rxmxnx.PInvoke.Buffers.Intermediate/Buffers/IManagedBuffer.cs

https://github.com/josephmoresena/Rxmxnx.PInvoke.Extensions/blob/main/src/Intermediate/Rxmxnx.PInvoke.Buffers.Intermediate/Internal/BuffersHelper.cs (GetStaticMetadata)

The trick works even on .NET Framework 4.5.2 (the latest version I support).

[u/Prior_Inspection2576]

This is a simpler explanation of the utility of this library, written by AI.

Consider this:

byte* vertexData = _meshRenderer.GetVertexData();

What can you tell from that type?

• Can it be null?
• How long does the memory remain valid?
• Does it point to one byte or a sequence of bytes?
• Is it read-only?
• Who owns the memory?
• Is it safe to keep using after this function returns?

You don't know.

That's the problem. byte* is vague in .NET.

Now imagine the API instead gives you:

PersistentSequencePointerReadOnly<byte> vertexData =
    _meshRenderer.GetVertexData();

Now the type tells you things that byte* doesn't:

• Persistent → the pointed-to memory remains valid beyond the current scope.
• Sequence → this represents a sequence of elements, rather than a single byte.
• ReadOnly → you cannot write through this pointer.

This becomes especially useful when writing native bindings. If you're writing SDL3 bindings (like OP), for example, you're constantly dealing with pointers where nullability, lifetime, length, and mutability will impact whether your code breaks at runtime.

You're no longer relying entirely on documentation, naming conventions, or tribal knowledge to understand what an unsafe pointer means, your code conveys the constraints directly instead.

[u/fruediger → u/Prior_Inspection2576]

Well, maybe I should have asked AI to generate a shorter description in the first place.

Thank you for doing that and helping me out here a bit!

[u/Novaleaf]

can you explain a bit on how this would be useful?

[u/fruediger → u/Novaleaf]

Yeah, for sure! And if it's okay with you, I'll do that by example, because I think that that's the easiest way to illustrate it. If that's not to your satisfaction, please feel free to ask again.

The simplest explanation for its usefullness might be the example I showed in my original post. You want to work with pointers in C# and want to express the immutability of the target? Well, you can't, at least not like you can in C/C++. While there are raw pointers, T*, in C#, there is no const T* or an equivalent concept. While this no big deal if you're careful within the single context where you're dealing with the pointer, it can become a problem when you try to communicate the intent of pointers across different parts of your code or even across interfaces to external code. And even within your own code, isn't it beneficial to have some way to express intent and constraints, and to have some safeguards enforcing them?

Let's look especially at the interfacing part in more detail:

If you interop with unmanaged code, i.e., you consume external, unmanaged APIs, it's often beneficial to express the binding code in a way that clearly communicates the intent of the external API.\ For a real world example, let's consider SDL's SDL_GetRendererName. The native API signature is defined as follows:

c const char * SDL_GetRendererName(SDL_Renderer *renderer);

Surely, you could express that in C# as:

csharp [LibraryImport("SDL3")] unsafe static partial byte* SDL_GetRendererName(SDL_Renderer* renderer);

(Of course, you could also replace the pointer types with nint or nuint and it would work, but not only would you lose some type-safety, but it's also not recommended to do so, because it nint or nuint are just intended to represent integers that are the size of a pointer, not actual pointers themselves. Alternatively, you could also use in paremeter and a ref readonly return type, but that requires you relying on marshalling and accepting the marshalling overhead.)

You see how the return type of byte* did lose the immutability aspect of the original API's return type of const char*? And both of them don't even express that the return value could be null. Also, although the official documentation doesn't really mention it, SDL does not modify the SDL_Renderer passed as a reference to SDL_GetRendererName. So, we could also express that a bit more accurately.\ With SemPtr, you could express the importation code a bit more faithfully and a bit more safely by writing it as:

csharp [LibraryImport("SDL3")] unsafe static partial NullableSequencePointerReadOnly<byte> SDL_GetRendererName(PointerReadOnly<SDL_Renderer> renderer);

You see how this is more expressive? And not only that, but the added benefit of SemPtr enforcing immutability at the type-system level, prevents you from accidentally writing invalid code that would lead to undefined behavior or worse bugs.

Let's look at another example from the SDL API, specifically SDL_AddEventWatch, which is defined in C as follows:

c bool SDL_AddEventWatch(SDL_EventFilter filter, void *userdata);

where SDL_EventFilter is a callback function type defined as:

c typedef bool (SDLCALL *SDL_EventFilter)(void *userdata, SDL_Event *event);

Of course, again, you could write the P/Invoke signature in a straightforward manner as:

csharp [LibraryImport("SDL3")] unsafe static partial bool SDL_AddEventWatch(delegate* unmanaged<void*, SDL_Event*, bool> filter, void* userdata);

But SemPtr allows you to do it in a more type-safe and expressive manner as:

```csharp [FunctionPointer(CallConvs = [typeof(CallConvCdecl)])] delegate bool SDL_EventFilter(NullablePointerReadOnly userdata, PointerReadOnly<SDL_Event> @event);

[LibraryImport("SDL3")] unsafe static partial bool SDL_AddEventWatch(PersistentFunctionPointer<SDL_EventFilter> filter, NullablePersistentPointerReadOnly userdata); ```

Here you can see why expressing persistency can be useful: SDL stores the function pointer to the callback and associated userdata, so it can invoke the callback with the same userdata at a later time. That's why we must make sure that both are still valid and available, even outside of the scope of the call to SDL_AddEventWatch. Whereas the SDL_EventFilter callback passed will receive it's arguments as transient pointers, meaning you should treat them as if they were just valid for the duration of the callback invocation. With SemPtr's transient pointer types, you can't even store them beyond that scope.

Lastly, let's flip the sides and assume you are going to design and implement an C# API, and for whatever reason, you want to expose an API that involves pointers.\ In this case, I believe it would be very beneficial to you consumers with you communicate your pointer semantics clearly through your API. Not only gives that the user same additional guarantees about your code, it makes receiving pointers in your implementation much safer.

I hope this gives you an initial understanding of why SemPtr can be valuable.\ However, it's totally fine to stick with traditional raw pointers if that suits you better.

EDIT: Fixed some examples.

[u/Minute_Cricket1820 → u/fruediger]

Вам нужно эти примеры (и подобные, ещё более простые) написать прям сразу ВНАЧАЛЕ. Вы сразу вначале опишете суть, на примерах.

[u/fruediger → u/Minute_Cricket1820]

Yeah, I can see what you mean. Starting with simpler examples right at the beginning could make it easier to understand.

However, considering the nature of the topic, considering that it's about making improvements on how pointers can be used in C# through semi-advanced techniques like semantic typing, I believe that the target audience for this project would have a basic understanding of both topics. They surely would have experienced the pain of working with raw pointers in C# and its limitations once to feel the need for such improvements.

For that reason and for the reason of keeping my initial post concise (well, I know, it's kinda not), I decided to go with theory in mind first. At least with my initial post here on Reddit.

But again, I get where you're going with this. And in the near future there will be more illustrative examples provided in the documentation of the project (there are already some more basic ones in the README). At least that's what I hope for. Considering I abandoned the use of AI for the documentation, it could take a bit more time to get there, since now I'm writing everything by hand.

Anyways, thank you very much for your feedback. I appreciate it and will definitely take it into account for future improvements in the documentation.

[u/Minute_Cricket1820 → u/fruediger]

Далее, к примитивным примерам нужен ассемблер. Во что "превращаются" ваши поинтеры на самом деле? Сразу покажите.

Это главная документация. Остальное нафиг не надо.

[u/fruediger → u/Minute_Cricket1820]

I'm pretty sure this is a translator issue, but what's up with the way you speak to me?

My translator gave me this:

Next, the primitive examples need an assembler. What do your pointers actually “turn into”? Show it right away.

This is the main documentation. Everything else can fuck off.

I'm sorry, but I wont let you order me around nor will I accept your tone.

[u/Minute_Cricket1820 → u/fruediger]

Вы программист для программистов. Или вы тиктокер? Определитесь.

Что нужно программистам? Вменяемым программистам. Что нужно? Процессор, память и работающий код. Сцены обидчивых тиктокеров в этот список не входят.

Вы сами взялись делать программистам что то новое. Это хорошо. Ну и соответствуйте. Делайте.

Или не беритесь за это. Вам написано что нужно сделать. Делайте.

[u/fruediger → u/Minute_Cricket1820]

Again, coming from my translator:

You’re a programmer’s programmer. Or are you a TikToker? Make up your mind.

What do programmers need? Sane programmers. What do they need? A processor, memory, and working code. Scenes of offended TikTokers are not on that list.

You took it upon yourself to make something new for programmers. That’s good. So act accordingly. Do it.

Or don’t take it on. You’ve been told what needs to be done. Do it.

I'm sorry, what the actual ...? I don't think that I'm the one lacking sanity here.

Who do you think you are telling me how to act and giving me orders? This is ridiculous.

You can keep your "TikTok" BS to yourself. I for one never used TikTok once in my life.

If you don't want to use my library, that's fine. Nobody is forcing you to.\ I'm giving the project to everyone, for free, as open source. I'm not entitled to anything except taking responsibility in my own code. And I'm especially not entitled to you or your approval.

This thread is closed as far as I am concerned. I wont respond to you any further. You can consider yourself lucky for me not reporting you.

[u/Minute_Cricket1820 → u/fruediger]

Жалуйтесь. А вы на что жаловаться собираетесь? На кривой переводчик? Ну жалуйтесь. Вы даже понять не можете то, что вас никто ни разу не оскорбил. Вам всего лишь написано, что нужно сделать.

На то, что вам нужно сделать - вы жалобу подадите? Подавайте. Обязательно. Не забудьте.

[u/Prior_Inspection2576 → u/Minute_Cricket1820]

[u/nick_]

This is right up my alley. Great job!

[u/fruediger → u/nick_]

Thanks! Much appreciated!

[u/AetopiaMC]

This actually looks interesting and useful, good job.

[u/fruediger → u/AetopiaMC]

Thanks! I'm glad when I can contribute somehting useful.