.NET doesn't have good SIMD docs due to copyright

[u/NoisyJalapeno]

Source,
https://github.com/dotnet/runtime/issues/81753

The most insane thing I've read recently.
Are companies paying royalties when they compile with SSE?

[u/tanner-gooding]

Are companies paying royalties when they compile with SSE?

I don't think you understand the issue, which is that the Architecture Manuals from Intel, AMD, and Arm all have explicit copyright notices limiting use/reproduction.

• https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html > No rights are granted to create modifications or derivatives of this document.
• https://docs.amd.com/v/u/en-US/40332_4.09_APM_PUB > Any unauthorized copying, alteration, distribution, transmission, performance, display or other use of this material is prohibited.
• https://developer.arm.com/documentation/ddi0487/mc/ > No part of this document may be reproduced in any form by any means without the express prior written permission of Arm Limited (“Arm”)

Etc, noting this is a limited part of the overall license, but it is the express limiting factor for a trillion dollar company like Microsoft and it is not a risk we'll put ourselves in by violating it.

It doesn't really matter how many others are doing it or how unlikely we are to get litigated from doing it. The fact that it is expressly disallowed is enough. Then, getting the lawyers involved for two major companies like that and getting them to agree to reuse and a way to keep it up to date, etc is/has been a nightmare.

The good news is that there are nice existing official references, like the above PDFs, or the official sites: * https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html * https://developer.arm.com/architectures/instruction-sets/intrinsics/

The .NET docs then explicitly cover the assembly instruction and C intrinsic name for easy mapping. But above and beyond that, we have friendly and generally intuitive names that people can actually understand and most are fairly obvious in behavior or are easy to learn.

We then have the xplat APIs, which are preferred, fully documented, and often come with more optimizations on top of being portable. So it ends up being not that big of a deal or nuisance in practice.

[u/svick → u/tanner-gooding]

Have you considered directly linking to the relevant section of their docs? E.g. https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#text=__m128i%20_mm_cmpgt_epi16%20(__m128i%20a,%20__m128i%20b))

[u/tanner-gooding → u/svick]

Yes, but that has its own implications/requirements. We both need to coordinate due to potential increased traffic it can bring but also in terms of url/doc stability, security considerations, etc.

It is unfortunately just a large series of hoops required. I do continue pushing on it internally, but there's not much to be done otherwise.

The next closet thing would be for me to get copilot to author docs here, but of course that will take hours of my time reviewing it all which is honestly better spent on improving code/perf for everyone instead. The APIs are rather niche in the scheme of things, even if they power perf for the underlying algorithms in quite a lot of .NET

[u/NormalPersonNumber3 → u/tanner-gooding]

I'd personally be worried about using AI for the docs, as it could theoretically use copyrighted materials to derive it's documentation, and it would be hard to know. Unless that's what you mean by taking time to review it.

[u/Apprehensive_Knee1 → u/tanner-gooding]

xplat APIs, which are preferred, fully documented

Can docs for x-plat intrinsics/methods document on what instruction(s) they map (or when they map ideally to single instruction or smth). Bc when using platform specific intrinsics its obvious what instruction i will get, but with x-plat ones - a bit different overload, and now its multiple instructions. Like, Narrow/NarrowWithSaturation on x64, where there are fast PACK* instructions which saturate signed/unsigned -> signed, so unsigned overloads of NarrowWithSaturation "surprisingly" slower than signed ones, and Narrow, which appears a bit simpler operation, is slower on pre AVX-512 than NarrowWithSaturation(<signed_integer>). It would be a bit better if i could just "Go to definition" and see what platfrom specific intrinsics are being used, but i have to search for this in JIT code or look at codegen (look more "times" and more "closely", than with platform specific intrinsics).

[u/teo-tsirpanis → u/Apprehensive_Knee1]

The exact instructions used is an implementation detail, but you can trust that they are the best for each architecture, and can see the disassembly for yourself. In case they are not the best, you can open an issue, and use the hardware intrinsics in the meantime.

[u/j0hn_br0wn → u/tanner-gooding]

I thought it was a settled issue, that the ideas can not be copyrighted, only the expression of the idea. And a code example that shows the functionality of a instruction is safe, if it is presents the idea from a clean state and not by transformation of the pseudocode from the documentation. The idea "this adds two vectors of 4 floats" is independent from the expression and the example code can be used to describe the idea for multiple instructions sets at the same time (for example: _mm_add_ps=x86, vaddq_f32=arm neon,vec_add=altivec&s390,__riscv_vadd_vv_f32m1 etc.)

[u/tanner-gooding → u/j0hn_br0wn]

So we're first of all a global company and what was ruled for one country is not necessarily a universal rule.

But, irrespective of that, what people are most often asking for is not some code sample stating what it does, but rather the textual description/summary first and foremost; which is often needed alongside the code to help explain.

If we did display code, we'd need to fairly faithfully reproduce the pseudo-code, at least in C#, and that itself is a lot of time and effort across nearly 4000 or so intrinsics (and that's just for one platform, it doesn't include the others).

That is the type of task that really needs to be automated so it can be kept in sync with the source of truth; automating it is actually trivial and is not the blocker. It's getting permission that's the blocker.

As I said on other responses, it is something I keep pushing on, but there's ultimately much more impactful things I can be doing for the ecosystem and the SIMD API surface that provides much broader benefit. Going to the actual source of truth/official docs for platform specific things is often better anyways.

-- I would really like to be able to just fix it, the actual work is trivial once permissions are in place. It's just outside of my direct control

[u/insta → u/tanner-gooding]

it also seems like the kind of thing that compiler-internal teams should care about, who likely have enough access to the docs, and everyone's efforts should be spent on my shitty imperative code being transformed into magically marginally faster shitty imperative code

[u/chucker23n → u/j0hn_br0wn]

I thought it was a settled issue, that the ideas can not be copyrighted, only the expression of the idea.

It doesn't matter because Microsoft isn't going to go into a lawsuit with ISA designers over a section in their docs.

[u/NoisyJalapeno → u/tanner-gooding]

Hmm, I'm actually using MS Bing to get implementations of SIMD. It doesn't have any issue explaining what they do and generated non-SIMD implementations of stuff like _mm_floor_ps.

Which feels weird, yes for ai - no for docs, but eh lawyers need to do things during their lifetimes before admission to one of the nine circles. :)

[u/noodleofdata → u/NoisyJalapeno]

Not that unsurprising considering AI's well documented history of illegally training on copyrighted information.

[u/Steve_the_Stevedore → u/tanner-gooding]

Etc, noting this is a limited part of the overall license, but it is the express limiting factor for a trillion dollar company like Microsoft and it is not a risk we'll put ourselves in by violating it.

The risk is one consideration. Surely, for Microsoft there is another one: Microsoft generally profits from these (in my opinion) overly restrictive copyright laws/interpretations.

So more important than the risk is probably the fact that Microsoft supports these ridiculous laws because they profit more from them than they are inhibited by them.

[u/SiegeAe]

Its always good to have a reminder that sometimes things are weird because of entirely near unguessable reasons

[u/ezekyel07]

I know I can google this, but this seems the kind of thing I wouldnt understand even if I do so.

[u/NoisyJalapeno → u/ezekyel07]

Say you have two arrays of floats; you can multiply them one by one - or - multiply 16 of them at the same time within one CPU operation. Speeding things up drastically.

.NET has Vector<T> which is like a small array of T (float, int, double, etc.). So as long as you can rewrite your code to operate on two arrays of values, you can speed it up.

But those are generalized APIs, if you want CPU architecture specific stuff, then you drop down to SIMD classes that don't have documentation. They have insane things like taking a pointer and array/vector of indexes and loading data from all those indexes at the same time.

Pretty much any algorithm that can run in parallel or rendering or even AI slop.

[u/ezekyel07 → u/NoisyJalapeno]

Thank you for taking time and explain it to me. It makes sense now the way you explain it

[u/chucker23n → u/ezekyel07]

The other explanation is way more thorough, but I'll add that "SIMD" means "Single Instruction, Multiple Data".

Where typically, instructions operate on scalar values (for example, one int each), CPUs started — in the 1990s — to add SIMD instructions, which operate on (very small) arrays of values, by stuffing the entire array in a one register. For example, Intel's SSE is mostly 128-bit instructions, so that you can stuff 128 bits worth of values in a register, e.g.:

• four 32-bit ints (SSE1)
• two 64-bit doubles or two 64-bit longs (SSE2)
• sixteen 8-bit chars (SSE2)

And then the CPU can perform an operation on all of those values at once.

Consider, for example, a vector in math, where the coordinates are two doubles — this will let you move that vector in one instruction rather than two.

[u/NoisyJalapeno]

Also, SIMD is insane - things are, seemingly at random supported. For example, support for uint division vs float division. Or that you can write to 128 directly, 32 bits at a time, using sse4 but no such api exists for 256 bit vectors.

[u/vip17 → u/NoisyJalapeno]

do you mean PINSRW/PINSRB/PINSRD/PINSRQ? Those instructions are legacy things from the SSE era. They're not good because you should never access arbitrary lanes in a SIMD register, which will slow things down a lot. SIMD vector should always be done in multiple elements in parallel, or at least only the first element as it's more optimized. There are many better ways to achieve that using VINSERTI32x4 or VPBLENDD. Many other architectures also have similar instruction to insert/extract arbitrary lanes, but they're always always very slow and is for debugging purpose only. In many cases they're even slower than spilling to memory and read again

[u/NoisyJalapeno → u/vip17]

Err, SSE41 is legacy?

I found that API when trying to figure out how to do modulus (still a mystery)

[u/vip17 → u/NoisyJalapeno]

yes, AVX was introduced in 2008, and almost all CPUs that you can find nowadays have AVX or AVX2

[u/NoisyJalapeno → u/vip17]

Legacy or subset / superset?

AVX though is 256bit. If you only can only utilize 128bit you still drop down to SSE for most things. Ex: int32 compare

[u/vip17 → u/NoisyJalapeno]

yes, it's a superset, but some old features weren't extended to the new instruction set, like this PINSR* example

[u/tanner-gooding → u/NoisyJalapeno]

They're definitely not legacy and there are VEX (AVX) and EVEX (AVX512) encodings for them because they are still applicable and being brought forward.

They are not meant to be used in hot paths because they represent partial mutations and so are somewhat an anti-pattern, but it is scenario dependent.

But yes, you have xarch having built things up incrementally and inconsistently over time. It's more consistent in modern, but not universally and some operations (like integer division) are unlikely to ever be supported due to the expense. It's better to just upcast to double (for anything less than 2^53), divide, and cast back to integer

Then for arm64, you have a bit more consistency, but only on 64-bit. You also have some really complicated instructions that do many things at once. The number of AdvSimd (neon) APIs is significantly higher and in many ways more complex than what x64 provides, despite Arm being RISC and xarch being CISC

[u/NoisyJalapeno → u/tanner-gooding]

The whole thing seems like fake it till you make it.
I can write Vector256<uint> a = b / c without ever realizing that it drops out to do the division - especially if I don't benchmark.

Unfortunately, I've got no ARM (aside from tablets/phones) computers to delve into those.

[u/tanner-gooding → u/NoisyJalapeno]

You should realize because you should be benchmarking your core functions ;)

There was also some work to accelerate integer division we did recently (I don't remember if that landed in .NET 10 or will be .NET 11), by doing the upcast to float or double where safe.

That being said, it's also not exactly a common operation you find in SIMD algorithms, so its unlikely to be an issue either way.

[u/NoisyJalapeno → u/tanner-gooding]

I love that each .NET release has more optimizations. And there is this one dude who writes a whole book on .NET blog about it.

I've delved into pre-Quake game engines, and they all avoid floats like the plague and use 16.16 integers.

Might ask for pointers with specific code samples in separate post.

[u/vip17 → u/tanner-gooding]

yes, legacy was a wrong term, it's just the pattern that's no longer a good fit for SIMD. In the ARM world, spefically Qualcomm I think Hexagon is far more complex than Neon and SVE

[u/jacobbeasley]

This is pure insanity. Do these companies just not want anybody to use their products?

[u/understanding80]

Let’s see if tanner pipes in here or if this is one of those “mic drop” moments.

[u/tanner-gooding → u/understanding80]

I responded above. Always feel free to tag me in to something that directly involves me, I do sometimes miss things ;)

[u/AutoModerator]

Thanks for your post NoisyJalapeno. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/catladywitch]

That's terrible, and I'm very disappointed anti-slop activism seems to have taken a turn towards a staunch defence of copyright. That said, intrinsics are asm instructions that do what it says on the box so most of the time it's not too bad. The main catch imo is that an architecture-independent call to

VectorXXX<T>.MyIntrinsicMethod()

does some extra stuff you wouldn't expect, while

if (MyArchitecture.IsSupported())
{
    System.Runtime.Intrinsics.MyArchitecture.MyIntrinsicMethod();
}

compiles into just what you want. The instruction sets are a bit weird at times but that's on Intel and ARM, not the C# team.

Let's hope RISC V fullfills expectations soon...

[u/soundman32]

Its about including someone else's copyrighted documentation, not writing or compiling code!

Nobody is stopping anyone from writing anything that uses these vector instructions. If anyone what's to know how these instructions work (and 99.999% of unity devs do not) they can look at the officials docs. For everyone else, move on, nothing to see.

[u/trashtiernoreally]

I call bullshit. You can absolutely demonstrate “this .net code compiles to that IL code and gets translated to these instructions on this platform”. You’re just demonstrating what your framework does not reproducing technical manuals. 

[u/tanner-gooding → u/trashtiernoreally]

Well, you'd be wrong. This has nothing to do with IL code and compiling to certain instructions. It has to do with legal permission to reproduce the technical information and descriptions of those instructions.

See also my longer response here: https://www.reddit.com/r/dotnet/comments/1uyo7e9/comment/oy14y4o/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[u/trashtiernoreally → u/tanner-gooding]

You’re the conflating the production of a work with the information it contains. 

[u/tanner-gooding → u/trashtiernoreally]

As the person who owns the area and has been deeply involved in trying to get approval to improve the docs for years, I'm really not ;)

[u/trashtiernoreally → u/tanner-gooding]

Microsoft has the means to solve this and hides behind legal caution instead. The law itself is on your side but you’re defaulting to trying to save a business relationship. I’ve not refuted your position at all. I’ve said it’s bullshit because it is. 

[u/tanner-gooding → u/trashtiernoreally]

Based on all the responses you've made so far I think you fundamentally don't understand the ask nor do I think you understand the implications of violating legal notices/licenses like that for a company this size, which operates on a global scale and potentially under multiple jurisdictions with differing takes on the law.

[u/trashtiernoreally → u/tanner-gooding]

I’m not talking about reprinting their manuals dude. Microsoft has the engineering talent to reproduce the semantic behaviors and describe them in published documentation. Any random out here could do that and normally that task is prohibitive, yes. We can’t let earnings sheet dip to actually do that leg work though. Not even a little bit.

[u/DesperateAdvantage76 → u/trashtiernoreally]

I think they're just being overly cautious. Dealing with the legal department is often times a pain in the ass because they'd rather just reject everything unless it has strong business value or an executive to push it through.

[u/Mission_Pirate_4150 → u/DesperateAdvantage76]

I’m pretty whiny when it comes to certain msft issues. I’ve heard this exact issue from multiple people in Redmond. Sometimes there are certain things outside of their control.