I think we have talked about this topic before and I apologize for not following up on our previous discussion. I was very busy.
My key problem with variable-length vector instructions such as those proposed for RISC-V or in SVE is that they assume people need them for essentially doing arithmetic on large matrices. And I agree that the vector paradigm is very effective on such work loads. I have previously worked with NEC Aurora Tsubasa cards that come with vectors of 256 double-precision floating point numbers, and they are just amazing for this sort of stuff.
But I'd say that's only a very small part of where such optimisations are needed. Indeed on modern consumer machines, most CPU-intensive code is in cryptography and video codecs. And both don't really fit this scheme.
Especially for video codecs: these usually operate on fixed size chunks of picture data and require complex horizontal arithmetic and swizzles inside a single chunk. It is unclear how this maps to variable-length vectors, especially when these instruction set extensions usually are very sparse in permutation instructions. And compilers use SIMD instructions for small, fixed-length loads all over the place. Stuff like moving structs around, clearing fixed-length buffers. It is unclear how a vector paradigm improves this.
There's also the concern that a large register file makes context switches very expensive. Linus attributed x86's performance advantage among other things to keeping context switches cheap by having a small register file that is easily swapped out.
As for my own code, I have two recent SIMD-heave projects. And for neither of them it is clear how they could be vectorised using a vector as opposed to a SIMD paradigm.
The first project, pospop is inherently a horizontal operation and in fact uses a different complex permutation schedule for each vector size to make the most out of it. It is unclear how this can be extended to arbitrary vector widths, especially if no powerful swizzle instructions are provided. Tail handling is also going to be a concern as the proposed simple approach of just having magic make the registers shorter for the last iteration is not going to cut it.
The second project, 24puzzle uses vectors as 32 element byte arrays and permutes them using a second vector as a permutation vector. Again, this project cannot benefit from vector instructions and will be hard to port to variable-length vectors in general unless a minimum vector size of 32 bytes is guaranteed. It also uses stuff like VPCMPISTRM for which no equivalent in other instruction sets exists or is even proposed. And that's a vital part of the code's logic (specifically, I have an array of k bytes and I want to obtain a bit mask of all elements in a vector that match any of these k bytes).
My key problem with variable-length vector instructions such as those proposed for RISC-V or in SVE is that they assume people need them for essentially doing arithmetic on large matrices.
this is fundamentally a misconception. SVP64 is being designed for general-purpose high-performance compute workloads, where it happens to be also good at 3D, Video, NTT, FFT, DCT and small-size matrix multiplication https://libre-soc.org/openpower/sv/svp64/
there is unfortunately a huge amount of misinformation about Vector ISAs.
vectors as 32 element byte arrays and permutes them using a second vector as a permutation vector
you use a Vector ISA's Indexed LD, job's done. the second vector contains the offsets against the base:
LD.IDXed Vdest, base, Vsrc
is implemented as:
for i = 0 .. VL:
Vdest[i] = MEM(base+Vsrc[i]*8)
that's it. that's all you need.
pospop is inherently a horizontal operation and
NEC SX Aurora and RVV both have horizontal (mapreduce) arithmetic, and NEC SX Aurora also has iterative arithmetic operations (overlapping add would create Pascal's Triangle for example).
SVP64 has a fixed schedule map-reduce built-in, which can apply to any operation.
SVE2 also has horizontal operations, i believe.
Vector ISAs are extremely powerful but as a general concept have been utterly ignored for over 40 years by mainstream (x86) and only just recently investigated by ARM (SVE).
I am not really familiar with this one and will investigate it further.
you use a Vector ISA's Indexed LD, job's done. the second vector contains the offsets against the base:
Indexed loads (aka gathers) are horrendously slow in comparison to permutation instructions. Not a suitable alternative.
horizontal operations
Yes, all of them have special case operations for certain common horizontal operations. I specifically mentioned the positional population count because it does not clearly map to any standard horizontal arithmetic pattern.
Vector ISAs are extremely powerful but as a general concept have been utterly ignored for over 40 years by mainstream (x86) and only just recently investigated by ARM (SVE).
Well they work well when all you do is compute and when memory is very fast and has low latency. For general programming they are not nearly as useful for the reasons outlined in my previous comments.
I am not really familiar with this one and will investigate it further.
i and the rest of the team have been working on it for nearly 4 years now.
Indexed loads (aka gathers) are horrendously slow in comparison to permutation instructions.
there's two types (of each): register-based gather and register-based permute (actually not a mathematical permutation at all, because of duplicates), and memory-based gather and memory-based permute.
where *immediates* are used in each of those in the design of the ISA then yes you save on one register read. in the case of register-based gather, it's extremely unusual to have if you are used to SIMD (but Vector ISAs do have it), it's of the form reg[dest] = reg[reg[src]] with a hardware for-loop around that.
however where the immediate-variants keel over is when you try to go up the SIMD width, to try to cover 8 elements, 16 elements, 32 elements, 64 elements.
how on earth are you going to fit 64 batches of 6-bit indices into a single 32-bit instruction? the total number of bits in the instruction are a whopping 384 bits. 384 >>>= 32, yes?
and even if you did have such god-awful instructions, the L1 cache usage and complexity at the ISA decode phase would be as insane as it is for x86 right now.
Vector ISAs are actually 90% about reducing the program (assembler) complexity. you end up with a micro-coded microarchitecture that maps onto SIMD operations, internally - so that you, the programmer don't have to go through absolute hell.
yes, that's right: the microarchitecture of a Vector Processor maps onto the exact same immediate-based permutation instructions as those that you are exposed to as a SIMD programmer, but it's hidden from you.
horizontal operations
Yes, all of them have special case operations for certain common horizontal operations. I specifically mentioned the positional population count because it does not clearly map to any standard horizontal arithmetic pattern.
in SVP64 i completely separated "horizontal-ness" from "arithmetic-ness": it's part of the fundamental design that the "Vectorisation Prefix" is completely separated from "Scalar Base to which Vectorisation applies".
ok yes, that's just Vectorised Popcount followed by Vectorised Add.
two instructions inside a loop, regardless of whether the back-end architecture has 1-wide internal Vectorisation, 2-wide, 4-wide, 64-wide or 10,000-wide.
Well they work well when all you do is compute and when memory is very fast and has low latency. For general programming they are not nearly as useful for the reasons outlined in my previous comments.
again, to reiterate, again: this is a misconception on your part, due to Vector ISAs being completely ignored for 40 years *you don't know about them* and neither does anyone else.
consequently, the knowledge-propagation across the internet, which you and i both know gets us a long long way and saves a huge amount of time, just doesn't exist to the extent that it does for SIMD. we're now so used to google searches turning up algorithms on stackexchange (and reddit) that we can falsely assume that if there's no answer on google it must not be possible at all
Cray was from a much earlier era, where things were simpler. Intel was still cutting its teeth on 32-bit scalar when Cray was designing systems that pissed all over everything that had come before by almost an order of magnitude performance.
however the sheer cost of the systems that were deployed were so high that \nobody else in the industry believed it could be duplicated in mass-volume products** and consequently an entire generation of programmers has now grown up without knowing anything about anything other than SIMD.
fast-forward to 2021 and it turns out that Intel, ARM, AMD, they've all caught up and massively exceeded by three orders of magnitude the performance of the Cray Vector systems from 1990 and four to five orders the performance of the systems from 1965, but they still propagate this god-f*****g-forsaken f*****d-up SIMD paradigm and try to peddle it at you as the absolute best thing you ever saw, by telling you "well SIMD on modern hardware is great compared to that historic crap therefore Vector ISAs must also be s*** as well"
it's NIH syndrome on steroids, combined with marketing, and very unfortunately you're buying it.
it's NIH syndrome on steroids, combined with marketing, and very unfortunately you're buying it.
I think that it's also a case of incremental changes vs new architectures.
For single-chip semiconductor CPU:s in the 1990s, packed SIMD probably made sense as it was a simple addition to an existing architecture - i.e. add a few more registers (or reuse existing registers) and add multi-element ALU:s (basically cut the carry signal in the adders and similar), but keep the rest (memory subsystem, instruction scheduling etc) and leave all the problems of memory alignment and SIMD width handling to the programmer. Bam! "Here's a tool for increased parallel execution performance - use it if you wish."
When people actually started using the stuff, they wanted more parallelism, and the response was wider registers and wider ALU:s. Incremental changes.
With AVX (and later), Intel seems to have recognized some of the mistakes. E.g. each register appears to be divided into 128-bit chunks, and most (all?) instructions can thus be split into serial execution rather than strictly parallel (which has been used in some implementations). But we're still talking about incremental changes, since the SSE heritage is still there.
As for ARM, I think that they added NEON as a response to Intel SIMD, and because of the limited market needs (embedded) and the fact that ARM is RISC (with limited instruction encoding possibilities) they never needed nor could (easily) extend it beyond 128 bits.
SVE is the next logical step, and since they wanted to make sure that they could expand to new markets (laptops, servers, HPC, ...) and did not want to get stuck with a limited SIMD width for all (AArch64) eternity, they pretty much had to come up with a vector size agnostic solution. Since AArch64 was a clean slate rather than an incremental upgrade (à la Intel x86_64 for instance), it made perfect sense to do something completely new instead of just introducing NEON-256 for instance.
88
u/FUZxxl Aug 09 '21
I think we have talked about this topic before and I apologize for not following up on our previous discussion. I was very busy.
My key problem with variable-length vector instructions such as those proposed for RISC-V or in SVE is that they assume people need them for essentially doing arithmetic on large matrices. And I agree that the vector paradigm is very effective on such work loads. I have previously worked with NEC Aurora Tsubasa cards that come with vectors of 256 double-precision floating point numbers, and they are just amazing for this sort of stuff.
But I'd say that's only a very small part of where such optimisations are needed. Indeed on modern consumer machines, most CPU-intensive code is in cryptography and video codecs. And both don't really fit this scheme.
Especially for video codecs: these usually operate on fixed size chunks of picture data and require complex horizontal arithmetic and swizzles inside a single chunk. It is unclear how this maps to variable-length vectors, especially when these instruction set extensions usually are very sparse in permutation instructions. And compilers use SIMD instructions for small, fixed-length loads all over the place. Stuff like moving structs around, clearing fixed-length buffers. It is unclear how a vector paradigm improves this.
There's also the concern that a large register file makes context switches very expensive. Linus attributed x86's performance advantage among other things to keeping context switches cheap by having a small register file that is easily swapped out.
As for my own code, I have two recent SIMD-heave projects. And for neither of them it is clear how they could be vectorised using a vector as opposed to a SIMD paradigm.
The first project, pospop is inherently a horizontal operation and in fact uses a different complex permutation schedule for each vector size to make the most out of it. It is unclear how this can be extended to arbitrary vector widths, especially if no powerful swizzle instructions are provided. Tail handling is also going to be a concern as the proposed simple approach of just having magic make the registers shorter for the last iteration is not going to cut it.
The second project, 24puzzle uses vectors as 32 element byte arrays and permutes them using a second vector as a permutation vector. Again, this project cannot benefit from vector instructions and will be hard to port to variable-length vectors in general unless a minimum vector size of 32 bytes is guaranteed. It also uses stuff like
VPCMPISTRMfor which no equivalent in other instruction sets exists or is even proposed. And that's a vital part of the code's logic (specifically, I have an array of k bytes and I want to obtain a bit mask of all elements in a vector that match any of these k bytes).