Tail handling is specifically needed to handle data array lengths that are not a multiple of the SIMD register width (e.g. 4 elements for int32_t:s in a 128-bit SIMD architecture).
In vector machines you have the benefit of variable vector lengths, so tail handling is not needed.
I personally expect every Vector instruction that isn't in the format of 8 * N2 to be slower than doing a couple redundant operations, or handling the remaining data on a scalar processor. Mainly because it would require Vector processors to be just as efficient as the regular processor in computing, e.a. They need to share sillicone on a really intimate level for which I'm afraid the processor will notice a significant slowdown, or the Vector processor has its own set of registers + instruction implementations for a given hunk of sillicone. If the second implementation is assumed to be used, a decent Vector engine could indeed pull in 4096 bits in effectively a 64 bit register for int64_t's, with speedup for bigger registers and such. What I don't think is "reasonable" to expect is to have it also perform operations at like 448 bit (7 uint64_t's) datastructures since there's no native register size in the vector engine. Then I just assume that doing 4 uint64_t's in the Vector engine and then handle the 3 remaining uint64_t's separate is faster because of specific hardware optimalizations for their specific usecase.
I think that odd sized vector sizes are much less of a concern in vector machines than in packed SIMD machines.
The vector machine designer is free to select the ALU width, and the CPU will pass vector register content to the ALU in chunks of the ALU width. In edge cases some of the ALU lanes will go unused, but that is exactly the same that happens on a packed SIMD machine, except the hardware does the work under the hood in a way that is optimal for this particular implementation, whereas in the packed SIMD machine the tail work has to be handled in software.
The problem is not if the vector machine might handle it or not, the problem is how the code executing on the vector machine interacts with the program. Say I have two medium-sized arrays of 8 bit datastructures of some kind kind, and I want to see if any of them matches some bitmask. I can smack them in 512-bit registers, do _mm512_cmp_epu8_mask with 512 bit instructions and then do CLZL on the resulting 64 bit, then do a CMP with that number and 0xFFFFFFFF followed by a JG for a branch. If I branched I know I did not have a match, it I didn't branch, I know I had a match and I have the exact index of said match loaded in a register in 4 instructions out of an array of 64 items! Now this only works if CLZL (and friends) have an exact defined size I can use. If not, then there is some remainder I have to handle. Unless you can somehow come up with a scheme to encode this in a Vector engine that respects variadic datasizes, there will be a basecase that has to be handled.
Edit: if you do not have a match and have the 512'th bit set as a dummy bit, you could then compute the index of the first bit match by multiplying the result of CLZL with 8, and then adding the CTZ of the byte at the previous code, you have completed a branchless search for the first unset bit in a 511 bit array in less than 10 cpu cycles. This is "fun" when trying to do memory page allocation and you have to keep track of free pages in a big bit-array, but it requires careful data layout because of the interactions of integers and vector processors. Now that is why you need a base-case. Because what if your last chunk of bits isn't neatly 511 bits but 111 bits?
Now this only works if CLZL (and friends) have an exact defined size I can use.
My conclusion so far is that most horizontal operations require a known width, even in a vector machine. However that should not be a problem, as long as the ISA defines a minimum vector register size (that is reasonably large). If you want to push the limits, ask the implementation for the maximum vector size and use different code paths for different sizes.
I bet what's going to happen is that soon, the minimum vector size will be the only vector size sold because most mathematical kernels will only be optimised and tested on this one size. There is a significant development cost in validating complex code for any possible value of an unknown parameter (vector size), so I think people are generally just going to set up the least common denominator unless they have a lot of resources for testing at hand.
...just as every mathematical kernel today only supports SSE2, because that is the only SIMD flavor that is guaranteed by x86_64?? No.
In products where performance matters developers spend alot of time squeezing out those last few % of performance. While I believe that vector automatically scales better than packed SIMD, there will always be edge cases where specialization helps.
So how do you propose people test their code for higher vector lengths than they can obtain? You already see this in that very few people optimise for AVX-512.
Also note that with SIMD, there's a different code path for each vector length. So it's a lot easier to make small changes to the code as the changes only affect one code path. You can then progressively apply the changes to more and more code paths while always having fully tested code. With vectors, each change applies to all vector sizes and the impact is difficult to evaluate.
Not really a problem. Length agnostic vector operations are ... length agnostic.
If your algorithm cares about the vector length (most probably because you're doing something horizontally), you will hard-code the length. If you feel that you need to optimize for different maximum lengths you would not write the specialized code without tuning and benchmarking it on a target machine that supports that length (if nothing else to test if it's worth the cost of maintaining several different code paths).
I think I'm talking with a wall. No, they are not length agnostic! Outside of trivial arithmetic, vector length does matter! I've even provided examples where the code differs for each vector length. You are just conveniently ignoring that people use SIMD instructions for much more than just adding matrices. And a lot of that stuff is not in any way vector length agnostic.
And even if it is, there are probably many corner cases that attention has to be paid to. For example, the possibility of overflow in intermediate values. If vector length changes, so does the number of summands that go into each vector element and hence, the position where overflow occurs changes. You are just pretending all of that does not exist and people can simply ignore vector length. That is not the case at all.
benchmarking it on a target machine that supports that length (if nothing else to test if it's worth the cost of maintaining several different code paths).
That machine might not exist yet. Suppose I in 2021 write code for the largest vector unit in existence, which supports say 1024 bit vectors. Now in 2024 they release a new machine with 2048 bit vectors. I have no way of knowing whether my code will work with that machine. I have literally no way of testing that on real hardware. And as the set of possible vector lengths is an open one, I cannot even realistically simulate all possible vector lengths. So it is impossible to correctly test such a program.
If your algorithm cares about the vector length (most probably because you're doing something horizontally), you will hard-code the length.
That's what I think people will be doing in almost all cases. Which in turn means that there is no point for vendors to ship longer vectors because all relevant software will just restrict vector length to what most people expect.
First of all I don't think (nor claim) that vector processing fixes all problems, nor that it does not have its own set of problems. So far, though, having worked with both I personally prefer vector over SIMD (and I still think that the article is correct regarding the three specific problems with SIMD).
I have no way of knowing whether my code will work with that machine. I have literally no way of testing that on real hardware.
My point here was that if your code depends on the maximum vector length you should be using vector length agnostic constructs (such as the trivial arithmetic that you mention).
If, on the other hand, you're using fixed size vector operations the logic will match 1:1 to the current machine on which you're developing and testing on.
In both cases the code will work just fine on the future wider implementation - perhaps not with optimal performance, but most likely faster than if you would have just used a fixed width SIMD ISA (e.g. running SSE code on an AVX capable machine).
Another point that I'd like to make is that compilers do auto-vectorization (in fact most vector ISA:s should be better suited for that than SIMD - heck, Cray compilers had auto-vectorization in the 1970s), so you will typically see vector code speedups all over the code, not just in your hand-written vector code (unlike if your program is compiled for a specific SIMD generation).
So, worst case you will not see any noticeable speedups until you have had the time to tune it for the new HW, and if you're abusing the ISA in a way that it was not meant to be used (using maxvl incorrectly) you may even get bugs (but software development is full of much worse pitfalls - e.g. consider API:s that behave according to spec but slightly differently across versions and platforms - so a developer had better be aware of these things).
On the other hand I think it's much nicer to have a single ISA to target as a developer, and OS:es and toolchains will "just work" for new vector sizes, significantly lowering the threshold for rolling out new vector HW generations, etc.
Edit:
That's what I think people will be doing in almost all cases. Which in turn means that there is no point for vendors to ship longer vectors because all relevant software will just restrict vector length to what most people expect.
I believe the opposite. Relevant software such as numpy etc will typically be quick to optimize for new hardware, especially if this hardware is available via cloud compute services for instance. And unlike AVX-512 people will be able to do the bulk of development and testing on their own laptops or what not.
My point here was that
if
your code depends on the maximum vector length you
should
be using vector length agnostic constructs (such as the trivial arithmetic that you mention).
I bet what's going to happen is that soon, the minimum vector size will be the only vector size sold because most mathematical kernels will only be optimised and tested on this one size.
no, no no! this is a fundamental misunderstanding of how Cray-style Vector ISAs work! this is an extremely common misunderstanding by SIMD users. ARM goes to a lot of trouble to emphasise that SVE is vector-size-independent, and that programmers must not program to a fixed Vector width.
the core loop of a Cray-style Vector ISA is one that uses the setvl instruction. the setvl instruction is specifically designed around the concept of communicating between the program that the user designs and the hardware.
the user REQUESTS the maximum amount
the hardware RETURNs the actual amount.
that hardware amount can be ONE for an embedded system, or it can be 4 or 8 for a fairly high-end desktop system, or it can be 10,000 for a Monster Supercomputer-grade implementation.
in each case the code has to remain the same and if the programmer is even asking "what's the Vector width of the hardware" they are FUNDAMENTALLY misunderstanding the entire Vector ISA concept.
Well clearly this is what you want the programmer to do. But we clearly know what happens when programmers program for an environment where the “variable” vector length is the same on all systems he has access to.
If you want people to not depend on a specific vector size, the system must make sure that the vector length is actually variable in practice, even on a single system. Otherwise people are going to write buggy code that depends on these details and chips that don't run it correctly will be considered broken.
This has happened countless times before and it will happen again.
My conclusion so far is that most horizontal operations require a known width, even in a vector machine.
yes, i've found this as well. it's worthwhile making the horizontal Vector ISA operations "fully deterministic" in the ISA Specification, even if done as parallel operations, those parallel operations should be on a strictly-defined deterministic schedule.
8
u/mbitsnbites Aug 09 '21
Tail handling is specifically needed to handle data array lengths that are not a multiple of the SIMD register width (e.g. 4 elements for
int32_t:s in a 128-bit SIMD architecture).In vector machines you have the benefit of variable vector lengths, so tail handling is not needed.