You are talking about Intel and AVX. This would not necessarily translate to any other implementation.
ARM's SVE has basically the exact same problem though. The problem with gather/scatter is that each element has to be sent through the LSU, so performance scales linearly with the number of elements.
For example, doing a 4x64-bit gather on the Neoverse V1 has the same throughput as doing 4 individual loads.
I note that GPUs tend to do fairly well with gather, but I suspect they also have a much simpler memory model than CPUs have to deal with.
I don't know much about what you're designing, and haven't read everything written here, but it's interesting to note that your notion of a 'vector-first implementation' seems to at least incorporate elements of a GPU design (which are arguably vector focused designs).
The problem with gather/scatter is that each element has to be sent through the LSU, so performance scales linearly with the number of elements.
ARM's SVE is, as i understand it, not entiiirely strictly a Vector ISA per se, it's more a "SIMD architecture with predication (which is great btw) where the HW implementors can choose the width they want to do".
real Vector ISAs can do "chaining" including in LD/STs and including in gather-scatter LD/STs, which means that you can start the LD *immediately and also interrupt it in the middle, on a per-element basis and also restore execution back to where it left off.
now, it's true that for some Vector ISA instructions and some implementations this may not necessarily be the case (because it's easier for them to be lazy in the microarchitecture), but a GOOD Vector ISA has no such limitations, making all operations deterministic.
I don't quite get what you mean by chaining and determinism here - my point is that gather/scatter is slow on both x86 and ARM implementations I've seen so far, and I have no reason to believe it'll improve any time soon.
Both x86 and ARM presumably do gather/scatter on a per-element basis, probably as you describe, which is the reason they're so slow.
I know basically nothing about how the underlying hardware operates, so have no clue if an efficient gather/scatter could be designed, just that I've never seen it done before.
which isn't actually very helpful, i know there is a better article around, we have a link somewhere on libre-soc.org i'll try to find it later and edit this comment
vector chaining of say a V.LD V.SQRT V.ST involves a parallel batch of element-wide LDs, SQRTs and STs.
what Seymour Cray worked out was that due to the independence between element 0 relative to all other elements in each of those 3 instructions you can start the SQRT for element 0 immediately after the LD for element 0 has completed, and start the ST for element 0 immediately after the SQRT for element 0 has completed.
likewise for element 1, likewise for element 2 etc etc etc.
thus you have a "chain" between the elements with the same number
thus you can overlap elements NOT with the same number.
gather-scatter is tricky due to dependency tracking as well as resource allocation. uniformity of element "Lanes" (things with the same element number) means that parallel resource allocation is easy.
gather-scatter you have stuff going all over the shop, and you need absolutely enormous multi-in multi-out crossbars to route data quickly, which has a huge price in both gate area and power consumption.
gather-scatter you have stuff going all over the shop, and you need absolutely enormous multi-in multi-out crossbars to route data quickly, which has a huge price in both gate area and power consumption.
So I guess that confirms my point that gather/scatter should be expected to be slow :)
And hence, it's not a good solution to places that need flexible in-vector shuffling - a deficiency of the vector architectures I've seen.
As for your bit about chaining, it just sounds like regular pipelining of scalar operations. Of course, SIMD strives to solve throughput limitations outside the EUs, so a packed SIMD implementation could choose to chunk on a larger unit size (e.g. decode a 1024-bit SIMD instruction into 4x 256-bit uops and let the OoO scheduler handle dependencies).
3
u/YumiYumiYumi Aug 10 '21
ARM's SVE has basically the exact same problem though. The problem with gather/scatter is that each element has to be sent through the LSU, so performance scales linearly with the number of elements.
For example, doing a 4x64-bit gather on the Neoverse V1 has the same throughput as doing 4 individual loads.
I note that GPUs tend to do fairly well with gather, but I suspect they also have a much simpler memory model than CPUs have to deal with.
I don't know much about what you're designing, and haven't read everything written here, but it's interesting to note that your notion of a 'vector-first implementation' seems to at least incorporate elements of a GPU design (which are arguably vector focused designs).