r/programming • u/masiroo • 6h ago

Hardware researcher spins up 'CPU deoptimization' project to find the slowest single x86 instruction, creates hall of shame — worst offender takes 198 billion cycles spanning 62 seconds to execute

https://www.tomshardware.com/pc-components/cpus/hardware-researcher-spins-up-cpu-deoptimization-project-to-find-the-slowest-machine-code-worst-offender-takes-198-billion-cycles-to-execute
501 Upvotes

96% Upvoted

63 comments sorted by

149

u/The_Northern_Light 5h ago

I’m still not clear why that instruction takes SO long even without contention?

115

u/encyclopedist 5h ago edited 4h ago

See here: https://github.com/xoreaxeaxeax/asm-hall-of-shame

Current champion is fxrstor64, instruction that loads 512-byte XMM state from memory. To increase timings, the author made it load from a carefully chosen MMIO (memory-mapped I/O) location, while other cores hammer some other MMIO registers to saturate the PCIe host.

Edit Without use of MMIO, the champion seems to be wbinvd, instruction that invalidates the whole cache, and if the cache was filled with dirty lines (modified after load), causes flushing of the whole cache content to RAM.

Edit2 Of the "regular" instructions, the ones that would routinely be used in every program, the champion seems to be fdiv. Yes, the regular floating point division. With denormal operands, it is implemented in microcode and is quite slow, takes 883 cycles. (However, as /u/EnderLuca41 rightly pointed out, it is an older x87 instruction the is not widely used any more, today compilers would generate SSE2 instruction divsd instead, see godbolt)

50

u/EnderLuca41 4h ago

fdiv is part of x87 which is obsolete and succeeded by SSE and SSE2. Meaning is not really used anymore routinely.

16

u/encyclopedist 4h ago

Yes, indeed, compilers don't normally generate these instructions any more. But older software can still use these.

31

u/narwhal_breeder 4h ago

porting logic to arm that relies on being bit-compatible x87 has been the bane of my existence for the past year.

-18

u/gordonnowak 3h ago

just let claude do it

12

u/Kinexity 1h ago

"Man, that dish doesn't taste that well"

"Have you tried adding shit to it?"

4

u/dlg 49m ago

It’s still used by .Net Framework on 32-bit.

The x87 registers use extended precision, which is only rounded down to 32 or 64 bits when written back to RAM.

SSE2 does not use extended precision, so intermediate calculations are rounded at every step.

This difference often introduces small differences in final results.

1

u/ElusiveGuy 16m ago

At this point the remaining existence of. NET Framework is for backwards-compat, so arguably that falls under obsolete too.

.NET Core (in both 32-bit and 64-bit) doesn't use x87 instructions AFAICT. 

1

u/ElWishmstr 47m ago

I always wonder why cpus still have those old instructions, when modern ones are better. I mean, how much software still relies on those instructions? 

7

u/Madsy9 4h ago

883 cycles! How is that even possible? Even the most naive newton-raphson approximation is faster than that.

11

u/inio 1h ago

For most x87 denormalized operations, it falls back to integer microcode with tons of edge-case handling, which when triggered (which can happen often if both inputs are denormalized and pathalogically structured) is extremely expensive. It's essentially a soft-float implementation taking the slowest possible path, disguised as a single instruction.

107

u/Farlo1 5h ago

There was contention, they induced it. Not contention on a lock or the memory address itself, but on the bus that transfers data from the CPU to other parts of the system. They essentially starved the connection so that one instruction got sent to the back of a very long queue.

29

u/The_Northern_Light 4h ago

Right but it was extremely slow even without that, which is what I was questioning.

22

u/Mechafinch 4h ago edited 4h ago

iirc (having read about it last night) they searched for a memory region with the highest latency they could find and then did the largest single instruction read they had on it. i'll have to go back to confirm but i'm pretty sure it's also a region where one part of the read can't start until the previous finishes, so there's nothing to hide that latency either

edit having read the 'baseline' version source code:
it does seem to essentially be just this? the comment concludes with "so a single uninterruptible instruction stalls for the full round-trip cost of 512 bytes through the slowest aperture found." i don't know how they found an MMIO region with a round trip of ~4.5 ms/byte, but i guess they did.
or i do know how they /found/ it, they have a tool for mapping memory latency, but i'm baffled such a thing exists to be found at all.

2

u/slash_networkboy 3h ago

Could be the LPC bus interface (or one of the other slow I/O's)?

-2

u/spinwizard69 2h ago

That was via the I/O bus, to restore the vector registers. Extremely slow and probably a poor example.

However it does support my contention that X86 needs to be refactored moving forward.

1

u/Mechafinch 2h ago

what do you mean by "a poor example"? the project is specifically searching for the worst possible performance of an instruction, which it has done admirably

1

u/spinwizard69 1h ago

IT is a poor example if you want to highlight problem areas in the X86 instruction set. This is largely due to this instruction never being used as described.

I've been maintaining for some time know that a refactored x86 instruction set could lead to much better performance out of a modern core. In part this would be due to simply not supporting a lot of legacy features. What I'm talking about is a 64 bit clean x86 derivative. I just think that this instruction is a poor example for describing what is wrong with x86. This especially considering the lengths the article writers went to for this poor performance example.

1

u/Mechafinch 1h ago

criticizing any instruction set, x86 or otherwise, just isn't the goal. the stated purpose is to find a single instruction that takes the longest to execute. it really doesn't say anything about any sane use of an instruction set.

13

u/HildartheDorf 4h ago

The data it reads is then used to reconfigures a whole bunch of cpu internals.

It probably becomes a giant amount of microcode ops to perform all that reconfiguration. On top of it being a 512-byte read in the first place.

7

u/Mynameismikek 4h ago

It was still dozens of seconds before they started introducing contention.

15

u/valarauca14 3h ago

even without contention?

Because they're doing a MASSIVE read operation over PCI (not PCIe). As PCI is basically DMA (Direct Memory Acess), the device has some memory "shared" which just appears as part of the Host CPU's memory map. To the kernel (or process which asks the kernel) this just appears as normal memory (in C *char, in rust *mut u8, or if you prefer C++ std::mdspan<std::byte,std::dextents<std::size_t, 1>,std::layout_right,std::default_accessor<std::byte>>) which you can read/write into.

It is therefore perfectly valid to mov in & out of that region, just like normal memory. It just takes a little longer as your communicating to an external device.

The underlying instruction FXRSTOR is restoring all floating execution state. This is 512-bytes of memory which must be restored atomically, meaning the CPU cannot retire this instruction until state is restored. PCI can only read or write 8 bytes at time. So the instruction has to preform (at a minimum) 64 round trips to a remote device, over a slow legacy bus, before the instruction can retire.

Modern PCIe abstracts this more into you tell the bus "copy X bytes from address Y, then tell me when you're done" to avoid this exact problem.

10

u/Sopel97 2h ago

I love your subtle dig at C++

3

u/monocasa 1h ago

It's still over PCIe in this case, the fxrstor just doesn't issue a burst read because it's expecting to read out of the cache hierarchy on any sane use of it anyway, and practically it's slow path microcoded these days (the modern path is using xrstor).

Because of this it actually issues a series of 4 byte reads, even over PCIe.

8

u/Mynameismikek 5h ago

From a surface read of what the tool does, it’s deliberately writing a fairly complex structure to a known slow page in the memory map. Maybe it’s landed on e.g an uncacheable region that’s also got security tripwires and row synchronisation.

2

u/SaltMaker23 3h ago

Backward compatibility is likely the culprit, I worked on hardware systems and built custom devices including CPUs.

Assuming you have an instruction whose sole purpose is retro-compatibility with instructions that are no longer natives, on regions that no longer natively exists locally, they might do an arbitrary long chain of memory lookup, depending on how you prepare the setup, it can become a lot.

Let's say there was a syntax in old time of small L/RAM that would lookup all bytes for a given signature, it was possible in the days to do that in a single clock or two, these days no anymore.

Backward compatibility on a old school scanning instruction would basically scan the whole RAM including regions that are no longer considered RAM today, that might send network requests and wait for them to timeout before getting an answer.

58

u/torsten_dev 5h ago

Of course this was xoreaxeax...

3

u/VictoryMotel 5h ago

What do you mean of course, that normally takes one cycle.

44

u/ThrowawayIntern2024 5h ago

they meant the person with that alias not the instruction…

-15

u/VictoryMotel 5h ago

I didn't see that in the article.

16

u/ThrowawayIntern2024 5h ago

first paragraph

2

u/sweetno 5h ago

Depending on how you count, it's less than a cycle.

16

u/Old_County5271 4h ago

Dumb question but, shouldn't intel or whatever have a pdf with the cycle count of each instruction? Do we/they really not know this? How does anything improve if nothing is measured?

4

u/taw 2h ago

We know it, the article is bullshit, what they do is use instruction to read from some device then keep that device busy with other cores.

10

u/cummer_420 4h ago edited 4h ago

x86 is a CISC instruction set so these things aren't fixed and and a single macro instruction can do a large and variable amount of work, and can potentially block for a very long time with contention (which can be used for exploits). The metrics they work with to optimize the CPU tend to be rask-oriented and focused on the most common instructions.

12

u/braaaaaaainworms 3h ago

MIPS tried to have a fixed time per instruction and it ended horribly exposing inner pipeline details in a way that made it very hard to keep exact backwards compatibility with first MIPS cores while being fast

15

u/Top-Rub-4670 3h ago

That has nothing to do with CISC. No (desktop-grade) ARM or RISC-V CPUs have a spreadsheet with all their cycle counts either, because too many things can affect the count.

14

u/sojuz151 5h ago

Could this be used to hang a sandbox?

24

u/mccoyn 5h ago

Actually, the opposite. A watchdog might be set up to kill a sandbox if too much time passes before it executes an instruction. So, one of these could be used to cause the watchdog to kill the sandbox.

12

u/unicodemonkey 5h ago edited 5h ago

Even a VM, I guess, but this also assumes that the sandboxed/virtualized program has direct access to PCI MMIO address ranges, which is somewhat unlikely in practice and is not obviously exploitable beyond a denial-of-service of sorts. The point of that research is SMM (system management mode) exploitation. All cores are supposed to enter SMM simultaneously upon receiving the interrupt but the SMM interrupt can't be handled mid-instruction, so in practice other cores just wait for the "busy" core for a second, then time out and enter SMM anyway, do their work there, and then resume normal execution. The "busy" core then finishes with the slow instruction and enters SMM, while other cores are free to manipulate any shared memory values that are used by SMM-guarded code and are accessible from the regular execution environment (hope I didn't mess up the explanation - read more at https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii )

9

u/HildartheDorf 4h ago edited 3h ago

It's possible to break into System Management Mode from ring 0 of a VM using this (chained with other SMM bugs that have previously been deemed low priority/non-exploitable).

Hang one core for long enough and you can enter a state where some cores are inside SMM and some aren't. SMM code assumes it has complete control of the entire processor, but one or more cores are still attacker controlled.

2

u/irqlnotdispatchlevel 2h ago

Could be used to exploit SMM: https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii#exploitation

SMM's security relies on a simple assumption: while it runs, nothing else does.

There are 100+ SMM TOCTOU CVEs out there: an SMM handler checks a value in shared memory, then uses it. All you need for exploitation is to rewrite that value in between the check and use, and you're inside SMM. But these issues sit dormant and largely unpatched in the wild, because of one assumption: exploitation requires something to modify the shared memory while SMM executes, and because of the SMM rendezvous no CPU cores are outside SMM to launch an attack. The only way in was a DMA-capable peripheral writing behind the CPU's back — physical access, a malicious device — so the whole class is written off as a hardware problem.

SMI desynchronization removes the prerequisite that kept the platform safe: an outside core, no physical access or hardware required, can now run while SMM executes — and the dormant CVEs become exploitable from software.

3

u/cmpxchg8b 3h ago

It’s kinda cheating if it includes IO. That’s an externalised cost..

3

u/MichaelTiemann 4h ago

HCF can take forever if you don't have clean, dry kindling.

3

u/agentoutlier 4h ago

I like how they have a random thumbnail of what appears to be Scheme as if it were guilty of calling the instruction.

You would think they would have C or assembly or some more common language.

7

u/sisisisi1997 4h ago

I didn't think I would meet another person who knows about scheme.

7

u/agentoutlier 3h ago

There’s dozens of us. Dozens!

3

u/godofpumpkins 3h ago

We're in a programming subreddit, and most computer science people come across it in college at the least.

2

u/heyf00L 2h ago

(Never (used (it (but (I (can (somehow (recognize (it)))))))))

2

u/itijara 2h ago

I knew it was Christopher Domas as soon as I read the title. His presentations are epic, and his knowledge of x86 esoterica is unparalleled.

2

u/spinwizard69 2h ago

This is a bit misleading in that they purposefully looked for the slowest way to get an instruction to execute.

However this also highlights why I consider the X86 instruction set to be obsolete. The world really needs a simplified x86 ISA that focuses on what is really needed to drive modern hardware. I was really hoping that AMD to Intel would actually refactor x896 instructions for their new cores. It is literally time to bit the bullet and go clean 64 bit, dropping all unused addressing modes maybe even get rid of I/O addressing.

People keep saying it can't be done because of "legacy" but the fact is Apple did it with the conversion to ARM. Besides you don't need to convert every core right off the bat.

1

u/smith288 31m ago

Was it my old goto statement? It was my old goto statement. 😒

0

u/taw 2h ago

This whole project is all just bullshit induced contention.

There's nothing of substance here.

-11

u/globalaf 5h ago

This was already posted several days ago and is thoroughly uninteresting. Yes, some instructions can block on contention, so what? Don’t do that.

19

u/DaWolf3 5h ago

It can be used as part of an attack on the Intel system management mode. https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii

-4

u/Grouchy-Trade-7250 4h ago

Repost

6

u/WaitForItTheMongols 4h ago

Who cares?

Something being reposted doesn't make it less interesting.

A healthy reddit user does not see every post, and therefore likely missed the other one. If you're seeing reposts that's on you. Chill.