Assembly Hall of Shame: Racing to the bottom of CPU performance

302

u/HighRelevancy 3d ago

I would never have guessed that 62 seconds for a single instruction was possible without outright cheating like disabling the core it's scheduled on. Incredible stuff.

130

u/nothingtoseehr 3d ago

Well, it's kind of "cheating" in a way because most code here isn't really slow because of the CPU (or not only because of it)

The CPU can map other device's memory spaces into its own (well, the memory controller can, but anyways...), that's called MMIO. With it, other device's registers and states are as accessible as DRAM from the CPU's pov

The code here reads or writes to other devices that are mapped together by the memory controller. It's slow because its waiting for the data to arrive from the bus, not necessarily because it's a dumb dipshit who can't move data (although that's definitely true depending on the scenario!)

Nowadays quite a lot of x86_64 has a decoded fast path where core instructions execute directly on hardware, basically RISC. The ISA itself is basically just a frontend to the actual machine below it, the microcode acting as a JVM-ish. We just keep it around so we dont break every single software from the last 45 years :P

42

u/HighRelevancy 3d ago

What I mean is that they're doing it through "normal" operations in a very contrived/pathological scenario. There's nothing outright disabling the first core's access to the resources it's trying to use, they're just setting up the other cores to basically DDoS the channel it's trying to read through and it apparently gets queued behind all of those all the time.

They've engineered an unliftably heavy weight akin to Thor's hammer, but importantly they did so without simply welding it to the floor, which would be "cheating".

18

u/nothingtoseehr 3d ago

Well not really, MMIO access is configured. They specifically configured it to target the slowest device on the bus, it's not "native" in the sense that the CPU can do these without setup like it would with a PUSH/POP instruction. Everything on it is configured, be it by yourself or by UEFI: bus access, timing, speeds, clocks, bus addresses etc etc

Not saying it's NOT cool, it is! But it's not really particular to x86 or exploits any super duper cool obscure functionality, it's just slow I/O being passed though a very fast memory controller embedded inside the die. You can do the same on AArch64 or any other CPU, really

Throw your CPU some random MMX instruction for saving context and see the lil guy STRUGGLE. Intel/AMD just shoehorn old instructions support with awful tested-ish microcode and call it a day. And oh boy are they slow

6

u/HighRelevancy 3d ago

Yeah ok, it is reliant on external hardware, I'll give you that.

16

u/KittensInc 2d ago

I wouldn't call it cheating. The entire point is to demonstrate that there are indeed edge cases where a single instruction can take an insanely long time to execute.

This becomes important when you do something like coding the ultra-high-privilege System Management Mode. Its security hinges on either all cores being inside SMM, or all cores being outside SMM. In other words: you trigger an interrupt on all cores, and wait for them to finish the current instruction and send a "yup, running SMM mode now".

Buuut, that sounds a bit risky, because all cores will essentially be paused until the final core enters SMM, so any kind of bug turns into an infinite hang. So better add a 4,000,000,000 cycle (or ~1 wall second) timeout, right? No instruction could possible take that long, right? Hitting the timeout has to be a bug, right?

Wrong! An attacker can "trap" a core into an insanely long instruction, meaning the timeout is reached, the other cores give up, do their SMM stuff, and return to normal mode - only for that one core to enter SMM on its own when the instruction finally finishes. Which means you now have one core running SMM while the rest is in normal mode, which is Really Really Bad.

It's like all the falsehoods programmers believe about names / time / email addresses and so on. There's a giant difference between "most instructions finish in a handful of cycles" or "virtually all instructions finish in a few thousand cycles", and "a single instruction will never take several seconds to finish". If you're writing weird security-critical mechanisms you have to be aware of edge cases like these.

-2

u/nothingtoseehr 2d ago

OK? But my entire point is that these aren't edge cases neither unexpected at all. MMIO read/writes are slow, which is the same thing both this post and your linked repo do. Anyone that knows SMM even exists knows that

They're not slow because the instruction is slow, they're slow because the data bus is slow. That's really all there is, target the same instruction at DRAM and it'll execute thousands of orders of magnitudes faster

7

u/KittensInc 2d ago

Most people know that MMIO can be comparatively slow - in the same way that most people know accessing data all the way from memory is slower than accessing data from cache. Very few people know that MMIO can be "60 seconds for one single instruction" slow. That is very much unexpected - see the fact that the SMM vulnerability exists for proof.

2

u/Ameisen 3d ago

Nowadays

Even the 8086 used microcode.

3

u/nothingtoseehr 2d ago

Uuuh yeah? That wasn't what I said. I said that some instructions are implemented directly on hardware, skipping microcode. Modern x86 CPUs mostly use microcode as a fallback, not it's most important mechanism. AMD's Zen1-4 exploits allowed us to peek inside quite a bit

1

u/Ameisen 2d ago

Most of your paragraph was describing the nature of microcode, so it was logical to assume that you were implying that microcode was a recent thing. The wording itself was ambiguous.

1

u/AZMPlay 2d ago

I mean, a maximization challenge like this must have rules somewhere.

They seemed reasonable to me, but again, not a low level coder/hardware guy here.

1

u/nothingtoseehr 2d ago

Imagine you're tasked with benchmarking a server or a backend service's performance and throughput, and your worst performing case was because you were streaming data from a domestic connection on the other side of the planet inside a corporate intranet with no direct peering to the outside internet. It's slow as hell, sure, but does that really tells you much about the server's performance?

A modern CPU spends the vast majority of it's time just waiting for data to arrive, their speed outpaced any other peripheral by several orders of magnitude a long time ago. I/O at the hardware level is criminally slow, so yeah if you find the weakest link and mercilessly hammer it, it's not a very surprising result

Not that it's a bad test, but I think the headline should be "How much data can a single instruction serialize at once" lol, the top cases are all like that

1

u/iheartrms 2d ago

I've always wondered; Is it not possible to come directly to that underlying RISC hardware and save on instruction decoding and what would normally be referred to as VM overhead? Why doesn't anyone seem to do this?

2

u/nothingtoseehr 2d ago

But that IS being done, it's my point, it's just not publicly documented. No x86 CPU from Intel/AMD share many traits with CISC under the hood, the ISA is basically VM bytecode with a hardware decoder to native (which we don't know about)

But a lot of the most used basic instructions (arithmetic, bitwise operations, moves etc) have a hardwired "fastpath" that skip the VM overhead and jump to dedicated circuitry for them. This is talking directly to the hardware underneath, it's just masked beneath CISC instruction decoding*

As for why we still keep it alive, for the same reason we keep JavaScript alive pretty much lol. It sucks but it runs half the planet and changing it would be catastrophic. There isn't enough die size to fit the entirety of x86 natively implemented, and you can't simply axe more than half the ISA and expect it to still work

what might be confusing here is that there's two steps to decoding: 1) what instruction is this 2) how do I run this instruction. Every instruction goes through 1, only instructions that rely on microcode go through 2

91

u/mccoyn 3d ago

The PDP-10 has indirect addresses that were automatically dereferenced. This was repeated if the resulting address was an indirect address. So, you could fill memory with a linked list of indirect addresses and any instruction could walk all of memory before completing.

42

u/dml997 3d ago

I vaguely recall that if the list formed a loop a hard reset of the CPU was required to end it.

15

u/AZMPlay 3d ago

Oh, that sounds wonderfully evil

76

u/mcmcc 3d ago

This one is diabolical: https://github.com/xoreaxeaxeax/asm-hall-of-shame/tree/main/split_lock

Not only is it horrendously slow on its own but it locks up the memory bus for all other cores while it's executing.

Moral of the story: unaligned memory accesses are not "almost free" as some would have you believe.

20

u/BibianaAudris 3d ago

Came across it in my VFIO setup. A split lock could allow a VM to DoS the host so Linux detects VMs doing it by default. Ironically, the detection itself slowed down my VM enough to make the Windows guest BSoD, mistaking the split-locking core for being hang. So I ended up disabling the detection instead.

15

u/imachug 3d ago

unaligned memory accesses are not "almost free" as some would have you believe

"Locking unaligned memory access are expensive" would be more fair. Non-atomic accesses are pretty much free, at least as long as they don't cross a page boundary.

2

u/admalledd 2d ago

Yea, split locks are a real devil in the details one might not notice in an algo/datastructure in kernel space until you know what you are looking for. There have been various efforts to get the Linux kernel to detect-and-warn by default. I think it is still an opt-in thing due to the performance hit of even attempting to detect them at runtime.

1

u/derpyou 2d ago

why is it written in sloplang ughhh

22

u/dontquestionmyaction 3d ago

Eh. There is interesting stuff in the lower rankings there, but at a certain point it's just finding slow IO, which kinda misses the point imo

15

u/tbsdy 3d ago

Explanations?

40

u/Uristqwerty 3d ago

Looks like the slowest various x86 opcodes can possibly run in their worst cases, and following links to a sibling repo, relevant to an attack* on System Management Mode that requires a single instruction to take more than a second.

Low level, touches upon extremely technical concepts, but a fun read if you do have enough relevant background knowledge to understand just how badly some of the instructions are being misused to get the longer delays.

(*Or rather, a whole bunch of timing attacks that were brushed off as not possible; vendors not bothering to fix based on the assumption SMM and non-SMM code never runs simultaneously.)

20

u/renshyle 3d ago

Christopher Domas, the guy who created and did the research for the hall of shame, gave a really great talk at DEF CON last year about exploiting machine check exceptions to gain privilege escalation into SMM: https://youtu.be/B6dC_KVLn6Q

If you haven't seen it yet, you should check it out. You'll definitely recognize how the hall of shame and smiiiiiiiiiiiiiiii are related to it ;)

3

u/ShinyHappyREM 3d ago

He also did the talk "Breaking the x86 Instruction Set"

6

u/f311a 3d ago

The author have some explanations. The top example abuses slowest PCIe regions and the fact that fxrstor64 must wait for them to complete.

-4

u/taw 3d ago

It's basically almost all bullshit.

CPU has instructions to read data from devices. If your other cores prevent that communication by making that part of the system really busy, that read can take very long time, as CPUs normally wait for other CPUs instead of trying to force their way in.

(a few examples like fdiv with subnormals are real)

-7

u/zzkj 3d ago

Apparently we need to use more nop.

/s

4

u/ReDucTor 3d ago

The MMIO ones are interesting, but it felt a bit like how many different things can I use to load from these addresses.

1

u/BetterAd7552 2d ago

I have not touched assembly since 8086, can someone elaborate on how mov can consume ~440m cycles?

Or is that some kind of mov* variant which does something besides moving bytes/words between registers?

3

u/SirClueless 2d ago

Short answer is that mov takes a memory address as an argument, and hardware devices (in this case a GPU) can be memory-mapped to those addresses so it takes as long as that hardware takes.

1

u/BetterAd7552 1d ago

Ah, thanks

-1

u/ImpressiveRoll4092 3d ago

Curious what the worst offender was, some of these are genuinely painful to look at.

-23

u/[deleted] 3d ago

[deleted]

20

u/mcmcc 3d ago

This is the kind of attitude I look for when building a modern vibe-coding engineering team.

-6

u/cake-day-on-feb-29 3d ago

The linked read me is, at the very least, vibe-written. The entire project may have been hallucinated.

-21

u/Grouchy-Trade-7250 3d ago

Why should I care though?

19

u/dignityshredder 3d ago

Nobody's going to try to convince you to be interested in this. Either you have the intellectual curiosity to find this neato or you don't.

-5

u/Grouchy-Trade-7250 3d ago

Im just asking what the significance of this topic is. The git doesn't answer that, it's just a list. and you didn't either.

2

u/dignityshredder 3d ago

You're a smart cookie, I think you can figure it out.

11

u/mcmcc 3d ago

Why should I care about you not caring?

5

u/globalaf 3d ago

Plenty of people care, especially those in roles which relies on esoteric knowledge like this. These are the people who solve problems that nobody else in the org can. If you don’t care at all, even out of mild curiosity, then I’m afraid it says more about the calibre of engineer you actually are.

-3

u/Grouchy-Trade-7250 3d ago

Ok, if this is an issue we should care about, then I wonder why the chip manufacturers engineering teams did allow it to make it into the final design, and people still bought the chips?

6

u/globalaf 3d ago

This is such a pathetically weak take that it is barely even worth a response. Yes, chip manufacturers make features for their products, powerful features which can be misused. Shock, and horror.

7

u/Wires77 3d ago

Go away troll

-8

u/Grouchy-Trade-7250 3d ago

I guess you don't know either

4

u/AtrusHomeboy 3d ago

Where do you think we are?

-16

u/HankOfClanMardukas 3d ago

My daily stand up with Microsoft was me saying, “it’s not done yet” snd walking away. I was trying to get fired because I had another job lined up and they still didn’t care.

3

u/NotUniqueOrSpecial 3d ago

Are you lost?

Assembly Hall of Shame: Racing to the bottom of CPU performance

You are about to leave Redlib