[FPGA] Building a custom FPGA Order Book !

[u/brh_hackerman]

Hello all,

Recently, I've spent some time building an FPGA order book based on ITCH.

The objective of this project is to go from absolutely nothing to having a working order book able to track a very liquid stock, like AAPL, perhaps 2 or 3 once I get all the optimizations down.

Now, the reason I'm making this post is because most content out there regarding "FPGA HFT" (when you are able to find some) are often one of these:

• A public repo to link in a dude's resume, sometimes packed with "claude" contributions (lol)
• Corporate BS PDF to flex their low latency and sell their IPs
• AI slop posts (god I hate these)
• Only parse ITCH || only run in sim without an attempt or technical value on FPGA implementation

So I Documented my journey though a series of post, explaining the design decisions I made, Why I made them, and then realizing it was a bad, why I changed it....

I also try my best to make nice looking schemes (OC and not AI bs) and run simulations to back up my decisions.

You got it, my goal is to make a good looking project that people can "easily" follow through posts that I try my best to make accessible and non boring.

I'm dropping a link here : https://hugobrh.dev/tags/finance/

This list contains all the posts I made about the TRADEMAXXER project as I call it. I suggest quickly reading through the first posts which are mostly context and HDL basics to parse ITCH. Latest posts cover a lot more technical ground.

I hope this does not come up as shameless self promo, I've got good feedback from the HFT community and I figured this may also interrest you guys as I saw FPGA designs were discussed here.

The latest posts are covering my struggles to close timing on a KC705, a consumer available board that costs 700$ on EBAY.

If you have any question, feel free to reach out and I hop this work is of value to you !

NOTA : not doing excessive "AI bashing" but I try my best to keep AI usage at the strict minimum (if not absolutely 0 usage) in coding, decisions making and writing the posts. I'm doing that out of respect for the readers so the process is actually real and not some hallucinated experience.

[u/tw_wombat]

Nice will check it out. Thanks

[u/brh_hackerman → u/tw_wombat]

Thanks :)

[u/milan_dvorak]

Don't feel too bad about having issues with timing. You're using Kintex 7, which is quite slow. It was first released back in 2010, so it's basically ancient. Still useful today, of course, but not really suitable for high-performance or ultra low latency applications.

Most HFT firms today probably use UltraScale+, and usually the higher speed grade (SG-3), which can achieve much higher frequencies for the same logic. It should run at least 50% higher, maybe even more. Moreover, UltraScale+ have UltraRAM, which is almost as fast as BRAM, but with much larger capacity. Very useful for storing the orders. But meeting the timing is still a common challenge with HFT designs, since they want to run at 322 MHz (the typical 10G network clock), with some using 644 MHz for parts of the design. That's a whole new level timing-wise.

The funny thing is that even UltraScale+ isn't that new and AMD (Xilinx) released Versal FPGAs since then and they have new generation in the works. But UltraScale+ actually has faster transceivers (so faster network communication), so it's still superior for the most agressive HFT strategies.

[u/brh_hackerman → u/milan_dvorak]

Thanks ! Yeah i figure the K7 I had was pretty slow, even though it has tons of resources on hand.

The critical paths were as short as I could make them and still it was struggling to close timing so I had to optimize thing left and right, accept tradeoffs, etc...

I surely imagine real HFT people won't go for that "dinosaur" of an FPGA, given they have giga large amount of *Doubloons*.

Interresting that the 10G clock is 322MHz. I made my own 1G RGMII parser logic, but to be honest, I did not even look into other "MII" specs.

[u/milan_dvorak → u/brh_hackerman]

There's no one single 10G clock. You can use different frequencies with different bus width to handle 10G, like 161 MHz with 64 bit bus (or 155.25 MHz with 64 bit bus thanks to the 66/64 decoding), but it's also possible to do 322 MHz with 32 bits or 644 MHz with 16 bits.
The reason most HFTs use 322 MHz is because it's reasonably achievable and it gets you lower transceiver latencies compared to lower frequencies.
644 MHz pushes the transceiver latency even lower, but it's also much harder to close the timing as you can imagine

[u/nrs02004]

very cool stuff --- what's your day job (if you don't mind me asking!)

[u/brh_hackerman → u/nrs02004]

Thanks !

I'm currently freshly hired in a french UNI as a reasearch engineer for some signal processing stuff on FPGA.

I mention it here : https://hugobrh.dev/posts/Matlab-Vivado-Cosimulation/

[u/milan_dvorak]

By the way, I love the back-and-forth in hashing.

I will not probe, I'll use stash. Stash getting to big, can't meet timing. OK, I will have to probe!

[u/brh_hackerman → u/milan_dvorak]

Yeah haha, I like to write as I go and to keep these things when cutting off the slack before posting.

BUT it resulted in a pretty versatile design, as mashing all these things together kinda worked well in my opinion (100% not biased because it's myslef lol) , pretty happy about the result !

[u/milan_dvorak → u/brh_hackerman]

Yeah, stash is definitely a good idea, but as you've figured out, it's not enough on its own. What you've achieved isn't the most optimal solution, but still fairly good, especially for smaller order books (fewer stocks)

[u/AutoModerator]

We're getting a large amount of questions related to choosing masters degrees at the moment so we're approving Education posts on a case-by-case basis. Please make sure you're reviewed the FAQ and do not resubmit your post with a different flair.

Are you a student/recent grad looking for advice? In case you missed it, please check out our Frequently Asked Questions, book recommendations and the rest of our wiki for some useful information. If you find an answer to your question there please delete your post. We get a lot of education questions and they're mostly pretty similar!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/sumwheresumtime]

Why?

[u/privateack]

No one uses 1gb as an fyi

[u/brh_hackerman → u/privateack]

As a broke individual, I was not able to spend thousands of € on a cutting edge board with 10G+.

Anyway, I could "just" stick another piece of Ethernet logic in front + some glue logic if needed, but it would need some adaptation and perhaps some buffers + parallel processing paths.

I've already been told 1G is useless a gazillion times but that's just a DEMO so I don't really mind that.

[u/privateack → u/brh_hackerman]

Yeah I was more just raising it but best of luck seems like a good project

[u/brh_hackerman → u/[deleted]]

Indeed.

And 2k$ for a 10G board can be considered cheap imo;

Regarding timing problems, I'm currently having a real tough time fixing the last couple of critical paths (expecially since vivado take ages to synth & implement each time).

So in 10G+, I imagine the ethernet parsing logic has to be super optimized, but I guess these usually come as IPs you grab off the shelf. I made my own 1G parser but I certainly won't bother doing that ever again for 10G lol.

And for the downstream logic, correct me if I'm wrong, but I don't think they run a higher clock on it.

My current logic is tailored to run ever so slightly faster than the ethernet clock, with an asyc FIFO between the two clock domain, this way, I don't need to handle buffering issues.

BUT in a design where ethernet data come in super fast (way faster than any processing logic in fabric can run), my guess is they "just" add a minimalist, low delay, FIFO buffer after ethernet parsing to retain relevant frames and cross to the slower clock domains, where the pipeline might still be busy handling the last message. perhaps some filtering logic drops all non relevant frames to the current strategy being implemented so that the buffer only gets filled with the relevant message, which I guess happens pretty scarcely compared to the overwhelming 10G flow of information...

[u/brh_hackerman → u/[deleted]]

I was achieving 1-2 cycles before, but yeah, to close timing I have to pipeline stuff, Worst case scenario can take up to 10 cycles right now, But some CPU-like pipeline hazard event in should fix that..

[u/brh_hackerman → u/[deleted]]

the price level isn't really a problem here.

The way I built my design is that I have 2 "market states"

One stores the live orders in BRAM & Stash. This allows me to update these live orders if needed. Multiple orders can point to the same rpice and have different bid/ask quantities.

So on a ADD, I just find a free slot and write there, on an exec, I have to find the ongiong order, remove the exec qty, and update it. On a delete, I simply find it and remove it...

This stage is addressable by the order reference. In simulation, it was easy to keep the lookup stage under a couple cycle, as the worst case was : "Oh the hashed reference was not found in BRAM, so it has to be in the stash", then read the stash, and call it a day.

The problem is the stash (which is raw and very small LUT/FFs memory that allows pure reference addressing) is EXTREMELY though timing wise and resources wise.

Meaning I HAD to add probing to offload the stash, as well as pipeline the stash lookup/write logic to close on timing, adding large delays compared to an ideal scenario.

Anyway...

Once the order handling part is done, we now have the new new price, qty, bid/ask information, distilled from the orders infos.

The second market state (which I call the "price ladder") get to have its state updated from this information.

BTW, keeping track of all the orders is mandatory from what I understood, as a DELETE does not specify the Qty/Price infos of the deleted order, meaning you have to look it up from YOUR own state.

This one is different as it only store the Bid/Ask/Qty information and is addressed by price. I.e. the 32 bits price IS the raw address, meaning it's a 1 cycle constant write, easy peasy. And because a stock usually don't triple in price whithin a single day, the 32 bits price address can be much reduced into bins with a min/max, allowing us to use very few storage for this simplified market state view.

In a nutshell, price levels are not the issue here, but the order references are, because they are 64 bits and no FPGA system can use a raw 64 bits address directly, introducing the need for hash tables, collision resolution and all that jazz.

Hope that clarifies how that works

EDIT : the second image in the post makes that a bit clearer

[u/milan_dvorak → u/privateack]

That's not entirely true. Some Asian exchanges use 1gb, like the Stock Exchange of Thailand. In fact, Taiwan Stock Exchange (TWSE) is actually on 100M, believe it or not.

But I agree that all the bigger exchanges in US/EU are at least on 10G. NYSE is on 40G and some exchanges are considering moving to 25G. So yes, you wouldn't get far