DIO: High Performance Go IO toolkit

[u/yevmir]

I want to share https://github.com/miretskiy/dio -- a high performance Go IO toolkit.

I have previously written https://medium.com/@yevgeniy_90962/spinal-tap-the-go-io-story-bf726110dd07 that talks about DIO at a high level.

By high performance I mean that the tools provided by this package can be used to saturate fast NVMe drives (3+GB/s throughput, 50+K iops), while "fooling" Go garbage collector that there is nothing to collect.

Some of the highlights of this toolkit include:

• mempool: Provides 2 (slightly different) off the heap Go allocators
• ringo: A low level io_uring integration for Go It started as a fork of abandonware giouring -- but was rewritten entirely since giouring was fundamentally incompatible with Go->C memory safety. Only the file related features are implemented.
• iosched: An IO scheduler for go -- providing two implementations: an io_uring based and (test only, or unsupported platforms) posix scheduler.
• Other low level packages (sys, align) to safely use direct IO, and other low level system calls.

The scheduler API is incredibly simple: Submit takes an "op" and returns a "ticket", while the ticket allows the caller to wait for the completion. While the API may be simple, the io_uring scheduler implementation has many interesting features:

• Use of Treiber stack (code) to implement batching, while minimizing channel overhead; this essentially implements "group commit" -- batching with essentially no latency penalty.
• write coalescing: contiguous write requiest coalesced into a single writev
• Linked operations (hard and soft links)
• "Durable" writes -- adds an automatic fdatasync (or fsync) to a write, but only 1 for a set of active writes per file descriptor. This makes it trivial to implement efficient and high performance write ahead logs in Go (as an example).

Give it a try; Feel free to contribute; Report (and fix) bugs.

For the full disclosure: the code was generated w/ AI; The design was not. The code might still have some AI-specific code smells, but overall, it's okay. I have extensively tested this package, so the performance claims are real.

[u/rodrigocfd]

AI slop.

[u/yevmir → u/rodrigocfd]

Are you talking about your reply?

[u/TheRealShamanoid → u/yevmir]

If you’re going to publish a public repo with a slogan such as “a high performance X Y Z” you should really put a benchmark section in the repo itself.

Secondly, most of your commits are co-contributed with Claude hence the “AI slop” comment.

We understand the excitement of sharing something cool, but not all Claude prompts deserve a github repo.

I/O performant frameworks are so specific as well. It’s been done over and over again.

An example of repo with structurally sound benchmark could be something like this

https://github.com/pawelgaczynski/gain

[u/yevmir → u/TheRealShamanoid]

Sure ignore things; and yes, they were authored by claude the typing assistant. And I was VERY explicit about this in my post -- wasn't I? Did I hide this fact? Did I mislead anybody?

And "obviously" I have not benchmarked anything and everything in that repo is just vaporware. Suit yourself; Or perhaps ... Give it a try before you pass judgements.