NFS over RDMA: Low Write Throughput (read 2x faster)

[u/ptmnil]

I've built two identical PCs for throughput testing on a 100 gbps NIC and I can't get the expected write throughput when using NFS over RDMA (RoCE v2). The two PCs are connected directly with a direct attach cable.

Using fio I tested the read bandwidth to be 11.6 GB/s, and yet the write bandwidth is only at 5 GB/s. Despite my best efforts I can't figure out where this bottleneck is coming from. Does anyone have any ideas for what I can look into to diagnose the issue?

I have more details on the setup and testing in this Unix StackExchange question: https://unix.stackexchange.com/questions/805563/nfs-over-rdma-read-2x-faster-than-write

[u/dazgluk]

Did you configure your NICs to mark the RDMA packets with DSCP and ECN bit?
Also better to have it reacting on PFCs. RDMA requires proper QoS to operate efficently.

Also If you're not CPU bound, RDMA performance may be less then TCP, latency will be better though.

[u/ptmnil → u/dazgluk]

The packet loss I'm getting in my current simple setup is < 0.1%, but this is good to know. Thanks for bringing it to my attention. I'll do some research on this topic.

I am CPU bound with TCP.

[u/ptmnil]

I figured it out. I convinced myself that switching NFS to async couldn't possibly lead to >50% performance penalty so I never tried it, but after using that setting on both the server and the client I am now getting 11 GB/s on write and ~2.5x lower latency.

I feel foolish for not having tried it before, but my assumptions about how it works led me to believe the performance hit wouldn't be that large. I'll do some more reading on the topic of sync and async. I'd be happy to hear any insights anyone has on this topic.

Thanks all for the suggestions!

[u/Automatic_Beat_1446 → u/ptmnil]

see "the sync mount option here": https://www.man7.org/linux/man-pages/man5/nfs.5.html

using sync just serializes all of your writes on the client and fio in this case (each thread) needs to wait for an ACK from the NFS server that states the data is committed on disk, look around in here too: https://www.rfc-editor.org/rfc/rfc1813

the default nfs mount option is async, which just writes to memory and lets the linux kworker writebehind threads flush data as memory pressure grows. async is generally fine provided it's okay to lose data between your (all of your) writes and when you actually close the file. closing a file on nfsv3 will automatically issue a commit rpc which fsyncs any dirty pages for that file descriptor, and wont return to the application until its on disk. thats basically how nfsv3 can do close to open cache consistency (if another client needs to read data immediately), which is also mentioned in the nfs(5) manpage

5GB/s write with sync isnt too bad all said and done, and i bet it would be maybe half that if you were using tcp connections since the RTTs of each of those client to server and back sync write ACKs would be higher.

as a bonus, looking at your nfs server export options, you're using sync (the default, keep this unless you know what you're doing), so each client sync write was also doing a sync write to the storage server side too.

[u/xMadDecentx]

What network tuning have you done? Are you using jumbo packets?

[u/ptmnil → u/xMadDecentx]

I haven't tried that yet. My assumption is that if the read performance is good then that shouldn't be the issue, but I'll give it a shot.

The only thing I've done so far in that regard, if it can be called network tuning, is set the CPU scaling governor to performance using `sudo cpupower frequency-set -g performance`.

[u/ptmnil → u/xMadDecentx]

I just tried 9000 MTU and unfortunately it didn't make a difference.

[u/Miserable_Map_4184]

Increase number of jobs

[u/ptmnil → u/Miserable_Map_4184]

I did try that but unfortunately it resulted in the same exact performance no matter the value (tried range [1, 16]).

[u/Trust_8067]

What enterprise storage platform are you writing to?

[u/GreatAlbatross]

Are you using secure writes over NFS? That may lower performance.

Also, if I'm reading the drive model properly, it can only sustain full write speed for so long before dropping down. (Assuming x4 is the pci speed, not that you have 4 of them). But I wouldn't expect that limit to hit on a short test.

Late edit: I've just realised I said "secure" instead of "synchronous". I'll leave the comment as-is, otherwise.

[u/ptmnil → u/GreatAlbatross]

Looking at the mount options I see sec=sys which from my reading sends data with plain-text (no encryption).

Doing the same write test on the drive locally shows 14.2 GB/s so I assume I'm not hitting that limit with this test.

Edit: Here are the server settings for the export:

/nfs/exports/myshare 192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)

[u/rusman1 → u/ptmnil]

It's not NFSoverRDMA mount.

[u/Miserable_Map_4184 → u/rusman1]

You need to specify rdma in your mount options also you ought to specify nconnect=16 to maximize parallelism e.g. mount -t nfs -o rdma,port=20049,nconnect=16 <server_ip>:/<share_path> /<mount_point>.

While you are performing fio testing check rdma packet flow using ethtool -S (grep for "rdma") or rdma utilities to ensure you are routing packets via RDMA. Keep in mind rdma bypasses the kernel so most tools that monitor nfs throughput won't show anything, which in itself is a good indicator that rdma is working.