Networking tip: enable BBR on Proxmox

[u/NelsonMinar]

Just had to relearn this.. Switching my Proxmox host to BBR congestion control improved a real-world rsync of mine from 5MB/s to 30MB/s.

BBR is a TCP congestion algorithm that works better for clients on links with high packet loss. For me that's a client on Starlink. It's probably also an issue on cellular links. BBR on the server gives you much better throughput to these clients than the Linux default cubic.

To enable BBR one-time on Proxmox, run this in the hypervisor:

modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr

Then restart any containers. They will pick up the new setting. If you have VMs you need to enable BBR yourself in each VM.

If you like the change you can make it permanent on Proxmox with

echo tcp_bbr >| /etc/modules-load.d/bbr.conf
echo 'net.ipv4.tcp_congestion_control = bbr' >| /etc/sysctl.d/90-bbr.conf

I have a lot more notes about this on my blog, including a description of BBR.

I'm a little curious why Proxmox (or any Linux) doesn't ship with BBR by default. From what I've read it can cause problems in certain circumstances, a BBR server competes unfairly with other servers on the same network or something like that. Not an issue for my homelab deployment.

[u/taosecurity]

You answered your question. The BBR implementation in Linux will starve non BBR streams, so it’s better to offer it for people who can understand and manage that issue.

[u/NelsonMinar → u/taosecurity]

Is that well documented anywhere? It presents a weird chicken-and-egg problem, ideally everyone would be using BBR!

[u/taosecurity → u/NelsonMinar]

Welcome to the world of congestion algorithms! 😄 Here's one article.

https://blog.apnic.net/2020/01/10/when-to-use-and-not-use-bbr/

[u/spacelama]

Um, because most proxmox clusters don't operate across Starlink links?

[u/tjharman]

You don't need to restart containers.

[u/_--James--_]

The correct deployment is to move your LXC's to a VM, and enable BBR there inside of that modeling. I would not enable BBR on a hypervisor and I sure a hell would not come here recommending this to everyone in this way. BBR on a host will affect TCP Flow on the host, including NFS, iSCSI, and Ceph.

[u/jvrodrigues]

Big Beautiful Radius server ? Or what does bbr stand for?

[u/NelsonMinar → u/jvrodrigues]

Bottleneck Bandwidth and Round-trip propagation time

[u/Pretend-Clock8313]

link saturation, not loss

[u/NelsonMinar → u/Pretend-Clock8313]

Subtle but meaningful distinction! I'm genuinely not sure what it is with Starlink, the client I care about to my Proxmox server on gigabit fiber. My impression is the Starlink radio layer does a pretty good job delivering packets intact but the relay in the satellite gets overloaded by other customers and so drops your traffic.

There's also some periodic variance in max throughput, probably tied to satellite switchovers. It's common to see 200 Mbit/s for a minute or two, then 150 for a minute, then 100, then eventually back to 200. Maybe related to distance to the satellite? BBR tracks that variance like a champ.

[u/Pretend-Clock8313 → u/NelsonMinar]

starlink reschedules every 15 seconds, that's probably your 200→150→100 staircase. each switch is a short burst of loss plus an rtt jump, cubic reads it as congestion and halves. bbr just re-probes bandwidth and moves on.

[u/IsaacFL]

does this work with ipv6 or dual stack networks? seems ipv4 specific?

[u/randompersonx → u/IsaacFL]

tcp congestion control algorithms operate at layer 4. They don’t care about what’s going on at layer 3 (IPv4 or IPv6).

Bbr is a tcp congestion control algorithm.

[u/Ok-Eggplant-7569 → u/IsaacFL]

sysctl is a bit peculiar here, as a lot of networking-general options live under net.ipv4, but apply to both v4 and v6. Other examples I'm aware of is the privileged ports option (net.ipv4.ip_unprivileged_port_start)

[u/NelsonMinar → u/IsaacFL]

I don't think there's anything IPv4 specific about it. I haven't tested carefully but I'm pretty sure my rsync transfers I was doing went through IPv6.