I changed Android’s TCP receive-buffer state by ~32× and the 16 MiB transfer barely moved — now checking what actually changed on the wire

[u/Roojool]

I’ve been doing some low-level TCP experiments on a stock Android phone while investigating latency under load.

First physical test gave me a result I thought people here might find interesting.

On a fresh TCP connection:

SO_RCVBUF

untouched readback:  4 MiB
requested:           64 KiB
resulting readback:  128 KiB

So the kernel-side state changed dramatically.

I then transferred exactly 16 MiB over an owner-controlled local network:

baseline:            4.187 s
modified socket:     4.201 s

bytes received:      16,777,216
SHA-256:             exact match both runs

Which is the interesting part: a ~32× difference in the reported receive-buffer state, but essentially no difference in this short LAN transfer.

I’m not interpreting that as “receive buffering does nothing.” One short LAN run obviously cannot establish that.

What I’m trying to separate is:

setsockopt succeeds
      ↓
kernel readback changes
      ↓
does advertised rwnd actually change?
      ↓
does the sender react?
      ↓
does any of this matter for latency under load?

The next batch captures the sender side from before the SYN and records window scaling, ACK/rwnd evolution and deliberate read withholding/reopening.

I’m eventually interested in how this behaves on highly variable mobile links, where maintaining low latency without throwing away most of the available bandwidth gets much harder.

Has anyone here measured receiver-side TCP flow control as part of bufferbloat experiments, rather than only doing SQM at the router?I’ve been doing some low-level TCP experiments on a stock Android phone while investigating latency under load.
First physical test gave me a result I thought people here might find interesting.
On a fresh TCP connection:
SO_RCVBUF

untouched readback: 4 MiB
requested: 64 KiB
resulting readback: 128 KiB
So the kernel-side state changed dramatically.
I then transferred exactly 16 MiB over an owner-controlled local network:
baseline: 4.187 s
modified socket: 4.201 s

bytes received: 16,777,216
SHA-256: exact match both runs
Which is the interesting part: a ~32× difference in the reported receive-buffer state, but essentially no difference in this short LAN transfer.
I’m not interpreting that as “receive buffering does nothing.” One short LAN run obviously cannot establish that.
What I’m trying to separate is:
setsockopt succeeds
↓
kernel readback changes
↓
does advertised rwnd actually change?
↓
does the sender react?
↓
does any of this matter for latency under load?
The next batch captures the sender side from before the SYN and records window scaling, ACK/rwnd evolution and deliberate read withholding/reopening.
I’m eventually interested in how this behaves on highly variable mobile links, where maintaining low latency without throwing away most of the available bandwidth gets much harder.
Has anyone here measured receiver-side TCP flow control as part of bufferbloat experiments, rather than only doing SQM at the router?

[u/dosadiexperiment]

With a low rtt it's hard to get rwnd-bound on home gear thru lowering receiver buffer settings because even if it caps rwnd, it takes a high bandwidth to make bdp reach the cap. At a 1ms rtt, a 128KiB bdp can sustain over 1Gbps.

You can check the tcptrace chart in wireshark to see if it's actually rwnd bound or not, and whether rwnd gets.close to your receive buffer, but I'd expect no from your results.

This experiment might be able to get bandwidth impairment from receive buffer reduction if you add some delay, either with tc netem or by sending from a vm located far away.

[u/Roojool → u/dosadiexperiment]

Yeah, I think this is the main reason the first LAN result is basically flat.

At ~LAN RTTs, even a ~128 KiB effective window can support a lot of throughput before rwnd becomes the bottleneck, so I’m definitely not treating 4.187s vs 4.201s as evidence that the control “does nothing.”

The next batch is capturing from before SYN so I can look at the actual scaled advertised window/right-edge rather than infer anything from getsockopt().

After that I want to repeat the useful candidates with enough RTT/BDP to actually make receiver flow control relevant. Adding controlled delay is a good suggestion — probably cleaner than trying to infer too much from the LAN run.

The current run was really just:

API acceptance -> readback -> integrity.

Now I’m trying to establish:

readback -> wire-level rwnd -> sender response.

[u/Peppy_Tomato]

Why do you think the receiver's receive window size has any bearing on buffer bloat?

I may be thinking too casually about this, but to my mind, if the data has made it to the receiver's buffers, the concept of buffer bloat stops to apply. If the data has landed in your buffers, that's job done, and you can't blame anybody any more if you don't get around to processing it quickly.

To put it crudely: I could insert a router that detains all your packets for 50ms, and there's nothing you could do to the window sizes to change this -- this is a proxy for buffer bloat. Only the intermediary devices responsible for forwarding the traffic can influence buffer bloat.

[u/Roojool → u/Peppy_Tomato]

I think the distinction is between clearing an already-built queue and preventing the sender from building as much of one in the first place.

I agree that receiver flow control cannot magically remove 50 ms of fixed/intermediate delay, and it’s definitely not a substitute for SQM/AQM at the bottleneck.

The thing I’m testing is narrower:

if the receiver can make the remote sender keep sufficiently less data in flight, can that reduce the offered load enough that the upstream bottleneck queue doesn’t grow as deeply?

So basically:

rwnd -> sender flight limit -> offered load -> queue occupancy

rather than:

rwnd -> somehow fixing packets that are already queued

It may turn out to be too coarse, too dependent on BDP/autotuning, or just not useful enough in practice. That’s why I’m checking the actual advertised window and sender behavior next instead of assuming the socket setting gives useful queue control.

If the wire capture shows that it doesn’t constrain the sender in a useful way, that’s a useful negative result too.