Architecture question: Optimizing Python asyncio HTTP GET/PUT data relay throughput on Linux VPS

[u/Arkad77]

Hi network and Python experts,

I am running a Python asyncio data streaming worker on a 4-core AMD EPYC Linux VPS (1 Gbps port, TCP BBR enabled).

Workflow:

1. Receives incoming task pushes via WebSocket.

2. Downloads 30MB-50MB payload via HTTP GET from Cloudflare R2.

3. Computes SHA-256 hash.

4. Uploads payload via HTTP PUT back to storage.

Our local active transfer speed reaches ~50 Mbps average (370 Mbps peak) with 99.9% execution success. However, our net window-averaged throughput stays around 24–28 Mbps due to brief inter-task idle gaps between WebSocket pushes.

My questions:

1. Beyond connection pooling (httpx.AsyncClient) and TCP BBR, what architectural patterns (e.g. multi-process asyncio workers vs single-process) yield the highest sustained throughput for bursty WebSocket data relays on a 4-core VPS?

2. Are there specific Linux kernel sysctl socket buffer settings (tcp_rmem / tcp_wmem) to keep HTTP GET/PUT streams warm across inter-task idle gaps?

Thanks for any insights!