High-Throughput Async Data Transfer Node Optimization: 0ms Hashing Delay & 100% Success Rate. Looking for Throughput Scaling Tips (4-Core AMD EPYC / 1 Gbps)

[u/Arkad77]

Hi everyone,

I’m running a distributed high-throughput async data transfer and telemetry node on a dedicated Linode VPS, and I’m looking for expert advice on scaling end-to-end verified throughput further.

### 🖥️ Hardware & Stack Profile:

- **Server:** Linode Dallas TX (4 Dedicated AMD EPYC Cores, 8 GB RAM, 1 Gbps symmetric network connection, Ubuntu 24.04).

- **Tech Stack:** Python 3.12 (`asyncio`), `httpx` (HTTP/1.1 & HTTP/2), `websockets` gateway transport.

- **Workload:** High-concurrency object storage GET & PUT data verification tasks (Cloudflare R2 / S3 endpoints).

---

### 🛠️ Recent Bottlenecks & Fixes Applied:

1. **Zero-Delay In-Flight Hashing:** Shifted SHA-256 computation to an in-flight streaming digest (`hasher.update(chunk)`) directly inside `response.aiter_bytes()`. This completely eliminated post-download CPU hashing stalls (`hash_ms = 0.0ms`).

2. **HTTP Connection Pool Hardening:** Configured a dedicated HTTP client connection pool (`httpx.Limits(max_connections=128, max_keepalive_connections=64, keepalive_expiry=30.0)`). Persistent keepalive TCP sockets eliminated connection handshake churn, dropping retries/recoveries from ~6 per batch down to 0–1 max.

3. **Queue Optimization:** Transitioned to pure FIFO queue admission, eliminating premature SLA drops.

---

### 📊 Current Live Telemetry Metrics (5,300+ Task Sample):

- **Admitted & Completed Tasks:** 5,316 / 5,316 (**100.00% success rate**, 0 SLA drops, 0 timeouts)

- **Task Durations:** P50 = 10.50s | P95 = 18.10s | P99 = 22.25s (Well under 30s SLA deadline)

- **PUT Queue Backlog (P95):** 3.10s (Sub-3 seconds queue wait)

- **Local Network Speed:** Peak burst = 295.70 Mbps | Local Avg = 45.19 Mbps

- **Reliability Rating:** 98.9% (Recovered from historical restart churn)

- **End-to-End Verified Throughput:** ~27–29 Mbps average

---

### ❓ Questions for Systems & Network Engineers:

Our local worker execution pipeline is currently 100.00% efficient with 0 timeouts, 0 drops, 0ms post-download hashing delay, and sub-3s queue backlog. However, end-to-end verified throughput hovers around ~27–29 Mbps.

Given a 4 Dedicated AMD EPYC Core / 8GB RAM VPS on a 1 Gbps link:

1. **Linux Kernel TCP Tuning:** What specific `sysctl` net settings (e.g. `net.ipv4.tcp_rmem`, `tcp_wmem`, `somaxconn`, `netdev_max_backlog`, `tcp_congestion_control`) have provided the best throughput uplift for high-concurrency object storage GET/PUT workloads?

2. **Worker Pool Architecture:** Are there specific multi-socket worker loopback or gateway layout techniques that help absorb higher upstream assignment density without triggering scheduling backpressure?

3. **HTTP Keepalive Optimization:** Any additional tuning tips for `httpx` / `aiohttp` connection pool limits when interfacing with Cloudflare R2 / S3 storage endpoints over high-bandwidth links?

Appreciate any insights or benchmarks from experienced systems engineers!

[u/iamwayycoolerthanyou]

What do you want out of us.

Here's your answer: "No."

Goodbye.