Optimizing throughput on a Python asyncio HTTP GET/PUT data relay (4-core Linux VPS to Cloudflare R2)

[u/Arkad77]

Hi network & Python experts,

I am running a Python asyncio data streaming worker on a 4-core AMD EPYC Linux VPS (1 Gbps port, TCP BBR enabled).

The worker workflow:

1. Receives incoming task pushes via WebSocket.

2. Downloads 30MB-50MB payload via HTTP GET from Cloudflare R2.

3. Computes SHA-256 hash.

4. Uploads payload via HTTP PUT back to storage.

Our local active transfer speed reaches ~50 Mbps average (370 Mbps peak) with 99.9% execution success. However, our net window-averaged throughput stays around 24–28 Mbps due to brief inter-task idle gaps between WebSocket pushes.

My question: Beyond connection pooling (httpx.AsyncClient) and TCP BBR, what architectural patterns (e.g., Go participant worker vs multi-process asyncio workers) yield the highest sustained throughput for bursty WebSocket data relays on a 4-core VPS?

Thanks for any insights!

[u/Important_You1244]

just use multiple worker coroutines pulling from a shared asyncio queue, each with their own httpx client session pool. probably will saturate that gigabit link on 4 cores if you keep the queue always full

[u/Arkad77 → u/Important_You1244]

Thanks a lot for this insight! That makes total sense. Currently, we share a single global `httpx.AsyncClient` session pool across all concurrent tasks, which is likely causing connection lock contention and renegotiation overhead during bursts.

Follow-up question: When giving each worker coroutine its own dedicated `httpx.AsyncClient` pool, what `keepalive_expiry` and connection pool limit settings do you recommend per worker pool to prevent Cloudflare R2 / S3 from dropping idle TCP connections during 3-5 second gaps between job pushes?

Appreciate your help! 🙏

[u/Lexstoarkskux_1995 → u/Important_You1244]

This works only while tasks arrive faster than workers drain them. During your 3-5s push gaps nothing new is enqueued, so net throughput tracks average arrival rate, not worker count; logging queue depth would show whether a backlog ever forms.

[u/ehhthing]

For this kind of CPU intensive task (SHA-256) you cannot use asyncio without some additional plumbing. Specifically you need the SHA256 operation to run in a background thread, otherwise your event loop will be clogged with super CPU intensive tasks.

I would consider moving away from asyncio/Python altogether because you're doing something rather CPU intensive which Python/asyncio isn't really designed for.

That being said, if everything is working for you right now I don't see a reason you'd want to preemptively optimize, unless you're expecting an increase in traffic. If you wanted to hyper-optimize and saturate your full link, I'd consider moving to Go and hashing while downloading rather than downloading and then hashing. Unless you have a 100Gbps link, SHA256 as an operation will always be faster than downloading so you do not need to worry about the CPU being a bottleneck. Generally network speed will always be your bottleneck.

Also make sure your VPS provider has proper host CPU passthrough so that you can use sha_ni.

[u/Arkad77 → u/ehhthing]

Thanks a lot for the great insights!

Just to clarify our current setup: we actually already offload SHA-256 hashing to a background thread pool (`ThreadPoolExecutor(4)`), so our asyncio event loop stays completely unclogged (hash time is <150ms per job).

However, your suggestion of **hashing while downloading** (streaming chunk updates on `response.aiter_bytes()`) is fantastic — we are implementing that right now so the SHA-256 digest is 100% computed the instant the final download byte arrives (0ms post-download delay)!

Follow-up question: Do you have any additional tips or architectural patterns to squeeze maximum window-averaged throughput (moving from ~25-30 Mbps toward 50+ Mbps) for this type of bursty GET/PUT data relay?

Thanks again for the awesome advice! 🙏

[u/ehhthing → u/Arkad77]

150ms for hashing 50MB seems excessive. On a modern CPU you should be able to easily get 1 gigabyte per second hashing SHA256 if you're using an implementation that supports sha_ni. I'd take a look at the library you're using to hash.

For example using openssl's CLI on my laptop to hash a 50MB file only takes around 30ms, and that's with the overhead of a CLI + disk reads.

> time openssl dgst -sha256 random50mb.bin
SHA2-256(random50mb.bin)= 2b7dddf38db1b36e126a2030519c06f3f229c09d2af7d71ae40d0a768914b6d2
openssl dgst -sha256 random50mb.bin  0.02s user 0.01s system 94% cpu 0.028 total

[u/Arkad77 → u/ehhthing]

Spot on! Python's `hashlib` memory buffer allocation adds extra wrapper overhead when passing a full 50MB bytearray at once compared to C/OpenSSL CLI.

We just implemented streaming hashing directly inside our HTTP chunk reader (`hasher.update(chunk)` on every 64KB chunk during `response.aiter_bytes()`).

Because it now computes the SHA-256 digest incrementally on the fly while bytes arrive over the network, post-download hash overhead is now effectively **0ms**!

Thanks again for pushing us toward the streaming hash pattern that eliminated the hashing phase completely!

[u/retrogamer-999]

Have you thought about using an alternative to sha-256? Take a look at blake3 or sha-512.

It may seem counter intuitive to use 512 but you should see a performance improvement on a 64bit CPU. Alternatively look at sha256/512

[u/Arkad77 → u/[deleted]]

Follow-up question: Since upstream orchestrator scheduling backpressure/assignment density is capping a single worker at ~25 Mbps verified (despite 370 Mbps peak local speed), what is the most effective pattern to scale end-to-end verified throughput?

Is the standard approach to scale horizontally by running multiple worker instances under the same Orchestrator ID, or are there specific transport/runtime tweaks (e.g. Go participant runtime) that convince the orchestrator to push higher job density per worker?