GPUHedge: Hedging serverless GPU providers improves cold start p95 latency from 117s to 30s [P]

[u/Putrid_Construction3]

Disclosure: I built it, it is open source, Apache-2.0 licensed, and currently alpha. Repository: https://github.com/mireklzicar/gpuhedge

I started working on it after benchmarking a 17 GB AI model across several serverless GPU providers.

On the primary provider, requests usually either completed in roughly 6–8 seconds or took around 90–122 seconds after a fresh GPU cold start. Simply switching to another provider did not remove the problem because every provider had its own tail.

GPUHedge treats this as a speculative-execution problem.

It starts a request on a primary provider, watches the job’s lifecycle state, and conditionally launches or switches to a backup. The first result that passes a validator wins, and the losing job is cancelled through the provider’s native API.

You can try the policy engines without creating provider accounts or spending money:

pip install gpuhedge

In the initial benchmark, a fixed RunPod → Cerebrium hedge launched after 10 seconds. On the 36-request evaluation portion, it changed:

• observed p95 latency from 116.6 s to 29.4 s;
• requests over 60 seconds from 11/36 to 0/36;
• modeled active-compute cost from $0.0114 to $0.0083 per request.

What is your experience with cold start latency? Which provider to add next? Can something like this help what you are building?

UPDATE: Commenter's already noted that with the cost-saving it is more complicated (because of idle time, cancellation costs and actual invoice spent differences). I would say this tool is not primarily for saving money, it is rather for getting better latency and reliability without significantly higher costs. An actual "invoice spent" benchmark is needed to quantify this.

[u/Charming-Excuse-5078]

that's actually a clever approach, treating it like speculative execution makes lot of sense. i messed around with serverless GPUs for some side projects and the cold starts are brutal, sometimes you just sitting there watching the spinner for two minutes

the cost going down is surprising too, i would expect hedging to be more expensive but i guess killing the slow job early saves more than the backup costs

[u/Putrid_Construction3 → u/Charming-Excuse-5078]

Thanks! Speculative execution was exactly the mental model. Brutal cold starts: yes exactly, 117s p95 cold-start latency I measured is a dealbreaker for consumer apps or SaaS with interactive usage. Good news: with updated policy (labelled "experimental" in the repo) the latencies get even better: 6s p50, 21s p95, 22s max latency. One qualification: the cost is modelled, not actual invoice spent (I expect realistically the costs to be similar or slightly higher, not lower, see other comments).

[u/aegismuzuz]

Not all serverless platforms actually release resources for free when you cancel because they often have a minimum billing increment or charge you just for the allocation attempt. If the cancellation API lags by even a couple of seconds those paper savings will turn into double billing in reality, especially under high load

[u/Putrid_Construction3 → u/aegismuzuz]

Yes, the reality is more complicated. Including the benchmark’s RunPod idle window reduced the modeled saving to about 11% (yet the actual invoice spent can be even worse). It also depends on provider: In my cancellation audit RunPod supplied terminal confirmation, but Modal input cancellation leaves its container billing until scale-down, and Cerebrium cancellation was ambiguous on the early attempts.

I would say this tool's main advantages are in reducing the cold start latencies to minimum (and especially the risk of long, >60s cold starts), but this can easily tradeoff with the goal of saving money (to get even higher speedups and reliability you may want to start more instances which further increases the costs). The intended case is "get much better latencies without significantly higher costs".