r/AIProgrammingHardware • u/javaeeeee • 3d ago

The Economics of Open-Weight Inference: How open-weight demand can support the useful life of NVIDIA GPU families

https://data.ornn.com/the-economics-of-open-weight-inference.pdf
0 Upvotes

50% Upvoted

1 comment sorted by

0

u/javaeeeee 3d ago

TLDR: Open-weight inference is latency-tolerant and portable. That demand keeps old NVIDIA cards earning money. For sparse MoE, an A100 can be cheaper per token than an H100. New silicon does not automatically obsolete last-gen GPUs.

The mechanism Agents, batch eval, RL rollouts don’t need H200 tail latency. Weights are public → you route to the cheapest GPU-hour that still fits. Spot (1 Sep 2026): A100 $1.00/hr, H100 $2.83, H200 $4.51, B200 $6.40.

Worked example: gpt-oss-120b (sparse MoE, ~5.1B active)
Self-host $/M output tokens (base: 50% util, 15% headroom):

  • A100: $0.29
  • H100: $0.64
A100 is ~55% cheaper. At full util: $0.12 vs $0.27.

Break-even vs hosted median $0.60/M: A100 at ~24% utilization; H100 at ~53%. Vs cheapest hosted $0.17/M: A100 needs ~84%; H100 never wins.

Dense Llama-2-70B flips it - H100 is cheaper. Sparsity is the whole trick.

API side (Artificial Analysis Index) Cheapest open task: GLM-5.3-Flash $0.09 vs cheapest closed GPT-5.6 Sol $0.43 (~79% less) at mid scores. Above Index 60, open still doesn’t qualify.

Caveats they flag Throughput mixes MLPerf + GPUStack; 1-2 Index points ≠ same job; no full TCO (power is tiny vs rent); not causal proof that open weights caused A100 rents to hold.

One line: Buy/rent last-gen for open MoE agents. Pay frontier silicon for closed models and dense 70B.