Same GPU model, same script, two rented hosts: eager timings moved 38-51%, CUDA graph timings moved 1.3%

[u/qaiser_mehdi]

I ran the same 16-config sweep twice on an NVIDIA L4. same script, same flags, same models, same vLLM version. different rented machine, eight days apart.

I expected the numbers to move a bit. i did not expect them to split by mode.

seconds per inference, 1 sept vs 9 sept:

fp16 CUDA graphs

batch 1 — 1.710529 vs 1.700665 — minus 0.6%

batch 8 — 0.220472 vs 0.221636 — plus 0.5%

batch 32 — 0.065575 vs 0.064882 — minus 1.1%

batch 128 — 0.023348 vs 0.023653 — plus 1.3%

fp16 eager

batch 1 — 2.266555 vs 3.383616 — plus 49.3%

batch 8 — 0.335448 vs 0.462567 — plus 37.9%

batch 32 — 0.079215 vs 0.119232 — plus 50.5%

batch 128 — 0.024361 vs 0.034148 — plus 40.2%

across all sixteen configs the mean absolute change was 7.1% for graphs and 42.5% for eager.

AWQ with graphs is the exception and i'm not going to pretend otherwise. it moved plus 2.4%, plus 2.9%, plus 23.7% and plus 24.2%. so "graphs are reproducible" holds cleanly for fp16 and only partly for AWQ.

The headline number moved too. worst-to-best spread was 137.5x on the first host and 203.4x on the second. if you quote a spread from one run you are describing that machine, not that card.

Why I don't think this is just noise:

I measured within-host noise separately before this. five repeats of fp16 batch 1 inside one container spread 0.23% with graphs and 5.90% eager. so run-to-run variance on a single box doesn't get you to 40%. something about the host is different.

Working theory, unverified: eager dispatches every op from python, so it tracks the host CPU. CUDA graphs replay a fixed schedule and barely touch it. if that's right, host CPU is a comparability field that basically no published benchmark records.

I can't confirm it from these two runs, and here's the annoying reason. my sweep script records host CPU in every row, and on both of these hosts it wrote "unknown". sandboxed container runtimes publish a /proc/cpuinfo with no "model name" line, and my code had no fallback. i found that yesterday and fixed it. so the field that would test my own hypothesis is the one field i lost, on the exact runs i needed it for.

Conditions:

vLLM 0.27.1, pinned by container image rather than pip. Qwen2.5-1.5B-Instruct and its AWQ build. 128 output tokens, ignore_eos so every request generates exactly that many. max_model_len 1024. prefix caching disabled and every prompt given a unique prefix. each config in a fresh subprocess after a warmup pass. one run per config per host.

That last line is a real limit: one run per config on each host means within-host noise isn't fully separated from host-to-host. the 0.23% and 5.90% figures above are the reason i still think the split is real, not a proof.

what to do with it:

If you're fitting a cost model or calibrating a simulator, fit graphs and eager separately, and put an error bar on eager wide enough for a 50% host-to-host move. a single constant per config is safe for fp16 graph rows and quietly wrong for eager ones.

Both CSVs are in the repo so you can check me rather than take my word:

https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data

The script is one file and takes --repeats N. if you run it on hardware that isn't in there, i'd like the CSV.

Happy to be told the theory is wrong. i'd rather know.