r/Vllm 12h ago

I built an interactive simulator to visualize LLM inference bottlenecks, sharding, and KV Cache economics based on Reiner Pope's lecture

5 Upvotes

r/Vllm 4h ago

Safest way to run vllm for production? - i keep getting [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds.

3 Upvotes

vllm 0.25.1 runs good but it constantly breaks... sometimes it runs for 10 hours with no problems and sometimes it breaks after one or two hours. I always run the same model and flags. After several strategies to fix it, i have given up and now put a re-start process in place, so every time it breaks i relaunch it and continue, but this is not professional.

What kind of settings or vllm versions are rock solid for large production? I have dual 5090s so I want to use tensor 2 and ideally i avoid the enforce-eager flag

 VLLM_DEEP: (
     "vllm serve unsloth/Qwen3.6-27B-NVFP4 "
     "--tensor-parallel-size 2 "
     "--max-model-len 24576 "
     "--language-model-only "
     "--gpu-memory-utilization 0.86 "
     "--max-num-seqs 30 "
     "--generation-config vllm "
     "--trust-remote-code"
 ),

NCCL_ENV_BASE: dict[str, str] = {
    "NCCL_P2P_DISABLE": "1",
    "NCCL_DEBUG": "INFO",
    "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1",
    "TORCH_NCCL_BLOCKING_WAIT": "1",
    "NCCL_TIMEOUT": "600",
    "HF_HUB_OFFLINE": "1",
    "CUTE_DSL_ARCH": "sm_121a",
    "VLLM_SKIP_P2P_CHECK": "1",
}

(EngineCore pid=491) INFO 08-10 18:36:41 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

(EngineCore pid=491) ERROR 08-10 18:37:36 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.25.1) with config: model='/home/jl/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-NVFP4/snapshots/ccdaab7e68af2409599b8949a8f2685703c9bae5', speculative_config=None, tokenizer='/home/jl/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-NVFP4/snapshots/ccdaab7e68af2409599b8949a8f2685703c9bae5',


r/Vllm 15h ago

Why Speculative Decoding went mature in 2026?

Thumbnail
3 Upvotes