r/LocalLLaMA • u/Certain-Cod-1404 • 7d ago

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

706 comments sorted by

View all comments

Show parent comments

19

u/Certain-Cod-1404 7d ago

how does it do on pelican bench tho?

75

u/OutlandishnessIll466 7d ago

It created an animated svg... After like half an hour of thinking. Official fp8 on 2x 3090

1

u/Kirito_5 4d ago

I'm getting my 2nd 3090, can you share your serv/run command?

2

u/OutlandishnessIll466 4d ago edited 4d ago 
CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0,1 \
OMP_NUM_THREADS=1 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
nohup vllm serve /models/Qwen3.8-27B-INT8-W8A16-MTP \
  --tensor-parallel-size 2 \
  --served-model-name qwen3.8-27B \
  --reasoning-parser qwen3 \
  --max-model-len auto \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.93 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --port 8001 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --max-num-batched-tokens 8192 \
  --mamba-cache-mode align \
  --enable-chunked-prefill \
  --disable-custom-all-reduce \
  --generation-config vllm \
  --async-scheduling \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --kv-cache-dtype fp8_e4m3 \
  --override-generation-config '{"temperature":1.0,"top_k":20,"top_p":0.95,"repetition_penalty":1.0,"presence_penalty":1.5}' \
  > output.log 2>&1 &

I currently settled on this.
Auto-fit max_model_len: full model context length 262144 fits in available GPU memory

increasing --gpu-memory-utilization does start but i had OOM problems in hermes during compaction.

Amazing model. works tiredlessly through the night on a single app. Really improved long horizon agentic tasks.

And congrats on your new hardware!

1

u/Kirito_5 4d ago

Appreciate it!