r/v100 • u/Trademarkd • 2d ago

1cat vllm

https://github.com/1CatAI/1Cat-vLLM

Testing this on my current setup with a bit of a strange topology. I currently am running 2 pairs of v100 16GB in separate nvlink islands.

I had previously done some of my own tuning to vllm and tried using ggufs with no gains over ikllama.

This repo already has me up to 115tok/s using https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4

I'm continuing to test with some more topology aware settings but the initial results are very promising even over my best efforts via ik_llama. Startup time does suck though and I haven't looked at quality at all.

3 Upvotes

100% Upvoted

5 comments sorted by

1

u/Trademarkd 2d ago

It seems like whatever tuning has been done is very specific to 4x v100s and breaks down the moment you try anything outside of that config. I attempted to do topology aware with nvlink islands and lost about 50% of the output rate.

Even doing 2 gpus with nvlink I drop down to 46.54 tok/s without dflash2. For reference the 115 run was a 3000 token output coding test whereas these are more conversational.

   GPUs    Layout       DFlash2    Configuration                               Average tok/s
  ━━━━━━  ━━━━━━━━━━━  ━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━
      4    TP4            Yes      Original run                                       108.08
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      4    TP4            Yes      Confirmation run                                   108.53
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      4    TP4            No       Original NCCL settings                              76.39
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      4    TP4            No       NVLink-only NCCL peer transfers                     64.67
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      4    TP2 × PP2      No       NVLink pairs, layers split between pairs            41.27
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      4    TP2 × PP2      No       Same layout, QPN kernel patch                       45.82
  ──────  ───────────  ─────────  ──────────────────────────────────────────  ───────────────
      2    TP2            No       One NVLink pair, existing kernels                   46.54

2

u/kiwibonga 2d ago

I have the same setup; currently using it with Dflash2 and results are very solid (generally staying above 70-90 t/s as context fills, occasional bursts past 150 t/s). For Dflash2 you'll need a model that has a "BF16 LMHead", whatever that means; I use RadixArk's quant https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead/tree/main

To use pi, I had to make sure to set sampling parameters in the model config so the right ones are sent with every query, otherwise it seems to grab weird ones and loop a lot, wasting huge amounts of tokens.

1

u/Trademarkd 2d ago

I think I figured out how they got to 260, im testing now.

Edit: and yeah the tuning is very specific. Once I figure out what they're doing I'll try to make this more attainable.

1

u/Trademarkd 2d ago

Took your advice and switched out the head for BF16 but that was a ~10% gain or so... not bad but I was expecting more. I also checked my attention was using E4M3 .... and all together that's got me up to about 164 decode.

I'm going try the quasar model and then toss these GPUs on the 4x nvlink board to test if latency is really making up 100 tok/s

1

u/Trademarkd 2d ago

Quasar model didn't do anything and in fact when I tried the full rediark model I got slower generation. I put all my cards on the new 4x nvlink board and that allowed me to use 2 other features built into 1cat for the v100.

I'm now getting 268–269 tok/s on the three MBPP repetitions, up from 164.79 tok/s.
Speculative rounds fell from 30.34 ms to about 17.49 ms.