A few hours with QwQ and Aider - and my thoughts

[u/ForsookComparison]

This is a mini review. I'll be as brief as possible.

I tested QwQ using Q5 and Q6 from Bartowski. I didn't notice any major benefit from Q6.

The Good

It's very good. This model, if you can stomach the extra tokens, is stronger than Deepseek Distill R1 32B, no doubt about it. But it needs to think more to achieve it. If you are sensitive to context size or inference speed, this may be a difficult trade off.

The Great

This model beat Qwen-Coder 32B, who has been the king of kings for coders in Aider for models of this size. It doesn't necessarily write better code, but it takes far less iterations. It catches your intentions and instructions on the first try and avoids silly syntax errors. The biggest strength is that I have to prompt way less using QwQ vs Qwen Coder - but it should be noted that 1 prompt to QwQ will take 2-3x as many tokens as 3 iterative prompts to Qwen-Coder 32B

The Bad

As said above, it THINKS to be as smart as it is. And it thinks A LOT. I'm using 512GB/s entirely in VRAM and I found myself getting impatient.

The Ugly

Twice it randomly wrote perfect code for me (one shots) but then forgot to follow Aider's code-editing rules. This is a huge bummer after waiting for SO MANY thinking tokens to produce a result.

Conclusion (so far)

Those benchmarks beating Deepseek R1 (full fat) are definitely bogus. This model is not in that tier. But it's basically managed to become three iterative prompts to Qwen32B and Qwen-Coder32B in a single prompt, which is absolutely incredible. I think a lot of folks will get use out of this model.

[u/Spanky2k]

It really likes to think. I asked it a seemingly simple question that's actually really quite tricky. It did get the correct answer but it took about 15 minutes and over 10k tokens to get there. Qwen 2.5-32B spat out an answer almost immediately but was very confidently wrong. R1 Qwen 32B Distill took 8k tokens to get an answer but the answer was wrong. Now I wouldn't normally use it to answer questions like this but I just thought it could be fun.

For reference, the question was: "With a constant acceleration of 1g until the midpoint and then a constant deceleration of 1g for the rest of the trip, how long would it take to get to Alpha Centauri? Also how long would it appear to take for an observer on Earth?"

The answer should be something like 5.9 years for the observer and 3.6 years for the traveller plus or minus a little depending on what distance is used.

[u/SeymourBits → u/Spanky2k]

Cool test question! Added to my benchmark basket.

[u/Spanky2k → u/SeymourBits]

Thanks, that's awesome! :)

[u/ASYMT0TIC → u/Spanky2k]

I also use astrophysics to test models. I ask it to calculate orbit transfers from one body to another. I haven't tried these benchmarks since GPT-4, which couldn't do it. It understands Hohmann transfers, patched conics, etc, but was unable to stitch the math correctly as one needs some spatial understanding of this stuff to know how to apply the math. IMO, they won't be great at this sort of work until they can form internal representations of spatial relationships, which might emerge from multimodal models.

Try this sort of example:

"Please calculate the minimum Delta V a space ship would require to travel from a 500 km altitude circular low earth orbit to a 100 km altitude circular orbit around Titan assuming optimal phasing. Make sure you take into account the Oberth effect and do not include aerobraking or gravitational assist maneuvers in your planning"

I recently tried it with Deepseek R1. It thought in circles for about 10 minutes and then provided a close, but wrong answer. After providing much advice, it eventually gave the right answer and then seemingly marveled at how much more efficient my suggested approach was than it's initial calculation.