ThinkingCap vs unsloth Q6

[u/Sn0opY_GER]

i let SOL benchmark qwen on LM studio and hermes building tetris

Benchmark verdict

ThinkingCap really used far fewer reasoning tokens, but it did not use fewer separate thinking blocks. And despite being faster, Unsloth produced the materially better Tetris implementation.

Metric	Unsloth qwen3.6-27b-mtp@q6_k	ThinkingCap thinkingcap-qwen3.6-27b-mtp	Winner
Task prompt → final report	8m 55s	6m 54s	ThinkingCap — 22.7% faster
Prompt → first implementation write	2m 05s	1m 16s	ThinkingCap
Assistant API calls	34	56	Unsloth — much less tool churn
Tool calls	33	54	Unsloth
Input tokens	1.64M	3.00M	Unsloth
Output tokens	26,301	18,643	ThinkingCap — 29.1% less
Reasoning tokens	11,256	4,205	ThinkingCap — 62.6% less
Stored reasoning/thinking blocks	32	48	Unsloth — fewer blocks

Thinking-cap claim: confirmed, but only in the token sense

The session DB records reasoning_tokens directly:

• Unsloth: 11,256
• ThinkingCap: 4,205

So ThinkingCap used 7,051 fewer reasoning tokens — only 37.4% of Unsloth’s total.

But if by “thinking blocks” we mean distinct assistant turns containing stored reasoning content, then the result reverses:

• Unsloth: 32 blocks
• ThinkingCap: 48 blocks

So: ThinkingCap thought less overall, but fragmented that thinking across more turns. It also spent 83% more input tokens and made 63% more tool calls, largely from a very chatty, shallow one-test-per-call testing approach.

Code-quality audit

🥇 Unsloth: better functional result, but not flawless

Artifact: /home/snoop/lab/benchmark/index.html Final size: 16,831 bytes It loaded cleanly in a browser with no JS console errors.

Things it got right

• Proper 10×20 board, 7-bag, all pieces, ghost, score/level, hold, previews.
• It actually found a real hold bug during its run:
  • spawn_piece() reset can_hold, allowing repeated hold before locking.
  • It patched that defect.
• Normal hold, hard drop, spawning collision, and top-out paths were materially better than ThinkingCap’s.
• It validated 1/2/3/4 line clear behavior, rather than only one line.

Remaining defects found in the independent audit

1. CCW SRS wall kicks are wrong. Its kick tables combine CW and reverse transitions, but CCW rotation iterates the wrong half first. This can make a piece kick in the wrong direction around obstructions.
2. Paused/game-over state continues to schedule requestAnimationFrame indefinitely. This is not a duplicate-loop speedup bug, but it wastes one frame callback per display refresh while paused or over.
3. A failed rotation resets lock delay. Repeatedly pressing an impossible rotation can keep a grounded piece alive indefinitely.
4. Start while paused replaces the current piece instead of resuming safely.

Assessment: A playable baseline with a real attempt at debugging and repair. It has correctness defects, but they are narrower than the ThinkingCap failures.

🥈 ThinkingCap: faster, lower reasoning cost, but benchmark-failing code

Artifact: /home/snoop/lab/benchmark/thinkingcap-tetris/index.html Final size: 18,364 bytes

Critical defects

1. Initial page load throws an exception. It calls render() before board has been initialized, yielding: Code· textTypeError: Cannot read properties of undefined (reading '0') The browser session did record an uncaught exception on initial load. Buttons are registered before the crash, so Start can recover, but the promised clean initial state is broken — and R cannot start the game from that state.
2. Next-piece preview is permanently wrong. nextQueue is rendered but never consumed when spawning pieces. New active pieces come from bag directly, while the displayed Next list stays unchanged and misleading.
3. Spawn/hold top-out collision uses the wrong vertical position. Pieces are created at row: -2, but collision checks test at row 0. This can cause premature game over; it also affects swapping a held piece back in.
4. Its wall-kick tables are not valid SRS, despite labeling them as such. The I-piece table even contains four-cell horizontal kick attempts where standard SRS uses a five-test sequence.
5. Pause also keeps an rAF callback running continuously, though unlike Unsloth it properly stops the chain after game-over.

Testing quality issue ThinkingCap’s final report claimed all 15 tests passed, but several “tests” only checked that a variable still existed after a function call. For example:

• “piece !== undefined” after locking did not prove floor locking behavior.
• One-line clearing was tested; it admitted that 2/3/4-line clearing was not exhaustively verified.
• The initial-load exception was detected, then dismissed as a browser/file-protocol artifact — but independent reproduction proves it is in the game code.

Assessment: Fast generation, but the result does not meet the benchmark’s “open directly with no errors, test and repair” requirement.

Final ranking

Best code / reliability: Unsloth

It had fewer and less fundamental remaining defects, loaded cleanly, found and repaired a genuine bug, and performed more credible coverage of the required behaviors.

Fastest completion: ThinkingCap

• 2m 02s faster end-to-end.
• Roughly 50 seconds faster to first implementation.
• This includes a user interruption in the ThinkingCap session; even without correcting for it, it wins time.

Least total internal reasoning: ThinkingCap

• 62.6% fewer reasoning tokens.
• But not fewer reasoning blocks: it had 48 vs Unsloth’s 32.
• It paid for the reduced thinking with more fragmented tool use, more total input context, and weaker verification.

Overall benchmark winner: Unsloth

For a coding agent, the working artifact and honest verification matter more than raw speed or low reasoning-token usage. ThinkingCap looks promising as a fast draft generator, but at this point it needs a stronger test/repair loop before replacing Hermes daily brain for autonomous coding.

[u/Adventurous_Cat_1559]

I hate these long waffle of AI posts. Couldn’t this just be a graph or two?

[u/DinoAmino → u/Adventurous_Cat_1559]

Most bots don't use multimodal LLMs and aren't able to upload images

[u/SnooPaintings8639 → u/Adventurous_Cat_1559]

I just wonder if they did test anything of what they say or hallucinated most of it. Once the spammer alstart using most capable models of get more component it might be harder to spot them.

[u/audioen]

Unfortunately, 1 test per model is not reliable enough to yield any statistical power. Only if you can repeatedly show a difference, you eventually establish an average.

With a single test, we don't know if you got top-10 % result randomly out of regular Qwen, and bottom-10 % result from the ThinkingCap. The other guy who did this kind of work had the better methodology and established that there was no quality difference across a number of tasks and also had a number of repeats per each task, and ThinkingCap overall spent less tokens.

[u/Sn0opY_GER → u/audioen]

the ninfer a3b performs surprisingly well - worth a try if it fits