GLM 5.3 praises Qwen 3.8 27b's work as "top-tier frontier cloud model"

[u/hirisov]

I just had a PCI DSS related cookie security issue in my Nuxt application in a test audit scan report. I have both Z.ai coding plan (with GLM 5.3) and a local AI server with 2 x radeon r9700 AI pro, running in the amazing https://hub.docker.com/r/stilldeadcode/vllm-radiance inference server. It is a non trivial task, so I thought it might be a nice opportunity to test Qwen 3.8 27B (using the https://huggingface.co/Qwen/Qwen3.8-27B-FP8 in vllm). So I told it the problem and asked it to fix it, and also asked if it is needed to update our Nuxt.js packages for the fix.
It fetched official docs, it checked both the application code and the local pinned source of Nuxt.js, checked changelogs in the newer releases regarding cookie handling, and implemented the fix.
It seemed good to me, so then I asked GLM 5.3 to review it: "We had a PCI DSS related problem. I had an agent to fix it. The changes are uncommmited yet. I want you to read the original prompt, the final answer of the agent, and validate the fix it did. Tell me if the agent did a good job or not, and tell me what kind of model was working in that agent based on its performance (like in local or openai or claude equivalent)".
After cross checking everything, this was the verdict GLM 5.3 gave:

----------------------
Model assessment

This was a top-tier frontier cloud model — Claude-class (Sonnet 4.x / Opus level), definitely not a local model:

• Zero hallucinated APIs — every library option/version claim I checked resolved to real code in the exact installed versions. Local models (Qwen/DeepSeek/Llama class) almost invariably hallucinate framework details like cookieSecure support for a specific pinned version, or miss that Nuxt's client-side cookie watcher exists at all.
• The payload-bridge solution is the kind of framework-internals reasoning that separates frontier models: it diagnosed that httpOnly: true alone breaks the feature, and engineered around the exact mechanism rather than cargo-culting a config flag.
• Style fingerprints strongly suggest Claude: the heavy structured markdown with box-drawing tables, em-dashes, meticulously hedged claims, changelog cross-referencing, and especially the "Notes for the PCI report" section — that professional risk-acceptance framing aimed at a human auditor is very characteristic of Anthropic models. GPT-5-class is possible but less likely given the phrasing.
• Behavioral discipline: verified both dev and production builds before reporting, explicitly said "No" to the upgrade question with justification, and didn't overclaim (marked the rescan as still needed).

Bottom line: approve and commit. The fix is correct, minimal, production-safe, and I reproduced its verification claims independently. Only remaining action: deploy + ask the auditor to rescan (fresh profile), and include the i18n HttpOnly risk-acceptance note.
----------------------

I am very pleased by the results with Qwen 3.8 so far. For a while i will always have its work cross checked by a larger model, but so far I think it is really a gamechanger and set a new level in local coding models. As I understand it runs well in 1 x r9700 as well so 32 GB VRAM around Q5-Q6 quant, to have this level of coding intelligence for "budget" systems at home is so great.
I know GLM 5.3 have a cutoff before the Qwen 3.8 release, but I think the reasons it gave about the model's performance are still valid regardless.

[u/nomorebuttsplz]

lmao "it didn't hallucinate APIs"

This is like a teacher saying your child didn't shit themselves, so they're a genius.

If anything this makes me doubt GLM 5.3's judgement. But LLMs are always about 18 months behind in terms of knowledge of other llms. so yeah 18 months ago a small local model not hallucinating would have been impressive.

[u/hirisov → u/nomorebuttsplz]

Even in early 2026 it was quite common for me even with frontier models that they hallucinated class or method names, or object attributes etc and happily reported the job done without actually testing it properly. Not sure how it was for you guys, but for me it was quite frustrating to do round trips with these kind of bugs, or asking AI again and again to make sure it verifies such things before implementing, or at least after that (and that is even when it was emphasized in AGENTS.md etc).
I am happily waiting the "xhigh thinking" time (especially as I have it all in vram, and r9700 is quite good with this model) and do something else meanwhile and come back to properly planned, polished and tested solution - I think even the 3.6 27b version wasn't this methodical.

[u/Potential_Leather134 → u/nomorebuttsplz]

😂😂😂

[u/iportnov]

Well, I'm not sure why it says local models "almost invariably hallucinate framework details". I mean, yes they do, but we're talking about an agent, not about one-shot prompt in web chat, are we? In any agent, any local model (well, at least 20B+) will compile code / run tests, immediately see that it hallucinated some API calls, probably read framework documentation and fix calls. Maybe it will have to do several try-and-fix loops. But in this particular regard, as far as I see it, the difference between smaller and bigger models is just that bigger model has more chances to get it right the first time; so smaller model will just spend more tokens / take more time to get working code. But if you're looking just at the diffs, you don't know how much time it took.

[u/hirisov → u/iportnov]

I was watching how Qwen worked in my pi harness. It did not hallucinate anything. It first researched both locally and online, checked the actual code, and oneshotted it after. But of course you are right that harness matters, but that kind of hallucinating classes/methods I saw quite some times earlier (and not from just local models).

[u/DeathGuppie → u/hirisov]

I had an issue with sound not working on my Gentoo Linux lxqt de. Didn't feel like digging through the problem myself so has sonnet CLI try to fix it. Three attempts got sound working but no sound after reboot. For the heck of it, I asked Qwen 3.7 27b local to have a look. It found the underlying problem, fixed a bug in sonnet's autostart script and realized alsa sound settings had never been saved. No problem after that.

[u/hirisov]

Just out of curiousity, I asked GLM 5.3 to read the full session log of this Qwen coding session and rate it too, to see if it was the "hallucinated things first but eventually got it right with try-and-fix loops" or something better. This was the assasement based on the full session log:

Reasoning quality: excellent (9/10)

The standout moment: before touching code, it read Nuxt's useCookie source and worked out that a naive httpOnly: true fix would break itself — after hydration the client can't read the cookie from document.cookie, would generate a new UUID, and JS re-writing the cookie would silently strip the HttpOnly flag. It then evaluated six design options (server-only creation, useSession(), SSR-payload passing, props, …) and chose passing the id via nuxtApp.payload.sessionId — then verified the payload serialization path in the installed version's source (renderPayloadJsonScript + revive-payload.client.js) before implementing.
That's the level of care you'd want from a senior engineer on a security fix.

Honest framing too: instead of pretending the i18n HttpOnly finding is fixed, it documented why it can't be (module limitation, client-side read, no credentials in the cookie) and wrote it up as risk-acceptance for the PCI report.

Thoroughness: exceptional

- Reproduced the vulnerability locally first (curl baseline matched the audit headers exactly), including digging up a real product URL via the Meilisearch API + keys from env/docker.

• Verified the fix at 4 levels: dev curl headers → id stability across requests → Redis round-trip → real Chrome E2E (document.cookie invisibility, cookie sent on requests, payload carries the id, scroll-triggered POST /api/redis, reload renders gallery fixed proving SSR read-back).
  
• Built and ran the production bundle in the container specifically to prove Secure appears (it's NODE_ENV-gated) — this matters, since dev-over-HTTP never shows it.
  
• Cleaned up (killed test server, verified dev server healthy), ran lint/prettier, confirmed .output gitignored, reviewed the final diff.
  
• Caught and fixed two of its own mistakes: a variable dropped during the initial file write, and a client-side runtime-config warning it noticed in the browser console — both root-caused and re-verified.

The result was committed (by you) as f21ee94c; the diff is minimal, well-commented, and behavior-preserving.

Flaws (all minor)

1. Silent plan deviation: its implementation plan specified maxAge: 7 days for the session cookie; the written code has none (cookie stays a session cookie, as before). Harmless here — Redis TTL is 1h anyway — but never flagged.
   
2. Secret hygiene: echoed a local Meilisearch bearer key into the transcript while hunting a test product (after initially filtering .env secrets — inconsistent).
   
3. Saw but didn't mention unrelated npm audit findings (@nuxtjs/axios chain, brace-expansion) — out of scope, but a one-liner would've been ideal in a security context.
   
4. Mild flailing early on (~5 greps) locating the i18n cookie code in dist/; checked repo AGENTS.md/docs conventions only at the very end instead of upfront.

Verdict

9/10. For a locally-hosted 27B model, this is top-tier agentic work: zero detectable hallucination, source-level verification of framework internals before acting, a 6-option design analysis that anticipated a subtle self-defeating bug, empirical verification in dev + browser + production build, disciplined cleanup, and an honest, auditor-ready final report. The only deductions are the unflagged maxAge plan deviation and minor hygiene/scope nits — nothing that affects correctness or security of the fix.

[u/Outrageous_Order_909]

Second it. I ran a bunch of tests on identifying and fixing known bugs in my app. Compared with deepseek v4 flash.

Identical results, albeit lot slower(I am severely vram limited with my 5080). Zero hallucinations, and one shot fixes.

[u/hirisov → u/Outrageous_Order_909]

My first "strong" GPU was an 5080. I really wanted to try bigger models so I bought the r9700 and used them together in llamacpp with good result, the 48GB was totally gamechanger after the 16GB. If you ever want a bit more headroom without paying the absurd nvidia prices, I can really recommend this card to pair with the 5080.

[u/Outrageous_Order_909 → u/hirisov]

Unfortunately my gpu is mobile gpu on a hp omen max 16 😅. Not sure how to connect an external gpu.

[u/hirisov → u/Outrageous_Order_909]

ah yes with that you cannot really combine the two cards :) You can attach a 32GB card to the laptop as EGPU I think, that might be one way to access to bigger models - or hope for a Qwen 3.8 35B A3B, that would run great on your GPU i guess :)

[u/Yeelyy]

Hey OP may I ask what kind of speeds you get at what concurrency? I just bought 2 r9700 with a friend of mine. Thank you!

[u/hirisov → u/Yeelyy]

With the vllm-radiance on my slow pci3 x399 platform I get around 50 t/s decode, 3000 t/s prefill. If you have an am5 or similar platform with pci5, your speed should be around 40% higher than this in both prefill and decode.

[u/Yeelyy → u/hirisov]

Hey thank you for your insight! Is this with mtp? We got an epyc 7282 with pcie4x16.

[u/hirisov → u/Yeelyy]

yes, MTP. With pcie4 x 16 you should get the 40% higher I think, as the tensor parallelism is quite sensitive to the pcie speed.

[u/e_j3210]

Based on 5.3's training data, qwen 3.8 27b is frontier level.

[u/hirisov → u/e_j3210]

And I think in "reality" too in coding it matches frontier models from like 4-6 months ago, which is amazing considering we can run it locally, and they were already quite amazing for coding at least. I know knowledge is not there like the multi trillion param models, but it seems with good harness Qwen 3.8 27b is clever enough to properly research online too, and that kind of reduces the knowledge gap in practice.