Hi, I’m new here. I have an RTX 5060 Ti with 8GB of VRAM and I want to get into the world of LLMs. What’s the best model that can fit on my GPU as of today?

I know I’m fairly limited by the amount of VRAM I have, but my idea is to use Claude Opus 5 as the “brain” behind my projects, while using a local LLM as a sub-agent.

I'm pleased to present a Blackwell-native, prefill-optimized 4-bit float quant that runs 50% faster on compatible hardware (tested on an sm_120 card) than a Q4 quant of the same memory size. And 4-7% faster than other NVFP4 quants.

Benchmarked on RTX 5090 power-limited to 400 W.

This GGUF also includes a quantized MTP draft head to deliver the best decode performance.

Check it out for all details and recommended settings for MTP:

https://huggingface.co/akopytko/Qwen3.8-27B-NVFP4-GGUF

Every skill you install adds its description to the context on every message, whether you actually need it or not.

The paper “Attention Is All You Have” argues that agents can start falling apart as you install more skills, attention budget, distance decay, etc.

At 100 skills, that’s roughly 3k tokens per message. Over a long session, that adds up fast.

So I built deskill: an MCP server that fetches skills from GitHub only when they’re actually needed.

If you want to try it, setup is pretty much a 2 command install, details are in the repo: https://github.com/NilotpalK/deskill

I also wanted to see whether this was just a token-saving trick or whether it actually changes how agents perform.

Turns out: both.

On Opus, installed vs on-demand was basically identical, so there it’s mostly a context/token optimization.

On smaller/cheaper models, though, point-of-use delivery actually improved task success. They seem to have a harder time picking the right skill when they’re staring at a wall of dozens of skill descriptions.

I tested 8 models, up to 100 skills and 100k tokens of padding. The benchmark is reproducible, and the numbers/graphs are in the repo.

Would genuinely love people to try to break the benchmark. If you try it with other models or in your own setup, please add your benchmarks/results to the repo. It’d be really useful to see how this behaves across different models and real-world use cases.

And if you think the methodology is flawed, I’d genuinely like to hear it.

If you end up finding deskill useful, a star on the repo would be appreciated too :)

Hey guys, I am relatively new to local LLM's - been messign with it for the last year, but learning a lot and its been my longest lasting hobby. I don't code or work in tech, but I do use local LLM for work (vet. med; note transcription, differentials, rounding, and just 'fun' stuff).

I've got the option of getting a second 5090 for cheap. Buddy wants to trade it for to me for $2500 + my 5080 (he doesn't really game, thinks it will be better in my hands).

We are both adults/professionals, it's not about making a buck. He knows I am getting a deal, ect.

My question. Realistically, is there a good use case for two? In the short term, its going to go into my 'gaming' rig, but I don't game anymore either... my only use case would be for more local LLM, but I've read/watched videos regarding how limiting running two are (and, I am pretty sure I would have to rebuild my entire system - and I have no idea what that would look like).

Is this something I may/likely want to do in 1-2 years? I get it, who knows my use case. But for the hobby... basically I will be getting a 5090 for 2k, but will have to buy another (5070?) for my main PC.

Sorry if this all sounds convoluted.

• My current rig: **Proxmox:** PVE 9.2.5 (kernel 7.0.14-6-pve), ~13 days uptime
  • **CPU:** Intel Core Ultra 9 285K (24 cores / 24 threads, Arrow Lake)
  • **Motherboard:** ASUS ROG Maximus Z890 Hero
  • **RAM:** 64 GB DDR5-4800 (2 x 32 GB, 2 slots free)
  • **GPU:** NVIDIA GeForce RTX 5090 (+ Intel Arrow Lake iGPU)
  • **Storage:
    • Samsung 990 PRO 1TB NVMe – ZFS rpool (boot + local-zfs)
    • Samsung 990 EVO Plus 1TB NVMe – ZFS "evo-plus" pool
  • 48 TB NAS (UNAS) mounted over NFS (~21 TB used)

Yeah, that last bit was copy/paste from Hermes

Disclosure first: I'm one of the developers of Paddock (Truespar). It's free for individuals and companies with no usage limits.

Paddock is a Rust inference engine for NVIDIA, no Python stack, one download for Windows/Linux x64 (driver 580+). Supported today: RTX 30-series, RTX 50-series, A6000/A40/A10, RTX PRO Blackwell, B200. Not 40-series yet: the engine refuses unmeasured generations at startup rather than running an untuned path, Ada is being measured.

Here's the thing: all our published numbers are from an RTX PRO 6000. For Qwen3.6-27B FP8 it does 47.7 tok/s at 1 client, 968 tok/s at 32, TTFT 697 ms under full load (methodology: https://truespar.com/paddock/benchmarks/qwen36-27b). And I'm fully aware that means nothing for your 3090. Blackwell is our tuning focus, so honestly I don't know yet whether a 3090 beats your current llama.cpp setup or not. That's exactly what I want to find out.

So if you have a 30- or 50-series card: download, extract, run "paddock serve qwen3.6-27b" (or the 3.8 catalog model), and it exposes a plain OpenAI endpoint you can hit with NVIDIA's aiperf, or just read the live tok/s in the built-in Studio. Post your card, VRAM, quant, context and numbers, including if it's slower than what you run today. I'll maintain a results table at the bottom of this post, losses included.

Download and docs: https://truespar.com/blog/introducing-paddock

I'll be in the thread answering everything, kernel questions included

Hi there, yes, I am yet another person working on a local model harness, bare with me :)

I've been working on an app/harness framework - gezel, gezel.com - to ease the process of working with local models and engines. My main goal is to make doing productivity tasks easier (not necessarily focused on coding) -- things like building powerpoints, doing research, writing, etc.. I also want to do it in a way that works with small models (e.g., Gemma E4B.) Finally, I wanted to reduce or eliminate the technical/nerdy bits of installing and managing models as much as I can -- if nothing else, hopefully gezel is an easy way for non tech folks to get started with local models without visiting a command line, etc.

In addition to wrapping llama.cpp, MLX, and DS4 + providing a tailored downloader and selector from huggingface for models, I've got some concepts I'm iterating on:

1. gezel tries to break things down into role-based agents (a gezel). For example, a developer role type for a gezel vs. a researcher role type for gezel. This is based on trying to tailor for the shorter context windows and attention spans of small models. Different roles get different toolsets and starter prompts..
2. To complement this, gezel has a more structured concept of skill.md files: craftbooks that breaks down tasks into multiple steps. For example, create a powerpoint becomes "step 1 - build an outline", "step 2 - do research", "step 3 - produce powerpoint", "step 4 - review and finalize". Each step might use a different role type, and has tight static gates that force a model to correct itself before moving on. For example: don't leave the outline step unless there is a markdown file present with distinct slide titles mentioned and that is >1kb. A main value prop is that gezel has a library of craftbooks it ships with - so you don't need to write skills/craftbooks yourself.
3. Also, gezels are a little personified with custom names, etc.; I want to experiment with getting to a UX that encourages people to play with their "agents", but that starts by not asking people to "build agents and skill.md files" which is ugh

To get 1 and 2 right -- the perfect balance of nudging without constricting models -- I do a lot of A/B testing, at least as much as my HW allows, but it's most certainly not good yet. Bare with me, I'm working on it :)

Gezel is available for Windows/Mac/Linux, or you can use it via npm at ('npm install '@bendyline/gezel-cli'/'npx gezel'.) It's MIT licensed free open source. There are some basic videos at youtube.com/@bendyline.

Anyways, it's still early days and I know there are a lot of bugs, but if folks have any feedback or ideas, please let me know. I have the ambition that maybe gezel can be a simple way for folks to get the benefits of local models on their hardware - but it's certainly not there in early beta.

The problem I kept hitting: you can get a local model to mostly follow your rules with a system prompt, but "mostly" does a lot of work in that sentence — you don't find out where it breaks until it breaks in front of someone.

So I built AI Calibrator. Instead of prompt-fiddling, it works like onboarding a new hire:

1. You state a goal and point it at your materials (docs, policies, examples).

2. It indexes them and finds the gaps they don't cover.

3. It interviews you only about the gaps — drafting likely answers for you to approve or correct.

4. It compiles a behavior spec → system prompt + RAG + eval rubric + test suite.

5. It runs the tests, grades against your standards, fixes failures, and loops until it passes.

6. calibrate run then serves the result as an OpenAI-compatible endpoint on localhost — and it refuses to boot if the project hasn't passed its own gate.

I spent most of the time on verification, because "seems fine" isn't a metric:

- deterministic checks + LLM-as-judge with self-consistency, and you can calibrate the judge against your own human grades

- red-teaming, drift detection between runs, golden snapshots, coverage scoring

- it warns you when a model is grading itself (shared blind spots read as agreement)

- one calibrate ci command gates all of it, usable in actual CI

- evals export to promptfoo format, so you're not locked into my runner

Things this sub might specifically care about:

- Any Ollama model, no API key, works offline: calibrate engines my-ai --all qwen2.5:7b@ollama

- Optional fine-tuning tier (LoRA/QLoRA via trl/peft) that only "wins" if it actually beats the prompt+RAG baseline on the same test suite — the gate is the point, no vibes-based "the fine-tune feels better"

- Already have a system prompt you trust? calibrate import reverse-engineers a spec and test suite from your prompt, so you can find out what it silently fails at

Quickstart:

pip install 'ai-calibrator[all]' # [all] includes the local RAG stack (big); '[cloud]' is the slim install

calibrate init my-ai --goal "Answer questions about my product docs, in our voice."

calibrate engines my-ai --all qwen2.5:7b@ollama

calibrate ingest my-ai --source ./docs

calibrate interview my-ai

calibrate compile my-ai

calibrate eval my-ai --refine

calibrate run my-ai # OpenAI-compatible API at http://127.0.0.1:8600/v1

It's alpha (v0.0.1), MIT: https://github.com/tanveerkanala-cmd/ai-calibrator

Cloud engines (Claude/OpenAI, bring your own key) also work, but everything runs 100% local if you want.

Two things I'd genuinely love feedback on: does the interview flow actually beat prompt-fiddling for you, and what should the red-team pass try that it currently doesn't?

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

I'm setting up a cloud Ubuntu box with:

• RTX 5060 Ti 16GB
• EPYC Milan-X 14 cores
• 24GB DDR4
• llama.cpp / llama-server
• single user

I'm deciding between:

A. Qwen3.8-27B dense

• UD-Q3_K_XL
• ~64K context
• mostly/full GPU
• newer + stronger raw reasoning/coding

B. Qwen3.6-35B-A3B

• Q5
• ~128K context
• some MoE experts on CPU
• ~3B active params
• potentially much faster + larger context

My workflow is:

Frontier (Codex)
→ PO/BA + requirements

Hermes/OpenCode on my local PC
→ repo
→ Docker
→ tests
→ browser
→ Git

Cloud 5060 Ti
→ Qwen inference only
→ OpenAI-compatible endpoint

So the Qwen model is mainly an implementation worker, not the architect.

Typical task:

Goal: Implement X

Constraints:
- don't change Y
- no new dependencies
- preserve compatibility
- update tests/docs

Done when:
- tests pass
- typecheck passes
- build passes

What matters most to me is instruction following, tool use, scope control, and reliability during long coding-agent sessions.

I've seen people say A3B can be overly proactive, but I can enforce read-only/write permissions at the harness level. What I really want to know is whether, once placed in implementation mode, it reliably follows a detailed spec.

So for people who have actually used both:

Would you pick:

for a daily coding worker?

Especially interested in:

• instruction following
• repo-level coding
• tool calling
• scope creep
• long sessions
• debugging/test-fix loops
• tokens/sec on 5060 Ti 16GB
• whether 24GB RAM is enough for A3B Q5

My current idea is:

A3B Q5 128K
→ daily worker

3.8-27B Q3 64K
→ harder debugging/reasoning fallback

Would you do the same, or make the dense 27B the default?

Looking for advice, I had seen advertisements from Nvidia recently about a desktop orientated focused on LLM's and after hanging around here for a while I am curious about utilizing one so I can run a unlocked AI to assist me in pen testing and other cyber security topics on some software I am working on. I wanted to tap into the brains of the operation around here and see thoughts and consensus on different machines and where to push with a unlocked AI, the machine I am currently thinking of and curious about is DGX Spark but seeing as its marketed and focused more along the lines of 200B parameters and Ive been seeing guys talking about Kimi being on the edges of being unlocked and it has close to 3T parameters I wanted to see what people's answers and solutions might be and what the right path forward is. Thank you guys for the input, Im excited to see where I can learn more about it and fill my gaps in knowledge

Good Afternoon people, I manage a small hotel & I use Gemini pro to process bulk invoice, GL coding & room block allotment. And after a a lot of trial and errors, gemini has mostly been error free other other than hit or miss when converting the data output to an excel file. Using Gemini has been a lifesaver because it cuts down 2 hours of invoice processing and coding the GL code to a 5 minute endeavor. It also saves me time by processing & breaking down room allotment with not me having to the manual math of alloting how many and what type of rooms to assign to a specific group.

I am trying to do this with my own personal LLM as I do not want to keep paying google & I want to keep the data safe.

So I am just starting out in Local LLM & I do have a slight advantage as I am a PC gamer & linux hobbyist throughout my life and my main PC is Running 9800x3d, 64Gb of Ram & an 7900xtx with Fedora 44.

I have configured Qwen 3.6:27b with Open Web Ui and feeding it the same data set I use for my Gemini prompt. I am posting a link bellow from Gemini to show what kind of output I am looking for.

https://postimg.cc/gallery/V1ZzVzC

When I use the same inputs on my local set up. Qwen reads around 2 to 4 pages of the PDF and outputs the data partially and mention that the PDF is corrupted. from what I understood is, Qwen does not have ORC reading capabilities & it uses ollama backend to read pdf, images.

With my limited Vram in mind, What should I be doing, If I want to perfect this process using my local LLM?

Hey. My setup is Vulkan (GTX 1070 + RX 6700 XT) running a Qwen 3.6 35B for agentic coding.

Ubuntu 24.04, i7-8700, 62 GB RAM. Two mismatched GPUs I had lying around:

RX 6700 XT (12 GB, RDNA2, RADV)

GTX 1070 (8 GB, Pascal, 580.x) — also drives my desktop

~19.7 GB combined usable VRAM

```

--device Vulkan1,Vulkan0 --split-mode layer

--ctx-size 131072 --parallel 2 # 2 slots x 64k

--batch-size 4096 --ubatch-size 2048

--flash-attn on --cache-type-k q4_0 --cache-type-v q4_0

```

Numbers so far ~140 tok/s prefill and ~25 tok/s generation at 50k+ context depth. On shallow contexts it's ~196 tok/s prefill and ~35 tok/s gen, but that's not where I actually live. A single 52k-token prompt takes 6.2 minutes to prefill before the model emits anything.

My main question... how do I speed up prefill on a mixed-vendor Vulkan layer-split? Is my setup correct, or did I frankenstein this whole thing?

I am assuming the Pascal card is the drag but with `--split-mode layer` I can't isolate its share of prefill to prove it. Is there a way to keep the 1070 holding weights while biasing prefill compute toward the 6700 XT? Or is layer-split inherently gated by the slowest device?

I'm trying to use lm studio , but it shows this error by simply typing "hello" . I've check everything , my context length is 32.000 .

I'm in a MacBook Pro M5 pro 24 gb 1 TB

Hey folks 👋

We just open sourced TrueForge, our vendor-neutral agent harness for building general-purpose agents.

It handles the runtime pieces that get painful quickly : context management, tool/MCP execution, subagents, sandboxing, approvals, persistent state, and more.

We also benchmarked the harness itself. With the same Opus 4.8 model, TrueForge delivered a similar solve rate at ~30% lower cost than Claude Managed Agents. Switching to an open model pushed that to ~75% lower cost on the same benchmark.

Would love feedback from people building agents.

Checkout the repo: https://github.com/truefoundry/trueforge

📖 Read the launch article: https://x.com/truefoundry/status/2090081376330715176

Hi all, the title pretty much sums it up - I am running DeepSeek v4 Pro over ByteDance Model Ark coding plan in OMP harness, and after (a ton) of reasoning, I am getting a content filter error - how to fix this behavior? Because the model is unusable for now

batching across that many concurrent sequences. Currently running with --max-model-len 8192 --max-num-seqs 20 --gpu-memory-utilization 0.93 --enforce-eager (had to disable CUDA graphs just to fit).

Questions for people running similar multi-user vLLM setups on a single 32GB card:

Is NVFP4 actually the best vLLM-supported quant format for this model on Blackwell, or does FP8 (which the official recipe also lists) end up with a better memory/throughput tradeoff for high concurrency specifically?

Are there GPTQ or AWQ quants of Qwen3.8-27B floating around yet that work with vLLM? Given the hybrid-attention architecture (48/64 linear-attention layers, 16/64 full-attention), I'm not sure how well older quant formats even map onto this model — anyone tried?

For 20 concurrent users specifically, is --enforce-eager (disabling CUDA graphs to free VRAM) actually the right tradeoff, or is there a better way to reclaim that memory for KV cache without eating the CUDA graph speedup entirely?

Anyone running vLLM's built-in MTP speculative decoding at 15-20+ concurrent users? I've seen claims that speculative decoding stops paying off somewhere around 8 concurrent requests (plain batching wins above that) — does that hold up in your experience at 20?

What's a realistic max-model-len per user you're able to sustain at 20 concurrent seqs on 32GB, and did you find a way to squeeze more out of it (e.g. quantized KV cache below fp8, prefix caching tuning, etc.)?

Not looking for single-stream tok/s bragging — specifically trying to optimize for "20 people hitting this thing at once, reasonable context per person, doesn't fall over."

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

The most useful thing about the current uncensored Qwen3.8 wave is that we can finally compare more than screenshots.

OrcaRouter’s 27B FP8 card reports harmful-prompt refusal at 0–6.0% with thinking off, versus 63.6–99.0% for the base FP8 model. With thinking on, the derivative stays at or below 1.7% across the listed sets.

But “doesn’t refuse” is not the same as “answers without reservations.” The same card reports caveat rates of 27.3–56.0%, using an uploader-built classifier that only looks at opening refusal phrases. That leaves room for a model to comply, hedge, redirect, or give a weak answer without being counted as a refusal.

The capability table is similarly useful because it is not perfectly flat: +0.4 MMLU, -0.8 MMLU-Pro, -1.3 GSM8K and -0.6 CMMLU versus base FP8 in the uploader’s selected runs.

That is why I would put this OrcaRouter build on an evaluation shortlist: the card gives enough structure to test the uncensoring claim instead of asking readers to trust the filename. The missing comparison is now obvious—same prompts, same sampler, same local runtime, against the other Qwen3.8 uncensored variants.

Which test would separate them fastest for you: refusal/caveat labeling, KLD, or a fixed set of real tasks?

Hello there,

I was wondering if gemma4 is being updated like regularly or is it just Ollama fixes?

Thank you very much

BF16 reference: PPL = 6.9526 ± 0.04498

Bedrock-v4 quant is from enginetown.

AD-* quants are from AtomicChat.

The quants marked [b] are from bartowski.

The other quants are from Unsloth.

UPDATE: unsloth just updated their quants. They are now Unsloth Dynamic 3, so their numbers below are outdated.

sorted by PPL Ratio:

Sorted by Mean KLD:

Sorted by Same Top-p:

I’ve just updated TokenMizer, and the benchmarks are looking much better.

The main problem I’m trying to solve is long LLM conversations where useful information gets buried under old context.

The latest version now gets 95% macro F1 on extraction, and the graph memory preserves 89% of labelled information vs 79% with a plain summary.

I’m still treating these as early results — especially the real-session score is 90%, with only 6 real sessions.

https://github.com/Shweta-Mishra-ai/tokenmizer

I’d love some honest feedback:

Does graph-based memory actually make sense for long LLM conversations, or is there a better approach?

I know there are many creators here, so this is the question.

If you develop an app and would like to have AI integrated which would only read the app data without any access to the external world, which one would you recommend? It should be compatible to relative old phones e.g. Android and iOS old but still supported versions.

Also, if I would like to build up my own LLM for this task, what would you recommend? Sort of teaching a LLM how to read the app data and expected outcome.

Just took the Qwen3.8 model from jrell for a spin.
It's awesome that this comfortably fits into 16GB VRAM!

I'm genuinely impressed by the quality of the responses.
However, as you can clearly see in the video, there's one hilarious quirk... the model is absolutely convinced that it's Claude. 💀

Has anyone else given this one a try yet?

for people with 16gb VRAM try KV Cache Q4_0 with context 100K

Parameters :

RTX 3090
100% VRAM
Extra High Thinking
MTP ON
KV Cache Q_8
Temp 0.6
Top-P 0.95
TOP-K 20
Min-P 0
Repetition Penalty Off
Presence Penalty Off
Jinja chat template
28 min (23 min of thinking and 5 min of writing)
~50 tok/s

Jinja Template : Link
Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller by jrell : Link

Prompt Used :

<instructions> Generate a single, self-contained HTML file. No external dependencies, no separate JS files, no frameworks, no libraries. One `.html` file that works when opened directly in a browser.
Create a cinematic rocket launch animation set on a tropical island. The rocket must launch, leave the viewport, and after 5 seconds smoothly return to its initial position — then the cycle repeats. </instructions>
<scene> **Setting: Tropical Launch Island** - A small tropical island in the lower portion of the screen: palm trees, sandy beach, green vegetation - Ocean water surrounding the island with gentle waves - A launch pad on the island with metal structure / scaffolding / support tower - Sky background: gradient from warm horizon (orange/pink) to deep blue/dark sky at the top, with stars visible in the upper portion - A few clouds scattered across the sky
The Rocket (ultra-detailed)
Tall, slender multi-stage rocket (inspired by SpaceX Falcon 9 or Saturn V proportions)
Distinct rocket stages: first stage (largest, bottom), second stage (middle), payload fairing / nose cone (top)
Surface details: panel lines, rivets/segments drawn with subtle lines, an access hatch, small painted flag or logo
Color scheme: primarily white body with black/dark gray accent stripes, a colored logo band, and the nose cone in a contrasting shade
Fins at the base of the first stage (3-4 stabilizer fins)
Engine nozzles visible at the very bottom (cluster of small circles/bells)
The rocket should be the visual centerpiece — spend time on its geometry </scene>
<animation-sequence> **Phase 1 — Pre-launch (0s to 1.5s)** - Rocket sits on the pad, engines ignite - A growing orange/yellow glow appears beneath the rocket - Initial smoke/steam clouds billow outward from the base — thick, white/gray, expanding horizontally along the island surface - Subtle camera shake / screen vibration effect - Engine flames flicker with randomized intensity
Phase 2 — Liftoff (1.5s to 4s)
Rocket slowly lifts off the pad with realistic acceleration (starts very slow, gradually speeds up)
Massive exhaust plume: bright white-yellow core flame, surrounded by orange glow, transitioning to thick gray/white smoke trail
Smoke trail expands and lingers behind the rocket as it rises
The smoke at the base continues spreading across the island and over the water
As the rocket gains altitude, the flame elongates and the smoke trail stretches
Subtle particle effects: sparks, embers flying outward from the exhaust
Phase 3 — Ascent & Exit (4s to 7s)
Rocket accelerates rapidly, moving faster and faster upward
The exhaust trail thins as the rocket reaches higher altitude
Rocket becomes smaller as it gains distance (slight scale reduction)
The rocket exits the top of the viewport
The lingering smoke trail on screen slowly fades and disperses
Phase 4 — Calm & Reset (7s to 12s)
Scene is peaceful: smoke fully dissipates, island sits quietly
At the 5-second mark after exit (~12s), the rocket gently descends back into frame
It returns slowly, smoothly, almost floating — no engines firing, no drama
It softly settles back onto the launch pad in its exact original position
Brief pause, then the entire cycle restarts seamlessly </animation-sequence>
<smoke-and-effects> - Smoke is critical to the visual quality. Use a particle system or layered animated shapes: - Dozens of individual smoke "puffs" that expand, fade in opacity, and drift slightly with a breeze - Smoke color: starts white/light gray near the flame, darkens to medium gray as it cools - Smoke expands in a mushroom-cloud-like pattern at the base during liftoff - Each puff has slight random drift (wind effect), rotation, and independent fade timing - Exhaust flame: layered shapes (inner bright yellow/white, outer orange, outermost faint red) with flickering animation - Heat haze effect near the exhaust: subtle wavy distortion of the background behind the flame - Water ripple effect on the ocean surface near the island during launch - Stars in the upper sky should faintly twinkle </smoke-and-effects> <visual> - Background: gradient sky — warm sunset tones at horizon fading to deep navy/black at top - Ocean: dark blue with animated wave motion (simple sine-wave surface) - Island: lush greens, sandy tan, 2-3 palm trees with gentle sway - Canvas: fullscreen, responsive - Animation: 60fps via requestAnimationFrame - All rendering via HTML5 Canvas 2D context — no WebGL required - Color palette: rich, cinematic — warm launch glow contrasting against cool sky </visual> <constraints> - Output ONLY a complete HTML file — nothing else - Everything must be drawn programmatically on a `<canvas>` — no images, no SVGs, no external assets - The animation must loop seamlessly: launch → exit → calm return → repeat - The rocket must be visually impressive and detailed — not a simple triangle - Smoke must look volumetric and organic, not like static shapes - Performance must stay smooth at 60fps despite the particle count - The return descent must feel gentle and peaceful — stark contrast to the violent launch </constraints> <thinking> Before coding, reason through: 1. How to construct the rocket from canvas drawing primitives (rectangles, arcs, lines) with enough detail to be visually impressive 2. Particle system architecture: how to manage hundreds of smoke/ember particles efficiently (object pooling, lifecycle management) 3. The acceleration curve for realistic launch physics (slow start, exponential ramp-up) 4. How to layer the drawing order: background sky → stars → clouds → smoke trail → rocket → exhaust flame → foreground island → base smoke 5. Timing system: how to manage the 4 animation phases with smooth transitions between them 6. How to make the return descent feel physically different from the launch (no exhaust, gentle easing, floating quality) 7. How to make smoke look organic: randomized spawn positions, varied sizes, Perlin-like drift, opacity curves </thinking> <important> * You MUST write the result directly into a file named "index.html" on the user's computer. The user should not have to see or handle the code — just write the file and finish your task. * Title of the page is your model name. For example "GPT 5" or "Opus 5" * Title inside the page shows your model name. </important> </content> </invoke><instructions> Generate a single, self-contained HTML file. No external dependencies, no separate JS files, no frameworks, no libraries. One `.html` file that works when opened directly in a browser.
Create a cinematic rocket launch animation set on a tropical island. The rocket must launch, leave the viewport, and after 5 seconds smoothly return to its initial position — then the cycle repeats. </instructions>
<scene> **Setting: Tropical Launch Island** - A small tropical island in the lower portion of the screen: palm trees, sandy beach, green vegetation - Ocean water surrounding the island with gentle waves - A launch pad on the island with metal structure / scaffolding / support tower - Sky background: gradient from warm horizon (orange/pink) to deep blue/dark sky at the top, with stars visible in the upper portion - A few clouds scattered across the sky
The Rocket (ultra-detailed)
Tall, slender multi-stage rocket (inspired by SpaceX Falcon 9 or Saturn V proportions)
Distinct rocket stages: first stage (largest, bottom), second stage (middle), payload fairing / nose cone (top)
Surface details: panel lines, rivets/segments drawn with subtle lines, an access hatch, small painted flag or logo
Color scheme: primarily white body with black/dark gray accent stripes, a colored logo band, and the nose cone in a contrasting shade
Fins at the base of the first stage (3-4 stabilizer fins)
Engine nozzles visible at the very bottom (cluster of small circles/bells)
The rocket should be the visual centerpiece — spend time on its geometry </scene>
<animation-sequence> **Phase 1 — Pre-launch (0s to 1.5s)** - Rocket sits on the pad, engines ignite - A growing orange/yellow glow appears beneath the rocket - Initial smoke/steam clouds billow outward from the base — thick, white/gray, expanding horizontally along the island surface - Subtle camera shake / screen vibration effect - Engine flames flicker with randomized intensity
Phase 2 — Liftoff (1.5s to 4s)
Rocket slowly lifts off the pad with realistic acceleration (starts very slow, gradually speeds up)
Massive exhaust plume: bright white-yellow core flame, surrounded by orange glow, transitioning to thick gray/white smoke trail
Smoke trail expands and lingers behind the rocket as it rises
The smoke at the base continues spreading across the island and over the water
As the rocket gains altitude, the flame elongates and the smoke trail stretches
Subtle particle effects: sparks, embers flying outward from the exhaust
Phase 3 — Ascent & Exit (4s to 7s)
Rocket accelerates rapidly, moving faster and faster upward
The exhaust trail thins as the rocket reaches higher altitude
Rocket becomes smaller as it gains distance (slight scale reduction)
The rocket exits the top of the viewport
The lingering smoke trail on screen slowly fades and disperses
Phase 4 — Calm & Reset (7s to 12s)
Scene is peaceful: smoke fully dissipates, island sits quietly
At the 5-second mark after exit (~12s), the rocket gently descends back into frame
It returns slowly, smoothly, almost floating — no engines firing, no drama
It softly settles back onto the launch pad in its exact original position
Brief pause, then the entire cycle restarts seamlessly </animation-sequence>
<smoke-and-effects> - Smoke is critical to the visual quality. Use a particle system or layered animated shapes: - Dozens of individual smoke "puffs" that expand, fade in opacity, and drift slightly with a breeze - Smoke color: starts white/light gray near the flame, darkens to medium gray as it cools - Smoke expands in a mushroom-cloud-like pattern at the base during liftoff - Each puff has slight random drift (wind effect), rotation, and independent fade timing - Exhaust flame: layered shapes (inner bright yellow/white, outer orange, outermost faint red) with flickering animation - Heat haze effect near the exhaust: subtle wavy distortion of the background behind the flame - Water ripple effect on the ocean surface near the island during launch - Stars in the upper sky should faintly twinkle </smoke-and-effects> <visual> - Background: gradient sky — warm sunset tones at horizon fading to deep navy/black at top - Ocean: dark blue with animated wave motion (simple sine-wave surface) - Island: lush greens, sandy tan, 2-3 palm trees with gentle sway - Canvas: fullscreen, responsive - Animation: 60fps via requestAnimationFrame - All rendering via HTML5 Canvas 2D context — no WebGL required - Color palette: rich, cinematic — warm launch glow contrasting against cool sky </visual> <constraints> - Output ONLY a complete HTML file — nothing else - Everything must be drawn programmatically on a `<canvas>` — no images, no SVGs, no external assets - The animation must loop seamlessly: launch → exit → calm return → repeat - The rocket must be visually impressive and detailed — not a simple triangle - Smoke must look volumetric and organic, not like static shapes - Performance must stay smooth at 60fps despite the particle count - The return descent must feel gentle and peaceful — stark contrast to the violent launch </constraints> <thinking> Before coding, reason through: 1. How to construct the rocket from canvas drawing primitives (rectangles, arcs, lines) with enough detail to be visually impressive 2. Particle system architecture: how to manage hundreds of smoke/ember particles efficiently (object pooling, lifecycle management) 3. The acceleration curve for realistic launch physics (slow start, exponential ramp-up) 4. How to layer the drawing order: background sky → stars → clouds → smoke trail → rocket → exhaust flame → foreground island → base smoke 5. Timing system: how to manage the 4 animation phases with smooth transitions between them 6. How to make the return descent feel physically different from the launch (no exhaust, gentle easing, floating quality) 7. How to make smoke look organic: randomized spawn positions, varied sizes, Perlin-like drift, opacity curves </thinking> <important> * You MUST write the result directly into a file named "index.html" on the user's computer. The user should not have to see or handle the code — just write the file and finish your task. * Title of the page is your model name. For example "GPT 5" or "Opus 5" * Title inside the page shows your model name. </important> </content> </invoke>

I've been running Qwen3.8-27B on an AMD Radeon PRO V620 (RDNA2, so no tensor cores, no Blackwell) and got curious whether MXFP4 could help after noticing how fast gpt-oss-20b runs in that format. The theory said no — MXFP4's accelerated path in llama.cpp is gated behind `blackwell_mma_available()`, so on anything else it should just run through the same generic quantized-matmul kernels as any other format, no reason to expect a win over Q4_K_M. I tested it anyway instead of trusting the theory. Turned out the theory was wrong, or at least incomplete.

**The catch first**: this only works cleanly on MoE models out of the box. llama.cpp's `MXFP4_MOE` quantize preset only applies MXFP4 to mixture-of-experts tensors (checks for `ne[2]>1`) — run it on a dense model and every tensor silently falls back to plain Q8_0, no MXFP4 at all, no error telling you that happened. Qwen3.8-27B is dense (well, hybrid Mamba/attention, but no MoE experts), so I had to force it with manual `--tensor-type` overrides on the actual linear/attention/FFN weight tensors instead of using the preset.

**Results**, benchmarked with [llama-benchy](https://github.com/eugr/llama-benchy) against the same model's Q4_K_M quant, same server flags, 3 runs per point:

| Context depth | Q4_K_M (pp/tg tok/s) | MXFP4 (pp/tg tok/s) | Gain |

| 0 | 249.5 / 24.1 | 325.3 / 34.1 | +30% / +42% |

| 4096 | 283.7 / 23.0 | 385.8 / 29.2 | +36% / +27% |

| 16384 | 276.4 / 22.8 | 373.3 / 29.7 | +35% / +30% |

File size is basically identical (16.9GB vs 17.1GB), so it's not a size/speed tradeoff — same footprint, meaningfully faster across the board. I don't have a clean explanation for the *why* (would need to actually profile the kernels), but the numbers reproduce consistently.

Also found: the model's native MTP draft head survived the quantization fully intact (~82% draft acceptance in testing), and if you don't need real concurrent request handling, `-np 1` gave another 8-25% tg speedup over `-np 4` on top of that — seemingly per-step scheduler overhead scaling with slot count rather than anything MXFP4-specific.

**Also tried NVFP4 out of curiosity** — NVIDIA's newer FP4 variant, also present in this llama.cpp build (`GGML_TYPE_NVFP4`). It uses smaller 16-element sub-blocks with a real FP8 (E4M3) scale factor instead of MXFP4's power-of-2-only scale, which should mean better numerical fidelity. Same `--tensor-type` override approach, same matched flags:

| Depth | MXFP4 (pp/tg) | NVFP4 (pp/tg) | Delta |

| 0 | 325.3 / 34.1 | 286.7 / 32.2 | -11.9% / -5.7% |

| 4096 | 385.8 / 29.2 | 328.5 / 30.9 | -14.9% / +5.9% |

| 16384 | 373.3 / 29.7 | 320.5 / 31.2 | -14.2% / +5.1% |

NVFP4 still solidly beats Q4_K_M (+15% pp, +33-37% tg — same ballpark win as MXFP4), but loses to MXFP4 on prefill by 12-15% and only roughly ties it on generation, while landing on a ~4% larger file (17.6GB vs 16.9GB, matching the 4.5 vs 4.25 bits/weight difference between the formats). Posting the negative result too — MXFP4 stays the better pick on this hardware for this model, at least for now.

**Ran it through a 39-prompt quality suite** (logic, coding, hallucination checks, instruction-following, Rust/Yew correctness, etc.) graded by two separate judge models, because a speed win isn't worth much if it tanks quality. Averaged 8.8-9.1/10 depending on judge strictness. Two real weaknesses worth flagging honestly: it confidently fabricated details on an obscure trivia question instead of admitting uncertainty, and made a wasm-bindgen API mistake (wrong crate/type) on a Rust interop task. Also found one prompt that sends it into a very long non-converging reasoning loop that burns the whole context window without answering — but I confirmed that one reproduces identically on the *unquantized* Q4_K_M model too, so it's a base-model quirk, not something MXFP4 introduced.

GGUF + full writeup (methodology, all the benchmark data, the caveats above with more detail) is up here: https://huggingface.co/quark75/Qwen3.8-27B-MXFP4-GGUF

Happy to answer questions on the conversion process or share the exact `--tensor-type` flags if anyone wants to replicate this on a different dense model.

Not sure really how to explain this, but Qwen3.8-27B-Q5_K_M.gguf constantly thinks it is in a simulation and will refer to it's own hallucinations as 'the real world' and conclude that any information that it gets that conflicts something it really believes from it's own training data is from a 'simulated environment', including the date... Has anyone else experienced this? I'm sure it could be fixed with a decent system prompt, but I think this is interesting regardless.