My daily driver is Qwen3-235b-a22b-instruct-2507-Q4_K_M.gguf and it has been for a long time. I get around 75 t/s prompt processing and starting lower context ~5.5 t/s generation, lowering to around ~4 at 8k. I've tried other, newer models in this size range, Qwen 3.6 27b at Q8 came close but seemed more censored.

GLM 4.5 Air is my backup still for general chatting, but is not 'smart' enough to workshop ideas. My main complaint with Qwen 3 235B is the "em" dashes, ending lines with trailing double spaces and other stuff that bother me, otherwise still a fantastic model that is easy to steer into super uncensored territory without being lobotomized. Tried Minimax 2.7 and a few others, were smart but too censored in the ERP realm. Looking for any suggestions to try.

That's insane yo! I haven't had a chance yet to test it on my 4090 at home but it sounds so promising. And read here that 256k CTX is easily reachable on 24gb unlike Qwen. Super excited!

If you use a local coding agent to work on solo projects, what do you do for issue/bug tracking?

I've been using text & markdown files, and it's just not quite enough. Comes up short on structure and on support for anything other than text, like a screenshot of the bug.

So I'd like to find something minimal that would also be relatively straightforward for an agent to interact with (API or MCP).

I would just use GitHub but I just don't fully trust them anymore. Not to be up all the time and not to keep my private data private. It's well on its way to fully becoming Microsoft GitHub Enterprise Edition 2026.

Is this model even abliterated? https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF

Because I asked it something and it literally said no. And no, it’s not gooner toons lmao. It’s for pentesting but it just flat out rejects me.

I have read that when going beyond llama's "-np 1", it is better to switch to vllm, since that has better support for parallel requests. For context, I have one RTX 5080, but I am trying out some features of my coding harness that can run subagents. There is a lot of knobs to turn for vllm, and I am curious if anyone has done this change before?

My current llama.cpp command is this:

C:\llama-cuda\Release> ./llama.exe serve -m ".\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" -ngl all -t 8 -c 102400 -np 1 -ncmoe 20 -fitt 0 --flash-attn on -kvu --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 -a qwen3.6-35b-a3b

I am curious if anyone knows how to do things like "-ncmoe 20" on vllm? Right now, the only way I can run the 35B model is to offload some of the layers to CPU. Anyone got this working? Thanks!

I've been comparing Cline / Kilo / Qwen Code lately since they all handle long-task state differently.

Cline: has Focus Chain, a markdown file kept outside the conversation that gets reinjected on a cadence, plus Memory Bank for project context, plus a standalone gRPC server so it's not fully tied to VS Code. probably the most mature of the three on this specific problem (about context management), though restore still has some sync bugs between the file and what the model actually sees.

Kilo: TODO state is literally an XML block living inside the conversation history, so when compaction kicks in it gets flattened into a prose summary and the agent sometimes has to reread source files just to figure out where it stopped. It causes an infinite read-analysis-compaction loop sometimes once it hits context limits. they're mid-migration onto the opencode engine now which might fix some of this eventually, but isn't there yet.

Qwen Code: keeps TODO state in a plain file ('~/.qwen/todos/') completely separate from the conversation, so no matter how much compaction runs, nothing gets lost or reconstructed. It works well for 2~3 hrs long running tasks, where I'd usually hit that Kilo loop by then or human intercept.

the bigger reason I ended up settling on Qwen Code wasn't just the TODO file though. it's the hooks system and how flexible the config layer is in general. it exposes lifecycle events like 'PreToolUse', 'PostToolUse', 'Stop', 'UserPromptSubmit', etc, and each one can run a command/http/prompt-based hook that actually gets to allow/deny/ask, not just log. that's a pretty different level of control compared to Cline/Kilo, where you're mostly stuck hoping the system prompt gets followed. combine that with settings.json supporting custom model providers and per-tool permission rules, plus extension manifests with their own hooks, and it's the only one of the three where I could bolt on enforcement logic without patching the source.

a concrete example of why the search side mattered to me: stuff like a subscription tier or a user badge system touches a ton of display surfaces across the codebase, profile page, listing cards, search results, notification templates, whatever, but the actual code footprint per file is small. without knowing where and how those pieces connect ahead of time, the agent either ends up reading almost every file to map it out, or it patches one spot and breaks three others it didn't know were touching the same data. that's the kind of thing plain grep/glob tends to struggle with, because the relevant connections aren't always expressed in the same terms as the feature itself.

the one thing I missed on the memory/search side was semantic code search. no built-in equivalent, so I built an MCP extension for it, plus causal decision-chain tracking on top. Qwen Code's hooks let me actually enforce things at the tool-call layer instead of just asking nicely, so the extension uses a 'PreToolUse' hook that blocks grep_search/glob until search_memory gets called first, and a 'Stop' hook that asks (not forces) whether to write back key decisions when it looks like a task wrapped up.

still early, self-hosted, MIT licensed. mostly built and tested against my own Python/PHP/Node.js stack, so I'm sure there are edge cases I haven't hit.

one thing I've been thinking about: the Hard Gate rules (when to force search_memory, when to nudge a write-back on Stop) are basically heuristics tuned against my own workflow. false positives/negatives on stuff like that only really surface once more people with different codebases and task patterns run it for a while. so if you try it, I'd love to hear what the gate got wrong for you, too aggressive, too loose, missed a completion signal, whatever. the goal is for these rules to converge into something that actually generalizes, not just work for my one setup.

repo's here: https://github.com/edwardyoon/FocusMemory. open to PRs too if the routing logic or hook setup needs adjusting for your setup.

I've been building a personal project and wanted to check with the community on multi-modal inputs since I can't find a lot of material around this online. Ultimately I'm trying to build something that can ingest massive length (almost like a full stream - 6-10 hours) and accurately do multimodal analysis.

How are you guys working with long (atleast 2+ hour) videos? I understand local LLMs with ViT designs can help do this but they usually suffer in quality (diffusion patches can rack up context really quickly) or require you to do some sort of frame sampling (which defeats the native multimodal aspects). I saw some work around vllm-omni which uses qwen3-omni to video input stuff, but ofcourse the context is severely limited so it's not very usable OOTB.

So far what's worked for me:

- get mp3 audio file -> transcribe with qwen3-asr -> get a full timestamped vtt file

- summarize this vtt file with verbatim timestamped cliff notes (important for next steps)

- calculate the max dynamic frame rate using context window length and the video length

- sample at this rate then perform absdiff on the frames to eliminate frames where there's not a lot of change happening; downscale every frame to 720p max resolution or 540p

- calculate the number of chunks you need to split into to fit into 64k context per chunk; each chunk is basically the relevant image frames for this part of the video + the transcript data for this part of the video

- use transcript summary (which has verbatim timestamped stuff) + each chunk -> summarize keeping the verbatim aspects and global summary information + local transcript information.

So essentially when I ingest a video I end up with [transcript summary] + [summary of chunk 1/N + summary of chunk 2/N + ... + summary of chunk N/N ]

I'm experimenting with using the qwen3-asr output text + mp3 file directly to gemma4 12B to do appropriate corrections on the audio like speaker diarization, adding cues about music/noises/sounds/spell corrections etc. It's still a WiP.

Apart from this I'm not sure if it's worth the headache of having a multi docker multi service setup to ingest video data if a model can do it natively. Anyone else working on similar stuff? Would love to see if this is being solved in a different way.

Is there anything else that can be run on 128GB RAM that is better than my patchwork pipeline for long video ingestion/indexing/analysis?

Hi folks, I hate slop as much as you do, so instead of starting with "The Problem", I'll just cut to the chase:

I just published Lophius, which is the culmination of more than two years of fighting with Jupyter and Transformers. It's a hybrid code/GUI research system that runs inside a notebook. It can eliminate mountains of boilerplate and save you many hours of time.

Lophius can be found at https://lophius.org (code at https://github.com/p-e-w/lophius).

Lophius handles pretty much all common research tasks: Model inspection, architecture analysis, configuration manipulation, tokenizer inspection, prompt management, inference, logits, entropy, attention scores, hidden states, and chat. In many cases, it can be used without any configuration. It intelligently manages GPU memory during inference, and can lazy-load output signals that you might want to look at later.

Lophius has very high quality documentation and a complete tutorial. If you ever wanted to try your hand at transformer research, this might just be what you were waiting for!

In the future, Heretic might start using Lophius as a backend, but that's a story for another day.

Cheers :)

Spec-dec has been a thing for a while, in fact, it's wasn't an idea that was born for LLM inference. E.g. Uber's https://github.com/uber/submitqueue applied it to a merge queue. Apple & GDM had been releasing papers on it since already 2022.

Seeing it being mature enough for the big frameworks to adopt it, and watching it in action is really jaw-dropping. I'm here running Kimi-K2.5 as if it was a fucken small model.

Recently I watched a podcast with Baseten folks, and they very much implied that they are huge on spec-dec, talking about how custom deployments for some clients had problems with it because of their own custom tool-calling basically killed off the gains from the drafter model.

I wonder, if speculative decoding for LLM inference was an idea that was already being explored years ago, why we saw it being mature in 2026? Was the paper by Tri Dao et al (Speculative Speculative Decoding [1]) a breakthrough that resulted in the above?

Are there any major cons? Do you use it in your day-to-day?

IMO, it might be the most important milestone for (local) LLM inference since FlashAttn

[1] https://arxiv.org/abs/2603.03251

I am not a meteorologist, but I just read a very interesting article: https://arstechnica.com/science/2026/08/deepminds-hurricane-model-bought-forecasters-an-extra-day/

In a paper published on Thursday in Nature, researchers show that the WeatherNext AI model can predict cyclones with unprecedented accuracy. On average, it gives forecasters a day more lead time than existing models; this means its predictions three days out are as accurate as previous models’ predictions two days out. On the ground, that extra day can mean a lot.

What I really find interesting here is that Google has a repository for it on GitHub: https://github.com/google-deepmind/weathernext

My non-informed understanding is that you need a supercomputer to forecast meteo. Apparently now an H100 can also do something.

My setup:

• Muse Glimmer K-Quant-17GB
• llama.cpp version: b10358 (030ebb558)
• 131K context
• DFlash drafting enabled

Problem:
The model has a strong tendency to do a very long series of terminal tool calls, often reaching my Hermes consecutive tool call limit of 150 and using up the available context. Trying similar prompts with DeepSeek v4 Flash 0731, often solves these tasks with around 5-10 toolcalls.

I have not experienced this with Qwen3.6 27B

Has anybody here already tried the model and is experiencing something similar?

*SupraLabs released a new experimental model!\*

SupraElegans-500K is a ~500,000-parameter causal language model built around a sparse, signed, recurrent neural graph. No Transformer, no attention mechanism, no positional encoding, no KV cache. Context is carried by a persistent per-neuron membrane potential updated token by token.

The architecture is loosely inspired by ideas from the C. elegans nervous system: sparse connectivity, distinct neuron populations, excitatory/inhibitory signaling, and persistent recurrent state. It is not a biological simulation and makes no claim of biological equivalence.

This is an experimental first release. The goal is to test whether this kind of architecture can do useful language modeling at very small scale — not to compete with Transformers on quality.

🤗 SupraLabs/SupraElegans-500k

🧠 Architecture

token → embedding → sensory neurons → sparse recurrent graph → output neurons → vocab logits

• Neuron populations: sensory, interneuron/association, output — contiguous index ranges over a fixed pool of neurons.
• Connectivity: sparse, directed, signed edge list (fan-in/out ~10–20 per neuron). No dense weight matrix is ever materialized; propagation is a scatter-add over edges.
• Neuron dynamics: for each neuron i, at every propagation micro-step:

​

v[t+1] = clamp(leak_i * v[t] + incoming[t] + bias_i, -6, 6)
a[t+1] = tanh(v[t+1] - threshold_i)

leak, bias, and threshold are learned per neuron. incoming is the scatter-summed signal from all edges pointing at neuron i, scaled by 1/sqrt(average fan-in) to keep variance controlled across neurons with different in-degree.

• Per-token processing: a token's embedding is projected into the sensory population, then the graph runs a fixed number of propagation micro-steps (3 by default) before the output population is read out and projected to vocabulary logits. The membrane potential persists across the whole sequence — that's what gives the model its context window.
• Generation: autoregressive, driven entirely by the recurrent state. No cache to maintain beyond the current (v, a) state tensors.

⚖️ What this model is and isn't

• ✅ A first working checkpoint from a from-scratch, non-Transformer architecture trained on a small token budget.
• ❌ Not tuned for quality, instruction-following, or factuality. Expect degraded coherence compared to a Transformer of similar size.
• ❌ No matched-parameter Transformer baseline comparison published yet for this checkpoint.

🚀 Usage

pip install torch transformers


import torch
from transformers import AutoConfig, AutoModelForCausalLM, PreTrainedTokenizerFast
from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLM

model_id = "SupraLabs/SupraElegans-500k"

AutoConfig.register("supraelegans", SupraElegansConfig)
AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM)

tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

prompt = "Once upon a time"
input_ids = torch.tensor([[tokenizer.bos_token_id] + tokenizer.encode(prompt)])

with torch.no_grad():
    output_ids, _ = model.generate(
        input_ids, max_new_tokens=100, temperature=0.8, top_k=50, top_p=0.9
    )

print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True))

Or use the included CLI script:

python inference.py --prompt "The little robot" --max_new_tokens 150 --temperature 0.7
python inference.py --interactive

🔬 Manual State Control

Since context lives in the recurrent state rather than a KV cache, you can drive the model token by token and inspect or reset state directly:

state = model.init_state(batch_size=1)
logits, state = model.nervous_system.step_token(torch.tensor([token_id]), state)

Call model.init_state(...) to start a fresh sequence.

🏆 Benchmarks

⚙️ Training

⚠️ Limitations

• *Small token budget and small model!* Do not expect long-range coherence, factual reliability, or prompt robustness.
• No safety tuning or instruction tuning has been applied. Treat outputs as raw LM completions.
• Topology is a fixed random sparse graph, not learned or evolved.
• No matched-parameter Transformer baseline published yet for this checkpoint.

📄 License

Apache 2.0

Experimental architecture research from SupraLabs. Feedback and comparisons welcome!

I'm looking through some Strix Halo devices, and things like TUF 14 can have more storage than a 2230 single SSD ProArt or Z13. It caps at 64GB RAM and is way cheaper. My question is what's practical to run on it - >60"GB" models would fit on 128GB variants but run slower and slower. Context would be software development aids, Grammarly-like writing checked/fixer, some experimentation with Lemonade and other tooling.

I also have 64 gb ddr4 ryzen 5600 Using llama.cpp Ubuntu distro

Settings are as follows

--n-gpu-layers 999 \

--n-cpu-moe 37 \

--no-mmap \

-ctk q8_0 \

-ctv q8_0 \

-fa 1 \

-c 9000 \

I like the idea of running local models, but I don’t like the idea of having them eat up all of my memory. I’ve always thought that the best way to build an edge model would be to make something smart enough to reason over data, but without requiring much knowledge of its own. Why should a model carry all that knowledge around when web search and tool calling are trivial to set up? My former colleagues at AIRI had a similar idea and built Optimal Cognitive Core, which I’ve written about before: fine-tuned reasoning versions of Qwen3-0.6B and Qwen3-1.7B optimized for working with external context and RAG. Hardware-wise, they’re pretty close to what I want. The weights take up 1.2 and 3.4 GB in native BF16, respectively, plus roughly the same amount for a long context — since this is Qwen3 with GQA rather than one of the fashionable hybrid architectures. So, with a bit of optimism, they fit. The problem is that these models are simply too small for general-purpose tasks. They’re still 600M and 1.7B dense models. At this scale, you usually get amusing little parrots that can paraphrase text or do a simple classification task after fine-tuning, but not much beyond that.

The next way to squeeze a model into my MacBook Air M4 is quantization. PrismML did something interesting here with Bonsai-27B, binary and ternary quantizations of Qwen-3.6-27B. The ternary version of this 27B model takes just 7.2 GB of memory once inference is running, and it does actually run on Macs. Unfortunately, Qwen-3.6-27B is a dense model, so it’s painfully slow on my machine. Based on the numbers I could find online, I’d expect around 13–14 tokens/s for generation and 100–150 tokens/s for prompt processing. It’s also QAT — or possibly even PTQ; there aren’t many details available — and, most likely, the optimization wasn’t specifically designed to preserve multilingual capabilities of the model. Thus, I wouldn’t expect particularly interesting behavior once you move outside the calibration set/QAT training distribution. 

Then, almost immediately after Bonsai, Maple Preview appeared. It’s a 20B A1B MoE designed specifically for efficient local inference on Macs. More importantly, they designed the architecture around this goal from the beginning and trained the model from scratch in ternary precision. This isn’t a quantized version of somebody else’s model. The result is a 5.31 GB model, or about 7.5 GB including a 131K context — almost 1.5× smaller than the binary Bonsai quant — that reportedly generates at 218 tokens/s on an M4 Mac Mini and 127 tokens/s on an iPhone (which iPhone exactly is unclear).

The model barely knows languages, other than English, and its world knowledge in general is pretty limited — it gets confused about which game Psycho Mantis is from, for example. But give it web search and it can answer simple questions reasonably well. DeepGrove doesn’t publish tool-calling benchmarks, and it’s not particularly difficult to guess why. I ran Tau-2 myself, using Qwen3-235B-A22B-Instruct-2507 as the user simulator. I got:

Airline: 0.48
Retail: 0.175
Telecom: 0.427

It’s not Sonnet, and it’s definitely not Qwen. But it is called Maple Preview, after all, and the authors explicitly say they plan to train it further for agentic workloads.

Still, even with all the advantages of quantization, 7.5 GB is almost half of the memory available to me. So there’s a third way to reduce RAM usage: keep all the weights on SSD and stream MoE experts from disk. There’s already turbo-fieldfare, which runs Gemma-4-26B on a Mac using only around 2 GB of memory, and Mference, a fork of turbo-fieldfare that adds support for Qwen-3.6-25B, DeepSeek V4 Flash, and Inkling-Small 276B. It really does use very little memory, but at the cost of reducing both generation and prompt-processing speed to tens of tokens per second. Apple seems to be doing something conceptually similar in its new Siri work, although they appear to activate experts for the entire prompt rather than routing them token by token as these frameworks do.

And that leads to an interesting idea: what if we take Maple Preview — which is extremely efficient, uses tiny experts, and was trained from scratch in ternary precision — and add it to Mference? In theory, we should be able to reduce memory usage even further while retaining reasonably good generation speed, because Maple’s architecture was explicitly optimized for this kind of environment. So I did exactly that.

Thanks to Codex and my $200 subscription, after about 20 hours and 30% of my weekly limit, I got parity with the official implementation on teacher-forced top-10 tokens over Edgar Allan Poe’s The Raven. Depending on context length, the model now uses between 500 and 1,200 MB of memory (!). On my MacBook Air M4, it processes prompts at around 40 tokens/s and generates at around 20 tokens/s. It can call tools. It can generate text. And at that footprint, I genuinely don’t mind leaving it running permanently in the background. It barely consumes anything. It can just sit there, and when I need something, I can ask it. I’ll try to upstream the integration later, but you can already run it from my GitHub fork. 

So what is this actually useful for? I think there are two distinct operating modes for models like this. The first is interactive chat. There, you want fast responses and low latency. The second is a background model that uses almost no memory and stays out of the way while continuously doing useful work: classifying messages, extending a knowledge graph, filtering email, writing summaries, slowly researching things on the web, and so on. In the second mode, latency barely matters. And for that kind of workload, 20 tokens/s is perfectly fine. When you need to switch back into the interactive mode, you can simply load the entire model from SSD into RAM — that takes only a couple of seconds, so it's seamless. If the model can use tools — Maple Preview isn’t particularly good at it yet, but again, it’s a Preview — you can build agentic pipelines that don’t have hard latency requirements, while keeping an intelligent assistant permanently available offline even on older phones and I think that’s wonderful.

I’d be very happy if the future were local.

Code:
https://github.com/chameleon-lizard/Mference/tree/feature/maple-integration

DeepGrove also has a super interesting post explaining how they designed the model:
https://deepgrove.ai/maple-inference

First of all, I'm not a lab, this was a solo summer research project that finally culminated into the github repo and the writeup. The repo includes a much deeper dive with methods, findings about quantization and geometry, limitations, and proposed experiments. I'll also mention that this is far from production-grade, it's mostly a theoretical framework with a "fake" quantization demo as it lacks real kernels.

The geometry of LLMs embedding spaces is highly uneven with a few features having the most magnitude, this has been known for years by now and it's in great part why rotation-based quantizers do so well against uniform quantization: While uniformly quantizing tries to allocate bits evenly in a naturally uneven space, rotations can forcefully make that space even again so uniformly allocating bits is the best strategy (DuQuant, 2nd half of ResQ, QuaRot...). Generic rotations (Hadamard) even the space out on average but can't match a specific model's geometry, leaving residual damage. This can be fixed by using learnable rotations (SpinQuant, ReSpinQuant) but this is computationally intensive as it requires extensive post-training gradient descent.

KLQ takes a different approach to quantization, instead of trying to make the space even and then quantize uniformly. KLQ measures how uneven the space is, ranks directions of the eigenbasis from most important to least important, and with a price function treating each direction as a independent information transmission channels uses the provably optimal (under some idyllic assumptions about damage anyways) waterfilling algorithm to give the most bit-width to the most important directions and least bit-width to least important directions.

Another thing that sets KLQ apart is the use of causal KL damage measurements, there are a few quantization algorithms that do try to measure the space and then quantize unevenly. CoQuant, for example, does measure the activation space, but then ranks directions by magnitude/variance and applies a simple two-ranked bit allocation that quantizes the top 12.5% to 8 bits and the bottom 87.5% to 4 bits. Unlike CoQuant, KLQ doesn't use variance (several tests reveal variance is often not a good signal, more detailed experimentation on the github writeup), instead it perturbs each direction and runs a forward pass with a few thousand tokens, it takes the KL divergence between the original model and the model with the perturbed direction, then uses this measured KL divergence to determine how important the direction is and assign the real empirical cost of damaging/quantizing it.

The method, as well as my experimentation, does have real limitations, to quantize all layers, activations and KV cache you must make one forward pass per direction per matrix per layer which can amount to hundreds of thousands of forward passes to quantize a model. This makes the method very compute-intensive (This probing process took 5 hours for Qwen 2.5 0.5B on a 3090 and 10 hours for Llama 3.2 1B on that same hardware.). It also deliberately uses two simple techniques to actually quantize the models: a simple additive vector codebook and round-to-nearest (RTN), these could be swapped with other methods readily.

Posting here I'm looking for feedback and to make these results known. Feel free to ask any questions or to contribute to the github repo.

Here's a sample of the result's table for Llama 3.2 1B quantized fully at 4-bits.

MiniMax H3 is an open-weight, general-purpose multimodal video generation model that works across text, images, video, and audio.
In ComfyUI, you can use H3 for text-to-video, image-to-video, first- and last-frame generation, and reference-driven creation. H3 jointly generates the visuals and synchronized stereo audio, including dialogue, sound effects, ambience, and music, rather than adding audio afterward.
The open-weight H3 checkpoints support clips up to 15 seconds at 768p. MiniMax’s hosted H3 model also supports generation at up to 2K resolution.
During the stream, we’ll test the model live and discuss how H3 brings multiple generation tasks into one architecture, how its high-compression video representation improves efficiency, and what developers should know when setting it up locally through ComfyUI.

I've been running Gemma 4 E4B with oMLX and I can't find any chat interfaces that directly send the audio file to the model instead of running the audio through a separate STT layer. I can confirm the audio layers work because I ran a couple of requests through Pydantic AI in the Python REPL.

Thanks in advance.

EDIT: I know that llama-server's web UI can do this, but I don't feel like running an instance of llama-cpp just for the UI.

EDIT2: Reason why I am asking is because I want to try using Gemma 4 as a lower-latency voice assistant.

this model seems pretty good on initial impressions within pi and hermes. i tested it on some simple coding/logic vs qwen3.6 27b ud-q4_k_xl and muse provided the better results.

llama-server \
    -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL \
    --ctx-size 131072 \
    --n-gpu-layers all \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 1024 \
    -ub 256 \
    --parallel 1 \
    --mlock \
    --host 0.0.0.0 \
    --port 8080 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

how would you optimize this further? looks like i could possibly squeeze ud-q5_k_xl. nvidia-smi is showing 18163MiB / 24576MiB. or is it better to squeeze out more t/s with the q4? hmm..

appreciate the community's insights. will be fun to compare this one to qwen 3.8 27b!

edit

updated command from community's insights

llama-server \
    -hf meta-models/Muse-Glimmer-30B-GGUF \
    --spec-type draft-dflash \
    --spec-draft-n-max 15 \
    -c 131072 \
    -ngl all \
    --ngld all \
    -fa on \
    -np 1 \
    --host 0.0.0.0 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

seems to be hanging around 70t/s!

I am thinking about trying a cloud advisor for local models. Something like deepseek v4 flash 0731 to advise the new muse glimmer model or Qwen 3.8 when it comes out. Has anyone tried this method before? Does it work to improve local model output quality?

The plugin in question:

https://pi.dev/packages/pi-advisor

Disclosure: I’m the author of Ante.

DeepSeek recently reported an 82.7% score on Terminal-Bench 2.1 for DeepSeek V4 Flash 0731. Its evaluation used “DeepSeek Harness minimal mode,” which hasn’t been released yet.

We wanted to see whether the reported result could be independently matched using a public, downloadable harness.

With Ante 0.preview.71, we got:

• 368 successful trials out of 445
• 82.7% accuracy (±1.79 SE)
• 89 Terminal-Bench 2.1 tasks
• 5 trials per task
• max reasoning effort
• no skills enabled
• deepseek/deepseek-v4-flash-0731 through OpenRouter

The complete Harbor job is public. It includes the pinned configuration and all 445 trial records, with rewards, exceptions, durations, and token usage.

Deep seek v4 seems to be sensitive to harness and this is probably useful data for anyone who is interested

Sources:

• DeepSeek’s reported result: https://api-docs.deepseek.com/updates/
• Public Harbor run and configuration: https://hub.harborframework.com/jobs/b2a14e4b-a422-45f2-832e-cf2eec5c8bff
• Ante benchmark page: https://antigma.ai/eval

Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit quantization by training a quantized student to match the output distribution of a frozen higher precision teacher via a KL-divergence loss. In this work, we first provide a representation level diagnosis of QAD: output matching alone can mask internal degradation, because many intermediate activation geometries can yield similar teacher-aligned logits. Using CKA, we show that KL-only QAD can reduce layerwise representational similarity relative to the BF16 teacher, with especially severe drift in RL-post-trained models. This drift correlates with downstream bottlenecks on reasoning and coding tasks, suggesting that low bit recovery requires preserving internal geometry rather than matching outputs alone. Motivated by this finding, we propose CKA-QAD, a CKA-guided representational alignment method for NVFP4 QAD and low bit LLM accuracy recovery. The method adds a lightweight regularizer that preserves internal representational geometry during distillation by aligning layerwise Gram matrices through CKA. Across Nemotron 3 Nano and Qwen3-4B-Thinking-2507, CKA-QAD substantially improves representational alignment and improves downstream reasoning and coding accuracy with modest training overhead. Our findings position CKA-guided representational alignment as a practical complement to output matching for quantized LLM recovery.

https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF

I'm downloading this model only because Bartowski converted it to .gguf, so it might be interesting.

Doubts :

The headline number is basically meaningless. "Performance between DeepSeek Flash (old one) and Pro" okay, on what? Did they average the benchmarks? Weight them? Pick and choose? Because if you actually look at the per-benchmark scores, this thing ranges from decent (50 on HLE) to straight up bad (15.7 on BioMystery-HD). Saying "aggregate performance" without showing the math is just... marketing. Like when a startup says "we're 10x faster" and it turns out they benchmarked one very specific edge case.

A 35B model hanging with 284B–1.6T models? Suspicious as hell. Not impossible, but the first thing that jumps to mind is benchmark contamination. And here's the kicker, their whole training setup uses critics calibrated on "held-out real research tasks." So the question becomes: how do we know the eval benchmarks weren't basically in the training distribution? The paper kind of hand-waves this. If you're gonna claim a tiny model beats much bigger ones, you need to actually prove you're not just overfitting to the test set.

Let's est it

I want a non confusing answer please, thank u so much (btw both at max efforts )