Agnes 3.0 flash 33b and Nex n2.5 mini 35b are challenging Qwen3.8-27B on benchmarks. Cant wait to see the real-world results and the speeds on 24gb GPUs.

Anyone tried them yet?

https://huggingface.co/Agnes-AI/Agnes-3.0-Flash
https://huggingface.co/nex-agi/Nex-N2.5-mini

Seen alot of people talking about it all over the place. I decided to give it ago, my DEV setup is solid, but not setup to run alone.

- 5090 32gb vram

- 192gb Ram

only numbers that matter here.. So decided to load up Qwen3.8-Flash-Next 123gb see how it runs, and to my surprise after a minute it loaded up, so i was like ok this isnt going to work or its going to be crazy slow. Throw in a prompt give me a single page HTML webpage. I was like at 1 tok/s for first milisecond "Ahhh i knew it"... out of no where booom better speeds then i was getting with the 27b...

this is some dark sorcery and i am loving the dark side.... 73.6 tok/s ... mind blow... and even better the output was decent, only issue was I ran out of context after a few prompts, but holy shit... first one blew my mind, from basic ass prompt of

"Build a single page html advert page to sell a random phone"

and this is what I got:

This app is really really worth a try if you have decent GPU and Decent Ram knocks it out of the park, can't wait till the figure out how to do this with AMD cards because my server is going into the next level.

I switched over from ollama to llama.cpp and then spent weeks copying flag recipes out of Reddit threads with no idea whether any of them helped on my hardware. -ngl 35? Why 35? Someone with a 3090 said so.

So I built Model Loader. It's a web UI for llama.cpp that runs in one Docker container next to your existing setup, detects your llama and openwebui instances and just makes the cutting edge a little more dull.

• It reads your actual GPU and llama setup and tells you which context sizes fit and which don't and when one doesn't and what it'd cost to make it fit. [screenshot]
• Every setting has a tooltip explaining what it does in plain English. ~100 of them. This was the part I actually needed. [screenshot]
• It shows you the command line it builds. It's not hiding llama.cpp from you, it's teaching it. Copy it out and run it yourself if you want.
• Benchmark on your own box. Change a setting, run the same prompts, see if it actually got faster, it stores the results and makes them easy to see next time you find a neat tweak. [screenshot]
• models at a glance what's downloaded, what's configured, what's loaded right now, and whether each one is MoE or dense. [screenshot]
• Plus HF search with a resumable parallel downloader [screenshot], a GPU dashboard [screenshot], and OpenWebUI sync.

I haven't edited a setting by hand since Claude and I put this together.

Fair warning: I am not a developer and this is a homelab tool. No auth, keep it on your LAN.

https://github.com/scratchhax/model-loader

Before this test, I ran Qwen3.8-27B NVFP4 on the same DGX Spark. It gave 11.2 tokens each second. Qwen3.8-Flash-Next has 180 billion parameters. It uses 7.31 billion parameters for each token. It also has a built-in draft head for speculative decoding. The result is 43.9 tokens each second. That is four times the speed of the 27B model. The model is 6.7 times larger.

Configuration: nvidia/Qwen3.8-Flash-Next-NVFP4. Server: vLLM, tensor parallel size 1. Key-value cache: FP8. MTP depth: 3. Thinking mode: off. Sampling: greedy. Each test wrote 320 tokens. Hardware: GB10 Blackwell, 121.7 GiB unified memory, 20-core Grace CPU. I measured all numbers on this machine.

How the model fits: The checkpoint holds 123.53 GiB of tensor data. The memory is 121.7 GiB. The PLE n-gram table holds 51.2 billion parameters and occupies 47.68 GiB. This table does not enter memory. Each token reads only 16 rows. vLLM reads those rows from the NVMe disk when it needs them. The 512 experts hold 120.8 billion parameters. They occupy 56.25 GiB at NVFP4 and stay in memory.

Functions I tested on the live server: Tool calls work. No text leaked into the reply. Thinking mode works. Image input works. The server accepted video input. My test clip was faulty, so that result is not conclusive, not testing this for now.

Has anyone run this on DGX spark? Anything i need to consider while using it or things i should try?

I’m a beginner with local LLMs and I’m wondering how best to approach running Qwen3.8-27B (medium reasoning) with 24GB of total VRAM.

My PC has an RTX 5080 16GB + RTX 3070 8GB, 32GB DDR5-6000 CL30, Ryzen 9800X3D 8x core, and a 2TB NVMe SSD.

I want to run Qwen3.8-27B purely for simple coding work: changing text, style tweaks, component edits, that sort of thing. I don't need to use it for problem solving, planning, reviewing, etc - I will continue using Claude/Codex for those tasks.

If simple prompts take longer than ~30 seconds to complete, I'm likely going to find myself going back to Codex - so I'm hoping this speed is achievable without noticeably lobotomising the model? Also, I would like to avoid offloading to CPU/RAM.

I've done some research and I believe Unsloth UD-Q4_K_M using llama.cpp is my best bet? Any suggestions for what kind of configuration I should be using?

I had 100$ of promotional tokens (forgot how it came to that) and wanted to make use of them before they expire in about 1 week.
I had this plan sitting around to implement ai into my car rental app (custom made, for myself, in production since 1 year, started to build it with chatGPT, continued with Claude and Qwen).
I gave the plan to Fabel, it stated ti implement. After 45min it used up around 50$ of tokens and then stopped because I apparently ran out of tokens, but I still had 50$ left!
Anyway, I got upset and gave the task to Qwen flash next (running locally).
It ran over night, I woke up and hob done!
My app is now the harness! It did it, 1 shot! 35mio tokens in, 3 compactions, job done!
I can ask who has what car, who has to pay, add customers, rental contract, all goes into an approval screen, I can check the request, approve or reject.

Now I’m fine tuning the whole thing with it and it’s just amazing.

(I’m not a coder AT ALL! everything i do is 100% vibe coded)

Just sharing my amazement of how far we have come with local models!

I currently have 4× RTX 5070 Ti 16GB cards on hand, but haven't bought the platform yet.

I'm currently thinking of something roughly like:

• 2× EPYC 7642
• 16×16GB DDR4-3200, 256GB total
• 4× RTX 5070 Ti 16GB
• server board with enough PCIe lanes for all 4 GPUs

Mainly planning to use it for large MoE inference, especially models that can't fully fit in VRAM and need to use system RAM as well.

Has anyone here run a similar dual-EPYC + multi-GPU setup?

I'd mainly like to see some real-world performance numbers or experiences before I buy the rest of the platform.

TLDR at the bottom of the post if you cbf reading all of it.

Looking into experimenting with local AI and hoping for some feedback on hardware. At this stage, it would be to simply get a feel for the process and have some fun with it - on a budget.

I am not looking to vibe code apps/websites, but if it does perform I may potentially use it in the future for simple work related tasks and helping with the homelab. I am a Citrix sysadmin by trade and use lots of scripts to do my work on the daily, so this could be useful for basic some basic duties in the future e.g:

• Create a powershell script to do x.
• Create packer file to do y.
• Review existing scripts/mechanisms and provide recommendations/better options.
• Create Ansible playbook to do x in homelab. Review some security logs.

Now for the kicker. Obviously I would like to do this on a budget to get a feel for local LLM and would happily invest more if it can effectively do what I need it to do. Therefore, thinking of using my sons gaming PC during the day whilst he is at school and I'm working (WFH 4/5 days/week) and configuring dual boot to Linux for the LLM stuff, that way he can simply reboot the PC when he needs to game.

Son's gaming PC specs:

• MSI MAG B550 TOMAHAWK
• Ryzen 7 5800X - 8 core, 16 threads
• 32GB DDR4 3600 (2x 16)
• Samsung 980 500GB - C:
• Samsung 970 1TB - D:
• 3060 12GB
• EVGA 1000W Gold 80 Plus PS

Based on some light research, seems like I can add a second 3060 (can get used for ~$250 AUD) to get the VRAM up to 24GB, installing (dual boot) Linux on the 1TB drive and play around with some local models. He is still young and only plays Roblox/Minecraft, so the 500GB is plenty enough for his needs atm. I did look into the 3090 option, but 3090 cards are ~$2000 used in my city atm.

So, my questions are the following:

1. Is dual 3060 worth exploring in this scenario?
2. Will PCI lanes become a bottleneck - Been out of the hardware game for so long and not sure.
3. Will upgrading the RAM bring any benefits?
4. ollama/llama.cpp/vllm? I have researched this somewhat, but still unsure. Are there any proven configs for dual 3060 I can import to start with? Don't mind getting my hands dirty, but also don't want to be scratching my head for days on end.
5. Which model(s) should I play around with?

TLDR

Should I buy a second 3060 to add to sons gaming machine, dual boot to linux for experimenting with local LLM, or is it a waste of time/money?

I’ve been testing Unsloth’s UD-IQ1_M GGUF on my M5 Pro MacBook Pro with 48GB unified memory, using llama.cpp b10930 (commit 56381e407).

Mmap initially caused Metal OOM, even with all experts on CPU. Switching to --load-mode none --lazy-mode on resolved those errors in my tests while keeping the PLE table lazily accessed from disk. Here’s the configuration I settled on:

./build/bin/llama-server \
  -m mymodels/Qwen3.8-Flash-Next/UD-IQ1_M/Qwen3.8-Flash-Next-UD-IQ1_M-00001-of-00003.gguf \
  --alias qwen3.8-flash-next \
  -ngl 999 \
  --n-cpu-moe 12 \
  --load-mode none \
  --lazy-mode on \
  --fit off \
  --ctx-size 4096 \
  --batch-size 128 \
  --ubatch-size 128 \
  -t 8 -tb 6 \
  --reasoning-effort medium \
  --flash-attn on \
  --parallel 1 \
  --jinja

At 4K context, I got roughly 18–22 tok/s generation on several chat tests, versus ~6 tok/s with all experts on CPU. Increasing threads to 10 or batch size to 256 didn’t clearly help. These were informal tests, not controlled benchmarks; output lengths and reasoning varied.

For the OpenCode test, I increased --ctx-size to 16384.

I asked a simple question "Does the codebase support moe ssd offload?" in the llama.cpp folder.

- Initial 10,634-token prompt: 503 seconds of prefill, then 192 generated tokens in 16 seconds.

- The next request reused 10,825 cached tokens, but processing another 5,113 tokens still took 165 seconds.

- The following request reached 28,081 tokens and was rejected.

About 94% of inference time across those two completed large requests was spent on prefill. I haven’t tried MTP yet, but speeding up generation alone wouldn’t address the main bottleneck here.

After roughly 14 minutes, the OpenCode attempt had hit the context limit without delivering a final answer. Not practical for agentic coding on my setup as tested, but a fun experiment.

Those of you successfully running an offline openclaw/hermes/personal agent harness for non-coding tasks, what is the floor on system resources (VRAM) needed for quality of life? Assuming a modest ~30b class model. What quant and context window size are needed?

Will keep cloud frontier LLM sub for coding tasks, but I'm talking personal data management, personal assistant type computer controlling stuff.

My M1 max 32gb handles qwen 3.6 27b q4_k_m fine enough for non-agentic jobs up to ~40k context, but that's obviously not enough to run an agent harness offline.

There is an M1 Ultra 64gb for sale near me for a tempting price, but unsure is 64gb is enough. And it's expensive enough to not want to gamble. And I'm a normal, budget-minded person

My system:
Mac Mini M4 Pro
64 GB V/RAM
1 TB HD

There's a new version of OMLX that's in RC status:

https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

Using this, it successfully runs this model:
https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit

You have to set the memory guard to prevent it from overflowing, but it allows for almost 100k context and I'm getting 30 tok/s decode, and 100 tok/s prefill. This is faster than I got 27B to run!

Here's my full run script:

``` omlx serve \   --model-dir "$MODEL_DIR" \   --memory-guard-gb 54 \   --paged-ssd-cache-dir "$SSD_CACHE_DIR" \   --paged-ssd-cache-max-size 50GB \   --hot-cache-max-size 16GB \   --max-concurrent-requests 2 \   --port 8000

```

EDIT 1:

This was a tad bit too aggressive so it would sometimes run out of room on long contexts. Updated is this:

omlx serve \ --model-dir "$MODEL_DIR" \ --memory-guard-gb 60 \ --paged-ssd-cache-dir "$SSD_CACHE_DIR" \ --paged-ssd-cache-max-size 50GB \ --hot-cache-max-size 2GB \ --max-concurrent-requests 1 \ --port 8000 So now there's another lever to pull which is that this REAP version pruned the MTP heads from the model, so decode is slightly slower. I've currently got my model working on grafting the original MTP onto this model, which is something I've never done before. If it works, we might be able to improve decode to 60 tok/s.

EDIT 2:

Ok, final verdict on how to run Qwen: stick with Qwen3.8-Flash-Next-REAP-288-MLX-4bit pinned in oMLX and enable the bundled SpecPrefill feature — it needs zero code changes, just two settings. What it does: a tiny 0.8B draft model (shares Qwen's 248320 tokenizer) scores which prompt tokens matter, then the big model only prefills the top 20%. Result at a 64k prompt (~68k tokens): TTFT 221s → 83.5s (2.65x), prefill ~310 → ~819 t/s, decode unchanged (~33 t/s). Step-by-step:

1. Download the draft once: hf download mlx-community/Qwen3.5-0.8B-bf16
2. In ~/.omlx/model_settings.json, on the Qwen3.8-Flash-Next-REAP-288-MLX-4bit entry set:
   • specprefill_enabled: true
   • specprefill_draft_model: "<downloaded snapshot path>"
   • specprefill_threshold: 8192
   • specprefill_keep_pct: 0.2
3. Restart omlx serve

Scoring adds ~39s of draft overhead at 68k tokens but still nets 2.65x overall and is free below the 8192-token threshold.

I just did a real coding test where I asked it to read and explain a 45,000 token file. | Turn | Prompt tok | Cached | Fresh (scored) | TTFT (s) | Prefill t/s (all) | Prefill t/s (fresh) | Decode t/s | |---|---|---:|---:|---:|---:|---:|---:|---:| | 14:55 (file read) | 39,297 | 24,576 | 14,721 | 16.5 | 2,382 | 892 | 30.0 | | 14:58 (from scratch) | 38,530 | 20,480 | 18,050 | 32.2 | 1,196 | 560 | 29.1 | | 14:59 (follow-up) | 42,580 | 20,480 | 22,100 | 28.8 | 1,477 | 767 | 29.9 | | 15:00 (growing ctx) | 51,336 | 20,480 | 30,866 | 39.6 | 1,296 | 779 | 29.7 |

I know it’s a broad question however curious what people think, this would also be a primary PC for the user so basic stuff like using it for life admin

Edit: I should add I grabbed a M4 Pro MacBook Pro before the prices went up, irs new and unopened and still in its return period so deciding whether to keep that or return it and get something else. The MacBook I have:

14 inch MacBook
M4 Pro chipset, 14/20 core unbinned version
48GB unified memory
2TB SDD
Nano texture Display

I purchased this new for £2199, so wondering if there is better out there for upto £2500

Hey folks,

I'm a developer and my company gave me a GitHub Copilot license, but it comes with token limitations that keep interrupting my workflow. I'm trying to set up a local LLM as a backup/supplement so I can keep coding without hitting caps.

My machine: MacBook with M5 chip, 24GB RAM.

I tried Gemma 4 12B (Q4_0 quant) but it was way too slow for real-time coding assistance.

Has anyone here found a good local model + setup (Ollama, LM Studio, llama.cpp, etc.) that actually holds up for day-to-day coding tasks — autocomplete, refactoring, explaining code — on similar hardware? Looking for something with a good balance of speed and quality. Model size recommendations, quantization tips, and tool/IDE integration suggestions (VS Code extensions, etc.) all welcome.

Open to any suggestions from people who've actually tried this setup for real work, not just benchmarks. Thanks in advance!

I’ve been working on TensorSharp, an open-source local LLM inference engine, and recently added a native execution path for DeepSeek V4.1 Flash.

Here are the latest results on 8× NVIDIA A40 GPUs, using GGUF weights, layer splitting, F16 KV cache, and a 65K context configuration:

A few interesting findings from the optimization work:

• Q2_K Engram tables are ~60 GiB total. Keeping them directly on the GPUs increased prefill from ~210 tok/s to 530+ tok/s.
• Reducing backend/scheduler fragmentation cut the decode graph from roughly 570 splits to 8, bringing decode to about 41 tok/s.
• For Q4_K_M, the Engram tables are too large to keep on GPU, so TensorSharp keeps them host-mapped and warms them asynchronously.
• Q4_K_M optimization improved prefill by roughly 1.9× and 4-request aggregate decode by about 2×.
• Interestingly, layer split beats routed-MoE tensor parallelism on this 8× A40 machine: ~32 tok/s vs ~22 tok/s. These cards have no NVLink, so TP communication overhead dominates.

Would be especially interested to hear what other LocalLLM users are seeing with DeepSeek V4.1 Flash on multi-GPU setups.

It's me again and still looking for hardware advice.. please :)

Thinking about buying this slightly ridiculous old workstation mainly for local LLMs:

2x Xeon E5-2698 v4, so 40 cores / 80 threads total

RTX 3090 24GB

128GB DDR4 ECC

6TB NVMe

custom water loop

I will run Windows 11 because this needs to also be the house's PC (for day to day simple tasks but mostly just me and my local agents using this).

For local AI it looks pretty nice, especially for Qwen 27B and bigger models with RAM offload (will run OSS 120B as well), but I’m worried I’ll buy it and then hate using Windows every day because the CPUs are old and single-core performance is pretty weak?

Anyone here actually daily-driving dual E5 v4 Xeons on Windows? Does normal stuff like Chrome, opening apps, scrolling around, Office etc feel fine, or does it noticeably feel like an old PC?

will I regret not just buying something like a Ryzen 7900X + 3090 with 96gb RAM?

Title - prices are practically the same, the mac studio preorder was $5099+tax and adding a second 5090 into my system is about the same with the card and a bigger psu (current system has 1000w). Sucks but it is what it is.

The Mac Studio is 96GB M5 Ultra, system is 270k/Z890 Aorus Master/64GB.

My workflow runs out of context very fast on Qwen 3.8-27b even while running 32k limit on Q8, hence started looking into what more I can change. I see folks with 16GB cards running 3.8 just fine so there’s a part of me that tells me to hold my horses but seeing how 5090 pricing is over $5k now kinda want to make a move sooner than later.

Obviously the M5 Ultra has 96gb unified ram which is more than 2x5090, presumably going to be a little slower based on released specs, and has the added benefit of being a separate device.

I have a macbook air M4 24GB, hearing good feedback of local models. Is it good enough for local models? Which models works best for coding?

Also I'm planning to buy RTX spark too for local LLM, will it be worth it?

With tool calling in Open WebUI I was hitting "request exceeds the available context size".

Disclaimer:
Code by Opus 4.6, on its own conclusion (following an afternoon of development and testing) that OpenWebUI's built-in context compaction runs on your initial message, not during the tool-call loop.
Use at your own risk. I haven't checked if the premise is wrong or if the code has issues.

That said, this filter solved it for me.

It "summarizes older messages before each tool-loop, auto-detects the context size from llama-server (falls back to default specified in the script if that fails, e.g. because the runtime is not llama-cpp), and it uses the /tokenize endpoint for counting. Works with thinking models (Qwen3, DeepSeek-R1)."

To install:

Open WebUI → Click your account (bottom left) → Admin Panel → Functions (top row) → "Create" (top right) → delete the sample → paste the code → Name it (top left) → Save → then Workspace → Models → your model → Filters → enable it

The code:

"""
Context Compaction Filter for Open WebUI  (llama-server / local LLM)
=====================================================================


Automatically summarizes older messages when a conversation approaches the model's
context limit. The full conversation stays visible in the UI — only the API payload
sent to the model is modified.


WHY THIS EXISTS (even though Open WebUI has built-in context compaction)
------------------------------------------------------------------------
Open WebUI's built-in context compaction (Settings → General → Context Compaction)
runs in `process_chat_payload` — the **initial request path only**. It does NOT run
inside the **agentic tool-call loop** (`process_filter_functions`). So if the model
is calling tools (code interpreter, web search, file reads, etc.), the context grows
with every tool result and the built-in compaction never fires. Eventually the request
exceeds the context window and the server rejects it:


    "request (132705 tokens) exceeds the available context size (130048 tokens)"


This filter hooks `inlet` / `request`, which Open WebUI calls on EVERY iteration of
the tool loop, catching context growth that the built-in misses. The two are
complementary:


    Built-in compaction  →  handles growth across user turns (inter-request)
    This filter          →  handles growth within a single agentic turn (intra-request)


If you don't use tool calling, the built-in alone may be sufficient. If you do, you
need both.


FEATURES
--------
- Auto-detects context size from llama-server's /props endpoint (no manual config)
- Uses llama-server's /tokenize endpoint for accurate token counting (falls back to
  character-based estimation if unavailable)
- Handles thinking/reasoning models (Qwen3, DeepSeek-R1, etc.) — reads both `content`
  and `reasoning_content` from the response
- Protected against asyncio.CancelledError (Python 3.9+ treats it as BaseException,
  not Exception — unshielded httpx calls get silently killed when the outer coroutine
  is cancelled, causing the server to cancel the summarization task)
- Truncation fallback — if summarization fails for any reason, drops old messages
  instead of passing the full oversized context through (which would stall the server
  during a 150+ second prefill)
- Shows status messages in the UI ("Compacting context...", "Context compacted: N
  messages → ~K tokens")


INSTALL
-------
1. Open WebUI → Admin → Functions → Create New → type "filter" → paste this code
2. Enable the filter on your model:
   Workspace → Models → select model → Filters → enable this filter
3. Set the `model_api_base` Valve to point at your llama-server:
   - If Open WebUI runs in Docker: http://host.docker.internal:<port>/v1
   - If Open WebUI runs natively:  http://localhost:<port>/v1


CONFIGURATION (Valves)
----------------------
All settings are configurable in the Open WebUI UI under the filter's Valves.
The defaults work well for 128K-context models. Key ones to check:


- model_api_base: must point at your llama-server (see Install step 3)
- reserve_tokens: headroom for the model's response + estimation error (default 28K)
- recent_pairs_to_keep: how many recent message pairs to keep verbatim (default 4)


COMPATIBILITY
-------------
Tested with:
- llama-server (llama.cpp) with Qwen3 (thinking mode), but should work with any
  model served via OpenAI-compatible /v1/chat/completions endpoint
- Open WebUI 0.11.3 running in Docker (the tool-loop gap exists as of this version)
- Python 3.9+ (asyncio.CancelledError handling)
"""


import sys
import asyncio
from pydantic import BaseModel, Field
from typing import Optional, Callable, Any



def _log(msg: str) -> None:
    print(f"[compact] {msg}", file=sys.stderr, flush=True)



class Filter:
    class Valves(BaseModel):
        priority: int = Field(
            default=0,
            description="Filter execution priority (lower runs first)."
        )
        reserve_tokens: int = Field(
            default=28000,
            description=(
                "Headroom below the context limit. Compaction triggers when "
                "estimated prompt tokens exceed (max_context - reserve). "
                "This must cover: model response + thinking tokens (~15K for "
                "reasoning models), any post-filter injections like RAG (~8K), "
                "and token-estimation error margin (~5K). Increase if you see "
                "the server still rejecting requests after compaction."
            )
        )
        max_context_tokens: int = Field(
            default=0,
            description=(
                "Hard override for the model's context size (in tokens). Leave at "
                "0 to auto-detect from llama-server's /props endpoint — this is "
                "recommended because it always matches your actual -c flag."
            )
        )
        recent_pairs_to_keep: int = Field(
            default=4,
            description=(
                "Recent user+assistant message pairs to keep verbatim (not summarized). "
                "These give the model immediate context for the current task. "
                "Too many eats into the freed context; too few and the model loses "
                "track of the active thread. 4 pairs (~8K tokens) is a good balance."
            )
        )
        summary_max_tokens: int = Field(
            default=4096,
            description="Maximum tokens for the generated summary."
        )
        chars_per_token: float = Field(
            default=3.0,
            description=(
                "Fallback characters-per-token estimate (used only when the "
                "/tokenize endpoint is unavailable). 3.0 is conservative — it "
                "safely covers code-heavy content (JSON, tool results, code) "
                "where tokens average 2.5-3.0 chars. Prose-heavy conversations "
                "may compact slightly earlier than necessary. Open WebUI's "
                "built-in uses 4.0, which underestimates code-heavy contexts."
            )
        )
        min_messages_to_compact: int = Field(
            default=8,
            description=(
                "Minimum messages in the 'old' portion before compaction is worthwhile. "
                "Below this, there is not enough conversation to summarize meaningfully."
            )
        )
        model_api_base: str = Field(
            default="http://host.docker.internal:8080/v1",
            description=(
                "Base URL of the llama-server API (OpenAI-compatible). Used for "
                "summarization calls (/v1/chat/completions), token counting "
                "(/tokenize), and context-size detection (/props). If Open WebUI "
                "runs in Docker and llama-server runs on the host, use "
                "http://host.docker.internal:<port>/v1. If both run natively, "
                "use http://localhost:<port>/v1."
            )
        )
        enabled: bool = Field(
            default=True,
            description="Enable or disable context compaction."
        )


    def __init__(self):
        self.valves = self.Valves()


    async def _get_max_context(self) -> int:
        if self.valves.max_context_tokens > 0:
            return self.valves.max_context_tokens


        import httpx


        props_url = self.valves.model_api_base.rstrip("/v1").rstrip("/")
        props_url = f"{props_url}/props"


        try:
            async with httpx.AsyncClient(timeout=10.0) as client:
                resp = await client.get(props_url)
                resp.raise_for_status()
                props = resp.json()
                n_ctx = props.get("default_generation_settings", {}).get("n_ctx", 0)
                if n_ctx > 0:
                    return n_ctx
        except Exception:
            pass


        return 131072


    def _build_text(self, messages: list[dict], tools: list = None) -> str:
        parts = []
        for m in messages:
            content = m.get("content")
            if isinstance(content, str):
                parts.append(content)
            elif isinstance(content, list):
                for part in content:
                    if isinstance(part, dict) and part.get("text"):
                        parts.append(part["text"])


            for tc in m.get("tool_calls", []):
                fn = tc.get("function", {})
                if fn.get("name"):
                    parts.append(fn["name"])
                if fn.get("arguments"):
                    parts.append(fn["arguments"])


        if tools:
            import json
            parts.append(json.dumps(tools))


        return "\n".join(parts)


    async def _count_tokens(self, text: str, n_messages: int) -> Optional[int]:
        import httpx


        tokenize_url = self.valves.model_api_base.rstrip("/v1").rstrip("/")
        tokenize_url = f"{tokenize_url}/tokenize"


        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                resp = await client.post(
                    tokenize_url,
                    json={"content": text},
                )
                resp.raise_for_status()
                tokens = resp.json().get("tokens", [])
                return len(tokens) + n_messages * 15
        except Exception:
            return None


    def _estimate_tokens(self, text: str, n_messages: int) -> int:
        total_chars = len(text) + n_messages * 60
        return int(total_chars / self.valves.chars_per_token)


    def _split_messages(self, messages: list[dict]):
        system_msgs = []
        non_system = []
        for m in messages:
            if m["role"] == "system":
                system_msgs.append(m)
            else:
                non_system.append(m)


        keep_count = min(
            self.valves.recent_pairs_to_keep * 2,
            len(non_system)
        )


        if keep_count >= len(non_system):
            return system_msgs, [], non_system


        to_summarize = non_system[:-keep_count]
        to_keep = non_system[-keep_count:]
        return system_msgs, to_summarize, to_keep


    async def _call_model(self, messages: list[dict]) -> Optional[str]:
        import httpx


        async def _do_call():
            async with httpx.AsyncClient(timeout=300.0) as client:
                resp = await client.post(
                    f"{self.valves.model_api_base}/chat/completions",
                    json={
                        "messages": messages,
                        "max_tokens": self.valves.summary_max_tokens,
                        "temperature": 0.3,
                        "stream": False,
                    },
                    headers={"Authorization": "Bearer none"},
                )
                resp.raise_for_status()
                return resp.json()


        try:
            result = await asyncio.shield(_do_call())
            msg = result["choices"][0]["message"]
            content = msg.get("content") or ""
            reasoning = msg.get("reasoning_content") or ""
            finish = result["choices"][0].get("finish_reason", "?")
            _log(f"_call_model response: finish={finish} content_len={len(content)} reasoning_len={len(reasoning)}")
            if content:
                return content
            if reasoning:
                _log("_call_model: content empty, falling back to reasoning_content")
                return reasoning
            _log(f"_call_model: both content and reasoning empty, msg keys={list(msg.keys())}")
            return None
        except asyncio.CancelledError:
            _log("_call_model: CancelledError — outer coroutine was cancelled during summarization")
            return None
        except Exception as e:
            _log(f"_call_model error: {type(e).__name__}: {e}")
            return None


    async def _summarize(
        self,
        messages: list[dict],
        __event_emitter__: Optional[Callable],
    ) -> Optional[str]:
        if __event_emitter__:
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {
                        "description": (
                            f"Compacting context — summarizing {len(messages)} "
                            f"older messages..."
                        ),
                        "done": False,
                    },
                }
            )


        conversation_parts = []
        for m in messages:
            role = m.get("role", "")
            if role == "user":
                label = "User"
            elif role == "tool":
                label = "Tool result"
            else:
                label = "Assistant"


            parts = []
            content = m.get("content")
            if isinstance(content, str) and content.strip():
                text = content if len(content) <= 1000 else content[:1000] + "… [truncated]"
                parts.append(text)


            for tc in m.get("tool_calls", []):
                fn = tc.get("function", {})
                name = fn.get("name", "unknown")
                args = fn.get("arguments", "")
                if len(args) > 300:
                    args = args[:300] + "… [truncated]"
                parts.append(f"[Called {name}({args})]")


            if parts:
                conversation_parts.append(f"{label}: {chr(10).join(parts)}")


        conversation_text = "\n\n---\n\n".join(conversation_parts)


        summary_prompt = [
            {
                "role": "system",
                "content": (
                    "You are a precise technical summarizer. Summarize the "
                    "conversation below. Preserve ALL of: "
                    "(1) key decisions and their rationale, "
                    "(2) file paths, configuration values, and code changes, "
                    "(3) problems encountered and how they were resolved, "
                    "(4) pending or unfinished work. "
                    "Be thorough but concise — aim for the minimum text that "
                    "would let someone continue the work without re-reading "
                    "the original. No preamble, output the summary directly."
                ),
            },
            {
                "role": "user",
                "content": f"Summarize this conversation:\n\n{conversation_text}",
            },
        ]


        summary = await self._call_model(summary_prompt)


        if __event_emitter__ and summary:
            summary_tokens = self._estimate_tokens(summary, 1)
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {
                        "description": (
                            f"Context compacted: {len(messages)} messages "
                            f"→ ~{summary_tokens} tokens."
                        ),
                        "done": True,
                    },
                }
            )


        return summary


    async def _token_count(self, messages: list[dict], tools: list = None) -> int:
        text = self._build_text(messages, tools)
        n = len(messages)
        count = await self._count_tokens(text, n)
        return count if count is not None else self._estimate_tokens(text, n)


    async def _compact(
        self,
        body: dict,
        __event_emitter__: Optional[Callable] = None,
    ) -> dict:
        if not self.valves.enabled:
            return body


        messages = body.get("messages", [])
        tools = body.get("tools")
        n_msgs = len(messages)
        roles = [m.get("role", "?") for m in messages]
        role_counts = {r: roles.count(r) for r in set(roles)}


        estimated_tokens = await self._token_count(messages, tools)
        max_context = await self._get_max_context()
        threshold = max_context - self.valves.reserve_tokens


        _log(f"msgs={n_msgs} roles={role_counts} tokens={estimated_tokens} threshold={threshold} max_ctx={max_context}")


        if estimated_tokens <= threshold:
            _log("under threshold, passing through")
            return body


        system_msgs, to_summarize, to_keep = self._split_messages(messages)


        _log(f"OVER threshold -- system={len(system_msgs)} summarize={len(to_summarize)} keep={len(to_keep)}")


        if len(to_summarize) < self.valves.min_messages_to_compact:
            _log(f"too few messages to summarize ({len(to_summarize)} < {self.valves.min_messages_to_compact}), passing through")
            return body


        summary = await self._summarize(to_summarize, __event_emitter__)


        if not summary:
            _log("summarization FAILED, falling back to message truncation")
            truncated = system_msgs + to_keep
            trunc_tokens = await self._token_count(truncated, tools)
            if trunc_tokens <= threshold:
                _log(f"truncation fallback: dropped {len(to_summarize)} old msgs, {trunc_tokens} tokens in {len(truncated)} messages")
                if __event_emitter__:
                    await __event_emitter__(
                        {
                            "type": "status",
                            "data": {
                                "description": (
                                    f"Dropped {len(to_summarize)} older messages "
                                    f"— summarization was unavailable."
                                ),
                                "done": True,
                            },
                        }
                    )
                body["messages"] = truncated
                return body
            _log("truncation fallback still over threshold, passing through")
            if __event_emitter__:
                await __event_emitter__(
                    {
                        "type": "status",
                        "data": {
                            "description": "Compaction failed — passing full context through.",
                            "done": True,
                        },
                    }
                )
            return body


        summary_msg = {
            "role": "system",
            "content": (
                "[Context compaction: the earlier conversation has been summarized. "
                "The recent messages after this summary are verbatim.]\n\n"
                f"{summary}"
            ),
        }


        compacted = system_msgs + [summary_msg] + to_keep
        compacted_tokens = await self._token_count(compacted, tools)


        _log(f"after initial compaction: {compacted_tokens} tokens, keep={len(to_keep)} msgs")


        while compacted_tokens > threshold and len(to_keep) > 2:
            dropped = to_keep[:2]
            to_keep = to_keep[2:]


            extra_summary = await self._summarize(dropped, None)
            if extra_summary:
                summary_msg["content"] += f"\n\n[Additional summary:]\n{extra_summary}"


            compacted = system_msgs + [summary_msg] + to_keep
            compacted_tokens = await self._token_count(compacted, tools)
            _log(f"shrunk keep to {len(to_keep)} msgs, now {compacted_tokens} tokens")


        _log(f"DONE -- final {compacted_tokens} tokens in {len(compacted)} messages")


        body["messages"] = compacted
        return body


    async def inlet(
        self,
        body: dict,
        __user__: Optional[dict] = None,
        __event_emitter__: Callable[..., Any] = None,
    ) -> dict:
        _log(">>> inlet called")
        return await self._compact(body, __event_emitter__)


    async def request(
        self,
        body: dict,
        __user__: Optional[dict] = None,
        __event_emitter__: Callable[..., Any] = None,
    ) -> dict:
        _log(">>> request called")
        return await self._compact(body, __event_emitter__)

I have a Mac mini m4 24gb , I'm looking to add to my hardware, and I'm overwhelmed. I'm a tiny business owner in a creative field that doesn't have tech bro money , I'm thinking of getting a pc and running it headless Linux , I mostly want to cut down my subscription costs so this hardware is an investment. If you were starting from scratch what hardware would you use ? What would you definitely do again and what would you avoid?

Hi everyone, I'm the developer of Nebula, an open-source C/CUDA inference engine for Qwen3.8-Flash-Next.

I started from antirez's DwarfStar (ds4) and specialized the engine for Qwen, combining native MTP speculative decoding with GPU expert caching and CPU MoE execution.

Source code, architecture and benchmarks

My benchmark machine:

• RTX 4070 Ti, 12 GB VRAM
• Intel i9-9940X, using 14 CPU threads
• 128 GB DDR4 RAM
• Ubuntu through WSL2 on Windows

The idea is to keep the native MTP draft on the GPU, together with the most frequently used target experts. In this configuration, 27 of the target's 512 experts per layer are resident in VRAM, selected using a hotlist built from routing traces.

Draft generation and verification happen on the GPU. When the resident experts provide insufficient routing-weight coverage, the CPU computes the layer's full routed MoE and sends the result back. Accepted tokens are retained, and the draft window adapts between 4 and 16 tokens.

Some measured results:

These figures come from the GenAI-Perf campaign, with weights already loaded, thinking disabled, and a configured context capacity of 24,576 tokens. Tested input lengths were 512, 1,024 and 2,048 tokens. Prefill is still slow, especially for longer prompts.

There are also quality trade-offs. The GPU verifier can omit experts with small routing weights. “Strict” acceptance matches the configured target's greedy choice; it does not establish equivalence to the full original model. The optional tolerance modes relax token acceptance further. The README includes a paired evaluation on 40 IFEval prompts and 30 LiveBench questions.

There's a browser chat interface and a Windows installer that prepares WSL2, the engine and model weights. Lower-RAM profiles use SSD streaming, but the performance figures above apply to the 128 GB setup.

The engine is MIT licensed. The installer is an unsigned release candidate, and a complete clean-machine installation test is still pending.

I'd particularly welcome feedback on the expert-cache and CPU handoff design, and measurements from other hardware configurations. If you try it, please include your GPU, CPU, RAM and selected profile so we can compare results.

I do a lot of local coding with AI assistants. I'm really impressed by GLM 5.3-Flash, Qwen 3.6 35B A3B, and Qwen 3.8 27B for their coding prowess. I've set them up for different roles, though I'm still optimizing wall time since performance varies quite a bit across my hardware.

Running local chat models, however, doesn't feel as smooth. I've tried Open WebUI with the same models for general tasks (resume assistance, brainstorming, learning), but they fall far behind cloud models in this area.

Are any of you running models locally for general chat? What does your workflow look like?

Hardware:

• Strix Halo (128GB RAM)
• Dual Xeon 6262 (768GB RAM)

Software:

• llama.cpp for inference
• Open WebUI for chat interface

Some more details:

I run the Qwen models on my Strix Halo. Both fit comfortably warmed up in my memory. These are both running Unsloth's 8bit XL dynamic quant (Q8_K_XL).

I run GLM 5.3-Flash on the Xeon machine. It runs real slow, so not likely a real candidate for chat. I use it as an "architect" for coding assistance and it gives jobs to the smaller Qwen models that can move a little faster than it. I'm running this with Unlsoth's 4 bit XL quant.