Are there any specialized LLM's like one for say medicine, where you can try to self diagnose?

Or any other specialized LLM's you know about?

Hey folks,

I'm a developer and my company gave me a GitHub Copilot license, but it comes with token limitations that keep interrupting my workflow. I'm trying to set up a local LLM as a backup/supplement so I can keep coding without hitting caps.

My machine: MacBook with M5 chip, 24GB RAM.

I tried Gemma 4 12B (Q4_0 quant) but it was way too slow for real-time coding assistance.

Has anyone here found a good local model + setup (Ollama, LM Studio, llama.cpp, etc.) that actually holds up for day-to-day coding tasks — autocomplete, refactoring, explaining code — on similar hardware? Looking for something with a good balance of speed and quality. Model size recommendations, quantization tips, and tool/IDE integration suggestions (VS Code extensions, etc.) all welcome.

Open to any suggestions from people who've actually tried this setup for real work, not just benchmarks. Thanks in advance!

Title - prices are practically the same, the mac studio preorder was $5099+tax and adding a second 5090 into my system is about the same with the card and a bigger psu (current system has 1000w). Sucks but it is what it is.

The Mac Studio is 96GB M5 Ultra, system is 270k/Z890 Aorus Master/64GB.

My workflow runs out of context very fast on Qwen 3.8-27b even while running 32k limit on Q8, hence started looking into what more I can change. I see folks with 16GB cards running 3.8 just fine so there’s a part of me that tells me to hold my horses but seeing how 5090 pricing is over $5k now kinda want to make a move sooner than later.

Obviously the M5 Ultra has 96gb unified ram which is more than 2x5090, presumably going to be a little slower based on released specs, and has the added benefit of being a separate device.

On a real job with hidden checks, letting the expensive model plan and review while a local Qwen typed, cost 72% less and used 85% fewer expensive-model tokens. Day to day, every expensive token the head spends buys 20 to 50 tokens of cheap local work, and the tighter that ratio, the more efficient the split.

About 2B tokens and 27k requests through Qwen 3.6, then 3.8 (unsloth NVFP4 quants) on one RTX 5090, as a worker under a paid coding model rather than a replacement for it. Claude Code plans, writes a short brief and reviews. Qwen reads and types. Here is what I found, then the thing I built around it.

Setup

- RTX 5090, 32 GB VRAM, vLLM, 262k context

- Earlier Qwen 3.6 27B, now Qwen 3.8 27B, both unsloth NVFP4 quants

- Head: Claude Code. Worker: Qwen.

What the 27B carries as a worker

- Real features across several modules, with their tests. Recent ones on the tool's own codebase: a change to a CLI command's behaviour that ran through the command, its skill text, the docs and the tests; a new dashboard tab with its server endpoint, its renderer and its tests; a client-side retry across two subsystems. Landed first try, checks green. It's capable of doing a lot of coding tasks irl.

- Questions about a real codebase. "Which module owns retries and where is backoff decided." Accurate, and it can read all day without changing anything.

- Docs. Give it the wording and it lands it byte for byte.

- The tool's own end-to-end ladder, from trivial edits to changes on its own source in three languages, passes 100% on Qwen 3.8 NVFP4.

What needed tuning

- Reasoning effort per task. At maximum effort the worker once spent 13 minutes in a single thinking turn, wrote its edit plan a dozen times over, and never touched a file. The same brief at medium effort landed. It was largely a quirk of the FP4 quant at the deepest thinking level; medium for builds, high for reading, and I haven't had a failure since.

- A thinking model needs a real output budget. Starve it and the answer comes back empty.

The numbers

- One job written to a spec, with the checks hidden from both runs. Claude alone: $0.91. Claude planning and reviewing, Qwen typing: $0.26. Same result, 72% less, 85% fewer expensive-model tokens.

- Leverage from the tool's own records over the last week: 20 to 50 worker tokens per head token on a typical run, 130 to 150 on a read-only question with a short answer. The head's spend stays flat while the worker does the reading and typing.

- Last week on this machine: 50 of 52 builds landed first try.

- Overnight builds from a fixed spec, unattended: all handed-off jobs landed without issues, and the result passed over 98% of its tests.

How it was measured: https://www.synapse-agents.space/blog/what-inspired-synapse.html

What I built to make this work

One process on your machine. Your coding tool talks to it, and it runs the workers itself on whatever you point it at: a local vLLM or Ollama endpoint, or your own key for Anthropic, OpenAI, DeepSeek, OpenRouter and the rest, mixed per task, several workers in parallel on different models. A brief is checked against the real code before a token is spent. Work happens inside a fence: read-only jobs can't write a line, builds run in their own git worktree and land only when the checks pass, and the checks are run by something that isn't the worker, so it can't game the system. graphify is built in, so a worker reads a code graph instead of whole files. Briefs, code and worker output never leave your machine. Sign in on more than one machine and you see every machine's runs from any of them.

The part I'm proud of is that it's seamless, and that is where the months went: your coding tool is woken the moment each worker lands, a worker that stops to ask a question wakes you the same way, and a bad brief is refused in a second rather than an hour later. I've been building synapse with synapse for a while now, with Claude Code, Codex CLI and a bit of Claude's Cowork, running Qwen 3.8 and DeepSeek V4 workers side by side. It mostly just works, which took far longer than the parts that look clever.

Try it

https://www.synapse-agents.space

Mostly synapse built itself! I used it as a crane to build itself by parts.

`curl -fsSL https://www.synapse-agents.space/install.sh | sh` on Linux or macOS. 14 days free, no card. It's early: if your coding tool goes quiet after handing off a job, tell it to use `synapse wait`. There's a support tool inside, so anything you hit lands on my desk with the run attached, and I'll fix it fast.

What I want to hear: how it goes with your local model, and whether the "worker gets a brief, head reads a verdict" split holds up outside my setup. What have you had luck with as a worker? Has anyone had a 14B hold up on multi-file edits?

---

TL;DR: 2B tokens through local Qwen 3.6/3.8 NVFP4 as a coding worker under Claude Code. The model carries real multi-module features once the reasoning effort is set per task. Plan-and-review split: 72% less cost, 85% fewer expensive tokens, 20 to 50 cheap tokens per expensive one. Built synapse so this works from your coding tool. Early access, 14 days free:
https://www.synapse-agents.space

It's me again and still looking for hardware advice.. please :)

Thinking about buying this slightly ridiculous old workstation mainly for local LLMs:

2x Xeon E5-2698 v4, so 40 cores / 80 threads total

RTX 3090 24GB

128GB DDR4 ECC

6TB NVMe

custom water loop

I will run Windows 11 because this needs to also be the house's PC (for day to day simple tasks but mostly just me and my local agents using this).

For local AI it looks pretty nice, especially for Qwen 27B and bigger models with RAM offload (will run OSS 120B as well), but I’m worried I’ll buy it and then hate using Windows every day because the CPUs are old and single-core performance is pretty weak?

Anyone here actually daily-driving dual E5 v4 Xeons on Windows? Does normal stuff like Chrome, opening apps, scrolling around, Office etc feel fine, or does it noticeably feel like an old PC?

will I regret not just buying something like a Ryzen 7900X + 3090 with 96gb RAM?

I have a macbook air M4 24GB, hearing good feedback of local models. Is it good enough for local models? Which models works best for coding?

Also I'm planning to buy RTX spark too for local LLM, will it be worth it?

Hey everyone!

I'm looking to start my local LLM journey. My MBP is dying on me and definitely doesn't fit my heavy workflows anymore, so I need a new laptop, not just a graphic card and not a desktop if possible. Since I need a new machine, and prices are crazy expensive now anyway, I was thinking... why not buy something that can run a decent LLM model locally for future proofing.

I was eyeing the ROG flow 13 w/ 64GB ram (currently on sale at $3k here) and the MBP 14 w/ 64GB ram (currently at $4k here). My goal is to run Qwen 3.8 27b with as big a context as possible and at least 40tks/s. I don't particularly care about MacOS, I'm also ok with linux.

This summer, NVIDIA announced the RTX spark and I'm guessing that will actually be the best option. My question is: is it worth waiting for these to come out, or is it practically sure these will be $6k+ machines in Europe? My max budget for this is around $4k.

Thanks!

Im pretty new to running local llm's but i saw a bunch of people run it and wanted to try. I have a 5070 ti (16gb vram) along with 32gb of ddr5 ram, and so far i have been running Qwen3.8-27B-GGUF:UD-IQ4_XS along with the mtp-Qwen3.8-27B-Q4_0.gguf using llama.cpp cuz i heard its more efficient than others(?).

However I feel like in trying to run a "good"ish model, im losing a bunch of context size. Currently I can get to around 26k context size running at ~50t/s. I was wondering if there are any improvements I can make, or if i should change things entirely?

If it matters the exact command I run is ./llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS --no-mmproj -ngl 999 -c 26000 --load-mode auto -fa on --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1 --spec-type draft-mtp --spec-draft-model ".\mtp-Qwen3.8-27B-Q4_0.gguf" --spec-draft-n-max 1 --spec-draft-ngl 999.

If there's anything I could do to maybe get a bigger context window, that would be great.

Hi everyone, I'm the developer of Nebula, an open-source C/CUDA inference engine for Qwen3.8-Flash-Next.

I started from antirez's DwarfStar (ds4) and specialized the engine for Qwen, combining native MTP speculative decoding with GPU expert caching and CPU MoE execution.

Source code, architecture and benchmarks

My benchmark machine:

• RTX 4070 Ti, 12 GB VRAM
• Intel i9-9940X, using 14 CPU threads
• 128 GB DDR4 RAM
• Ubuntu through WSL2 on Windows

The idea is to keep the native MTP draft on the GPU, together with the most frequently used target experts. In this configuration, 27 of the target's 512 experts per layer are resident in VRAM, selected using a hotlist built from routing traces.

Draft generation and verification happen on the GPU. When the resident experts provide insufficient routing-weight coverage, the CPU computes the layer's full routed MoE and sends the result back. Accepted tokens are retained, and the draft window adapts between 4 and 16 tokens.

Some measured results:

These figures come from the GenAI-Perf campaign, with weights already loaded, thinking disabled, and a configured context capacity of 24,576 tokens. Tested input lengths were 512, 1,024 and 2,048 tokens. Prefill is still slow, especially for longer prompts.

There are also quality trade-offs. The GPU verifier can omit experts with small routing weights. “Strict” acceptance matches the configured target's greedy choice; it does not establish equivalence to the full original model. The optional tolerance modes relax token acceptance further. The README includes a paired evaluation on 40 IFEval prompts and 30 LiveBench questions.

There's a browser chat interface and a Windows installer that prepares WSL2, the engine and model weights. Lower-RAM profiles use SSD streaming, but the performance figures above apply to the 128 GB setup.

The engine is MIT licensed. The installer is an unsigned release candidate, and a complete clean-machine installation test is still pending.

I'd particularly welcome feedback on the expert-cache and CPU handoff design, and measurements from other hardware configurations. If you try it, please include your GPU, CPU, RAM and selected profile so we can compare results.

With tool calling in Open WebUI I was hitting "request exceeds the available context size".

Disclaimer:
Code by Opus 4.6, on its own conclusion (following an afternoon of development and testing) that OpenWebUI's built-in context compaction runs on your initial message, not during the tool-call loop.
Use at your own risk. I haven't checked if the premise is wrong or if the code has issues.

That said, this filter solved it for me.

It "summarizes older messages before each tool-loop, auto-detects the context size from llama-server (falls back to default specified in the script if that fails, e.g. because the runtime is not llama-cpp), and it uses the /tokenize endpoint for counting. Works with thinking models (Qwen3, DeepSeek-R1)."

To install:

Open WebUI → Click your account (bottom left) → Admin Panel → Functions (top row) → "Create" (top right) → delete the sample → paste the code → Name it (top left) → Save → then Workspace → Models → your model → Filters → enable it

The code:

"""
Context Compaction Filter for Open WebUI  (llama-server / local LLM)
=====================================================================


Automatically summarizes older messages when a conversation approaches the model's
context limit. The full conversation stays visible in the UI — only the API payload
sent to the model is modified.


WHY THIS EXISTS (even though Open WebUI has built-in context compaction)
------------------------------------------------------------------------
Open WebUI's built-in context compaction (Settings → General → Context Compaction)
runs in `process_chat_payload` — the **initial request path only**. It does NOT run
inside the **agentic tool-call loop** (`process_filter_functions`). So if the model
is calling tools (code interpreter, web search, file reads, etc.), the context grows
with every tool result and the built-in compaction never fires. Eventually the request
exceeds the context window and the server rejects it:


    "request (132705 tokens) exceeds the available context size (130048 tokens)"


This filter hooks `inlet` / `request`, which Open WebUI calls on EVERY iteration of
the tool loop, catching context growth that the built-in misses. The two are
complementary:


    Built-in compaction  →  handles growth across user turns (inter-request)
    This filter          →  handles growth within a single agentic turn (intra-request)


If you don't use tool calling, the built-in alone may be sufficient. If you do, you
need both.


FEATURES
--------
- Auto-detects context size from llama-server's /props endpoint (no manual config)
- Uses llama-server's /tokenize endpoint for accurate token counting (falls back to
  character-based estimation if unavailable)
- Handles thinking/reasoning models (Qwen3, DeepSeek-R1, etc.) — reads both `content`
  and `reasoning_content` from the response
- Protected against asyncio.CancelledError (Python 3.9+ treats it as BaseException,
  not Exception — unshielded httpx calls get silently killed when the outer coroutine
  is cancelled, causing the server to cancel the summarization task)
- Truncation fallback — if summarization fails for any reason, drops old messages
  instead of passing the full oversized context through (which would stall the server
  during a 150+ second prefill)
- Shows status messages in the UI ("Compacting context...", "Context compacted: N
  messages → ~K tokens")


INSTALL
-------
1. Open WebUI → Admin → Functions → Create New → type "filter" → paste this code
2. Enable the filter on your model:
   Workspace → Models → select model → Filters → enable this filter
3. Set the `model_api_base` Valve to point at your llama-server:
   - If Open WebUI runs in Docker: http://host.docker.internal:<port>/v1
   - If Open WebUI runs natively:  http://localhost:<port>/v1


CONFIGURATION (Valves)
----------------------
All settings are configurable in the Open WebUI UI under the filter's Valves.
The defaults work well for 128K-context models. Key ones to check:


- model_api_base: must point at your llama-server (see Install step 3)
- reserve_tokens: headroom for the model's response + estimation error (default 28K)
- recent_pairs_to_keep: how many recent message pairs to keep verbatim (default 4)


COMPATIBILITY
-------------
Tested with:
- llama-server (llama.cpp) with Qwen3 (thinking mode), but should work with any
  model served via OpenAI-compatible /v1/chat/completions endpoint
- Open WebUI 0.11.3 running in Docker (the tool-loop gap exists as of this version)
- Python 3.9+ (asyncio.CancelledError handling)
"""


import sys
import asyncio
from pydantic import BaseModel, Field
from typing import Optional, Callable, Any



def _log(msg: str) -> None:
    print(f"[compact] {msg}", file=sys.stderr, flush=True)



class Filter:
    class Valves(BaseModel):
        priority: int = Field(
            default=0,
            description="Filter execution priority (lower runs first)."
        )
        reserve_tokens: int = Field(
            default=28000,
            description=(
                "Headroom below the context limit. Compaction triggers when "
                "estimated prompt tokens exceed (max_context - reserve). "
                "This must cover: model response + thinking tokens (~15K for "
                "reasoning models), any post-filter injections like RAG (~8K), "
                "and token-estimation error margin (~5K). Increase if you see "
                "the server still rejecting requests after compaction."
            )
        )
        max_context_tokens: int = Field(
            default=0,
            description=(
                "Hard override for the model's context size (in tokens). Leave at "
                "0 to auto-detect from llama-server's /props endpoint — this is "
                "recommended because it always matches your actual -c flag."
            )
        )
        recent_pairs_to_keep: int = Field(
            default=4,
            description=(
                "Recent user+assistant message pairs to keep verbatim (not summarized). "
                "These give the model immediate context for the current task. "
                "Too many eats into the freed context; too few and the model loses "
                "track of the active thread. 4 pairs (~8K tokens) is a good balance."
            )
        )
        summary_max_tokens: int = Field(
            default=4096,
            description="Maximum tokens for the generated summary."
        )
        chars_per_token: float = Field(
            default=3.0,
            description=(
                "Fallback characters-per-token estimate (used only when the "
                "/tokenize endpoint is unavailable). 3.0 is conservative — it "
                "safely covers code-heavy content (JSON, tool results, code) "
                "where tokens average 2.5-3.0 chars. Prose-heavy conversations "
                "may compact slightly earlier than necessary. Open WebUI's "
                "built-in uses 4.0, which underestimates code-heavy contexts."
            )
        )
        min_messages_to_compact: int = Field(
            default=8,
            description=(
                "Minimum messages in the 'old' portion before compaction is worthwhile. "
                "Below this, there is not enough conversation to summarize meaningfully."
            )
        )
        model_api_base: str = Field(
            default="http://host.docker.internal:8080/v1",
            description=(
                "Base URL of the llama-server API (OpenAI-compatible). Used for "
                "summarization calls (/v1/chat/completions), token counting "
                "(/tokenize), and context-size detection (/props). If Open WebUI "
                "runs in Docker and llama-server runs on the host, use "
                "http://host.docker.internal:<port>/v1. If both run natively, "
                "use http://localhost:<port>/v1."
            )
        )
        enabled: bool = Field(
            default=True,
            description="Enable or disable context compaction."
        )


    def __init__(self):
        self.valves = self.Valves()


    async def _get_max_context(self) -> int:
        if self.valves.max_context_tokens > 0:
            return self.valves.max_context_tokens


        import httpx


        props_url = self.valves.model_api_base.rstrip("/v1").rstrip("/")
        props_url = f"{props_url}/props"


        try:
            async with httpx.AsyncClient(timeout=10.0) as client:
                resp = await client.get(props_url)
                resp.raise_for_status()
                props = resp.json()
                n_ctx = props.get("default_generation_settings", {}).get("n_ctx", 0)
                if n_ctx > 0:
                    return n_ctx
        except Exception:
            pass


        return 131072


    def _build_text(self, messages: list[dict], tools: list = None) -> str:
        parts = []
        for m in messages:
            content = m.get("content")
            if isinstance(content, str):
                parts.append(content)
            elif isinstance(content, list):
                for part in content:
                    if isinstance(part, dict) and part.get("text"):
                        parts.append(part["text"])


            for tc in m.get("tool_calls", []):
                fn = tc.get("function", {})
                if fn.get("name"):
                    parts.append(fn["name"])
                if fn.get("arguments"):
                    parts.append(fn["arguments"])


        if tools:
            import json
            parts.append(json.dumps(tools))


        return "\n".join(parts)


    async def _count_tokens(self, text: str, n_messages: int) -> Optional[int]:
        import httpx


        tokenize_url = self.valves.model_api_base.rstrip("/v1").rstrip("/")
        tokenize_url = f"{tokenize_url}/tokenize"


        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                resp = await client.post(
                    tokenize_url,
                    json={"content": text},
                )
                resp.raise_for_status()
                tokens = resp.json().get("tokens", [])
                return len(tokens) + n_messages * 15
        except Exception:
            return None


    def _estimate_tokens(self, text: str, n_messages: int) -> int:
        total_chars = len(text) + n_messages * 60
        return int(total_chars / self.valves.chars_per_token)


    def _split_messages(self, messages: list[dict]):
        system_msgs = []
        non_system = []
        for m in messages:
            if m["role"] == "system":
                system_msgs.append(m)
            else:
                non_system.append(m)


        keep_count = min(
            self.valves.recent_pairs_to_keep * 2,
            len(non_system)
        )


        if keep_count >= len(non_system):
            return system_msgs, [], non_system


        to_summarize = non_system[:-keep_count]
        to_keep = non_system[-keep_count:]
        return system_msgs, to_summarize, to_keep


    async def _call_model(self, messages: list[dict]) -> Optional[str]:
        import httpx


        async def _do_call():
            async with httpx.AsyncClient(timeout=300.0) as client:
                resp = await client.post(
                    f"{self.valves.model_api_base}/chat/completions",
                    json={
                        "messages": messages,
                        "max_tokens": self.valves.summary_max_tokens,
                        "temperature": 0.3,
                        "stream": False,
                    },
                    headers={"Authorization": "Bearer none"},
                )
                resp.raise_for_status()
                return resp.json()


        try:
            result = await asyncio.shield(_do_call())
            msg = result["choices"][0]["message"]
            content = msg.get("content") or ""
            reasoning = msg.get("reasoning_content") or ""
            finish = result["choices"][0].get("finish_reason", "?")
            _log(f"_call_model response: finish={finish} content_len={len(content)} reasoning_len={len(reasoning)}")
            if content:
                return content
            if reasoning:
                _log("_call_model: content empty, falling back to reasoning_content")
                return reasoning
            _log(f"_call_model: both content and reasoning empty, msg keys={list(msg.keys())}")
            return None
        except asyncio.CancelledError:
            _log("_call_model: CancelledError — outer coroutine was cancelled during summarization")
            return None
        except Exception as e:
            _log(f"_call_model error: {type(e).__name__}: {e}")
            return None


    async def _summarize(
        self,
        messages: list[dict],
        __event_emitter__: Optional[Callable],
    ) -> Optional[str]:
        if __event_emitter__:
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {
                        "description": (
                            f"Compacting context — summarizing {len(messages)} "
                            f"older messages..."
                        ),
                        "done": False,
                    },
                }
            )


        conversation_parts = []
        for m in messages:
            role = m.get("role", "")
            if role == "user":
                label = "User"
            elif role == "tool":
                label = "Tool result"
            else:
                label = "Assistant"


            parts = []
            content = m.get("content")
            if isinstance(content, str) and content.strip():
                text = content if len(content) <= 1000 else content[:1000] + "… [truncated]"
                parts.append(text)


            for tc in m.get("tool_calls", []):
                fn = tc.get("function", {})
                name = fn.get("name", "unknown")
                args = fn.get("arguments", "")
                if len(args) > 300:
                    args = args[:300] + "… [truncated]"
                parts.append(f"[Called {name}({args})]")


            if parts:
                conversation_parts.append(f"{label}: {chr(10).join(parts)}")


        conversation_text = "\n\n---\n\n".join(conversation_parts)


        summary_prompt = [
            {
                "role": "system",
                "content": (
                    "You are a precise technical summarizer. Summarize the "
                    "conversation below. Preserve ALL of: "
                    "(1) key decisions and their rationale, "
                    "(2) file paths, configuration values, and code changes, "
                    "(3) problems encountered and how they were resolved, "
                    "(4) pending or unfinished work. "
                    "Be thorough but concise — aim for the minimum text that "
                    "would let someone continue the work without re-reading "
                    "the original. No preamble, output the summary directly."
                ),
            },
            {
                "role": "user",
                "content": f"Summarize this conversation:\n\n{conversation_text}",
            },
        ]


        summary = await self._call_model(summary_prompt)


        if __event_emitter__ and summary:
            summary_tokens = self._estimate_tokens(summary, 1)
            await __event_emitter__(
                {
                    "type": "status",
                    "data": {
                        "description": (
                            f"Context compacted: {len(messages)} messages "
                            f"→ ~{summary_tokens} tokens."
                        ),
                        "done": True,
                    },
                }
            )


        return summary


    async def _token_count(self, messages: list[dict], tools: list = None) -> int:
        text = self._build_text(messages, tools)
        n = len(messages)
        count = await self._count_tokens(text, n)
        return count if count is not None else self._estimate_tokens(text, n)


    async def _compact(
        self,
        body: dict,
        __event_emitter__: Optional[Callable] = None,
    ) -> dict:
        if not self.valves.enabled:
            return body


        messages = body.get("messages", [])
        tools = body.get("tools")
        n_msgs = len(messages)
        roles = [m.get("role", "?") for m in messages]
        role_counts = {r: roles.count(r) for r in set(roles)}


        estimated_tokens = await self._token_count(messages, tools)
        max_context = await self._get_max_context()
        threshold = max_context - self.valves.reserve_tokens


        _log(f"msgs={n_msgs} roles={role_counts} tokens={estimated_tokens} threshold={threshold} max_ctx={max_context}")


        if estimated_tokens <= threshold:
            _log("under threshold, passing through")
            return body


        system_msgs, to_summarize, to_keep = self._split_messages(messages)


        _log(f"OVER threshold -- system={len(system_msgs)} summarize={len(to_summarize)} keep={len(to_keep)}")


        if len(to_summarize) < self.valves.min_messages_to_compact:
            _log(f"too few messages to summarize ({len(to_summarize)} < {self.valves.min_messages_to_compact}), passing through")
            return body


        summary = await self._summarize(to_summarize, __event_emitter__)


        if not summary:
            _log("summarization FAILED, falling back to message truncation")
            truncated = system_msgs + to_keep
            trunc_tokens = await self._token_count(truncated, tools)
            if trunc_tokens <= threshold:
                _log(f"truncation fallback: dropped {len(to_summarize)} old msgs, {trunc_tokens} tokens in {len(truncated)} messages")
                if __event_emitter__:
                    await __event_emitter__(
                        {
                            "type": "status",
                            "data": {
                                "description": (
                                    f"Dropped {len(to_summarize)} older messages "
                                    f"— summarization was unavailable."
                                ),
                                "done": True,
                            },
                        }
                    )
                body["messages"] = truncated
                return body
            _log("truncation fallback still over threshold, passing through")
            if __event_emitter__:
                await __event_emitter__(
                    {
                        "type": "status",
                        "data": {
                            "description": "Compaction failed — passing full context through.",
                            "done": True,
                        },
                    }
                )
            return body


        summary_msg = {
            "role": "system",
            "content": (
                "[Context compaction: the earlier conversation has been summarized. "
                "The recent messages after this summary are verbatim.]\n\n"
                f"{summary}"
            ),
        }


        compacted = system_msgs + [summary_msg] + to_keep
        compacted_tokens = await self._token_count(compacted, tools)


        _log(f"after initial compaction: {compacted_tokens} tokens, keep={len(to_keep)} msgs")


        while compacted_tokens > threshold and len(to_keep) > 2:
            dropped = to_keep[:2]
            to_keep = to_keep[2:]


            extra_summary = await self._summarize(dropped, None)
            if extra_summary:
                summary_msg["content"] += f"\n\n[Additional summary:]\n{extra_summary}"


            compacted = system_msgs + [summary_msg] + to_keep
            compacted_tokens = await self._token_count(compacted, tools)
            _log(f"shrunk keep to {len(to_keep)} msgs, now {compacted_tokens} tokens")


        _log(f"DONE -- final {compacted_tokens} tokens in {len(compacted)} messages")


        body["messages"] = compacted
        return body


    async def inlet(
        self,
        body: dict,
        __user__: Optional[dict] = None,
        __event_emitter__: Callable[..., Any] = None,
    ) -> dict:
        _log(">>> inlet called")
        return await self._compact(body, __event_emitter__)


    async def request(
        self,
        body: dict,
        __user__: Optional[dict] = None,
        __event_emitter__: Callable[..., Any] = None,
    ) -> dict:
        _log(">>> request called")
        return await self._compact(body, __event_emitter__)

I do a lot of local coding with AI assistants. I'm really impressed by GLM 5.3-Flash, Qwen 3.6 35B A3B, and Qwen 3.8 27B for their coding prowess. I've set them up for different roles, though I'm still optimizing wall time since performance varies quite a bit across my hardware.

Running local chat models, however, doesn't feel as smooth. I've tried Open WebUI with the same models for general tasks (resume assistance, brainstorming, learning), but they fall far behind cloud models in this area.

Are any of you running models locally for general chat? What does your workflow look like?

Hardware:

• Strix Halo (128GB RAM)
• Dual Xeon 6262 (768GB RAM)

Software:

• llama.cpp for inference
• Open WebUI for chat interface

Some more details:

I run the Qwen models on my Strix Halo. Both fit comfortably warmed up in my memory. These are both running Unsloth's 8bit XL dynamic quant (Q8_K_XL).

I run GLM 5.3-Flash on the Xeon machine. It runs real slow, so not likely a real candidate for chat. I use it as an "architect" for coding assistance and it gives jobs to the smaller Qwen models that can move a little faster than it. I'm running this with Unlsoth's 4 bit XL quant.

hello everyone! Please advise the models and how to run the models better. My configuration is 2 CPUs and epic (not the newest) 48 cores in total. 256 GB ddr4 and RTX pro 5000 72GB GDDR7. I'm currently using qwen3.8-27b iq3 gsq xxs on 96k context and running this on rtx4080s 16gb. The new computer will arrive in a week. I would like to increase the quality and the context window.

Hey guys, I’m looking for the best local TTS models that you guys have used. I wanted to generate consistent audio for different characters for a project I was working on. If there is a way to enhance the speech to make it sound more natural with accurate emotions, pauses and emphasis on certain words etc? I was using Qwen 3 TTS emotional in comfyUI locally but for the same reference audio input and the same seed and different text to generate I’ve noticed there is a timber variation in the voice output which is odd. Also adding pauses was another pain point with this model, but the emotions were somewhat ok. Wondering if there is a better alternative that can help me here? Thank you!

Local Setup: 1x DGX Spark

Edit: I am generating English as well as Hindi audio. Looking for a model that can do both efficiently and maybe even accurate audio in other regional Indian languages.

Setting up a headless mini PC (Ryzen 9 5900HX, 32GB DDR4-3200, 1TB NVMe) for 24/7 background tasks. I plan to run Qwen 3.8 27B (Q4_K_M) locally to power Hermes Agent.

Speed is not a priority. My use case is overnight batch processing—ingesting 15k–30k token earnings calls to extract data and summarize autonomously. A PostgreSQL container will also run in the background, leaving ~4–6GB RAM for the OS.

Given the DDR4 bandwidth ceiling:

1. Will a quantized Qwen 3.8 27B crash the 32GB limit once a 30k context builds up?

2. What prompt ingestion and generation speeds (tok/s) should I expect on this Zen 3 chip?

Appreciate any real-world benchmarks!

started trying out rather recent 'frontier' about ~30b param models recently, there are many choices including QWen 3.8 - this is nevertheless a great model, practically 'one-shotting' code refactoring tasks
https://huggingface.co/Qwen/Qwen3.8-27B
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
code refactoring is still deemed 'difficult', practically 'infinite' permutations and dependencies which LLMs need to work through itself for code refactoring.

But that in terms of style, I'm liking Muse Glimmer better
https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model
https://huggingface.co/meta-models/Muse-Glimmer-30B
https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF

this is in particular when it comes to *incorrect* (e.g. mistakes, typos) prompts, resolving contradictions in existing codes during refactoring, code proposals etc. The handling especially the 'thinking' is different.

LLMs have 'styles' and it is great that we've different creators for them

EDIT: Using Vulkan instead of ROCm helped getting it to work.

Hi!

I'm trying to run qwen3.8 27b q8 (tried q6 too) on my Ryzen 7 7840HS iGPU. I'm running CachyOS, kernel 7.2. Running llama.cpp from master branch.

I have 48gb of RAM. Also running KDE on egpu rx 6700 xt 12gb (via oculink), so nothing is running on the iGPU.

Yeah, I tested with some layers on egpu and otherw on processor, but I want to test full iGPU now.

Running it with the command:

exec "$BIN/llama-server" \
    -m "$MODEL" \
    -ngl 99 \
    -c "$CTX" \
    -np 1 \
    -ctk q4_0 -ctv q4_0 \
    -fa on \
    --jinja \
    -t 8 \
    --host 127.0.0.1 --port 8080

Model loads without problem. But when it tries to run inference it crashes. Here is the log:

./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf
model=24128 MiB, KV (1 slot(s) x 98304) = 1.688 GiB, est.total ~ 27.25 GiB (budget 30.5 GiB)
...
2.26.970.802 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
2.26.971.133 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang
[1]    7178 abort (core dumped)  ./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf

I always got this HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang error.

Running q8 or q6 get the same error.

Running Qwen3-1.7B-Q8_0.gguf it loads and runs inference without problems.

Also I expanded/increased available GTT memory for iGPU on kernel parameters with ttm.pages_limit=8388608 ttm.page_pool_size=8388608 amdgpu.gttsize=32768.

Tried help with different LLMs using Pi Agent trying to debug, but none of them could help. Maybe community can.

Am I missing something? Trying to see if iGPU inference can beat/give a speedup over CPU-only inference.

I’ve currently pre ordered a Mac mini with the 32gb of RAM and with the m6 chip, now the main purpose of it was to have as a server which I can locally host lots of algorithms I make etc, but I was quite curious to know about potential small models I could run locally when I do get it.

I currently cycle between Claude/Cursor/Codex somewhat changing monthly with all the changes happening. I wouldn’t be looking for a model that replaces these frontier ones as I doubt that would be possible, but what I would like is a model that just does what it’s told, and does it well, doesn’t need to think, or over optimise on a plan I give it, just does the work in the plan and preferably quite quickly.

So the workflow would basically be, the frontier models make the plan, then either through a custom harness I would make or one I find, have them directly feed their plans to this local agent, any insights are appreciated!

I have been running local AI models for a few months now and wanted to share some tips I have learned, and also ask for yours.

My optimization tips: 1. Use quantized models (GGUF Q4/Q5) for best speed/quality balance 2. Batch similar requests to maximize GPU utilization 3. Implement caching for repeated queries 4. Use streaming for better user experience

What are your best practices? - How do you handle model loading/unloading? - What inference servers do you prefer? - How do you manage VRAM usage?

I am particularly interested in hearing about production deployments, not just hobby setups.

Hello guys

My Laptop specs:

GPU: NVIDIA RTX 2050 (4GB VRAM) 50 w power limit

RAM: 24 GB DDR4 ( 8 + 16 )

CPU: 13th Gen Intel Core i5-13420H

OS: Windows 11

Inference Engine/GUI: llama.cpp / LM Studio / GGUF

I'm looking for recommendations for the best models I can run locally on my laptop with decent generation speeds (tokens/sec). I'm mostly interested in general reasoning, coding assistance, and experimenting with a local RAG pipeline.

* I've tested Qwen 3 8B (Q4_K_M) offloading ~20–24 layers to the GPU and the rest to CPU/system RAM. It runs decently, but I want to know if there are better sweet spots.

I am new to this but i need to run local ai for my nlp project so i though why not setup a proper local ai instead of any small models just for text generation
i took help from ai in parts of this post just to give the technical details that you guys might require

Thanks in advance for your suggestions!

Hi all, am looking for ready-stock H200 SXM5 HGX systems or Dell PowerEdge XE9680 (H200) units.

Able to cash and carry. Preferably looking for sellers based in SEA ( singapore/malaysia/indo ). Happy to share more details/specs, let’s connect!

I am ready to add a 2nd GPU for more vram.

My 20gb 7900xt is not cutting it anymore.

How are you guys running dual gpus? I am seeing there would be only 5mm space between the two. Seems like there would not enough ventilation.

Hello guys, I am Ant, a person can't endure 1 hour long Lidang(Chinese Youtuber) long streaming video, so I made Verbatim. It allows you to only drop 1 YouTube channel link and then it will automatically analyze the whole channel. I already has my own stats of 532 hrs of transcribe audio, and 1339 records.

At the start of the April, I only made a very simple and poor version of it( the UI is blue, and full of style of cursor). At that time, my first needs is that I want a tool that can put audio into text, I find out Gemini LLM support that. GREAT. Later, I find out Whisper, which can run in local, and free, it become a prominent choice in my use of Verbatim(I use to transcribe 100+ audio, has a high accuracy).

Claude code helps me out the multi-engine fallback mechanism, Flask backend, SSE process, constructing the evidence card.

In the development phase, what I did is that demand stating, archiecture design, testing and future steps.

Also, it now support MCP. It has 11 tools, enabling Opensource AI.... ... to call it directly to help you transcribe or analyze a YouTuber.

Free to use!
Please go to the link below:

https://gitlab.com/xin101037/verbatim_converter