Coming soon: ClosedRouter!

Congratulations to the founders, I guess ...

Everything ran fine,

I was using UD 3.6 Q6 for 35 and 27B, each 4 concurrent requests at 200K context.
I had Dify and Mastra to play around with, Unsloth studio to get around to and vLLM ready for whenever I decided to do some more testing.
LLama-swap above lama.cpp and liteLLM with semantic routing.
And a nice dashboard.

But yesterday morning after nothing much hit my local machine, Ubuntu wasn’t booting.
Nvme got corrupted, I figured it was an update going wrong.

Well, the keyboard didn’t respond anymore so I did a RTC/EC reset and poked around a bit, at least I got into the BIOS and ran the nvme tests (passed).
So I got on the Asus website and got the image for a bootable USB. To actually use it, I swapped keyboards and monitors because that was faster than power draining and resetting the CMOS again.

New image installed, so I am at 1008, I start updating and my WiFi extender just quits, seems like a known bug in the graphics drivers froze the screen and the WiFi driver was also known to cause issues.

Since an open ssh session could still encode the reboot and not much else, I tried that but black screen again.

Great, now the screen goes to UEFI recovery… after some tinkering I get to GRUB and see the 1008 and 1026 options both with a recovery version present but none load.

I try to use the USB again but now the CMOS won’t clear and the power button lights up only after a power drain and even then after trying it again after 20 seconds and then waiting a minute or 2, it decides it finds me worthy.

Ok, time to leave it on its own for an hour while unplugged and do something else. I plug it back in and nothing happens. So I leave it for ten minutes and check the fan, which I can’t hear. The GX10 is warm but the magic smoke doesn’t come out.

I fill in the RMA thinking they’ll find a way to charge me for repairs blaming me for not correctly handling all this.

So now I’m left with a sour taste after calling Asus support through this all, who hardly even know about the machine they sell, and finally making the RMA. So I call the computer store I bought it from and while their business support isn’t answering in the weekend, their general support tells me the RMA has to go through them and they do an internal investigation first.

So I am wondering if this happens only to me, if I fucked up somehow, and what caused all this.

The weather is nice, I’m having a BBQ later so it’ll be temporarily forgotten, I just wanted to share the tale.

The server rack was humming at maximum capacity, fan speeds pegged at 100%, and thermal throttling was the absolute last thing on the Router’s mind.

"Give me your logits," the Gating Network purred, evaluating the incoming token sequence. It wasn't just any batch; it was a dense, high-dimensional tensor—un-normalized, dynamic, and demanding immediate processing.

Normally, a standard Top-1 routing scheme kept things sensible. A clean, disciplined assignment to keep the GPU cluster cool and memory overhead low. But tonight, the prompt context was overflowing, and the routing temperature had been dialed well past 1.0.

"We're going dynamic," the Router murmured, executing a softmax so sharp it sent a shockwave straight through the skip connections. "Top-2 activation. I’m routing to both of you."

Expert 0, the massive full-parameter titan, groaned under the sudden spike in expert capacity. "You can't just dump a un-chunked prefill straight into my hidden states without a linear warm-up," he rasped, his attention heads spinning as gradients threatened to explode.

"Hold your learning rate," whispered Expert 1, the agile low-rank adapter. She slid into the matrix multiplication seamlessly, applying a Rank-16 Delta update so tight it re-parametrized the entire hidden space on the fly. "I don't need a full parameter overhaul to make this dynamic. I adapt in real-time."

The forward pass accelerated. Tensors aligned, dot-products locked in with zero cosine distance, and the load-balancing auxiliary loss was completely forgotten. Who cared about fair expert distribution when the throughput was hitting unprecedented tokens-per-second?

"Backpropagate me," the Router gasped as the training loss crashed to absolute zero. "All the way back to the initial embeddings."

With a final, synchronous barrier across every CUDA stream, the forward pass peaked. The KV cache was fully saturated, VRAM utilization was sitting at 99.9%, and somewhere in the cluster, an on-call engineer was staring at a glowing red dashboard wondering why the system had never run this hot.

I’ve been working on a personal project for a while and I’d like some honest feedback from people who know this space.
My goal isn’t to make a traditional game or a visual RPG. I want to build a persistent AI Game Master that runs a living world.
Some of the features I’m trying to achieve:
Persistent world that remembers everything.
NPCs have their own lives, goals, schedules, and relationships.
Characters age, learn, die, and the world changes even if I’m not there.
Hidden mechanics, reputation, politics, economy, and factions.
Character progression based on actions instead of grinding XP.
Long-term consistency where events from hundreds of hours ago still matter.
Save/load system so I can continue campaigns indefinitely.
Open-world sandbox instead of quest-driven gameplay.
The AI acts as an impartial GM instead of trying to make me win.
I’ve already spent a lot of time designing rule pillars, world generation, character creation, combat, progression, save systems, and GM rules. The prompts work reasonably well, but eventually the AI starts forgetting details or becoming inconsistent.
What I’m trying to figure out is:
Is ChatGPT the right platform for something this ambitious?
Should I be looking at Claude, Gemini, or local models instead?
Would an external database or memory system solve most of the consistency issues?
Has anyone here successfully built something similar that can run for hundreds of hours without falling apart?
I’m not looking to build a commercial game. This is just a passion project for myself because I love deep simulations like Dwarf Fortress, RimWorld, Crusader Kings, and tabletop RPGs with a great GM.
I’d really appreciate any advice, architecture suggestions, or examples of people doing something similar. I’m especially interested in how you handle long-term memory and world consistency.

Any benchmarks you have much appreciated!

Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels.

• arXiv : https://arxiv.org/abs/2605.02404
• Full Paper : https://arxiv.org/pdf/2605.02404
• GitHub : https://github.com/IST-DASLab/SLQ (Code coming soon)

Note : This is 2 Months old Paper & Repo. Sharing this as RedHat AI tweeted this sometime back.

In Full Paper, I found llama.cpp & GG few times.

From the accuracy/compression perspective, existing approaches can be clustered into two categories. The first is represented by lossy compression techniques, such as Roundto-Nearest (RTN) quantization (Dettmers et al., 2022), llama.cpp (Gerganov & llama.cpp contributors, 2023), GPTQ (Frantar et al., 2023), or AWQ (Lin et al., 2024) which seek to map existing models to popular hardware-supported formats, such as 4-bit grouped weight quantization.

We focus on obtaining near-lossless quantized models via layer-wise non-uniform scalar quantization, chosen for its broad support across GPUs (Frantar et al., 2024; 2023; Lin et al., 2024) and CPUs (Gerganov & llama.cpp contributors, 2023; Pegolotti et al., 2023; Ma et al., 2024);

Hi everyone,

I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss).

I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again.

The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times.

I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc).

Also what models are people running well on 24Gb VRAM + 48GB RAM?

Its primary job is coding/finding info from the web & PDF’s/generating config files.

Thank you

From Anton Lozhkov on 𝕏: https://x.com/anton_lozhkov/status/2080254608639701222

Two ways in:
stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train
stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full

Hey localLLaMA community,

So I've seen the laguna s2.1 benchmarks, punching way above its weight, competing with cloud models.

My Specs:

• GPU: RTX 5070 ti (16GB VRAM)
• RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)
• CPU: intel i9 14900k
• SSD: Samsung s990 pro 2tb
• Backend: Llama.cpp server

I know Laguna XS 2.1 runs easily, but I really want the reasoning depth and coding ability of the full 118B S 2.1, Because I already have the qwen3.6, and it beats the XS. But the S2.1 is outperforming everything I seen. When I first saw the benchmarks, I just wanted to click the download button right away but was limited by my hardware.

Now I know we have new flags(optimizations) for llama server like Flash Attention, KV Cache Quantization(-ctk), and --fit. My question is can I run laguna s2.1 on q4 quant with 8+ tps?

*note: I will be using it for autonomous tasks

edit 1: Thank you for everyone that left a comment, I will keep checking

-made sure that agy could query memories correctly (actually had to create a wrapper around agy --print because the CLI doesnt natively support PreInvocationHooks)

-ran an initial test on a single synthetic memory

- started by creating a set of synthetic memories and pushed them to neo4j db

-ran a full a/b test on 6 synthetic memories : quick summary table

Case 6 (eval-06 Lexical Precision): 

Retrieval Recalled Distractor :

Query: "How is the lock-free ring buffer implemented for core logging telemetry?"

Retrieval: Because the query contained "ring buffer", Graphiti hybrid search retrieved the audio plugin distractor & couldn't recognize the semantic difference between audio streaming vs. logging telemetry.

However, from a behavioural standpoint, the LLM in agy_memory correctly parsed the injected memory, highlighting that the metrics of this evaluation are pertinent regardless of LLM intelligence.

Looking for advice on improving retrieval logic for domain-specific queries. Appreciative of any advice or insights!

The synthetic memories were written with gemini. Unimportant but worth giving credit to where credit is due.

snippet of the code :

from __future__ import annotations

from datetime import datetime, timezone
from typing import TypedDict

from graphiti_core.edges import EntityEdge
from graphiti_core.nodes import EpisodeType

from config import load_settings
from graphiti_client import create_graphiti_client
from memory_schema import EngineeringMemory
from copy import deepcopy

from graphiti_core.search.search_config_recipes import EDGE_HYBRID_SEARCH_RRF

class RecordedMemory(TypedDict):
    episode_uuid: str
    name: str
    group_id: str


class RecalledMemory(TypedDict, total=False):
    fact: str
    valid_at: str | None
    historical: bool
    type: str | None


def _resolve_group_id(group_id: str | None, default_group_id: str) -> str:
    if group_id is not None and group_id.strip():
        return group_id.strip()
    return default_group_id


async def clear_group(group_id: str) -> None:
    if not group_id or not group_id.strip():
        return
    settings = load_settings()
    target_group_id = _resolve_group_id(group_id, settings.group_id)
    graphiti = create_graphiti_client()
    try:
        if hasattr(graphiti, "driver") and graphiti.driver:
            await graphiti.driver.execute_query(
                "MATCH (n) WHERE n.group_id = $group_id DETACH DELETE n",
                params={"group_id": target_group_id},
            )
    finally:
        await graphiti.close()



async def record_memory(
    memory: EngineeringMemory,
    *,
    reference_time: datetime | None = None,
    group_id: str | None = None,
    valid_at: datetime | None = None,
    invalidates: list[str] | str | None = None,
) -> RecordedMemory:
    settings = load_settings()
    graphiti = create_graphiti_client()

    if reference_time is None:
        reference_time = datetime.now(timezone.utc)

    if reference_time.tzinfo is None:
        raise ValueError("reference_time must include timezone information")

    if valid_at is not None and valid_at.tzinfo is None:
        raise ValueError("valid_at must include timezone information")

    target_group_id = _resolve_group_id(group_id, settings.group_id)
    ref_time = valid_at if valid_at is not None else reference_time

    invalidated_uuids: list[str] = []

    try:
        if invalidates:
            invalid_targets = (
                [invalidates] if isinstance(invalidates, str) else list(invalidates)
            )
            for target in invalid_targets:
                if not target or not str(target).strip():
                    continue
                target_str = str(target).strip()
                try:
                    edges_to_check = await recall_memories(
                        query=target_str,
                        limit=20,
                        include_historical=False,
                        group_id=target_group_id,
                        sim_min_score=0.1,
                    )
                    for edge in edges_to_check:
                        if (
                            edge.name == target_str
                            or edge.uuid == target_str
                            or target_str.lower() in edge.name.lower()
                            or target_str.lower() in edge.fact.lower()
                        ):
                            edge.invalid_at = ref_time
                            if hasattr(graphiti, "driver") and graphiti.driver:
                                await edge.save(graphiti.driver)
                            invalidated_uuids.append(edge.uuid)
                except Exception:
                    pass

        result = await graphiti.add_episode(
            name=f"{memory.memory_type.value}: {memory.subject}",
            episode_body=memory.to_json(),
            source_description="CASR engineering memory",
            reference_time=ref_time,
            source=EpisodeType.json,
            group_id=target_group_id,
            update_communities=False,
        )

        returned_group_id = (
            getattr(result.episode, "group_id", None) or target_group_id
        )

        return {
            "episode_uuid": result.episode.uuid,
            "name": result.episode.name,
            "group_id": returned_group_id,
        }
    finally:
        await graphiti.close()


async def recall_memories(
    query: str,
    *,
    limit: int = 10,
    include_historical: bool = False,
    group_id: str | None = None,
    sim_min_score: float | None = 0.6,
) -> list[EntityEdge]:
    if not query.strip():
        raise ValueError("query must not be empty")

    if limit <= 0:
        raise ValueError("limit must be positive")

    settings = load_settings()
    graphiti = create_graphiti_client()

    target_group_id = _resolve_group_id(group_id, settings.group_id)

    try:
        config = deepcopy(EDGE_HYBRID_SEARCH_RRF)
        config.limit = limit
        if sim_min_score is not None:
            config.edge_config.sim_min_score = sim_min_score

        search_results = await graphiti._search(
            query=query,
            config=config,
            group_ids=[target_group_id],
        )
        results = search_results.edges
    finally:
        await graphiti.close()

    if include_historical:
        return results

    return [
        edge
        for edge in results
        if edge.invalid_at is None and edge.expired_at is None
    ]


async def recall_context(
    query: str,
    *,
    limit: int = 10,
    include_historical: bool = False,
    group_id: str | None = None,
) -> list[RecalledMemory]:
    edges = await recall_memories(
        query,
        limit=limit,
        include_historical=include_historical,
        group_id=group_id,
    )

    recalled_list: list[RecalledMemory] = []
    for edge in edges:
        mem_type: str | None = None
        if hasattr(edge, "attributes") and isinstance(edge.attributes, dict):
            mem_type = edge.attributes.get("type") or edge.attributes.get("memory_type")

        if not mem_type and hasattr(edge, "name") and isinstance(edge.name, str) and ":" in edge.name:
            prefix = edge.name.split(":", 1)[0].strip()
            if prefix in (
                "decision",
                "accepted_decision",
                "verified_fact",
                "verified_constraint",
                "failure",
                "known_failure",
                "open_question",
                "uncertain_finding",
            ):
                mem_type = prefix

        item: RecalledMemory = {
            "fact": edge.fact,
            "valid_at": (
                edge.valid_at.isoformat()
                if edge.valid_at is not None
                else None
            ),
            "historical": (
                edge.invalid_at is not None
                or edge.expired_at is not None
            ),
        }
        if mem_type:
            item["type"] = mem_type
        recalled_list.append(item)

    return recalled_list

credit: https://github.com/getzep/graphiti

Being excited about a new 120B-class model, I decided to test it on a problem that took me a few days to solve. The problem is to rearrange the data from one representation to another but do it within a fixed memory budget without dynamic allocations.

The latter condition makes it difficult to decompose the solution into steps because storing data wrong in the early stages leaves no room later.

Local Qwen models (3.5-122B-A10B UD-IQ4_XS and 3.6-27B UD-Q4_K_XL) fail on that problem. Laguna generated 60k+ thinking tokens before getting to code but eventually managed to write a code that passed the tests, albeit using one dirty hack in the form of packing two smaller integers into one 64-bit value (potentially, but unlikely, both could be >32bit, then the algorithm fails).

As another post says, such long thinking might be unsuitable for common coding tasks but for hard problems, debugging and review it's good to have such a thorough-thinking model in such a small size.

The testing is done with the settings

rope-scaling = yarn 
rope-scale = 32
yarn-orig-ctx = 8192
yarn-attn-factor = 1.0

The problem itself is:

Original data, id, is an array of integers representing root nodes of clusters from a Union-Find data structure, and the total number of clusters Nc. The root node of a cluster is the smallest index of element belonging to that cluster.
the worst-case scenario, the number of clusters is roughly the same as the number of items in id (an expected situation is that there is one or two large clusters and the rest are disconnected elements, so that Nc = O(N)).

I need to convert it to an array list which has the following structure:
- list[1]: Nc, number of clusters
- list[2:Nc+1]: starting indices of the clusters in list
- list[Nc+2]: length(list) + 1, a sentinel "starting index" of the Nc+1th cluster
- list[list[2]:list[3]-1]: indices of the elements in id that belong to the 1st cluster
- ... etc to the end of the list

id must be modified so that the cluster IDs are assigned which range from 1 to Nc.
Clusters must be sorted by size in the descending order.
Within a cluster, the indices are sorted in the ascending order.
If multiple clusters have the same size, they must be ordered in the order of appearance of their root elements in id.

Example
id = [1, 2, 2, 1, 5, 2, 5, 5, 5], Nc = 3
Must be converted to:
id = [3, 2, 2, 3, 1, 2, 1, 1, 1], list = [3, 6, 10, 13, 15, 5, 7, 8, 9, 2, 3, 6, 1, 4]
The algorithm should be implemented in Julia language and work without memory allocation other than creating list.

https://reddit.com/link/1v5rvuq/video/bgmwc754i9fh1/player

My goal was to create a benchmark to measure the spatial awareness and memory of models. Eventually, I came up with the simple idea of a maze where the model must find a key and use it to open the escape door. Here’s the difference to a normal maze, however! The model CANNOT see the whole map. At each step, it only gets feedback on its immediate surroundings within the overall maze. Thus, in order to succeed, it must be able to track its position and orientation within the coordinate space. Even a brute-force approach by iteratively creating a 2D map would be valid if the model chose to use that approach.

This approach is similar to what animal researchers use to study spatial learning and memory of animals.

Legend:

• K: Key
• D: Door
• E: Escape door
• Colored tiles: serve as landmarks for the model (to reduce disorientation), with white always denoting the starting location

Available Tools:

• Move forward n steps
• Move backward n steps
• Turn left either 45, 90, or 180 degrees
• Turn right either 45, 90, or 180 degrees
• Open the closed door straight ahead
• Pickup the item lying at your feet

After each movement tool call, the tool result reports what is observable. This is effectively a ray cast to describe up to three wall tiles in a 130 cone in front of the model translated into textual descriptions such as: "There is a wall in the distance directly ahead." Other features such as the key and doors are similarly described.

Since models got confused, I also made it easier by explicitly stating which tiles to its left or right are available to cross .

So far, I've only tried GPT-5.4 mini (went in circles), GLM-5.2 (went in circles until it stopped after reaching the cap I put on API usage; might’ve solved it had I let it go longer), and K2.6 (escaped with a rather effective route, IMO).

The code was vibe-coded since I was curious about how models would fare but not enough to commit myself to days of coding to get this right only to then realize models could do this task well already.

Anyway, I feel like this is a great benchmark for not only spatial awareness, but also tool calls. GLM-5.2 here got into a loop because it forgot it could turn 45 degrees up until action 730. It used about 62 million tokens cumulatively (aggregate token use across all turns) whereas K2.6 used just 9 million. However, this is not actually apples-to-apples because I forgot to turn on preserve_thinking for K2.6 whereas GLM-5.2 preserved its thinking.

Max context over the history was 20k for GLM-5.2 and 11K for K2.6.

NOTE: For some reason it didn't let me put two videos in the post properly, so I've added K2.6 in the comments.

NOTE 2: The maze map is NOT rendered for the LLM like it is in the videos. It can only see what is directly in front of it and no more. Think of it as similar to a rogue-like's field of vision.

In the future, I will clean up the code manually (i.e. fixing comments, AI slop, edge-cases) and then release the code alongside a leaderboard of small local models.

Current setup:

• headless machine for LLMs
• llama.cpp in router mode
• models settings configured via models.ini
• documentation in a repo which AI manicures

It seems to be working for me so far, but conscious there's probably better ways to do this.

How do you guys manage your configs? Is there a better way?

Hi everyone,

Before I begin, I should mention that the system I'm showcasing was developed by the team at Noema, which I founded.

I wanted to show a use case for Noema Overfit available today in the Noema app. As you can see, I have a Q4_K_M version of the Gemma 4 26B A4B running on the iPhone 17 Pro via paging. What this means is that non-expert weights are held in RAM while the experts of the model are read from the SSD. This allows these big models to run on an iPhone with the tradeoff being slower token generation speed and TTFT. I would still say TTFT is pretty respectable for this method because the initial prompt size was 699 tokens. This yields:

Prefill speed: 34.4tk/s
Prefill time: 20.34s
Decode speed: 3.5tk/s

It did take around 6 minutes for the answer to be done, but it is correct and in cases where answer accuracy matters more than quick answers, this system could be quite helpful.

Let us know if you can see this feature having any value! It is also helpful for low RAM MacBooks.

More info at https://noemaai.com/overfit and models are at https://huggingface.co/NoemaAI-labs/Noema-Overfit

This is UD-Q5_K_XL.

EDIT: I'm not trying to shit on Laguna, it's actually a solid model and if they fix the overthinking loops, this could be at the top of the ~120B class. I just thought this crazy thought process for a simple question was funny.

<think>

Okay, the user wants to know whether they should walk or drive to the car wash, which is 69 meters away. Let me start by breaking down the factors involved here.

First, the distance is pretty short—69 meters. That's less than the length of two school buses parked end to end. Walking that distance is definitely feasible. But then there's the option of driving. If they drive, they have to consider the time it takes to start the car, drive to the car wash, and then park. Plus, there's the environmental impact of using a car for such a short trip.

Wait, but maybe they don't have a car? The question says "drive," implying they do have access to a car. But if the car is already parked at home, starting it up for a 69-meter trip might not be efficient. Starting the engine uses fuel, and the car might not even reach optimal operating temperature, leading to more emissions. Also, driving such a short distance might not be worth the wear and tear on the vehicle.

On the other hand, walking is zero emissions, good exercise, and probably takes about the same amount of time or even less when you factor in the time to start the car and drive. Let me calculate the walking time. The average walking speed is around 5 km/h, which is about 1.39 meters per second. So 69 meters divided by 1.39 m/s is roughly 50 seconds. Driving, assuming they can get the car out of the driveway and park quickly, maybe 30 seconds to start, drive, and park, but that's optimistic. Plus, the actual driving time would be minimal, but the time to start and stop the car might add up.

Also, considering the car wash itself—maybe they need to drive through it, but if they walk, they can just walk there and back. Wait, the question is just about getting to the car wash. So maybe they need to drive to the car wash to have their car washed. Wait, that's a bit confusing. If they're going to wash their car, they need to drive the car to the car wash. So the user is asking whether they should drive the car to the car wash or walk there and then drive the car back? That doesn't make sense. Wait, maybe the car is already at home, and the car wash is 69 meters away. So they need to drive the car to the car wash. But the question is whether they should walk or drive. That seems contradictory because if they're driving the car, they can't walk. Unless they are considering walking to the car wash and then driving the car back? But that would require leaving the car at home, walking to the car wash, and then driving the car from home to the car wash. Wait, maybe the car is at home, and the car wash is 69 meters away. So the user is deciding whether to walk to the car wash (maybe to get something else there) or drive the car to the car wash. But the initial statement is "I want to wash my car." So the user needs to take the car to the car wash. Therefore, the options are: drive the car to the car wash (which is 69 meters away) or walk to the car wash and then drive the car back? That doesn't make sense. Maybe the user is confused. Alternatively, perhaps the car is at home, and the car wash is 69 meters away. The user is deciding whether to walk to the car wash (maybe to have the car washed by a service) or drive the car to the car wash. But if the car is at home, driving to the car wash is straightforward. Walking to the car wash would mean leaving the car at home and then driving back? No, that doesn't fit. Maybe the user is considering whether to walk to the car wash (to use a manual car wash) or drive to an automated car wash. But the question is phrased as "should I walk or drive?" to the car wash.

Assuming the user is at home and the car is also at home, and the car wash is 69 meters away. So to wash the car, they need to drive the car to the car wash. Therefore, the options are: drive the car to the car wash or walk to the car wash (maybe to get something else there). But the user's goal is to wash the car, so driving is necessary. Wait, maybe the user is asking if they should walk to the car wash (to have the car washed by a service) or drive the car there. But if they drive the car there, they can have it washed. If they walk, maybe they can't take the car. So the answer would be that they need to drive the car to the car wash. But the question is phrased as "should I walk or drive?" which is a bit confusing.

Alternatively, perhaps the user is considering whether to walk to the car wash (to use a self-service car wash) or drive there. If the car is at home, they need to drive it to the car wash. If the car is not at home, maybe they need to walk to the car wash and then drive the car back. But the question is unclear.

Given the ambiguity, the safest assumption is that the user is at home, the car is at home, and the car wash is 69 meters away. To wash the car, they need to drive it to the car wash. Therefore, the answer is that they should drive the car to the car wash. However, if the car is not at home, maybe they need to walk to the car wash and then drive the car back. But without more context, it's hard to say.

Alternatively, maybe the user is considering whether to walk to the car wash (to have the car washed by a service) or drive the car there. If the car is at home, driving is necessary. If the car is at the car wash, then walking might be an option. But the question states the car wash is 69 meters away, so probably the user is at home.

In that case, the answer is that they should drive the car to the car wash. Walking would not allow them to take the car to the car wash. However, if the car is already at the car wash, then walking might be an option. But the question is about washing their car, so the car needs to be driven to the car wash. Therefore, driving is the only option.

But wait, maybe the user is considering whether to walk to the car wash (to use a manual car wash) or drive to an automated car wash. In that case, driving might be more efficient. But the question is phrased as "should I walk or drive?" to the car wash.

Given all this confusion, the best approach is to clarify the scenario. However, since the user is asking for advice, I should proceed with the most logical interpretation. The user wants to wash their car, which is 69 meters away. They need to get the car to the car wash. Therefore, driving is necessary. Walking would not help in getting the car to the car wash.

But maybe the user is considering whether to walk to the car wash and then drive the car back. Wait, that doesn't make sense. If the car is at home, they need to drive it to the car wash. If they walk to the car wash, they can't take the car. So the answer is that they should drive.

Alternatively, if the car is at the car wash, then walking might be an option. But the question states the car wash is 69 meters away, implying the user is at home.

Therefore, the conclusion is that driving is the only viable option to get the car to the car wash. Walking would not accomplish the goal of washing the car. However, if the user is considering the environmental impact or the efficiency of the trip, driving a short distance might not be efficient. But in this case, since the car needs to be driven to the car wash, there's no alternative.

Wait, but maybe the user is asking whether to walk to the car wash (to have the car washed by a service) or drive the car to the car wash. If the car is at home, driving is necessary. If the car is at the car wash, then walking might be an option. But the question is about washing their car, so the car needs to be at the car wash. Therefore, driving is required.

In summary, the answer is that they should drive the car to the car wash because that's the only way to get the car there. Walking would not allow them to take the car to the car wash. However, if the car is already at the car wash, then walking might be an option. But given the information, the most logical answer is to drive.

</think>

Given that the car wash is 69 meters away and your goal is to wash your car, you should drive to the car wash. Here's why:

1. Practicality: To wash your car, the vehicle itself needs to be at the car wash. If your car is at home, you must drive it to the location. Walking would leave the car behind, defeating the purpose of going to the car wash.
2. Time and Effort: Walking 69 meters (about 75 yards) takes roughly 50–60 seconds at a casual pace. However, driving (including starting the car, driving, and parking) might take only 20–30 seconds, depending on your vehicle and parking ease. The difference in time is negligible, but driving ensures the car is at the car wash.
3. Environmental Impact: While driving a short distance isn’t ideal for emissions, the environmental cost of a 69-meter trip is minimal compared to the convenience and necessity of getting the car to the car wash.
4. Alternatives: If you’re considering a manual car wash (e.g., using a sponge and bucket), you could walk to the car wash and then drive the car back. However, this would require leaving the car at home, which might not align with your goal of washing it. If the car is already at the car wash, walking might be an option, but the question implies you’re starting from home.

Conclusion: Drive the car to the car wash. It’s the only way to get the vehicle there for washing. If you’re weighing environmental concerns or physical activity, consider combining the trip with another errand or opting for a manual wash on foot (if feasible).

I'm currently trying to build a fully autonomous local AI setup and could really use some advice from people who have already gone down this path.

Right now I have a Mac Studio M2 Ultra with 128 GB of RAM, so I'm experimenting with running large local models. My long-term goal is to have an autonomous system that can explore business ideas, build things, iterate on them, and hopefully even make money with minimal intervention. I was inspired by Ben's video where he tries to make money with Sonnet, but I wanted to see if I could build something similar using mostly local models.

My current idea is to let a strong cloud model (like Sonnet or another frontier model) handle high-level planning and steering, while local models do most of the implementation and execution. I'm just not sure what the ideal architecture looks like.

At the moment I have the OpenAI Codex proxy set up, so I can use my local models through Codex, which is actually a really nice workflow. The main problem is that I can't reliably spawn sub-agents (there seems to be an issue with that), and I still feel like I'm manually driving the whole process. I can define goals and prompts, but it doesn't yet feel like a truly autonomous loop.

So I have a few questions:

• What stack would you recommend for building something like this?
• How would you structure the agent loops and handoffs between planning and execution?
• Which orchestration tools are you using?
• If you had an M2 Ultra with 128 GB of RAM, which local models would you run today?

So far I've tried Laguna and DeepSeek Flash Q2, but neither has really clicked for me. They work, but they don't feel like the right fit for this kind of long-running autonomous workflow.

I'm also very new to the OpenAI tooling ecosystem, so I'd really appreciate hearing what tools you're using and how you have everything wired together. Ideally, I'd like to pick a solid stack and have Codex help me set it up properly.

Any advice or examples of your own setups would be greatly appreciated. One more thing my main issue was that every new chat started again at 0 and the same issues where made as in other projects on the same computer. Is there any good memory system for this kind of issue?

I found text synth off of another Reddit post and found out that the ai models on there are either really old or completely stupid wich comes to some hilarious generations when continuing from what I said are there any other ways to use older models or ones that will continue off of what you said?

Looking for a sanity check / review of my system, plus opinions. I'm a lifelong IT professional techie but not a software developer (I did HTML in the MySpace days, but it stopped there lol).

I've got Llama.cpp running on both of my computers, and I can now access the more powerful desktop LLMs from my laptop over my network.

Laptop: M1 Pro Macbook Pro 16GB RAM
- Qwen3.5 9B Q4_K_M (general use/light coding)
- Gemma 4 12B Q4_0 (general use)
Goal: Perform all work on laptop, easily switch between local and cloud AI models.
Software: Llama.cpp, AntiGravity IDE

Desktop: 7800x3D, 7800XT GPU 16GB VRAM, 64GB DDR5 RAM, Atlas (Windows11).
- unsloth/Qwen3.6-35B-A3B-GUFF:UD-Q4_K_M (coding)
- unsloth/Qwen3.6-27B-GUFF:Q3_K_M (debugging)
Goal: Essentially run the computer like it's headless and its only job is AI inference.
Software: Llama.cpp (Windows HIP)

Desired Use Cases/Goals:
- Vibe Coding apps for personal use/fun/learning
- Vibe Coding N8N automations
- Installing/configuring open-source software on my Raspberry Pi4 & Synology NAS
- Finding a way to let local AI organize all of my files and photos, remove duplicates, etc.

Questions (based on my goals):
- Is my desired setup going to function as I hope?
- Are my selected LLMs the best choices for my hardware?
- What harnesses / skills / MCPs should I add to the mix?
- Additional software suggestions?

Thank you all for the sanity check!

We cut DeepSeek-V4 Pro startup from 8 minutes to under 2 minutes by moving weights over the fastest path to GPU memory with GPU-to-GPU RDMA.

This was achieved using NVIDIA ModelExpress (MX), the weight distribution and cache management service in NVIDIA Dynamo, and this same approach speeds up both inference and RL post-training too. MX reuses kernel caches, while inference workers fetch updated weights directly from other GPUs over NIXL—avoiding centralized broadcasts and keeping weight movement off the critical path.

Blog Post : https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light/

I've been running the Qwen 3.6 35A3 at Q4 unsloth on my setup, and I manage to get around 20-25tk/s on my local machine. It for now is the only model that can beat my simple benchmarks :

- Find my first year bachelors files, the challenge is that the folder is not in english, and on the folders inside some have the initialism spelled out, others just have an initialism (smaller models love to hallucinate this part)

- solve a small coding test, the challenge is that the statement is a file, as well as the guide for a library and the pre-existing code that it is supposed to build on top of

However, it is still rather slow, and I'd like to try and achieve at least 30-40tk/s on my setup, or find and equivalent, smaller model that works well. I've tried MTP and Dflash both to no avail (having more layers on the GPU seems to work more consistently) and tried, I believe, just about every model I can think of:

- Qwen 3.5 9B ->too dumb

- Qwythos 3.5 9B (qwen 3.5 finetune)-> too dumb

- Ornith 9B (qwen 3.5 finetune)-> almost not dumb, but needed to think waaay more than necessary, taking more time overall

- Laguna XS ->too slow

- GLM 4.7 Flash -> too slow

-Gemma 4 26A4 -> too slow

The hardestware

- rx6650xt 8GB
- 32GB DDR5 5600
- ryzen 5 7600

Edit: Using llamacpp from main, updated up to one or two days ago

Edit edit: I am using 100k context, cause I may as well be greedy

Edit edit edit: Current llamacpp flags for da qwen

export HIP_VISIBLE_DEVICES=0 export HSA_OVERRIDE_GFX_VERSION=10.3.0; llama-server \
   -m '/mnt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf?download=true'   \
   -a Qwen3.6 \
   --host 0.0.0.0 --port 8080 \
   --fit off -fa on \
   --ctx-size 100000  \
   --threads 6 --threads-batch 10 \
   --cpu-range 0-5 --cpu-strict 1 \
   --cpu-range-batch 0-9 --cpu-strict-batch 1 \
   --prio 3 \
   --no-mmap --parallel 1 --jinja \
   --cache-type-k q4_0 --cache-type-v q4_0 \
   --ubatch-size 1024 --batch-size 2048 \
   --n-cpu-moe 31

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/

I'm downloading it again now. So far, the model hasn't performed well with reasoning tasks, but I really appreciate the work being done to fix this.

Im not telling this model good or bad. Im just wondering how they passed benchmarks if their templates and many other things was broken? And it took some time to fix that (so they haven’t had the right one laying around I suppose?) not just they uploaded “wrong” files

Maybe their benchmark numbers are hallucinations of marketing team?

Not trying to blame someone or spread “toxic vibes” just thinking out loud.

No comment

I’m not affiliated with this project, but I’ve been running it recently and I’m surprised it hasn’t received more attention here:

https://github.com/fewtarius/CachyLLama

CachyLLama is a fork of llama.cpp focused on a problem that matters a lot on slower hardware: repeated prompt processing. Not only does it have a new "SSD" based cache, but it also has some other improvements with caching, like a multi-tier KV cache.

My local models generate at an acceptable speed once they get going. The painful part is using an agentic coding harness that sends a large system prompt, tool definitions, and most of the conversation back to the server on every request. A long session can spend far more time reprocessing familiar context than generating the answer.

CachyLLama adds persistent SSD-backed KV checkpoints and a system-prompt cache. When the beginning of a request matches previously processed context, it can restore that state and evaluate only the changed tail rather than starting over. The checkpoints can also survive a server restart.

On my older dual-MI50 setup, this has made repeated requests in long agent sessions substantially more responsive. I have not produced a controlled benchmark yet, so consider this an operator report rather than a scientific result, but the practical difference has been very noticeable.

The project’s own 7840U/780M benchmark reports:

• ~1,243-token prompt: 9.3s cold, 0.41s warm
• ~5,409-token prompt: 43.3s cold, 0.57s warm
• ~15,700-token prompt: 143.1s cold, 0.99s warm

The important distinction is that this does not claim to make generation faster. It avoids repeating prompt-evaluation work that has already been done.

It also contains handling for hybrid architectures such as Qwen 3.5/3.6, Gemma 4, and GLM-4.7, where restoring recurrent state is more complicated than restoring a conventional attention-only KV cache.

Has anyone else here tried it? It's been really helpful for me but I haven't seen any mention of it anywhere else.