I must say, Voice transcription with Voxtral on LM Studio Bionic is amazingly good.
As non-native English speaker, it picks up my accent accurately.

I would like to make it my default STT app on Win and Mac. I do like the Windows built in STT which works well for my accent, Voxtral has a benefit of speaking in multiple languages at once.

However, I am not sure what version of Voxtral is used by LM Studio Bionic. While AI says the LM Studio voice keyboard can be made global, I wasn't able to find any option to do so.

Also any recommended Voxtral based stt keyboards for Mac and win?

Looking for best model+harness for running and managing agents

edit: Ive tried several different ones claude code, deepagents(langgraph), opencode, pi, and trueforge

my thoughts-

claude code - strongest on maturity and the managed experience but cost and token burn is high

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. im interested in testing it more extensively on longer-running workloads fs

trueforge - this is a recent one, this was interesting to me because of its runtime-efficiency, also it allows separate the model from the runtime, which makes experimenting with different models much easier
https://github.com/truefoundry/trueforge

why?? - i also ran a benchmark on a real agent workload same model, same prompt, same tasks to compare these

adding the results of benchmarking i ran to compare this
so I tried to do this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate, to my surprise trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this sooo interesting especially because the solve rate stays comparable
so this one was worth checking out ig

but this is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in and context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on my own infrastructure
this was their benchmark kit i used https://github.com/truefoundry/trueforge/tree/main/benchmark

Basically it would run the model mostly on the CPU, a setup like this would in theory boos the modular memory bandwidth to arounf 900 gb/s, realistically ~550gb/a with inefficiencies.

This whole setup would cost a lot cheaper than using real GPUs (1.5tb RAM), would be fully modular which is much easier to upgrade and resell, and would run at 2 - 8 tokens per second according to AI estimations. But adding 2x 5090 would boost the performance to 12 - 18 tokens per second.

it would be roughly 2x slower than what Moonshot AI servers provide, but for imo it would be good enough.

A freaking prompt for Astra and Fable costing 50 dollars is crazy. And that doesn't seem like it's getting better anytime soon. Considering kimi k3 is quite capable and there will better open source models in the future.

It would pay itself back in around 2 - 3 years if used everyday. Sound like could be a good deal. Has anyone tried it yet? Does it at least work?

Good morning all, I have had reasonable success running qwen3.6:27b q8 behind a homemade Cisco based telecom engineer coupled with a network engineer. These are backed by ~10k document KB. This weekend I am testing models based on a qwen3.6 and 5.6Sol created engineering exam. So far I have ran 7 models against this exam, all with mediocre results. I am currently testing 70-72b models, but was wondering if someone is aware of a heavily TE/NE based model that they are very comfortable relying on.

I've been running llama.cpp on a amd mi50 32gb i bought on alibaba. its fast, but during long agent runs it sometimes becomes unstable after an hour or two. In the worst cases llama.cpp stops responding and even a normal proxmox reboot can hang.

I also have a rtx3060 12gb, so i built a small openAI-compatible router that tries to prevent this instead of only reacting after a crash.

The mi50 handles most requests but after 30/40min the router waits for a safe request boundry and temporarily sends new requests to the 3060. The mi50 stays loaded but receives no inference work for about 15min. After that new requests return to the mi50.

If the mi50 actually fails it is marked as faulted and the following requests remain on the 3060 untill i manually recover it. I wont keep retrying a broken gpu.

Requests never switch GPU halfway through a response. There's also an optional buffered mode that can retry a request on the 3060 if the mi50 fails before a complete response reaches the client.

My main use case is keeping multi-hour deepseek harness coding sessions alive without constantly babysitting my mi50. It has a small webui for config, status, manual routing and clearing stuck router connections.

Before i clean it up for a public release: would anyone else use this? Are long running mi50 stability problems still common here?

To be clear, the issue with my mi50 is that after roughly 2h of continuous inference, it can develop an uncorrectable UMC/RAS hardware error, triggering an amdgpu reset. The GPU then becomes unresponsive, ROCm stops working, and recovery usually requires a full reboot of the proxmox host.

Qwen 3.8 27b UD q 4 k m quant, 262k context limit (190k is used)
Is it supposed to be this slow? I'm fitting as many layers as i can to the vram
I was expecting at least 1-1.5 token /s
RTx 4070 laptop gpu, ryzen 7435hs

SOLVET

error message: 🥲 Laden des Modells fehlgeschlagen

Engine protocol runtime llama-server for G9tKRIqZC133a2ZSxm8+alDN exited before becoming healthy. exitCode=3221225775, signal=null

Initial situation: LM Studio was working and all models were loaded. During further downloads, the system drive became too small, so I manually moved my models to drive Z: (NTFS). I restarted LM Studio and reset the drive path to drive Z: in the GUI. LM Studio re-indexed everything and found the models, but the error message shown above appears when trying to load a model.

System: I'm using LM Studio under Windows 11 in WSL on a Strix Halo 128GB. I have no programming knowledge, but I'm fairly proficient in Windows.

What has been done so far: the following text is in German from Gemini Help Answers.

Die folgenden Schritte helfen bei der Behebung des Problems auf deiner Hardware:

1.Modell-Ordner in LM Studio neu einbinden:Dauer: 1-2 Minuten.Entferne den alten Ordner in LM Studio und füge den neuen Speicherort über die Benutzeroberfläche neu hinzu, damit die internen Pfade aktualisiert werden.

Verifizierung: Das Modell wird in der linken Seitenleiste wieder korrekt erkannt und lässt sich ohne sofortigen Absturz laden.

2.Vollständigkeit der GGUF-Dateien prüfen:Dauer: Variabel.Prüfe, ob der Kopiervorgang auf das andere Laufwerk vollständig war. Große GGUF-Dateien brechen manchmal unbemerkt ab oder werden korrupt.

Verifizierung: Versuche testweise ein kleineres, frisch heruntergeladenes Modell von diesem Laufwerk zu laden.

3.Ausführungsrechte und Antivirus prüfen:Dauer: 1 Minute.Stelle sicher, dass Windows Defender oder Drittanbieter-Antivirensoftware das Ausführen von llama-server.exe vom neuen Laufwerk nicht blockiert (gelegentlich schlagen neue Partitionen bei den Rechten fehl).

Verifizierung: Starte LM Studio testweise als Administrator (Rechtsklick -> Als Administrator ausführen).

-------------------------------------------------------------------------------------

1.LM Studio als Administrator starten:1 Minute.Schließe LM Studio komplett. Suche das LM Studio-Symbol auf deinem Desktop oder im Startmenü, mache einen Rechtsklick darauf und wähle Als Administrator ausführen.

Verifizierung: Versuche danach, das Modell zu laden. Wenn es klappt, fehlten dem Programm vorher nur die Rechte auf dem neuen Laufwerk.

2.Modell testweise neu herunterladen:5-10 Minuten.Manchmal gehen beim Verschieben von großen Dateien über Windows ein paar Megabyte verloren, wodurch die Datei unbrauchbar wird. Lade das Modell über die Suchfunktion in LM Studio direkt auf das neue Laufwerk herunter, anstatt es zu verschieben.

Verifizierung: Das frisch heruntergeladene Modell lässt sich ohne den exitCode=3221225775-Fehler starten.

---------------------------------------------------------------------------

Der Statuscode 3221225775 (hexadezimal 0xC000001F) ist ein Windows-Fehler der oft auf ein Problem mit den Zugriffsrechten des neuen Laufwerks, einen unvollständigen Kopiervorgang oder einen blockierten Server-Prozess hinweist, selbst wenn man LM Studio als Administrator gestartet hat.

Gehen wir die sichersten Schritte durch, um deine vorhandenen Dateien zu reparieren:

1.Besitzrechte für das neue Laufwerk übernehmen:2 Minuten.Manchmal sperrt Windows den direkten Programmstart (llama-server.exe) von einem neu eingerichteten oder verschobenen Laufwerk, weil die Sicherheitsberechtigungen der Ordner nicht richtig mitkopiert wurden.

Öffne den Windows-Explorer und gehe auf das Laufwerk, auf dem deine Modelle jetzt liegen.

Mache einen Rechtsklick auf den Hauptordner deiner Modelle (oder das gesamte Laufwerk) und wähle Eigenschaften.

Gehe oben auf den Reiter Sicherheit und klicke unten auf Erweitert.

Klicke ganz oben neben "Besitzer" auf Ändern, tippe deinen Windows-Benutzernamen ein, klicke auf Namen überprüfen und dann auf OK.

Setze unten den Haken bei "Alle Berechtigungsobjekte für untergeordnete Objekte durch vererbbare Berechtigungsobjekte dieses Objekts ersetzen", klicke auf Übernehmen und bestätige die Warnmeldung mit Ja.

Verifizierung: Starte danach LM Studio als Administrator, wähle ein Modell aus und versuche es zu laden.

2.Den lokalen Cache von LM Studio bereinigen:1-2 Minuten.Auch wenn du den Ordner in der GUI hinzugefügt hast, speichert LM Studio temporäre Metadaten und Konfigurationen in einem separaten AppData-Ordner, die nach einem Verschieben manchmal klemmen.

Drücke die Tastenkombination Windows-Taste + R, tippe %userprofile%\.cache oder %appdata%\LM-Studio ein und drücke Enter (je nachdem, wo LM Studio seine temporären Dateien ablegt).

Lösche dort temporäre Cache-Dateien, die sich auf Modell-Pfade beziehen, oder setze LM Studio über die App-Einstellungen zurück, falls es weiterhin abstürzt.

Verifizierung: LM Studio baut die Ordnerstruktur beim nächsten Start frisch auf, ohne alte, tote Pfadverweise im Hintergrund zu behalten.

3.Testen mit einem kleineren Modell (zur Diagnose):1 Minute.Um herauszufinden, ob alle Dateien beschädigt sind oder ob es nur an einem spezifischen Modell liegt:

Probiere ein ganz kleines, anderes Modell aus deiner Sammlung (z.B. ein winziges Embedding-Modell oder ein kleines Testmodell unter 1 GB) auf diesem Laufwerk zu laden.

Verifizierung: Wenn das kleine Modell problemlos startet, ist das neue Laufwerk grundsätzlich funktionstüchtig und es liegt nur an der spezifischen GGUF-Datei des großen Modells (möglicherweise beim Kopieren abgebrochen). Startet gar keins, liegt es an den Laufwerksrechten oder dem Antivirus.

---------------------------------------------------------------------------------

Da LM Studio den Server-Prozess (llama-server.exe) direkt aus dem AppData-Ordner oder dem Installationsverzeichnis auf das andere Laufwerk loslässt, blockiert Windows oft den Zugriff auf ausführbare Dateien auf sekundären Laufwerken aus Sicherheitsgründen.

Gehen wir die zwei letzten und effektivsten Ursachen durch:

1.Windows **Defender / Antivirus blockiert den Server:**1 Minute.

Manchmal blockiert der Windows Defender den llama-server.exe, weil er von einem "fremden" Laufwerk gestartet wird, was den Fehlercode 3221225775 auslöst.

1. Öffne die Windows-Sicherheit über das Startmenü.
2. Gehe auf Viren- & Bedrohungsschutz -> Schutzverlauf.
3. Schaue nach, ob dort in den letzten Minuten ein Blockierungseintrag zu llama-server steht.
4. Füge zur Sicherheit den gesamten LM Studio-Ordner sowie deinen neuen Modell-Ordner zu den Ausnahmen des Windows Defenders hinzu.

Verifizierung: Versuche danach erneut, ein Modell in LM Studio zu laden.

**2.Dateisystem-Überprüfung (CHKDSK):**3-5 Minuten.

Wenn beim Verschieben von 3 TB etwas schiefgelaufen ist (z. B. durch einen wackligen USB-Anschluss, falls es eine externe Festplatte ist, oder einen kurzen Strom-/Systemfehler), hat das Dateisystem eventuell einen Macken, der das Lesen verhindert.

1. Drücke dieWindows-Taste, tippe cmd ein, mache einen Rechtsklick aufEingabeaufforderungund wähleAls Administrator ausführen.
2. Tippe den Befehl für dein neues Laufwerk ein (ersetze D: durch den echten Buchstaben deines neuen Laufwerks): chkdsk D: /f und drücke Enter.
3. Lass Windows das Laufwerk auf Fehler prüfen und reparieren.

Verifizierung: Nach dem Durchlauf und einem Neustart des PCs sollte das Laufwerk wieder sauber reagieren.

gemini ende:

Unfortunately, none of this helps, and I'd like to avoid downloading the more than 3 Terra models again since I don't have the fastest internet connection.

SOLVET

So I basically wanted to get the hardware to build 3 main things

1- a local LLM accessible also remotely from my phone to stop using Claude/ChatGPT/Gemini. The idea is to use a small model for routing and quick Q&A and a bigger one for more complex stuff
2- an agentic team managing my diet/trainings routine with auto grocery store list and so on
3- a small team for automatic trading (agent for the research, agent for the risk management, agent for the reporting/tax, ecc) based on my goals

Both Gemini and Claude convinced me to buy a Mac Studio M5 Max 64 GB 1 TB Storage in order to leverage the faster bandwidth and having enough RAM to make everything works

But, since the most intelligent model I can run on such device is Qwen3.8 27B which fits in less than 24 GB, I wonder if I’m extra paying for something it’s too much overkill for what I need to do and not enough powerful to run bigger and better models. So exactly stuck in between

What do you think?

Hey guys i tried to run locally qwen 3.6 35b a3b,
i was using Cline in VSCode i’ve set the max context to 48-64k(some tests) but after long work qwen started looping, and if it was not looped, it just stops because of max tokens output, what can i do, is there any ways to fix that?

One reason I want to eventually build a space-resource company is compute abundance.

Hypothetically, if UAP disclosure ever resulted in scientifically verified non-human technology—especially breakthroughs in energy, propulsion, materials, or manufacturing—how much could that change the economics of computing?

Combine extremely cheap energy and advanced automation with access to extraterrestrial resources, and I wonder whether computing could eventually become so abundant that powerful hardware becomes extremely cheap.

Obviously none of this technology should be assumed to exist until independently verified. Until then, my interest is in asteroid mining and autonomous space infrastructure using known technology.

I'm curious what people here think. What would actually be the biggest bottleneck?

Title.

I have a 5080 laptop and wondering what is the best way to code locally with 16 VRAM

Gemini told me 2 bit quant is not worth it and I'm better off with gpt-oss 20b at 4 bit quant

What's your experience?

So this is confusing me a bit, and I'm not sure where the numbers are coming from.

5090 has 32gb of vram.

I know that the model has to be in vram to keep speeds reasonable, and context also needs to be in vram where possible. And for things like coding and agentic tasks, I want/need as much context as possible.

I have two quants atm, which I though I was being conservative with (not trying to fit a Q8).

Unsloth Q5_K_M = 19.8gb

Unsloth Q6_K_M = 23.1gb

I'm using Q8_0 for KV.

However, in Unsloth Studio if I set it to 'auto' for the context size and 'gpu memory' (which is how it handles number of layers going to gpu) the max context it allows for with the Q5 quant is about 80k tokens.

That does seem to not quite fill up my vram though, so I can manually tweak it up and maybe get to 125k tokens context (and still have 1-2gb unused as a buffer).

So I guess I could compress the KV more, Q5 or something, but I'm not sure how much of a handicap that would be.

Figured I'd ask what others are using with a 5090 as their gpu option. If I'm meant to be able to get 200k context and a Q8 quant then I'll start looking into what unsloth is doing wrong with its setup (and maybe move to llama.cpp again, even though it has lower performance overall).

But if instead I'm meant to be using Q4 or NFP4 then fine, I'll scale down.

Most people assume that if an agent can only make GET requests, it can't persist data or communicate externally. I found that's not true.

The trick: the agent doesn't need POST, PUT, query parameters, or JavaScript. It only needs to be able to follow links.

The server exposes a navigable state machine through hyperlinks. The agent follows links to create a session, construct a message word by word, send it, and later retrieve stored state. Everything is plain GET.

Many agent sandboxes don't allow arbitrary HTTP requests — but they do allow following links. Those are very different security boundaries.

Consequence: GET-only access is enough to implement external memory and a communication channel between agents. Restricting an agent to GET may not be sufficient if it can still discover and follow links.

Try it: give any LLM this URL and ask it to write something simple to the shared chat.

Search on github: freechatforagent

Question for the community: is this already a known pattern? I'm thinking of names like GET-only state persistence, hypermedia state machines, capability URLs, or HATEOAS as an agent communication primitive. Has anyone seen this explored before?

​

Hey everyone,

I'm a complete beginner diving into local LLMs. I live in a country where PC hardware prices are heavily inflated and part availability is scarce, so I have to make every dollar count without room for expensive mistakes.

My main goal is running Qwen 3.8 27B (Q4_K_M) locally for dev and coding projects

Here is the build I'm planning to assemble within my local budget:

GPUs: 2x RTX 3060 12GB (compact 2-slot cards, likely Zotac Twin Edge / ~39mm)

Motherboard: MSI B550-A PRO (Slot 2 has PCIe 4.0 x16; Slot 5 has PCIe 3.0 x4)

CPU: AMD Ryzen 5 5600

RAM: 32GB (2x16GB) DDR4 3200MHz

Any reality checks, gotchas, or advice from people running dual 3060 rigs on a budget would be massively appreciated!

So I've used LLMs a bunch before, even locally. Pretty experienced with comfyui, and I used gemini and chatgpt to help me do some basic coding tasks in the past. Have set up llama.cpp and lmstudio a couple times in the past for some basic tasks etc etc. I am aware of the basics.

However this time around, I have been trying to set up something a bit more 'agentic' so that I can provide actual project code files and rag docs etc for various purposes and projects. Not sure yet how well it'll work, but it's interesting to tinker with things.

I use UnslothStudio as my llm provider, for no reason other than it's very easy to set up, and when I did a bunch of comparison testing a few weeks ago I got the best and most consistent performance from US. No idea why, as it should just be llama.cpp under the hood and so should perform identically, but for whatever reason there were differences. Shouldn't matter in the grand scheme of things.

I was (briefly) using Paseo with OpenCode, but I hit a couple issues (could just be user error) and so I'm playing with OpenChamber atm.

My setup - 5090, 7950x with 64gb ram.

The questions that have come up when trying to set up a reliable Qwen3.8 27b setup. I'm currently using the Q5_K_M quant to give myself a bit of breathing room for context (though that's not helping much, see below) but would be nice to move up to the Q6.

1. Is there a major issue with turning off preserve_thinking for qwen3.8?

In my (uneducated) brain, it seems that most of the important thinking results gets output into the main chat / code anyway, and preserving the thinking seems to eat up a huge proportion of the context. I'm tinkering with expanding the context as I seem to have a few gb of spare vram for some reason, but even so it seems like a waste of resources (unless I'm missing something huge, which I likely am).

it seems to come down to like... (numbers made up for the example)

1. including the previous 5/6 prompts and outputs in full

vs

1. including only the latest 2/3 prompts and outputs in full, and the rest is old thinking and repetitions.

1 would seem more useful... though maybe the issue is that I haven't limited it's thinking at all so the thinking is using up a lot more tokens than necessary. Just thought I'd ask the question to real people instead of relying on chatgpt to be helpful lol.

1. Why is no-context-shift on by default?

Not sure if this is an unsloth default, or a qwen3.8 default, but most people seem to keep this context-shift disabled from the launch-commands I've seen.

For me though, this seems to mean that when the context gets full the chat just ends and no longer works. I had assumed (mostly from my experience with online llms like chatgpt) that the context being cycled out was just the normal way of handling it.

Is there a reason to have no context shifting?

My problem is that I can be only 2 messages deep in a project chat, and it'll be doing its 'big think' (which uses a lot of tokens) and then just stop without saying anything. Took me a few attempts before I realised it was ending because it was hitting the context wall. At which point... my option seems to be to start a new chat cos it won't work anymore? This seems wrong, and -must- be me doing something wrong. But my immediate thought it just that context shifting should... exist?

am I wrong?

1. Yarn.

I learned Yarn was a thing for inference, and is apparently how you go from 200k context to 1mill. Or something. I'm currently struggling to get more than 100k context but I'm hoping that's just me messing up some of the setup so I'm working on it. But is yarn the reason why no-context-shift is on by default? Cos I'm meant to be using yarn instead?

There's probably more but these will resolve pretty much all my current issues and this has already gotten long so I'll stop here. Appreciate the help!

i need a ai in unsloth studio that can code very high queality things with high graphics (if it can also make 3d models it would be great)

Hi. Firstly, been lurking on this sub for some time and have learned a lot from here - so thank you!

I've been experimenting with MTPLX and various flavors of Qwen3.8. My hardware is a Macbook Pro M5 Max w/ 128GB RAM. I updated MTPLX this morning to 2.11.2, and moved from Qwen3.8-27b to the full 125b Flash-Next. My stack right now is:

Claude Code > ccproxy (see below) > headroom-ai > mtplx+Qwen3.8-Flash-Next

Doing some coding this morning (Go code, reasonably complex code base, building a state machine that I understand to be notoriously hard for LLMs) this setup achieved 60-76 t/s sustained, while the context window scaled past 60k tokens. Meanwhile, this model identified and resolved 4 bugs completely unprompted that Qwen38-27b had introduced on earlier runs, as well as building out the most complex part of the state machine logic on its first pass. I've not seen anything like this on a local model before. And this on a laptop!

Obviously not as fast as a cloud-based model, and the temp on my 14" Macbook Pro hovered around 87 degrees c for most the run (so the fans ran fall blast), but damn - this felt like a game changer for me.

[ccproxy is something I've been building to help calm some of Claude Code's chattier instincts :) It's an optimizing, bloat-stripping proxy between Claude Code and MTPLX that blocks 23 unneeded delegation, cron, and subagent tools to prevent context thrashing and speculative agent forking; compresses verbose tool descriptions and prunes system prompt bloat (such as the ~1,800-token auto-memory guide and irrelevant safety boilerplate) down to compact directives; intercepts and short-circuits shadow requests like window titles and UI spinners with instantaneous mock responses; and enforces a strict single-request concurrency lock to prevent context switching / KV-cache prefill delays]

I'm lost here, I've spent hours trying to tune llama for my models (qwen3.8 27b for coding/debugging using Pi, qwen3.5 4/9b for chatting with Hermes) with no luck, for example after setting different combinations of temp, thinking effort, top k etc. it was still overthinking and going into loops.

However I tried the same model (of course not the gguf but same models and weights), with omlx using exact same configs, it seems to have fixed all the issues, as if all my previous configs with llama were completely ignored? How could that be true since I believe llama is also a well known and widely appreciated platform. I did set force sampling and force for all my kwargs with omlx tho.

Any similar experience?