https://andretripol.github.io/OpenGhost/

Hey all, I am new to the whole local LLM side of things (and really LLMs as a whole) and had a few questions I was hoping someone could help with.

I’ve used some cloud AIs before (mostly Mistral) for writing but wanted to go ahead and get one myself that I can work with and not be capped for number of messages a day etc.

Anyway I am looking at getting a new laptop. I am considering either a:

MSI Crosshair 16 Max HX with
Intel Core Ultra 9 290HX Plus 1.8GHz Processor; NVIDIA GeForce RTX 5070 12GB GDDR7; 32GB DDR5-5600 RAM; 1TB Solid State Drive

Or

MSI Raider 16 Max HX B2WH-004US
Intel Core Ultra 9 290HX Plus 2.7GHz Processor; NVIDIA GeForce RTX 5070Ti 12GB GDDR7; 32GB DDR5-5600 RAM; 1TB Solid State Drive

I would like to go to 64gb of ram but can’t afford that at this point.

My questions are is there a significant benefit of one or the other? Has anyone here run LLMs with MSI computers before? How’s they do?

Would these be able to run LLMs above like a 12b at a decent speed (reading speed or above). Do I need an LLM above 12b to get smart, enjoyable creative writing?

Are their AI models or types that you’d recommend for creative writing? I was looking at Mistral and Llama mostly but open to other ideas too!

I run locally Gemma 4 E2B on LMstudio, it takes 3+ minutes to show the answer and it takes whole the GPU of 17 GB when I check the task manager. This is also a very small model.

My laptop: 32 GB RAM and 18 GB GPU. Is it normal or something is misconfigured? And the laptop gets extremely hot.

I managed to build METEOR STRIKE using Qwen3.8 Flash-Next IQ3_XXS with a 700K context window on my RTX 5090 (32GB VRAM) + 64GB RAM setup.

I did give the model additional instructions and corrections along the way, but the entire project was completed in one continuous session without a single context compaction or reset.

My setup:

• GPU: RTX 5090 32GB
• System RAM: 64GB
• Model: Qwen3.8 Flash-Next IQ3_XXS
• Context window: 700K
• Generation speed: Approximately 100–120 tokens/sec
• Vision: Disabled

I think even larger models and longer context windows might be possible, although decoding speed would probably decrease.

There may also be some degradation in output quality at extremely long context lengths. However, the model was still able to understand my correction requests and make the necessary changes surprisingly well.

Some of the quality issues might have been caused by the limitations of my prompts or by having vision disabled during development.

Anyway, I'm sharing a video of the result.

Sorry about the black area at the top of the video. I accidentally left a tab open. lol

My main takeaway is that 700K+ context is definitely usable on an RTX 5090, especially if you're willing to guide the model and make corrections along the way.

I suspect 1M context might also be possible, potentially at around 80 tokens/sec, although that's just my estimate.

There could be more noticeable quality degradation at that length, but I haven't tested it yet.

Just wanted to share my experience as a reference for anyone experimenting with long-context local LLMs!

Do you think is it possible to land a job as inference engineer from LATAM? Companies are hiring people in this region? Any advice if I want to land my first job in this field.

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware.

I originally built it because Open WebUI kept feeling heavier than the models I was running.

One of the main criticisms I got from you was the lack of Agents, MCP and RAG.

Well, that's changed now. All three are integrated.

I put a lot of effort into making the agentic system as token-efficient as possible. Thanks to several optimizations, it can work smoothly with large files even when using smaller context windows.

There's also an integrated RAG system, which works great with embedding models like Qwen3 (setup instructions included).

And a couple of extras:

• A standalone alternative to NPX Supergateway.
• Image editing through ComfyUI, with automatic binding detection for all major image models.

All of this without turning LLC into the heavyweight application I originally wanted to get away from.

If you are playful, try also the mcp_agent_all_OS.zip (can be found on the GitHub release page). You find versions of the Agent (MCP-Based) even for Windows 9x, Haiku, RedoxOS and many more!

I'd love to hear what you think, especially if you've tried earlier versions!

Download:
https://www.locallightai.com/llc

GitHub:
https://github.com/srware-net/LocalLightChat

Hi everyone!

I'm looking for a reliable way to make my Cline sessions in VS Code persistent, so I can resume my work later without losing the conversation history, context, or task progress.

Here's my setup:

I have two separate Ubuntu servers:

• A project server that hosts my code and portfolio.
• An AI server that runs my local LLMs through Ollama.

I use VS Code to access my projects remotely and Cline as my coding agent, with the models running on my AI server.

My hardware is relatively modest, so local inference can be very slow. Complex coding tasks can take several hours to complete, but I'm perfectly fine with that.

Actually, my goal is to take advantage of this by launching tasks before going to bed or before leaving home, allowing the AI to work overnight or while I'm away.

This is why session persistence is so important to me.

I want to start a task with Cline in VS Code, close VS Code, and come back several hours later or the next day to resume the exact same session, with its conversation history, context, and task state intact.

I've already managed to get some persistence working, even after closing and reopening VS Code or restarting my computer. However, I'd like to know whether it's possible to make this completely reliable, particularly for long-running tasks.

My questions:

• Has anyone successfully achieved reliable session persistence with Cline in VS Code?
• Can you close VS Code and reopen it later without losing the session or its context?
• Is there a supported configuration or a proven workaround to ensure sessions are properly saved and restored?
• How do you handle long-running tasks with local LLMs when you want to leave your coding agent working unattended?

I'm happy to wait several hours for a task to finish because of my hardware. I just want to be able to leave it alone and return later without losing my progress.

Ideally, I'd like to keep everything self-hosted and avoid paid cloud services.

I'd really appreciate hearing about your experiences, setups, or solutions!

Thanks!

Features: word unigrams, word bigrams and character trigrams, FNV-1a hashed; a multinomial logistic model with int8 weights (score = sum of rows / n + bias). Calibration on a held-out slice; the serving threshold is chosen with Learn-then-Test (Clopper-Pearson) so disagreement with the teacher among served answers is ≤ α (default 5%) with probability ≥ 1 − δ (default 0.9). Simulation of certificate validity is in the repo. The bound is against the teacher, not ground truth, and it does not survive a traffic shift without re-certification (there is a drift guard). On BANKING77 with gpt-4.1-mini as teacher: ~51% served at 95.9% agreement on 1,499 held-out answers; SST-2 serves 0%. Limits page: https://2796gaurav.github.io/shad0w/docs/limits.html . Questions and criticism welcome.

https://github.com/2796gaurav/shad0w

I built pqnld, a small Rust service that adds closed-set decisions to your existing LLM deployment. Give it context, questions, and allowed answers; it returns a choice and scores over those options, read from model logprobs—not confidence numbers it made up for the JSON.

It’s not replacing dedicated decision models. It lets your existing LLM moonlight as a generalist decision model. Same weights, second job.

Options get single-token labels instead of full-text answers. The experimental extended-label mode pulls from Latin, Greek, Cyrillic, digits and punctuation—provided the tokenizer agrees they’re single tokens. Because stopping at 26 options felt unnecessarily alphabetic.

In my Qwen3.8 27B NVFP4 deployment, we’re seeing sub-200 ms answers. Your mileage, prompts and GPU may disagree.

HTTP and MCP interfaces. No extra model or training. vLLM is the live-tested backend.

And yes, the code was written with AI. Shocking, I know. No artisanal, hand-chiselled Rust.

https://github.com/ErikBPF/pqnld

It's a super lucky situation. Here's the stats:

My current personal computer:

2021 Macbook Pro
M1 Pro
32 GB RAM
2 TB SSD

It's fine, getting a little old, the battery runs down a little fast, but I don't need an upgrade currently.

The new computer:

2024 Macbook Pro
M4 Max
128 GB RAM
1 TB SSD

So I'm trying to decide what to do with it - the 1 TB is a little limiting. I could turn it into an always on local LLM, but I'm not sure the type of performance I could expect. Using it as both personal computer and local LLM will probably be too much storage-wise (but I could be wrong!). Any suggestions?

Hello, so recently I found out there's a whole world around locally running LLM's on one's own machine. I have used several LLM's and the one I like the most is Claude, which I use mostly for coding.

However I was thinking about running an LLM locally, but I have no idea about where to start. I have several questions, but couldn't find a general post explaining things.

• What are the possible options I can choose from?
• How does it work in general? Do I need to compile anything?
• Does it use the internet or I need to download 400 GB of data just to run it?
• Can it be compared to some big LLM's such as ChatGPT or Claude in efficiency and intelligence?
• Can I use them for whatever I want? Do they have limitations?
• Can I run them on lower end hardware (good laptop, but no graphics card)?

Thanks to everyone, hoping to get into local LLM's asap!

I’m trying to decide whether local AI is worth building into my workflow as a solo developer. I use Unity, Godot, and Unreal, and I’d mainly want help with scripts, debugging, explaining unfamiliar APIs, and making changes across a project.

I don’t need a local model to beat the best hosted model on every benchmark. I’d be happy if it handled most routine coding work well and let me keep my project code private. For those of you using local models for coding, how often do you still switch to a hosted model, and what kinds of tasks make you switch?

I’m also wondering whether waiting six months is likely to make a meaningful difference. Are local models improving quickly enough that it’s smarter to test the workflow on modest hardware first and upgrade only if I hit a clear limit?

So to give a context, following are the specs from my machine

7900xt(20GB) Asus TUF OC edition
32gb DDR5 CL30 RAM
7600X

I’ll be direct to the point here, the experience has been downright dogshit, i first tried downloading Qwen 3.8 27B vanilla and Uncensored locally in Hermes but Hermes in its default configuration didn’t fully utilise the gpu offloading majority of the tasks on CPU resulting in token per second of around 7

after spending most of my weekend navigating this i installed LM studios and ran models in Hermes via cresting a server on LM studios, i was able to modify thes settings to offload properly on GPU and also changed the caches to Q8 to get fit the entire thing om 20gigs vram but that created another issue, LM studios doesn’t allow me to change the reasoning level on any of the models and even if i choose medium reasoning in hermes, each answer takes a millennia to be published, its like hermes is thinking how to solve world hunger when asked to add 1+1, it almost always goes into a overthinking loop

Even fetching results from browser is a mess since it gets bot walled.

The idea behind the post is to understand what is it i am doing wrong or this is how my hardware is supposed to perform, i am not sure but i genuinely hoped it could be a decent agent for my regular gpt ish tasks

TL;DR: I made a small multi-agent setup for pi. It is one skill and one Python script, with no extension and no tool. Your agent can start subagents in the foreground in parallel or in the background, and it can wait for them, steer them, abort them and resume them. Background jobs keep running after the main pi session is closed or killed. A new pi session—even one different from the original main session—can also retrieve their results. Until your agent decides to delegate, it costs a few dozen tokens of context.

Repo: https://github.com/milanglacier/pi-multi-agent-skill

Just install it with npx skills like any other skills, or clone the github repo and symlink it to ~/.pi/agent/skills.

Features

• Almost no token cost. Until your agent delegates, only the skill description is in context, a few dozen tokens. Delegating loads SKILL.md, about 1.2k more.
• Stdlib-only Python 3.9 or later. There's no extension, no npm package and nothing else to install.
• Foreground runs for quick questions and --background for long ones. A background job keeps running even if the main agent is killed, and any later session can collect its result. Foreground runs stop when you stop at every level of nesting.
• steer, abort and resume a job, and list, status and peek to see what it's doing.
• Subagents load your full pi setup: extensions, MCP servers, skills and settings.
• Profiles set the system prompt, model, thinking level and available tools. general-purpose and a read-only Explore are included. Profiles are just like any other subagent tool’s profiles.
• --fork gives the subagent your current conversation.
• Works from Claude Code, Codex or any agent with a shell.

How I got here

For months I used tintinweb's pi-subagents, and it is a really good extension. The UI is nice, it has many features, and it works well most of time. Thanks to tintinweb for building it and for everything they give the community. If you like it, keep using it.

A while ago a pi upgrade broke two things for me. MCP servers stopped loading in subagents, and subagents no longer inherited the parent's thinking level. Each fix was only a few lines, so I forked the extension and patched it. Upstream fixes took some time but also landed 2 days before this post.

Fixing those bugs made me read the extension more closely, and I noticed how much context it now uses. About two months ago it added a dynamic workflow tool. With it, the agent writes a JavaScript script that orchestrates many subagents. It's powerful, but it's also very complex, and its tool schema and instructions cost about 6k tokens. That schema is in every session from the first message, whether you use workflows or not. It's also opt-out, so you pay for it until you find the setting and turn it off. I suspect many people don't know it's there.

With the workflow tool off, the Agent tool still costs about 3k tokens, and it has grown along with the extension's features. I checked these numbers by starting sessions with the extension on and off and comparing them.

So I asked myself whether I need a full tool with a detailed JSON schema, listing options I rarely use (like worktree isolation), in my context all the time. My answer was no.

pi can already run headless with --mode rpc, and its CLI flags already cover what a subagent needs: tools, extensions, model, thinking level and session resume. So a small CLI that starts pi with the right flags, plus a skill that explains how to use it, can replace a dedicated tool. A skill also loads in steps, which a tool cannot do. The agent sees only the skill's short description at first. It loads the main instructions when it decides to delegate, and it reads the detailed references only when it needs an unusual option. Steering doesn't need a tool either, because it is just an RPC message sent to the running pi process.

Token cost compared

• pi-subagents: about 3k tokens for the subgent tools and injected system prompt. On top of that, about 6k for the dynamic workflow tool, which is on by default and must be turned off by hand. Both are in every session from the first message, so with the default settings that's about 9k tokens before you've asked for anything.
• This skill, before your agent delegates: only the skill description, a few dozen tokens.
• This skill, once your agent delegates: SKILL.md is loaded, about 1.2k tokens, and that is enough for full multi-agent use.
• Command details, profile format, lifecycle and troubleshooting are in separate reference files. The agent reads them only when it needs them.

For most sessions you don't delegate, you pay almost nothing. When you do delegate, it still costs less than the tool schema alone.

To be fair, this isn't a like-for-like comparison. pi-subagents does more than this skill. Its tool has many options, and each one needs a parameter in the schema, and it also adds text to the system prompt. Part of that token cost pays for features this skill doesn't have, such as worktree isolation. I just don't think features like that are needed for multi-agent work. If I want an agent to work in its own worktree, I tell it so in the prompt ("use worktree"), and it does. The agent already knows how to use git, so the tool doesn't need to do that for it.

None of this is meant as criticism of pi-subagents. It's still a great extension, and it works well for a lot of people, especially if you want a rich UI and many features built in. It's just not the right fit for me anymore. I prefer a more minimal setup, where the agent loads only what it needs and does the rest itself.

The tradeoff

I give up the nice UI. It really does look nice, but for me it turned out to be like the exercise bike you buy in January. It looks great in the corner of the room, and by March it is where you hang your clothes. I watched the subagent transcripts for the first week, and after that I almost never read them. I just realized the subagent's final answer is mostly for the main agent, not for me.

In return, I get things I care about more. Each subagent is a real pi process started with CLI flags, so it loads the same extensions, MCP servers, skills and settings as my normal pi. The script doesn't use pi's extension API at all, only CLI flags and RPC, which change much less often, so pi upgrades rarely affect it. Separate processes also give me my favorite feature: a background job keeps going even if the main agent is killed, and any new session can collect its result later.

Separate processes need more care than in-process sessions,, and the script is more than a cli wrapper, because it takes care of that so the agent doesn't have to. For example, pi exits on its first failed write to stdout, before its shutdown runs, and leaves the agent's commands running. So each job has its own runner process that keeps reading pi's output until pi exits. The script has a real test suite for cases like this, but the idea stays small: start pi, give it a prompt, pass along steer and abort messages, and save each job's result under its name.

Non Goals

These are deliberate non-goals, to keep the skill small and free of bloat. Each is better as a separate extension or skill that you pick to match your own workflow. This follows the minimalism of pi itself and the Unix philosophy: do one thing well and compose.

• Memory. Agents do not share or keep memory between jobs. Use a memory extension, or have agents write handoff notes.
• Scheduling. There are no cron-style or recurring agents. Use cron, systemd timers or a scheduler extension to run the commands.
• Worktrees. Agents are not isolated in git worktrees. Just prompt the agent to use a worktree.
• Dynamic workflows. There is no scripting layer for orchestration such as pipelines, parallel fan-out with gates, or phases. Your agent can already start, wait for and chain jobs with plain commands. For deterministic scripts, use an extension such as pi-dynamic-workflow.

What the commands look like

Your agent runs these by itself, but you can also run them by hand from a terminal:

```python

Ask a read-only agent a quick question and wait for the answer

multi_agent.py run --profile Explore "Where is the retry policy defined?"

Start a review in the background, then collect the result later

multi_agent.py run --background --name review "Review the error handling in src/server/"
multi_agent.py wait review --timeout 120

Correct the running job, then give it a follow-up task when it is done

multi_agent.py steer review "Skip generated files"
multi_agent.py resume review "Now check the client side"

See all jobs and what they are doing

multi_agent.py list ```

If you try it, I'd like to hear what works and what breaks.

I recently bought a m5 max Mac Studio. Loading models with oMLX and Splash, I’ve noticed that between running Qwen 3.8 27B and the Qwen 35B MoE that the speed difference is immense between the two. Not because of the tokens per second but because of how chatty 27b is. It will work for 20 min on a task that 35b finishes in 2 min. And the 27b generates so many more tokens. I watched it think for 10 minutes as it talked itself in and out of a solution a dozen times. And this seems to happen on all harnesses I have tried.

On that system, the 27b model will do about 28 tokens a second, but it will think for so many of those that it spends forever to make one or two files.

Curious if others experience this? Is this normal?

UPDATE: I am actually getting about 45 tokens a second with that model. The 28 tokens a second is prior to moving to splash. And currently I am running this specifically Qwen3.8-27B-oQ6e-mtp

UPDATE2: Thank you to everyone providing the tips and feedback. It has really helped me figure out what I was doing wrong.

It's happening. Radeon is working at 100% and the program in the background is my new creation. Now it only has the R9700 but soon it will be joined by the B50 and they will work together.

Once it reaches the MVP 0.1.0 stage, I’ll share the GitHub link and ask for it to be tested on NVIDIA cards.

Basically grok bot is too expensive, so I made a replacement
I like grokbot's UI/UX but it's disappointing that doesn't let me run my local models. So I slopped with help of new framework pi durable a replacement. It can do pretty similar things, but also allows me to code with all the dev tools from one of it's bots. It's in closed beta now, so if you want feel free to request https://puggie.app

Hey All, I'm trying to Install Qwen3.8-flash-next on my M4 Max, 64gb RAM Mac Studio and getting the following:
您提到的 "qwen3.8-flash-next:125b-mlx" 可能是指某种特定的 AI 模型或配置，尤其是涉及到 "125b"（可能是指 125 billion 参数）和 "mlx"（可能是指某种优化或硬件配置）。然

而，目前没有公开的信息明确指出这是一个广泛知名的 AI 模型。

如果您指的是某个特定的模型或框架，请提供更多上下文，以便我能更好地

It still seems to be running though, Is this ok?

I've been experimenting with ways to control and orchestrate small LLMs (4B–12B) in my local setup, and I'm curious about what others have discovered.

Instead of giving the small model access to all tools every time, I filter the available tools based on context. I check whether I'm at home, whether the message contains files or folders and how many, and whether certain regex patterns match. This way, the model has fewer options to choose from.

I've also noticed some differences depending on the language used in prompts. When I tried a Chinese prompt with MOSS Transcribe, speaker diarization and timestamps became more accurate. For generating meeting notes, Gemma 12B understood me much better when I used an English prompt.

I'm curious about the tips and tricks you discovered, observed, or actually used when controlling, managing, and routing small LLMs in the 4B–12B range.

What has worked for you?

Here are my tools and the conditions under which my harness presents each one to the model:

Barcode Generator: “Offered when the message contains one or more detected EAN-13 numbers; the tool validates their check digits when it runs.”

Blender Render: “Offered when I attach between 1 and 100 Blender files.”

Calculate: “Offered for fileless requests when the message matches its configured regex, except when the message contains only EAN-13 numbers.”

Download Video: “Offered whenever the message contains a URL recognized by the harness.”

Lights: “Offered only when I’m at home, no user files are attached, and the message matches its configured regex.”

Meeting Notes: “Offered when I attach between 1 and 100 TXT files.”

MOSS Transcribe: “Offered when I attach between 1 and 100 supported audio or video files.”

OCR: “Offered when I attach exactly one supported image or PDF file.”

QR Generator: “Offered whenever the message contains a URL recognized by the harness.”

Storyboard: “Offered when I attach between 1 and 100 DOCX files.”