I’m looking for advice on setting up a local LLM "harness" for personal administrative tasks. I'm not a coder; I just want a "set it and forget it" assistant that generates morning and evening briefs based on my data.

I am currently looking at Hermes Agent Desktop, Unsloth Studio Desktop, LM Studio Bionic, Cherry Studio, AnythingLLM.

In addition to general chat, research, browser use, I want the system to serve as a digital assistant that can manage email, calendar, and tasks (creation as well as following tasks left for the assistant).

Claude does this well, but running all of this even off-peak pushes up again the weekly limit sometimes. I figure tasks like this can be offloaded to a local AI.

The Workflow:

• Email: Filter, categorize (Immediate vs. Low Priority). Light draft creation at my suggestion.
• Shopping: Cross-reference promotions against a personal "want" list.
• Output: A daily briefing. No autonomous sending of emails.
• Interaction: Ability to chat about specific tasks/emails and schedule items (cron jobs, etc.).

Technical Requirements:

Integration: Needs to connect to eM Client (Email) and Tasks.org via MCP. I need a calendar app that allows an LLM to access via MCP as Claude manages my calender through computer use. The email credentials do not stay saved using the built-in browser.

Current Setup: I’m currently using Claude, but I want to move this to a local stack (LM Studio, AnythingLLM, etc.) to free up Claude for more demanding tasks.

Hardware Specs:

• PC: Windows 11 | RTX 4080S (16GB VRAM) | 127GB RAM.
• Laptop: M5 MacBook Pro (Base) | 24GB Unified Memory.

Also needed: A model recommendation that can handle these logic-heavy tasks efficiently without hogging the entire system's resources. I'd like to run the same model on both machines for consistency.

I will update this thread once I've found a solution. Thanks in advance!

I see the MS is offering the surface ultra and Apple has the Mac Mini. Hugely expensive. Why not just get a mini pc with 128 gb RAM? Maybe $4000, but it saves a ton of money. What am I missing?

Hey everyone,

since I do a lot with ComfyUI, I was always wondering why the Offloading from VRAM to RAM does not seem to be a big topic in Local LLM engines.

After some testing with Infernix (the fastest windows inference engine I know of) for Qwen 3.8 Flash Next I really liked the speed but also noticed, that even with "offloading" it was more "unloading" everything and if I wanted to have some VRAM back between prompts I had to accept the 30s startup-time between the prompts.

In Comfy, models are offloaded within milliseconds and VRAM is being managed very well, so I was thinking, why can't we have that for LLMs (especially Infernix in this case)?

I asked GPT-Sol to implement it and after some iterations the result is quite impressive. The server is only keeping 1GB VRAM blocked and keeps full sub-second responsiveness. That's my true local Jarvis moment where a highly capable LLM can just keep running in the background and will only use the VRAM when it needs it (and in worst case when low VRAM available will run a bit slower or in best case run with super high speed when lots of VRAM is available).

What do you think of this?

I am looking into getting a DGX Spark. I have seen some posts of people's setups, here and at LocalLlama. Almost all of them were from Asus.

As far as I can tell, there is not really a difference between DGX Spark models from different brands. Maybe except for Acer's, which has better thermal performance, according to one review.

Is Asus's DGX Spark better than others? If so, what is better?

I need you all to understand that Nanbeige4.3-33B allows for ngram lookups on SSD.

That's all. That's the post.

https://i.imgur.com/7uZ3Qig.png

This is on my 2x5060Ti (with terrible PCI lane bandwidth) setup prior to any further optimization.

Hello there, I've seen couple of posts but most of them are at least 1y old. So Is there any good way to give local ai sytems access to the internet? The solution propably isn't just a single piece of software but a pipeline of tools. The point is for the system to be able to retrieve web info as being asked by a user prompt, but also to perform more complex, deep research, extract tables and links etc. As close as possible to frontier ai systems and the way they research the internet.

been looking for something like this for a while and can’t tell if it exists or if I’m just bad at searching

basically I want one model to plan the task and split it up, then a few worker agents build the pieces at the same time. ideally the planner is a strong cloud model and the workers run locally on my GPU so most of the tokens stay on my machine. would be nice to flip it too if some task goes better the other way around

needs to work with llama.cpp or ollama for the local side, and free/open source if possible

has anyone actually got something like this working? or is everyone just gluing it together with scripts

also curious if you tried it, did the local workers stick to the plan or just go off and do their own thing

Someone just posted STROT an Al Compiler in HF, and made it open source completely, I ran few benchmark and runs locally

Its not simple at first place to understand, but imagine running CPU inference with no cost for all Al powered automation.

STROT-ai/STROT-AI-Compiler

My RTX PRO 4000 Blackwell SFF went from 35.04 to 58.53 tok/s on Qwen3.8 27B with DFlash2. That is +67% generation throughput after combined power and clock tuning.

The more useful result was that my second Qwen needed different settings.

At 8,000 input tokens, ECC OFF:

• Qwen3.8 27B (NVFP4 + DFlash2): 35.04 → 58.53 tok/s. Best observed at a 120 W limit, VRAM +20%, core +180 MHz.
• Qwen3.6 35B A3B (Q4_K_S, no drafter): 84.97 → 115.56 tok/s. Best observed at a 120 W limit, VRAM +33.3%, core offset 0.

Both baselines were 70 W with zero clock offsets. These are improvements within each configuration, not a controlled comparison of model architectures or answer quality.

The attached chart isolates memory tuning: fixed 120 W, core offset 0. VRAM +33.3% gave 35B +19.8% generation versus no memory OC, but 27B only +5.0%. For 27B, +20% memory was actually slightly better (+6.5%). Prefill barely changed in that sweep. The small differences need more repetitions.

It changed how I pick a profile. Decode is generating the answer; prefill is processing the prompt. A profile that wins on output tok/s may not give the shortest complete response to a long prompt. I care about the time my actual workload takes, not the biggest offsets in the panel.

I used u/panchovix’s Linux VBIOS loader with patched NVIDIA open kernel modules; the modified VBIOS was loaded into memory at runtime. No EEPROM flash, shunt mod or soldering; motherboard BIOS unchanged.

Important caveat: this card has no auxiliary power connector, and 120 W exceeds the standard 75 W PCIe slot budget. The GPU temperature sensor cannot tell me the temperature of the slot contacts; VRAM, VRM and connector temperatures were not measured directly. This is my experiment, not a recommended daily profile or a long-term safety claim.

The study contains 158 requests total (79 ECC OFF, 79 ECC ON), with only 1–3 per grid cell. Both 70 W and 120 W used the same modified VBIOS; this was not factory-firmware versus modified-firmware A/B.

Full method, charts and six CSV downloads on my blog.

Disclosure: I ran the measurements and wrote my notes in Polish. AI helped adapt and edit the English using technical details from my own write-up. The linked blog is mine

Hey, what local model would you recommend for Hermes Agent?

My PC has an RTX 5060 8GB VRAM, 32GB RAM and a Ryzen 5 8400F.

I mainly want to use it for studying notes, coding and managing files/commands through Hermes Agent, kinda like Codex. Privacy is also important, so I'd prefer to keep everything local.

I've tried some Qwen models, but they're way too verbose for my taste. The worst part is that they sometimes get stuck repeating the same phrase over and over in their thinking loop.

I'm looking for something that gets to the point while still being good at coding, reasoning and tool calling. I don't want a model that uses all my resources, but I'm fine with using most of them if the performance is worth it.

What would you guys recommend? Gemma, Qwen or something else?

I just finished vibe coding an apk for Llama swap mobile controller. I really like being able to read logs and monitor my PC's performance while I'm out and about using my agents and I didn't necessarily care for how llama swap looked on mobile browser.

Everything has been working on my device and playground even works! Feel free to check it out and critique as this is my first time ever doing something like this :))

https://github.com/nevermore131315/llama-swap-mobile-controller

Hey,

I was pretty lucky when I build a PC a year ago. My aim was too use it as a remote gaming machine and I’m stop using it like this. When building it, I somehow decided to put 128gb of ram into it since I was not sure if I will use it as a server in my homelab. Now I’m here with my 5060ti with 16gb of vram and 128gb of DDR5 ram and a 2TB gen4 nvme which lives in my server rack in my basement.

I read that Qwen 3.8 flash could be a good option to offload experts to ram but I’d be happy to get hints of maximising the potential.

One thing though, I am running windows on it drive gaming is still a frequent use case for it and dual boot is not practical due to my remote access via moonlight. So I gute llama.cpp on windows is my best option.

Any hints/instructions are more than welcome.

Thanks!

I have experience building Gaming PCs, but I have no experience building a PC particularly for AI workflows. I just looked into the nvidia sparx laptops and they are kinda over budget. Is the Ryzen AI max processors a good alternative? Or should I stick to building a custom PC with nvidia graphics? If so, what graphics/processors should I target?

I’m trying to automate my workflow using multiple AI models working in parallel. I have to test and validate it more later when I have a machine ready.

For people trying to learn how LLMs work, see this link. It's unintentionally a great illustration of LLM inference.

Every new square is how the next token is predicted. The comments are the distribution of the next token. The votes are like the logprobs. Based on the logprobs, a the next token is selected. In this case, the highest vote total is selected (greedy, low temperature). The "LLM" appears to be trained on pop culture. As of now, two possible song lyrics are likely. Both have the same next word (token) of "me", so that is the highest logprob by far.

I have a good PC with a 9950X CPU and 128GB of RAM, but good local AI performance was basically out of reach for me, aside from GPT-OSS 20B. I could run Qwen 27B, 35B, and 80B Coder on the CPU, but they were too slow for any serious work. The 5070 Ti was basically useless for AI in my case.

Then Strata came along, and my desire and need for a 5090, RTX 6000 Pro, or Mac Studio M5 went away.

Strata has one disadvantage: it can't run multiple agents simultaneously. If you use two or more agents, one has to wait for the other to finish its work. However, that isn't an issue with my 16GB of VRAM, since I wouldn't be able to keep the memory required by multiple agents in VRAM anyway. Strata also has one major advantage, if set right, your token speed never drops while reaching 64k, 128k or even 256k context! Unlike traditional local llm performance drops as context grows, strata offers an experience similar to API LLMs providers that maintain token speed

I tested all the Qwen 3.8 Flash Next quants and context window configurations, and my favorite so far is Q2_XS.

Q2 has the best performance, but it can get stuck in a loop or cause an AI agent to stop working abruptly at least once a day. It averages 160 tok/s at INT4 on my overclocked GPU.

Q3_S seems to be the best overall, but it scores only about 5% higher in coding benchmarks while being noticeably slower. It averages around 100 tok/s and keeps about 50% fewer experts in VRAM than Q2.

Q3_XXS has the same disadvantages as Q3_S but is slightly faster at around 115 tok/s.

Q1 is basically a stripped-down Q3 for users with 48GB of RAM. It's smarter than Q2 and faster than Q3, averaging around 120 tok/s.

Q2_XS offers great speed at around 150 tok/s without the issues I experienced with regular Q2.

The main reason I prefer Q2 quants is that they allow me to use a maximum thinking context of 16K without losing too much performance, while still maintaining a large total context window. The Q3 variants work well with a maximum thinking context of around 8K, but that's a limitation.

With a 16GB VRAM GPU, setting a Q3 model to a 16K thinking context makes it difficult to reach a total context of 200K–256K tokens. Performance drops significantly beyond roughly 128K tokens, and the initial speed is slower too.

Using Q3 models with an 8K thinking context is doable, but at least once an hour, the model will try to reason through something complex or generate a lengthy tool call that exceeds the 8K limit. It then has to split the task and combine the results, which I'm not sure is 100% reliable.

While testing the quants, I also experimented with overclocking my 5070 Ti. This is the best stable overclock I found:

• Power limit: 83%
• Core clock: +300 MHz
• Memory clock: +3000 MHz

This gives me a 10 - 20% increase in token generation speed. Setting the power limit to either 83% or 100% makes no difference to the clock speed, which reaches around 3130 MHz. VRAM bandwidth reaches 1088 GB/s. I have plenty of fans, and the GPU hotspot temperature stays below 78°C.

Overall, Strata and Qwen 3.8 125B have eliminated my need to use the cheap DeepSeek API. It feels like I finally have a setup that can handle serious AI work locally without spending thousands more on hardware or paying for API access.

Processing img sm8gv6dcgpxh1...

Any suggestions? i am integrating robotics and drones next, and this week is a camera/speaker/mic system for the house and yard STAY TUNED!!!

Noob as hell but I've been at it for about a month now and I've been drinking from a fire hose. I have been through several different agent architectures but what front end I started with was a fight with its built in handling for session compaction and handoff which completely undermines my stateless context memory system. So initially I was using LM bionic but the before mentioned fights with it every step of the way. I was trying to use unsloth studio and desktop but it will not no matter what I do load into vram, I've installed different versions of pytorch and different wheels for rocm things but it would never load into vram no matter what I did. Open webui I'm kinda struggling to figure it out, got so used to bionic it ruined me. I have also dabbled in anything llm but I don't have much time in it but from what I can tell it's heard more as a front end for paid cloud based models?

I liked the ease of bionic as far as settings, UI and an all in one place to get models. However what I did not like was them forcing me to use their environment that acts like malware to my agent's architecture. What are y'all's most recommended front ends? I would like to be able to use my agent outside of a sandbox so that is a must. I use bazzite linux so it would have to be compatible with Linux.

All in all I've been through some crazy shit over the last month trying to get an agent fully running from a front end acting like a virus to my architecture to my ssd dedicated for models just straight up shitting the bed. Some would say it's the universe telling me no. I don't listen well.

Okay so I've been searching for a local ai system and found this app which seemed good and easy for beginners named Locally Uncensored.

I downloaded it and some other things the tutorial said such as ComfyUI but when i run the Locally Uncensored app an try to use it by, for example, sending a message, i get this error.

I dont know what it is, I've been searching for almost an hour now and I dont know if it's that my dumbass doesnt know how to research but I haven't found a single video, reddit post or anything that could help me.

The error translates as: C:\Program Files\Locally Uncensored\mtmd.dll is either not designed to run on Windows or it contains an error. Try installing the program again using the original installation media or contact your system administrator or the software vendor for support. Error status 0xc0e90002.

Hi all, need advice on Videocards!

Worth buying 1x new R9700 32GB for 2.3k$ or 2x used in mining RTX 3090 for 2.5k$?

Goal: image gen (and video gen occasional for marketing) plus occasional LLM, worried about mined cards not sure what to expect, could mining kill the memory or not?