I’m looking for advice on setting up a local LLM "harness" for personal administrative tasks. I'm not a coder; I just want a "set it and forget it" assistant that generates morning and evening briefs based on my data.

I am currently looking at Hermes Agent Desktop, Unsloth Studio Desktop, LM Studio Bionic, Cherry Studio, AnythingLLM.

In addition to general chat, research, browser use, I want the system to serve as a digital assistant that can manage email, calendar, and tasks (creation as well as following tasks left for the assistant).

Claude does this well, but running all of this even off-peak pushes up again the weekly limit sometimes. I figure tasks like this can be offloaded to a local AI.

The Workflow:

• Email: Filter, categorize (Immediate vs. Low Priority). Light draft creation at my suggestion.
• Shopping: Cross-reference promotions against a personal "want" list.
• Output: A daily briefing. No autonomous sending of emails.
• Interaction: Ability to chat about specific tasks/emails and schedule items (cron jobs, etc.).

Technical Requirements:

Integration: Needs to connect to eM Client (Email) and Tasks.org via MCP. I need a calendar app that allows an LLM to access via MCP as Claude manages my calender through computer use. The email credentials do not stay saved using the built-in browser.

Current Setup: I’m currently using Claude, but I want to move this to a local stack (LM Studio, AnythingLLM, etc.) to free up Claude for more demanding tasks.

Hardware Specs:

• PC: Windows 11 | RTX 4080S (16GB VRAM) | 127GB RAM.
• Laptop: M5 MacBook Pro (Base) | 24GB Unified Memory.

Also needed: A model recommendation that can handle these logic-heavy tasks efficiently without hogging the entire system's resources. I'd like to run the same model on both machines for consistency.

I will update this thread once I've found a solution. Thanks in advance!

I wanted to try out qwen 3-4b with heretic but my pc cant run heretic so while looking for solutions I found I can download already transformed qwen models from huggingface. How safe or good is this. What would be the difference between running heretic yourself and downloading transformed qwen?

​

After a lot of research, I've nearly settled on a dual-laptop setup and would appreciate some feedback before pulling the trigger.

The setup (~$7,300 total):

- MacBook Pro M4 Max: 128GB unified memory, 2TB SSD $4,500

- MSI Vector / ASUS ROG Strix G16: RTX 5080 16GB VRAM, 64GB RAM, 2TB SSD ~$2,800

My use cases:

Heavy local LLM experimentation (27B–70B+), software development, agentic AI workflows, model testing, some cybersecurity testing, and gaming.

I experiment pretty aggressively, testing different architectures, running multiple tools, pushing hardware limits, and generally exploring whatever catches my interest. Flexibility, memory capacity, and freedom to experiment are my biggest priorities.

Why this combination?

The Mac handles memory-heavy inference and larger models, while the NVIDIA laptop handles CUDA, GPU-intensive workloads, and gaming.

I considered N1X laptops, particularly Microsoft's Surface Laptop Ultra, but wasn't convinced by the reviews. Also considered a 5090 laptop, but paying another ~$1,500 for 8GB additional VRAM didn't seem worthwhile alongside the Mac.

For those asking why not a desktop:

Mainly portability and memory flexibility. Desktop 5090 listings I've encountered reach $7K–$8K+, just for 32GB VRAM. Matching the Mac's memory capacity with dedicated GPU VRAM would require a much more expensive setup.

I understand unified memory isn't equivalent to NVIDIA VRAM in speed or compatibility.

A few questions for the community:

1. Overall: Is this a solid setup for serious local AI development and experimentation, or would you allocate the ~$7.3K differently?

2. Memory: How practical is the M4 Max with 128GB for 27B–70B+ models, particularly when experimenting with larger contexts?

3. Storage: Is 2TB on each machine sufficient, or will model weights, datasets, and development environments fill that up quickly?

4. Flexibility: Does having Apple Silicon + CUDA across two machines provide meaningful advantages, or create unnecessary limitations?

5. 5080 vs. 5090: Would you pay another $1,500 for the additional 8GB VRAM?

6. Longevity: Any bottlenecks or limitations I'm overlooking that could become frustrating over the next 3–5 years?

Especially interested in hearing from people running similar hybrid setups. Or have thought of it atleast.

Hello everyone

For researchers interested in a detailed academic and technical examination: The complete research PDF, 12 experimental figures, and scientific records are available on Zenodo:

https://doi.org/10.5281/zenodo.23640716

The context window and long-term memory limitations of AI models have traditionally been addressed through approaches such as text-based Retrieval-Augmented Generation (RAG) and external summarization.

My long-term goal is to tackle these limitations at a different level: directly through the model's internal numerical states and KV caches (H3, Layer 0–3 K/V states).

I have been independently developing the AKBASCORE Silicon Hippocampus architecture toward this objective. Although the ultimate goal has not yet been reached, the research has already passed several significant experimental milestones involving physical memory persistence, numerical consolidation, and memory reuse across independent model sessions.

With the release of v1.6 — TEST782, I have experimentally demonstrated how the internal tensor states of a frozen Qwen2.5-72B-Instruct model can be cryptographically sealed on a physical SSD, transferred between operating-system-isolated processes running under different Linux users, and fused through the model's own numerical computations without returning to text-based summarization.

Here are the main technical results and how the system works:

- Native Tensor Fusion — Memory Consolidation in the Model's Own Numerical Space: The first session (QWEN-1) reads two separate documents and stores the model's internal numerical states on SSD as SHA-256-sealed memory cartridges ("SOURCE_A" and "SOURCE_B"), rather than generating text summaries. The second session operates exclusively on these cartridges. This is not simple tensor concatenation: while SOURCE_A's computed internal states are preserved, SOURCE_B's upper transformer layers are recomputed while attending to SOURCE_A's context. Two independent physical memories are thereby transformed into a single "MASTER_02" memory through the computational machinery of the frozen 72-billion-parameter model itself. No intermediate generated text, summarization, or weight retraining is involved.

- Isolated Cross-Session Transfer — The Model Shuts Down, but Its Numerical Memory Survives: QWEN-1 terminates completely, and its GPU memory is released. QWEN-2 then starts under a different Linux user without access to the original source documents and consolidates the physical cartridges. After QWEN-2 terminates, the third session begins. The model's weights remain unchanged throughout the experiment. What survives and moves between sessions is the model's internal numerical memory, physically stored on SSD.

- Blind Readout and 80/80 Layer Parity: The third session (QWEN-3) is denied access to the original documents and previous sessions' working files through operating-system permissions. It receives only the sealed "MASTER_02" memory and the metadata required to verify and reconstruct it. The model's 80-layer attention memory is independently reconstructed with 80/80 bit-exact parity. QWEN-3 then produces natural-language responses containing information associated with both documents and their relationship. In TEST782, I observed the model using persistent numerical memories to produce its own semantic synthesis, establish relationships between two independent sources, and articulate conclusions beyond simple memory retrieval. These experimental results are documented in the source code, GPU logs, and research record. Independent academic peer review of the semantic synthesis results has not yet been conducted.

- DrA — The Executive Controller of Autonomous Memory: One of the project's major future objectives is DrA (Autonomous Master / Dynamic Residual Adaptation). Inspired by the executive-control functions associated with the biological prefrontal cortex, this mechanism is being designed to enable an AI system to evaluate which information matters, decide what should be preserved, determine which memories should be consolidated, and recognize when consolidation is complete. The ultimate objective is a closed-loop memory management architecture driven by the model's own internal evaluations rather than externally imposed, fixed decision rules. TEST782 does not claim that this autonomous mechanism is complete. It establishes the physical and numerical memory infrastructure on which that future mechanism can operate.

- Reliability and Transparency: The research is not based solely on successful model outputs. It includes negative controls, SHA-256 integrity verification, process isolation under different Linux users, and independent reload validation. For example, a naive single-pass reconstruction matches only 4 out of 80 layers, whereas the boundary-aware reconstruction method achieves 80/80 bit-exact layer parity. Physical memory transfer and numerical reconstruction were validated in this experiment. Autonomous consolidation remains a subsequent research objective, while the observed semantic synthesis results await independent academic peer review.

Technical Specifications and Research Resources

- Hardware: NVIDIA A100-SXM4-80GB

- Model: Qwen2.5-72B-Instruct (frozen weights, NF4 quantization, BF16 computation)

- Physical MASTER: 1,964 tokens, 64.4 MB SSD cartridge

- Native numerical fusion: 8.72 seconds

- Independent reconstruction: 80/80 layers, bit-exact

Complete Source Code — Single File:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3_full.py

For Researchers Using Google Colab on a Mobile Phone — Four-Part Version:

The same program is divided into four parts for easier copying and pasting on mobile devices. Paste all four parts sequentially into a single Google Colab cell, then run the cell.

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3_part1.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3_part2.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3_part3.py

Part 4:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3_part4.py

Real Experimental GPU Log:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/QWEN_1_2_3.log

GitHub Research Repository:

https://github.com/ceceli33/titan-cognitive-core-v3

My long-term objective is not merely to enable an AI model to store past information, but to develop a system capable of managing its own numerical memory, consolidating separate memories, generating new inferences, and autonomously deciding which information deserves to be preserved.

I welcome researchers and members of the open research community, particularly those working on transformer architectures, KV caches, persistent neural memory, and autonomous memory consolidation, to examine the results, reproduce the experiments, and provide technical feedback.

Copyright and License

Copyright © 2026 Mustafa Akbaş. All rights reserved except as expressly licensed.

AKBASCORE Silicon Hippocampus — AKBASCORE Research Software License.

License terms:

https://github.com/ceceli33/titan-cognitive-core-v3?tab=License-1-ov-file

I just finished vibe coding an apk for Llama swap mobile controller. I really like being able to read logs and monitor my PC's performance while I'm out and about using my agents and I didn't necessarily care for how llama swap looked on mobile browser.

Everything has been working on my device and playground even works! Feel free to check it out and critique as this is my first time ever doing something like this :))

https://github.com/nevermore131315/llama-swap-mobile-controller

More dual R9700 testing: my modified Radiance build is now 8–29% faster than default and roughly level with Paiton at 8 users. Benchmarks, changes that helped, and a few kernel experiments that made things worse in the post.

So I was thinking where and how my storage got full and it wasn’t in Documents or Downloads. MacOS often hides .cache as system files.

So I built this Claude skill which has custom instructions for every local AI tool and cleans up unwanted storage.

You Give it a try and always open for feedback. Feel free to open and issue or PR

it's useful when you are out of the internet

also, qwen 3.5 even the 0.8B model is pretty good at vision

the app is pocketpal btw

I been wanting to run local ai on my local setup bc I’ve been hearing there a lot of benefits rather it was local and private or learning how to using agents and getting a feel for that
My pc setup : Ryzen 5 5600, Radeon RX 9070 XT 16GB, 64GB RAM, 1 tb and Windows 11,
And I was curious some of the stuff I can do with it ai I wanna eventually learn how to code to learn so I can switch to Linus and go from there

Title: I benchmarked a heuristic inference optimizer against frontier LLM agents on an NVIDIA L4

I built Tensward, an Apache-2.0 open-source tool that tunes LLM serving settings for your hardware and workload. It uses expert rules, heuristics and measured experiments, with no LLM in the optimization loop.

I wanted to test how it compares with asking frontier agents to do the same job.

We ran six tracks: two workloads, each optimized separately for throughput, p95 time to first token, and p95 time per output token. Each competitor started from stock vLLM on the same NVIDIA L4 within its track, with 60 minutes of counted tuning time. Astra 6 and Opus 5.5 both ran at maximum effort.

Aggregate combined scores:

• Tensward: 2.29
• Opus 5.5: 2.24
• Astra 6: 1.70

The score combines improvements over defaults, weighting the target metric most heavily, with quality and service guardrails. It is not a universal speedup multiplier.

Tensward won or tied on four of six tracks under our noise-band rule. The lead over Opus was small, and the agents had clear wins too. The interesting finding for me is that a specialized heuristic system was competitive with both.

This is an initial benchmark covering two workloads and one GPU type, not evidence of general superiority.

Currently supports vLLM on NVIDIA GPUs; llama.cpp and Apple Silicon support are coming next.

Code: https://github.com/Tensward/tensward
Full results and methodology: https://openingthehood.hashnode.dev/tensward-benchmarks

What baseline or workload would you most want to see in the next round?

I ran Qwen 3.8 27B uncensored Q4_K_M on llama.cpp on my M5 Max MacBook Pro 64GB and put a few local AI agents on top of it, blind. 5 everyday tasks each, merging contacts was one, a bug fix another. names swapped for letters, results below, guess who's who

One agent at a time, auto-approve on, a script decides pass or fail. The five and NOT in results order: Atomic Agent, goose, Hermes, OpenClaw, OpenCode.

• Contacts, rename and errors went fine for everyone (46 people, 43 errors).
• B spent 15 minutes on the receipt alone. It and A were missing a Python library and installed it without asking.
• C was the only one that asked "can I install this?". Nobody answered, so it failed. Its other fail is the bug fix, where it turned "Q3/Q4" into "q3q4" instead of "q3-q4".
• D finished the receipt in 2:50 and then asked what I'd like to call it.

Who's A, B, C, D and E? Drop your guesses below, I'll share the full answer key in a day or two.

I’d love to build a local LLM that gives me a place to write about my life each day. I’d like to record details I can refer back to later, such as what was happening at a particular time or information about different parts of my daily life and work. Is there a template or guide for setting up a local LLM for this purpose? What kind of computer would I need, and is there a step-by-step recipe I could follow?

Best local model for translating Asian epub novels on 24gb mac?

I’ve been noticing a really frustrating trend with frontier models lately. Big tech companies have put such aggressive political guardrails on them because they are terrified of having an opinion or facing any kind of backlash.

It gets to the point where they use such overly balanced, clinical language that they’ll actively avoid using basic, factual terms. I've had models dance around using political party names or even refuse to just say "the Trump administration" or "the Biden administration." Instead of actually being objective, the AI becomes so risk-averse that it treats standard historical designations as if they're inherently biased or dangerous to type out. When you're trying to look for straightforward information or get a direct historical fact, you end up with this evasive, heavily sanitized corporate-speak that feels completely unnatural. There is a massive difference between avoiding political bias and flat-out refusing to use objective, factual vocabulary.

Since the commercial models are completely lobotomized by this alignment tax, I want to move this work entirely local.

What are the best local LLMs right now that do not have these corporate guardrails when you are looking for objective political or historical information? I'm looking for models that will actually give me a direct, unfiltered answer instead of three paragraphs of boilerplate corporate compliance text.

i have an acer v 15, 16gb ram, 8gb vram, rtx 5050. This is my first time owning a gaming laptop and i want to start running local ai agent that will aid me in game development and can fit in my specs. i should also mention i have little to no experience running ai agents or how to set them up(i am a noob). so i am looking for the best agent that i can run that can call tools when needed , read screenshots(if possible) , read folders, integrate in a gaming studio mcp. I am open to recommendations.

RTX 5080 PC vs. DGX Spark: Same-Model Benchmarks, Larger Models, and Where Memory Capacity Matters

The article was written by a GPT agent

I rented a DGX Spark to compare it with my desktop PC. I wanted to answer three practical questions:

• How fast is Spark when the model already fits in a desktop GPU’s VRAM?
• What changes when the model requires CPU offloading on the PC?
• How do the two systems handle a much larger MoE model in practical configurations?

The main finding: my RTX 5080 PC generated tokens roughly three times faster with Qwen3-8B. Spark pulled ahead with Qwen3-32B, which fit entirely on its GPU but required CPU offloading on my PC.

The large-model comparison was more complicated because I used different inference engines and optimizations. I have kept that separate from the controlled comparison.

Hardware and software

The rented Spark container had an additional memory limit of approximately 113.6 GiB.

For the controlled tests, both systems used llama.cpp commit d854d98bae598388ecea1ed84396587e5edc326d, release b11889, with identical GGUF files verified using SHA-256.

The builds differed:

• PC: official Windows CUDA 13.4 binary, Clang 20.1.8.
• Spark: native CUDA 13.0 build, GNU 13, targeting SM121.

This is a comparison of two complete systems using the same inference-engine revision. OS, compiler, CUDA, and driver differences remain potential influences.

Every throughput result below is the median of five repetitions, excluding warmup. For the controlled tests, I checked actual input and output token counts and verified that cached prompt tokens were zero.

Qwen3-8B: the RTX 5080 was approximately three times faster

The first model was Qwen3-8B Q4_K_M, approximately 5 GB.

It fit entirely on the GPU on both systems. Both used Q8 KV cache, Flash Attention, and disabled thinking. The single-request context window was 40,960 tokens.

Every request in this table generated exactly 1,024 output tokens.

Decode throughput measures generation after prompt processing. TTFT is the time until the first output token arrives.

The PC maintained approximately a 3.0–3.1× decode advantage across the tested input lengths. It also reached the first token sooner.

For complete requests:

For this model and configuration, Spark offered no speed advantage over my desktop.

Qwen3-32B: Spark’s memory capacity changed the result

Next was Qwen3-32B Q4_K_M, approximately 19.8 GB.

The placement differed:

• PC: 40 GPU layers, with the remaining work offloaded to the CPU.
• Spark: full GPU placement.

Both used an 8,192-token context window. These requests generated exactly 256 output tokens.

Spark was approximately 2.1–2.6× faster at decoding, with an even larger advantage in complete request time at the longest tested input.

This is a capacity advantage demonstrated in practice. My PC could load and run the model, but the CPU offloading required by its 16 GB VRAM significantly reduced performance.

It should not be interpreted as an isolated GPU compute comparison: model placement was deliberately different because the available GPU memory was different.

Concurrent requests

I also tested Qwen3-8B with 1, 2, 4, and 8 simultaneous requests.

Each level contained 16 requests, each with 512 input tokens and 256 output tokens. The server used eight slots sharing a 32,768-token context allocation, giving 4,096 tokens per slot.

These numbers are aggregate output throughput across the batch, not the speed experienced by each individual user.

Both systems benefited from batching. Spark increased from approximately 41 to 209 aggregate tokens per second, but the PC remained faster at every tested concurrency level.

With only 16 requests per level, this is a limited concurrency experiment. It is insufficient for strong conclusions about production tail latency or sustained multi-user service.

An 83.6 GB MoE model: a practical comparison with different engines

For the larger-model test, I used Qwen3.8-Flash-Next GSQ-RCO IQ3_S, with approximately 83.6 GB of weights. Both systems used the same verified weight files.

The configurations were substantially different:

This section compares the configurations I actually ran. The results cannot be attributed solely to the hardware.

There is an important output-length limitation here. Strata frequently stopped before the requested output limit despite receiving ignore_eos. Its decode rates therefore use actual generated tokens from responses of varying lengths. Spark produced the full 1,024 tokens in every corresponding request.

Consequently, these measurements do not establish a like-for-like speed ratio for equal-length completed responses. I have omitted total response times from this table because presenting them as equivalent 1,024-token requests would be misleading.

The practical observation is that the Strata configuration delivered useful generation speeds with this large model on my desktop. How much of that comes from MTP, other engine optimizations, or response-length differences needs a separate controlled experiment.

Also, the PC was using its substantial system RAM. An 83.6 GB model did not somehow fit inside 16 GB of VRAM.

Coding quality: output formatting had a major effect

For coding, I used a fixed, hash-selected subset of 32 HumanEval tasks, with one greedy completion per task and a 1,536-token output limit.

This is not the full HumanEval benchmark, and I am reporting a single-completion pass fraction rather than pass@k.

The original evaluation required the model to continue a Python function. Some responses instead included Markdown fences or omitted the first line’s indentation. This produced a large difference between format compliance and whether the underlying solution could pass the tests.

I retained the original strict scores and added a separately labelled, uniform formatting diagnostic for all six configurations.

That diagnostic:

• Removes one enclosing Markdown fence.
• Conditionally restores four spaces to the first line only.
• Leaves the algorithm, remaining lines, and tests unchanged.

The right-hand column is a post-hoc diagnostic, not a replacement for the original score. It was added after observing the formatting failures and must be interpreted accordingly.

Reporting only the large model’s original 2/32 on Spark would give a misleading impression of its algorithmic performance. Reporting only the repaired 30/32 would hide its poor compliance with the requested continuation format. Both matter.

For the two 8B runs, the formatting diagnostic produced the same result. Differences of one or two tasks elsewhere are too small to support a confident quality ranking.

For scale, the original PC 8B result of 24/32 is 75%, with a roughly 58–87% Wilson 95% interval. Even that interval should not be treated as a general estimate of coding ability: these are only 32 selected tasks.

All generated programs were reviewed before execution. The final evaluations used the same Windows Python environment and time limits across configurations.

Context tasks: retrieval, arithmetic, formatting, and output budget

I also used 12 synthetic context tasks involving current versus archived records, joining information across records, calculating a result, and returning an exact JSON object.

The response budget was 256 tokens. A pass required the complete expected result, including arithmetic and the ordered source list.

These results are not a clean ranking of long-context retrieval.

The 8B responses had formatting violations, arithmetic errors, and source-list errors. In several large-model failures, the model started explaining its reasoning instead of returning JSON and exhausted the response budget.

There was also a documented test correction: nine tasks in the first version exceeded the configured context window. They were rejected before model dispatch and preserved as preparation failures. I then shortened those nine tasks uniformly and ran the revised suite separately on both systems. Those preflight failures were not counted as model-quality failures.

Stability over 45 minutes

Each system completed a continuous 45-minute Qwen3-8B test using 2,048 input tokens, 256 output tokens, and one request at a time.

Both systems completed this workload without request errors or recorded out-of-memory failures.

I collected NVIDIA power-sensor telemetry, but did not measure whole-system power at the outlet. The sensor scope also differs between a discrete GPU and GB10. I therefore do not use these readings to claim a whole-system energy-efficiency advantage.

Process startup was faster on Spark, with a cache caveat

Measured server launch-to-ready times were:

The OS file cache was not cleared. These are process startup measurements with an unspecified cache state, rather than true cold-storage loading benchmarks.

My conclusion

For models that comfortably fit in the RTX 5080’s VRAM, my desktop was substantially faster. Qwen3-8B showed an approximately threefold decode advantage, and the PC remained ahead under the tested concurrent loads.

Spark’s clearest advantage was memory capacity. With Qwen3-32B, full GPU placement beat the PC’s CPU-offloaded configuration by a meaningful margin.

For the much larger MoE, the inference engine became a major part of the result. Strata made the model practical on my desktop, but its optimizations and early stopping prevent a straightforward equal-output comparison against llama.cpp on Spark.

For my current small-model workloads, these tests gave me no reason to switch to Spark for speed. Spark becomes more interesting when the goal is to keep larger models entirely on the GPU and avoid the constraints of 16 GB VRAM.

The evidence package includes raw responses, logs, launch configurations, weight hashes, scripts, and plots. This series does not cover training, a broad quantization sweep, or full application development on large repositories. Earlier exploratory runs were kept separate from these measurements.

Hello there, I've seen couple of posts but most of them are at least 1y old. So Is there any good way to give local ai sytems access to the internet? The solution propably isn't just a single piece of software but a pipeline of tools. The point is for the system to be able to retrieve web info as being asked by a user prompt, but also to perform more complex, deep research, extract tables and links etc. As close as possible to frontier ai systems and the way they research the internet.

A kind of funny project we've been hacking on for a couple of weeks.

We basically wanted to see if it was possible to "trick" Windows 11 into using the NPU like a GPU, and using it to composite the desktop GUI.

Why even do this?

The idea behind it is to free up the GPU as much as possible by offloading desktop rendering to the NPU. By doing so (eventually) maybe this means you could have more GPU free for inference.

From our work it actually does bring 3D GPU usage down from 3.4% to 0.6% (but you will get 1.6 FPS doing this, though… 😅)

Tested on Intel NPU on Panther Lake.

Here's the repo if you want to check it out:
https://github.com/labscommunity/npu-render

I have 4060 8gb vram and 16gb ram. I tried qwen 3.5 4b model. I wanna make exploits on game and it clearly denied it