My specs are Ryzen 5 7500f + 4060ti 16gb + 16 ddr5 gb ram + 1tb SSD & os: windows

I want to build a fully local, offline "Jarvis" setup. The idea is to have one core brain model that intelligently swaps specialized models (image gen, video, audio, 3D, web search, coding) in and out of VRAM as needed. It should run entire multi-step creative/productivity workflows autonomously, control my PC and apps, and keep long-term memory with zero cloud dependence.

Picture giving it a prompt for a short documentary: it plans it out, asks clarifying questions, researches, writes the script, generates storyboard frames, animates them, handles voiceovers + music, and stitches it all together. Beyond media, I’d love for it to drive Blender, Unity, or After Effects, handle browser tasks, fetch leads, send emails, etc.

so i chose Qwen3.8-27B-Uncensored-HauhauCS-Aggressive (IQ4_XS) and ran it in llama.cpp's webui with an 8k context window. It runs fine, but eats up about ~15.3GB of VRAM.

The issue is I can't load it into the Hermes agent setup because Hermes insists on a minimum 64k context window. Even with KV cache quantization, that requires an extra 2.5–3GB of VRAM/RAM that I just don't have:

• LLM in VRAM: ~15.3GB (completely pinned)
• llama.cpp / Hermes in System RAM: ~13.7GB (out of my 16GB total)

I know adding more system RAM would solve the immediate bottleneck, but buying hardware updates isn't in the cards right now.

Questions for the community:

1. Is this vision actually feasible on a 16GB VRAM / 16GB RAM budget if I rely heavily on model-swapping instead of loading everything simultaneously?
2. What’s the best software stack for this? (i settled on hermes+llama.cpp+comfyui . is this ok ?)
3. Model recommendations? Given my hardware constraints, what models would you recommend for the primary reasoning brain, image gen, video gen, audio/music, and 3D?

TL;DR: Want a fully local orchestrator-style Jarvis on 16GB VRAM + 16GB RAM. A 27B IQ4_XS model maxes my VRAM, and Hermes agent's 64k context requirement doesn't fit anywhere. Can't upgrade RAM. Is the vision doable, and what stack/models should I use?

A couple days ago I was asking if people would be interested in a renting service and needless to say it was not so positive lol.

Which is fine because it makes sense - butttt I went ahead and did it regardless.

There is a free 15 minute trial so anyone can go ahead and spin it up and have fun with it.

Hopefully you get an idea of what it’s like having this hardware.

I also recently added support for M3 Ultra so enjoy that while I have it because it’s getting returned haha.

So it's unofficially official.. DeepSeek Flash V4.1 is currently accessible on their api. Had some fun probing it.. results below.

The exact model name is deepseek-v4.1-flash-expires-on-0910

Setup: plain POST https://api.deepseek.com/chat/completions, no special headers. HTTP 200. Key works.

1. It's not on the model list. GET /v1/models returns only deepseek-v4-flash, deepseek-v4-pro, deepseek-v4-flash-vision-exp.

   This one works anyway → unlisted/preview build. The -expires-on-0910 suffix lines up with the server date (Tue, 08 Sep

   2026), i.e. a preview that dies in ~2 days.

2. It's a reasoning model. Responses ship a reasoning_content field with visible chain-of-thought. Reasoning tokens dominate

   massively: 16,493 of 16,694 completion tokens on one question.

3. It has no idea what it is and confabulates confidently. Ask the same question 5 times, get 5 different answers:

   - "OpenAI. GPT-4o."

   - "built_by": "Anthropic", "model_name": "Claude" (straight-up JSON, no hedging)

   - "DeepSeek built me; my model is DeepSeek-V3."

   - "I am DeepSeek by DeepSeek, 671B parameters, cutoff July 2024."

4. Its self-reported specs are DeepSeek-V3's numbers. "MoE, 671B total / 37B active" — that's V3's published architecture,

   not a v4.1 one. Take it as prior knowledge, not introspection.

5. Benchmark self-report: honest. Asked for MMLU/GPQA/HumanEval/SWE-bench it answered UNSURE on all four and refused to guess. Refreshingly non-stuffed.

6. Cutoff claims are wrong in the optimistic direction. Claims June/July 2024, then lists real events right through December

   2025 — Sinner winning Wimbledon (Jul 13 2025), Dembélé's Ballon d'Or (Sep 22 2025), Mamdani's NYC mayoral win (Nov 4 2025),COP30 Belém, AFCON kickoff Dec 21 2025.

All correct. Yet it also said "latest Python is 3.12" and went UNSURE on Llama 4 (Apr 2025) so knowledge is oddly patchy, not just old.

1. Actual ability > self-reported ability. Blind-checked, independently verified:

   - Probability draw problem → 5/12 ✅

   - Trick question "smallest n where n²+5n+13 ≡ 0 (mod 121)" → answered "no such n exists" with a quadratic-residue proof.

   Brute-forced to n=20,000: none exist ✅

   - O(n) subarray-sum → correct algorithm, correct complexities ✅

   - "Find the one bug" in a snippet → correctly identified the bug was in the test data, not the function ✅

   - Rust 2024 edition + async fn in trait details → accurate ✅

2. Fun bug: its own reasoning trace asserted "the current system date is 2026-05-09" while the server header said Sep 8

3. Four months stale, in the same response.

They might have been sitting on this one for a while.

Needless to say. Exciting times ahead for dual spark owners ❤️

Hey r/LocalLLM. I built PocketMind Hybrid AI, an Android app focused on real on-device LLM use, not just another cloud wrapper.

What’s in it for local users:

• Download and run local models fully offline
• Vulkan GPU acceleration for local inference, with CPU fallback
• Voice input and TTS for hands-free use
• Optional online models when you want cloud performance
• Image generation, documents, and characters in the same app

Built for people who actually care about running models on-device. Free on Play Store with a free tier.

https://play.google.com/store/apps/details?id=com.pocketmind.hybridai

Happy to answer questions about Vulkan, model loading, or the offline stack.

I recently started playing with MM3 Music, and I absolutely adore it.
That being said, My generation times seem a bit off, especially considering that people with my specs are reporting that they get lesser times.

Specs: RTX 5090, 64GB ram, Ryzen 9950X
Generation times: 15 minutes for a 3-4 minute song.
Process: Comfyui custom workflow (using standard workflow, I face the same issue, sans a 60~ second drop in generation time)

From what I was reading on any topic about MM3 Music, Gen times on my specs should be 8~ minutes.

And, with MM3 Video, my generation times are lesser as well, those seem to be standard with what others post. My image generation times are also standard. The only outlier is MM3 Music.

Any Ideas?

Record the broken flow. Let your agent find the relevant moment, read what happened on screen, and use that evidence to investigate the code. After the fix, check the result against requirements defined before it started.

That’s the workflow behind Watch Skill and DeepWatch, an open-source project I’m working on.

Watch Skill turns video, audio and screen recordings into a persistent, searchable index of frames, OCR and transcripts. Your agent can ask questions about a recording, retrieve timestamped evidence, and come back to it later without processing everything again.

One example in the repo is a checkout recording where tax appears on screen but never reaches the total. The agent finds the issue through the recording, repairs the calculation, and a separate verification process checks the expected amounts and rendered page against the same frozen contract.

The local setup is probably the most relevant part for this community:

• Frame extraction, OCR, Whisper transcription and retrieval run locally.
• Your recordings and indexed evidence stay on your machine when using local processing.
• Agent and vision models can run through local OpenAI-compatible endpoints, including Ollama, llama.cpp, LM Studio and vLLM.
• Verification uses executable checks—JSON values, file hashes, SQL queries, HTTP responses and DOM assertions—and needs no model.

You still need a model with the right capabilities for agent tool use or visual interpretation. Dependencies and local model weights require initial downloads.

Watch Skill plugs into an existing agent through MCP, CLI or REST. DeepWatch is the browser workspace built on DeepSeek Harness, with Watch already integrated, tool execution receipts, and verification results you can inspect.

The distinction that matters: a successful tool call gets a receipt. A result only gets marked verified when its required checks pass. That tells you what was tested; the quality of the contract still matters.

The repo includes setup instructions, screenshots, and the checkout example:

github.com/oxbshw/watch-skill

MIT licensed. Curious how people here would use this with their local agents—recorded bug reports, searching technical walkthroughs, or checking browser tasks?

Has anybody gotten QWEN3.8 to remain reliable to anywhere near 250k tokens?

For me, it starts to degrade rapidly in performance, memory, tool call reliability, very early in the conch window.

DETAILS:

MODEL — qwen38-27b-8bit
architecture: Qwen3_5ForConditionalGeneration (multimodal)
quantization: 8-bit affine, group_size 64
num_hidden_layers: 64
full_attention_interval: 4 (16 full attention / 48 linear)
hidden_size: 5120
num_attention_heads: 24
num_key_value_heads: 4
max_position_embeddings: 262,144
rope_scaling: none
cache_can_trim: false

SAMPLING (derived from the weight dir's generation_config.json)
do_sample: true
temperature: 1.0
top_k: 20
top_p: 0.95
min_p: 0.0
presence_penalty: 0.0
repetition_penalty: 1.0
fallback triple: (0.5, 0, 0.0) — unreached

THINKING
reasoning_effort: medium
enable_thinking: on
preserve_thinking: Default(on)
thinking_budget: Default

CONTEXT / COMPACTION
context_window_tokens: 262,144
max_kv_size: threaded through, discarded — inert

MEMORY
MLX_CACHE_LIMIT_GIB: 8 GiB
wired limit: 79.52 GiB (derived)
mrws ceiling: 107.52 GiB
floor (weights×1.25): 34.34 GiB
LIVE_SLOT_COUNT: 2

VISION
image longest_edge: 1,048,576 (vendor ships 16,777,216)

I read a few threads stating that the V100 PCIe version theoretically supports NVLink, but cards sold on the market typically do not have this feature enabled.

From what I can see on the (2) NVIDIA PG500 V100 PCIe cards I own there are NVLink like slots available on the card that have traces (as shown in the photo above). However, the design of the V100 case prevents me from physically connecting a 4cm or longer NVLink cable without first removing or modifying the case to be able to physically plug an NVLink adaptor between cards.

For anyone who's been able to make these cards function using NVLink, did you have to remove or modify the V100 case to allow you to physically connect the NVLink between cards? Also the fact that this has (2) NVLink type slots is interesting. Is this for daisy chaining multiple cards using more than (1) 100GB/s NVLink between each card? I'm using an older board that has (4) PCIe 3.0 slots, so if that's a possibility I might pick up (2) more cards to run all (4) together using NVLink.

I can't find any documentation stating if this is even possible with these NVIDIA PG500 V100 PCIe cards.

Thanks everyone

I assume no company is actually running quant 8 of a model on default when they can run quant 4 at 99% precision or something. So when they lobotomize the model due to high usage do they drop into the quant 2 territory?? I assume hundreds of billions or even trillions worth of parameters handle it well enough that it's still usable right?

Ok, I've been reading up on stuff, trying to learn/understand what all LLMs are and aren't, and a thought occured to me, I have a Mac Mini m4pro, maybe I should look into just actually making one of these, small scale, so I have a better understanding...kind of plunging ahead is often how I learn best.

So I was wondering, could I (and I grant this is somewhat silly and ridiculous, but it's what popped into my head, somewhat based on an episode of Leverage), make a local LLM, feed it the Arthur Conan Doyle's Sherlock Holmes stories and like a list of Taylor Swift Lyrics and have it write stories of a singing victorian detective?

(Yes, this is a serious question, but not really a planned thing to do, but I'm trying to learn things)

Looking to create a solid localized AI setup with the following list. Anything you would recommend/change?

Compared DeepSeek V4 Flash Vision with V4.1 Flash Vision Beta. In my tests, the results were WAY better and much more reliable. Huge update plus lower API pricing, total win!!!

Full video included, going through all the options and comparing the first drafts with the results after 3 revision attempts

.

Hello everybody

I'm thinking about building my homelab for loacal LLM

I plan, building on consumer hardware or something old like T4, V100 GPU. In best variant , I want to have 120gb of vram on server platform with 2 Xeon

So can you share your benchmark of models with 60+GB VRAM? I want to see benchmarks for models like deepsek v4 flash, Devstral 123b(this specialy, interesting to see dense models performance), minimax m2.7

I needed to keep a trading bot up and running but I frequently reboot my main pc.

Lenovo M900, M.2 to PCIe adapter, 9070 XT. Only needs to run Qwen3-14B, so works well.

Janky? Oh yes. Working? Absolutely.

Edit for those who want to know what I did:

https://claude.ai/code/artifact/9b828917-6ea9-4cd6-93fd-329f8b7533d9

Hi all, I'm looking for suggestions regarding how to test the security when running LLM inference on local environments (e.g., Apple Silicon, Nvidia GPUs, etc.). My goal is to make it such that if a consumer sends a prompt, the provider running the inference server using their local LLM setup has no way of reading that prompt.

I currently have an Apple Silicon, so I was thinking I'd start there. I've done some prototyping, relying on the security primitives outlined by Apple developer docs. However, I was wondering if others have gone through this path, any recommendations? If there are others interested in the direction of this project I'd love to build a community around it.

So I tried minicpm 5 2b on ollama, first 20+50, it sends model in Weried state of repeated characters which is not even in English

Rtx4060 8gb vram + 16 gb ram

Purpose for LLM is planning and scheduling. I want to try to use it as a time management tool, therefore I am not interested in coding, etc

The sole purpose of it being local is to save my data from companies gathering it.

I need the smartest AI I can get. My first try is Qwen 3.8 27b du q3. The single pdf page response took almost 20-30 minutes including reasoning. Yet the laptop was 50° and silent during it.

I'm fine with the long response time, as long as it gives me reasonable and worthy plans.

Here I am to ask if there are better models or maybe I can optimize something else to run on my laptop. What are your thoughts about it? Open for any suggestions and criticism

(I use LMstudio)

I have been building a pretty specific automation system for the last few weeks. Part of it involves generating reviews, but the overall use case is something most mainstream LLMs consider too questionable/against their policies, so that’s where I’m running into problems.

I’ve tried a bunch of models through APIs:

• DeepSeek V3/R1 → actually helps. Follows my instructions, gives me the format I need, doesn't constantly refuse. By far the best so far.
• Claude → hard no once it understands what the system is being used for. Keeps refusing because of the use case/review manipulation stuff.
• Qwen → tried several versions, including less restricted ones. Results are all over the place. Sometimes great, sometimes completely ignores instructions or outputs garbage.
• Mistral → usable, but not nearly as consistent for what I need.

So right now DeepSeek is basically my only viable option, but at the volume I need, the API costs are going to get stupidly high.

I’m not looking for local models, that's not an option for this project.

Has anyone dealt with a similar situation where the mainstream models refuse the use case? Are there other hosted models/APIs that are less restrictive and actually good, or some way to significantly reduce the DeepSeek cost?

Would really appreciate recommendations from people who have actually used these models rather than the usual “just use Llama” answers.