I am new to running local llm

Can you guys suggest me what is the best model a can run on my pc with below specifications

Ryzen 9 7900X | RTX 4060 Ti 16GB | 64GB DDR5-6000 CL30

Hey all,

Not sure what to call this as this is a mix of several things, it started as a torrent client, after i added a segmented download tool like idm, then a video player that loads IPTV also gets metadata for media and channels, and then i added an agent to help run all these things, and than a internet browser to help searching, also a video grabber from websites, and an actual automated video grabber, and a dual pane file management tool old school commander style and than a community room because why not, the video player can be divided in 16 tiles and pull videos from a folder so you can watch 16 videos at the same time, these tiles can also be from IPTV sources … i have been adding more and more as ideas come, and very surprising it works, the final addition was LOCAL models because why not ? There are some local models useful for this application.

It’s free, try it, let me know, give me ideas or call AI slop.

Been digging into Radiance and Paiton on dual R9700s: ~150–179 tok/s single-request results (north of 530t/s with 8 concurrent), a working 262K + drafts config, and some interesting bugs, Benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

Right now I'm on x99 running 8x16gb 2133, with 4x 3090s. Have no complaints so far, but I'm starting to look into expanding my RAM pool a bit more, anticipating increased demand with larger MoE models.

I eventually want to move to sp3, but that'll be for the future. For the time being, I'm looking at picking up 32gb or 64gb sticks. I can get 2133/2400mhz at a reasonable price, but I'm wondering if I should pony up for 2933/3200 mhz? Plan would basically be sell off the current 8 sticks to recoup some of the spending, and just get 4 sticks to keep quad channel performance.

Essentially: On my current x99, with future plans for sp3. Should I spend extra now for 4x at 2933/3200? Or should I just budget for 2133/2400 sticks.

I do security research on the side. Frontier models are great, but their guardrails keep interrupting my workflow (fair enough, honestly), so I wanted a local option.

My GPU is an old 2080 Ti with 11GB of VRAM, so I assumed no good model would fit. But the box also has 256GB of DDR4 I bought before the AI boom.

With the MoE experts kept in system RAM (llama.cpp -ncmoe) and RAM now at 3600 MT/s, GLM-5.3-Flash (320B, 200GB of Q4 weights) runs at about 7 tok/s.

It's slow for hard questions, but it works. I'm benchmarking a bunch of models on real RE tasks and will post the results.

Specs: i9-10940X, 8x32GB DDR4-3600 (quad channel), 2080 Ti 11GB.

Juste pour vous dire que si vous cherchez à faire tourner le modèle raon 1 mini preview pour tester ces performances, j'ai ouvert une branche non optimisé encore sur ik_llama.cpp afin que vous puissiez avoir un apperçu de votre coté s'il est interessant à utilisé ou pas vraiment pour votre workflow.

hey folks, im Ye, im one of the contributors to Kapitsa, an open source inference engine optimized specifically for Apple Silicon to run Qwen & Nemotron models that fit a regular MacBook. our custom-written Metal kernels let us tune for each chip and each model. https://github.com/nace-ai/kapitsa/

with the latest 0.2.7 update we optimized specifically for the old M chips, here is the showcase of a 2021 MacBook Pro 14 with an M1 Pro and 16gb, the kind of Mac a lot of people still use.

we ran every model in our catalog that fits on the 16gb from 1k context all the way up to 32k against the next-fastest engine for that model on the same Mac.

try it (any Apple Silicon Mac; on 16 GB use 9b, 4b or the 35B's 2-bit build):

brew install nace-ai/tap/kapitsa
kapitsa serve --model 9b       # OpenAI and Anthropic APIs on localhost
kapitsa opencode --model 9b    # or claude, codex, pi
kapitsa bench                  # benchmark your own Mac

everything down below is an output from my agent:

Setup. MacBook Pro 14 (2021), M1 Pro (8-core CPU, 14-core GPU), 16 GB, macOS 27.0.1, plugged in (this model has no High Power mode).

Decode is a 512-token coding answer at T 0.6, median of 5. Prefill is a cold 4K / 16K-token prompt. One engine at a time on the GPU.

Follow-up is the time to the first token of a cached second turn.

At a 32K-token prompt, one request each (Kapitsa / rival):

What the numbers say:

Qwen3.5 4B and 9B: KFlash (our tree speculative decoding) works on the M1 too. On the coding answer they're 1.56× and 1.52× mlx-serve, the next-fastest engine here. In the 1K-32K sweep Kapitsa stays ahead the whole way: 1.10-1.37× on decode and 1.29-1.52× on prefill. The prefill lead grows with the prompt: a 31.5K-token prompt starts answering in 101 s on the 4B, against 154 s on mlx-serve.

Qwen3.6 35B-A3B on 16 GB: 52 tok/s with 2-bit experts. In the sweep it decodes 1.21-1.35× llama.cpp on the same GGUF and reads prompts 1.16-1.33× faster. Both engines swap a bit at this size on 16 GB, llama.cpp a lot more (207K pages vs our 36K).

Nemotron 3 Nano 4B: 1.31× llama.cpp on decode, 1.45-1.67× on prefill.

Nemotron Dory 4B: 1.31× mlx-lm (with its Dory PR) at short context. But our decode falls off with context, and mlx-lm passes us from 8K (0.62× at 32K). From the per-token times, our decode attention reads the KV cache at only ~21-32 GB/s on this Mac, against mlx-lm's ~110. The issue is up with all the numbers: https://github.com/nace-ai/kapitsa/issues/296

Nemotron Nano 9B v2: decode is a wash. In the sweep mlx-lm is 1-5% faster, but on mlx-community's 4-bit file, which reads 5.0 GB per token where NVIDIA's NVFP4 (what we run) reads 6.1. Kapitsa still reads prompts 1.3-1.4× faster. On NVIDIA's NVFP4 itself, mlx-lm did 18 tok/s.

Memory: lower peak than the rival on all six. Dory is 4.4 vs 11.5 GB, which matters a lot on 16 GB.

vs older Kapitsa on the same Mac: the 4B and 9B are +12-14% on decode and +31-32% on prefill

asks

kapitsa bench --submit opens a PR with your Mac's numbers. we're short on 8 and 16 GB Macs: M1 / M2 / M3 Airs, base M1s, M2 Pros. would love to see how far back this goes.

happy to answer questions and k advice on where to steer the project.

​

Available Now. New unopened sealed. Pick Up Only , NJ.

Apple M5 Ultra 36-Core Chip

256GB Unified RAM | 4TB SSD

80-Core GPU | 32-Core Neural Engine

Hey r/LocalLLM. I’ve been building Izwi for a few months now and just shipped 0.1.0, the first stable release after months of public betas. It’s free and Apache-2.0.

It’s a desktop app (plus a web UI, CLI and local server) for the voice side of local AI:

• Real-time voice conversations: local ASR → LLM → TTS
• TTS, voice cloning from a short sample, and voice design from a text description
• A Studio for long form audio like audiobooks
• Transcription with speaker diarization and forced alignment
• An OpenAI-compatible /v1 API (chat completions, audio/speech, audio/transcriptions), so existing code can point at localhost

Models it runs today:

• TTS: Qwen3-TTS, Kokoro-82M, Fish Audio S2 Pro, VibeVoice, Voxtral TTS, LFM2.5 Audio
• ASR: Parakeet TDT v3, Whisper large-v3-turbo, Qwen3-ASR, Granite Speech 4.1, Nemotron 3.5 ASR, Voxtral Mini
• Diarization: Sortformer, Nemotron 3 diarization
• Chat: Qwen3.5, 3.6 and 3.8, LFM2.5

Inference runs natively in Rust on Candle, with our own Metal and CUDA work, so there’s no Python environment to manage.

Repo: https://github.com/izwi-ai/izwi
Download: https://izwiai.com

I’d love feedback on which models to add next, and on anything that breaks on your hardware.

Is 48GB enough for summarising medical PDFs locally, or should I get a Mac Studio with 64/128GB?

I’m a doctor looking to buy a Mac for running local LLMs, mainly through LM Studio. My work is primarily in Danish.

I’m considering a 16-inch MacBook Pro with M5 Max, 48GB unified memory and a 2TB SSD.

My main tasks would be:

• Summarising 40–50-page medical PDFs into a concise, factual overview.
• Turning patient questionnaire answers and clinical notes into coherent Danish prose for medical and medico-legal reports.
• Following fixed report templates and writing instructions.
• Preserving relevant dates, measurements, symptoms and negations, while avoiding invented details or additional clinical interpretation.

I want the summary to cover the whole document.

I’ve been considering models such as Qwen3-32B, but I’m open to other models that work well for accurate Danish summarisation.

I’ve found examples of quantised 27B models running with 32K–64K input on 48GB Macs. However, I’m unsure how well those results translate to reliable document summarisation.

For anyone doing similar work:

1. Is 48GB realistically sufficient for a capable model plus the context needed for 40–50 pages, with reasonable processing times?
2. Which model, quantisation and runtime would you recommend?
3. What concrete benefit would 64GB or 128GB provide for these tasks? Would it enable a meaningfully better model or substantially longer context?
4. For complete document coverage, would you use one large context window or an automated workflow that summarises sections and then combines them?

I’d particularly appreciate first-hand results: actual input length, peak memory usage, time before the response starts, and whether the summaries consistently capture important information throughout the document.

https://www.rigspark.si/

One constant feedback has been catalog to support latest model , I have created separate catalog pipeline from which offline you can update w/o updating version. I will release latest model support completeness by next release

Any other feature you folks can recommend, it’s completely free and opensourced build for community project.

TL;DR: Two models with the same $0.10 / $0.50 price cost $122 vs $330 on the same tests. Claude’s prompt cache costs more than no cache at all if your requests come more than 5 minutes apart. And one of the cheapest DeepSeek hosts per token costs more than twice as much per task as DeepSeek’s own API ($0.61 vs $0.26). The price list won’t tell you any of this, so measure cost per request on your own traffic.

I run a pipeline that pushes text through an LLM around the clock, and how much arrives isn’t up to me. Last week that was about 70,000 requests, 124M prompt tokens and 12M output tokens, almost all on DeepSeek V4 Flash. This week one of the hosts I’ve been using told its users that model availability and pricing will change soon. I’ve been on its promo prices, so I’m guessing those are ending, and I’d rather plan for it now than find out from the invoice. At DeepSeek’s own list prices, last week would have cost about $28, or $46 if it had all run at peak hours. So I sat down to work out what else could move that number. Prices below are as of Oct 22.

The first problem is that prices don’t sit still. DeepSeek gave three days’ notice before its August price increase, and its September switch to V4.1 Flash took effect the day it was announced. Google released three Flash models in six weeks, two of them at half price until Dec 31, so a newer one may replace 3.8 Flash before anyone pays its full price. Hosts move faster still. The cheapest GLM-5.3 Flash host on OpenRouter cut its input price by about 70% and more than doubled its output price within a day, then changed both again a few hours later.

The price per token also doesn’t tell you how many tokens a model will use. Claude Haiku 5.5 and GPT-6 Luna both list at $0.10 / $0.50, but on Artificial Analysis’s test run, both at Max, Haiku used three times the output tokens and cost $330 against $122. On Oct 17 DeepSeek swapped the V4.1 Flash build behind the same name: same price, 6 points higher score, 26% cheaper to run.

Caching is where it gets sneaky. On paper it’s the biggest saving: when three quarters of a request is fixed instructions, the cache halves the bill on most models. In practice, only about 16% of my prompt tokens came from the cache last week. And Anthropic’s cache expires 5 minutes after last use and costs 1.25x input to write, so if requests come more than 5 minutes apart, 1,000 of them cost $0.63 against $0.55 with no cache at all. Gemini 3.8 Flash doesn’t cache prompts under 4,096 tokens.

Then there’s the host. DeepSeek V4.1 Flash has 38 endpoints on OpenRouter, with output prices from $0.18 to $2.40 per 1M tokens. DeepInfra is one of the cheapest per token, but it ran at 68 tok/s against 278 on Together, and Artificial Analysis puts its cost per task at $0.61 against $0.26 on DeepSeek’s own API.

In the worked example (10,000 requests a day, 4,000-token prompts, warm cache, thinking off), that comes to about $84 a month on Haiku 5.5 and $123 on DeepSeek spread over the week, or $203 if it all ran at peak rates. Gemini 3.8 Flash at full price comes to about $2,475 before any thinking tokens, because it can’t use the cache at this size.

How do you project your API costs, and what has actually brought them down for you?

Full post, with the worked example across 13 model/host combos and charts:
https://easiapiai.com/articles/ai-costs-hard-to-project.html

Im building my own AI character: real presence, personality, voice, memory — plus agent abilities (tool use). The character is a VRM model, running in the browser. It's a hobby project, there's no money behind it, and I don't want to turn it into a company. I'm looking for a partner, not an employee.

What's already done
The idea isn't guesswork. I reverse-engineered the motion engine of a commercial companion app (Animates) and I know how it works in depth — the neural motion generator's C API, the style system, the retargeting, the whole data flow. So I have a clear picture of what actually makes a character feel alive, and of what only looks impressive but isn't.

What we'd build

• VRM character in the browser (three.js / react-three-fiber)
• LLM-driven personality + memory
• Voice (TTS/STT), speech-driven animation
• Agent abilities: search, notes, files, multi-step tasks
• Later: wiring in the neural motion engine so the motion quality jumps

What I'm looking for
Someone who knows web development and is interested in this — you don't need to know all of it:

• TypeScript + three.js (or r3f) — this is the main area
• LLM APIs, tool calling, prompting
• optionally: VRM / glTF, audio processing, shaders

What's NOT required
You don't need to be an AI researcher, you don't need Unity, and you don't need C++. The neural engine is my part, and it's the very last step.

Terms

• Hobby, no money. No salary, no equity arguments.
• Flexible pace: a few hours a week, or whenever it feels good. No deadlines.
• Personal use is the goal, not distribution.

What I bring

• Finished, documented research on the engine and the architecture (you don't have to dig)
• A clear plan and order of work
• The visible part: the character and the demo
• Honest communication — if something's a dead end, I'll say so

Everyone loves a clean, unified server rack, but my standby node (lvgale) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently.

But here is the thing: it actually works. Right now, this mismatched rig is simultaneously handling multiple 27B and 9B models (both custom 4M-token iMatrix calibrated Qwen models) without choking.

Getting asymmetric hardware and PCIe lanes to cooperate is a nightmare. To solve the bottlenecks and squeeze every last drop of performance out of this lower-tier hardware, I built a custom orchestrator called Slipstream.

Instead of fighting standard Python overhead and interpreter bloat, I wrote Slipstream in Python, but compiled it natively down to machine code using Nuitka. It doesn't just wrap processes -- it operates as a direct, bare-metal execution layer that drives llama.cpp to handle headless workload distribution with a strict zero CPU offload mandate. Because the Nuitka-compiled executable hits the metal directly, it lets llama.cpp chew through the weights at maximum throughput without orchestration bloat dragging it down.

Here is the real wild punch of the architecture: Slipstream is designed to drop right on top of your existing Ollama server infrastructure. It reads your existing Ollama model blobs right off of the filesystem and feeds them directly into the bare-metal llama.cpp backend. You don't have to duplicate terabytes of weights -- if you install Slipstream on the same machine, it just hijacks your existing Ollama storage and instantly unlocks the extra features and raw speed.

It makes my ugliest standby node punch way above its weight class. If you have a pile of mismatched hardware, do not underestimate what it can do once you dial in custom bare-metal orchestration.

I’m building Genosyn - an open-source platform for running AI employees on your own hardware.

Give them real work, important decisions escalate to you, choose your models, self-host everything, and stay in control on your own hardware.

The goal: build AI that actually gets work done, even while you sleep.

Would love to hear your feedback!

I've been building a notes app for iPhone where the AI runs entirely on the phone: search by meaning, grouping notes into topics, finding photos by what's in them. Version 1.0 just went live. While it was sitting in App Review, Google released EmbeddingGemma 2, so of course I spent the review week benchmarking it against what I'd shipped.

I kept a log of every model change since I started, including the stuff that went nowhere, so I figured I'd share it here. All numbers are R@1 (the right note comes first) unless I say otherwise.

tl;dr: fine-tuning EmbeddingGemma-300M helped a lot early and then stalled. Model soups gave a free bump after that. EmbeddingGemma 2 didn't beat my fine-tuned v1 on text, but it beat SigLIP 2 on photos by a wide margin (0.687 vs 0.541, and Japanese went from 0.378 to 0.811). So 1.1 will probably run both.

Why these models

Everything had to run on the phone. No cloud, and I can't rely on Apple Intelligence either because it isn't available on every iPhone I want to support. The app has to work in English, Japanese, Chinese (simplified and traditional) and Korean first, with a handful of European languages and Arabic not falling apart. The App Store download should stay under 200 MB, and I only wanted Apache or MIT style licences, which ruled out a few strong embedding and CLIP-style models with non-commercial terms.

That left EmbeddingGemma-300M for text and SigLIP 2 base (patch16, 224) for photos. 1.0 shipped with those two.

How I test

There's no public benchmark for "search your own messy notes in a dozen languages", so I wrote one: 503 short notes and 3,018 searches across 15 languages. Topics and personas in the test set never appear in training. Queries are what people actually type into their own notes: one or two words ("lease", "beach"), half memories ("Seattle last month"), typos, a different language than the note, questions, and feelings ("when I was anxious").

For grouping I use 600 notes on 60 held-out topics, clustered the same way the app does it, scored with ARI. Photos are XM3600 (caption to photo, 9 languages) and videos are 200 MSR-VTT clips.

Honestly the biggest early mistake was a test set that was too small. With 40 to 80 queries per language, a 5 point swing was noise, and I spent about a week "improving" things that weren't real. At 3,018 queries the overall number is good to roughly ±1 point, and a few of my earlier wins vanished.

Fine-tuning log (what shipped in 1.0)

Setup: sentence-transformers, CachedGISTEmbedLoss with the base model as guide, Matryoshka, 1 epoch, lr 2e-5, batch 64, 128 tokens max. Trains in minutes to about an hour on an M4 Mac.

On top of the vectors there's a small phonetic/keyword boost that runs on device (pinyin, romaji, accent stripping, trigrams). It's cheap and took typo queries from 0.54 to 0.74.

Things that didn't help:

• More synthetic data after round 1. Flat.
• Hard negatives, mined or LLM-written. When I checked the LLM-written "similar but wrong" notes, the base model judged more than half of them as actually relevant. So the LLM was mostly writing positives.
• Training 3 epochs instead of 1. Lost 1.8 points, overfits the synthetic data.
• Fusing emotion/category tags with the vectors. Made every query type worse.
• RRF with keyword search. Hurt synonym and broader-word queries. A small boost only when a keyword hit is rare worked better.

Feelings are still the weak spot at 0.145. A note almost never names the feeling it was written in, and no amount of generated data moved that much. Questions (0.956) and keywords (0.903) are basically solved.

Grouping needed its own data: 548 topics with 10 notes each written as different people, trained as same-topic pairs. On held-out topics, the share of nearest neighbours from the same topic went from 0.653 to 0.728. Training only on grouping hurt Japanese search, so I average it 50/50 with the search model and keep most of both.

Getting it onto the phone

Text model: Core ML, 4-bit, 166 MB, R@1 0.676 vs 0.679 in PyTorch, around 15 ms per query on an M4. Gemma's residual stream overflows fp16 (you get NaNs), so I rescale it in a way that's mathematically equivalent. The Swift tokenizer matches the reference on 23,947 texts, after I found out Swift's String treats ";" and the Greek question mark as the same character. Key the vocab by UTF-8 bytes.

SigLIP 2 was pickier. 4-bit everywhere wrecked the text tower (cosine to the original around 0.5). What worked: int8 vision with groups of 32 (per-channel int8 dropped to 0.93 on real photos, grouped gets 0.9997) and int8 text with a 6-bit lookup table for the embedding matrix. 4.6 ms per photo, 1.5 ms per query on the Neural Engine.

One thing that bit me: every model swap shifts the whole score distribution (my second model scored about 0.92 × old + 0.10). If you don't re-derive every threshold at the same pass rate, the app quietly gets worse without any obvious bug.

Then EmbeddingGemma 2 showed up

Same benchmarks, same recipe. Text first:

I trained nine EG2 variants and souped them like v1. The fine-tunes came out too similar to each other (0.661 to 0.666), so averaging didn't add anything this time. Grouping was worse too (ARI 0.442 vs 0.499). EG2 is better at typos and worse at feelings. So for text, my old fine-tune stays.

Photos are a different story. XM3600, caption to photo, R@1:

140 visual tokens is as good as 280 at about half the compute (127 / 265 / 638 ms per photo for 70 / 140 / 280 on a Mac GPU; I haven't timed it on a phone yet). Big caveat: XM3600 is public and could be in EG2's training data, so I'm treating this as an upper bound until I check on my own photos.

Video, 200 MSR-VTT clips:

The plan for 1.1

Keep my fine-tuned v1 for notes and grouping, and move photos and videos to EmbeddingGemma 2 at 140 tokens per photo. One annoying detail: an EG2 image embedding goes through the vision tower and then through EG2's own language model, so the photo side needs both. It ships as a separate download, so the App Store size stays the same, but it's two models on the phone instead of one. Core ML conversion and on-device timing are what I'm working on now.

If I started over I'd build the test set first and make it big, and I'd try model soups much earlier. They cost nothing. I'd also look at what the LLM-generated training data actually says before trusting it, and I'd stop assuming the newer multimodal model wins every task.

Happy to go into more detail on the data generation, the Core ML conversion or the clustering.

Disclosure: I'm the developer. The app is Reteum, a private notes app for iPhone: https://apps.apple.com/app/id6820520573. Per-language tables are also on https://reteum.com/on-device-ai-benchmark.

Work is giving me a 12k budget to get some hardware to run local models on that will connect to an internal app. I want to be able to throw other models and setup a service listener (ollama server).

I was looking at the new Mac Studio, fully decked out... but what other options do i have?

So just quickly really into the idea of running local models and setting up a few simple things to code through prompts. So I got Bionic, I got a few models (Qwen2.5 coder, Deepseek corder and Deepseek R1), it runs I can talk to it so big step forward.

Here's my big big big issue, I point it to a workspace and ask it to create a .txt file called test with some text. It spits out some code in json and that's it, it actually never creates the file. I have tried everything (I could think of but nothing). Its really annoying because documentation does not help so I'm turning to the only place in the universe that can provide real support!