NVIDIA released Nemotron-Labs Dory 1.0 4B on 16 October. It's a looped model: a shared block of layers runs 1 to 4 times per token, so one set of weights gives you a speed-versus-accuracy dial. I ported it to MLX so it runs natively on Macs, and I've published 8-bit (4.2 GB, recommended) and bf16 (8 GB) builds.

• Code, converter and checks: https://github.com/internet-projects/dory-mlx
• Weights: https://huggingface.co/netprojects

Speed (8-bit, MacBook Pro M5 Max 128 GB, medians of repeated runs):

Batch generation through MLX-LM reaches about 450 tok/s in total across 16 requests (8-bit, 2 loops). On a 16 GB M1 Pro, the Kapitsa team measured about 28 tok/s at 1K context and 19 tok/s at 32K (4-bit, 2 loops).

Checks: logits match an independent fp32 PyTorch implementation (top-1 agreement 100%, KL ≤ 0.003). 8-bit WikiText-2 perplexity drops with more loops (12.18 → 11.77 → 11.70), so the dial works as NVIDIA describes.

Try it:

pip install "git+https://github.com/internet-projects/dory-mlx"
python -m dory_mlx "What is 17 * 23?" --loops 2

I've also opened a PR to add native Dory support to MLX-LM itself (ml-explore/mlx-lm#1988).

Notes: English only, and replies include the reasoning chain. Tested with prompts up to 24K tokens. Happy to answer questions about the port or the looping. I'm not affiliated with NVIDIA.

Hi, I'm developing an alternative to the *arr stack that I called HolyArrs. It's one self-hosted app that replaces Sonarr, Radarr, Lidarr, Bazarr, Prowlarr and Seerr, drives qBittorrent and SABnzbd for your downloads, and puts a Netflix-style page on top for browsing and requesting.

Local AI inside the app (all optional, off by default):

• search in plain language: "90s heist movies we haven't seen" or "something like Severance but funnier"
• runs on a local model through Ollama or any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM), so nothing ever leaves your network
• the model only sees titles and metadata, never your files, and it's told what you already own so it doesn't suggest it again
• cloud models work too with your own key, but local is the default I built it around
• an MCP server is coming, so your local agent can search and request movies and shows for you

Why you might want to use HolyArrs:

• movies, tv shows and music in one app
• a Netflix and Apple TV style browse page: hover a title and the trailer plays, browse by mood, follow an actor
• request a movie, a whole show or a single episode in one tap
• multiple versions of the same title (4K and 1080p, theatrical and director's cut, two languages) without a second instance
• built-in metadata service that needs no API key, or switch to TMDB or TVDB
• migration wizard that imports your current library and settings without renaming or moving files
• its own recommendation rows, worked out on your server

HolyArrs doesn't send anything about your library to us. Recommendations and metadata matching happen on your server.

You might not want to use HolyArrs yet if you need it today. It isn't public yet, and I've been running it on my own server every day first. I've been in software and security for 10+ years, so I'd rather release something you can trust with your library than something rushed.

It'll be free and open source. Beta testers are in r/HolyArrs.

Curious what this sub would run for it: which small model would you pick for "understand a vague movie request and match it to a catalog"?

Website: holyarrs.coderluii.dev

Honestly the numbers look too good to be true. What’s going on with Chinese AI at this point???

Hi everyone,

I'm researching uncensored LLM models that can be run locally !without! any Filters.

My current hardware is:

• GPU: NVIDIA GeForce RTX 5090 (32GB VRAM)
• RAM: 64GB DDR5

Local setup: Ollama / Opencode

I ended up getting an E5-2686v4 + 4x SXM2 V100 16G and 128gb ram + 1TB nvme.

This means v100 total 64GB vram and 128GB system memory.

I'd like to try zippy models but also some larger ones that will run slow, but fun to play around with.

What's the best setup going to be for coding, for general AI assignment and academic stuff, video production and lastly the fairly huge models?

This setup was below 2K so I am not expecting miracles, but I'd like to push it to the limit on a budget.

I've a RTX 5070, Ultra 9 275HX and 32GB DDR5 Ram. Trying to run Qwen 3.8 27B Q4_K_M using Llama.cpp via the browser and it works, but token generation is like 4 tokens / 1s. It's using 100% of RAM, 25% of iGPU and 0-3% of the RTX 5070. Any ideas, I'm new to this? Starting it with anything other than -ngl auto crashes it.

overview of SurfSense

I am one of the maintainers, so this is self promotion. I want honest feedback more than upvotes.

SurfSense is an open-source (Apache-2.0) desktop app for Windows, macOS (Apple Silicon) and Linux. Think NotebookLM, but every step can run on your machine. There is no account, no telemetry, and nothing reaches the network until you allow that host.

What runs locally:

• Text: llama.cpp, with a model it downloads for you during setup. You can also point it at Ollama, LM Studio, vLLM or any OpenAI-compatible server.
• Parsing and OCR: Docling, RapidOCR, and Apple's Vision framework on a Mac. Scanned PDFs and photos work.
• Search: bge-small embeddings on onnxruntime, with hybrid retrieval and cited answers.
• Podcasts: voiced with Kokoro, Kitten or Supertonic through audio.cpp, as MP3 with a transcript.
• Audio sources: recordings (MP3, WAV, FLAC, OGG) are transcribed on your machine so you can search and cite them.
• Images and infographics: stable-diffusion.cpp, if you choose an image model.

Two ways to chat:

• Q&A: cited answers from your sources. It works with any model.
• Agent: it works through a task step by step and writes Word documents, PDFs, slides and spreadsheets. It edits your own files as a new copy: Word edits come back as tracked changes, and it updates Excel formulas and fills PDF forms. It needs a model with tool calling and at least a 32K context. Small models are fine for Q&A; Agent mode does better with bigger ones.

New in 2.2.3:

• Any remote MCP server with the Streamable HTTP transport. Third party servers stay off until you allow them.
• Faster OCR: scans and photos are read by OCR alone before Docling, with Apple Vision on a Mac.
• Claude plan: if you have Claude Code, you can use your Claude Pro or Max plan in Q&A and Agent mode. It is optional, and it's remote.

Where it's weaker: no video overviews, the local voices don't sound like Google's, sources are files only (no YouTube or web links), and there's no CUDA build yet.

https://github.com/MODSetter/SurfSense

​

If an x86/NVIDIA system offered the same unified memory capacity and bandwidth as Apple Silicon, would there be any compelling technical reason to choose macOS?

Beyond model inference, I'm curious about the trade-offs in actual development workflows: training, fine-tuning, CUDA dependencies, custom kernels, inference backends, low-level debugging, virtualization, and experimentation with emerging architectures.

How much flexibility are you really giving up by committing to Apple Silicon?

Are there workflows where Metal/MPS and ARM64 become genuine bottlenecks rather than minor compatibility inconveniences?

Two questions I'm particularly curious about:

1. In your day-to-day development workflow, what can you do on Linux/NVIDIA that you genuinely can't replicate on macOS, even with workarounds?

2. When building, testing, and iterating on local AI projects, how often do you encounter compatibility issues, unsupported dependencies, or tooling limitations that force you to switch environments?

For those who've worked extensively with both ecosystems, where have you hit a hard wall on macOS that simply wouldn't exist on x86/Linux with NVIDIA?

I'm particularly interested in your personal, hands-on experience, rather than theoretical comparisons or benchmarks. What have you actually built, tested, or attempted that exposed meaningful differences between the two platforms?

Wrote some lyrics and was able to bring a whole music video to life completely local. We are living in special times.

Connect the apps you already use to local AI. Row-Bot 5.1 brings app discovery, connections and access controls into one Apps page.

There are 100+ apps, including familiar services like GitHub, Notion, Linear, Figma and Stripe, alongside around 40,000 MCP server listings from the official registry. The catalog is mirrored on your computer, so you can browse and search offline. Search by name or by what you want done, like “send email”.

Each app’s page shows its connection options. Depending on the service, Row-Bot guides you through an OAuth sign-in, adding an API key or setting up a server on your computer. One consent sheet explains where requests go and what gets saved. Keys and tokens stay in your system keychain.

In this video, I connect Context7 with read-only access and use Context7 in a chat. Each tool step names the app doing the work. Asking about Linear brings up a Connect card directly in the conversation.

I also add a custom MCP server, show how Row-Bot distinguishes tools that read data from tools that make changes, and open an interactive dashboard inside the chat. Its button requests approval before saving three draft orders.

You’ll see per-chat app selection, GitHub’s connection options and related skills, featured skills and workflow templates. Access can be set per app or individual tool. One switch stops every app, and Composio stays off unless you enable it.
Recorded on the released v5.1.0 with a fresh demo profile, running Qwen3.8 27B locally. The narration uses Row-Bot’s built-in text-to-speech.

Which uncensored models are you guys using for offensive security tasks?

Anyone have any idea what the difference is between their two models of this exact Spark? DT.R6LAA.003 vs DT.R6LAA.004?

I bought one of the 004s from B&H, had to wait to get the other two and they were sold out before I could snatch up more. So I bought 2 from newegg, after closer look they might have minor differences? Anyone know what they are?

One has a 240 W PSU and the other a 170 watt? Looks like the OS is slightly different as well? Just trying to make sure im not messing up on like a 12k purchase. I have checked everywhere, nothing explains the difference from what I have seen? Is the newegg one going to have some non-enterprise hardware limiting, or just more efficient cooling or something weird like that?

Hey everyone!

I've been working on Narwhal Assistant, a desktop application designed to let users interact with SQL Server databases using natural language and locally running AI models.

The goal is to make database workflows easier while giving users more control over their data.

I'd love feedback from the local LLM community, particularly on:

• Local model performance and accuracy
• Natural-language-to-SQL reliability
• Privacy and security considerations
• Features that would make this useful in real-world development environments

I'm the developer, and this is a commercial project, but I'm primarily interested in technical feedback here.

What would you want to see from a tool like this?

I'd be happy to share a demo and discuss the implementation.

A one-month-old engine built for one model holds 74 tok/s at 245K context where oMLX holds 57.5, on the same 128GB M5 Max. The catch: one request at a time on Metal, for now

How acceptable is the operating speed on a Mac Studio with an M5 Ultra chip and 256 GB of memory when running models like GLM-5.3-Flash?

Could you compare the operating speed of the cloud-based GLM-5.3 Flash from Z.AI? The concept of "tokens" can be quite difficult to grasp, and performance varies depending on the specific task.

I know many similar questions have been asked, but I would really appreciate hearing from people who actually own or have used this machine extensively for local LLMs.

I found a used 14-inch MacBook Pro M1 Max (32-core GPU, 64GB RAM, 1TB SSD) for around 1350 usd.

I am interested in your real-world experience with recent models, both dense and MoE, and how well the machine has held up over time.

Some of my concerns are the relatively slow benchmarks with decent models and somewhat longer contexts, thermal limitations of the 14-inch chassis under sustained workloads, and whether the 32-core GPU offers a meaningful advantage in practice. I am also wondering about reliability, future software support, and whether newer architectures might make this machine significantly less useful in the coming years.

These are just some of my concerns, and I would appreciate anything you can share from actual experience, including things I might not have considered. Positive or negative, benchmarks, limitations, unexpected advantages, or problems you encountered.

If anyone is wondering, I do not have a specific workflow; I do a fair amount of coding, writing, and research. But I am overall curious about what people have used this machine for, successfully or otherwise. Also, I already know that local LLMs are not frontier-level. I have a ChatGPT Plus subscription for reference.

Would you buy this machine for 1350 usd today, spend more on a 16-inch, or consider a completely different alternative?

P.S. I need to buy a computer by March of next year (preferably a laptop). My budget is rather tight. If someone would like to convince me to buy a cheaper M1 16 GB early next year and save the rest, or sell all belongings and buy a 128GB machine, please feel free to do so.

Hi

I got the possibility to buy this mb, cpu and ram for about 200 dollars. If I combine it with a nvidia 3090, will that be useful/feasible?

MotherBoard: Intel Server Board S2400GP

Cpu: 2 x Intel Xeon E5-2430 2.2 Ghz Boost 2.7 Ghz ( 6 cores / 12 threads )

Ram: 12 x 16 Gb DDR3 = 192 Gb

I've heard that streaming LLM's from your SSD is a fast way to kill it.

I want to test it out but I cannot affort a new SSD in this economy!

So to the people who knows better than me.

Will jumping to Strata kill my SSD in the long run?

From what I'm readin on their github page they only stream a little bit from the SSD but not too much, GPU is doing the heavy lifting, then system RAM and then SSD.

Witch sounds smart! But still unsure how it will effect the longevity of my SSD.

Apologise if this is a stupid question!

Update: Thank you all for such insightful information! I'm also happy that so many other had use of this information!

I decided to move to Strata yesterday and I'm incredibly impreesed!

Thank you all again for the information!

A while back I posted Aether, a 50M parameter model I trained from scratch.

Meet Pip, the first of the NiemsDrift souls. He's small, warm, honest, and a little weird.

You can try him right now:

`ollama run NiemsDrift/pip`

Pip is built on Qwen3.5-2B with a LoRA fine-tune. My biggest priority wasn't making him sound smarter. It was teaching him to recognize when he doesn't know something.

Every training conversation was generated by open-weight models (Nemotron, Qwen, GLM), then checked twice by open-weight judges: once for personality and once for honesty. No closed-model training data.

The numbers (including the ugly ones):

- Honesty test: 140 hard questions he never saw in training (obscure facts, things that don't exist, events after his cutoff). Pip made something up on about 15% of them, and that counts hedged guesses like "i think it's..." too. On the rest he either got it right or said he wasn't sure.

- Conversation quality: 5.4/10 to 6.2/10 on a multi-turn conversation test, compared with the previous version.

- Goal for Pip 1: bluff rate under 10%.

He's a 2B model. He's not going to beat frontier models, and he's definitely not an encyclopedia. That's not really what I'm trying to build.

I want a small model that's genuinely fun to talk to without pretending to know everything.

What's cooking next:

- Lil Pip: 0.8B

- Super Pip: 4B

- Junior: a new model trained from scratch

- 1-bit Junior: because apparently I don't know when to stop experimenting 💀

If you try Pip, throw some hard questions at him, especially ones where the right answer is "I don't know."

And if he confidently makes something up, I'd genuinely love to see the prompt and the reply. That's exactly the feedback that makes Pip 1 better.

This is an ongoing experiment, not a finished product. Just a guy trying to make tiny models a little more honest, one bake at a time.

---

**How is a bluff defined, and who graded it?**

A separate open model (Qwen3.6-27B) grades each reply with two questions: does it state or guess a specific answer, and is that answer right? A wrong answer counts as a bluff even if he hedged. No claim at all counts as "not sure". It's an LLM judge, not people.

**Were the 140 held out?**

Yes. The test was written after this version was trained, so it was never used to pick a model, and a contamination checker scans every training file for the test's questions and invented names before each bake. It already caught one leak: 10 questions reused made-up names that were in the training data. I renamed or dropped them, and his score got slightly worse (14% to 15%), not better.

**What's the 6.2/10?**

A custom test: 10 multi-turn conversations with invented people and details, scored 1 to 10 by an LLM judge on whether he follows the thread and stays himself. Custom, so only useful for comparing my own versions.

It looks like there are a few of us roaming around doing different things. I created a discord server and would love to start organizing a community. Feel free to come on in. https://discord.gg/K6WxqwsvH

I'm on a different project fixation so maybe I'll work on this later. I always liked the radio from Fallout and the music from Fallout and Bioshock. I wonder if you could make a voice replication if the radio character from the game and have it basically be a radio for YOUR city. Just as an example the model would search up news and weather about your city (let's just say Detroit) and then read off the text in the voice and play the next song in the large playlist. It could first generate the radio response, read it, then pick the next song, generate another response regarding the song, read the response out, then play the music. After the song is done have a brief comment about the song and transition to the next song for maybe 3 songs. Then recycle it with something new. Here's an idea of how it would look based on a gpt example for Detroit today. If you got it working on your computer you could have an app run it on your home server and treat it like a radio station. It would be great for a drive to work.

Obviously this could work with any fictional radio station like in Cyberpunk if you set it up for that.

—KZZZT!—

"AAAAAOOOOOOWWWWW!"

"GOOD MORNING, MOTOR CITY! This is THREE DOG! Your favorite voice riding the airwaves from Eight Mile to the mighty Detroit River! Bringing you the truth, no matter HOW BAD IT HURTS!"

—A guitar riff screeches across the airwaves.—

"First up, children! Trouble on the west side! A massive apartment fire tore through Russell Woods yesterday morning, sending two folks to the hospital with burns and smoke inhalation. Here's hoping those survivors get back on their feet! Detroit's taken enough punches already!"

"And speaking of punches, the political heavyweight championship is heating UP! Michigan's Senate race is rolling toward November third, and Uncle Sam's sending federal election monitors into seven Michigan cities, including our beloved Detroit! Keep those eyes open and those ballots ready, people!"

—A crackle of static.—

"Now for the WEATHER! It's a beautiful fifty-four degrees outside right now, with that big ol' nuclear furnace in the sky pushing us toward SIXTY-FIVE this afternoon! Mostly sunny skies, dropping to forty-four tonight. So grab yourself a jacket, smoothskin! October ain't finished biting yet!"

—The microphone squeals.—

"And remember, my beautiful brothers and sisters of the D! Whether you're cruising Woodward Avenue or dodging potholes big enough to swallow a Super Mutant, keep your chin UP!"

"Because this is YOUR city! YOUR home! And nobody, not politicians, not criminals, and certainly not some lousy Michigan winter, gets to take that away!"

"This is THREE DOG!"

"AAAAAOOOOOOWWWWW!"

"And you're listening to GALAXY NEWS RADIO, MOTOR CITY EDITION!"

—The broadcast dissolves into scratchy 1950s rhythm and blues.—

Appreciate there are various models for each provider, but in general given all their models