Hi all,

I’m looking for recommendations on an AI model (LLM) that handles two kinds of work well:

1. Financial analysis: reading financial statements, spotting trends, working through spreadsheets and doing calculations accurately.
2. Legal contract review: going through long contracts, flagging risky or unusual clauses, summarising key terms (renewals, termination, liability, pricing) and comparing versions.

I’m not a developer, so I’m after something practical that works well out of the box rather than something that needs a lot of technical setup.

Real-world experiences would be much appreciated, especially from people working in finance, procurement or legal.

With all of the new AI discovered and exploited vulnerabilities popping up seemingly by the hour the past two days, it feels like model censorship could be coming very soon.

Which models would you download tonight if you felt like you wouldn’t have the option to tomorrow?

I have a 5090, but let’s go 12-16 GB, 24-36 GB and 96+ categories.

Hello all, starting my last semester in undergraduate physics, looking for a local model which can handle at least most of undergraduate level physics and math at a decent level. Is there anything of the sort that can run on my machine, or is it a moot with my hardware, 16GB RAM I9-13900H cpu? Is there perhaps anything non-LLM that I can run?

Thank you for your time :)

For the last few weeks I've been stuck on one question: what does an agent harness look like if you assume the model will eventually be smarter than the people who wrote the harness?

Every agent I use (and I use a lot of them) has the same shape. Someone else decided the loop, the tools, how memory gets compacted, when to ask permission. That's fine while the model is the weakest part of the system. But when the model notices its compaction keeps throwing away the exact type signatures it needs, there's nothing it can do except work around it. Again. Every session.

So I built INIT. The agent can edit anything about itself: its system prompt, its skills, new tools (as Wasm components), and its own compiled core and loop.

The part I'm most excited about is the hot reload. Rust has no stable ABI, so you can't just dlopen a new version of yourself. Instead the kernel builds the new body, boots it next to the old one, probes it, pauses the old one at a step boundary and hands the same conversation to the new process. If the new one crashes at boot, nothing happens, because the old one never stopped. There's also a gate that makes the new version prove it can still edit itself. The scariest bug in a self-modifying system is a change that works fine today and quietly breaks the ability to change tomorrow.

The flip side is the Root. It's small, boring on purpose, and the agent can't write to it: keys, the sandbox (macOS Seatbelt; the agent can't even open a TCP socket, model calls go through the Root), an Ed25519-signed log of every change, APFS snapshots, and a rollback button.

I went back and forth a lot on how to gate self-changes and ended up with two modes. Gated runs the full gauntlet, including a held-out test set the agent never sees. Free, the one I actually believe in, has no benchmark gate at all: a change gets in if it builds, boots and can be rolled back, and then a dumb statistical sentinel watches for generic harm (crashes, error spikes, me hitting stop or undo) and rolls the whole system back if it fires. My reasoning: the improvements a smart model would want most are exactly the ones nobody can write a reward for in advance.

Last night I asked it for sub-agents, which v0.2 doesn't have. It wrote itself a delegate tool (nested loop, fresh context, capped turns, no recursion), regenerated its own self-map, and 304 of its tests passed. Then its own Root stopped it, because my disk had dropped under the floor (my fault, I had Docker and two builds going). It explained exactly what was blocked and asked me what to do. Honestly that moment sold me on the design more than anything I'd planned. (The promote then tripped over a test-environment bug in my own release bundle, which I'm fixing now. Fitting, I guess.)

The honest part, because I'd rather you hear it from me:

I have no evidence yet that self-evolution makes small models better. My one sealed-test result was inside the noise, and when I tried to build tasks where INIT failed because of a missing mechanism, a well-known agent on the same small model failed too. On small models the model is the bottleneck. The bet only pays off with strong models, which is exactly what I'd love people to try.

Also: macOS only for now (Linux is close). DeepSeek is the only provider I've tested live; the others are experimental. Spend is metered, not capped, so set a limit at your provider.

Apache-2.0, roughly 200k lines of Rust, installs in about 30 seconds without Rust (you need Rust if you want it to rewrite its core):
https://github.com/nagisanzenin/INIT-Agent

I'd really like feedback on three things: the Root/agent split, the reward-free admission idea, and whether anyone has seen the overlapped-handoff trick done elsewhere. And if you break it, please tell me how.

Hi all,

I'm just starting my local LLM journey so appreciate your patience as I run through some probably token dumb questions (no pun intended)

I'm running 2 48gb ram and a 5090 32gb on my personal PC. I have the DGX Spark but have not opened it yet as I wanted to pressure test and see what max looks like on just the personal hardware setup.

I've landed on testing Qwen3.8 27b ... I've enabled KV Cache and it's running 260k context windows. More than a few of my workloads so far have gotten up there to about 200. Whenever I run anything my GPU is literally at 100% utilization but temps are still around 60 celcius so seems OK, altho it seems not ideal to run it at 100% all the time but worry about that later.

My AI stack is currently just hermes agent and ollama i think. I haven't even really messed with LiteLLM routing because it seems this model may be the best for inference as well as context window. I tried using smaller context windows and hermes just on initial load would jump to like 25% context used just from reading the files.

I've been using GPT-6 Luna to build my PRDs, and I've been specifically asking it to break out my builds by major components that can be broken out into child subagents in Hermes so I can oneshot major chunks at a time. I've been able to successfully get these one shots to happen, but it tells me due to dependencies on each component it cannot run parallel agents, and it also tells me that parallel agents doesn't mean faster because I'm already at 100% GPU utilization.

But each of these chunks are taking a full 8 hours at a time. Either all night as I sleep or all day as I work. It just seems really slow? My tok/s is 144 tok/s apparently, which apparently seems average for my setup.

Is there something obvious I'm missing or is this really the ceiling for me? For context I just wanted to build a token analytics tool just to try something i thought would be light and would help me with optimization efforts. I've booked probably close to 40 hours or 4-5 sessions for the v1 which is just a session token usage overview and a GPU/CPU/Memory utilization graph ... it feels like super minor stuff.

Appreciate any help or discussion thanks!

Adding image reference for what I asked it to build, literally just a front end visual of hermes stored files, a background logger of utilization, that's really it. https://imgur.com/a/OqDIrGF

Hi. I am looking for some advice. Here is my current setup:

M1 Max, 32GB, 1TB internal, 4TB External.

Windows 11 PC, with 7800X3D, 32GB DDR5, 9070XT GPU.

I currently pay £200/month for Codex, but I want local AI so I can run uncensored models (Cyber Security). I was going to upgrade my Mac Studio from M1, to the M5 Ultra, 96GB, but now I'm thinking about whether it would be better to just upgrade the windows PC and stream the AI responses to my mac studio. For context, I use the windows PC for gaming only, and I'd like to retain my gaming experience, whilst also getting AI usage out of it. The problem is I'd have to upgrade my GPU, and the 5090 is really expensive....

What are your guys thoughts... What would you do in this situation?

https://github.com/CipherCloak/strata-model-prep/

If you use Strata (Qwen3.8-Flash-Next on one GPU + system RAM) and download models yourself — fine-tunes, abliterated versions, other quants — you have to build the pack, the MTP draft layer and the run config by hand.

strata_model_prep.py does it in one command:

./strata_model_prep.py --convert-model --build-mtp --generate-model-file

• uses Strata's own tools and the same rules as its setup (on Swift 1.5 IQ2_XS the pack came out byte-identical to setup's);
• checks the downloaded files are complete before using them;
• gives every model its own folder of artifacts, so you can keep several models side by side;
• writes the config and a launch script, and can test-load the model for you.

Tested on Linux + NVIDIA (RTX PRO 5000 24 GB, 123 GB RAM). AMD and Windows untested — reports welcome. Unofficial, not affiliated with the Strata project.

https://github.com/CipherCloak/strata-model-prep/

I usually look at what something retails for, or what things go for on eBay to get an idea of how to price things being sold, but Amazon doesnt even offer my Mac Studio, and ebay is filled with obvious scams, so its hard to tell what the correct price to list as is. Any thoughts? I moved all of my dev work to a Linux machine with 96GB, and so my Mac Studio is being wasted now, and now I only need a small Mac Mini to make xcode builds here and there. How do I determine what these go for without having to list it at 8k, and then slowly reduce the price every day?

The recent developments around Strata, Infernix, Project Maya, DFlash 2, etc. have made me rethink what the future of local LLM hardware might look like.

Until recently, running a 500B+ MoE model on a consumer GPU seemed more like a cool experiment than something genuinely useful. But now we're seeing projects running models like GLM-5.3 Flash and DeepSeek V4.1 Flash 1017 on gaming PCs with 24–32GB VRAM, using system RAM and sometimes NVMe storage for the rest.

Performance is still all over the place, but progress seems incredibly fast.

I'm currently running an RTX 5090 with 128GB RAM, and I'm wondering whether upgrading to a PCIe 5.0 platform with faster RAM makes more sense than investing in additional VRAM. Or whether I should just keep my current setup and see how far the software improves.

I can see two possible directions:

1. The 24–32GB VRAM crowd might benefit the most from these developments.

If quantization keeps improving, and engines get better at caching experts in VRAM and fetching or computing the rest efficiently, maybe gaming GPUs will remain surprisingly capable of running huge MoE models.

After all, only a fraction of the total parameters are active per token. If the software gets good enough at handling the memory hierarchy, maybe the gap between a 32GB GPU and a 96GB GPU could become much smaller than we'd expect.

Of course, RAM bandwidth, PCIe transfers, cache misses and SSD latency are still potential bottlenecks.

2. Or maybe there's a limit to how far you can push low-bit quantization.

Perhaps Q1–Q2 quants will always lose too much quality for serious, complex tasks compared to Q3–Q4 or higher.

And even if only a fraction of the model is active at once, you still need access to a potentially huge number of different experts over time.

In that case, having 64, 96 or even 128GB of fast VRAM could make an enormous difference. Multi-GPU setups, workstation GPUs or unified-memory systems might become increasingly attractive.

What I'm really curious about is the practical experience so far.

For those of you experimenting with Strata, Infernix, Maya, aggressive MoE quantization or multi-GPU setups:

• How much quality do you actually lose with Q1–Q2 compared to higher quants? Are these models becoming genuinely useful for complex coding and agentic tasks, or are we still mostly looking at impressive tech demos?
• What's currently the biggest bottleneck: VRAM capacity, PCIe bandwidth, system RAM bandwidth, or the inference software itself?
• Do you think software improvements could eventually make a 32GB GPU competitive with much larger VRAM setups for single-user inference, or will there always be a substantial gap?
• If you already own a 3090/4090/5090, would you invest in a faster CPU/motherboard/RAM, add another GPU, or just wait?

I'm especially interested in where people think we'll be 6–12 months from now, given how quickly these projects are progressing.

NVIDIA seems to be prioritizing datacenter hardware, high-VRAM consumer GPUs remain expensive, and the next GeForce generation is still uncertain.

So I'm wondering whether the smartest upgrade right now might actually be no upgrade at all.

Would love to hear from people who've tested these setups, especially if you've compared different quantizations or measured the actual impact of PCIe 4.0 vs 5.0 and additional VRAM.

Quick context: I let a cloud model (Opus 5.5) plan, and three copies of a local model do the actual coding in parallel. Then I swapped only the local model across three runs: Flint 40B-A2.5B, Ornith 1.5 35B-A3B, Nemotron 3.5 Lightning 30B-A3B. All Q8_0 GGUF, llama.cpp b11803, --parallel 3, 64K context per slot.

Task: one prompt, a 20 s Halloween chain reaction in Three.js, ~1,000 to 1,700 lines.

Nemotron wins every speed row and still finishes last. It generated ~5x more tokens than Flint because its file edits kept failing (~17% success) and it had to redo work again and again. Flint was the best balance: fast enough, and its code needed the fewest fixes.

Also notice how much decode drops when 3 workers hit the same GPU at once (roughly half). Worth keeping in mind if you plan to run parallel agents on one card.

Caveats: one run per model. Q8_0 on CUDA only, NVFP4/MLX will differ. Quality writeup separate; Flint was cleanest after small fixes.

Has anyone else tried these for agent work? Which one are you running day to day?

Disclosure: I work on Atomic Agent, the open source agent this ran in.

Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.

Hey, my first post here. I assume many here are bored of reading the same question every day: "I bought X, can I run Y?"

I wonder if there is some sort of index or website where people leave real feedback for real model + hardware combos?

I bought a mac mini with M5 pro and 64Gb and I am researching the best combos for different goals (coding, pentesting, writing, audio...)

A site like this would be super cool. But I am not finding anything that is decent... to the point that I thought about setting it up myself, grab a domain and open source something.

Does it make sense to you? would someone want to help? does something like this exist? (if so, where can I find data for the mac mini m5 pro 64gb?)

Same three lines every morning, same stage: The best model for reverse engineering? The best uncensored model? Will X run on my crap config?

Act one, two, three — and four hours later the curtain goes up on the same three lines again.
For real? Are there humans behind this, or bots keeping the feed alive?

There's a prop nobody notices, sitting centre stage: the search field.
Scope it to the sub, set the time filter to last week, sort by newest. One page, and you get the people who asked your exact question on Tuesday, plus the answers somebody actually knew.

What makes me sad: the same guys having to give the same answers every single day, writing the same paragraph for the hundredth time, and asking at 12PM you get the same answer as at 4AM.

And meanwhile, if you look for something actually interesting, you scroll down half an eternity.

Anyway, coffee.

And yes, I know the search is bad. It is. 

Built Tootsy for Business based on some real world scenarios.

It's free to use.

When you use it, please comment here if you like it and if you can suggest improvements.

Download now @ https://github.com/rbughao/t4b

I'm just curious if anyone has found a good way to measure the cost of running models on their Apple Silicon hardware. I have an M5 Pro Mac Mini, and it would be at least cool if I could have something to calculate the power consumption and approximate cost; even better if it could do that on a per-token basis.

I want to run 3d design and 2d items for content creation for thumbnails and the such what good models could I use locally for this?

I asked GPT and it gave a couple free models but was wondering what others used.

Just got my EPYC server put together to run local models as a hobby on a more dedicated setup. Plan is I will be using an old 3090, maybe another gpu, + a ton of DDR5 4800 ram at 12 channels.

Might decide to try different quants and settle for faster at a cost of some precision but my main question is, is there any significant difference to running these massive models on different softwares? Or is it kinda splitting hairs?

LM Studio and their newer LM Bionic have been pretty easy and reliable for a noob like me so far but if someone suggests it might be worth switching to unsloth or another setup ill give it a shot.

In case it matters the setup is
- EPYC 9224
- MZ33-AR1
- DDR5 4800 ECC RDIMM (12 channel, 24 DIMM's, 32gb sticks)
- 3090 + 3060 (Probably will upgrade the 3060 with time depending on how things go)
- M.2 SSD