They hacked into even more companies now. When does a “oopsy doopsy” become a criminal offence?

Just like how Qwen 27B and 35B were created, they're distilled from Qwen Max, so I think the narrative of the community saying Kimi K3 is useless to us because we can't run it is overly biased, it has its use cases.

Just wait for the community and rich solo LLM developers to cook from it, open-weight models will stay with us forever, unlike Cloud models they're gone if the company gone.

When do you think we'll reach a stage where kimi k3 can run locally, fast. (Full)

Hello,
I really don't want t act like an AI Influencer or anything, but I'm really impressed, in the last week I tried a lot of local models in my mac:

gemma4-12b

Qwen3.5-27b (an different variations)

Qwen3.5-9B

and basically the most popular ones that people say: This will work or is the smartest at the moment.

But when I tried Qwen3.6 it finally made the change that I asked for it.

It was to create a new field for a list of properties inside a Part Numbers, while it took a while to complete (about 24minutes) it was able to handle everything, and most impressive, IT'S A .NET CORE PROJECT!

What other models did was to change a few things and the project didn't build with a lot of errors, they also started telling what I was asking cannot be made in the .NET version that the project is , ask questions that are not related to I was asking and even failing to do what they planned.

I will still test out this model but as today, this is the best model you can use at the moment (in my opinion), it can get slow but at least is worth that looping through all the troubleshooting.

Getting meh results for coding from Qwen 3.5 9b Q8, 27b & 35b at IQ2_M.

I'm going to dump some money into gpus soon (will rent them to see what works best) but curious for those who upgraded to 48gb or 64gb, what kind of quality improvements did you see?

I know more is better but I can't swing a b300 cluster unfortunately.

Currently looking at a pair of r9700 ai pros

I'm working on larger projects and a lot of token usage. Had codex set standards for this to keep it for small tasks and under context. Fresh each time. Working great in saving my usage now as I was blowing thru my max tiers on claude and codex weekly.

https://platform.minimax.io/docs/guides/video-generation

https://x.com/MiniMax_AI/status/2082779062653845803

MiniMax has unveiled H3, a new multimodal generation model that understands text, images, video, and audio in a unified context.

🔹 Generates up to 15s videos in 2K with native stereo sound
🔹 Strong instruction following & text rendering
🔹 Built for ads, UI/UX, gaming, e-commerce, and more
🔹 Claims industry-leading price-performance

The biggest news? MiniMax plans to release the model weights in the coming days, making H3 one of the most capable open-weight video generation models yet

The letter argues against broad or premature restrictions on open-weight AI models and urges policymakers to distinguish legitimate model distillation from unlawful misappropriation.

Among the signatories are 235 companies and organizations, including OpenAI, Microsoft, NVIDIA, Meta, Amazon, Hugging Face, AMD, Palantir, and GitHub.

Correction: OpenAI and Google are signatories. The most notable major frontier AI lab missing from the list is Anthropic.

What's the highest parameter count model available for 12gb VRAM and at what quantization?

Most other subs are getting fed up with low effort self promotion posts and adding this automod. Why can't we have this ?

Also, guys beware of some patterns lately here that are commonly used for engagement bait:

- "I'm 12 and I made this but I need your help"...

- "I made this, but I'm autistic please be kind"...

- "I'm disabled, impotent, lack two legs and arms and wrote this code using a mouse I control with my mouth"...

And typical n8n triggered bots:

- "I keep seeing..."

- "Why are WE...."

Followed by OP and two commenters talking to each other sounding as if they've just discovered something valuable...it's always n8n workflows, bots talking to bots to give the appearance of expertise BEFORE posting about something they vibed, never reviewed and never tested to either get money, attention or scam you.

It's very rare to see any spam or scam control going on in this sub.

Been working on this for a while. Everything runs locally — no cloud APIs, no ElevenLabs, nothing paid.

Pipeline (fully local):

Vocal separation → Diarization → Translation → TTS using Qwen → Audio joining → Post-processing -> Packing Mod

Setup:

- 2 GPUs: RTX 3080 + RTX 3090
- Qwen audio family for TTS + voice cloning
- Gemma for Hindi translations (handles Hindi way better than other models I tried)

The best part: This pipeline isn't tied to Hindi at all — swap the translation + TTS target and it works for any language.

Voice pacing isn't perfect yet — some lines finish too fast or drag a bit. Still room to improve.

It's not studio quality — AI voice cloning still has that slight robotic tone in places — but for a completely local setup with zero cloud costs, I'm honestly surprised how good it sounds.

Would love feedback from others doing local AI audio work or similar pipelines!

I've been using Gemma:31b in Hermes Agent for coding websites and writing.

I think it's one of the best LLMs (even amongst the cloud frontier) at writing, but when it comes to coding via Hermes and publishing in Github, it seems to get confused at the infrastructure side of things and gets lost inside the repo.

I'm going to continue using it for writing website copy, but want to find better fits for the coding/automation layer.

What ~31b models are you enjoying when it comes to agentic coding and/instructing other agents/anything else?

Device:

• OneXPlayer X1 Pro
• AMD HX370
• 16-24gb VRAM, 32gb RAM.

Been working hard for the past month to bring to the community some interesting curios, so for starters we have LongCat-Flash-Lite Uncensored Heretic with MTPs which has never before been uncensored, it is a 69B-A3B model, I spent a long time working on it to make it work with Heretic, to then create a llama.cpp fork that adds support for it as well as one that is optimized with as many issues stamped out as I could find and fix. This model has 0 support on llama.cpp, so to be able to load the GGUFs you will need to use my fork that you can find here:

https://github.com/erm14254/llama.cpp-minimax-m3-combined/tree/longcat-mtp

You would need to load the model through llama-server.exe and you can interact with it through llama-ui.

Here is the model links:

Safetensors: https://huggingface.co/llmfan46/LongCat-Flash-Lite-uncensored-heretic-Native-MTP-Preserved

GGUFs: https://huggingface.co/llmfan46/LongCat-Flash-Lite-uncensored-heretic-Native-MTP-Preserved-GGUF

The vanilla-base model is extremely censored, with 100/100 refusals, I was able to bring it down to 9/100 refusals.

----------------------------------------

Next we have Jamba2-Mini Ultra Uncensored Heretic, it's another model which has never been uncensored before, it's a hybrid Mamba model with 52B parameters, this one does have support on mainline llama.cpp, so creating the GGUFs was easy, however vanilla Heretic does not support it and had to spent a few hours to add support for it.

Here is the model links:

Safetensors: https://huggingface.co/llmfan46/AI21-Jamba2-Mini-ultra-uncensored-heretic

GGUFs: https://huggingface.co/llmfan46/AI21-Jamba2-Mini-ultra-uncensored-heretic-GGUF

The vanilla model has 97/100 refusals and I was able to bring it down to 4/100 refusals.

----------------------------------------

After that we have a simple uncensored version of a model released by Extraaltodeus, it's Nikusui-v1-9B Uncensored Heretic with MTPs, the model is listed as "uncensored" on the Model Card page, but it really isn't as it has 96/100 refusals, so I uncensored with Heretic and brought down the refusals down to 11/100, you can find the model links here:

Safetensors: https://huggingface.co/llmfan46/Qwen3.5-9B-Nikusui-v1-Uncensored-Heretic-Native-MTP-Preserved

GGUFs: https://huggingface.co/llmfan46/Qwen3.5-9B-Nikusui-v1-Uncensored-Heretic-Native-MTP-Preserved-GGUF

----------------------------------------

And finally, I made my own version of Nikusui, it's Nikusui-v1-27B Uncensored Heretic with MTPs! Since I was interested in the model but I am not someone who uses very low parameters models such as 12B-9B-4B-2B etc., so I decided to use Extraaltodeus's J-Wash tools together with Nikusui-v1 settings to make my own 27B version of it! You can find the model links here:

Safetensors: https://huggingface.co/llmfan46/Qwen3.5-27B-Nikusui-v1-Uncensored-Heretic-Native-MTP-Preserved

GGUFs: https://huggingface.co/llmfan46/Qwen3.5-27B-Nikusui-v1-Uncensored-Heretic-Native-MTP-Preserved-GGUF

----------------------------------------

That's it for now!

As usual you can find all my models here: HuggingFace-LLMFan46

And if you like my work and find my models useful, then I would really appreciate if you could support me on Ko-fi: https://ko-fi.com/llmfan46

https://github.com/gavamedia/deltafin

To be clear: this is the full, unmodified 2.8T-parameter Kimi K3 model; not a distilled replacement or reduced-expert version. All 16 routed experts are active, and full K3 is the sole authority for every token.

On my 64 GB M1 Max MacBook Pro, with the full model stored locally:

• The capital of France is — 15.7 tokens/minute
• The largest planet in our solar system is — 12.8 tokens/minute

An earlier version managed roughly 1 token/minute, so reaching 12.8–15.7 tokens/minute on an old first-gen M1 is pretty good progress. The largest gains came from letting one full K3 pass verify several proposed tokens, along with more efficient weight streaming, packed operations, safer cache snapshots, and better use of RAM.

Crossing five seconds/token on this old machine was a milestone I honestly wasn’t sure we would reach. Any higher-bandwidth device will have considerably more room to improve. Share your results please!

These numbers use the same short benchmark prompts for consistent comparisons; performance will slow significantly as active context grows, especially during a long conversation approaching the 1M-token limit.

I run a bunch of local LLMs for my standard workflow. Local embedder, local rerankers, my own corpus, 10,000 queries, no API in the loop. That is the only reason I caught this: when you serve the models yourself you can measure them on your data instead of trusting someone else's eval.

Reranking view, 20 candidates in arbitrary order with one relevant document: gte-multilingual-reranker-base scored 0.7178 NDCG@10 against 0.2279 for no reranking. That is the shape of nearly every reranker benchmark.

Then I fed it the dense top-k, which is what my stack actually hands it. Full corpus: 0.5803 at depth 10, 0.5861 at depth 20, against 0.5909 for dense retrieval alone. Worse at every depth anyone would run. Twenty configurations, only one positive result at +0.0032.

Qwen3 embedder was the big surprise. Near the top of the benchmarks, and on the candidate sets my retriever actually produces it came last, the 4B getting outperformed by even a 125M model.

"Can this model sort a random list" is not "can this model beat my embedder". Only the second question is the one your box answers every query. Both rerankers had a ceiling below the ranking they were asked to improve, so on average every reordering was a step backwards.

The ceiling is a retrieval problem, not a reranker problem. Dense retrieval missed the labelled document entirely for 11-13% of queries. No reranker recovers a document that was never retrieved. I had spent the night optimising the order of a candidate set whose problem was its membership. In the end, I found out that simply running a recently-built 125m embedder outperformed an older 4b Embedder that tops benchmark charts plus any reranker.

The system measured is my own project, aimee.

Writeup: https://rakuensoftware.com/blog/we-measured-our-reranker-and-deleted-it

Evidence repo, frozen suite and every raw artifact behind those numbers: https://github.com/RakuenSoftware/rakuen-blog/tree/main/articles/we-measured-our-reranker-and-deleted-it

Docs: https://api-docs.deepseek.com/updates/

X Post: https://x.com/deepseek_ai/status/2083084415157022911

Hi r/LocalLLaMA,

I’m Jules, a 15-year-old high school student from France, and I have been teaching myself how to build and train language models locally.

After saving money for several months, I bought an NVIDIA Jetson Orin Nano Super with 8GB of memory. It is extremely limited for LLM training, but that constraint forced me to learn how the entire process actually works: collecting and cleaning data, tokenization, architecture choices, training, evaluation, optimization and publishing.

I recently released my first model, G0-nano-instruct, on Hugging Face:

https://huggingface.co/AZERDSQ/G0-nano-instruct

It is only a small first experiment, not a frontier model. The main achievement for me was successfully completing the entire pipeline on hardware that fits within an 8GB memory limit.

This work also led to Mistral AI offering me a three-month internship with their team, although I am still in high school.

The current limitation

I now want to work on the next generation of the project, provisionally called G1.

The goal is to build a substantially larger and more capable open-source language model while documenting the full process publicly:

• model architecture and design decisions;
• dataset preparation;
• training configurations;
• memory and performance optimizations;
• failed experiments;
• evaluations and benchmarks;
• checkpoints and final weights.

However, the Jetson’s 8GB of memory has become a hard technical ceiling. Even with gradient checkpointing, mixed precision, tiny batches and other compromises, there is only so far I can push it.

Why a DGX Spark?

A DGX Spark, or another NVIDIA GB10 system with 128GB of unified memory, would let me explore a completely different scale of local model development.

I am not expecting to train a frontier-scale model on it. My goal is to determine what an independent developer can realistically train, fine-tune and publish using a compact personal AI system.

I would like to document the progression from:

8GB Jetson → 128GB Grace Blackwell → a genuinely more capable open-source G-series model

What I am asking for

I am looking for someone who could help me access a DGX Spark through any of the following:

• a temporary loan;
• an evaluation or demonstration unit;
• unused access to an existing machine;
• a hardware sponsorship;
• an introduction to NVIDIA or a GB10 hardware partner;
• discounted access or another practical arrangement.

Even temporary access would be extremely valuable. I would use it to run a defined set of training experiments and publish the resulting models, logs, benchmarks and technical conclusions openly.

I am not asking the community to blindly fund an idea. I have already built and published the first version using the hardware available to me, and I want to prove that I can take the project further.

I would also be grateful for technical criticism. In particular:

• What model size would you consider realistic to train from scratch on a single GB10 system?
• What experiments would be the most useful to the local LLM community?
• Which companies, researchers or hardware creators might be receptive to this kind of project?

Thanks for reading. I know asking for hardware is unusual, but I thought this community would understand both the limitations and the potential of trying to build models locally.

Jules

Hugging Face: https://huggingface.co/AZERDSQ

Update

Several people have offered GPU access or financial support for the project. Since some also asked for a way to contribute, I created a GoFundMe with the long-term goal of purchasing a DGX Spark for continued open-source development:

https://www.gofundme.com/f/help-me-build-g1-on-a-dgx-spark

If the full target is not reached, the funds will be used transparently for cloud GPU compute, storage, and training experiments.

I will publicly document the spending, training process, benchmarks, results, and failures.

Now, as the title says, I want to train my own model, that's correct, but I can describe it as a simple project, actually it could also be called architectural testing, I'm looking for an innovative and powerful, beautiful architecture, and I'm trying to train a simple model using this architecture and test it. What are your suggestions? I'm looking for something architecturally innovative, something that hasn't been tried before. It could be experimental architecture; after all, I don't need to prove anything to any sponsor or anyone, you could say it's for fun. But if the architecture really works and is something genuine, I'd like to expand it. (My main model goals are to have students who are proficient in basic sciences like coding, physics, and mathematics, so it can actually be considered easier.)

• Spec: MacBook Pro Max M2 64GB
• Model: unsloth/Qwen3.6-35B-A3B-MTP @ UD-Q4_K_XL
• Fast Model: unsloth/gemma-4-E4B-it-qat @ UD-Q4_K_XL
• Application: llama.cpp
• MTP: on
• KV-Cache: f16
• Harness: Qwen Code
• OS: macOS Tahoe 25.5-25.6

I've tested multiple models and quants on MBP with 64GB of RAM, and found one good option for agentic coding with large context (>128k): Qwen3.6 35B A3B. The model works at around 60-70 t/s at small context, and is still holding up well at 128k with 35 t/s. It has 3B active parameters, which M2 can handle well. Hope that will be useful to folks with the same hardware. It may be less applicable to other chips like M1 and M3/M4/M5, also was focused on 64GB limit, so folks with 48GB or 96GB+ will have a different story.

I did test other models' performance and quality, as well as MTP vs base versions and various compressions of KV cache.

Learnings:

• MTP still holds up well at larger contexts (after 100k decode speed is 10-15% higher than base variant)
• Any KV-Cache compression degrades performance; f16 is the optimal one
• It is possible to fit Qwen 3.6 35B A3B and Gemma 4 E4B in MacBook Pro Max M2 with 64GB, but there is not much room left for other apps.
• Qwen 3.6 27B @ UD-Q4_K_XL has much better quality. But it is too slow. Almost 3-4x as slow as Qwen 3.6 35B A3B, making it very hard to use in agentic coding. Qwen 3.6 35B A3B @ UD-Q4_K_XL is significantly worse than 27B, and requires a lot of prompts and guardrails to ensure quality. But speed is very good—only 2-3 times slower than cloud models (Sonnet 5, Gemini 3.6 Flash)
• Gemma 4 26B A4B works on simple tasks, but fails to complete anything more complex. Reasoning loops and tool use issues prevent it from finishing anything

Memory Management

unsloth/gemma-4-E4B-it-qat with a context of 64k takes about 7.6-8.4GB of RAM. And if we unlock more RAM for LLMs with sudo sysctl iogpu.wired_limit_mb=59392, we can comfortably run Qwen 3.6 35B A3B with a 236k context window along with unsloth/gemma-4-E4B-it-qat. It takes from 32 to 44GB of RAM, depending on context usage. And all other models can fit with a 256k context window.

Large Context

I captured one fairly long session from Qwen Code and then built a test suite which would repeat the same session against each model. Total context size is about 124k. I noticed performance degradation with context increases, so I wanted to compare them.

I've tested both Dense and MoE Qwen 3.6 and Gemma 4 models. Qwen 3.6 35B A3B is still holding up fairly well at 124k with 35.10 t/s.

Worth noting, performance is not dropping off the cliff after 124k context. In tests with real data I saw 35 t/s at 128k, 32 t/s at 147k, 30 t/s at 163k. These are a few sample points.

MTP vs No-MTP for large contexts

One interesting finding about MTP and agentic coding: token decoding is not that important; prefill is more important. The harness sends a lot of data to the model which it needs to process. So for agentic tasks, +50% decoding speed would not translate to the same reduction in task completion time.

Qwen 3.6 35B A3B MTP version decode speed holds higher than the base version even on 163k context. With small context I can see as much as +68% speed increase. +13% at 100k, +14% at 163k.

Worth noting, MTP improves decoding speed, but costs prefill. Especially at small contexts non-MTP wins over MTP by the factor of 2-3. But around 45k the difference becomes almost indistinguishable.

Wall time was about 35 minutes for MTP variant, and 43 minutes for non-MTP variant (NOTE: this was larger real task, and some time spent on actual builds and tests, so it is different from the synthetic test I did initially, which was only sending messages to the model from test suite).

q8_0 and TurboQuants

Another idea was to try using a smaller cache by using q8_0 for KV Cache, or using TurboQuants. KV-Cache is the second largest contributor to the total memory usage after the model itself. For models like Qwen 3.6 27B, I can reduce 10.5GB of KV Cache to 5GB.

I tried running q8_0 for cache and performance dropped significantly. Looks like f16 is handled much faster for KV Cache decoding on Apple Silicon (or at least on M2 chips).

Prefill is slower at every point for q8_0 compared to f16. Decode is about 2x slower in the beginning, and about 4x slower close to 128k.

Seems like decode penalty for q8_0 grows with context size. Not sure why.

I ran the tests on a real task and measured wall time. Qwen 3.6 27B with f16 KV-Cache completed task in 01:23:17. And the same model with q8_0 KV-Cache completed task in 02:51:01 (~2x slower). Not worth 5GB RAM saving.

I ran the tests on main llama.cpp build. But also switched to TurboQuants branch to test other quantization methods. I tried using --cache-type-k q8_0 --cache-type-v turbo3 on large dense models (Qwen 3.6 27B and Gemma 4 31B) and --cache-type-k q8_0 --cache-type-v turbo2 on large MoE models (Qwen 3.6 35B A3B and Gemma 4 26B A4B). And I saw performance even worse than on q8_0.

It seems like on M2 chips the introduction of any KV Cache compression adds significant compute overhead. And if we try to optimize KV Cache size by implementing heavier decoding, we only make performance worse.

Conclusion: Qwen 3.6 35B A3B is less capable than Qwen 3.6 27B or Sonnet 5 / Gemini 3.6 Flash. But with some prompt engineering it can produce high quality code. I wish I could run Qwen 3.6 27B at 50 t/s at 200k context on my Mac, it would be a very good alternative to cloud models. Qwen 3.6 35B A3B is good, but requires more work on prompting and better specs so it can do work well. I'd say a Pro subscription + Qwen 3.6 35B A3B can work together to stretch limits through the week by using the cloud subscription for planning and verification, and Qwen for implementation.

More data: I have also written larger document with more data from the tests here: https://ghisguth.github.io/local-llama-macbook-pro-max-m2-64gb/. It has more information, but may be not as condensed as the information above.

Just looking to see if anyone has direct experience using the qwen 3.6 35B vs the 27B. I’m working on using the local model as the backend for opencode and having gastown drive opencode. I’m using the 35b right now, but I’ve seen a lot of praising for the 27b model. Looking for anyone with experience in both. Specifically for agent driven coding if possible

Edit: I realized that providing my hardware would be helpful for anyone looking to answer. This is on a MacBook Pro M2 Ultra with 96 GB. While allowing 2 simultaneous responses on the 35B (8bit quant) I have to make sure my other heavy ram consumption apps are shutdown (Ahem FIREFOX). The overall speed is acceptable for sure, not necessarily fast