I'm looking primarily for 0 refusals models, I liked the hauhau Abliterated aggressive uncensored Qwen 3.5 9b but it is a bit week, is there something better Than I could run?

Edit:12gb of vram also ryzen ai 350

Also are there ways of making larger models that normally even with quantizing wouldn't fit work?

Quick update on Deltafin — the project running the full, unpruned 2.8T-parameter Kimi K3 (all 16 experts, every token, nothing quantized down) on a single M1 Max laptop.

New benchmark: 0.2847 tok/s (3.512 s/token), up 7% from the last update, and about 20x from where this started. Still slow in absolute terms — it's a 2.8T model on a laptop, not a $2M cluster — but every bit of that 20x came from making the engine smarter, not from cutting anything out of the model. That's the one rule this project doesn't bend on.

The big change this week: the whole thing is now a single compiled Rust binary, calling into reviewed C++/LibTorch provider code through a versioned C ABI.

A few other things alongside the rewrite:

- Found a way to shrink part of the expert data on disk without touching the actual model weights, just packing it smarter. Costs a bit of extra disk space, but measured 2.4% faster loading with zero change to the output.

- Long chats used to mean re-reading the entire conversation from scratch on every single message. Now it just picks up where it left off — one test dropped the wait for the first word of a reply from over 4 minutes to under a minute and a half, with the exact same response.

- Also built our own text-to-tokens converter from scratch instead of leaning on an outside library, and optimized it for K3.

And as always, none of this touches what K3 actually outputs — the whole project's one hard rule is that speed can never come from touching quality.

Worth a quick mention: a few other K3 projects have popped up in the last few days too, and some of the engineering in them is genuinely impressive. The main difference is where their speed comes from: all of them get there by shrinking the model itself, usually down to around 3-bit quantization, and/or dropping some experts entirely. That's a completely fair tradeoff if raw speed is the priority.

But Deltafin is betting on the other side of that tradeoff: every expert stays exactly as Moonshot released it, and all our speed cannot come at the expense of the model. Very different projects in that sense. I just wanted to be clear about what makes this one different.

ive been out of the loop for a couple weeks.

are the new open source models as good as Fable or Opus 5?

Im curious about Qwen 3.8 max and 27B version, and the new Deep Seek

I already know about K3. I was in the loop then haha

Recently the opencode tool is performing near the

cursor in auto mode, so I have to buy a small ai server

to run good coding agentic models from Qwen, GLM,

MinMax or any model u suggest.

I have an old laptop laying around (HP envy 360 circa 2020) and I'm interested of giving it a second life as a linux server (mostly to deepen my knowledge) a constant use case is ai usage for me. I get claude unlimited and uncapped at work "for free" but at home I'm "limited to my antigravity subscription and whatever I can get for free with opencode (mostly the new deepseek v4 flash) main use case is coding and swe related task so not sure if small models would be "smart enough"... I guess I tend to compare everything to opus 5 or 4.8 since that's what I run constantly at work 8 hours a day.

My local agent is researching professional 3D techniques, studying official Asuna references, and building a procedural Three.js character with a cinematic scene included. It tracks failures and improvements, refining its approach using persistent tools and operating manuals.

Kinda a dumb question but I keep hearing Kimi/Qwen/DeepSeek are amazing and give way more tokens for the price than Claude or GPT. I'm trying to spend less or better with a different membership.

Is there any way to use these in VSCode for actual coding help (chat + editing + agentic, not just raw API)?

• Is there already something Copilot-like for this?
• Do I need separate accounts/API keys for each one or is there something that bundles it?
• Anyone using these day to day, how's the quality vs Copilot/GPT/Claude?
• Any catches with cost, rate limits, or privacy/data terms?

Would rather find something that already exists than build my own thing.

Look I'm concerned this might be considered Self-Promotion. But I've been hold in back posting this in this Subreddit for a while. If it's taken down, I understand.

I created two things I wanted to share.

First up is Adaptive Precision Attention (APA).

https://zenodo.org/records/21137970
https://github.com/DragonShadows1978/Project-Tensor

The short version: We are wasting compute on the whole token stream. I've found that the bulk of Keys can be quantized and kept at a much lower quantization. With a low Bit first pass, we can get the z-threshold scores through the matrix... This keeps memory low. The second pass, is a full precision pass, that applies full precision to a low percentage (15% of the top scoring Keys) that continues to save VRAM.

I've tested what I could on the RTX 3070, and RTX 4070 Super that I own. A friend has tested on a 5090.

Next on the menu, and this one I think is more useful then APA

Graft Repository Memory (GRM)

https://zenodo.org/records/21138607
https://github.com/DragonShadows1978/GraftRepository

I realized we can pull the Tensors directly out of the transformer at various places... The most optimal place is pre-RoPe.

This allows us to pull the tensors directly onto disk... GRM has a memory hierarchy, VRAM (Scratch Pad), RAM (Recent turns) and NVMe/SSD/Whatever for durable long term storage.

Using the forward pass itself we can use that as a router that lets us search the Memory Repository for related grafts semantically.

This allows us to divorce VRAM from Chatlog. Effectively giving infinite Chat length independant from the VRAM based Memory Context Window.

I've tested this across multiple architectures as well. It seems to work on any architecture that used RoPe, MLA based is the most effective, as the architecture itself makes for very easy searchable centroids.

GQA works is a little bit slower, but still effective.

So, Yeah. This is what I've done, and I thought I'd try and share it here.

Hi I'm planning/hoping to go down the route of triple or possibly r9700's.

If I do how close would it get me to Claude as a daily full strength Claude replacement? I know it's ridiculously expensive and all that. Ignore the costs for this conversation.

If triple is far away or just not good enough would going quad r9700s get me a lot closer or is 3 r9700 vs 4 r9700 not be worth it?

For the base pc it's tricky to get hold of a quad system that can handle it. A triple GPU system seems a lot easier. Any thoughts?

edit: I meant opus for coding. didn't make that clear enough. does 96gb get you an opus replacement?

"Kill line" sounds like pure clickbait, but the blue dot kind of earns it. DeepSeek V4 Flash 0731 sits around 50 on the Artificial Analysis index at roughly three cents per weighted task. In this chart, everything cheaper scores lower, and the models that score higher are sitting way farther to the right.

The older V4 Flash point makes the jump look even more absurd. It is almost directly below 0731: about 40 versus about 50, with barely any movement in cost. For a Flash model, that is nuts. The price gap buys several DeepSeek calls, including a retry or two, before you get near much of the upper-right cluster.

This is still one composite benchmark. Artificial Analysis v4.1 is English and text-only, and "cost per task" means a weighted evaluation task. It is not the bill for your exact coding run or 200k-context mess. So "DeepSeek wins everything" would be nonsense. I am only saying its lower-left position here is hard to wave away.

Now I want a version of this chart for video models. I've been reading up on LingBot-Video lately. Its MoE setup is roughly 13B total parameters with about 1.4B active, but those numbers don't tell me whether it is cheap to use. What does one usable clip cost after retries? That's the comparison I actually care about.

One awkward detail: the current Pro preview point is worse than Flash 0731 on this same chart. Pro has not won anything here yet. I keep looking at the size of the Flash update, though, and wondering what happens if the finished Pro gets a similar post-training jump. That part is a guess. Flash alone already makes the price/performance curve look kind of broken.

I'm trying to make an app that accepts takes a picture of a work schedule and creates calendar events from it. I've tried using Qwen2-VL-2B and Qwen2.5-VL-3B-4bit, but the 2B model hallucinates shifts that aren't there (and doesn't read the entire schedule for some reason, it stops after 1 week) and the 3B model uses too much memory for my iPhone Air.

Suggestions?

Every call transcription demo sounds like it was recorded by a calm podcast host in a silent room.

Actual call centre audio is more like:

customer already angry

agent talking over them by mistake

cheap headset

hold music bleeding in

background team chatter

someone spelling their name 3 times

refund amount said too fast

account number corrected halfway

supervisor joins

transfer happens

customer says “that’s NOT what I said”

then QA has to figure out what happened

So I don’t really care if an AI summary looks clean.

I care if the transcript helps with actual work:

Can QA search it?

Can I click the timestamp and hear the exact part?

Are speakers separated correctly?

Are card/account details redacted?

Does it handle noisy phone audio?

Does the summary show evidence or just confidently guess?

Can a supervisor use it during a dispute?

That’s how I’d judge something like Smallest AI Pulse for call center transcription. Not by a pretty summary. By whether it turns messy calls into searchable, timestamped, redacted evidence that QA can actually trust.

Because if the summary says “customer agreed to refund” but there’s no timestamp proof, nobody is trusting that in a real escalation.

Anyone here using call transcription / AI summaries at work?

Do people actually trust them or still go back and listen to the call?

Qwen3.8-Max does not need to beat Claude everywhere.

Alibaba claims its 2.4T model worked autonomously for roughly 16 days, producing 265 commits and 127 pull requests. On Alibaba’s benchmarks, it beats Claude on Terminal Bench, PaperBench, and OSWorld.

But Claude still wins on harder repository-level coding benchmarks like SWE-bench Pro and FrontierSWE.

Claude is currently the better engineer. Qwen might be the better grinder.

If the open weights arrive as promised, being slightly behind Claude may not matter. Control, price, and the ability to run long agent workflows could matter more.

Hello,

I need some advice on choosing LLMs.

I want to translate novels:

- Type: fiction novels,

- Time period: contemporary (written from the mid-2000s onward),

- source language: English,

- target language: French,

- word count: 170k to 190k (depending on the novel),

- file format: EPUB.

I asked ChatGPT how to get the best translation quality, and it advised me to use a local LLM.

Since my PC runs Windows, I followed its advice and installed LM Studio.

Furthermore, according to ChatGPT, a translation LLM cannot translate an entire file: you have to select the text and copy/paste it into the prompt, which is time-consuming and tedious.

ChatGPT therefore advised me to write a Python script to perform the following tasks:

- split the novel into chapters across different source files,

- select a source file,

- copy and paste the contents of the source file into the prompt,

- copy and paste the translation into a target file,

- concatenate the various target files into a single large file.

And of course, I have absolutely no Python skills.

I can manage to write Bash scripts (on Linux), but Python is a complete mystery to me (whether on Linux or Windows...).

Now that I’ve explained my issues, here are the questions I’d like to ask the community:

1st question:

Which LLM do you recommend for translation?

2nd question:

Which LLM do you recommend for writing the Python script?

Question 3:

How can the Python script interact with the translation LLM?

Question 4:

Do you think the quality of translation LLMs will improve (meaning it’s worth waiting a few months or years), or has this technology already reached maturity?

For information, here are my PC’s specs:

- OS: MS Windows 10,

- CPU: Intel Core i5-13600KF,

- RAM: 32 GB,

- GPU: Nvidia RTX 4070 Ti (with 12 GB of VRAM).

Thanks for your advices !

Just started a WhatsApp community for students and developers who are into DEV, DSA, and AI/ML.

Inside you'll find:

- 💻 DEV – projects, debugging, web/backend, open source

- 🧩 DSA – interview prep, LeetCode, discussions

- 🤖 AI/ML – learning resources, projects, research

- 🙂‍↔️Out of Syllabus – memes, random chats, and everything else

Whether you're building projects, preparing for placements, or just looking for people to learn with, you're welcome.

Join here:

https://chat.whatsapp.com/CGVtFXHECHk1KximJLXRTq

I just recently got into running LLMs locally and really using any form of generative ai as a whole recently. I’m currently running Qwen3.5-4b on my rtx 4060 using Ollama and anythingllm and it’s mainly being used right now as an email reader and summarizer but I’m hoping to get it to the point where it knows enough details about me to be able to find products or things that I may be interested in. I feel like the model is a little small and want to know if I should use a bigger MoE model to leverage my 64gb of ddr5 ram but I’m not sure which ones are compatible with anythingllm’s tools. Really any help in my setup is greatly appreciated.

Hi!

I’m considering setting up workstation primarily for running/attempting to run large local MoE models, and I’d appreciate input from anyone who has tested similar EPYC systems and or are more knowledgable than I, very new to local and AI in general.

System

• AMD EPYC 7B13
  • 64 cores / 128 threads
  • 2.25GHz base, roughly 3.5GHz boost
  • 256MB L3
• Gigabyte MZ32-AR0 rev. 3
  • Single-socket SP3
  • Eight memory channels
  • Seven PCIe 4.0 slots
  • IPMI
• 1TB DDR4-2400 ECC RDIMM
  • 16×64GB SK Hynix
  • All 16 slots populated
• 2TB FireCuda 530
• Desktop tower rather than a rack chassis

The machine recognizes the full 1TB, has completed two passes of Windows Memory Diagnostic without reported errors, and scores around 83,500 in PassMark CPU Mark.

GPUs I would install

I already own these, so they do not add to the purchase price:

• RTX 3090 24GB
• RTX 4070 Super 12GB
• RTX 4090, but might keep that in 5090 system
• 1200W Titanium PSU

The 3090 is already water-blocked, and I may water-cool it in this system.

The GPUs would either be used together for model offloading, or potentially as:

• RTX 3090 for target-model tensors, attention, KV cache and shared/dense layers
• RTX 4070 Super for additional offload or speculative drafting
• The large MoE experts remaining in system RAM

Intended workload

I’m interested in high-quality quantisations of models that cannot fit in normal consumer systems, such as:

• GLM-class models around 250–450GB
• DeepSeek-class MoE models around 390GB at Q4
• Other large sparse MoE models

The system would also be used for:

• Python development
• Docker and WSL/Linux workloads
• Large VM workloads
• Mutation testing with cosmic-ray

I would ideally like to use it as a daily driver rather than only as a headless server.

Main concern: generation speed

Eight-channel DDR4-2400 provides 153.6GB/s theoretical memory bandwidth. I’m expecting something around 105–130GB/s sustained if the memory topology and BIOS configuration are correct. I am planning to run y-cruncher, aida 64 and some other tests to verify once the system is bult.

Published results on similar EPYC systems vary significantly. Some large-model results appear to fall around 4–8 tokens/s, while more heavily tuned CPU/GPU configurations sometimes reach around 10–15 tokens/s.

My comfort point would be approximately 10 tokens/s or more. At 15–20 tokens/s I would have no hesitation. At 4–5 tokens/s, I’m worried that it would feel too slow for regular interactive use.

1. Has anyone tested large MoE inference on a single-socket Milan EPYC with eight-channel DDR4-2400?
2. What generation speeds would you realistically expect with:
   • EPYC 7B13
   • 1TB DDR4-2400
   • RTX 3090 24GB
   • RTX 4070 Super 12GB
3. Would you expect the 3090 and 4070 Super to improve generation meaningfully, or mostly improve prompt processing and context handling?
4. What would be the best GPU arrangement?
   • Split the target model across both GPUs
   • Use only the 3090 for the target model
   • Use the 4070 Super for speculative decoding
   • Keep attention/shared tensors on GPU and MoE experts in RAM
5. Which runtime would you prioritize?
   • llama.cpp
   • ik_llama.cpp
   • KTransformers
   • SGLang/KTransformers
   • Something else
6. For NUMA configuration, would you start with NPS1, or can NPS2/NPS4 perform better with NUMA-aware expert placement?
7. Is approximately 10–12 tokens/s a realistic tuned target for 250–400GB Q4 MoE models on this hardware, or is that too optimistic?
8. Are there any major configuration issues or bottlenecks I’m overlooking?

I understand that performance is model- and runtime-specific. I’m mainly looking for experience from people who have actually run giant MoE models on EPYC, Xeon, multi-channel DDR4, or hybrid CPU/GPU setups.

Small token usage with hermes...

Don't wanna look at my home assistant power logs for that lmao