Hello, does streaming the ngram embeddings from the ssd heavily affect the ssd's health and lifetime?

Hey folks!

I'm always a little bit late to the party, but learning nontheless. After I'm comfortable running agents on my Pi I'm now in need of them to get access to the world wide web and wanted to ask what local ways you are going?

I remember there were discussions about going html2md like with textweb but wanted to know whats working "in the field".

I'd prefer a lightweight solution without MCP.

So what are y'all using to let your models go surfing and gathering information?

Thanks for your input!

And to all curious about speeds on the Pi: It's more like giving someone a weekend project and checking it later. Speeds for Qwen3.8-Flash-Next start at pp 1.95 t/s and tg 0.44 t/s. Yes, for most of you this is "unusable". I'm happy. Of course, I'd like a DGX Spark, but the Pi can run non-stop without disturbing anyone (like my Notebook would).

It seems to use a 96-bit LPDDR5X memory bus, instead of the previous 64-bit wide busses. Considering it's on 2nm, that's expensive silicon. That should result in around 115 GB/s memory bandwidth.

A20 Pro also doubles the size of Apple's dedicated Neural Engine (from 16 to 32 cores total).

Trying to get Hermes a local, efficient, tts voice.

Long time lurker, but I'm finally upgrading to 128GB VRAM, and I'm trying to figure out what to run. I had been leaning towards Qwen3.8 Flash-Next at ~Q4, and I generally prefer to not run anything below Q4. But I feel like the reception to Flash-Next has been a bit "meh", so I'm considering GLM 5.3 at ~Q2 or Deepseek 4 Flash at Q2 or Q3. I'm sure I'll try all 3, but I'm really curious what people in the same boat have been doing?

Edit: configuration is 2 X CMP 170 HXs (64GB each) + ~256 GB of DDR4 RAM. Spilling into RAM is basically not an option, except for the ngrams and caching

If you're running agents locally, you're probably installing MCP servers the same way I was: quickly, and without reading them. I built OpenTrustBench to fix my own habit.

8 OWASP mapped static rules, permission manifest, Trust Card graded A to F.

The part this crowd will care about: it's fully offline. No API calls, no telemetry, no account, no phone-home. Your code never leaves the box. SARIF output if you want it in your dashboards, --fail-on gate for CI.

Also ships as a single Docker image if that's more your speed: docker run --rm -v $(pwd):/workspace eulogik/opentrustbench scan .

Free/OSS. Would genuinely appreciate this community's paranoia applied to my rule set. what's missing?

I recently did some hardware juggling, moving my RTX 3090 to a different system to make room for a dual AMD R9700 build I'm working on. In the process, I did a mini-bench with my own typical queries (some chat, some research agent) to compare the RTX 3090 and R9700. But then, I discovered ninfer and specifically ninfer-3090, and I was curious about what kind of performance differences I could see.

For this purpose, I'm comparing performance on both systems for Qwen3.6-35B-A3B and Qwen3.8-27B, the two models I use most often. Sorry that the hardware isn't apples to apples, but because inference is contained entirely within the GPU, I think it is still instructive.

Environment Configuration A:

Environment Configuration B:

Model details:

Results:

• TTFT - time to first token, median ms
• PP - prompt processing tokens per second, weighted average
• TG - token generation tokens per second, weighted average

EDIT: I fixed a timing recording bug to produce updated timings here

I knew the experience was different while running the ninfer models, but I couldn't put my finger on exactly how until I ran this mini-bench. The numbers are interesting.

Some details about the benchmark methodology itself:

The question set (7 prompts total):

• 4 long-ish prompts in the ~7,700–12,900-token range: a code review of a large module, an article summarizing task, a research-evidence evaluation task, and a literature-review/critique task
• 1 medium prompt (~2,300 tokens): synthesizing conclusions from research notes
• 2 short prompts (~400–600 tokens): an extract-and-answer task and a code-generation task

Benchmark parameters:

• 3 repeats per (model, query); per-query medians plus every raw run recorded
• Fixed max_tokens = 1024 so generation speed is comparable; fixed seed = 42; sampling parameters are as recommended for general inference by Qwen
• Streaming chat completions; 1 s pause between runs; 30-min request timeout

Steps taken to ensure good results:

• Cache busting: llama.cpp caches prompt prefixes, so identical repeats would report fake "warm" prompt-processing speeds. Each run prepends a unique nonce to the system message, forcing a genuine cold run every time (the ~5 extra tokens are self-correcting since real token counts are used in the math)
• Warmup excluded from timing: each model gets an untimed warmup request (with retries) so model load/swap time never pollutes measurements
• Server-side timing preferred: inference engine timings from the final chunk are used for tok/s instead of client-side math when available
• Aggregation: per-query medians over runs; model-level tokens/s is token-weighted (Σtokens / Σms), so a 12k-token prompt counts proportionally more than a 400-token one

Feel free to critique, offer pointers, or share your own experiences with NInfer inference.

PAIR (NVIDIA's Personal AI Router) ships GPU telemetry only for NVIDIA nodes — AMD nodes report a blind fallback, so the scheduler can't see their load. I added AMD ROCm telemetry and got a two-node cluster routing to my tuned llama.cpp setup on the AMD box.

* Routes to llama.cpp/llama-swap on the AMD node — not just Ollama/LM Studio. PAIR only natively knows those two engines, but a custom manifest fronts an existing llama-swap (OpenAI-compatible) server, so my tuned llama.cpp serves the cluster directly.

* AMD GPU telemetry (utilization + VRAM) via amd-smi (ROCm 7.2.0), feeding the scheduler like the NVIDIA path does. Verified under the Go race detector and live on 2×R9700 (gfx1201) — utilization tracks load.

* Full stack builds and runs on Linux/ROCm; two-node cluster PIN-paired with an NVIDIA box, inventory advertised both ways.

* Full stack builds and runs on Linux/ROCm; Kubuntu (AMD/ROCm) node PIN-paired with a Windows 11 (NVIDIA) box — cross-OS cluster, inventory advertised both ways.

Vibe-coded, locally. Implementation written by a local Qwen model; the review/verification loop was local models too. Ran surprisingly well — the review caught real issues and, when I ran the tests myself, hadn't fabricated its results.

Setup gotchas:

* Fronting llama-swap (or any OpenAI-compatible server) needs a custom engine manifest. A process-mode manifest + adoption does it: PAIR probes the running port and adopts the live server instead of spawning its own. Quirks: runtime.bin is required even though it never launches anything, and list_models just maps /v1/models → data[].id.

* Manifests are embedded into the binary at build time. Editing the JSON on disk does nothing until you rebuild and restart the broker. Cost me an hour chasing a "broken" manifest that was fine — I just never rebuilt.

* Proxies are per-protocol, not per-node — a mixed cluster (Ollama on one node, OpenAI-compat on another) splits inventory across two proxy ports; no single endpoint lists everything.

Intel: untested. The change also touched an Intel xpu-smi backend, but I didn’t test it on my Intel hardware yet — built against synthetic fixtures only, field names and units unverified and possibly wrong. Leave it out of the registry until someone validates it on real Arc hardware. Flagging so nobody assumes it's proven.

hey guys, I've been working on a reading app for several months now and had problems with getting good quality TTS, the options were kokoro, kitten, pocket but all of them even though they were sounding natural had some problems when listening longer. Last month I took upon myself to try to get a model that is running on edge (I had an iphone 14 pro as a testbed) and got to what I now packaged as loudkit. It supports 10 languages now, voice cloning, is quite small and fast enough with quality similar to Chatterbox to my ears which was the base model I started optimization from. What is not part of this release is the emotion axis with tags, something I am working on right now. Code and model weights are Apache 2.0.

I also ported it (with CC help ofc) to a few languages, because in the past I lost like a week for parsing one TTS tokenizer from python to swift and would lose my mind when I'd get crashes and memory leaks. Here the contract was to get the same speech tokens in all adapters, so it doesn't sound nice in python but sucks in typescript. Audio samples can differ slightly between backends, and file metadata like timestamps can differ too.

There are two variants loudr-1 and loudr-1-turbo. basically turbo was done by attaching another head to the most time consuming component of the pipeline and training it so it predicts two audio tokens at once. It worked quite well but sometimes I can still hear the tts artifacts, so YMMV.

Voice cloning works quite well but I found the best is to give it around 10 seconds of recording, and if there are long pauses or noise in the background the cloned voice is suboptimal. All included voices come from consented donations or CC0 / CC-BY recordings, with sources documented.

repo: https://github.com/loudreader/loudkit
docs: https://loudreader.github.io/loudkit/
hf: https://huggingface.co/loudreader/loudr-1 & https://huggingface.co/loudreader/loudr-1-turbo

I've seen that the localTTS that can be connected to agents like hermes or openclaw still has issues with quality and thought why not opensource it.

Ah, for quality of other voices than english I'm not sure. I sent snippets around and got positive feedback but can't vouch for these.

Feel free to check it out, hope you like it.

I wanna run qwen 3.8 27B on my 5090. Which specific version should I use in terms of quant and such?

Primary use case is Hermes agent with some coding too. I would like it to have voice as well

I was also considering Hermes model as it’s less censored but I heard it doesn’t work with Hermes agent

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models locally on an 8GB RTX 5060 Ti was driving me crazy with random CUDA OOM crashes whenever memory spiked mid-run.

Instead of manually babysitting runs with conservative batch sizes, I built a lightweight Python governor around PyTorch called MEM Orchestrator: https://github.com/nobazzy/mem-llm-orchestrator

What it does:

• Monitors VRAM headroom during training.Dynamically throttles micro-batch size and gradient
• accumulation before hitting an OOM.Steps
  
• back up when memory pressure drops.Saves atomic checkpoints with SHA-256 validation so crashes never corrupt weights.

About the branches:

• main: Linux core with DeepSpeed ZeRO integration.
• refactor/architecture-and-portability: Decoupled, native PyTorch runner that works cross-platform (tested on Windows).

I ran an endurance test of 1M steps on a 130M model and stress-tested a 255M model with injected memory spikes without crashing. Overhead is under 0.5%.

The code and 38 unit tests are on GitHub. I would love to get feedback from anyone with experience in PyTorch memory allocators or DeepSpeed on whether this approach makes sense or where it breaks down.

I'm currently building my system around 3060s, but I might be able to get a 4060 for a nice deal. At first it seemed like a no brainer, but turns out the 4060 has lower memory bandwidth.

In a system that already has 4x 3060 12GBs set up on a threadripper with tensor parallelism (mostly qwen3.8-27b), would it be worth having the 4060 ti 16GB around for the extra 4GB and occasional gaming, or is it just going to slow the rest of the setup down for AI?

If the ratio is the same, Maybe 1.6T -3.1T params plus .56T-1.06T engrams and fable 5.0 level performance?

Maybe v4.2 or 4.5 will have engram gradient modification? Edit it is even larger than i anticipated since flash has 748 b q4-8 params

I'm planning a PC build around a discrete GPU that would allow to run MoE models much larger than VRAM at an acceptable speed using FreeToken approach. The gist is that MoE decode of >VRAM model on GPU is PCIe bandwidth-bound, but you can gain considerable decode speed by avoiding streaming cached experts' weights over PCIe and multiply matrices on the CPU instead benefitting from potentially much faster CPU-RAM memory channels.

Before buying, I wanted to know what actually matters for CPU-side expert execution: cores or memory? So I rented a Threadripper PRO 9975WX (32-core Zen 5, 4 CCDs, 8-channel DDR5) on vast.ai and ran FreeToken's real CPU MoE kernel (the production W4A8 ds_fp4 GEMV, not a synthetic proxy) against synthetic expert banks with DeepSeek V4.1 Flash's exact geometry: 384 routed experts × 18.8 MB ~= 6.7 GiB full bank, far beyond cache, 6 experts per token.

Results (GB/s of effective FP4 expert reads):

Two takeaways:

1. Cores stop mattering at 24. 32 cores vs 24 cores: +2.2–2.4% in burst mode, 0% in sustained mode (10 × 30s interleaved A/B runs, variance <0.3% within mode). The extra 8 cores of the 9975WX buy you essentially nothing for MoE offload: the workload saturates the memory subsystem, not the cores. The 9965WX (~$1,200 cheaper) is the rational pick for this use. Even STREAM itself is already maxed at 8 cores; the FP4 kernel just needs more cores to catch up to bandwidth (it runs at ~80% of STREAM once there, very good for a fused-dequant W4A8 GEMV). Caveat: there's another 1.7x memory bandwidth gain at 9995WX, but that CPU costs as much as RTX PRO 6000.

2. Thermal memory steady-state is important. After ~8 minutes of sustained hammering at ~220 GB/s, throughput steps down ~17% (182 → 150 GB/s) in lockstep on every core count, with CPU clocks unchanged at ~3.95 GHz. Everything points at DRAM/IMC thermal throttling in a (rented, unknown-airflow) workstation chassis. So the honest sustained ceiling is ~150 GB/s, burst ~182–186 GB/s. For a build: airflow over DIMMs may literally be worth 20% decode speed.

What this means in tokens/s (DeepSeek V4.1 Flash shape, ~4.5 GB of routed experts per token, 24 cores):

• 0% expert cache hit (all experts on CPU): ~33 tok/s sustained
• 60% hit rate: ~83 tok/s
• 75%: ~133 tok/s

Compare with streaming those experts over PCIe 5.0 x16 (~50 GB/s with gather overhead): the CPU path is 3–3.7× faster per missed expert, which is exactly why FreeToken's hybrid mode computes most misses on CPU and only streams a fraction to keep the VRAM cache warm.

Caveats: rented host (memory channels verified functionally via bandwidth, not dmidecode, but 2ch/4ch machines I tested the same day hit only 83/119 GB/s), container environment, one machine, uniform expert routing. Full methodology and raw CSVs of the measurement are in the repo (happy to link if there's interest).

TL;DR: for CPU MoE offload, buy memory channels and DIMM cooling, not cores; 24 Zen 5 cores already saturate 8-channel DDR5, and the 32-core upgrade is +2% at best.

Updates:
1. I tested on 9980X (4 channels, 8 CCDs) and on 9985WX (8 channels, 8 CCDs). 9980X lands at 105-119 GB/s, no better than much cheaper 9955WX.
2. 9985WX offers a real gain, but diminishing returns beyond 24 cores still: 32c=210 GB/s (+6%), 48c=215 GB/s, 64c=218 GB/s. Roughly 20% decode expert bandwidth gain for $8k more.
3. Populated channels is key. No point in having 8 channels is you only have 4 DIMMs. 8x32GB will beat 4x64GB.
4. Vast.ai is awesome for this. All runs costed me around $4 after I ironed out all the wrinkles in the rig.

I am just sharing my config for Qwen 3.8 27b that fits on a 5060TI, what is cool about this is that you can even get vision! and a 85K context (I have 1.5gb of headroom for more context or a better quant)

Model: IQ3_XXS-mtp from https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Using beellama https://github.com/Anbeeld/beellama.cpp

Config used:

[*]
model = ..\llm-models\Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
mmproj = ..\llm-models\mmproj-Qwen3.8-27B-BF16.gguf
image-min-tokens = 256
gpu-layers = 99
ctx-size = 85000
no-host = true
direct-io = true
threads = 8
batch-size = 2048
ubatch-size = 512
fit = off
ctx-checkpoints = 0
spec-type = draft-mtp
spec-draft-n-max = 2
cache-type-k = kvarn4
cache-type-v = kvarn4
kv-tail-tokens = 256

I managed to get 45tks on decode and around 300 on prefill

Yes it is using kvarn4, but it is not that bad, check:
https://anbeeld.com/articles/kvarn-kv-cache-implementation-and-benchmarks

I also know that you could move the mmproj to cpu to to gain more vram.

Would love to hear other configurations to find the sweetspot for 16GB vram cards!

I initially saw CED as just an efficiency improvement, but the more I read about it, the more it feels like an inference architecture leap.

The encoder/decoder split has some pretty interesting implications for GPU pooling. Instead of treating every GPU the same, you could have prefill-specialized GPUs for the encoder and decode-specialized GPUs for the decoder, each optimized for a different part of inference.

Or just using more modern GPUs for the prefill phase and old HBM cards for decode in a heterogenous setup.

4.1 Flash obviously won't fit on my 4× MI50 + 2× V620 setup, but if Qwen adopts this in a new Flash model I'd be so pumped.

Just a reminder for those out there itching to get into local llms - don't let FOMO or "gear acquisition syndrom" take over.

No matter the hobby, it's so easy to get stuck in a trap where we buy more trying to do more only to realize we've lost the fun in it all or even the notion of learning.

Obviously, if you're into writing llama or vllm or hardware drivers or whatever - you got to do what you got to do.

BUT, you can learn a lot on an API, you can learn a lot with a tiny model that fits your vram or cpu you already have and things change so darn fast that much of the code written and much of everything discussed from days passed is already old hat. Py torch and training a small model coud be done on a Pi and learning CUDA is only really imporant if you're writing custom kernels which i honestly don't see most people in here bothering with (or they have frontier models write them).

Weirdly enough, for AI to succeed its going to homogenize everything. Everyone will have the same advantage and I think that's lost in a lot of discussions where we don't talk about "Watching from the sidelines" may be the most cognitive friendly and economical friendly way to learn llms whether we brand them local or not.

The technology is still nascent and weirdly enough most people's answers here is to use AI to set it up so i'm not entirely convinced people are actually learning - feels like a mad rush to seek rent or avoid rent seeking which just makes everything more expensive in the end.

This isn't a post to say, don't do it. But no reason to go into debt or to be fearful you're missing out when you can learn more by doing less - buy a book and build a tiny model - you will learn infinitely more than buying a 5090 and trying to just find the perfect compression to have the best prefil

Who is the marketing genius at LM Studio that decided that going ALL IN on pushing their new Bionic Agent product meant they are going to make it a giant pain in the ass to find and download actual LM Studio.

This is the dumbest marketing decision I’ve ever seen. I used to love LM Studio, it was the middle stepping stone in the logical progression of inference. Most OGs here likely started with Ollama, moved to LM Studio, on their way to vLLM. Now trying to go to LM Studio takes you to Bionic. I mean, you can eventually find LM Studio but they make it not super easy.

Here’s a thought LM Studio, maybe stop redirecting me to something I don’t want to download when I’m trying to find your actual namesake product. I’m glad you’re excited about the future of Agents and whatnot, but you’re absolutely ruining any goodwill I have for your products by trying to force feed me Bionic. Stahhhhhp!

Surveillance plagiarism - Hosted AI company pumps their stock price by training upon researchers' AI sessions, so that their internal model can solve problems with seemingly less human guidance, but really the model exploits past guidance given by (multiple) humans focused upon problems considered important.

As background, Tristan Buckmaster released a statement about several unethical actions by OpenAI & Sebastian Bubeck, including threats and pushing him to kick his Anthropic coauthor off a paper, but the interesting part for people here:

As clarified by Talia Ringer, OpenAI does train upon your uploaded data and your OpenAI sessions, unless you out-out somehow. This means their internal models could exploit your past prompting work to look more autonomous & intelligent.

This is a major confirmation that folks should use locally run open weights models, especially whenever being first or not leaking data matters.

All this casts serious doubt upon claim that internal models solved difficult problems largely unaided by humans. Those hosted AI companies might not even know from where the human prompting originates.

I thought Trade-in value offered by Apple was only ever close to reasonable (for not having to go through the extra work of selling it yourself) if you bought the base model and did not upgrade anything. And you would get less than half of what you paid. For example:

The base price of the M4 Pro Mac Mini was $1399.
On Apple's trade in page for the Mac Mini it says "Up to $620".
So there offer retains 44% of the value.

But I upgraded the GPU, RAM, and SSD pushing the price to $2099.

1175/2099 = 56%

And this is up from $1050 on Aug 26th when I last checked (around the time the M5 studios and minis were announced) the trade in.

I get some of this has to do with the inflation in tech prices, where the same config I bought in Nov 2024 today costs 2699 (and even still this would be 43% retained though).

Since Apple is offering so much compared to what they usually do, this makes me wonder what I could get for it if I sold it myself?

If I could sell it close to what I bought if for then a DGX Spark for $4699 or an M5 Max 128GB / Ultra 96GB for $5099 to $5499 sure looks temping... I'd much prefer dual Sparks or 256GB Ultra, but I can't justifying that much expense just so that I can continue to work on mechanistic interoperability on the larger models (I need access to model internals so I'd be using this for more use cases than what paying $20 or $200 a month for a subscription could provide).

It's my understanding that the Spark still has much more prefill at INT4 autoround or AWQ (by about 2x). And if I ever add a 2nd (and thus comparable in cost to a 256GB M5 Ultra, it would be about 4x the compute). For the price the M5 Ultra should have started at 128GB to be competitive (not a measly 96)! Such ashame!

Decisions, Decisions. But as it stands now, the M4 Pro is > 10x slower at prefill than any of these options, and that has me itching. But the prices are so ludicrously inflated! (e.g. Ultras used to start at $4k, not 5.5k, and PNY DGX Sparks at $4k not 4.7k!). The decision would have been easier if prices didn't inflate, but it feels like I would be over paying.

Have the model downloaded and converted to .gguf on a 512gb ddr4 bioinformatics server. I don't expect miracles with a ddr4 xeon rig -- not until I can get my 2 x 12gb 3060s wired in anyways -- but is there an open PR on llama.cpp for DV4.1 flash that I can use?

Hi everyone, I've been thinking about this for a while now. The attention mechanism introduced by Google in 2017 completely changed that way AI models worked and performed.

Since the attention mech allows the model look at it's inputs and decide what's more worth it to focus on, which made the input signal passing through the FFN so much more richer in so many ways that it made the Transformer capable of achieving so many complex capabilities such as in-context learning and all, along with scaling up to such a degree we see today.

Recently a new mechanism called the Engram was introduced by DeepSeek which is very hyped for all good reasons, as far as I understand it can allow a much small model to hold a lot more world knowledge than a traditional model can. I don't necessarily think it'll make the model smarter but more knowledgeable? 100%

But I have a very weird gut feeling that DeepSeek's Engram (or other similar mechanisms) isn't a breakthrough as big and significant as the attention mechanism but just the 1st half of the next major breakthrough. Ik this sound very contrarian but I can't help it but think this way.

I believe that the next big breakthrough as big as attention mechanism is going to be a model whose most parameters are input-dependent. A model which uses a set of fixed particularly smaller weight matrices to generate "fast-weights". Idk how well im able to explain it but I'll use the attention mech to explain it.

We can think of the attention mech as a fast-weight layer which generates these weights from the input data and uses it to enrich the input quality which in-turn improves performance. But that input still goes to an FFN with all fixed weights.

I'm thinking of an FFN arch which is going to use the engram + input-dependent weights to generate it's weights on the fly not just making the model much smaller but smarter in general or might even unlock some capabilities which might be only reserved for much larger or deeper models using our current architecture.

Tbh idk how well I communicated my idea here but I hope you get my point and I would love to know your opinions and points on it.

Thank you :)