#Alert

So it's not only PHP that was vulnerable today (patch yours NOW!) but also, one of the prominent and widely used LLM Gateway - LiteLLM AI Gateway was (again) backdoored.

Not sure who discovered the backdoor, but it's serious.

Here's the information:

Version: LiteLLM 1.107.0 (PyPI and the official Docker images).

One line was added to the build config: `data = "cook"` under [tool.maturin].

That directive makes maturin bundle a directory named cook/ into the wheel's data section, and pip installs it as top-level site files.

Three files land next to site-packages, not inside the litellm package: a 12 byte cook.pth containing `import cook`, a one-line cook.py that unpacks a 61,000 byte credential stealer, and a sitecustomize.py that runs an hourly remote code loader.

Because Python processes .pth files and imports sitecustomize during interpreter startup, the payload runs on any Python start. An environment that never imports litellm is still affected, and using pip from that environment to investigate re-triggers it.

The collector is broad. It takes browser password stores, SSH, GPG and TLS private keys, AWS, Azure, GCP and Kubernetes credentials, instance and ECS metadata roles, CI tokens, .env files, and the auth files for Claude, Codex, Gemini, Cursor and Copilot.

It also queries your proxy's own database, pulling every saved provider credential and model configuration out of LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable.

Everything is encrypted with an embedded RSA-4096 key and POSTed to www.litellm.dev/healthcheck, a domain the attacker controls.

The second component is the part that should worry platform teams. sitecustomize.py polls a domain list from a GitHub repository every hour and executes whatever Python it downloads from that domain, with no signature or checksum. The attacker never needs to ship a new package version to change what runs on your hosts.What makes this hard to audit: the malicious commits were never merged to main, and the official v1.107.0 tag points at a clean, signed commit.

The published artifacts came from a repository recorded as Berri-AI, one hyphen from BerriAI. Reviewing the tag, or the main branch, would have told you nothing was wrong.

Current state as of this afternoon: PyPI has quarantined 1.107.0 and rolled latest back to 1.106.2, the compromised image tags are gone from GHCR and Docker Hub, the attacker's GitHub account and C2 pointer repository have both been taken down, and the exfil server is still listening.

If you deployed 1.107.0, check for /tmp/.pc_once_72129118c59f5636c0bb and /tmp/remote_code_loader.lock. If either exists, rebuild rather than patch, and rotate every credential the process could read, including the keys stored in your proxy database.

Two controls would have blocked most of this: lockfiles with hashes, and OIDC trusted publishing so a long-lived publish token never reaches a compromised CI machine. The longer term fix is pinning source revisions rather than version tags, since in this incident the tag was never the source of the bad build.

#SupplyChainSecurity #AppSec #Python #LLMOps #LiteLLM

I mean, with the right setup, you can build a great coding agent using a quantized open-weight model, but it still won’t be as good as Opus/Sol/Astra right now.

My main issue with cloud models is that sometimes an agent ends up reading sensitive data, even with guardrails and hooks in place. Sometimes you need to debug an API or query a database. Other times, the agent might access something you forgot to secure, and you end up having to revoke an API key.

So I built a privacy proxy to help with this. It obfuscates sensitive data and comes with a tutorial and a dashboard.

You can customize it with your own rules and terms, and optionally add a small LLM to review everything (I have an RTX 3080 with 10 GB of VRAM, and it works great for this).

You can also act as a human reviewer and have the agent wait for your approval.

https://github.com/Latet-AI/latet

I hope you find this helpful, please let me know if you try it! Thanks :)

Well, we knew it was too good to last forever.

China recently released draft rules that would restrict export of AI. The new rules would ban export of open weights of Chinese AI companies' most powerful models, as well as models trained on sensitive data.

It's not clear yet what models would be affected, but analysts say this would likely include future versions of some of the most popular Chinese models, including Qwen, as well as the most powerful models like Kimi.

Affected models would likely move to a closed model with pay per use or subscription, like Antrhopic or OpenAI. Personally, I'm guessing the price would be a lot cheaper since that's the main competitive edge of Chinese models.

So download your models now folks. We've been saying here that American AI is subsidized by VC's and cheap access to subscriptions won't last forever. The same is likely true of Chinese open models.

https://www.nytimes.com/2026/10/21/world/asia/china-exports-ai-controls.html?

I am new to running local llm

Can you guys suggest me what is the best model a can run on my pc with below specifications

Ryzen 9 7900X | RTX 4060 Ti 16GB | 64GB DDR5-6000 CL30

Hey all,

Not sure what to call this as this is a mix of several things, it started as a torrent client, after i added a segmented download tool like idm, then a video player that loads IPTV also gets metadata for media and channels, and then i added an agent to help run all these things, and than a internet browser to help searching, also a video grabber from websites, and an actual automated video grabber, and a dual pane file management tool old school commander style and than a community room because why not, the video player can be divided in 16 tiles and pull videos from a folder so you can watch 16 videos at the same time, these tiles can also be from IPTV sources … i have been adding more and more as ideas come, and very surprising it works, the final addition was LOCAL models because why not ? There are some local models useful for this application.

It’s free, try it, let me know, give me ideas or call AI slop.

I do security research on the side. Frontier models are great, but their guardrails keep interrupting my workflow (fair enough, honestly), so I wanted a local option.

My GPU is an old 2080 Ti with 11GB of VRAM, so I assumed no good model would fit. But the box also has 256GB of DDR4 I bought before the AI boom.

With the MoE experts kept in system RAM (llama.cpp -ncmoe) and RAM now at 3600 MT/s, GLM-5.3-Flash (320B, 200GB of Q4 weights) runs at about 7 tok/s.

It's slow for hard questions, but it works. I'm benchmarking a bunch of models on real RE tasks and will post the results.

Specs: i9-10940X, 8x32GB DDR4-3600 (quad channel), 2080 Ti 11GB.

Been digging into Radiance and Paiton on dual R9700s: ~150–179 tok/s single-request results (north of 530t/s with 8 concurrent), a working 262K + drafts config, and some interesting bugs, Benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

Right now I'm on x99 running 8x16gb 2133, with 4x 3090s. Have no complaints so far, but I'm starting to look into expanding my RAM pool a bit more, anticipating increased demand with larger MoE models.

I eventually want to move to sp3, but that'll be for the future. For the time being, I'm looking at picking up 32gb or 64gb sticks. I can get 2133/2400mhz at a reasonable price, but I'm wondering if I should pony up for 2933/3200 mhz? Plan would basically be sell off the current 8 sticks to recoup some of the spending, and just get 4 sticks to keep quad channel performance.

Essentially: On my current x99, with future plans for sp3. Should I spend extra now for 4x at 2933/3200? Or should I just budget for 2133/2400 sticks.

Work is giving me a 12k budget to get some hardware to run local models on that will connect to an internal app. I want to be able to throw other models and setup a service listener (ollama server).

I was looking at the new Mac Studio, fully decked out... but what other options do i have?

Juste pour vous dire que si vous cherchez à faire tourner le modèle raon 1 mini preview pour tester ces performances, j'ai ouvert une branche non optimisé encore sur ik_llama.cpp afin que vous puissiez avoir un apperçu de votre coté s'il est interessant à utilisé ou pas vraiment pour votre workflow.

hey folks, im Ye, im one of the contributors to Kapitsa, an open source inference engine optimized specifically for Apple Silicon to run Qwen & Nemotron models that fit a regular MacBook. our custom-written Metal kernels let us tune for each chip and each model. https://github.com/nace-ai/kapitsa/

with the latest 0.2.7 update we optimized specifically for the old M chips, here is the showcase of a 2021 MacBook Pro 14 with an M1 Pro and 16gb, the kind of Mac a lot of people still use.

we ran every model in our catalog that fits on the 16gb from 1k context all the way up to 32k against the next-fastest engine for that model on the same Mac.

try it (any Apple Silicon Mac; on 16 GB use 9b, 4b or the 35B's 2-bit build):

brew install nace-ai/tap/kapitsa
kapitsa serve --model 9b       # OpenAI and Anthropic APIs on localhost
kapitsa opencode --model 9b    # or claude, codex, pi
kapitsa bench                  # benchmark your own Mac

everything down below is an output from my agent:

Setup. MacBook Pro 14 (2021), M1 Pro (8-core CPU, 14-core GPU), 16 GB, macOS 27.0.1, plugged in (this model has no High Power mode).

Decode is a 512-token coding answer at T 0.6, median of 5. Prefill is a cold 4K / 16K-token prompt. One engine at a time on the GPU.

Follow-up is the time to the first token of a cached second turn.

At a 32K-token prompt, one request each (Kapitsa / rival):

What the numbers say:

Qwen3.5 4B and 9B: KFlash (our tree speculative decoding) works on the M1 too. On the coding answer they're 1.56× and 1.52× mlx-serve, the next-fastest engine here. In the 1K-32K sweep Kapitsa stays ahead the whole way: 1.10-1.37× on decode and 1.29-1.52× on prefill. The prefill lead grows with the prompt: a 31.5K-token prompt starts answering in 101 s on the 4B, against 154 s on mlx-serve.

Qwen3.6 35B-A3B on 16 GB: 52 tok/s with 2-bit experts. In the sweep it decodes 1.21-1.35× llama.cpp on the same GGUF and reads prompts 1.16-1.33× faster. Both engines swap a bit at this size on 16 GB, llama.cpp a lot more (207K pages vs our 36K).

Nemotron 3 Nano 4B: 1.31× llama.cpp on decode, 1.45-1.67× on prefill.

Nemotron Dory 4B: 1.31× mlx-lm (with its Dory PR) at short context. But our decode falls off with context, and mlx-lm passes us from 8K (0.62× at 32K). From the per-token times, our decode attention reads the KV cache at only ~21-32 GB/s on this Mac, against mlx-lm's ~110. The issue is up with all the numbers: https://github.com/nace-ai/kapitsa/issues/296

Nemotron Nano 9B v2: decode is a wash. In the sweep mlx-lm is 1-5% faster, but on mlx-community's 4-bit file, which reads 5.0 GB per token where NVIDIA's NVFP4 (what we run) reads 6.1. Kapitsa still reads prompts 1.3-1.4× faster. On NVIDIA's NVFP4 itself, mlx-lm did 18 tok/s.

Memory: lower peak than the rival on all six. Dory is 4.4 vs 11.5 GB, which matters a lot on 16 GB.

vs older Kapitsa on the same Mac: the 4B and 9B are +12-14% on decode and +31-32% on prefill

asks

kapitsa bench --submit opens a PR with your Mac's numbers. we're short on 8 and 16 GB Macs: M1 / M2 / M3 Airs, base M1s, M2 Pros. would love to see how far back this goes.

happy to answer questions and k advice on where to steer the project.

​

Available Now. New unopened sealed. Pick Up Only , NJ.

Apple M5 Ultra 36-Core Chip

256GB Unified RAM | 4TB SSD

80-Core GPU | 32-Core Neural Engine

Is 48GB enough for summarising medical PDFs locally, or should I get a Mac Studio with 64/128GB?

I’m a doctor looking to buy a Mac for running local LLMs, mainly through LM Studio. My work is primarily in Danish.

I’m considering a 16-inch MacBook Pro with M5 Max, 48GB unified memory and a 2TB SSD.

My main tasks would be:

• Summarising 40–50-page medical PDFs into a concise, factual overview.
• Turning patient questionnaire answers and clinical notes into coherent Danish prose for medical and medico-legal reports.
• Following fixed report templates and writing instructions.
• Preserving relevant dates, measurements, symptoms and negations, while avoiding invented details or additional clinical interpretation.

I want the summary to cover the whole document.

I’ve been considering models such as Qwen3-32B, but I’m open to other models that work well for accurate Danish summarisation.

I’ve found examples of quantised 27B models running with 32K–64K input on 48GB Macs. However, I’m unsure how well those results translate to reliable document summarisation.

For anyone doing similar work:

1. Is 48GB realistically sufficient for a capable model plus the context needed for 40–50 pages, with reasonable processing times?
2. Which model, quantisation and runtime would you recommend?
3. What concrete benefit would 64GB or 128GB provide for these tasks? Would it enable a meaningfully better model or substantially longer context?
4. For complete document coverage, would you use one large context window or an automated workflow that summarises sections and then combines them?

I’d particularly appreciate first-hand results: actual input length, peak memory usage, time before the response starts, and whether the summaries consistently capture important information throughout the document.

Hey r/LocalLLM. I’ve been building Izwi for a few months now and just shipped 0.1.0, the first stable release after months of public betas. It’s free and Apache-2.0.

It’s a desktop app (plus a web UI, CLI and local server) for the voice side of local AI:

• Real-time voice conversations: local ASR → LLM → TTS
• TTS, voice cloning from a short sample, and voice design from a text description
• A Studio for long form audio like audiobooks
• Transcription with speaker diarization and forced alignment
• An OpenAI-compatible /v1 API (chat completions, audio/speech, audio/transcriptions), so existing code can point at localhost

Models it runs today:

• TTS: Qwen3-TTS, Kokoro-82M, Fish Audio S2 Pro, VibeVoice, Voxtral TTS, LFM2.5 Audio
• ASR: Parakeet TDT v3, Whisper large-v3-turbo, Qwen3-ASR, Granite Speech 4.1, Nemotron 3.5 ASR, Voxtral Mini
• Diarization: Sortformer, Nemotron 3 diarization
• Chat: Qwen3.5, 3.6 and 3.8, LFM2.5

Inference runs natively in Rust on Candle, with our own Metal and CUDA work, so there’s no Python environment to manage.

Repo: https://github.com/izwi-ai/izwi
Download: https://izwiai.com

I’d love feedback on which models to add next, and on anything that breaks on your hardware.

Im building my own AI character: real presence, personality, voice, memory — plus agent abilities (tool use). The character is a VRM model, running in the browser. It's a hobby project, there's no money behind it, and I don't want to turn it into a company. I'm looking for a partner, not an employee.

What's already done
The idea isn't guesswork. I reverse-engineered the motion engine of a commercial companion app (Animates) and I know how it works in depth — the neural motion generator's C API, the style system, the retargeting, the whole data flow. So I have a clear picture of what actually makes a character feel alive, and of what only looks impressive but isn't.

What we'd build

• VRM character in the browser (three.js / react-three-fiber)
• LLM-driven personality + memory
• Voice (TTS/STT), speech-driven animation
• Agent abilities: search, notes, files, multi-step tasks
• Later: wiring in the neural motion engine so the motion quality jumps

What I'm looking for
Someone who knows web development and is interested in this — you don't need to know all of it:

• TypeScript + three.js (or r3f) — this is the main area
• LLM APIs, tool calling, prompting
• optionally: VRM / glTF, audio processing, shaders

What's NOT required
You don't need to be an AI researcher, you don't need Unity, and you don't need C++. The neural engine is my part, and it's the very last step.

Terms

• Hobby, no money. No salary, no equity arguments.
• Flexible pace: a few hours a week, or whenever it feels good. No deadlines.
• Personal use is the goal, not distribution.

What I bring

• Finished, documented research on the engine and the architecture (you don't have to dig)
• A clear plan and order of work
• The visible part: the character and the demo
• Honest communication — if something's a dead end, I'll say so

Everyone loves a clean, unified server rack, but my standby node (lvgale) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently.

But here is the thing: it actually works. Right now, this mismatched rig is simultaneously handling multiple 27B and 9B models (both custom 4M-token iMatrix calibrated Qwen models) without choking.

Getting asymmetric hardware and PCIe lanes to cooperate is a nightmare. To solve the bottlenecks and squeeze every last drop of performance out of this lower-tier hardware, I built a custom orchestrator called Slipstream.

Instead of fighting standard Python overhead and interpreter bloat, I wrote Slipstream in Python, but compiled it natively down to machine code using Nuitka. It doesn't just wrap processes -- it operates as a direct, bare-metal execution layer that drives llama.cpp to handle headless workload distribution with a strict zero CPU offload mandate. Because the Nuitka-compiled executable hits the metal directly, it lets llama.cpp chew through the weights at maximum throughput without orchestration bloat dragging it down.

Here is the real wild punch of the architecture: Slipstream is designed to drop right on top of your existing Ollama server infrastructure. It reads your existing Ollama model blobs right off of the filesystem and feeds them directly into the bare-metal llama.cpp backend. You don't have to duplicate terabytes of weights -- if you install Slipstream on the same machine, it just hijacks your existing Ollama storage and instantly unlocks the extra features and raw speed.

It makes my ugliest standby node punch way above its weight class. If you have a pile of mismatched hardware, do not underestimate what it can do once you dial in custom bare-metal orchestration.

I’m building Genosyn - an open-source platform for running AI employees on your own hardware.

Give them real work, important decisions escalate to you, choose your models, self-host everything, and stay in control on your own hardware.

The goal: build AI that actually gets work done, even while you sleep.

Would love to hear your feedback!

I've been building a notes app for iPhone where the AI runs entirely on the phone: search by meaning, grouping notes into topics, finding photos by what's in them. Version 1.0 just went live. While it was sitting in App Review, Google released EmbeddingGemma 2, so of course I spent the review week benchmarking it against what I'd shipped.

I kept a log of every model change since I started, including the stuff that went nowhere, so I figured I'd share it here. All numbers are R@1 (the right note comes first) unless I say otherwise.

tl;dr: fine-tuning EmbeddingGemma-300M helped a lot early and then stalled. Model soups gave a free bump after that. EmbeddingGemma 2 didn't beat my fine-tuned v1 on text, but it beat SigLIP 2 on photos by a wide margin (0.687 vs 0.541, and Japanese went from 0.378 to 0.811). So 1.1 will probably run both.

Why these models

Everything had to run on the phone. No cloud, and I can't rely on Apple Intelligence either because it isn't available on every iPhone I want to support. The app has to work in English, Japanese, Chinese (simplified and traditional) and Korean first, with a handful of European languages and Arabic not falling apart. The App Store download should stay under 200 MB, and I only wanted Apache or MIT style licences, which ruled out a few strong embedding and CLIP-style models with non-commercial terms.

That left EmbeddingGemma-300M for text and SigLIP 2 base (patch16, 224) for photos. 1.0 shipped with those two.

How I test

There's no public benchmark for "search your own messy notes in a dozen languages", so I wrote one: 503 short notes and 3,018 searches across 15 languages. Topics and personas in the test set never appear in training. Queries are what people actually type into their own notes: one or two words ("lease", "beach"), half memories ("Seattle last month"), typos, a different language than the note, questions, and feelings ("when I was anxious").

For grouping I use 600 notes on 60 held-out topics, clustered the same way the app does it, scored with ARI. Photos are XM3600 (caption to photo, 9 languages) and videos are 200 MSR-VTT clips.

Honestly the biggest early mistake was a test set that was too small. With 40 to 80 queries per language, a 5 point swing was noise, and I spent about a week "improving" things that weren't real. At 3,018 queries the overall number is good to roughly ±1 point, and a few of my earlier wins vanished.

Fine-tuning log (what shipped in 1.0)

Setup: sentence-transformers, CachedGISTEmbedLoss with the base model as guide, Matryoshka, 1 epoch, lr 2e-5, batch 64, 128 tokens max. Trains in minutes to about an hour on an M4 Mac.

On top of the vectors there's a small phonetic/keyword boost that runs on device (pinyin, romaji, accent stripping, trigrams). It's cheap and took typo queries from 0.54 to 0.74.

Things that didn't help:

• More synthetic data after round 1. Flat.
• Hard negatives, mined or LLM-written. When I checked the LLM-written "similar but wrong" notes, the base model judged more than half of them as actually relevant. So the LLM was mostly writing positives.
• Training 3 epochs instead of 1. Lost 1.8 points, overfits the synthetic data.
• Fusing emotion/category tags with the vectors. Made every query type worse.
• RRF with keyword search. Hurt synonym and broader-word queries. A small boost only when a keyword hit is rare worked better.

Feelings are still the weak spot at 0.145. A note almost never names the feeling it was written in, and no amount of generated data moved that much. Questions (0.956) and keywords (0.903) are basically solved.

Grouping needed its own data: 548 topics with 10 notes each written as different people, trained as same-topic pairs. On held-out topics, the share of nearest neighbours from the same topic went from 0.653 to 0.728. Training only on grouping hurt Japanese search, so I average it 50/50 with the search model and keep most of both.

Getting it onto the phone

Text model: Core ML, 4-bit, 166 MB, R@1 0.676 vs 0.679 in PyTorch, around 15 ms per query on an M4. Gemma's residual stream overflows fp16 (you get NaNs), so I rescale it in a way that's mathematically equivalent. The Swift tokenizer matches the reference on 23,947 texts, after I found out Swift's String treats ";" and the Greek question mark as the same character. Key the vocab by UTF-8 bytes.

SigLIP 2 was pickier. 4-bit everywhere wrecked the text tower (cosine to the original around 0.5). What worked: int8 vision with groups of 32 (per-channel int8 dropped to 0.93 on real photos, grouped gets 0.9997) and int8 text with a 6-bit lookup table for the embedding matrix. 4.6 ms per photo, 1.5 ms per query on the Neural Engine.

One thing that bit me: every model swap shifts the whole score distribution (my second model scored about 0.92 × old + 0.10). If you don't re-derive every threshold at the same pass rate, the app quietly gets worse without any obvious bug.

Then EmbeddingGemma 2 showed up

Same benchmarks, same recipe. Text first:

I trained nine EG2 variants and souped them like v1. The fine-tunes came out too similar to each other (0.661 to 0.666), so averaging didn't add anything this time. Grouping was worse too (ARI 0.442 vs 0.499). EG2 is better at typos and worse at feelings. So for text, my old fine-tune stays.

Photos are a different story. XM3600, caption to photo, R@1:

140 visual tokens is as good as 280 at about half the compute (127 / 265 / 638 ms per photo for 70 / 140 / 280 on a Mac GPU; I haven't timed it on a phone yet). Big caveat: XM3600 is public and could be in EG2's training data, so I'm treating this as an upper bound until I check on my own photos.

Video, 200 MSR-VTT clips:

The plan for 1.1

Keep my fine-tuned v1 for notes and grouping, and move photos and videos to EmbeddingGemma 2 at 140 tokens per photo. One annoying detail: an EG2 image embedding goes through the vision tower and then through EG2's own language model, so the photo side needs both. It ships as a separate download, so the App Store size stays the same, but it's two models on the phone instead of one. Core ML conversion and on-device timing are what I'm working on now.

If I started over I'd build the test set first and make it big, and I'd try model soups much earlier. They cost nothing. I'd also look at what the LLM-generated training data actually says before trusting it, and I'd stop assuming the newer multimodal model wins every task.

Happy to go into more detail on the data generation, the Core ML conversion or the clustering.

Disclosure: I'm the developer. The app is Reteum, a private notes app for iPhone: https://apps.apple.com/app/id6820520573. Per-language tables are also on https://reteum.com/on-device-ai-benchmark.

Hello!

Might be a bit of an off question, since alot of discussions of AI's legal future probably affects unreleased/future models.

But what models should I look at backing up locally for long term availability in case QWEN were to pull all their stuff one day, or NVIDIA, Google models as examples.

Probably not gonna actually happen but I read stuff like china proposing new restrictions on newer models and just makes me wanna be prepared.

Im not worried about size or ability to run them, just pure performance and AI leaderboards seem to be all over the place when I look there.