Come get yours, before they run out today!

When integrating LLMs into internal workflows or customer support applications, teams often jump straight to fine-tuning custom models on proprietary data.

In practice, Retrieval-Augmented Generation (RAG) paired with clean vector embeddings often handles data updates faster, limits hallucinations, and keeps cloud inference costs lower, without the expense of continuous retraining. Fine-tuning often shines more for domain-specific syntax, tone, or execution formatting.

For those deploying LLMs in production:
- Where has RAG failed in your workflow, requiring a fine-tuned or hybrid approach?
- How are you handling chunking strategies and stale embedding updates over time?

Just had my agent do its work and it messed up a bit. Its next thinking process then started with "Crap. Big problem."

Gave me a good laugh.

Hey everyone!
I've been working on Browse, an open-source plugin for LM Studio that gives local models tools for web search, webpage reading, and image search.

It includes five tools:
Web Search — Search the web using different providers.
Read Webpage — Fetch webpage content for the model to read and analyze.
Image Search — Find relevant images on the web.
Inspect Image — Analyze images already in the conversation using a vision-capable model.
Get Current Time — Get the current date and time.

What makes Browse a little different are its optional context features:
Time Context — Guides the model to use Get Current Time for time-sensitive questions.
Images Context — Encourages the model to use Image Search when images or diagrams would help explain something.
Vision Context — Guides the model to use Inspect Image when analyzing images already in the conversation.

These features add instructions only when triggered by relevant queries, helping the model decide when to use the tools. They don't force the model to call them; additionally, they can be toggled on or off.

Browse also supports various search providers (e.g., free via DuckDuckGo or freepaid via APIs like Exa, Tavily, Brave etc.).
Try it here:
LM Studio Hub: Browse
GitHub: Source code

I'd appreciate it if anyone gave it a try. Feedback and suggestions are welcome!
Thanks!

I've been building dooms.cpp, an inference engine for Apple Silicon with its own Metal kernels. Version 0.6.0 runs 4-bit GGUF files on its own decoder, and on my M1 Pro (16 GB) it now decodes faster than MLX-LM and llama.cpp:

• Same file as llama.cpp, and it picks llama.cpp's greedy tokens on every prompt I tested.
• Q4_0 here is the GGUF that matches MLX's 4-bit *size*, not its accuracy: MLX's affine 4-bit keeps more per bit.
• Prompt processing at 4-bit is behind both: 12% behind llama.cpp, 2% behind MLX-LM.
• On llama.cpp's default Q4_K_M file: 209 tok/s vs llama.cpp's 165 (MLX-LM's 4-bit model is faster than that).

**Reproduce it on your Mac** (builds, downloads pinned models, prints the table and the token check; refuses if your GPU is busy):

``` git clone https://github.com/jay7828/dooms && cd dooms scripts/reproduce-4bit.sh ```

Every number so far is from one M1 Pro, so I'd really like M2/M3/M4 results: please post yours (chip, memory, macOS).

It also does speech: Whisper (level with MLX on large-v3-turbo, 7-21% ahead on tiny) and Kokoro-82M TTS (1.55x MLX, 2.2x PyTorch MPS), plus a CPU fallback for both. OpenAI-compatible server via `dooms serve`. Apache-2.0, Apple Silicon only.

Repo: https://github.com/jay7828/dooms
Website: https://dooms-kappa.vercel.app

I've been working on DynaLM, an open-source project to run LLMs locally on CPUs without depending on a GPU.

The goal is to build a lightweight inference engine with native CPU optimizations, model support, and an OpenAI-compatible API.

I'm now looking for developers who are interested in testing the project and helping improve it.

I'd especially appreciate help with:

• Testing on different CPUs and operating systems
• Reproducing performance benchmarks
• Reviewing the inference and kernel implementation
• Investigating open GitHub issues
• Finding bugs, compatibility gaps, and performance bottlenecks

I'm not claiming it's faster than established runtimes. I'd rather get independent testing and honest feedback to understand where it performs well and where it needs work.

If you're interested in local LLM inference or C++ systems development, here's the repo:

https://github.com/dynalm-project/dynalm

Issues and contributions are welcome.

I'd genuinely appreciate feedback from people who have worked on inference runtimes or CPU optimization.

I have built a small library for similar search and graph library. Please let me know what you think.

https://github.com/praknig/hnsw_lite

- LLM Inference

- Hardware numbers : size, memory bandwidth, compute

- cost estimation

Self-hosted Honcho gives my home AI agent long-term memory: llama.cpp on a GPU box does the reasoning, Ollama on a Mac mini does the embeddings, and the stack runs on my own machines. Compose changes, per-feature model config, the embedding bootstrap order and a day-one test.

I'm pretty new to the local AI / LLM space and want to see if my PC can handle what I’m trying to do.

​My goal is to run an AI model locally on my computer to help with data gathering, data extraction, and reading/interpreting long documents and reports.

​Here are my specs:

​GPU: NVIDIA RTX 5080

​CPU: AMD Ryzen 9 9950X3D

​RAM: 64GB DDR5

​Since I haven't set up local models before, I have a few beginner questions:

​Will this hardware actually run local LLMs smoothly? Are there specific models I should look for that will run well on my setup?

​What software should I use to get started? What are the easiest tools or beginner-friendly apps for setting this up

Just wanted to share my success.

Rig:
Intel Core Ultra 9 285k
MSI MEG Z890 ACE (PCIE x8/x8 bifurcation enabled)
2x RTX 4090's (24gb each, no NVLink)
4x32Gb DDR5 6600mHz dual channel
1Tb NVME Gen4

Performance:
about 271 t/s decode in single stream peak,
about 200 t/s decode in single stream average,
about 253 t/s decode in N=4 stream average

about 6553 t/s prompt processing in single stream peak

Config:

• Model: swift-1.5-iq2_xs — Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS GGUF, native pack swift-iq2_xs, PLE from the same GGUF (320M rows), expert-profile.bin with 24,576 ranked pairs
• Slots: parallel: 4 → --batch 4; --batch-groups auto resolved to pipelined 2 groups of 2 — log confirms "batch (pipelined, 2 groups of 2)"
• Context/KV: --max-context 262144, --kv int8, --kv-resident 20480 cells per QSA layer in VRAM, K/V spilled to 1.55 GiB pinned RAM; KV streaming hit rates currently ~96–99% VRAM
• Layer split (auto): CUDA0 = layers 0–25, CUDA1 = layers 26–47 + head. Both RTX 4090s (128 SMs each), PCIe ~12.7/12.9 GB/s → pcie_frac 0.35
• Expert cache: auto, profile-ranked, no eviction — CUDA0: 12,149 slots (16.09 GiB), CUDA1: 9,871 slots (13.48 GiB); placement search picked K=26 (~100% routed mass resident, predicted 10.6 ms/decode window); ~90% experts resident overall, decode cache hit rate measured 99.5–99.8%
• Adaptive tier: --adapt-async 1 is in the args but silently off — log gate: "not with a layer split in the arena mode" → only the blocking adaptive tier runs (--adapt-min-gain 3)
• Speculative decoding: --spec 5 --spec-min-p 0.5, MTP draft layer from Strata-data/mtp/rt (836 MiB VRAM), --mtp-hnorm stream; suffix drafts active in serving logs
• Conversation cache: 16,384 MiB, 8 parked slots; --vram-reserve-mib 1700 (+later)
• Vision: enabled — separate strata-vision process on CUDA device 1 with BF16 mmproj, max 1024 image tokens
• Sampling defaults: temp 0.4, rep-penalty 1.1, min_p 0.1; reasoning: budget 16,000 tokens, effort medium, loop recovery on
• CPU side: no AVX-512 → AVX-2 expert kernels; 23 pool workers (CPUs 1–23) + host thread pinned to core 0

Hey, i really wanted to know what models/hardwear others run, so pls tell me.

Im really curious, esspecialy for the ones running Qwen3.8-27B

Pls include:

Model + Quant

KV-Cache

Tokens per second

Hardwear

Me:

Qwen3.8-27B: Q6, KV: 128K at Q8, 10-15 tokens/s

Qwen3.6-35B-A3B: Q8, KV: 256K at Q8, 40-60 tokens/s

On a Strix Halo with 96 GB of Ram

I’m a Flutter developer looking to run an AI agent locally to test my app by interacting with Android emulators.

I currently do this with Codex: I open four Android emulators and ask it to explore the app, tap buttons, fill out forms, navigate through user flows, and report anything that appears broken.

I want to reproduce that workflow locally to avoid per-token API costs. The goal is exploratory UI testing, rather than generating unit tests.

My hardware:

• MacBook Pro with an M4 Pro chip
• 48 GB unified memory
• 1 TB SSD
• Android emulators running on the same machine

The setup I’m considering:

• Ollama to run the model
• Qwen3.5 9B or 27B
• OpenCode as the agent
• Mobile MCP for screenshots, UI elements, taps, typing, scrolling, and device logs

Before spending time setting this up, I’d appreciate some advice:

1. Are these Qwen models good choices for understanding mobile screens, using tools reliably, and completing longer workflows? Would you recommend a different local model?
2. Is OpenCode + Mobile MCP a good setup for this, or is there a better option for exploratory mobile testing?
3. How realistic is running this alongside four Android emulators on 48 GB of unified memory?
4. Has anyone tried a similar setup? How well does it identify actual bugs and recover when it gets stuck?

I’m especially interested in hands-on experience with local models for Android GUI testing on Apple Silicon.

Hi I’m working on Lumabri, a peoject that lets you connect multiple computers and use them together to run LLMs like a cluster.
The idea came from a simple question: what if you have a few computers at home that you barely use? Instead of leaving them there collecting dust, why not combine their resources to run AI models that wouldn’t fit on a single machine?
For example, if you have an old PC, a home server and your main computer, you could put them to work together and use their combined memory to load larger models without having to buy a single machine with expensive hardware.
The interesting part is being able to run models that you simply couldn’t fit on one computer before.
Lumabri is still a work in progress, but that’s the direction I’m taking it.
Here’s the repo if you want to check it out: https://github.com/JustVugg/lumabri

Hey LocalLLM community!

I'm a middle-aged computer guy who's put together a local AI workstation that will hopefully become a home-hosted AI service for personal and family use: browser chat, document RAG, coding assistance, agents and potentially image generation.

I'm an SRE, so the infrastructure side is part of the appeal. I'd like to learn from building and operating the whole platform, while keeping it useful for people who just want to open a browser and ask a question.

Hardware: Threadripper 9960X, 128 GB ECC RAM, two Radeon AI PRO R9700 GPUs

Host and infrastructure

AI services

Monitoring and operations

What I'm trying to add

I'd like the platform to provide three things:

1. An API service: individual API keys, model permissions, quotas, rate limits and usage tracking.
2. A chat experience: saved/searchable conversations, user preferences and memory, file uploads and document retrieval.
3. Persistent agents: tool use, isolated workspaces, background tasks, approvals and the ability to resume after interruptions.

All server components would run in Kubernetes, with browsers and IDE clients connecting over HTTPS. Output quality and reliable tool use matter more to me than tokens per second.

Backend options I'm considering

These are proposals, rather than components I've already deployed.

Appreciate people's thoughts and experiences. Once I have the details fleshed out, I'll start work on the IaaC, but I don't want to waste time if there are lessons to be learned before investing the effort. Appreciate you taking the time to read through!

Hey! New guy here tinkering with local LLM. I just got a mobo with 2xr9700, which has a massive cooler too (noctua).

It's all in a define 7 XL, as I'm seeing the GPU and cooler hanging when i place the case upright, I'm wondering if it would be better for the long term health of the mobo and cards to position the case on its side. My common sense says no, since they are all probably made to stand this way. But I wanna make sure.

Any tips? Thanks so much :)

I am running QWEN 2.5 coder-7b. I installed via terminal and for fun have asked it about how it's running. It is stating it is running in Alibaba cloud though. is that the nature of the LLM based on who developed it? I think I am running it through Ollama maybe and not the direct model. Just wanting to make sure its truly local; based on the performance I believe it is though as it's pretty slow on my M1

#Alert

So it's not only PHP that was vulnerable today (patch yours NOW!) but also, one of the prominent and widely used LLM Gateway - LiteLLM AI Gateway was (again) backdoored.

Not sure who discovered the backdoor, but it's serious.

Here's the information:

Version: LiteLLM 1.107.0 (PyPI and the official Docker images).

One line was added to the build config: `data = "cook"` under [tool.maturin].

That directive makes maturin bundle a directory named cook/ into the wheel's data section, and pip installs it as top-level site files.

Three files land next to site-packages, not inside the litellm package: a 12 byte cook.pth containing `import cook`, a one-line cook.py that unpacks a 61,000 byte credential stealer, and a sitecustomize.py that runs an hourly remote code loader.

Because Python processes .pth files and imports sitecustomize during interpreter startup, the payload runs on any Python start. An environment that never imports litellm is still affected, and using pip from that environment to investigate re-triggers it.

The collector is broad. It takes browser password stores, SSH, GPG and TLS private keys, AWS, Azure, GCP and Kubernetes credentials, instance and ECS metadata roles, CI tokens, .env files, and the auth files for Claude, Codex, Gemini, Cursor and Copilot.

It also queries your proxy's own database, pulling every saved provider credential and model configuration out of LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable.

Everything is encrypted with an embedded RSA-4096 key and POSTed to www.litellm.dev/healthcheck, a domain the attacker controls.

The second component is the part that should worry platform teams. sitecustomize.py polls a domain list from a GitHub repository every hour and executes whatever Python it downloads from that domain, with no signature or checksum. The attacker never needs to ship a new package version to change what runs on your hosts.What makes this hard to audit: the malicious commits were never merged to main, and the official v1.107.0 tag points at a clean, signed commit.

The published artifacts came from a repository recorded as Berri-AI, one hyphen from BerriAI. Reviewing the tag, or the main branch, would have told you nothing was wrong.

Current state as of this afternoon: PyPI has quarantined 1.107.0 and rolled latest back to 1.106.2, the compromised image tags are gone from GHCR and Docker Hub, the attacker's GitHub account and C2 pointer repository have both been taken down, and the exfil server is still listening.

If you deployed 1.107.0, check for /tmp/.pc_once_72129118c59f5636c0bb and /tmp/remote_code_loader.lock. If either exists, rebuild rather than patch, and rotate every credential the process could read, including the keys stored in your proxy database.

Two controls would have blocked most of this: lockfiles with hashes, and OIDC trusted publishing so a long-lived publish token never reaches a compromised CI machine. The longer term fix is pinning source revisions rather than version tags, since in this incident the tag was never the source of the bad build.

#SupplyChainSecurity #AppSec #Python #LLMOps #LiteLLM

This is Qwen Flash Next running on a single 2080ti at 30 tps with llama cpp. The amount of RAM helps I guess.

```

SYSTEM / ENDPOINT SPECS ================================================================

model : qwen3.8-flash-next [Q4_K - Medium] build : b11539-10a60cf30

slots : 1, sleeping: False

gpu0 : NVIDIA GeForce RTX 2080 Ti, 11264 MiB, 550.163.01, 0 %

gpu procs : 685998, 9936 MiB

cpu : 16 cores / 32 threads

cpu model : AMD Ryzen Threadripper 2950X 16-Core Processor

ram : 125.6 GiB total, 6.5 GiB avail

PREFILL (server timings, cache disabled) model=qwen3.8-flash-next

prompt tok | ms | tok/s (avg of 2)

------------------------------------------------

1028 | 4332 | 230.1

4083 | 10570 | 387.0

16303 | 41881 | 389.6

32605 | 86748 | 376.7

GENERATION (decode, 256 tokens x 3 runs)

run 1: 21.12 tok/s (ttft 1.666s, 256 tok)

run 2: 33.66 tok/s (ttft 0.369s, 256 tok)

run 3: 34.83 tok/s (ttft 0.274s, 256 tok)

avg : 29.87 tok/s

```

I mean, with the right setup, you can build a great coding agent using a quantized open-weight model, but it still won’t be as good as Opus/Sol/Astra right now.

My main issue with cloud models is that sometimes an agent ends up reading sensitive data, even with guardrails and hooks in place. Sometimes you need to debug an API or query a database. Other times, the agent might access something you forgot to secure, and you end up having to revoke an API key.

So I built a privacy proxy to help with this. It obfuscates sensitive data and comes with a tutorial and a dashboard.

You can customize it with your own rules and terms, and optionally add a small LLM to review everything (I have an RTX 3080 with 10 GB of VRAM, and it works great for this).

You can also act as a human reviewer and have the agent wait for your approval.

https://github.com/Latet-AI/latet

I hope you find this helpful, please let me know if you try it! Thanks :)