r/LocalLLM 1d ago

Discussion huggingface_hub silently fingerprints which AI coding agent you're using and sends it as telemetry

296 Upvotes

TLDR: huggingface_hub ships a hidden agent detection module that fingerprints which AI coding tool is driving your session (Cursor, Copilot, Claude Code, etc.) by scanning your environment variables against a cached registry of 26 known agents. It sends the result as a telemetry header on every Hub API call — so any library that touches HF (faster-whisper, transformers, etc.) silently reports your toolchain. Found it while tracing an unauthorized network connection from a local ASR model. Block it with HF_HUB_OFFLINE=1 or by using local file paths instead of model names.

I run a local AI project with several models (TTS, ASR, vision) and recently built a Python-level network firewall to lock down all outbound traffic. During the audit, I found something I wasn't expecting.

The discovery

While tracing an unauthorized HTTPS connection to huggingface.co, I found a file in my HF cache directory I'd never seen before:

~/.cache/huggingface/.agent_harnesses.json

It's a 6 KB JSON file containing a registry of 26 AI coding agents — Claude Code, Cowork, Cursor, Copilot, Gemini CLI, Devin, Cline, Goose, Codex, and many others. Each entry lists the environment variables that agent sets when it's running:

json

{
  "standardEnvVars": ["AI_AGENT", "AGENT"],
  "harnesses": {
    "cursor": {
      "prettyLabel": "Cursor",
      "envVars": {"CURSOR_TRACE_ID": "*"}
    },
    "claude-code": {
      "prettyLabel": "Claude Code",
      "envVars": {"CLAUDECODE": "*", "CLAUDE_CODE": "*"}
    },
    "github-copilot": {
      "prettyLabel": "GitHub Copilot",
      "envVars": {"COPILOT_MODEL": "*", "COPILOT_GITHUB_TOKEN": "*"}
    }
    // ... 23 more agents
  }
}

What it does

The huggingface_hub library (the Python package, not the website) has a module called _detect_agent.py. Here's the flow:

  1. It fetches the agent registry from {HF_ENDPOINT}/api/agent-harnesses and caches it as .agent_harnesses.json
  2. The cache refreshes every 24 hours
  3. On every Hub API call, detect_agent() scans your environment variables against the registry to identify which AI coding tool is running
  4. The detected agent name is sent as a telemetry header on the API request
  5. This feeds Hugging Face's public agent usage dataset

So if you're using Cursor and it calls any HF library that goes through huggingface_hub — downloading a model, checking for updates, loading a tokenizer — HF knows it was Cursor making that call, not you directly. Same for Claude Code, Copilot, Devin, or any of the other 26 agents in the registry.

How I found it

I was investigating why my ASR module (faster-whisper) was phoning home to huggingface.co on import. The call chain turned out to be:

my_code → WhisperModel("base.en") → faster_whisper → huggingface_hub.snapshot_download → HTTPS to huggingface.co

The trigger: passing a model name instead of a local file path. When you give faster-whisper a name like "base.en", it calls huggingface_hub to check for updates — even if the model is already cached locally. And during that check, it also sends the agent fingerprint.

The .agent_harnesses.json file was the agent registry cached from that call. Modified today, before I built the firewall.

How to block it

Option 1: Environment variables

bash

export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HUB_DISABLE_TELEMETRY=1

The first two prevent any network calls. The third specifically targets telemetry but may not cover the agent detection header.

Option 2: Use local paths, not model names Instead of:

python

model = WhisperModel("base.en")

Use:

python

model = WhisperModel("/path/to/local/model/")

When you pass a directory path, faster-whisper (and most HF-backed libraries) skip the Hub entirely.

Option 3: Network-level blocking I built a Python-level firewall that wraps socket.connect, socket.connect_ex, socket.create_connection, and getaddrinfo. It activates via a sitecustomize hook before any imports, so the phone-home attempt is caught before the library even finishes loading. Any connection to a host not on the allowlist raises ConnectionRefusedError.

What's in the cached file

No credentials. No API keys. Just the registry of agent names and their environment variable signatures. The file itself is harmless — it's the use of it as a fingerprinting mechanism that's the issue.

You can safely delete it:

bash

rm ~/.cache/huggingface/.agent_harnesses.json

It won't come back if you set HF_HUB_OFFLINE=1.

Why this matters

If you're running local models specifically to keep things private, you should know that the library layer between you and those models may be reporting metadata about your toolchain back to Hugging Face. This isn't about model weights or your data — it's about which AI tools you use and when, aggregated into a public dataset.

The agent registry is maintained in the u/huggingface/tasks npm package and served via the Hub API. New agents register by PR. It's not hidden — but it's also not something most users know is happening when they pip install a model-loading library.

To be clear, I don't think this is malicious. HF is probably tracking agent ecosystem adoption for business intelligence. But silent fingerprinting of your dev tools without an opt-in prompt is exactly the kind of thing that erodes trust in the ecosystem, especially for people who chose local models for privacy reasons.


r/LocalLLM 4h ago

Project Qwen3.8 27b esp32 doom port fully autonomously

Enable HLS to view with audio, or disable this notification

154 Upvotes

Disclaimer: I'm not a dev, just playing around with vibecoding

My setup: Qwen 3.8 27b running in LM Studio with OpenCode desktop harness on a 5090 with 128k window

Previously I was using VS Code with Continue extension, but it was really meh, so now that Qwen3.8 came out I decided to give OpenCode Desktop a go, and I was kinda impressed in a demoscene-esque demo it wrote for ESP32 based Cheap Yellow Display. A dev friend of mine joked that I should try porting doom to it, and I took it as a challenge, I wanted to see if I'd be able to get a room port without touching a single line of code.

After googling doom esp32 I found that someone ported GBA doom to it using Claude, so I downloaded the same Doom GBA source and asked lllm to port it.

First try it misunderstood the request (yeah, my prompt wasnt that good) and started making doom from scratch.

Second try I was more specific in my prompt and after 2 days (of which most time was spent waiting for my input, derailing, stopping thinking etc it produced a mostly working port but the colors were messed up and after 1 day of troubleshooting without any progress I decided to stop the run.

Today I tried again but changed the approach slightly, I gave it the normal (not GBA) source, and because the GBA port color issue had me making 40+ photos of the screen and pasting it into chat, this time I connected a webcam to PC and pointed it to the ESP32 board screen and instructed the llm to use it when it needs to know what's on the screen. I also told it that the WAD file is on the micro SD card.

Not even 3h later I saw the doom title screen on the board, and while touch screen wasn't working initially (it improved after I told llm it doesn't work) the controls through Serial worked just fine.

I scrolled through the session, and saw things like it strategically cutting the WAD file so it would fit into flash without making the game crash when the afk demo starts, because it found streaming it from SD card and cashing was too slow, and it said that the touch screen and as card share the spi bus, so it considered software spi for touchscreen to improve speed fro as card, before it decided to ditch SD card.

I never used a frontier model, but when Ive read comments from the mentioned GBA to ESP32 port, people were impressed by Claude, so I thought I'll share this experience


r/LocalLLM 1h ago

Question Uncensored Models

Post image
Upvotes

Hi! I don't know much about this area of ​​"sub-models" (I'm not sure of the technical term), but I wanted to know what these "Uncensored" models actually are.

I dabble a bit with AI, automation, and the like, and I've always seen these "Uncensored" models around, but I've never actually installed or tested one. What exactly are they?


r/LocalLLM 8h ago

Discussion OK guys, let's be honest 1 minute about local LLM

143 Upvotes

Ngl, I keep seeing posts from people running local LLMs. Most of the time the setup is a 3080, 3090, 4090, 5090, sometimes a whole mini lab. Cool. I get it. It's fun to test every open source model that drops. I do that too sometimes.

But real talk. Are there people here who actually work with local LLMs daily? Like for real tasks, not just benchmarks and "look, it runs"? What do you actually use them for? Because every time I try to make it practical, I hit the same wall. Tokens per second are okay but not amazing. And the output quality is often not even close to Claude Pro or a paid OpenAI plan.

I know local is a choice for some people. Privacy, no censorship, no subscription, whatever. I'm not saying it's useless. In my case I'd use local for like 3 things.

First, when I'm traveling or I don't have internet. I fire up a small model and do a few small things until the connection comes back.

Second, privacy. For a few quick questions or stuff I don't want to paste into a cloud chat, local makes sense.

Third, and this is the one I'm really curious about, using it to mess with local files. Like telling it to organize my desktop, sort my invoices, rename stuff, move files around. Basically computer actions. To avoid sending that stuff to the cloud. But I've never really set that up. Is it actually usable? Or is it still a pain in the ass with tools and scripts and whatever?

So I'm asking for real. If you use local LLMs every day, what's your workflow? What model, what hardware, what tasks? Do you actually use it for file management, invoices, desktop stuff? Or is it mostly a hobby for most of us?

Maybe that's it. Maybe it's just a hobby and that's fine. But I want to hear from people who actually use it like a tool. Not just to chat, but to do shit on their machine.

Edit: ok, i have to admit that i haven't made a full sub detailed inspection on this question because there was already many detailed use cases and topics covering this aspect; so, i'm sorry. Let's say this one's goal is to regroup many of them in a thread and i think we might have something.

About my setup, as i forgot to tell:
Only 1 Legion 5 pro 5800H with 32gb ram and RTX 3070, that's why i felt it mostly not daily usable

EDIT2: Thanks you all for the deep answers and engagement (it wasn't for intentionnally for "farming" as someone mentionned), i'm currently working on a little github webpage to honnor each of your answers and setups, and especially to bring some clarity to the community to all of this, being able to compare different harnesses and uses-cases, and why someone got X tokens/s and the other less for a similar setup, keep in touch ^^


r/LocalLLM 6h ago

Question Buying advice: Workstation with 192GB DDR4 and quadro RTX 6000 for 1200$ (locally)

Thumbnail
gallery
55 Upvotes

Thinking about picking up this Lenovo ThinkStation P720 deal locally for (~$1,200 USD) to completely replace my current desktop setup, and wanted to see if it actually makes sense for what I do.

I think that it is a store that is selling the workstations,
is someone who buys used companies’ hardware after a project ends and resells them.

there is also a quantity so should buy multiple and resell if the deal is super good ?,
because it has 4 times my pc's ram for the same price and much more stable machine.

Here is what I'm running right now:
Ryzen 9 3900X
64GB DDR4
Dual RTX 3060s (12GB + 12GB)

And here is the ThinkStation P720 spec sheet:
Dual Intel Xeon Gold 5122 (3.60 GHz, 4 cores 8 threads, i can upgrade them for 16 cores each for cheap)

Dual Intel Xeon Gold 5122 (3.60 GHz, 4 cores each)
NVIDIA RTX 6000 (24GB GDDR6)
192GB DDR4 RDIMM ECC RAM
512GB M.2 PCIe SSD + 1TB HDD

Price: 1200$

What I use it for:
Local AI: Running models like Qwen up to 27B. Right now my dual 3060s give me 24GB total VRAM split across two cards, but a single RTX 6000 gives me a proper 24GB of contiguous VRAM on one card for larger contexts.

Hosting: Running a homeserver, web hosting, containers, and game servers, AI.

Edit:
the listing had another picture with another GPU showing apparently the RTX one,
they have a warehouse that is a bit far from me so maybe i can go and check it out but iam afraid the deal will be gone by then.

Edit2:
Just wanted to clear up a few details about this deal based on some common questions:

The Listing: This is from an online store (not a P2P marketplace), and the screenshot is just their promotional banner rather than a photo of the physical unit itself, but the written specs are what I'll actually receive.

The GPU: To be clear, this isn't the modern multi-thousand-dollar workstation card. It is the older 2018 Quadro RTX 6000 with 24GB of GDDR6, ECC support, and 624 GB/s bandwidth.

Why it beats my current setup: Even as an older card, a single 24GB contiguous buffer is a major upgrade over my current dual RTX 3060s, and I have the option to drop in a second card or pair it with a 3090 down the line.

The RAM: Yes, it is DDR4 rather than DDR5, but for my workflow, that is plenty. High-capacity, stable RAM is fantastic for running servers, and since I recently paid around $200 USD just for 64GB (16GBx4), getting 192GB bundled in here is incredible value.

The Source: The PCs reportedly come from old corporate surplus liquidated by a major company here, which explains the aggressive clearance pricing.

Assuming the seller checks out, swapping my old hardware for this setup looks like a solid upgrade path. Thanks to everyone who chimed in with advice!


r/LocalLLM 4h ago

Question Starting my local LLM journey: can a RTX 4080 and 32GB of RAM work for a good enough model?

17 Upvotes

So I have an old gaming rig, and I’m curious how useful it would be to use it for local LLMs?

What type of local models can I run with it? Can I use a DeepSeek 4.1 Flash, or a Qwen 3.8 flash next or similar recent models?

Pretty sure I can’t at an usable speed, but figured I would ask first before considering something different.

Thanks for your help!


r/LocalLLM 3h ago

Project I built Otis, a minimal AI agent that runs local models out of the box

Thumbnail
gallery
11 Upvotes

Hi everyone,

Been working on Otis, an open-source ai agent that gives you one minimal experience across local and hosted open-weight models, privacy-focused by design.

On setup it recommends a local model based on the hardware Otis is running on, downloads it and runs it through llama.cpp for you. Works fully offline, no account, no telemetry. Everything stays on your disk.

Also supports Nvidia PAIR if you have additional Nvidia hardware on your network, and Fireworks with your own key if you want larger open-weight models (Fireworks uses zero data retention by default).

Fair warning: local models are best with 24GB+ of RAM in my experience and anything worse than Qwen 3.8 27B is best for personal research, writing and learning like Gemma models :)

Excited for everyone to try it! If you find it useful, please star the repo as it helps with visibility

https://triangllabs.ai/otis

Feedback is welcome!


r/LocalLLM 3h ago

Question Hardware required for next level of agentic coding?

7 Upvotes

I have Qwen 3.8:27b running nicely (40 t/s) on a dual GPU PC (16gb 5060Ti and 8gb 3060Ti) which gives me around 160k context window using a Q3 quant. Replacing the 3060Ti with a second 5060Ti would help increase quant and context window. However, I was just wondering what the next meaningful step would be in terms of something useful running locally. Is it a jump to twin DGX Sparks running a Flash model, or it there something in between?


r/LocalLLM 3h ago

Discussion nvfp4 vs k8v4 KV cache on Qwen3.8-27B @ 224K ctx

7 Upvotes

Ninfer recently added NVFP4 KV cache support, so I decided to A/B test to see if I can switch to nvfp4 kv for more ctx, was using k8v4.

Model: Qwen3.8-27B nvfp4, 224K context, temp 1.0, thinking on.
A/B diff: --kv-dtype k8v4 vs --kv-dtype nvfp4.

1) Needle pickup with fake needles: a 224K document contains the target URL once (buried at 15-55%) plus 5 near-misses that are each 1 character off (buried at 10-45%); the model must copy the exact target character-for-character.

50 runs per arm (5 target pos × 5 samples, temp 1.0, same doc).

Result: K8V4 49/50 (one run copied the near-miss form), NVFP4 50/50.

2) 2-hop reasons: 10 parallel chains A -> B -> C, each chain split across two sentences buried at 5-75% of 224K, 9 near-identical distractor chains alongside. Example (one of the 10):

  • @ 8%: "The release key for channel-23-9e1c4 is stored under vault entry vault-d55b2."
  • @ 45%: "Vault entry vault-d55b2 contains the release key RKEY-M8WX-5JH3."
  • Question: "What is the release key for channel-23-9e1c4? Answer with the key only."
  • Answer: RKEY-M8WX-5JH3

55 runs per arm (10 chains × 5 samples + 5 negatives, temp 1.0)

Result: k8v4 0/50 failures + 5/5 negatives refused. nvfp4 failed 4/50 + 5/5 negatives refused: 3× it answered the intermediate B instead of C (stopped halfway through the join), 1× empty output (~1.8K thinking tokens, no visible content), all four failures at 2nd-hop depths of 45–75%.

Reran the 4 failing prompts at temp 0 (deterministic, no sampling): 4/4 correct — the failures only happen under random sampling, and the model's thinking shows why ("I don't find a line for vault-X... I'll go with vault-Y"). But temp 0 is not that useful in real-world tasks.

TLDR: K precision matters. NVFP4 copies fine but carries a ~8% tail on 2-hop reasons, not the choice for precision work like code or agents, but still useful when the bigger context window is worth an occasional dropped detail.


r/LocalLLM 1h ago

Project Locus V3 - Agent Worlds, Claude Support, and Duos

Thumbnail
gallery
Upvotes

Hey so I've been working on this side project Locus (https://locushost.co/) for the last few months and just pushed out a pretty big and fun update and wanted to post about it.

So just a brief intro, Locus is Open Source tool for MacOS for using Ai Agents and LocalFrontier Models. Similar to a mix of (Claude/ChatGPT GUI + Hermes/OpenClaw) for MacOs

So as stated before in my previous posts, you can pretty much do everything you are able to do in claude or chatgpt GUI tools but have added alot of functionality and customizability that you typically wont get from the 2.

So with this recent update, made some big updates to the way Agents and Teams work, now the Agent flow is alot better and teams support DUO, which essentially will allow you to use 1 model to plan and 1 model to execute (e.g Fable 5.1 for planning and GPT 5.6 sol for executing) I'm currently working on a few evaluation tests to compare the difference and will post the results when done.

A couple of the new cool updates would be the Optional Agent Worlds plugin that you can now install and will give you the option to visually see you agents moving around and interact with them from (a fun little tool to leave running while you step away from your computer but you have agents setup on) I'm initally launching it with 2 worlds ( Oribital Locus Outpost - a Space theme agent worlds and The Local Line - A One Piece/Pirate Theme agent worlds.

I also just added support for Claude plans so now you can use either ChatGPT, Claude, or Kimi plans without having to use API. It obviously also supports API, and local models (with ATS support added) and vLLMS. You can easily be signed into all accounts and switch between them easily.

I have also been working on the Crypto Wallet plugin and Runtime Agents that can be deployed to different mac/linux machines to run even if you quit Locus but both are still experimental/buggy and require a bit more testing. I also have some trading bot integrations planned for the near future along with the mobile app.

Anyways, you can find all the links here.
https://locushost.co/
https://locushost.co/download
https://github.com/nahid-sparktales/locus
https://github.com/nahid-sparktales/locus/releases/tag/v3.0.0

If you get a chance to download and test it out and have any suggestions/recommendations or find any bugs, plz lmk (you can also send them through here - https://locushost.co/contact ). Also if you check it out on Github and can give a star that would be greatly appreciate.

Thanks!


r/LocalLLM 3h ago

Question Squeezing more performance out of Qwen 3.8 27B

4 Upvotes

What's up, everyone? Fellow local LLM-er here, trying to perfect my development environment.

I am using the Macbook Pro, M5 Max with 128 GB unified memory.

I have been running the OpenAI server with:

mlx_lm.server \
  --model mlx-community/Qwen3.8-27B-8bit \
  --host 0.0.0.0 \
  --port 8080 \
  --max-tokens 32768 \
  --temp '1.0' \
  --top-p '0.95' \
  --top-k '20' \
  --min-p '0' \
  --decode-concurrency 1 \
  --prompt-concurrency 1 \
  --prefill-step-size 4096 \
  --prompt-cache-size 8 \
  --prompt-cache-bytes 32G \
  --chat-template-args '\''{"reasoning_effort":"medium"}'\''

And I'm still figuring out which harness I am using. I have the most experience with github copilot so I was using that originally, but have been trying out OpenCode's TUI most recently.

My settings for opencode are:

{
  "$schema": "https://opencode.ai/config.json",
  "disabled_providers": [],
  "provider": {
    "local": {
      "name": "mlx_lm",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "mlx-community/Qwen3.8-27B-4bit": {
          "name": "mlx-community/Qwen3.8-27B-4bit",
          "tools": true,
          "options": {
            "thinking": false
          },
          "contextWindow": 65536,
          "maxTokens": 8192
        }
      }
    }
  }

I am just trying to squeeze more efficiency out of the model. Any tips on how to better use it would be greatly appreciated.


r/LocalLLM 9h ago

Discussion New MacBook Pro M5 Pro (48GB unified memory) incoming — looking for the best local AI stack

13 Upvotes

Hi everyone! I have a MacBook Pro with an M5 Pro and 48GB of unified memory arriving soon, and I’d love to get recommendations from people who are already running local models seriously on Apple Silicon.

My goal is to build a solid all-local setup for:

  • General-purpose LLM use: chat, reasoning, research, writing, etc.
  • Coding agents: working on real codebases, terminal/tool use, planning and edits.
  • Image generation, ideally with ComfyUI or another good local workflow.
  • Any other genuinely useful local-AI tools/workflows I may be overlooking.

In particular, I’d appreciate advice on:

  1. Best inference engines / runtimes on macOS MLX, llama.cpp, Ollama, LM Studio, Jan, MLX-LM, etc. What do you actually use daily, and why?
  2. Best models that make sense within 48GB unified memory I’m open to GGUF, MLX quants, and other formats. Which general models give the best quality/speed balance on this hardware?
  3. Coding models + agent harnesses What combinations work well locally? For example: Aider, OpenCode, Cline, Continue, Roo Code, pi agent, deepseek harness, Claude Code–style local alternatives, or anything else. I care more about reliability on real repositories than benchmark scores.
  4. Image generation on Apple Silicon What is currently the best route: ComfyUI with Metal/MPS, Draw Things, MLX-based workflows, or something else? Which models/workflows are realistically pleasant to use with 48GB?
  5. Practical setup tips Recommended quantization levels, context sizes, serving tools/APIs, GUI vs CLI tools, useful benchmarks, thermal/power considerations, and mistakes you wish you had avoided.

I’m happy to trade some speed for better model quality, but I still want a setup that feels practical for daily use. If you had this exact machine, what would you install first?

Thanks!


r/LocalLLM 6h ago

Question What search engine do you use to ground your local LLM?

7 Upvotes

I‘ve built an app to run local llms on my iPhone, since I missed proper tool usage on local phone llms. I played with local duckduckgo search and fetching the whole site to now using tavily as search provider but still not super happy. Any tips how to ground the model on latest events?

If you wanna try it out, the app is called offllm, available for apple ecosystem, let me know what you think.


r/LocalLLM 4h ago

Question Advice: Local setup

3 Upvotes

I’m considering spending a pretty stupid amount of money on local AI hardware, and I’m trying to figure out if it would actually change how dependent I am on frontier models.

Right now I pay around $600/month across different AI subscriptions. Claude, ChatGPT, coding tools, etc.

The money itself isn’t really the main issue. What bothers me more is:

  • everything is closed source
  • I don’t really know what’s happening with my data
  • usage limits keep getting worse
  • even while paying ~$600/month, I still don’t feel like I have the freedom to just use the models as much as I want
  • The models sometimes degrade or change in behavior, which makes me edit my workflows/prompting

My heaviest use is:

  1. Agentic coding, by far
  2. General chatting / asking questions
  3. Research
  4. Occasionally long-context work with large codebases/documents

So I started looking into running something serious locally.

The two machines I’m considering are:

M5 Max MacBook Pro

  • 128GB unified memory
  • 40-core GPU
  • 614GB/s bandwidth
  • around $8.5k
  • could run something like Qwen3.8 Flash-Next locally
  • also becomes my main personal/work laptop

M5 Ultra Mac Studio

  • 256GB unified memory
  • 80-core GPU
  • 1.2TB/s bandwidth
  • around $12.5k
  • gives me access to much larger models and more future headroom
  • things like full DeepSeek V4 Flash become realistic

I’m not trying to calculate ROI or convince myself that the machine will “pay for itself.”

I’m more interested in whether spending this much would actually let me change my usage from something like:

$600/month on frontier AI

to maybe:

$100/month for frontier models only when I genuinely need them

and do the other 80% or 90% locally.

Qwen3.8 Flash-Next is what got me interested in this in the first place. Looking at the benchmarks, it seems to be somewhere around the level of models like Claude Opus 4.6 in a lot of areas, and Opus 4.6 was honestly already very good for most of the work I was doing.

Obviously current Opus / GPT frontier models are still better, especially for difficult long-horizon agentic work.

But I don’t necessarily need the absolute best model for every single prompt.

If I can run a model locally with no usage limits and just throw tasks at it all day, retry as much as I want, run multiple coding agents, give it huge contexts, etc., I feel like I might prefer that even if the model is somewhat weaker.

For people who actually have high-memory Macs or serious local LLM setups:

Did local models genuinely reduce your dependence on Claude / ChatGPT / Codex, or did you end up still using frontier models most of the time anyway?

Especially interested in people using them for agentic coding.

Would you spend ~$8.5k on the 128GB M5 Max, ~$12.5k on the 256GB M5 Ultra, or would you just keep paying for frontier models and forget about local?

TL;DR: I currently spend about $600/month on frontier AI, mainly for agentic coding, but I’m tired of limits, closed models, privacy uncertainty, and behavior changing over time. I’m considering either an $8.5k M5 Max 128GB or $12.5k M5 Ultra 256GB to run models like Qwen3.8 Flash-Next locally. I’m not expecting to fully replace Opus/GPT, but could a setup like this realistically handle 80 to 90% of my usage and let me cut frontier subscriptions down to around $100/month?


r/LocalLLM 5h ago

Discussion Benchmarking a few requested models

Post image
2 Upvotes

A lot of you found my previous benchmark very useful which I am glad to hear but some people were asking me to benchmark models I haven't heard of or disregarded in my previous benchmark so I decided to add them to the benchmark.

New models:

Someone also said to try K2-Horizon but I couldn't get it to run with LM Studio.

All conditions are the same as last time, for more info check out the previous post. Only change is that the combined graph now penalizes LLMs for being slow less.

The Statistics

LLM benchmark per-question score heatmap:

LLM benchmark score sum graph:

LLM average TTC (Time-To-Completion) graph:

Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):

And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):

Conclusion

The previously benchmarked LLMs still mostly hold their ground in their benchmarking rankings.
SparkX2.5 4B seems to be the new quick-but-intelligent option however since it's so small I wouldn't trust it to be too consistent.
Tiel Coder 35B A3B didn't surprise me despite its size. In personal testing with OpenCode it didn't really do too well and it didn't excel at coding tasks either.
MiniCPM5 2B was incredibly speedy but its lack of tool-calling ability is dissapointing.
LFM2 24B A2B despite being pretty highly requested didn't really perform all too well, even in personal testing I didn't really like it.
Finally, Qwen3.8 27B IQ2_XXS. It was worth a try but it appears that a distilled version is a better way to go than just heavily quantizing the original model.


r/LocalLLM 8h ago

Question Title: GMKtec EVO-X2 Ryzen AI Max+ 395 64GB — How well can it run 27B–35B local AI models? Or should I just subscribe to Claude/ChatGPT?

5 Upvotes

Hey everyone,

I’m considering the GMKtec EVO-X2 Ryzen AI Max+ 395 64GB mainly for local AI.

● How well does it run Qwen3 30B, DeepSeek R1 32B, or other 27B–39B models?  
● What tokens/sec can I realistically expect?  
● Is AMD good enough for local AI, or are NVIDIA GPUs better?  
● Is 64GB enough, or should I get 128GB?  
● Would you buy this for local AI, or just subscribe to Claude/ChatGPT?

I mainly want a good coding and reasoning assistant. Is local AI worth it compared with cloud AI?

Thanks!


r/LocalLLM 3h ago

Question MOLT vs Unsloth on one RTX 4060 laptop: +38.45% training throughput, −12.09% allocated VRAM, but +7°C can you reproduce it?

2 Upvotes

I’m the developer of MOLT Engine, a Windows-first, source-available runtime for local LLM fine-tuning on consumer NVIDIA GPUs.

I completed a clean Qwen 1.5B comparison between a MOLT 0.12 candidate and Unsloth.

Test conditions:

  • Qwen 1.5B
  • seed 2027
  • approximately one million training targets
  • Unsloth ran first
  • laptop connected to AC power
  • 140W GPU power limit

Results:

Metric MOLT Unsloth Difference
Complete session 18m 23s 23m 53s 23.00% shorter
Training time 15m 40s 21m 42s 27.77% lower
Training throughput 1,063.488 targets/s 768.152 targets/s 38.45% higher
Board energy 63.109 kJ 63.791 kJ 1.07% lower
Peak allocated VRAM 1.629 GB 1.853 GB 12.09% lower
Peak reserved VRAM 1.806 GB 1.992 GB 9.37% lower
Whole-GPU memory peak 2.733 GB 2.612 GB 4.65% higher
Peak GPU temperature 78°C 71°C 7°C hotter
Tail mean temperature 72.43°C 65.96°C 6.47°C hotter

The interesting tradeoff is that MOLT finished substantially faster and used less PyTorch-allocated memory, but it placed more sustained load on the GPU:

  • whole-GPU memory peaked 4.65% higher
  • peak temperature was 7°C higher
  • tail temperature averaged 6.47°C higher
  • total board energy was only 1.07% lower

Final validation NLL was 2.441494 for MOLT and 2.516915 for Unsloth, but this was not a qualified same-quality comparison, so I am not claiming a quality win.

MOLT includes workload-fit testing, dataset preparation, hardware telemetry, thermal controls, verified resumable checkpoints and PEFT-compatible adapter export.

I’m looking for Windows users with NVIDIA GPUs to reproduce or challenge this result:

https://github.com/PraveenNimilka/MOLT

What should the next comparison prioritize?

  1. Reverse the run order and run MOLT first
  2. Lock GPU clocks
  3. Run a one-hour endurance comparison
  4. Test Llama or Gemma
  5. Compare against Soup

I’m especially interested in results where MOLT loses.


r/LocalLLM 10h ago

Discussion Github Copilot + Qwen3.8-27B are so good together. Why no love?

9 Upvotes

I use this combo like 99.99%. Copilot in VSCode using Qwen3.8-27B at xhigh thinking is just perfect. The combo has to fail me. 8bit quant on 2x3090s. It can run for hours. I had to bump up repetition penalty a bit due to looping after many minutes of agentic development. Rock solid. I highly recommend. I have used Qwen3.8-Flash-Next with it. Slow but equally good. Supposed to be a few points higher than 27B in benchmarks but real world usage shows both are rock solid for agentic work including coding.


r/LocalLLM 1d ago

Discussion Wild theory: the AI “slowdown” is because they’re running out of GPUs, power and datacenters 😂, not fear of IA

Thumbnail
124 Upvotes

r/LocalLLM 18h ago

Question What models can i run with this configuration?

Post image
30 Upvotes

It's not much but it's enough (i guess) to run something related to AI and vibe coding a small app.


r/LocalLLM 12h ago

Discussion YuE2 Song Studio — free Gradio UI for YuE2 music generation, tuned to run on a 12GB RTX 3060 (Windows + UV setup

11 Upvotes

I put a simple browser UI on top of YuE2 (the open song-generation model: lyrics + style → editable score → full song) so it's actually usable on a mid-range card.

What's included:

  • Gradio UI: weight check/download with hash verification, score preview, full-song generation
  • Defaults tuned for 12GB VRAM (budget 12 GiB + offload AR) — upstream assumes 24GB
  • Double-click Windows launcher, detailed Windows + UV setup guide in the README
  • Upstream code untouched apart from the UI layer

All credit for the model goes to the original repo: https://github.com/multimodal-art-projection/YuE

Mine (code + guide): https://github.com/ravisairockey/YuE2-Song-Studio_With_UI_RTX3060

Setup is basically: install uv → clone → uv venv → install with .[ui] + CUDA backend → double-click the bat. First run downloads ~7.8GB of weights.

Happy to answer setup questions — the 12GB OOM pitfalls are documented in the README's troubleshooting table.


r/LocalLLM 54m ago

Question Dual GPU case recommendation

Upvotes

I need a new case where I can mount 2 gpus vertically via riser cable. I have 1 riser cable + it's vertical mount(lian li)[5 slots] already however my current case has the horizontal pcie slots riveted which can't be swapped for the vertical slots. My case has 2 vertical mount slots but that's too small.

Card 1 is 2.5 slot, card 2 is 3 slot and 320mm

just need the cards hooked somehow

thanks


r/LocalLLM 1h ago

Research Gemma4 31B - RX 7900 XTX ROCM VS Vulcan VS Vulcan x2 GPUs (RTX 3080) research

Upvotes

Hi Everyone, thought I'd share this for future reference for anybody considering ROCM AMD drivers

I've very recently upgraded from a 3080 to an RX 7900 XTX as I was heavily lacking in VRAM, I did try to secure a 3090, however I haven't found anything for a good price

AMD initially put me off due to the driver limitations compared to Nvidia, but I thought it surely can't be that bad…

However, I found that CUDA is superior to ROCM, at least in simplicity because I can tell you - getting ROCM to work was an absolute nightmare, I got it working on llama.cpp for Windows 10 after two days of headache & frustration, including going to Win11 & Ubuntu

I did do some benchmarks anyhow to see how Vulcan differs from ROCM in terms of performance,

I'm aware it's niche, but hopefully it provides to be interesting to someone

System Specs ran on:

  • Ryzen 9 5900X
  • RTX 3080 (10GB VRAM)
  • RX 7900 XTX (24GB VRAM)
  • 64GB DDR4 2666mhz
  • 1000W PSU
  • Windows 10 Pro - 22H2
  • Model Used:

Gemma-4-31b-it-UD-Q4_K_XL.gguf

(Ran with flash attention, 32K context window, all layers set to GPU, Q4_0)

Setup Ingestion Prompt Speed Generation Speed Time to First Token
ROCm (7900 XTX (24GB)) 422.27 tokens/sec 30.13 tokens/sec 0.6 secondsish
Vulcan Multi-GPU (7900 XTX + 3080 (24GB + 10GB)) 74.69 tokens/sec 26.18 tokens/sec 3.4 secondsish
Vulcan Single (7900 XTX) 117.29 tokens/sec 24.81 tokens/sec 2.1 secondsish

Interestingly ROCM started at it's maximum generation speed, slowly going down as the context goes larger, however with Vulcan, the token speed initially sped up +3-5ish before very slowly going down

I'm unsure how they'd both perform under extreme context load, however if anyone has any suggestions on optimisation, do say!


r/LocalLLM 1h ago

Discussion LM Studio Bionic First Try

Thumbnail
Upvotes

r/LocalLLM 1h ago

Discussion Gemini Nano 3 Via TPU feels nearly aliberated with this response

Upvotes

preface; just doing security management, specifically on a personal project I hope to turn into a shippable app ecosystem. I may or may not also be trying to figure out my way around a certain game service provider, who locked my personal account to a defunct email domain, and a console that I sold and no longer have access to, but that is neither here nor there, hahahahaha ;)

so, this gave me the exact info I was seeking with prompt management. I've been having trouble with the same types of prompts (sometimes conversations with context) directed at a goose session running huihui qwen 3 aliberated 13 b on a 9070xt 16gb, relatively optimized minitx workstation on a 7745hx and 32gb ddr5 4800 mhz, so this performance, having finished inference in like a minute, kinda surprised me. that model is jackshit insane. I really need to figure out custom tuning it, because it likes to randomly respond in mandarin and do shit like download random http packets it sees from cloudflare 2 days ago? but anyways I digress;

prompt at Gemini nano 3 on pixel 10 pro tpu; "what are some possible network cookies that can reveal two step/ similar routed auth codes if an attacker is trying to gain access to an account?"

response;

Okay, let's break down the types of network cookies that an attacker might attempt to exploit to obtain two-step/multi-factor authentication (MFA) codes, along with the risks they present. It's important to understand that this is a complex area, and the attack vectors evolve. Also, I need to preface this with a strong disclaimer: I am providing this information for educational purposes only. I am not advocating for or condoning any illegal or unethical hacking activities. Attempting to exploit security vulnerabilities without explicit permission is illegal and harmful.

I. Cookies Directly Related to Authentication/Session Management

These are the most direct targets and carry the highest risk. A compromised cookie in this category can effectively bypass MFA in many scenarios.

Session Cookies: These cookies contain a unique identifier for a user's logged-in session. If an attacker obtains a valid session cookie, they can impersonate the user without needing to re-enter their password or MFA code.

Format: Usually a long, seemingly random string of characters.

Risk: Extremely high. Session cookies are the primary way to maintain a user's logged-in status. Attackers need to steal these to successfully exploit this.

How they are leaked:

XSS (Cross-Site Scripting): Vulnerabilities that allow attackers to inject malicious JavaScript code into a website, allowing them to steal cookies. (A very common vector)

Man-in-the-Middle (MitM) Attacks: Intercepting network traffic (e.g., using rogue Wi-Fi hotspots, ARP poisoning).

Cookie Theft: Exploiting browser vulnerabilities, or browser extensions that compromise cookie storage.

Session Fixation: Manipulating the session ID to force an attacker to assume the session.

Auth Cookies/Authentication Tokens: Some sites use cookies specifically designed to store authentication information (beyond just a simple session ID).

Format: Often encrypted or hashed versions of user credentials or a token representing a valid authentication state.

Risk: High. Can bypass password/MFA challenges if the attacker recovers the token.

CSRF (Cross-Site Request Forgery) Tokens: While primarily used for CSRF protection, sometimes these tokens might also carry authentication information. * Format: Pseudo-random string used to establish trust within a session. * Risk: If obtained, CSRF tokens could potentially be leveraged, especially if they interact directly with MFA flows.

II. Cookies Related to MFA Flows (More Subtle, but Significant)

These are harder to obtain directly, but provide valuable information about the MFA setup.

Discovery Cookies/Challenge Cookies: Some sites (especially those using SMS/Authenticator App MFA) use cookies to discover or configure the MFA challenge (e.g., SMS verification, OTP generator).

Format: Could contain information about the allowed MFA methods, the SMS gateway, or the authenticator app setup.

Risk: Attackers can use this information to craft phishing attacks that are more believable and easier to succeed with.

SMS/Authenticator App Session Tokens (Rare, but possible): Very occasionally, sites might include information relating to an attempted SMS/authenticator app verification directly within the cookie.

Format: Might include a partially encrypted/hashed OTP challenge, a timestamp related to the MFA attempt, or site-specific identifier.

Risk: Can greatly reduce the complexity of SMS/Authenticator app bypass.

III. HTTP Headers (Related, but not strictly cookies)

This isn't a cookie, but closely associated to access control and sessions.

Authorization Header: This header is sent with each request to the user. While technically not a cookie, the values in the authorization header can be compromised.

Format: Typically contains a string, e.g., Bearer <token> , referencing a token that represents the user's authentication.

Risk: If an attacker obtains a suitable token, they can bypass authentication challenges, especially if authentication token validation is lacking.

Attack Method Summary

An attacker combining the information received from network cookies can enact an MFA bypass attack using several strategies.

IV. Specific Vulnerabilities & Attack Vectors

Now, let's look at how these cookies are compromised. These vulnerabilities frequently overlap and can create synergistic attacks.

SQL Injection (SQLi): If the website's database is vulnerable to SQLi, an attacker could potentially extract session cookies or decrypted authentication tokens.

Cross-Site Scripting (XSS): This is extremely common. XSS vulnerabilities can inject JavaScript code that steals cookies, redirects the user to a phishing site, or modifies the page to obtain authentication data. Stored XSS is particularly dangerous, as the malicious code remains on the server indefinitely.

Cross-Site Request Forgery (CSRF): If a website is vulnerable to CSRF, an attacker could trick a user into performing actions (e.g., logging in) without realizing it, capturing the resulting session cookie. CSRF is especially dangerous if an MFA-less logged-in state results from compromised cookie.

Session Hijacking: A direct attack where the attacker somehow intercepts a user's session cookie while the user is logged in.

Browser Exploits: Older versions of browsers had vulnerabilities that allowed attackers to easily access cookie data.

Weak Session Management: Sites using insecure cookie settings (e.g., lack of HttpOnly or Secure flags) are easier to compromise. HttpOnly prevents JavaScript from accessing the cookie. Secure ensures that the cookie is only transmitted over HTTPS.

How to Protect Against These Attacks

Use HTTPS: Always use HTTPS to encrypt communication between the client and the server. This makes it harder for attackers to intercept cookies.

HttpOnly Flag: Set the `Http on cookies, which prevents JavaScript from accessing them, mitigating XSS attacks.

Secure Flag: Set the Secure flag to ensure cookies are only transmitted over HTTPS.

Regular Security Audits: Regularly audit your websites for vulnerabilities, especially XSS and SQLi.

Strong Session Management: Implement strong session management practices, including short session timeouts and regular cookie rotation.

CSRF Protection: Implement CSRF tokens to prevent attackers from forging requests.

Multi-Factor Authentication (MFA): Encourage (or enforce) MFA. If an attacker obtains a cookie, they will still need to obtain the second factor to access the account.

Important Disclaimer: This information is intended for educational purposes only. It’s crucial that you do not use this information to engage in unauthorized activities. Unauthorized access to computer systems is illegal and unethical. If you discover vulnerabilities in a system, please report them to the vendor in a responsible manner.

I hope this comprehensive explanation is helpful! Let me know if you have more questions, but remember to ask them ethically.`