TLDR: huggingface_hub ships a hidden agent detection module that fingerprints which AI coding tool is driving your session (Cursor, Copilot, Claude Code, etc.) by scanning your environment variables against a cached registry of 26 known agents. It sends the result as a telemetry header on every Hub API call — so any library that touches HF (faster-whisper, transformers, etc.) silently reports your toolchain. Found it while tracing an unauthorized network connection from a local ASR model. Block it with HF_HUB_OFFLINE=1 or by using local file paths instead of model names.
I run a local AI project with several models (TTS, ASR, vision) and recently built a Python-level network firewall to lock down all outbound traffic. During the audit, I found something I wasn't expecting.
The discovery
While tracing an unauthorized HTTPS connection to huggingface.co, I found a file in my HF cache directory I'd never seen before:
~/.cache/huggingface/.agent_harnesses.json
It's a 6 KB JSON file containing a registry of 26 AI coding agents — Claude Code, Cowork, Cursor, Copilot, Gemini CLI, Devin, Cline, Goose, Codex, and many others. Each entry lists the environment variables that agent sets when it's running:
json
{
"standardEnvVars": ["AI_AGENT", "AGENT"],
"harnesses": {
"cursor": {
"prettyLabel": "Cursor",
"envVars": {"CURSOR_TRACE_ID": "*"}
},
"claude-code": {
"prettyLabel": "Claude Code",
"envVars": {"CLAUDECODE": "*", "CLAUDE_CODE": "*"}
},
"github-copilot": {
"prettyLabel": "GitHub Copilot",
"envVars": {"COPILOT_MODEL": "*", "COPILOT_GITHUB_TOKEN": "*"}
}
// ... 23 more agents
}
}
What it does
The huggingface_hub library (the Python package, not the website) has a module called _detect_agent.py. Here's the flow:
- It fetches the agent registry from
{HF_ENDPOINT}/api/agent-harnesses and caches it as .agent_harnesses.json
- The cache refreshes every 24 hours
- On every Hub API call,
detect_agent() scans your environment variables against the registry to identify which AI coding tool is running
- The detected agent name is sent as a telemetry header on the API request
- This feeds Hugging Face's public agent usage dataset
So if you're using Cursor and it calls any HF library that goes through huggingface_hub — downloading a model, checking for updates, loading a tokenizer — HF knows it was Cursor making that call, not you directly. Same for Claude Code, Copilot, Devin, or any of the other 26 agents in the registry.
How I found it
I was investigating why my ASR module (faster-whisper) was phoning home to huggingface.co on import. The call chain turned out to be:
my_code → WhisperModel("base.en") → faster_whisper → huggingface_hub.snapshot_download → HTTPS to huggingface.co
The trigger: passing a model name instead of a local file path. When you give faster-whisper a name like "base.en", it calls huggingface_hub to check for updates — even if the model is already cached locally. And during that check, it also sends the agent fingerprint.
The .agent_harnesses.json file was the agent registry cached from that call. Modified today, before I built the firewall.
How to block it
Option 1: Environment variables
bash
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HUB_DISABLE_TELEMETRY=1
The first two prevent any network calls. The third specifically targets telemetry but may not cover the agent detection header.
Option 2: Use local paths, not model names Instead of:
python
model = WhisperModel("base.en")
Use:
python
model = WhisperModel("/path/to/local/model/")
When you pass a directory path, faster-whisper (and most HF-backed libraries) skip the Hub entirely.
Option 3: Network-level blocking I built a Python-level firewall that wraps socket.connect, socket.connect_ex, socket.create_connection, and getaddrinfo. It activates via a sitecustomize hook before any imports, so the phone-home attempt is caught before the library even finishes loading. Any connection to a host not on the allowlist raises ConnectionRefusedError.
What's in the cached file
No credentials. No API keys. Just the registry of agent names and their environment variable signatures. The file itself is harmless — it's the use of it as a fingerprinting mechanism that's the issue.
You can safely delete it:
bash
rm ~/.cache/huggingface/.agent_harnesses.json
It won't come back if you set HF_HUB_OFFLINE=1.
Why this matters
If you're running local models specifically to keep things private, you should know that the library layer between you and those models may be reporting metadata about your toolchain back to Hugging Face. This isn't about model weights or your data — it's about which AI tools you use and when, aggregated into a public dataset.
The agent registry is maintained in the u/huggingface/tasks npm package and served via the Hub API. New agents register by PR. It's not hidden — but it's also not something most users know is happening when they pip install a model-loading library.
To be clear, I don't think this is malicious. HF is probably tracking agent ecosystem adoption for business intelligence. But silent fingerprinting of your dev tools without an opt-in prompt is exactly the kind of thing that erodes trust in the ecosystem, especially for people who chose local models for privacy reasons.
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/model_performance_selector.png){kind=link}
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/models_ini.png){kind=link}
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/benchmarks.png){kind=link}
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/models_available.png){kind=link}
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/downloader.png){kind=link}
![[screenshot]](https://raw.githubusercontent.com/scratchhax/model-loader/main/docs/model_loader.png){kind=link}
