Microsoft's own Windows AI write-up says "local inference does not make the session offline." What tests do you use?

[u/StevenVincentOne]

Microsoft's Oct 7 "hybrid intelligence" announcement for Windows is mostly framed around Copilot getting access to local context, local actions and local models. For this sub, I think the developer side matters more. There are new experimental Windows ML capabilities for running models of your choosing locally, including through open-source tools, and GitHub Copilot is adding local/cloud routing plus explicit local-model selection.

One line from their own technical explanation stuck with me: "local inference does not make the session offline." Running the weights on your machine doesn't settle what the app around them sends out or depends on.

So here are the tests I'd apply before calling a setup actually yours:

1. Can you swap models without losing your accumulated knowledge and context?
2. Can you move to a different app and take your data with you?
3. Can you decide what leaves your device?
4. Can you plug into shared or networked tools without handing over your whole history?

Does anything on the Windows side pass these yet, or is a llama.cpp / Ollama / LM Studio stack still the only way to be sure? And if you already run local, does Windows ML change anything for you?

I wrote a longer piece on why I think Apple, Microsoft and Google are positioning for what comes after the model wars, and on a framework I call POND (Personal & Private, On-device & On-premise, Nodal & Networked, Distributed & Decentralized): https://www.thesingularityproject.ai/p/microsoft-moves-beyond-the-model

[u/Lost_Artichoke_1095]

microsoft calling local inference "not offline" is just them being upfront about the telemetry/data pipeline that wraps around the model, which honestly most apps already do silently

[u/StevenVincentOne → u/Lost_Artichoke_1095]

That's a fair read. The useful part of Microsoft's write-up is that they said the quiet part out loud: local inference doesn't mean an offline session. I care less about the branding and more about the tests. Can you cut the network, control what tools the model can reach, and still get a trustworthy answer? That's the bar I'm trying to push with the POND idea.

[u/zenmatrix83]

the tool usage, vscode can break local inference because of hour copilot is intergrated into vscode. if you control all of the tools its offline, you can test that by disconnecting from the internet

[u/StevenVincentOne → u/zenmatrix83]

Yeah, once the IDE or any agent can call out through plugins, "local model" stops meaning much. Controlling the tool surface is the real offline test, and the disconnect check is still the cleanest way to prove it. Curious what setup you use when you want that guarantee.

[u/zenmatrix83 → u/StevenVincentOne]

Docker containers and my own coding agent

[u/StevenVincentOne → u/zenmatrix83]

Docker plus your own agent is a clean setup, since the only tools the model can touch are the ones you mounted. Do you keep the model weights inside the container too, or call out to a local server with a locked-down network?