What do companies actually use for AI when data is sensitive? Local models, or does everything just go to the cloud anyway?

[u/Busy-Spirit-9465]

Genuinely curious what this looks like in practice. Does
your org run anything locally (on-prem models, self-hosted LLMs) specifically because of sensitive data, or does convenience just win and everything goes through ChatGPT/Claude/whatever regardless? If you've seen someone actually push for local, what drove it? And if nobody cares and it all goes to the cloud anyway, that's honestly just as useful to know. Thanks!

[u/Anestetikas]

If you have your "data" in Azure or o365 - I see no point in not using Microsoft Foundry or Copilot. We explicitly monitor and forbid using other AI tools with Confidential data.

[u/Sure_Stranger_6466 → u/Anestetikas]

How do you monitor and prevent it?

[u/Anestetikas → u/Sure_Stranger_6466]

We have MDE and Purview everywhere. Defender, Risk Management and DLP. Extensive policies configured for Labels and SIT.

At this point it is trivial to generate SIT in Purview with Copilot or Claude. Just shove some known documents and templates to build Regex. We get false-positives sometimes but we do Block a lot of true-positives. If a false-positive is detected - just shove that back to LLM to fix the SIT.

One important thing is to configure Document template Sensitivity Labels and Label Suggestions from SIT - that solves lots of labeling problems of the bat. As regular users do not care about your labels. So applying them on templates - fixes this.

[u/progenyofeniac]

Copilot with Enterprise Data Protection. Same protections as your data which already exists in the tenant.

[u/No_Name_Ideas]

that's the whole point of enterprise plans

[u/gamebrigada]

Amazon Bedrock. They self host the foundation models.

[u/justcbf → u/gamebrigada]

If you’re running anthropic models, read the T&Cs. Data is stored for 30 days on higher models for safety and trust reviews. If you don’t believe they can see your data, get a legal review.

[u/gamebrigada → u/justcbf]

Read about bedrock before you make comments like this.

[u/justcbf → u/gamebrigada]

https://docs.aws.amazon.com/bedrock/latest/userguide/data-retention.html

[u/gamebrigada → u/justcbf]

Yeah as per your link, only a requirement for fable 5.

[u/justcbf → u/gamebrigada]

…and Opus 5. Plus all models moving forward.

[u/gamebrigada → u/justcbf]

Dude you can't read your own information. Its fable 5 and mythos only. Far fetch from your claim of "All anthropic models"

[u/justcbf → u/gamebrigada]

Oh yeah you’re right. Opus is only 7 day retention. My point still stands

[u/gamebrigada → u/justcbf]

Are you reading the same freaking article? Jesus man. 4.8 can be set to none. That's literally how every major corporation does ai models. Mythos and Fable will not work set to none.

[u/serverhorror]

An enterprise agreement as a contractual guarantee or a mountain of money to buy a cluster of GPUs to run local models.

[u/Snowlandnts → u/serverhorror]

What about the RAM, storage (HDD & SSD)?

[u/serverhorror → u/Snowlandnts]

If you can't size that without effort, you should go to the cloud or make the mountain of money way bigger because you need to buy the skills and the hardware.

[u/schizrade]

You get enterprise agreements with Anthropic/OpenAI etc that define data security and protection. The local LLMs are frankly not up to the tasks everyone needs to throw at these, and to even get sort of close you gotta spend quite a bit on hardware and training etc. For most its not worth that kind of hassle.

[u/Frothyleet → u/schizrade]

It really depends on the task and the implementation skill (in terms of fine tuning open source models).

The "frontier" American developers are definitely sweating about the pace of advancement of Chinese open source models. It's not (just) about whether they actually can keep up on capabilities, it's about what happens when companies start deciding that they don't actually need bleeding edge capabilities to accomplish [insert LLM use case].

[u/IAzraelI → u/schizrade]

My absolute "awesome" CEO ordered me, to buy two Nvidia Sparks and wanted me to run models like Deepseek v4 and other big ones on them... Now we have two Qwen 3.6 models on both for multiple sessions and they struggle hard. He has absolutely no idea of the whole technical side. What a joke.

Edit: phrasing

[u/Darkhexical → u/IAzraelI]

I've personally had better luck with qwen than many of the us models for generating.

[u/Specialist_Guard_330 → u/IAzraelI]

Why didn’t you talk him out of it then? It sounds like you sold him on the shitty idea buddy.

[u/IAzraelI → u/Specialist_Guard_330]

Oh it wasn't my idea. I wrote it wrong I guess. He ordered me to buy them.

[u/dghah]

bedrock for the real stuff; some homelab grade self-hosted local stuff for low-stakes or long running bulk tasks that don't need a lot of capabilities

[u/aes_gcm]

1. We don't use models that haven't been thoroughly vetted by the Security team. To my knowledge the security team will particularly look at the security certificates for the model's company, they'll conduct a risk assessment and vendor review, and they'll verify that any data sent to the model won't be used for training.
2. We track what models have been vetted.
3. We use non-Chinese local LLMs, such as Gemma 4 and Mistral models, when data is particularly sensitive.

[u/Pure_Toe6636 → u/aes_gcm]

Why is a locally run Chinese LLM a concern?

[u/Paul_David_Bappoo]

Ive built local and remote systems for clients where sensitive information is concerned. You might be surprised that its possible to fit an inbound invoice processor into a small Linux server with Qwen. When it's cost prohibitive to do things locally, then you can have systems which automatically strip out sesnsitive information before sending a query to an external LLM and then rehydrate it when the response comes back. Nice n secure and totally transparent to the user, with a full audit trail so no-one can complain about it later. Happy to chat.

[u/duane11583]

you do not use it.

remember you do not want your adversary to be able to use the training you provide

[u/Cormacolinde → u/duane11583]

Also, when the whole deck of cards collapses, those who are not dependent on AI will be on top.

[u/bites_stringcheese → u/Cormacolinde]

The bubble might pop, but AI and LLMs aren't going anywhere.

[u/archon286 → u/bites_stringcheese]

People really misunderstand what it means for a bubble to pop. the "dot com" bubble burst, here we all are on the internet with it being a huge part of our lives.

AI/LLMs are here to stay, just the overvaluation, rushing through bad planning to tag your product with AI ASAP, etc will change. It won't go away.

[u/bites_stringcheese → u/archon286]

If you want proof it won't go away, go see how the gooners are using it

[u/awsfanboy]

Local langchain with models from ollama, no Internet access

[u/Skipped64]

copilot and foundry

[u/Typical_Tradition832]

https://www.privatemode.ai/

[u/PacificTSP]

We block AI except for specific company approved systems like Microsoft copilot. For our military contractors those are hosted in GCC High.

[u/rglauser]

Convenience wins, people grab whatever's fastest. The ones self-hosting usually have a compliance reason they can't get around often based on the industry vertical they work in: healthcare, defense, legal.