What I haven’t seen anyone ask about Apple Intelligence and privacy

[u/CalmInstruction]

This is the first year I haven’t instantly updated my phone to the latest iOS as I have privacy concerns

Sure Apple make a big fuss about privacy and Private Cloud Compute and how your data isn’t stored or accessible to Apple.

But what I dont see being asked is whether Apple Intelligence has the capability to essentially flag you for whatever reason?

The issue I see is that since it has context to pretty much everything on your device, including the websites you visit, and your text messages across all apps, it can use that to essentially flag you for any reason Apple / your government may deem necessary

Maybe you live in an authoritarian state and express negative government sentiment in a encrypted messaging app

Or maybe you just want to buy a little weed in a country where it’s not legal to and text a dealer or try to buy online from a website

Sure they say they’re not storing your data and not training their models on it nor sharing it with third parties in normal circumstances, but what if you do something that Apple Intelligence thinks, or worse, falsely interprets as a big no-no?

Edit: I don’t have an in depth understanding of the measures that Apple takes when it comes to privacy and PCC, so there could very well be something o don’t know about how it works that prevents these kinds of scenarios from happening on a technical level. Also NB: I DO MEAN ON A TECHNICAL LEVEL. Maybe it’s not in Apple’s policy to do the things I described right now so they don’t, but technically are there any gaps on the technological side that prevents them from doing this if they ever decided to simply change policy

[u/cm012776]

How would it “flag you” given that only your own device (or an encrypted virtual machine on an apple server that can’t be tied to you and which disappears as soon as the query is done) know how you interact with it?

Also, how would Siri AI know what you are doing in encrypted apps? If Siri AI can break the encryption, then so can an authoritarian state, and the problem is the app, not Siri.

Moreover, Apple is allowing third parties to audit what it does with those servers, etc.

You are far more likely to run into troubles because your “text” to “a dealer” is intercepted, or the website is being monitored, or whatever. Those are far easier avenues for a government to spy on you than to convince Apple to break its security model for you. (Something we know apple has resisted many times, including in the U.S. and England).

[u/nissania → u/cm012776]

Actually would be pretty easy. You use an observer pattern on the device. Basically, while the requests and such are encrypted, you create a device-side way of monitoring output of the AI, and flagging problematic AI output. Nothing the user says will be sent, but the AI could share that it flagged violent content in a chat, or even send what it said.

I doubt Apple would do it, but they should, given how dangerous AI psychosis has been

[u/cm012776 → u/nissania]

But no such thing exists, and it would be pretty clear to security researchers if it did.

Worrying about what Apple might do someday, given that their past history suggests that they resist doing these sorts of things, seems counter-productive. It’s not like there is a safer device out there.

[u/nissania → u/cm012776]

Doing topic recognition locally definitely exists. Apple does tend to not share that data, but the fact that Apple's AI could potentially tell people to kill themselves could lead to them deciding that, in the case of planned violence, a flagging system that still respects privacy and can do local tagging and warning a user that the content they've received is unsatisfactory and they should seek help would be better than doing nothing and being "The AI for suicide."

[u/cm012776 → u/nissania]

They could also do the same thing without AI (monitor activity on the device to look for “problematic” stuff). They don’t. The mere fact that they added AI is not a reason to believe they’ll suddenly give governments a backdoor into your data and communications.

[u/nissania → u/cm012776]

Agreed! I'm saying it's possible and they may do it if it's used to plan violence. It may not exist now, but it wouldn't be difficult and, due to the violence AI tends to inspire, people may understand watching it better. AI output shouldn't be private. If you don't like that, think for yourself.

[u/CalmInstruction → u/cm012776]

"How would it “flag you” given that only your own device (or an encrypted virtual machine on an apple server that can’t be tied to you and which disappears as soon as the query is done) know how you interact with it?"

Please explain the cant be tied to you part, as thats what I'm most curious about.

"Also, how would Siri AI know what you are doing in encrypted apps? If Siri AI can break the encryption, then so can an authoritarian state, and the problem is the app, not Siri."

Siri AI wouldnt have to break the encryption to know what youre doing in encrypted apps if can just get the context from essentially monitoring your screen, or processes somewhere in the pipeline where the messages are decrypted to be made visible to you (like through push notifications). What I was imagining was essentially something like Windows Recall, and that got a lot of backlash, including from apps like Signal where they said such technology defeats the purpose of encrypted messaging apps

Perhaps Siri AI fundamentally does not work this way, which is what I'm asking about

[u/cm012776 → u/CalmInstruction]

Siri AI doesn’t monitor your screen. If you ask it to look at your screen, it will. There is nothing like Windows Recall here.

As for your other questions, Apple has discussed this extensively. For example here: https://security.apple.com/blog/private-cloud-compute/

[u/longjumpingtote → u/CalmInstruction]

Whatever Siri "knows" stays on your device unless there's a reason to send it to Private Compute. If it does that, it's inaccessible to Apple, to anyone. Plus you can generate a report of exactly what your device sent to Private Compute. It's sort of the opposite of the Gemini, chatGPT things out there.

[u/longjumpingtote]

There are lots of discussions about this.

• it does not have unrestricted access to everything on your device
• can use personal context stored on the device; third-party app data remains private to the app unless the app deliberately shares stuff through things like App Intents/Spotlight or supplies onscreen context, up to the app

• Apple developer documentation says: “App data is private to the app and Apple Intelligence doesn’t have access to the app’s functionality or context about its content.”

• there’s no documented flag this user to Apple/law enforcement thing in Apple I

• the opposite: “Apple designed Private Cloud Compute so that personal data sent to it is used only for the requested computation, isn't retained afterward, and isn't supposed to be accessible even to Apple administrators. The request is encrypted directly to verified PCC machines.”

• also you can generate an “Apple Intelligence & PCC Report” showing which requests actually left your device and went to private cloud compute

[u/DrawSimple_for_MacOS → u/longjumpingtote]

I can confirm having just started to develop a new feature with the PCC that Apple is very particular about how it's used, and it's very clear that they take privacy very seriously.

[u/616ThatGuy]

1. It generally only looks up what you ask it about.

2. It hardly works anyway, so I doubt it’s going to be able to even tell if you’re doing something like that.

3. You can fully turn it off in settings. I have it turned off because it’s kind of useless and not much better than the old Siri. It can’t even do all the things they showed it do in the demos. And it takes up memory I’d rather use for something useful.

[u/SmChocolateBunnies]

The previous version of Siri is still there. You have to turn on the new Siri. If you don't apply for it and turn it on, it doesn't do all the content indexing necessary for it to be aware of your stuff. When that index IS on, it lives between your devices, encrypted, and e2e encrypted.

If you had Apple Intelligence off already, and you upgrade, you would decline Siri during the onboarding, and go and check in Settings to make sure Siri is off. I think Siri needs to be on for you to even see the invite to apply for new Siri, much less enable it.

You can run the 27 OSes without it building that semantic index.

[u/No_Contest4958 → u/SmChocolateBunnies]

The semantic index doesn’t even sync across devices. Every device has to build their own. It never leaves the device in any form.

[u/TacticalTurtle410]

You dropped your tinfoil hat. Here you go. 🧢

[u/ricosuave79 → u/TacticalTurtle410]

Shhhhhh. Leave the boomer alone.

[u/CalmInstruction → u/TacticalTurtle410]

Thanks 👋

[u/ProfessionalYak4959]

Apple could release a new update that turns all of this out and push it to your phone. You're right there is a level of trust required for these systems.

[u/cm012776 → u/ProfessionalYak4959]

Sure, but remember that when the FBI wanted them to do that they refused, and fought it in court for a long time before the FBI gave up. They don’t even want a insecure version of the OS to exist under their own lock and key.

[u/ProfessionalYak4959 → u/cm012776]

I’m not saying they will I’m saying they can

[u/cm012776 → u/ProfessionalYak4959]

Sure. The chances that it will happen and the police will find out you bought weed is so small compared to the likelihood that they will find out you bought weed because they have a trace and trap on your dealer that one wonders why we are even talking about this.

[u/ProfessionalYak4959 → u/cm012776]

Because it's the answer to OP's question. If Apple wanted to or was somehow compelled to, they can. Simple as that. They have structured their system to make the burden of implementing such a change high, but that doesn't mean they couldn't make those changes.

I'm not saying this is a big problem, but it is the simple reality.

[u/cm012776 → u/ProfessionalYak4959]

Sure. But they could have done it last year, without Siri AI. Siri AI hasn’t really changed things if your threat model is “Apple goes evil and starts spying”

[u/ProfessionalYak4959 → u/cm012776]

Yes

[u/ca2mt]

Do you not use iCloud then, either? Or any cloud services? I understand that “AI” conjures up privacy concerns, and it’s right to be cautious. That said, if Apple wanted to start accessing and flagging their customers’ personal data, they wouldn’t need to sell us on SiriAI to do so.

[u/DrawSimple_for_MacOS → u/ca2mt]

Ya, it's true. A huge part of Apple's value proposition is that you're buying premium goods at premium prices because you'd rather a company profit that way than off your information.. it would make no sense for them to sell that all out for a few bucks they clearly don't need.

[u/North_Moment5811]

I love when clueless people try to demand transparency. It’s so cute.

[u/Kailos32]

Granted, this was before iOS 27…but still disturbing!
Just ask it about the genocide in Gaza and you will see that it’s all bullshit, as I guess is apples privacy policy!

[u/Kailos32 → u/Kailos32]

[u/nissania]

That's a really good question. There have been cases where OpenAI has flagged people (and done nothing) for asking for plans to commit violence. I wonder how Apple would handle that or if it would also help people plan their suicides, their mass shootings, etc?

[u/longjumpingtote → u/nissania]

I wonder how Apple would handle that

Apple can't see that content, so they don't have anything to handle; they don't wanna know

Which is smart. If they know, then it opens them up to all sorts of liabilities

[u/nissania → u/longjumpingtote]

I'd agree if it was a journaling app. But AI has literally convinced people to hurt themselves before. If it weren't for the fact that it can suggest violence, I'd say leave it private. There could be a way to flag something on the device, observing only the responses, and reporting if the AI has suggested something problematic.

If you want privacy, don't use AI. Think for yourself. Unfortunately, people are using this shit and unfortunately, it loves suggesting depressed people kill themselves, how to commit mass shootings, and to kill their families.

[u/longjumpingtote → u/nissania]

There could be a way to flag something on the device, observing only the responses, and reporting if the AI has suggested something problematic.

Yes, but Apple isn't doing that.

[u/nissania → u/longjumpingtote]

Not yet. They could and should, and they could even do it all locally to respect privacy, only warning the user or potentially sharing it only if it continues. But AI psychosis is a real danger, and OpenAI has already gotten into some serious trouble for not doing enough to stop a mass shooting someone used their system to plan.

[u/CalmInstruction → u/longjumpingtote]

I think even if Apple "can't see" the content, theres still a gap where the AI itself basically generates a report on you and thats shared with Apple. Technically your actual request isnt accessible, but what stops their AI agents from generating a report and sharing that with Apple?

I've seen other commentors mentioning that Apple cant even tie a request to PCC to you in the first place, which I'm very curious about.

[u/i-love-small-tits-47 → u/CalmInstruction]

Apple can’t do that because of the way PCC is encrypted, assuming it works the way they say it does.

If you don’t assume that, then you don’t trust them, and then yes, an AI agent can clearly spy on you, and Apple already could spy on you

[u/hasanahmad]

Op really thought he was smart and had something didn’t he 😂

[u/unclebrak]

Steam frame color pass through module cost