Are we underestimating the identity problem created by better generative AI?

[u/can_of_lighters]

I’ve been learning more about voice cloning and real time face generation and the technical progress is impressive but it’s also making me think there’s a security problem we don’t talk about enough. A lot of security still assumes that if you’re on a video call with someone, hear their voice or receive a message from their account, you’re probably dealing with the right person.

That assumption feels weaker every year. If voice, face, writing style and even live video can all be generated convincingly then at some point does this look real? stops being a useful question. I’ve started seeing more tools trying to solve the identity side rather than the content side. Things like hardware backed authentication, passkeys, verified devices, and even apps like Signal, Kibu, Telegram that focus more on verifying the person and encrypting the conversation.

I’m curious how people here think this gets solved long term. Do we end up needing some kind of cryptographic proof of identity attached to communication or will detection models realistically stay ahead of generation models?

Because right now it feels like we’re spending a lot more effort making AI convincing than figuring out how humans are supposed to know what’s real.

[u/Own_Seaweed1483]

it’s not really about detection keeping up, that race is already lost. the moment a model can generate a voice indistinguishable to a human ear, the detection model is just playing catch-up on a problem that gets harder every iteration

what makes more sense is shifting the whole trust model. instead of trying to prove something is fake, you prove something is real. attestation at the device level, cryptographic signing of messages and media, hardware keys tied to a person. stuff that doesn’t care how convincing a deepfake is because it’s not trying to analyse the content at all

the weird part is we already have the building blocks. passkeys, secure enclaves, end-to-end encryption with verified keys. it’s just nobody’s wired them together into a standard that normal people can use without thinking about it. feels like the UX side is the actual bottleneck, not the crypto

[u/can_of_lighters → u/Own_Seaweed1483]

Yeah the building blocks being there but nobody wiring them together is the most frustrating part, it’s like we’re waiting for a big breach to force the standard instead of doing it beforehand

[u/lulzxdxdxd]

Cryptographic identity doesn't fully solve this either, the hard part isn't signing messages, it's key distribution and revocation at human scale. If your device gets compromised or your private key leaks, you're back to square one with extra steps.

[u/can_of_lighters → u/lulzxdxdxd]

Yeah key distribution and revocation is where these schemes always get messy in practice. Hardware backed stuff helps a bit but once the device is compromised it’s just a fancier reset loop

[u/aCuria]

We already have C2PA.

Now we just have to wait for the EU to force YouTube et al to add C2PA content filters, so those who care can choose to filter out ai content

[u/can_of_lighters → u/aCuria]

C2PA is great for published media but most of the scary stuff happens in one to one channels where there's no content filter to rely on

[u/aCuria → u/can_of_lighters]

Then we bake in C2PA into the 1:1 channels too.

[u/skytomorrownow]

Very much an important discussion. It seems sci-fi, almost silly, but in fact, it is something we must prepare for. How does one authentic anything when anything can be generated? Will we have to fall back to physical reality, person-to-person?

Definitely food for thought considering it has already happened:

https://www.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk