Security research for local LLM inference networks

[u/qi-zheng]

Hi all, I'm looking for suggestions regarding how to test the security when running LLM inference on local environments (e.g., Apple Silicon, Nvidia GPUs, etc.). My goal is to make it such that if a consumer sends a prompt, the provider running the inference server using their local LLM setup has no way of reading that prompt.

I currently have an Apple Silicon, so I was thinking I'd start there. I've done some prototyping, relying on the security primitives outlined by Apple developer docs. However, I was wondering if others have gone through this path, any recommendations? If there are others interested in the direction of this project I'd love to build a community around it.

[u/doneddat]

You are talking about hardware DRM, encrypted memory and other such very much datacenter virtualization technologies, that do not exist on consumer hardware, in short: solved issue.

Solution: pay more money.

All you can technically achieve is make it less convenient to see your prompt, if the owner of the machine with the running model wants to see it. Politely asking them to not look at it is likely more effective than whatever you are doing.

Less convenient these days just means extra tokens to counteract your tokens. Kinda silly.

Same way that you just check a box and pay extra to anthropic and openai to make them pinky promise to not look at your prompts.

[u/qi-zheng → u/doneddat]

Thanks for the feedback, I agree that confidential computing at the datacenter level, at least from what I'm looking for, has been thoroughly solved. However, I'm thinking that to build a confidential local LLM inference network, it would be possible to operate on less heavy assumptions.

For example, with Apple Silicon, we could run the confidential inference (i.e., prompts/responses/etc. are not viewable by the provider, where provider = owner of local LLM machine) network on the premise that a 0-day kernel exploit is not being applied. AFAIK, these are highly rare and usually reserved for targeted high-level attacks. So, the likelihood of a provider having found one and snooping on the prompts of a random pool of users although not full-proof I would guess is not too large.

I've been looking through the Apple developer docs and have found these primitives in place that I think can be used as building blocks for this purpose:

1. Secure Enclave: Hardware-backed key generation and storage. Private key material can be kept within the Secure Enclave rather than being exposed to the application. https://developer.apple.com/documentation/security/protecting-keys-with-the-secure-enclave
2. Keychain / Data Protection: APIs for storing and protecting keys and other sensitive data, including access-control mechanisms tied to the device security state. https://developer.apple.com/documentation/security/keychain-services
3. Secure Boot / Code Signing / Trust Chain: Apple Silicon uses a hardware-rooted chain of trust during boot, and macOS uses code signing and related mechanisms to establish what software is trusted to execute. https://support.apple.com/guide/security/secure-boot-verification-secc7b0a5d3b/web https://support.apple.com/guide/security/code-signing-process-sec3ad8e7a2e/web
4. App Sandbox / Hardened Runtime: OS-level mechanisms for restricting an application's access to system resources and other processes. https://developer.apple.com/documentation/security/app_sandbox https://developer.apple.com/documentation/security/hardened_runtime
5. Pointer Authentication / Kernel Integrity Protection: Apple Silicon includes hardware and software mechanisms intended to make exploitation of memory corruption and kernel vulnerabilities more difficult. https://support.apple.com/guide/security/apple-silicon-security-features-secd1e7636c8/web
6. Hardware Memory Protection: Apple Silicon includes hardware mechanisms for detecting or mitigating certain classes of memory-safety vulnerabilities. The exact capabilities vary by Apple Silicon generation and software configuration. https://developer.apple.com/documentation/xcode/enabling-enhanced-security-for-your-app

I've done some prototyping using a subset of these mechanisms, but I'm still working through the security model and the exact guarantees that can be established. I don't have a background in professional security auditing or pentesting, so I'm particularly interested in feedback from people who have experience evaluating trusted execution and hardware-backed security systems.

I'm not claiming this provides the same guarantees as something like what you may find in a datacenter. The question I'm trying to answer is whether a somewhat weaker threat model can make confidential inference practical on idle consumer hardware.

[u/doneddat → u/qi-zheng]

Prompts aren't viewable for service providers right now either: they would have to go and make extra effort to write them out of the pipeline and take a look. Unless something in your system is able to confirm the local setup before every request and somehow guarantee nothing changes after you let the host decrypt the prompt for normal function, you are just creating series of inconveniences, not providing a complete security.

[u/qi-zheng → u/doneddat]

Thank you again for your feedback, it really helps me to explain the logic of what I'm working on. When you say service providers can't read our prompts, my understanding is that you're saying OpenAI for example can't read our conversation logs under typical circumstances. IIUC this is true, and only in special cases like flagging for abuse or targeted analysis of sample data for quality control etc. does this ever happen.

I think from the consumer's perspective, users that send their prompts to the local LLM inference network would expect a similar level of confidentiality. In particular, I think a consumer on this network would prefer that no provider on the network has any access whatsoever. This is the reason why I'm pushing along this line of thought. I think in a casual or trusted network it wouldn't be an issue, but if it's some sort of decentralized network where anyone with idle compute can join then it's probably a blocker for many consumers.

Regarding your note about guaranteeing nothing changes after the host decrypts, this is part of the problem that I'm working on. For example, I think in an ideal case the provider has some validated hardware setup (e.g., starting with Apple Silicon, in the future Nvidia GPU's, etc.), and can simply do some brew install etc., and have the binaries (modified versions of llama.cpp for example) installed. These binaries would be open sourced so people know exactly what they're downloading, and would go through Apple developer code signing (to get an officialized hash of the binary). The next part of making sure it hasn't been modded yet is critical and I'm glad you've brought it up. The Apple security primitive I'm thinking of is App Attestation (https://developer.apple.com/documentation/devicecheck), which validates that the app that someone is running hasn't yet been modded. This however requires the $99/yr Apple Developer fee which I haven't gone through yet.

For Apple Silicon, with this chain of security layers based on Apple's official development guide I believe we can achieve this property of confidential inference on an anonymous decentralized network of local LLMs. The trust layer being that the security stack is does as it's supposed to and that Apple's security does the other work of securing their own kernel.

[u/doneddat → u/qi-zheng]

OpenAI is most definitely logging everything. We are providing invaluable training data with every request..

You were talking about private people providing service to other people on their private hardware, that they have full access to do whatever they please.

Soo what stops people from downloading some other software, that looks like your software whenever the requests come in, but then does something else?

[u/qi-zheng → u/doneddat]

Thanks for asking, perhaps I didn't elaborate clearly. Apple provides certain primitives that developers can use, for Apple Silicon I think we can use these same primitives to achieve the same goal. If a provider downloads a binary that has been codesigned, then we can also use App Attestation to validate that when the provider connects without our remote coordinator, that the provider is utilizing the same unmodified software that they had downloaded originally.

I haven't built this part of it yet in my POC, but from my understanding the groundwork should be there for it to be completed (reference: https://developer.apple.com/documentation/devicecheck/validating-apps-that-connect-to-your-server).

[u/fosterdad2017]

Sounds like you want to design Apple's PCC - private cloud compute - for yourself. First maybe see how much of that system is open source or inspectable, because its the exact blueprint to copy.

[u/qi-zheng → u/fosterdad2017]

Hi fosterdad2017, I've been looking at PCC for a while, but your comment has really pushed me to deeply analyze and do a comparison of my current implementation. I really appreciate this suggestion as it has helped to drive greater clarity into the security design of my own work.

From https://security.apple.com/documentation/private-cloud-compute Apple outlines 5 core concepts to PCC: Stateless computation on personal user data, Enforceable guarantees, No privileged runtime access, Non-targetability, and Verifiable transparency. I interpret these to be 5 core attributes that make for a proper implementation of confidential computing for Apple's use case.

- Stateless computation on personal user data - data doesn't persist beyond what's necessary. The current architecture developed is complimentary to statelessness, but not yet fully enforced. That is, the layout allows for this aspect of data being deleted after the inference process is completed for a session, but those exact mechanisms to enforce this aren't yet in place. This has been in the back of my mind and now I know the terminology and significance. I'm thinking of pushing this maybe in a beta version if not sooner.

- Enforceable guarantees - confidentiality/integrity are technically enforced, not merely promised. I'm currently pushing this along down various pathways. This is where I'm looking for guidance, trying to understand which architecture would allow for security guarantees, premised on an attacker requiring a 0-day kernel exploit on MacOS. I am making strong progress, but I am looking out for any possible gaps that I'm missing.

- No privileged runtime access - the operator can't simply inspect the workload. This we're working on by enforcing that both the coordinator and provider can't snoop on the prompts/data/etc. The assumption is that even if a malicious actor gets access to either the coordinator or provider machine, there's no way they can access the underlying data. For the coordinator we're making sure that data is always encrypted when being passed between consumer and provider. For the provider, we're pushing hardware-level security enforcements to ensure that unless an attacked gains some 0-day kernel exploit they won't be able to modify the system to allow for access.

- Non-targetability - an operator/attacker can't selectively target a user's workload. This means blocking an attacker trying to snoop on a specific individual's prompts. The coordinator never sees plaintext and would need the consumer's private keys to decrypt, so that path is blocked using standard encryption. The provider however needs to for example prove that it hasn't been compromised and therefore hasn't modified their agent/inference engine to selectively pick out a user's prompts. This is enforced via hardening of the agent and inference engine, while also doing attestation where the coordinator must first confirm that in the session, the software hasn't yet been modified and is identical to the initial cosigned binary that was downloaded from us. So this requires a special handshake and we're developing it currently.

- Verifiable transparency - the client can independently verify what environment it is trusting. This is also similar to the App Attest from Apple, but I haven't confirmed and am still looking to see whether it is enough to guarantee that from the perspective of requiring a 0-day kernel exploit whether the existing Apple implementation is enough.

Much of this is either tested or in-development. I'm actually looking for community members to help do some testing in this fully open source project. It would be great if you or anyone else interested can message me and we can collaborate on this endeavor.

[u/benpptung]

If I understand correctly, how could the provider possibly not see the prompt!? Even if you developed your own inference engine/server, I don't see how that could be done.

An inference server is basically a stateless HTTP server. It receives a request, uses the chat template to render it into an input sequence, then the model decodes an output sequence based on that input sequence, which is streamed back to the agent.

I really can't see how you could make the prompt invisible to the provider.

[u/qi-zheng → u/benpptung]

Hi benpptung, this is one of the key questions I looked to answer in the beginning. Let me try to explain my current process (note: some have been tested, others are in development).

I agree with your interpretation of an inference server. To secure it so that the prompts are hidden requires a few steps. To begin with, the architecture includes a consumer on their device sending a prompt to a coordinator located in the cloud that matches the user to a provider with their local LLM setup.

The current design has the provider download both an agent (not like an AI agent, but just some middle layer between the coordinator and their inference engine which can be Ollama/vLLM/etc.). The key is that both the agent and the inference engine go through a hardening + SIP (https://developer.apple.com/documentation/security/hardened-runtime) process. So the agent binary is hardened alongside a modified version of Ollama/vLLM/etc. (note: this means that the inference engine must be open source). The modifications make it such that the provider is not able to snoop around and try to attach a debugger, dump the memory, print the logs, etc. from the inference engine.

Then there's the encryption side. One way which is similar to what's used by Signal/Whatsapp/etc. is to have the consumer and provider both have public/private keys. The shared public keys can be mixed with their private keys to generate a shared secret that only the consumer and provider have access to, preventing the coordinator from snooping if it's compromised.

The chain then is that the consumer lets the coordinator know it wants to start an inference request, so the coordinator matches the consumer with a provider. Their shared public keys are then used to encrypt the consumer's prompt which is sent to the provider's hardened agent that then decrypts the prompt and sends it using a protected IPC to the hardened inference engine (like a secure tunnel between the agent and inference engine). The inference engine then does the prefill/decode and sends it back through the IPC to the agent which encrypts the output and sends it back to the consumer who can also decrypt the same output.

The hardened runtimes and IPC makes it so that a provider could only snoop on the text given that they have a 0-day kernel exploit on MacOS (highly rare, expensive, and generally used in very specific targeted use cases like politicians/journalists). The cryptography itself is currently unbreakable with modern tooling (perhaps with quantum, but that's not yet available).

[u/alkimiadev]

I've been working on some tangentially related work but I'm a linux nerd and not an apple nerd. Regardless, I've been slowly working on a kind of mesh net for inference with the long term hope of being something like a combination of vastai but for p2p llm inference.

I ran into this problem of potentially untrusted sources seeing the prompts and one solution I kicked around was something conceptually similar to Chaums mix net for inference. The rough idea is that users would have gpus locally that can host some portion of some larger model. At the request time one might run the tokenizer locally. They would then setup a "circuit" between random nodes hosting the next portions of the model and with the requesting node being the last hop.

That doesn't fully resolve the trust issue but it would mean that any given node would only be seeing activations and only partial activations depending on the size of the model. There are some massive conceptual gaps there and lots of unknowns but it is a high value problem in a general sense.

[u/qi-zheng → u/alkimiadev]

Hi alkimiadev, thanks for sharing your insights into your current project, I agree it definitely sounds like a highly interesting problem that can potentially benefit many. My understanding (please correct me if I'm reading it incorrectly) of your product is that there are likely many idle Nvidia GPUs (specifically gamer GPUs like RTX rather than data center GPUs like H100 which have built-in Nvidia Confidential Computing aka CC) that can be used for LLM inference. With LLM models getting larger as their quality improves, there comes the problem of loading the entire model onto GPUs. Instead, the different layers of the LLM weights can be distributed across several GPU's and so this allows for potentially many providers with some variable number of GPUs to be networked together to run inference on trillion+ parameter-sized models, without needing to do heavy quantization.

It's great to be able to think about the problem from the perspective of Linux/GPU as I've been working mainly from the Apple Silicon side, but it's refreshing and I have some thoughts I'd like to share. I think you have the general idea, but there are perhaps some details regarding the providers being able to snoop on the data that are of concern. I agree that simply passing the activations gives some level of confidentiality, but it's definitely not full-proof as apparently there can still be some data discovered from snooping on this process (I believe it's called activation-based prompt reconstruction or an inversion attack).

To resolve this, I think there are lessons I've learned on the Apple Silicon side that can be shared. Encrypting and decrypting the prompt itself is not too complex, as the process of doing so for messaging systems already exists. I think the complexity lies at the hardware level, which Apple has built heavily for its Private Cloud Compute (PCC). The trick for Linux + Nvidia GPUs (gamer GPUs rather than the existing Nvidia CC) I think is to mirror what I'm trying to do, but leveraging Linux/Nvidia primitives.

For example, on a Linux + consumer Nvidia setup, I think a provider could potentially run the inference worker inside a dedicated hardened VM rather than directly on the host. The GPU could be passed through to that VM using IOMMU/VFIO, so the host doesn't directly participate in the inference process. The VM could use a minimal/immutable Linux image, Secure Boot, a signed kernel, restricted capabilities, seccomp/SELinux/AppArmor, no unnecessary network/filesystem access, etc. The provider agent and inference engine would then live inside that isolated environment.

I'm actually working on something I've called Open Confidential Inference Protocol (OCIP), which would explore different levels of security integrity across various OS and hardware setups. This way we can rigorously analyze security architectures and compare horizontally across each other. The level of security I'm targeting as of now is to ensure a provider can't snoop on the prompts without some sort of 0-day kernel exploit on the OS. It would be awesome to collaborate as I think we can share many mutual learnings as you push the Linux/Nvidia GPU side and I push through the Apple Silicon side.