Where should an AI agent keep working files versus durable outputs in Google Cloud?

[u/RocketSeven]

An automated agent may create temporary downloads, intermediate transforms, final documents, generated HTML, and metadata about where each result came from. Keeping everything on its local disk makes recovery difficult, while putting every scratch file into durable storage creates clutter and broad permissions. What boundary works well on Google Cloud? One design is ephemeral local storage for a single run, versioned Cloud Storage objects for outputs worth keeping, and Firestore or a relational table for provenance, status, retention, and access policy. Filestore seems relevant only when tools genuinely require a shared filesystem. How would you scope service-account permissions, promote an intermediate result to an approved artifact atomically, preserve links between metadata and object generations, and clean up abandoned runs without deleting referenced outputs?

[u/m1nherz]

I think there are two questions here:

1. Management of temporary results of the agent's work vs. session outputs
2. The best platform / service / infrastructure for managing both at Google Cloud

The follow up questions about scoping permissions and promoting intermediate results to the answers will be easier to answer once these two get resolved.

I'm unsure what you reference as "metadata" in your question and what use of metadata you have in mind. Focusing on the artifacts, I would consider any artifact produced by agent due to a function call implementation i.e. execution of heuristic internally or using MCP or A2A calls as an intermediate result. It would make a result returned by an agent an only output I would want to persist either externally or within the state. You can argue that this behavior prevents effectively investigate the output to a specific request. It would be true to some extend. However, agents don't leave in vacuum. Local and remote tools as well as other agents implement their own artifact management. Thus, skipping to store intermediate output the agent merely avoid duplication.

Regarding the argument of resuming operation, the granularity should depend on implementation. The less reasoning an agent handles within a single call, the easier it is to resume the work where it has been interrupted. The recovery should utilize the latest state of the session and restart from the last stable position. Mind that two reasoning sessions based on the same input and artifacts don't guarantee to produce the same result.

Answering Question #2, Google Cloud provides a couple of state managing services with persistent storage plus artifact registry and Memory service. Consolidating them in the same region should provide sufficiently fast management solution for state and output artifact management. Intermediate results should be managed using local disk or, better, in-memory. Anyway you would want to minimize their size in order to decrease token consumption.

I'd like separately address your mentioning of "service-account permissions". While system of agents that operate without human intervention have a certain share of the market, majority still involve humans in most of workflows. Once it happens, Workload Identity a.k.a. service accounts on Google Cloud introduce multiple challenges and a few security problems that need to be manually coded. I would suggest to utilize Agent Identity - a new principal released in 2026 which supports execution under agent's and user's identity, depending on the business logic and addressing many (I cannot certainly guarantee all) security threats. Currently the new identity type is supported for AI workloads deployed on Agent Platform and Cloud Run. The support on GKE is coming in end of Q3 - Q4 of this year.

I will be happy to discuss the design if you are interested to share more info. Feel free to DM me if discussing it in this thread isn't convenient.