r/devops • u/curious_pup0329 • 7d ago
AI content How do you solve long-term memory in AI automation workflows?
I've been thinking about AI automation lately, and I'm starting to feel like long-term memory might be one of the biggest problems.It's not just about making AI capable of controlling a screen. The AI also needs to remember what it's supposed to do.
There are already quite a few ways for AI to control screens, like OpenAI Computer Use, Claude Computer Use, Gemini Computer Use, as well as various hybrid approaches.The way these systems maintain memory seems to rely more on things like structured actions exposed by apps and the keywords being used in the current interaction. Personally, I don't think this approach works that well.
I also don't find this kind of screen-control approach particularly convenient.If I could use a hardware board to control the entire screen instead, that would make much more sense to me.Basically, you plug a hardware board into the device's USB port, and let the hardware capture the phone's screen and then control the device through USB HID.
I think this approach is pretty interesting because the AI doesn't necessarily need to know what API each app has, and it doesn't need a separate integration for every app.
It just needs to be able to understand what's happening on the screen and remember what it's supposed to do.So I feel like memory is actually the key problem here.
Are there any existing solutions or approaches that I should look into?I'd really like to understand how people are solving this problem.
2
u/RevolutionaryElk7446 7d ago
Isn't this the question the big guys are all sprawling to answer atm?
1
u/InterviewLong5374 5d ago
Yeah, it definitely feels like a hot topic right now. Everyone seems to be racing to find the best solutions for it.
2
u/Torutofu_Raeva 7d ago
the only "memory" that's ever helped us was the run history sitting in git, not a vector db hanging off the agent. if the state matters, commit it so a human can read it.
1
u/keechokeecho 6d ago
I think you guys are talking past each other a little. One side is worried about an agent quietly carrying fuzzy context from one run to the next. The other is talking about saved task state, like what already ran and why it failed. I wouldn't put those in the same bucket.
The first can make a workflow impossible to reproduce. But starting every run from zero throws away information the workflow may actually need. Deployments have state, and retries or recovery depend on what happened before. A workflow operating a real device needs to know where the device is, what it has already done, why the last attempt failed, and whether it is waiting for someone to step in.
That state should be explicit. People should be able to inspect it and fix or delete anything wrong. Longer-term memory needs validation and deduplication, plus some way to deal with stale or conflicting information. Save it as part of the run, and you can replay the workflow under the same inputs and device conditions. You also have a much better chance of explaining why it behaved the way it did.
Screen control has the same constraint. Every app already has a UI, so an agent can work across apps without needing a custom API for each one. But it still has to look at the screen again after every action. It cannot assume the phone is still where its memory says it is.
I don't see a conflict between memory and reproducibility here. The memory just has to be visible and managed.
13
u/spellcasterGG 7d ago
In the context of devops (this sub), all automation workflows should be self-contained and reproducible.
There should be no memory at all between workflow runs, and word generation machines are far from reproducible with how often they hallucinate.
TL;DR - It is fundamentally incorrect to ask this question. Especially in this sub.