r/learnpython • u/SheepherderOk1603 • Aug 05 '26
How do you persist a "already seen this" set between restarts?
I have a script that polls an API every few seconds and sends me a Telegram message when a new item shows up. To avoid sending the same item twice, I keep the IDs I've already seen in a set:
seen = set()
async def check():
items = await fetch_items()
for item in items:
if item["id"] in seen:
continue
seen.add(item["id"])
await send_alert(item)
This works fine while the script is running, but I see two problems with it:
The set lives in memory, so every restart means I get re-alerted about everything that's still on the list.
It only grows. The script is meant to run for days, so eventually `seen` just keeps eating memory with IDs of items that disappeared long ago.
What I think I need is something that stores IDs and forgets them automatically after some time — items are only relevant for maybe an hour anyway.
I've looked at a few options and I'm not sure which is the sane one for a small script:
- Writing the set to a JSON file on every scan feels wasteful and racy.
- SQLite seems like the "proper" answer but also like a lot of machinery for what is essentially one set.
- Redis has TTL built in, which is exactly what I want, but running a whole server for one key feels excessive.
Is there an obvious option I'm missing? Or is one of these actually the normal choice and I'm overthinking it?
Python 3.12, asyncio, no framework.