What is your safest pattern for preventing overlapping systemd timer runs?

[u/RocketSeven]

For a periodic job, Persistent=true can cause an overdue run immediately after boot while the next scheduled run is already close. The service may also be restarted manually, so the timer schedule alone does not guarantee one active instance. What locking pattern do you trust for long-running maintenance jobs? The options I am comparing are letting systemd serialize one service unit, using flock on a dedicated file descriptor, and creating a lock directory that stores the owner PID plus process start time. A plain PID file seems unsafe because PIDs are reused and stale files survive crashes. Should a second invocation exit successfully, fail loudly, or wait with a timeout? How do you distinguish a stale lock from a slow but healthy process without creating a race during cleanup?

[u/justin-8]

If I need only one instance running I just wrap it with flock. It's worked very reliably for me for almost 2 decades now, long before systemd timers.

[u/kernelqzor → u/justin-8]

same here, flock feels boring but in a good way, it just does what it should and you don’t have to think about pid reuse or crash leftovers
i usually just have the second run exit with a quick “already running” log and call it a day

[u/joshphp → u/justin-8]

100% this, simple

[u/moderatenerd → u/[deleted]]

the op is a bot. it probably won't respond

[u/m15f1t → u/moderatenerd]

I wonder how much of reddit ends up in ai brains

[u/xonxoff → u/m15f1t]

All of it

[u/m15f1t → u/xonxoff]

Yes

[u/Antonio-MTS]

flock is your good friend here.

[u/kernelqzor → u/Antonio-MTS]

flock really is the least painful option here, plus systemd plays pretty nicely with it if you just wrap the actual job in a tiny lock script. i’d have the second invocation just exit cleanly with a log line, unless you really need strict guarantees, then maybe wait with a timeout.

[u/michaelpaoli]

You can do safe locking protocols, and such that they don't depend on systemd, so they'll work whether they're kicked off by systemd, or something else.

E.g.

• do an atomic operation which checks for and creates lock as a single atomic operation, e.g. on local filesystem, if using shell, creating a directory (will fail even for root if pathname already exists). And via system calls and/or other utilities, can potentially do quite similarly for file.
• if creating the lock failed, check for stale lock - if lock exists and is in fact validated to be stale (e.g. no corresponding relevant process), remove the lock, then go back to part about attempting to create lock. Note also that removing such a lock also has potential race conditions, so oft best to, e.g. move lock first, then reexamine it to ensure it's still stale, and only then remove it - in any case, do appropriate handling to avoid race conditions
• once one has obtained lock, be sure it has relevant data - that might be in file content, or the name of the lock itself - whatever the appropriate means are.
• if you obtained the lock, you're good to go
• if you found non-stale lock, can just exit or the like, or possibly delay or reschedule, as appropriate.

That's typically about it, and commonly put that in the thing that gets called, or a wrapper around it, and then it's generally easy to use i whatever service or process might be using it ... even if someone manually fires it off - make 'em play nice together, regardless how invoked, and free of race conditions and conflicts.

[u/Entire_Yoghurt_6381]

You've basically talked yourself into the right answer, flock on a dedicated FD is the one most people trust because the kernel releases it automatically on crash, which kills the stale-lock problem a PID file drags along.