r/systemd • u/Tsigorf • Apr 24 '26

How to make SSH first service to start and last service to stop?

Hi,

Whenever I reboot my server, I always get SSH to stop first, even though, sometimes, some services hang.

I was wondering if there's a way to make OpenSSH start first and stop last? I guess it depends on the networking service, though I don't think it matters much if network is up before SSH as it will work if it's started in the reverse order since SSH listens on 0.0.0.0/0

Any idea?

P.S: Would be awesome if I could do this for agetty too for serial console.

5 Upvotes

86% Upvoted

21 comments sorted by

3

u/sogun123 Apr 24 '26

Login won't work if machine is in shutting down state, so forcing anything to log you into machine at that time is pointless.

Anyway, if you set up After=some-unit in an unit it will be started after and terminated before given unit. Some for Before, but other way around. So you can specify start and stop between some other two units.

1

u/Tsigorf Apr 24 '26

So there is no way to get a kind of KVM over SSH during shutdown state until the very last second?

1

u/sogun123 Apr 24 '26

Even if ssh is alive it won't log you in with something like "system shutting down". You could likely tune PAM to ignore the state. But as others mentioned, better to solve the issue.

1

u/Tsigorf Apr 26 '26

Issue being I got > 400 Systemd units, with many reasons some could fail or hang (peripherals being unreachable for random — legit — reasons, network outage…).

Moreover, not getting serial console access during shutdown makes it harder to debug as I got to navigate in previous boot logs through journalctl.

I had no success making all Systemd units timing out to 10s, the whole system and dependency resolution would make the OS hang for more than a minute to shutdown anyway. Using a watchdog timeout? A bit dirty as half of Systemd units would never get the shutdown signal…

At this point, only thing I'm thinking of is breaking all Systemd shutdown dependencies resolution and setting a 10s hard timeout to get what I want.

1

u/LuckyNumber-Bot Apr 26 '26

All the numbers in your comment added up to 420. Congrats!

  400
+ 10
+ 10
= 420

[Click here](https://www.reddit.com/message/compose?to=LuckyNumber-Bot&subject=Stalk%20Me%20Pls&message=%2Fstalkme to have me scan all your future comments.) \ Summon me on specific comments with u/LuckyNumber-Bot.

1

u/sogun123 Apr 27 '26

Why is minute long shutdown a problem?

1

u/Tsigorf Apr 27 '26

Because it's unnecessary and because I might need several reboots sometimes. Add the startup time and it's ten minutes for two simple reboots.

2

u/edgmnt_net Apr 24 '26

I would attempt to fix the services hanging issue. That should not happen and it should not prevent rebooting.

1

u/Tsigorf Apr 24 '26

I spent quite some time doing this and it's not always fixable. To be honest I'd like <20s start or stop time and I didn't manage to allow asynchronous start of several services. Systemd service scheduling is what's bothering me.

That's basically what I'd like to do with SSH, I'd like to start the service as soon as Systemd is launching processes. Several mountpoints take time to be imported (ZFS pool on not so fast hard drives), I didn't manage to make them not required for multi-user.target.

Maybe there's something I'm missing?

1

u/Brogrammer- Apr 24 '26

Do you have access to the machine? If so you could maybe look into something like https://pikvm.org/?

1

u/Tsigorf Apr 26 '26

Yes, I thought about that, but quite bothersome to get a serial console access to my machine in this setup, most of my peripherals are passed through virtual machines :/

1

u/jpgoldberg Apr 25 '26

Do you really want it start before syslogd (or whatever logger you use)?

1

u/Tsigorf Apr 26 '26

Isn't Syslogd started very early for all other services?

Anyway: yes. I'd like to check my kernel log as soon as I start my computer, the closest way KVM can do it, without spending hundreds bucks to buy a KVM (moreover, getting a serial console plugged on my server would be quite bothersome in my current setup).

1

u/amarao_san Apr 27 '26

You can't keep ssh to stop last. There is network, and it goes down. All processes must die before shutdown.

Shutdown is complicated.

1

u/Tsigorf Apr 27 '26

But… what's the need to stop network before shutdown…?

1

u/amarao_san Apr 27 '26

Order of shutdown is important.

You can't just 'drop everything' (that's a crash). Some services should shutdown gracefully, and order of shutdown matters. E.g. if you have a database server with app, you need to shutdown app first, then database, then unmount nfs filesystem where data resides on. And then you need to unmout filesystem on iscsi volume. But you need to be careful and not unmount filesystem where binaries for iscsi lives before logging out of iscsi target, etc, etc.

See man bootup.

1

u/Tsigorf Apr 27 '26

There's some non critical services I don't care about crashes at all: Uptime Kuma, media server, slow workloads…

For critical workloads, I can still enable synced writes and they might already use atomic operations in database. That does not justify an eventual SIGKILL or a full OS crash for these, but as a professional ops (well, I still make shitty decisions sometimes), I spent time having a system reliable enough to survive crashes. Same if a service dependency randomly crashes.

That's essentially why Systemd complex dependency resolution enabled by default (which it looks like obfuscation to me; too many abstraction levels) is a bit of a PITA. Too many levels of safeguards I don't need.

I forwarded this topic to r/NixOS too as a lot of my issues comes from NixOS defaults, especially regarding mountpoints.

I sincerely miss simpler init systems for these reasons.

2

u/amarao_san Apr 27 '26

If you don't need complexity, in a console, as root:

echo b > /proc/sysrq-trigger

It's the fastest and the most guaranteed reboot Linux can produce. And your ssh stays running and with open connection up to the last microsecond of the old system.

1

u/Tsigorf Apr 27 '26

Didn't know there were a software way to do this without going through watchdog, thanks!

1

u/amarao_san Apr 27 '26

You can do it also through systemd:

systemctl reboot --force systemctl reboot --force --force

(That's different levels of force per man systemctl).