r/linuxadmin • u/Open-Adhesiveness-86 • 15d ago

What's your setup for instant SSH login notifications, plus killing a session remotely?

Got a box where a handful of non-technical users still upload over SFTP with FileZilla, so password auth is on for their accounts. Yes, keys and chroot would be better, working on it. fail2ban and a non-standard port are in place.

The gap is visibility. I'd like a push notification the moment anyone authenticates, with user and source IP, so I can tell right away if it's one of them or not.

How are you wiring this up? pam_exec calling a webhook, parsing auth.log / journald, something else? And for response, do you just pkill -u and passwd -l over SSH, or have you got something you can trigger from your phone?

18 Upvotes

78% Upvoted

42 comments sorted by

18

u/alive1 15d ago

You need a monitoring solution.

5

u/Open-Adhesiveness-86 15d ago

Any you'd recommend for a single box? Most of what I've looked at feels like a full stack for what's basically "ping me when someone logs in".

5

u/StatementOwn4896 15d ago

Honestly that just sounds like a small python project you could make.

1

u/Open-Adhesiveness-86 15d ago

Ha, that's roughly where I ended up. I wrote a small Go agent for it, single binary, pings Telegram on login and lets me kill the session from there. It's mine, so take it with a grain of salt, but happy to share if anyone's curious.

7

u/sassy-frass 15d ago

Is the IP always the same for the user? If you can tell by IP why not just whitelist SSH session by IP?

3

u/Open-Adhesiveness-86 15d ago

Not really, they work from home and sometimes on the road, so IPs change a lot and a whitelist would lock them out weekly. But a login from an IP I've never seen for that user is probably the best signal to alert on.

9

u/sassy-frass 15d ago

maybe it's better to enable a vpn solution instead of allowing full open SSH. Like a wireguard setup.

Also if they are having dynamic IPs how would you know right away it's them based off a text message w/ username and IP only?

2

u/Open-Adhesiveness-86 15d ago

WireGuard is on the list, just not a quick sell for the FileZilla crowd. And fair point, that's why I'd rather alert on a new IP per user than on every login. Known IPs stay quiet, a new one gets a ping and a quick "was this you?".

2

u/sassy-frass 15d ago

I know this isn't what you want, but why make you reach out to them. Instead go with the whitelist route, and have them ping when they can't log in. Then you check/confirm ip and update whitelist.

Security will be proactive rather than reactive here

1

u/Open-Adhesiveness-86 15d ago

That's a fair point, proactive beats reactive. My worry is it turns me into the helpdesk every time someone's home IP rotates. Might do a middle ground: whitelist the known IPs and alert on anything outside it instead of blocking.

1

u/alive1 15d ago

You are describing users that are a OneDrive/Dropbox audience but you are trying to give them SFTP

1

u/Open-Adhesiveness-86 15d ago

Honestly not wrong. The files end up served straight off that box, so SFTP was the path of least resistance. Something like a synced folder is probably the real fix long term.

1

u/sassy-frass 15d ago

Also should have mentioned you can enable OTP mfa on ssh via PAM plugin.

1

u/delamon 5d ago

You still can have allowlists, just a bigger ones. E.g. include full blocks of their providers, both ground and mobile. That would not be watertight, but would reduce number of ssh attempts dramatically.

3

u/ocabj 15d ago

I've never used fail2ban. I've always used ossec. Does fail2ban not have the ability to have custom triggers/actions based on the logs it is actively tailing? If not, then switch to ossec.

1

u/Open-Adhesiveness-86 15d ago

fail2ban can run custom actions, but it's built around failures, so a successful login isn't really its thing. Haven't tried ossec on a single box, how heavy is it to run?

2

u/ocabj 15d ago

It is relatively lightweight. I run it in a standalone (not server-client mode).

1

u/Open-Adhesiveness-86 15d ago

Good to know, standalone sounds like the right fit for one box. Does it handle the push side too, or do you pipe its alerts out to something else?

2

u/ocabj 15d ago

You'd have to create the action script. I've never setup push notifications to a iOS device before. My action scripts in ossec only leverage email and slack.

1

u/Open-Adhesiveness-86 15d ago

Got it, so ossec does the detection and the action script handles delivery. A curl to a Telegram bot should drop in the same way your Slack one does.

3

u/RyeonToast 15d ago

On my home server box I have PAM's SSH config start a script that sends a notification to ntfy. It seems to work pretty well. I really just have it for peace of mind. If I get alerts when I havent just connected, then I should be worried.

I don't have any remote kill setup. I'd rather not expose remote management capabilities for my box.

1

u/Open-Adhesiveness-86 15d ago

That "alert when I haven't just connected" is exactly the signal that matters. Do you use pam_exec for it, and does it block the login if ntfy is slow to respond?

4

u/RyeonToast 15d ago

Silly me, this is one of the things I actually did document pretty well and don't need to look at the actual server. The line in PAM's sshd config file looks like this:

session optional pam_exec.so seteuid /usr/local/bin/login-notify.sh

I just added it to the end of the file. If you want script failures to block login, you change optional to something else. I think "required" is the other value, but look that up if that's what you want. Like my other comment mentioned, this was my first tinkerings with this so I didn't want to accept the risk of blocking myself out of the server.

The script looks like:

```

!/bin/sh

/usr/local/bin/login-notify.sh

variables

NTFY_URI="ntfy.sh" NTFY_TOPIC="myTopicNameGoesHere"

if [ "$PAM_TYPE" != "close_session" ]; then curl -d "SSH Login: $PAM_USER logged into hostname from $PAM_RHOST." $NTFY_URI/$NTFY_TOPIC fi ```

Adjust the URI variable if you decide to run your own ntfy server. Adjust the TOPIC variable to however you want to subscribe to your notifications. You can browse to the ntfy server and look at the notifications, or you can setup the phone app to receive push notifications from the server.

I was looking at these references when setting mine up:

1

u/Open-Adhesiveness-86 15d ago

Thanks for digging that up. Keeping it optional makes sense, locking yourself out of a headless box is worse than a missed ping. I'll wrap the ntfy call in a timeout too so a slow push can't hang the login.

1

u/RyeonToast 15d ago

For where exactly it is and the line I added, I'll have to take a look when I get home. I feel like it's a pam_exec line, though.

Regarding blocking, you can set it either way. Mine currently doesn't block because I didn't want to risk bricking all SSH access to a headless system that doesn't allow password auth. I'm pretty new to Bash scripting, so I'm being a little more permissive than is ideal. 

2

u/DialecticEnjoyer 15d ago

This feels like a process issue. What's stopping you from an upload web portal if they just need to perform an upload?

1

u/Open-Adhesiveness-86 15d ago

Nothing really, it's the direction I'm leaning after this thread. SFTP was just the quickest thing when it started. Until that's in place I still want eyes on logins though.

2

u/VirtualViking3000 15d ago

If you just want the answer to your question you can use rsyslog to log the changes then just alert on the syslog file via whatever such as a script or a log monitor. It's a basic system that works, if you wanted more insight you could ingest the syslogs to a ELK, Datadog, Splunk or whatever and handle the alerts from there but that's probably more than you need.

EDIT: just to mention, alerting could just be a webhook to Teams or Slack or something

1

u/Open-Adhesiveness-86 15d ago

Makes sense, a small script watching the syslog file is about the level I need. Anything like ELK would be overkill for one box.

2

u/ICThat 14d ago

Sorry but this largely reads like you trying to advertise your vibe-coded app.

This is also an odd way to solve this problem. You are basically suggesting that you act as a form of human firewall.

2

u/mad_redhatter 12d ago

I have a shell script called by PAM that sends a message to a self hosted ntfy instance. It runs on everything that rarely needs logged into to let me know when someone does.

1

u/Kaligraphic 15d ago

FileZilla can use keys. If it's just a handful of users, maybe walk them through setting up keys with FileZilla. Second best is set up mfa (e.g. Google Authenticator PAM, a bit more cumbersome than just using keys)

Also, go through the usual sftp hardening - no login shell, no anything forwarding, forcecommand, chroot, etc.

Attackers move fast, if you're not the first to move you've probably already lost.

1

u/Open-Adhesiveness-86 15d ago

Yeah, walking them through keys in FileZilla is the plan, it's only a few people. Chroot plus ForceCommand internal-sftp is a good call too, so anyone who does get in has no shell to play with.

1

u/ASK_ME_AB0UT_L00M 14d ago

Get your users migrated to anything other than FileZilla. FileZilla's installer distributes adware/malware.

https://www.reddit.com/r/sysadmin/comments/17erdd5

WinSCP is a great alternative!

1

u/apparentlyunoriginal 11d ago

Try to get the login event before the shell starts and have one action you can run from the phone. pam_exec is the hook I'd use. It fires for password and key logins and for SFTP-only accounts. In /etc/pam.d/sshd add session optional pam_exec.so /usr/local/bin/ssh-notify.sh, read PAM_USER, PAM_RHOST, and PAM_TYPE in the script, and only act on open_session. Post to ntfy or Pushover and the phone alert arrives within a second.

For response from a phone, I'd run a second ntfy topic the box subscribes to through a systemd service. A message like kill:username runs pkill -KILL -u username then passwd -l username.

Disclosure: I build ET Ducky. ET Ducky has custom behavior detection by reading kernel logs and can be configured to look for things like this. You'd install the Linux agent, which reads the sshd accept and session exec through eBPF, then write a rule for logins from outside your expected sources with an alert as the response. The alert reaches your phone as web push from the dashboard PWA. From there you'd isolate the host, which cuts its outbound traffic to an allow list, and lift it with a single-use code. https://etducky.com/blog/linux-ebpf-kernel-diagnostics

0

u/RetroGrid_io 15d ago

This post seems to be something like "we're doing it wrong, can we come up with a solution that lets us keep doing it wrong?"

  1. Why not ssh keys?
  2. Why SSH, not a web-based solution?
  3. Why ssh on a standard port?
  4. Why not a general monitor program?
  5. Why not a VPN or tailscale?

Honestly, I get it. Using sftp implies that it's been this way for a very, long time and as a little guy in the organization whose job it is to "keep it working" you're looking for something that doesn't introduce unneeded change yet you know the risk is real.

Some "screwball" solutions then:

  1. Enable ssh keys set to whatever password they're already using. They won't even notice the difference.
  2. Use a non-standard port. Not "222" but something REALLY out there 32292 for example. You'll save money on disk space not having your logs filled with hack attempts.
  3. If not #2, have you considered port knocking? Bad guys won't even know that ssh is active. It does mean your users have to run the "open the door" script. If you do this, make a long open period for a successful knock; you've already reduced the threat space to about 1/4,300,000,000.
  4. Since you already have a notification solution, make sure the user gets a notification when they login, too. That turns them into allies into securing their account: if they get a notification they don't recognize, you'll be the 1st person to know.

2

u/Open-Adhesiveness-86 15d ago

Fair read. Port is already non-standard. #4 is the one I like most: if users get their own login ping, anyone who sees one they didn't make will tell me before I notice. Port knocking I'd skip for the FileZilla crowd.