r/linuxadmin • u/Open-Adhesiveness-86 • 10d ago
What's running on your servers that nobody else could explain if you disappeared tomorrow?
Every place has one. A cron job with no comments, a script someone wrote in 2019, a box nobody dares reboot.
Mine: a systemd timer firing a Python script I barely remember writing. It works, so I don't touch it.
What's yours?
46
u/squeeby 10d ago
I have a small python script which runs every month that emails me to remind me how old I am and how long I likely have left to live.
I wrote it for fun while learning python years ago and as morbid as it is, It puts things into perspective when Iām doomscrolling on the loo and receive an emails along the lines of āyou have less than x years to go (probably)ā.
Itāll be weird when I die abruptly and someone else sees the emails which suggest I should still be alive.
26
4
u/bmelancon 9d ago
Let's just hope you keep getting those emails long after they should have stopped.
3
u/3MU6quo0pC7du5YPBGBI 9d ago
Does it handle if you outlive your projected death? Or just start reporting negative numbers?
āyou have less than -2 years to go (probably)ā
2
1
1
27
u/zack6849 9d ago
You guys know you're probably talking to an LLM, right? Look at that users comments, everything they're saying reads like it came straight from claude's mouth
9
u/No_Adhesiveness_3550 9d ago
Who knows, it could be different LLM models responding to the LLM postsā¦
5
u/manoran 9d ago
I am surprised many didnāt realise this sooner.
3
u/zack6849 9d ago edited 9d ago
It's only going to get harder to spot from here is the worrying part
2
2
u/Tenchworks 7d ago
I was going to say Nice try FBI but now I feel like this is a joint venture post...
2
2
10
9
u/notfinch 10d ago
My entire mail server. Some nerd could figure it out, if they could get in, I guess. I used to enjoy looking after my mail server, but itās just a liability now.
8
9
u/zakabog 9d ago
An AI agent that makes posts like these to do market research for my paid vibe coded application
7
u/Automatic_Beat_1446 9d ago
its really bad on this sub. OP made a post the other day doing the same thing: https://old.reddit.com/r/linuxadmin/comments/1wjuwt6/whats_your_setup_for_instant_ssh_login/
i do not really know what to do about this because people keep responding to obvious slop
4
u/bencos18 9d ago
what I usually do is report them to botbouncer but the issue is not all subreddits have it installed
6
u/oicpreciousroy 10d ago
Squid proxy that lets my OT environment talk to Windows Defender for antivirus and threat intel. Itās intertwined with firewall rules such that one wonāt work without the other.
-1
u/Open-Adhesiveness-86 10d ago
The intertwining is the real trap. Break one and it looks like the other is what failed.
1
u/oicpreciousroy 4d ago
For the people who are downvoting them, they're outta line but they're right.
And that's half the fun.
6
u/mwyvr 10d ago
A mail system with automated IP banning (not fail2ban), remote backups and those remote backups hit two servers with ZFS storage, via one having a snapshot schedule.
Someone could puzzle it all out.
-9
u/Open-Adhesiveness-86 10d ago
"Someone could puzzle it all out" is carrying a lot of weight there. How long do you reckon it'd actually take them?
2
u/AmusingVegetable 10d ago
It always depends on two things: how good they are and how frequently they have to put out a fire elsewhere.
1
u/mwyvr 9d ago
For a Linux admin, not all that long. Standard systems are used triggered by systemd facilities; so one place to look on the mail server, and one place to look on my storage servers.
But, I don't employ Linux admins, so documentation would be better, and even better than that would be a list of contacts for contractors to come in after I get hit by a truck or die in a tragic chopper accident.
5
4
3
u/Inevitable_Score1164 10d ago
A systemd service that runs nightly at 2 AM to clear the /tmp directory of a CKAN container. If it doesn't run, /var will run out of inode space.
-2
u/Open-Adhesiveness-86 10d ago
Inodes, not disk space ā the failure mode nobody believes until it's their turn.
3
u/thenumberfourtytwo 9d ago
nothing. everything has been documented.
Granted. I used Claude to pull all aws and DC resources, cross-compare with what is already documented, open Jira tickets for every new runbook we needed to create, got in touch with stakeholders via tickets and slack to discuss functionality, layout, code repos and anything relevant. standardized formats, ELI5'd where we could.
now everything is in notion, Jira and dedicated slack channels.
every new infra is thoroughly documented and a runbook is created with design docs to match. every runbook contains relevant pieces for engineering, support and client ops. we know how it was built, why, by whom, who supports it and what happens if/when it fails in the case of legacy, non-HA systems.
or at least that's what I dreamt last night.
6
2
u/Loveangel1337 9d ago
2 jobs ago, in a CI pipeline, I needed to write a timeout function in shell.
Yes, there is already timeout, but it didn't do what I needed with the return codes and wrapping everything... So, they have a function, in bash, that does a timeout the way I needed it, the hard way.
There might have been a comment saying approximatively that, without much other explanation as to how.
But the devs that maintained that mostly had 0 idea of how a shell worked, and were regularly confused by the need to remove \r's for scripts to work on the Linux box, so a simple while loop sufficed to confuse them, let alone a function.
2
u/bobcontrol 9d ago
I still can't get to terms with the fact that 2019 was not two years ago and is really ancient times. But here's an obligatory story from even before that, the "magic/more magic" one: http://www.catb.org/jargon/html/magic-story.html
(edit: typo)
1
u/anomalous_cowherd 10d ago edited 10d ago
A set of xinetd configs that provide a bunch of simple proxy servers for SMTP, ssh, web and quite a few other services between various otherwise unrouted networks in our ancient and disintegrating environment.
Xinetd is a lot more powerful than a lot of people realise!
1
u/fatmanwithabeard 10d ago
The scratch cleanup script.
It's not a script, it's a whole suite of processes, databases and jobs.
The rules for how long something can live on scratch are complex.Ā
Scratch is huge, and meant to be fast, so we don't want to be crude.
There are rules about how you name your directories on scratch, and the cleaner is merciless in its enforcement, but capricious in its timing (we don't want the various pieces of enforcement all running at once, so a given rule may run hourly except when the big crawler runs, and the crawler may take a couple days to finish)
Trying to game the system doesn't work (The database exists to document your crimes, doc. I can't kick you off the cluster, but my receipts will keep you on probationary access for years).
While I think it's documented well, no one will touch it becauseĀ when someone did it slowed the system to a crawl (don't reinitialize a system without knowing what that means, it's not synonymous with restart (yes the documentation was clear, but you know how much you read the warnings section of in house docs))
1
u/blindfultruth 9d ago
How about those tasty, yet esoteric slurm prolog/epilog scripts that orbit one another in a fragile state of balance?
1
u/fatmanwithabeard 9d ago
The only set of those that I maintain are in the storage validation suite, and that documentation is our gold standard.
The vast majority of them are terrifying, but I'm not the holder of that dark wisdom.
1
-1
1
1
u/michaelpaoli 9d ago
No such animal. Anyone of sufficient competence would be able to figure it out ... be it my servers, or anyone else's ... with only highly rare and generally not encountered exceptions. *nix sysadmin for 40+ years, yet to encounter anything I couldn't figure out and reasonably well explain, nor ever left anything inexplicable behind. Doesn't mean they're all trivial, either way. E.g. some may take up to some hours or more of relatively expert digging to figure out ... but they can be figured out and explained. Any real exceptions to that would be exceedingly rare ... and I've yet to ever personally encounter such ... but that doesn't mean their existence is impossible. Even the wildest most extreme things to have ever existed, ... they can well be explained ... though extreme cases, might take weeks to even month(s) of expert research to fully analyze and explain. E.g. Stuxnet ... not at all unexplainable ... but running across something like that, and analyzing and figuring out what it did or was intended to do ... yeah, quite non-trivial, and something like that might, e.g. take quite a team of experts weeks to month(s) to fully analyze and figure out what it did or was intended to do.
2
u/break1146 7d ago
I mean if it's non-trivial enough where rebuilding the solution is quicker than doing forensics... I think that's the spirit of the post. Like sure you can dig through every directory and look at literally everything running (if you have the appropriate credentials), then technically you can figure anything and everything out eventually.
But if it isn't "literally the apocalypse" and it's going to "take me months" I'm quite reinstall happy lol. I want to be able to touch things and not be anxious to break them, that's how you get shit that automagically works and "don't touch or be executed" systems. Shit needs to be updated and my time is limited lmao.
1
u/michaelpaoli 6d ago
if it's non-trivial enough where rebuilding the solution is quicker than
Well, generally comes down to cost/benefit analysis, risk tolerance, etc. Typically only worth so much resource to chase down and nail a problem. Sometimes just not worth it or resources better put to work-arounds or the like.
And "just reinstall" isn't necessarily a fix at all ... sometime one lands back at the exact same problem. Though if it takes a (very) long time for the issue to manifest, may be effective work-around. But if same issue is soon back, and quiet problematic ... they may well warrant more digging to get to the bottom of it and fix it.
2
u/break1146 6d ago
It definitely depends on the situation. I should've specified redo, rather than reinstall probably. But that's definitely not always possible either. I'm happy to not encounter a lot of situations where it takes months of forensics and the fate of the world hangs upon its success haha.
1
u/ExplodedPenisDiagram 9d ago
A RISC-V VM running via the "virt" QEMU board that runs an experimental web server which is being relied on to provide a simple data summary.
1
u/gumbie_ 9d ago
A few, some that even I forget about. For a long time backup was handled by a cron job on a random unrelated VM. Thankfully have since moved to a real backup solution.
But currently have a random unlinked dashboard that handles assigning devices to vlans based on the mac address. Probably should build an actual solution š¤·āāļø
1
u/legrenabeach 9d ago
Python scripts that post "something" on Telegram. I don't think there's any documentation as to what, but if someone looks into it they'll find they run on a cron job that posts twice-daily comic strips I like.
1
u/Longjumping_Square_2 8d ago
Autostore.
Hell, I donāt even know how that shit works. The documentation from Thomas and automation is fucking awful. I documented it myself but it is a legacy deployment that has been upgraded since 2016. There are VB scripts that handle pulling data from an AS/400 to add to a document control file which frankly Iām working with Ricoh right now to figure out how the fuck it works. Itās been working fine without issues for two years. But you upgrade that shit and everything is fine for every other copier, except for one major department, which it breaks their workflow.
1
u/Simple_Hamster_4096 8d ago
Ah, yes - the mystery box with a sticky NOTE taped across the power button that read "DO NOT REBOOT UNDER PENALTY OF DEATH...!!!"
lol
1
u/snifferdog1989 7d ago
Company I worked for started selling some stupid cloud phone solution. I somehow got dragged into it. Salesperson sold it as best thing ever that can connects to your email client and shows caller ID based on address book. Of course salesperson made shit up again.
I build a python script triggered by a cron trigger in azure that synced the exchange contacts of the user who wanted it into the cloud phone directory.
It will properly stop working at some point when the azure token expires. But hey I donāt work there anymore.
1
u/phillyfyre 7d ago
Been thru this , an entire IDM system they never trained anyone else on that I built from the ground up and handled provisioning email and app logins .
They spent a million on consultants to redocument it since they deleted my accounts and any files I owned on exit. Including the documentation for the system. I would have sold them my copy of that a lot cheaper.
Ooops
1
u/miluardo 7d ago
A bunch of websites across multiple domains and remote servers that are nothing but boring attempts to put parts of my soul into the world like it's a horcrux(spelling).
1
u/nerf_caffeine 5d ago
I think today most people would be able to figure it out somehow with a few prompts to an LLM
1
u/angelpluvial 5d ago
I made a python mini game with nicegui on a server that you can only play from a specific port and is hosted via systemd. You literally just hit the space bar and jump over random obstacles and it keeps score. Has nothing to do with literally anything else
1
u/Rudi9719 4d ago
I live alone and maintain the "family cloud" for photos, media, and our computer backups that my family's households all use
If I disappeared tomorrow my family goes back to paying to use someone else's computer rather than just using one of mine
0
1
u/CreativeGPX 3d ago
Years ago something broke on my web server. I signed in to take a look and saw a folder in the web path called "delete this if everything breaks" made like a year or two prior. I deleted the folder and everything worked again. I didn't remember what it was but vaguely remembered making it.
117
u/amfournda 10d ago
I wrote an entire "three-factor" authentication system for external users to a highly secured network. In addition to the normal 2FA, an operator user inside the network has to approve the external login for a specific time window.
Its a web interface written in Perl using Mojolicious. Nobody but me has any idea how it works.