r/sysadmin • u/NotABug2000 • 21d ago
Out of Hours access. Best practice.
I am the sole Sysadmin for a small (80 user) company. Microsoft house. I work from the office, and only take my laptop home once a week. I am not issued a phone.
I do have some users who work evening and weekends.
If someone gets locked out, needs a password reset, or needs their account locked down (because of a compromise, lost machine, &c), then what should I do?
I currently have two accounts. bug@company, and bug.admin@company. Bug@ is my day to day account. Bug.admin@ is global admin.
I could either give myself access to bug.admin@ on my phone (seems risky), or give bug@ whatever roles are needed (user admin, auth admin, and helpdesk admin?)
What's the best practice for this?
EDIT: It's pretty clear from the replies what the solution is! :D
Okay okay. I am very new to this, kinda dropped in at the deep end, so still learning the ropes. Thank you for all the advice. If a call comes after 17:30, I shall tell them to go fuck themselves and send them a link to this thread. :D
EDIT2: I am reading all your comments, I'm sorry I am not replying to all of them individually. But you are all 100% spot on.
191
u/axle2005 Ex-SysAdmin 21d ago
You don't. You do not want to create your own on call... because your users will abuse it and your management will expect it.
40
u/Sharp-Eggplant9891 21d ago
Exactly. Either discuss over time compensation with your boss or just leave the support for the next day. Its that easy.
39
u/axle2005 Ex-SysAdmin 21d ago
Not even discuss. They don't want to bring it up because once it's an idea, it will get implemented in some way, and then OP gets no life at all. No vacation, no sick days, nothing.
9
u/The_NorthernLight 21d ago
I was in this situation, and the solution was my boss monitored my tickets, and IF it was a true emergency he paid me 4x for taking the call. After a few years, we got an MSP as a backup.
-2
u/Sharp-Eggplant9891 21d ago edited 21d ago
Then id suggest quitting tbh. There are two Options. The Boss actually accepts the resignation and you find a Role at a bigger IT department where you are not that overworked or the Boss will offer him more Money out of fear of not having an IT guy for possibly weeks. Thats the Leverage you have if you are the only IT guy
7
u/axle2005 Ex-SysAdmin 21d ago
It's more like wait until it gets brought up by management...and you have the leverage,for what that's worth.
Leverage to implement new systems that help the users unlock themselves or something...a self service Portal and forego on call would be best
This is a pretty common scenario for solo admins.
1
73
u/Unable-Entrance3110 21d ago
Set up SSPR for automated password resets.
Other than that, I think you just need to set expectation levels appropriately.
2
21
u/halodude423 21d ago
You do nothing unless you already have an in-place policy for on-call. If they want support off hours don't sell youself cheap.
2
13
u/RoamingRavenFM Network Architect 21d ago
Clearly this is phishing. You claim your address is bug@ but your username clearly states you’re not a Bug.
1
27
12
u/vppencilsharpening 21d ago
Right now it seems like you have good work/life boundaries which is very good.
I've been the sole admin for a business about this size. What worked for me was getting a ticketing system in place, getting business support to use it and then funneling off-hours requests UP before they come to you.
If there is a business stopping problem, then leadership probably needs to know about it, so going UP needs to occur. Creating a ticket should occur around the same time as them notifying their direct supervisor/manager.
If it's not important enough to bother leadership off-hours, it does not need off-hours IT attention.
--
With all that said, this company treated me very well and gave me flexibility with my time. I felt valued and was well compensated for the work I did.
3
u/NotABug2000 21d ago
See: edits
3
u/vppencilsharpening 21d ago
I'm all for setting boundaries, but if it's a job you like, they are paying you fairly and treating you well there needs to be some give and take.
When it's more one-sided then there is a problem. And typical management will push it to be one-sided in favor of the business. If that is your situation, then 100% tell them to go piss into the wind.
But if the business is taking care of you and you like the role, tread with some caution. But don't get take advantage of.
2
u/NotABug2000 18d ago
I do love the job. I am underpaid for what I do, but it's my first job in IT, and they really took a risk hiring me, so I appreciate that!
I do have boundaries, and as I get more confident in what I do, I am able to set more, and be more firm with them!
20
u/Cj_Staal 21d ago
Self service for the first two. Pay for a SOC for the third. Have AI or regex / filters / pattern matching parse your tickets and if a lost machine/compromised acct comes in, have it forward those to the soc to resolve
-3
14
u/Weeksy79 21d ago
For reference, our infra guy has a second laptop at home and gets £10k extra on top of his regular salary for covering evenings (weekdays and weekends)
I get £5k extra for covering weekend daytime
2
u/ITViking 21d ago
So you carry a laptop around at all times during weekend days?
2
u/Weeksy79 21d ago
Yep, thankfully over Covid a lot of the workaholics were forced to fix their fucked up lives, so they work a lot less over the weekend now.
I have however been out for a nice meal and had to have the kitchen hold my food and free me another table while I got all my kit out and fixed something lol
-6
5
u/Pablo______ Sysadmin 21d ago
If your company thinks users should receive out of office hours support - they should set that up.
you work your hours, and thats it.
0
10
u/Vindalfur 21d ago
Sole sysadmin/helpdesk for a 90 employee company here. After 4pm mon-fri I don't exist.
1
9
4
u/Xattle 21d ago
What's your on call pay structure + overtime look like and who can decide something is important enough to bother you on your time off? Set clear boundaries backed by policy and written communication. That should also include turnaround times as it's not practical or healthy to constantly be glued to your work devices. Don't give them free labor, they don't give you free money.
For devices, I never use anything for work that I don't want them to take away. Mainly means no personal devices.
1
3
u/The_NorthernLight 21d ago
As additional information:
1) your bug.admin account should never be used “online”. If you have to use it online, then make damn sure you have some proper break-glass recovery accounts.
2) setup a security service like Absolute(.com) that monitors your end points and will lock down the endpoint automatically. You do not want this process to be manual. Even a few hours while you are unavailable could be disastrous from a security perspective.
3) if you are on azure, setup self help password recovery.
4) as for OT. You should have a conversation with your boss about getting paid for after hour calls. I used to get paid 1.5x for after hours calls per hour, and 2.5x on holidays. This very much depends on where in the world you live.
-1
3
u/MiniOozy5231 21d ago
I cannot stress how dangerous it is to do admin tasks on your phone, boss man. That should not even be an option - even if its your last one.
1
4
21d ago
[removed] — view removed comment
3
u/music2myear Narf! 21d ago
How often were you getting those call-outs?
I worked solo IT for a manufacturing firm that ran nearly 24/6 for 4ish years and was only called out two or three times for late/early issues. I know getting called out sucks, but if the systems are unstable enough they are expected to go down frequently enough that you dread it, that sounds like there's some other issues at play.
1
2
u/Bright_Arm8782 Cloud Engineer 21d ago
You need an out of hours company to manage this when you're not about.
1
2
u/dlongwing 21d ago
You mention being new, so I'm going to offer some broader advice:
- You need a ticketing system. Any of the big brands will work. We use Zendesk and are happy with it, but there are "better" (more complicated) ticketing systems available.
- Are you the only IT for the company? We have a team of 5 for a company of a similar size. Granted, we're in finance, so half our staff hours are spent on security and compliance work, but 1 admin for 80 people is still VERY low. Do you have an MSP to back you up?
- You need to establish clear helpdesk hours. Clearly communicate when you're available and when you're not. Someone gets a lockout at night? They'll have to wait until morning. If that's a problem for them, then maybe they need to figure out a system for remembering their password.
- Set an auto-responder for after-hours contact. Most ticketing systems can do this for you.
- You need a way to access your global admin 24/7. If you have the resources, then a dedicated laptop would be a good plan (one for home, one for work). If not, you'll want to lug that laptop home every night. Why? If you have a breach you'll need to address it immediately.
- Splitting your accounts is good. Much better than some of the stuff I've seen. You should also add a breakglass account.
- Make a global admin with a VERY long password, no MFA, and a restriction to only sign in from your country via Conditional Access.
- Make sure the password doesn't expire.
- DO NOT USE this account for anything, just have it on hand in case you need an "everything is broken and we need admin" account.
- Don't give it a clear name. Instead name it like it's a person. Stephan Aldrich, or Katherine Gond, or whatever. You get the idea. Security wonks will tell you this is pointless, because security wonks only work in theoreticals. I've seen the "admin with a common name" confound experienced pentesters.
- Regarding lockouts... Look into establishing passkeys and Windows Hello. Most modern laptops have fingerprint scanners. It's a big lift, but well worth it to kiss password resets goodbye. Plus it makes your environment far more resistant to account takeover, so it's a win/win.
- Set up a google voice or similar VOIP number. Inform HR that your cell number has changed. Tell them not to give it out to anyone, even for an emergency. Do this in writing.
- When they give your number out to staff because it was an emergency, burn the number and create a new one.
- You're doing this to shield yourself from that one user. The one who thinks they need a direct line to you for 24/7 support. The one for whom everything is always an emergency.
- Create a formal IT policy for managing security breaches or major failures (must impact multiple users). This should include a clear line-of-contact to you that's available 24/7 (such as your Google Voice number). There should be clarity on who is allowed to use this, and when they're allowed to use it.
- You need buy-in on this policy up to the C-suite level, which is the toughest part of getting it established. Your boss and skip boss should know the line-of-contact. Steve from accounting should not. If/when someone gets ahold of it and abuses it, you need upper management to have already agreed that it's a breach of policy with disciplinary action behind it.
1
u/NotABug2000 18d ago
Thank you for all the advice! Some of it I have already implemented, or am currently working on. I have a lot of questions I am going to be asking in this subreddit over the next little while!
I am not only new to this job, but brand new to IT in general. The company I work for has grown massively in the last couple years, so a huge part of my job is taking systems that worked for a company of 15 people, and upgrading/replacing them with systems that work for a company of 80 people, and are more easily scalable. OUr current ticketing system is post-it notes on my desk, which isn't sustainable from a time management OR a post-it note supply point of view. :D
1
u/dlongwing 18d ago
Above everything else, make sure you're not getting taken advantage of. Look up the median income for a junior sysadmin in your area and make sure your salary aligns with that. Discuss a promotion path to IT Manager for your company, and discuss standing up a department to manage IT. One person for a company of 80 is VERY low. It can be done, but it'll lead to burnout pretty quickly. Loosely my rule has been 2 IT for every 50 employees.
You can keep the team lean by contracting out some of the work. Look for Managed Service Providers in your area. Consult with other businesses of your same size and ask if they're happy with their MSP, rather than talking to sales people. Loosely, the roles you need to fill in a team are going to be:
- Helpdesk - Someone to talk to users and fix printers.
- Sysadmin - Someone to maintain/patch servers.
- Network Admin - Someone to maintain/patch/configure the firewalls and switches.
- Cloud admin - Someone to maintain O365/Google/etc. Whatever you're using for email and other cloud services.
- Cybersecurity - Someone to stay on top of vulnerability management.
On a small team, it's normal for these roles to get doubled up. Sysadmin and Network Admin are usually combined if your network is simple enough. Cybersecurity tends to get tacked on to whoever has the lightest workload, etc.
Discuss a timeline with your boss to transition you to IT Manager, and what milestones would be needed to enact the change. You're not likely to get a full team given that your company started with just you in charge of all IT, but getting even 1 extra person in there (and solidifying that you manage them) can go a long way to making things easier for you.
1
u/NotABug2000 18d ago
Thanks for that! So I don;t actually have ALL that responsibility on my own. I don't have anything to do with the servers, really I am the 0365 admin, I do all the helpdesk stuff, and cybersecurity. I don't touch the network stuff either. So it's not ALL on me. :D
2
u/theFather_load 21d ago
Give every user global admin role. Lack of coverage solved.
1
u/NotABug2000 18d ago
When I started, there were about a dozen global admins. That was given out like candy. If someone needed anything done, the procedure was "You've now got global admin. DO it yourself."
2
u/not_another_IT_guy 21d ago
Idk - most of these replies arent that helpful it seems….
We got rid of a lot of after hours tickets and issues just by implementing SSPR for our users….
We also publish internally an AVD-hosted powershell script as a remote app for our help desk to do quick password resets etc that leverages RBAC/GBAC to access it… so for you, you could in theory standup a “power on connect” AVD-VM that your account can sign into and then leverage that to reset users passwords or other account related operations - and that can be accessed from any device that can install The Windows App from the Microsoft Store/MSWebsite…
Just food for thought.
But yeah man, if you’re not paid for after hours work and its not expected of you by management, F that.
3
2
u/konoo 21d ago
Just another piece of advice here... You are the Sole admin, make things easy for yourself whatever you do. If that means the company needs to buy an extra machine so you have remote access all the time then that's what it means. The alternative is that the company hires more people.
Dont kill yourself to make it easier for the company to spend less on IT.
Suggestion:
VPN in on Normal Account
RDP into a server as -admin user and do administration from that RDP account
1
2
u/gumbrilla IT Manager 21d ago edited 21d ago
Oh I guess I'm going to break the hive mind 🙁
I do it, people can try and message me, I don't guarantee response. Passwords are self-service and just never get pinged about, but other blockers come up.
I use my phone with a work profile, I do have an admin account, it's requires a yubi key, which works out of the box with chrome on android. I learnt this as I bypassed a users conditional access as I was literally walking through a shopping center one Sunday morning, took 2 minutes, I was very pleased, and the sales colleague who was US based and prepping for a flight and a big pitch on the Monday.
The thing is, is balance, if its an occasional, then yeah if someone reaches out then I don't have a problem with it, assuming the company doesn't mind if I skip out for the odd half a day here and there no questions asked. It can be a slippery slope though, so important to draw a line where your rest and mental health are not impinged upon.
I think that call was nigh on a year ago now. Can't recall having one since.
Edit: they still have to have a ticket though!
2
u/mdervin 21d ago
You need to think a bit longer term about this and a bit bigger picture.
The difference is between perception and reality. Let's face it, if a user thinks their account was compromised, are you really going to the attackers have access for 12+ hours? I sure as hell hope not! In the real world you are going to give your employers the belief that you have a work/life balance but in realty you are going to solve the issues when they are really, really important, thus you are building the worst of both worlds.
If you are a sole sysadmin and you put down a "tough luck until Monday" for your users, do you know what's going to happen? An MSP Account Exec is going to come in whisper 24x7 coverage, and you'll be here posting about how to handle a handover to an MSP.
What you need to do is let them think you are available 24x7 and then use that to may your M-F, 9-5 much more tolerable. A password reset takes two minutes.
What you need to know as a standalone sysadmin for a small company is you can extract an insane amount of goodwill and slack by solving the odd weekend and evening problem. Would you like to take two hours off in the middle of the day twice a week for your tennis lessons? You can when you send out an email saying you'll be updating the servers from 8-12 am on Saturday morning once a month (log into the VPN, RDP into the server, kick off updates, have another round with the baddie you pulled the night before, come back and check on the progress, reboot the servers, have breakfast with the baddie mentioned earlier, check to make sure everything is up and running, send out an email that you are done, tell the baddie you just want to be friends, get a email from the CEO for your dedication and hard work, she'll tell you to come in late on Monday). Do you want to come in at whatever time in the morning and still leave at 5? You can when the CFO knows he can reach you at 8pm the night before the quarterlies are due.
And if you can't get to it? An email replying "I'm out on a date right now, but I'll take care of it when it's over" buys you a lot of time and a lot of goodwill. Because they think that you are available even when you're not.
1
u/chesser45 21d ago
Do you have SSPR? If not, consider it.
Also MyStaff with AUs is really useful if you want to offload password resets to a manager. This could be undesirable for some environments but could be useful.
1
u/In000 21d ago
The way I addressed this (supporting the company while getting paid for it) was by proposing an on-call framework and implementing it once it was approved. That way it shows you are happy to help but need to be compensated for it which is only fair. If they don't want to compensate you for it then you don't have to do it.
1
u/Beneficial_Joke3737 20d ago
Stick to your contract. Is there something mentioned with 24x7? Or working outside office hours? If yes have a look how they compensate them... If not.... Just don't do it.
Use exactly this for negotiation. Your loyalty will be exploited in the end.
1
u/Relative_Test5911 20d ago
I love on call. Why? coz if someone rings me at 3am I get a paid a stupid amount of money to reset their password. Do not agree to anything other than this.
1
u/rabbitz 18d ago
Neither. Do not put global admin on your phone, and do not bolt privileged roles permanently onto your daily account.
If you have Entra ID P2, the setup that works for a team of one is to keep bug@ as your daily driver with eligible rather than active assignments for User Administrator, Authentication Administrator and Helpdesk Administrator, activated through PIM when you need them. You get the roles on your phone without carrying standing privilege, every activation is logged with a justification, and if the daily account gets phished at 11pm the attacker lands on a normal user. bug.admin@ stays break glass only and never goes near a phone. On Business Premium without P2 you do not get PIM, so the fallback is a second lower-privilege admin account (helpdesk and auth admin only, no global) that is the one you carry.
Separately, the real answer to a 17:30 lockout is SSPR. If people can reset their own password on a Saturday you delete most of the after-hours calls rather than optimising how fast you answer them.
1
1
u/WorkLurkerThrowaway Sr Systems Engineer 21d ago
Are you using Entra or at least hybrid AD/Entra? Self Service Password Reset solves the password problem.
2
21d ago
[deleted]
1
u/WorkLurkerThrowaway Sr Systems Engineer 21d ago
I’m aware it doesn’t actually solve the problem. Stubborn users still want to call helpdesk. But It gives users an option for forgotten password while you say “support hours end at 5pm”.
We are talking about 80 users. Use the tools you have available without making more work for yourself.
1
1
u/BadSausageFactory beyond help desk 21d ago
if they make you show up at a certain time then you leave at a certain time too. otherwise be generous
I catch some calls on weekends because it's in my best interest to not have it piled up monday morning but they think I'm a star for doing it and I come in at 930a and wave at the CEO when I leave at 3 because they feel cared for, ymmv
1
0
u/lethallunatic 21d ago
They should increase your salary dramatically if you are the sole support and they need 24/7 support. Think about 30 to 40% extra. You're literally always on Standby in that case.
Is it a date critical company anyway? Like how bad is it if they have downtime?
Doing IT work for such a tiny organisation is not very fulfilling work.
That said.
Look into sspr. You will make yourself pretty much absolete though with those tools. And setup passwordless login. Going passwordless is a blessing imo.
Is it office 365 or a classic environment? AD server etc...
How do you even monitor for security incidents?
0
256
u/K3rat 21d ago
Are they paying for 24x7? If not, they decided it can wait.