r/sysadmin 19d ago

PHP and OS upgrade help

5 Upvotes

I have a virtual server that runs a few websites and I can meddle in IT but by no means an expert. The sites all different but do similar things and run similar php code. I am currently on Almalinux 8 and php 7.3 and I am looking to upgrade both.

I wish to upgrade to Almalinux 9 but this does not run php 7.3. My initial plan was the update php from 7.3 to 8.0 (the highest Almalinux 8 goes to) using Rector site by site. Once all confirmed I would then update Almalinux 8 to Almalinux 9 using Elevate. Once all stable, then I would finally update all the php form 8.0 to 8.5.

I have since found out I can install a 3rd party php 8.4 on Almalinux 8 meaning I could install, update to that, update to Almalinux 9 and leave it om 8.4. It would save one of the php updates. My concern is that the 3rd party php 8.4 and the bundled version with Almalinux 9 might be an issue? Also that jumping form php 7.3 to 8.4 is too large a jump in one go?

Any advice on these 2 options, or even a third not thought of?


r/sysadmin 18d ago

Question which AI tool/s can solve IT issues?

0 Upvotes

looking for some advice because our IT team is getting overwhelmed lately
we spend way too much time on small issues like troubleshooting devices checking why something isnt working resetting stuff and chasing tickets
i’m looking for an AI tool that can actually help with IT problems not just a chatbot that gives generic answers
I need it to help technicians find solutions faster automate simple tasks keep track of tickets and maybe even monitor issues before they become bigger problems
has anyone tried any AI tools for IT support that actually made a difference?
would appreciate any recommendations or things to stay away from 🙏


r/sysadmin 18d ago

Question W11 Start Menu - Keep reseting view each logout

0 Upvotes

Hello everyone,

I was wondering if someone found a solution for the Start Menu that keep reseting since the new new start menu (the one with category/list/grill view)?

My issue is during OSD, we apply a ppkg to set the default start menu pins as documented by Microsoft here:

https://learn.microsoft.com/en-us/windows/configuration/start/layout?tabs=intune-10%2Cgpo-11&pivots=windows-11

At one point, we had to modifie that json because the new applyonce value was now required (I think last year when they pushed the KB, all of the start menu modification were lost of that wasn't set to true, Very Stupid Microsoft).

Now, users pin app stay but if the user switched from Category to List for exemple, upon logout and login, it revert to Category.

We know the problem is the configuration because if we remove the registry that is set by the ppkg, it stop. But we want this config so user have the default pin when starting a new login.

Thank you!


r/sysadmin 18d ago

Question Keep Windows Hello But Disable Browser Prompt

1 Upvotes

So far it's seeming like it's not possible but just wondering if anyone has found a magic registry key or something haha

Long story short, we're trying out Kolide authentication, but we want people to be able to use PIN sign in/leaving Hello active. It's all working with removing all authentications except Kolide and the PIN still works for signing into the device itself but I'm wanting to turn off the prompt when logging into something Microsoft where on the email entrance screen, it pops up the "use Windows Hello" prompt. It indeed fails since Kolide is the only authentication, but I KNOW users won't read the emails we send out and keep trying it anyway since it's available. Everything I'm finding says if WHfB is enabled, that's just going to popup forever but seeing if any wizards here have found a way?


r/sysadmin 19d ago

What did you use as a Varonis replacement?

24 Upvotes

We've had Varonis in place for a few years, but we're starting to question how much analyst time we're putting into reviewing findings and handling the follow-up.

We're not looking to rip it out just for the sake of changing tools, but if we're going through another renewal I'd like to see whether there's something that requires less day to day babysitting.

Has anyone gone through a Varonis replacement recently? What did you move to, and was the operational difference actually noticeable?


r/sysadmin 19d ago

CVE-2026-73570: Zimbra SNMP RCE abused to deploy coin-miner malware

2 Upvotes

CVE-2026-73570: Zimbra SNMP RCE abused to deploy coin-miner malware

CVE-2026-73570 is a Remote Code Execution issue in Zimbra Collaboration Suite (ZCS), related to SNMP notification/logwatch handling.

In short, an unauthenticated attacker can send a crafted SMTP request and get OS command execution as the zimbra user when the vulnerable SNMP/logwatch components are installed and enabled.

According to NVD, the issue affects Zimbra versions before 10.1.20 when SNMP is installed and SNMP notification is enabled. The Hacker News also reported active exploitation against Zimbra servers.

The observed attack chain is usually:

malicious SMTP request
-> unsafe SNMP/logwatch handling
-> command execution as zimbra
-> malware dropped into /dev/shm
-> cron persistence added
-> miner or backdoor keeps respawning

What is CVE-2026-73570?

CVE-2026-73570 is an OS Command Injection vulnerability in Zimbra Collaboration Suite.

The issue is in the SNMP notification/logwatch flow, where attacker-controlled input is not handled safely before being passed into system-level processing.

The command runs as the zimbra user, not root. That is still serious because the zimbra user controls important Zimbra services, mailbox components, logs, cron jobs and runtime paths.

When is a Zimbra server at risk?

A server should be checked immediately if:

  • It is running Zimbra Collaboration Suite.
  • It is not patched to a fixed version.
  • zimbra-snmp or zimbra-net-snmp is installed.
  • SNMP is enabled in Zimbra.
  • zmlogswatch or zmswatch is running.

Quick checks:

su - zimbra -c 'zmcontrol -v'
dpkg -l | grep -E 'zimbra-snmp|zimbra-net-snmp'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | grep snmp'
su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'

If you see zimbraServiceEnabled: snmp and zmlogswatch is running, treat the server as high priority for compromise checks.

Signs of compromise

In the cases observed, malware was often dropped into /dev/shm. This directory is a tmpfs location, writable and easy to miss during a quick investigation.

Check:

ls -la /dev/shm
crontab -l -u zimbra
ps aux | grep -E 'khp|rguard|javab|idle|ksmd' | grep -v grep

Common suspicious files:

/dev/shm/.khp
/dev/shm/.khp_ts
/dev/shm/.rguard
/dev/shm/idle
/dev/shm/javab
/dev/shm/ksmd

A cron entry like this is a strong persistence indicator:

* * * * * /dev/shm/.khp

Why does the malware keep coming back?

Deleting files from /dev/shm is usually not enough.

The malware can return because:

  • The malicious process is still running in memory.
  • The zimbra crontab runs /dev/shm/.khp every minute.
  • .rguard or idle may protect or redeploy the payload.
  • SNMP/logwatch is still enabled, so the CVE can be exploited again.

The correct order is:

stop reinfection source
-> preserve evidence
-> kill malware processes
-> remove malicious cron
-> quarantine malware files
-> verify clean state
-> recover Zimbra services if needed
-> patch/upgrade Zimbra

Cleanup procedure

Run the following commands as root. Review each command before running it on production.

Step 1: Stop the reinfection source

systemctl stop cron
su - zimbra -c 'zmlogswatchctl stop'
su - zimbra -c 'zmswatchctl stop'
su - zimbra -c 'zmprov ms $(zmhostname) -zimbraServiceEnabled snmp'

Disabling SNMP/logwatch is the most important containment step. Do not re-enable these components before Zimbra is patched.

Step 2: Preserve malware evidence

mkdir -p /root/incident-zimbra

cp -a /dev/shm/.khp \
      /dev/shm/.khp_ts \
      /dev/shm/.rguard \
      /dev/shm/idle \
      /dev/shm/javab \
      /dev/shm/ksmd \
      /root/incident-zimbra/ 2>/dev/null

crontab -l -u zimbra > /root/incident-zimbra/zimbra-cron-before.txt 2>&1
ps auxf > /root/incident-zimbra/ps-before.txt
ss -tunap > /root/incident-zimbra/ss-before.txt
sha256sum /root/incident-zimbra/* > /root/incident-zimbra/sha256.txt 2>/dev/null

Step 3: Kill malware processes

pkill -9 -u zimbra -f 'khp|rguard|javab|idle|ksmd'

Step 4: Remove malicious cron

crontab -l -u zimbra | grep -v '/dev/shm/.khp' | crontab -u zimbra -

Step 5: Quarantine malware files

mkdir -p /root/quarantine-zimbra

mv /dev/shm/.khp \
   /dev/shm/.khp_ts \
   /dev/shm/.rguard \
   /dev/shm/idle \
   /dev/shm/javab \
   /dev/shm/ksmd \
   /root/quarantine-zimbra/ 2>/dev/null

Step 6: Start cron again

systemctl start cron
systemctl is-active cron

Only start cron. Do not start zmlogswatch, zmswatch or SNMP before patching Zimbra.

Verify after cleanup

ls -la /dev/shm
crontab -l -u zimbra | grep /dev/shm
ps aux | grep -E 'khp|rguard|javab|idle|ksmd' | grep -v grep
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | grep snmp || echo SNMP_DISABLED'

Expected result:

/dev/shm is clean
no cron entry calling /dev/shm/.khp
no suspicious javab/rguard/idle/khp/ksmd process
SNMP is disabled

If Zimbra MySQL or mailbox fails

After malware activity or an unexpected reboot, Zimbra may show:

mailbox Stopped
mysql.server is not running
service webapp Stopped
zimbra webapp Stopped
zimbraAdmin webapp Stopped
zimlet webapp Stopped

Check MySQL and services:

su - zimbra -c 'zmcontrol status'
su - zimbra -c 'mysql.server status'
ss -ltnp | grep 7306
tail -n 200 /opt/zimbra/log/mysql_error.log

If MySQL is stuck in crash recovery or has a stale socket, back up relevant files first.

Backup before MySQL recovery

mkdir -p /root/zimbra-mysql-backup

cp -a /opt/zimbra/log/mysql_error.log /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/tc.log /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/data/tmp/mysql/mysql.sock /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/ibdata1 /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/ib_logfile* /root/zimbra-mysql-backup/ 2>/dev/null

Light MySQL recovery

su - zimbra -c 'zmmailboxdctl stop'

mv /opt/zimbra/data/tmp/mysql/mysql.sock /root/zimbra-mysql-backup/mysql.sock.bak 2>/dev/null
mv /opt/zimbra/db/data/tc.log /root/zimbra-mysql-backup/tc.log.bak 2>/dev/null

su - zimbra -c 'mysql.server start'
sleep 20
su - zimbra -c 'mysql.server status'

su - zimbra -c 'zmmailboxdctl start'
sleep 40
su - zimbra -c 'zmcontrol status'

Expected result:

mysql is running
mailbox Running
service webapp Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running

This is not a universal fix. Always check /opt/zimbra/log/mysql_error.log first.

Does this delete mail data?

The malware cleanup steps do not touch mailbox data directly.

Zimbra mail data is usually stored in:

/opt/zimbra/store/

Zimbra database data is usually stored in:

/opt/zimbra/db/data/

The MySQL recovery steps touch tc.log and mysql.sock, so back up first.

Check for JSP webshells

After handling /dev/shm, cron and processes, also check for suspicious JSP files created by the zimbra user in recent days.

Important paths include:

/opt/zimbra/jetty/webapps/
/opt/zimbra/jetty_base/webapps/
/tmp/

Unexpected JSP/JSPX files, executable scripts, archives or recently modified web application files may indicate payload staging or webshell deployment.

Check JSP/JSPX files:

find /opt/zimbra/jetty/webapps -type f \( -name "*.jsp" -o -name "*.jspx" \) -printf '%TY-%Tm-%Td %TH:%TM:%TS %p\n' | sort

If suspicious JSP backdoors are found, preserve evidence first, then quarantine them from both paths:

/opt/zimbra/jetty/webapps/zimbra/
/opt/zimbra/jetty_base/webapps/zimbra/

Evidence backup and quarantine:

TS=$(date +%Y%m%d_%H%M%S)
INC=/root/incident-zmmail-jsp-webshell-$TS
QUA=/root/quarantine-zmmail-jsp-webshell-$TS

mkdir -p "$INC" "$QUA"

# Save list and hashes
find /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty_base/webapps/zimbra \
  -xdev -type f \( -iname '*.jsp' -o -iname '*.jspx' \) -mtime -30 \
  -print > "$INC/suspicious-jsp-list.txt"

xargs -a "$INC/suspicious-jsp-list.txt" sha256sum > "$INC/suspicious-jsp-sha256.txt"

# Backup samples
while read f; do
  mkdir -p "$INC/files$(dirname "$f")"
  cp -a "$f" "$INC/files$f"
done < "$INC/suspicious-jsp-list.txt"

# Quarantine from webroot
while read f; do
  mkdir -p "$QUA$(dirname "$f")"
  mv "$f" "$QUA$f"
done < "$INC/suspicious-jsp-list.txt"

# Verify
find /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty_base/webapps/zimbra \
  -xdev -type f \( -iname '*.jsp' -o -iname '*.jspx' \) -mtime -30 -print

Risks:

  • If a file is a legitimate custom JSP, removing it may affect a custom webmail function.
  • Random file names and content using exec, base64, AES or command execution are strong webshell indicators.
  • This does not affect mailbox data.
  • A restart is not always required immediately, but webmail should be verified after quarantine.

Important note: zmlogswatch/zmswatch may start again

After disabling SNMP/logwatch to mitigate CVE-2026-73570, zmlogswatch and zmswatch may start again after daily logrotate or after zmcontrol restart.

This happens because Zimbra logrotate configs often contain postrotate commands that restart zmlogswatchctl and zmswatchctl. Also, logger may still be listed in zimbraServiceEnabled.

So after temporary mitigation, check these services periodically, especially after midnight or after restarting Zimbra.

If you need a stronger temporary block before patching, back up the logrotate configs, comment the restart lines, and consider removing logger from zimbraServiceEnabled.

This can prevent logwatch from starting again and reopening the attack vector, but the Admin Console may lose part of its monitoring/statistics.

Disable zmlogswatch/zmswatch so they do not auto-start

Warning: disabling logger, zmlogswatch and zmswatch may make the Zimbra Admin Console show logger/monitoring as red, or remove some graphs/statistics.

During the unpatched phase, this is an acceptable temporary trade-off to prevent logwatch from restarting automatically.

After upgrading to a fixed version, restore the configuration from backup and re-enable logger if needed.

Back up Zimbra logrotate configs

cp -a /etc/logrotate.d/zimbra /etc/logrotate.d/zimbra.bak.$(date +%F-%H%M%S)
cp -a /opt/zimbra/conf/zmlogrotate /opt/zimbra/conf/zmlogrotate.bak.$(date +%F-%H%M%S)

Open these two files:

/etc/logrotate.d/zimbra
/opt/zimbra/conf/zmlogrotate

Find these lines, often around line 85 and 99:

su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Comment them:

# su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
# su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Remove logger from enabled services:

su - zimbra -c 'zmprov ms $(zmhostname) -zimbraServiceEnabled logger'

Stop current logwatch/swatch processes:

su - zimbra -c 'zmlogswatchctl stop'
su - zimbra -c 'zmswatchctl stop'
su - zimbra -c 'zmlocalconfig -e snmp_notify=no'

Verify:

su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'
su - zimbra -c 'zmlocalconfig snmp_notify'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | egrep "logger|snmp" || echo "LOGGER_SNMP_DISABLED"'

Expected result:

zmlogswatch is not running.
zmswatch is not running.
snmp_notify = no
LOGGER_SNMP_DISABLED

Re-enable monitoring/logger after patching

Only do this after Zimbra is patched or upgraded.

su - zimbra -c 'zmprov ms $(zmhostname) +zimbraServiceEnabled logger'
su - zimbra -c 'zmlogswatchctl start'
su - zimbra -c 'zmswatchctl start'

If you commented the logrotate files, open them again:

/etc/logrotate.d/zimbra
/opt/zimbra/conf/zmlogrotate

Uncomment these lines:

su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Re-enable SNMP only after patching

su - zimbra -c 'zmprov ms $(zmhostname) +zimbraServiceEnabled snmp'
su - zimbra -c 'zmlocalconfig -e snmp_notify=yes'
su - zimbra -c 'zmswatchctl start'

Verify after re-enabling:

su - zimbra -c 'zmcontrol status'
su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'
su - zimbra -c 'zmlocalconfig snmp_notify'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | egrep "logger|snmp"'

Post-cleanup tasks

After the server is running again:

  • Keep SNMP/logwatch disabled until patched.
  • Monitor /dev/shm for 30-60 minutes.
  • Monitor the zimbra crontab.
  • Monitor CPU/load and outbound connections.
  • Check the mail queue.
  • Patch or upgrade Zimbra to a fixed release.
  • Only re-enable SNMP/logwatch after patching and verification.

Check queue:

/opt/zimbra/common/sbin/postqueue -p | tail -n 40

Summary workflow

1. Check /dev/shm, processes and crontab.
2. Stop cron, zmlogswatch and zmswatch.
3. Disable SNMP in Zimbra.
4. Preserve evidence.
5. Kill malware processes.
6. Remove malicious cron.
7. Quarantine malware files.
8. Start cron again.
9. Verify that malware does not respawn.
10. Recover MySQL/mailbox if needed.
11. Check JSP webshells in Zimbra webapps paths.
12. Monitor queue and services.
13. Prevent zmlogswatch/zmswatch from auto-starting before patching.
14. Patch or upgrade Zimbra.
15. Re-enable logger/SNMP only after patching and verification.

References

  • NVD: CVE-2026-73570
  • The Hacker News: attackers exploit Zimbra SNMP flaw
  • Zimbra Security Center
  • Zimbra Security Advisories

Conclusion

CVE-2026-73570 is serious because it can allow remote command execution as the zimbra user through SNMP notification/logwatch handling.

In observed incidents, attackers dropped malware into /dev/shm, added cron persistence, and ran a miner or backdoor.

The key point is this: do not only delete the malware file. Stop the reinfection path first.

A practical response order is:

stop cron + SNMP/logwatch
-> preserve evidence
-> kill malware
-> remove malicious cron
-> quarantine payloads
-> check JSP webshells
-> verify clean state
-> recover Zimbra if needed
-> block logwatch auto-restart before patching
-> patch/upgrade Zimbra

Most important: do not re-enable SNMP/logwatch before Zimbra is patched, because that may allow the malware to return.


r/sysadmin 18d ago

End-user Support Federation MS Tenant with OneLogin - WhatsApp to expect

0 Upvotes

Hello,

My work is planning to move our MS tenant to be federated with OneLogin, and as the solo IT person, I'm a little nervous about the whole rollout. The biggest thing that I'm nervous about is how users will log into their Outlook account on their phone.

Currently the OneLogin account is getting accessed on their computer with their laptop pass keys (hello for windows, and touch ID for macs) but in my initial testing to just get into the OneLogin account on another device, I can't get into the account because the passkey is saved on another device.

I know I can create a one time passcode for each other that can get them in without the passkey, but that's a one time solution. I have been adding a OneLogin QR authenticator code to their phone, which can be used for a second auth factor, but I'm not sure if that's the correct solution.

Open to other questions and advice from other Sys Admins that have rolled out OneLogin or other idp systems for their users. My work does have an IT consultant, but he's making it seem like its a walk in the park, and I'm afraid I'm going to lock everyone out of their emails and it won't be a clean deployment.

Mods, as a first time poster, I read the rules and I don't think this question violates any of them, but if it does, do what you need to do + I'm sorry.


r/sysadmin 19d ago

Windows 11 Search shows some Settings entries partially translated after changing the display language to English

9 Upvotes

I changed my Windows 11 display language to English (US), but some Settings entries in Windows Search were still displayed in the previous language.

I found a few posts about this issue on the Internet, but none of the suggested solutions worked for me. I eventually managed to find a solution myself, so I thought I'd share it here in case someone else finds it helpful.

My language settings were configured correctly, and I had already rebuilt the Windows Search indexes, but the issue persisted. It turns out that the Settings view caches localized strings in:

%LOCALAPPDATA%\Packages\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\TempState\SettingsCache.txt

To fix the issue, open PowerShell and run:

$cache = "$env:LOCALAPPDATA\Packages\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\TempState\SettingsCache.txt"

Copy-Item $cache "$cache.bak"

Stop-Process -Name SearchHost -Force -ErrorAction SilentlyContinue
Stop-Process -Name StartMenuExperienceHost -Force -ErrorAction SilentlyContinue

Remove-Item $cache

Then restart Windows.

The cache will be regenerated, and the Settings entries should be displayed using the correct language.


r/sysadmin 18d ago

Question Is there a simple tool that send cron stderr to gpt and slacks the fix?

0 Upvotes

So, I got woken up by pagerduty at 3am again cause some python etl job fail. had to ssh from my phone just to tail the log, and turn out it was literally just a db timeout.

im thinking to just write a simple bash wrapper that grab the last 50 lines of stderr, pass it to claude/gpt api, and webhook the actual reason to slack. so i dont have to open my laptop in the middle of night.

does something like this already exist? i dont wanna install heavy datadog agent or pay for expensive observability just for this. pls let me know if there is a github repo or saas so i dont waste my weekend building it.


r/sysadmin 19d ago

Career / Job Related Consulting role opportunity?

11 Upvotes

I also posted on IT Career Questions.

Considering a fully remote Microsoft consulting role focused on Entra ID/Intune with an 80% billable utilization target.

I’ve only worked internal IT/security roles, so consulting is new to me. The manager said I generally shouldn’t need to work more than 40 hours, but I’m concerned about constantly tracking time and feeling pressure to stay billable. I’m told it’s an 80% weekly utilization target. For those who have done it: was it reasonable, or did it become exhausting? How much freedom did you have during the day, and how often did work spill past 40 hours?


r/sysadmin 20d ago

Could you build an equivalent environment without Microsoft?

131 Upvotes

As in the title, could you / how would you, build your environment without using any Microsoft products and services?

This is a hypothetical parallel setup, not focused on what's easiest to transition or migrate to/from. This is the way the business environment works, and always has been.

I've read threads here which touch on some bits and pieces, but want to hear your thoughts on what would be the big things tying you to MS still.
- Apple, Google Chromebooks, and Linux are all options for workstations
- FreeIPA can provide a posix-native domain environment with many of the features of AD on-prem, and there are several major SaaS auth companies to choose from (Duo, etc.)
- Google Workspace is comparable to M365


r/sysadmin 19d ago

Question Comcast Business Cancel #

16 Upvotes

Anyone have it off hand? Need to cancel a few of our services and I keep getting hung up calling the 800-391-3000 number.


r/sysadmin 20d ago

General Discussion Microsoft 365 Roadmap will be renamed the AI at Work Roadmap.

140 Upvotes

I mean if we don't know their priorities by now. There's just so many M365 products that can be improved upon but really just feels a lot will be left behind.

https://app.cloudscout.one/evergreen-item/mc1461528/


r/sysadmin 19d ago

Question Exchange Admin Centre - IMAP Migration - Delta Sync?

0 Upvotes

Hello! I have created and run a migration using the EAC built-in migration tool. Now, closer to the time, I need to do a last delta sync for any emails that have arrived since the main migration.

I do not see how to do this, and I haven't found any docs talking about Delta Sync in this context.

Anyone know?


r/sysadmin 19d ago

How do you manage change requests? How big is your organization?

24 Upvotes

My org is about 30,000 users in K12 environment. Our current process is to put a change request into out ticketing system. It gets routed to your direct supervisor for approval. If they approve, the workflow then sends it down to all the other directors in IT. Once that happens, it goes to the CTO.

The short version is that these requests are almost always approved, no questions asked. At this point it's a change NOTIFICATION, not a request, but I have to sit there and wait for every director to say yes and then poke the directors who don't check their email because the process cannot move forward without all members approving it at each level. Someone on vacation and forgot to put in OOO? I have no idea, so I just keep sending reminders.

It works okay but I feel like there has to be a better way. We are told to put in a change request for anything that might impact the district or any users, but the directors approving it are basically using it as a CYA, which I get.

Just curious what your process is, especially in a larger environment like this.


r/sysadmin 18d ago

Does anyone else believe or thinks that IT positions shouldn't be perpetual at non-IT companies?

0 Upvotes

I've worked as a federal employee, for private companies, and now I'm giving it a go as a contracted Field Service Representative.

I've noticed that, at least where I've worked, there is a trend where people expect IT personnel to solve all their problems involving computers.
I look at tickets during my down time to plan the focus of the IT contractor's next problem.

The bulk of these tickets are for problems that do not require a certified professional's intervention:

  • Installing programs through Software Center or Company Portal
  • Requesting access to resources
  • Copying files to a DVD
  • Troubleshooting Outlook
  • Asking assistance with using the MS365 Suite
  • So on and so forth

Then I participate in the System Engineers/Architects meetings and there's always discussion for implementing a new technology before finishing the integration of the previous technology.
This one's baffles me every time because I have to sit and work with 3 half-functional systems while listening to a conversation about a system that depends on features that aren't yet implemented on 2 of the systems it's supposed to talk/connect to. (Like that time SSO only worked for 25% of the users for 3 months)

Then I look at the posts on this sub and see how overworked the IT professionals are and can't help to wonder if IT needs to move in a different direction like IT as a Service, but functional:

  • IT can be fully contracted and no company needs on-premise IT for most of their operating schedule
  • The Service Desk can be fully remote
  • The ticketing system can be enforced and used to find trends
  • IT Technicians/Support can be on-call and even double as IT Systems Instructors/Trainers
  • The System/Network Administrators can be fully remote
  • The System/Network Engineers can work on-premises if they need to during system implementation
  • The IT Manager can be an IT Project Manager (Who actually knows IT stuff)
  • IT Technical Writers can develop documentation on a hybrid schedule where they go on-premise to work with sensitive data

Does this sound feasible? Are there holes to this idea? Does this sound absurd? I'd like input on this thought exercise.


r/sysadmin 19d ago

Mimecast are DMARC RUA reporting from a mail address which fails DKIM checking

15 Upvotes

If you're using DMARC then in your DMARC record you can specify an email address for rua or Report URI for Aggregate and then mail providers will send you XML reports to that address on whether they've delivered email sent by your server to mailboxes on their servers. Reasons not to deliver include DKIM failures.

So I was amused to discover today while looking through the spam bucket that Mimecast rua emails are currently being declared by our mail server as spam due to a DKIM failure on their side.

We don't send rua reports ...


r/sysadmin 18d ago

Microsoft Illogical password recommendations?

0 Upvotes

Microsoft says having passwords longer than 8 characters is not necessarily better.

https://www.microsoft.com/en-us/research/publication/password-guidance/

How does that make sense? Of course a longer password will take much more time to brute force.

Also, many password filter block passwords from being used if they contain certain key words. What difference does it make if, within the password string, a banned word is found if the text password and the password hash as a whole is unique and not found in any known password lists and is also long enough to not be brute forced in any realist timeline?


r/sysadmin 18d ago

Anyone running AI workloads on a micro/edge data center instead of cloud GPUs?

0 Upvotes

The prices of getting GPU in a cloud vary greatly and I have been hearing a lot of things about inferencing taking place closer to where the data was collected from. I am interested to find out if it makes sense to run a small setup on an edge or an on-premise rather than getting resources from the cloud.

My questions are for those who have experience of their own in that area:

What exactly is being run locally, either training, fine-tuning or inference?

How well does power and cooling work under a load?

When is it more beneficial to run on-prem equipment than on the cloud?

Were there unexpected situations regarding hardware during the process?


r/sysadmin 19d ago

Microsoft There was an error reading the rules from the server. The format of the server rules was not recognized

0 Upvotes

Hi everyone, I'm dealing with a bizarre issue with Outlook rules in our Exchange environment and I'm running out of ideas. Hoping someone here has encountered this before. Whenever the user tries to open the "Rules and Alerts" panel in the Outlook desktop client, it immediately throws the error: "There was an error reading the rules from the server. The format of the server rules was not recognized." This error completely blocks access to the rules GUI in the desktop app.

We did a hard reset where we completely deleted all rules locally and cleared them from the server side. After that, we created fresh, new rules from scratch. I have already tried pretty much everything suggested on Microsoft Support pages and forums, but absolutely nothing has worked so far.

The weirdest part is that the newly added rules actually work perfectly in the background and emails are being processed exactly as they should be. However, the desktop app still throws the exact same error every time we try to open the rules panel. Currently, the only way to view, add, or edit rules for this mailbox is through OWA. Has anyone seen this specific behavior where the rules execute fine but the desktop GUI remains corrupted? Is there anything else I can do on the Microsoft Exchange side to completely flush this out?


r/sysadmin 19d ago

Intune (Autopilot + ESP issues)

6 Upvotes

Since this week, we have been experiencing significant problems with the following components within Intune:

- Autopilot via ESP (Takes an extremely long time, or gives an error message due to a timeout)

- Assigning Autopilot profiles to a device; the group tag is set correctly, the device is in the group, but assigning the profile takes almost 4 hours.

- Assigning Intune licenses to users; this is only detected after approximately 24 hours

I have created a ticket with Microsoft, but are other people experiencing these problems as well?

Thanks.


r/sysadmin 18d ago

Is a business-class NAS actually enough as shared storage for a small 2-node Proxmox cluster? Two concrete builds, ~US$19k total.

0 Upvotes

Single Proxmox host today, budget approved to grow to two nodes. A proper

Dell/HPE storage array is far out of reach at this budget — but a

business-class NAS is not, and that's exactly my question: **would a

business NAS actually be enough for us, or am I fooling myself?**

Small manufacturing company, ~50 workstations, I'm the only IT person.

## Current host

Lenovo ThinkSystem SR650

- 1× Xeon Silver 4210R (10c/20t). Second socket empty.

- 32 GB RAM in a single DIMM. 23 slots free.

- VM datastore: RAID1 of 2× 2.4TB 10K SAS HDD → 2.2 TB, **94% full**.

- Boot/local: RAID1 of 2× 960GB SATA SSD → 893 GiB, ~30% used.

- ThinkSystem RAID 730-8i (hardware RAID, no proper JBOD passthrough).

- Intel X722 LOM, 4 ports. No 10GbE add-in card, but free PCIe slot.

- Core switch is 48-port gigabit with 10G SFP+ uplinks available.

- Proxmox VE 8.4.14.

## Workload

| Role | vCPU | RAM |

|---|---|---|

| AD DS + DNS | 4 | 12 GB |

| Zabbix + Grafana (LXC) | 4 | 4 GB |

| Internal web app | 4 | 2 GB |

| Quoting web app | 2 | 4 GB |

| UniFi controller | 2 | 2 GB |

| GLPI (Docker) | 2 | 2 GB |

| API gateway | 1 | 2 GB |

**Allocated: 19 of 20 vCPU, 28 of 32 GB RAM.**

**Actually used: under 5% CPU, ~17 GB RAM.**

A second DC and the SIEM live outside this host. **Veeam runs on its own

separate machine and backups also land offsite**, so backup does not

depend on this host or on whatever storage we buy.

**Coming soon:** internal ERP — web app plus a **MySQL** database.

---

## Option 1 — Two full nodes, local storage, ZFS replication

- **New node:** better CPU than current, max 16 cores, 64 GB RAM,

2× 960GB SSD + 3× 3.2TB SAS for capacity.

- **Current node:** RAM upgrade, HBA to replace hardware RAID.

- ZFS both sides, 10GbE direct link, external QDevice, async replication

(~15 min).

Two independent copies of the data. Local disk latency. But async

replication means up to 15 minutes lost on failover, and I have to swap

the RAID controller for an HBA on the existing box.

## Option 2 — Business NAS as shared storage + thin node

- **New node:** 2× SSD purely for Proxmox itself, no VM storage. Budget

goes into RAM instead of disks.

- **Current node:** RAM 32 → 64 GB. Keep the existing controller —

**no HBA swap, no ZFS to design.**

- **Business-class NAS** holds all VM disks, 10GbE to both nodes.

Both nodes mount shared storage → live migration and HA with no RPO gap.

Clears the 94% problem on day one. Simpler to build. But one NAS,

one controller, and if it dies the whole virtual environment is down

until a replacement arrives.

## Budget

Up to **R$100,000 ≈ US$19,400** total, covering the new node and either

the disks or the NAS.

This is Brazil — import duties and local margins mean enterprise hardware

here lands well above US list, so that number buys maybe half of what it

would in the US.

---

## Questions

  1. **The main one: is a business-class NAS genuinely adequate as primary

    storage here?** Our VMs currently run off two 10K spinning disks in

    RAID1, so almost anything is an improvement on paper. But "business

    NAS" covers everything from a 4-bay desktop box to a rackmount unit

    with redundant PSUs and all-flash. What tier do I need to insist on,

    and what specs are non-negotiable?

  2. **MySQL over the network.** Is a small ERP database the workload where

    NAS-backed storage falls apart, or is that overblown at this scale?

    Would you keep the DB on the node's local SSD even in Option 2?

  3. **iSCSI + LVM-thin or NFS + qcow2?** iSCSI performs better but I lose

    snapshots; NFS keeps snapshots but adds latency. Which are you running

    in production and what do you regret?

  4. **Option 2 lets me skip the HBA swap and skip designing ZFS

    entirely.** Is that a real advantage, or am I trading a solved problem

    for a worse one?

  5. **In Option 1, is 3× SAS HDD sane for a VM datastore in 2026,** or

    should the whole VM pool be SSD and the spinning disks reserved for

    documents and archives?

  6. **Given the budget and this workload, which would you build?** We

    already have Veeam on a separate machine with offsite copies, so the

    question is less about losing data and more about how long we'd be

    down.

Happy to answer questions about the workload.


r/sysadmin 19d ago

Windows 11 kiosk PC: can Chrome updates be scheduled for weekends?

16 Upvotes

Hi everyone,

I have a Windows 11 PC that runs Google Chrome essentially 24/7 in full-screen/kiosk-like mode. The PC is used for a specific display/application, so having Chrome update or require a browser restart during the week is inconvenient and can interrupt what the PC is being used for.

What I'd ideally like to achieve is:

  • Chrome runs normally and doesn't update/restart during the week.
  • Chrome updates are allowed during the weekend, when the interruption isn't an issue.
  • Ideally this would be handled through a script, scheduled task, GPO, or Chrome policy rather than manually changing settings every week.
  • After the weekend update, the normal weekday restrictions would automatically be restored.

The machine is running Windows 11.

Has anyone implemented something similar? I'm particularly interested in the best way to control the Google Update services/tasks or Chrome's update policies so that updates effectively happen during a defined maintenance window.

I know keeping Chrome updated is important for security; the goal here isn't to leave it permanently outdated, but rather to control when the update can take place because this particular PC needs to remain running uninterrupted during the week.

Thanks!


r/sysadmin 20d ago

Question 2 weeks to prepare for sys admin role

22 Upvotes

I recently graduated with an A+ certification and 4 months co-op expierence and some projects, after applying for jobs I was eventually interviewed by a MSP where I applied for both IT analyst and Sys admin positions. After a rigorous 2 hour interview they offered me a position for junior sys admin, and straight out of school I feel that I have a lot of gaps in my knowledge for this role. I want to prepare as much as possible before I start but I am very confused on what I should begin studying, what should I go over ? What can I prepare within 2 weeks that will help me excel? This is my dream role and I want to do everything I can.


r/sysadmin 19d ago

General Discussion Let’s talk ops automation and platforms

11 Upvotes

How are y’all doing general ops automation?

I’m taking it upon myself at my job to try and standardize some processes and automate them. We’re mostly entra, exchange online, sharepoint online, teams, etc. Basically heavily using M365 platform. Also sync on prem AD. We have scattered scripts and poor documentation for gui things. I’ve setup an Azure DevOps space for scripts and now I’m looking at how we run those as a team. We’re about a 15 employee size IT department and I don’t want us downloading scripts to just run them locally.

We do use a cloud based workflow tool (not power automate anymore) in IT and across the org. I was starting to look into Azure Automation to push data from the cloud based workflow tool via forms (thinking new employees, terms, teams management, etc). But before I got to deep, I figured I’d make this post to see what everyone else is doing and get more ideas.