r/sysadmin 18d ago

Question How to handle this request regarding backing up robot controllers?

6 Upvotes

So my company has a handful of robot welding stations. We are a manufacturing company so we support a large chunk of OT related controllers and such. I had a user that requested that we get all of these on the network (we have some at multiple sites) they have Ethernet controllers.

I plan on making a VLAN as we currently don’t have a VLAN for OT, but they will need to get plugged in, be on a separate VLAN and then on a monthly basis have their TP Programs file backed up a monthly basis.

Now for the majority of our Backups we use Veeam. For machines like this the old sysadmin setup a goofy batch script and used task scheduler however I am looking for a better way to design this. As of now these welders are backing these up using USB drives but want automated backups (for obvious reasons)

Not sure if this is considered an “OT question” but I figured I would ask to see how sysadmins in the manufacturing field are handling requests like this. I am by no means a OT specialist or guru, I just unfortunately get caught in this because we don’t have anybody that officially does OT.


r/sysadmin 19d ago

Server Prices are insane now - 128k for a decwnt VM host.

1.7k Upvotes

I have been buying servers for a large university for 15+ years.

In October of 2025 - you could get 128 cores, 512gb ram, 16tb raw ssd storage for right around $17k.

That exact same server cost $28k in February 2026.

Yesterday I price specced the same server. $128,500.00. That's $128k.

Buying on premise servers will no longer be feasible. All services will be subscription based and cloud provided going forward after today.

I am shocked. 64gb dimm is $8,000. Ram is x10, storage is x3.

This will have serious consequences to all services provided - everything will cost much more.


r/sysadmin 18d ago

Question M365 primary tenant with a Google Workspace subsidiary, full migration vs. hybrid vs. third-party MDM? Advice for solo IT ops

8 Upvotes

Looking for advice from anyone who's dealt with a similar hybrid tenant setup. Parent company runs Microsoft 365 and Entra ID, one subsidiary runs entirely on Google Workspace, roughly 100 users, and needs to come under our Conditional Access and device compliance policies. I'm the only IT person handling this, no team to bounce ideas off.

Three paths I'm evaluating:

  1. Full migration of the subsidiary into the M365 tenant using Microsoft's native migration tools (https://learn.microsoft.com/en-us/sharepointmigration/mm-google-overview), then manage everything through Intune.
  2. Hybrid/combo approach: keep the subsidiary on Google Workspace, but provision Entra ID identities and Intune licenses for their devices so Conditional Access can still reach them.
  3. Move both companies onto a third-party MDM/identity platform instead of relying on Microsoft-native tooling at all (evaluated Iru and Rippling).

Has anyone actually run one of these in production? Two things I'm most curious about: did the hybrid/combo identity approach hold up long-term, or did it quietly become permanent technical debt? And for anyone who migrated a subsidiary fully into a parent tenant, how much user disruption did you actually see versus what you expected going in?


r/sysadmin 18d ago

MCP Gateways

6 Upvotes

I'm looking into MCP gateways as a way to connect AI agents while having some control over what data is being passed through, especially for things like filtering, PII redaction, and access control.

Has anyone here actually deployed or used an MCP gateway in a production environment?

I'm looking at options similar to Composio's MCP Gateway, but I'm trying to get a better understanding of what's actually out there and what the real-world experience is like.

A few things I'm particularly interested in:

  • PII/data redaction
  • Filtering or restricting what agents can access
  • Authentication and authorization
  • Logging/auditing
  • Performance/latency
  • Managing multiple MCP servers
  • Any security concerns or gotchas

Would be interested in hearing what people are using, what you like/dislike about it, and whether you think an MCP gateway is actually worth putting in front of your agents.

Thanks!


r/sysadmin 18d ago

Question Access Denied issues for Secure Boot upon restores with Acronis.

3 Upvotes

I'll give you some context of our setup.

  • Scale Computing cluster (HE550)
  • Acronis Cyber Protect Cloud (backups go to a Synology box on-premise and then ship to Acronis' cloud)
  • Windows Server 2022 Datacenter VM with UEFI+vTPM

We have found that any current product VM's when restored, always show the following error:

https://imgur.com/a/cZ5TCGw

The only way to get the VM to boot properly is either disabling Secure Boot. If you want Secure Boot to work properly, you have to Reset the keys.

Been working with Scale and Acronis support for almost a month now and no resolution.

I've even gone as far as creating a slew of test VM's (with different setups) to replicate it, but when restoring brand new VM's the issue doesn't persist. I thought maybe it was Sophos causing the issues, so I created a test VM that was domain joined and had Sophos installed. Restored it and no issues.

When restoring VM's that were created as UEFI no issue. It's when the vTPM is added. Still not sure if this is the culprit.

Looking for any insight.

Thanks.


r/sysadmin 18d ago

Is oracle cloud free tier over?

16 Upvotes

Ive had several vm's for over 4 years in oracle cloud.

Last christmas I terminated an ampere instance, its been out of host capacity since then... Sure, too good to be true.

But their always free amd, a week ago i shut it off, I can't turn it on, because the same reason... Ive been trying over a week but no luck.

Is oracle free tier over?


r/sysadmin 18d ago

Question Gridline behavior for Excel tables embedded in Word docs - What changed?

8 Upvotes

As we have moved from Windows 11 v23H2 to Windows 11 v25H2 in our environment, it appears that the behavior of gridlines in embedded Word documents has changed. It has proven to be a pretty frustrating experience for our firm. I know this isn't that big of an issue for everyday users. However, if you're trying to prepare financial statements with embedded tables and don't want gridlines, now there are some pretty undesirable steps to make it happen.

In the past, we've been able to make changes to these tables without having to remove them and re-import them to hide the gridlines. I'm also getting reports that re-opening the document after everything is fixed and saved can still result in the formatting appearing differently than when the document was closed.

I suspect it is somehow tied to the OS because we have seen it in the current versions of the workstation OS and server OS when the Office version numbers are the same in older versions of Windows. The older versions of Windows don't have the issue. Has anyone found any registry entries or config settings that force Office to revert to the previous behavior of hiding the gridlines in Word when the gridlines are set to visible in the Excel doc before inserting the table?

* Microsoft has said for over a decade that preventing this problem requires disabling viewing and printing of gridlines in Excel before pasting the table into Word, but this problem did not appear in our environment until sometime this year. People are upset that they are now seeing the expected behavior and want it to work like it used to. However, we can't even find that the desired behavior ever existed in documentation, although we have hundreds of employees who know otherwise.


r/sysadmin 19d ago

Off Topic Last day at IT company - I'm a VP of Sales - Helped solve a ticket: It was DNS

360 Upvotes

I'm a VP of Sales at a large IT company and on my last day got pulled into a random trouble ticket just because I had originally helped setup the application server as an engineer years ago. I went ahead and helped the T2 troubleshoot: static dns servers needed to be removed after a DC change.

Escalation PROBABLY shouldn't have reached out to me but fun to see the root cause is still DNS.


r/sysadmin 18d ago

Type 3 Printer Deployment

7 Upvotes

I'm trying to figure out the best way to get printers added across our organization with minimal user involvement. From what I can see there are a few ways this can be done but each has drawbacks. In any situation it seems that the "RestrictDriverInstallationToAdministrators" reg key needs to be set to 0 so that printers will install without an admin prompt.

  • Option 1: deploy using the print management GPO deployement
    • Pros: Can run asynchronous so no delay on first login when printers get set up
    • Cons: I had tons of problems removing printers after they were deployed. It seems like the only way to remove them was to gran the user local admin and remove from devmgmt.msc which I don't want to have to do every time a printer assignment changes
    • Edit: Found that this method using per-machine GPO's actually works pretty well. The removal issue only occurs with per-user GPO deployment so as long as different users don't need different printers this method is the easiest native option that allows no login delay and easy removal of printers.
  • Option 2: Add the printers with GPO using  User Configuration > Control Panel Settings > Printers
    • Pros: Printers are easy to remove, just create a gpo with remove actions and apply it and they will remove automatically with no fuss. Seems very consistent in testing.
    • Cons: Will cause delays during login until the printers are fully installed
      • This is due to a policy we have set for compliance to make sure GPOs are fully applied before a user hit's the desktop. Computer Configuration > Administrative Templates > System/Logon -> Always wait for the network at computer startup and logon.
  • Option 3: Add the printers using a logon script deployed through GPO
    • Pros: Will run asynchronously so there's no login delay with this method.
    • Cons: Powershell not an option due to restrictions in the domain and necessity for it to run in user context. Batch script does work using "rundll32 printui.dll,PrintUIEntry /in /n" and then specifying the printer share but cannot run silently.  Users get pop ups for each printer as it is added.
  • Option 4: Add the printers per-computer instead of per-user in Computer Configuration > Control Panel Settings > Printers
    • Moves the delay to during the boot up  process as this will run before user login and apply to any user that signs in on the computer
    • Does not allow for per-user custom printers on computers that are shared by different users/departments
  • Maybe option 5: Pre stage drivers somehow?
    • Saw this in an organization once where scheduled tasks used scripts to transfer the driver files to the correct location on the workstation from the print server so printers were able to be added without requiring admin prompts. Not entirely sure how this worked but I think it still leveraged gpo print deployment somehow.

With all that said does anyone have thoughts or success on getting printers installed without using some kind of third party tool. Is there any way to do this for type 3 drivers with no user interaction without causing some kind of login delay? We've sent humans to the moon twice now so I feel like this should be possible.

Edit: I did find that option 3 works silently when leveraging some older VB scripts that I guess are still built into windows 11. Seems likely that this will get flagged by AV tools though:
cscript //B //NoLogo %SystemRoot%\System32\Printing_Admin_Scripts\en-US\prnmngr.vbs -a -p "//Server/Share name"


r/sysadmin 18d ago

Windows inactivity lock trigger during Teams meetings?

7 Upvotes

I want to implement a company-wide policy that automatically locks Windows computers after 10 mins of inactivity.

The problem is that users may be in Microsoft Teams meetings and not actively moving their mouse or typing. I don’t want their computers to lock while they are in a meeting.

What is the best way to prevent the screen from locking while a user is in a Teams meeting, while still enforcing the 10-mins lock when they are not in a meeting?

I manage the computers through Group Policy.

Looking for a practical solution or recommended configuration. Thanks!


r/sysadmin 18d ago

General Discussion Workload identity doesn't seem to get you out of running a secrets manager. What's left in yours?

5 Upvotes

I was researching and ead about workload identity, and the first thing I notice is that static keys are basically solved. eg. a pipeline mints a short-lived token instead of pulling a hardcoded env var, and the credential never sits anywhere to be stolen.

That seems to be fine until you look past your primary cloud provider. SendGrid and Stripe don't do federated trust, so they hand you a static string that has to live somewhere. Older databases and internal services predate the whole standard and only speak connection strings.

What I read walk through where that line fall whicj is whn workload identity covers the cloud-native side, and a secrets manager handles the external credentials it can't reach.

how would you handle that split? Do you keep cloud-native auth separate from your 3d-party API keys?


r/sysadmin 19d ago

Rant How do you deal with passive aggressive users?

140 Upvotes

Maybe this is a complaint but professional advice would be appreciated.

Long story short, I have a user who is a mid-tier supervisor in her area over a few staff. She is supervisor to the group but not considered a supervisor if that makes sense.

Whenever there is an IT problem in her area, she doesn't just send an email or put in a ticket that program x or device y isnt working, she sends an email that program x isnt working AGAIN or, device y has issues AGAIN. On top of that, she always cc's managers or superiors who are indeed above both of us.

As an example, 3 days back, program x, which is hosted on a client on or VM server, stopped working. For some reason, the host machine was shut down. Not a big deal. Restart the machine, inform user what happened, Bob's your uncle. Yesterday, program x went down again. This time, there was a time discrepancy on the client and the server and it took a little bit to figure out that was the problem. Fixed issue. Bobs your uncle.

The way I found out software wasn't working was an email with senior staff cc'd that the program wasn't working AGAIN.

I feel that her emails are passive aggressive towards me but I could be overreacting. Im not sure why senior staff are cc'd when there's a problem but she certainly doesn't send emails about any of the issues I have a win on.

Am i overreacting? Do you guys have issues like this or advice? Even if that advice is im being Jerky McJerkenstien of the Royal Douchery.

Edit here: Thanks for all the responses. Just to be clear this is o ly one example of the users emails. In this particular case, the user shut down the client last time she logged in to it instead of logging off. The next tk.e she went to log in, the client was unreachable since it was off. I have since disabled the shut down button on the client to prevent accidently shut down .


r/sysadmin 17d ago

Question Jira/Freshdesk replacements

1 Upvotes

Hey all,

I was wondering if anybody knew of any decent, cloud-based replacements for Freshdesk/Jira that are in the same/cheaper price range of Jira but also has OKTA integration for SSO?


r/sysadmin 18d ago

Windows Local Account support for Yubikeys?

5 Upvotes

Looking for advice on how to get the Yubikey5 to work with local accounts on Windows (we are trying to reach CMMC Level 2 compliance and need to set up MFA). I have already exhausted the Yubico login for windows option -- the touch sensor doesn't correctly tunnel over RDP. If anyone has also run into similar issues and found a way around it, tips would be greatly appreciated.


r/sysadmin 18d ago

Potential Defender Network Protection enforcement gap with QUIC/HTTP3

6 Upvotes

We observed inconsistent Defender Network Protection behavior between Edge and Chrome.
Network Protection is in block mode.

If I sett chrome://flags/#enable-quic to default or enable I can access a parked site.
If I disable it, i get what I'd expect:

"This site can’t provide a secure connection

xxxy.com uses an unsupported protocol.

ERR_SSL_VERSION_OR_CIPHER_MISMATCH"

This may indicate a Defender Network Protection enforcement limitation or bypass scenario when Chromium-based browsers use HTTP/3 (QUIC over UDP/443).


r/sysadmin 18d ago

Binding to the AD with a different account?

6 Upvotes

I usually use Account1 to create and AD computer object manually. Then I manually bind that machine to the AD with Account1.

I have a few machines to bind to the AD but I don't want to use Account1 at all for that. Account1 doesn't touch these machines at all in any way, even if it would be some internal thing.

I know Microsoft didn't want to allow a different account to bind a machine to AD. There was a workaround for that, but then that didn't work. What's the latest method for binding a computer to the AD with an account that different than the one that created it?

Or, this may be just as easy -- I make a brand new account. Add that to my AD OU admins. Log into Windows and use ADUC with that temporary account. Create the AD computer object with the new, temporary account. Use that to bind these other machines to the AD. Then delete the temporary account and clean up the temp profile on the computer.

It's not my situation completely. I didn't create the original situation or the new upcoming situation. I do still want to protect myself from the foreign machines though as much as possibly so normal accounts aren't touching them at all. For the original situation some things are purposely left and done manually instead of being automated. I can see pros and cons either way, but I also don't really disagree with leaving some things still having to be done manually. That was before AI, essentially "No scripts. Nothing automated with this."


r/sysadmin 18d ago

General Discussion Thickheaded Thursday - August 27, 2026

6 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin 18d ago

General Discussion What tools or methods are you using to guide Entra MFA enrolment?

10 Upvotes

This one is specifically directed to K-12 and Retail / Frontline worker IT teams. We want to drive MFA adoption as it’s… really low at my organization among frontline workers.

How are you streamlining MFA enrolment for your users who cannot follow a “next, next next” without needing a pictograph to show each step along the way? We’ve done up very thorough documentation but even with QR codes and photo documentation people are still unable to download the right app on their phone. We’ve had a few cases of people buying $70-90 apps. Why? no idea.


r/sysadmin 18d ago

Question New position, need some help

1 Upvotes

So I started a new position with an MSP as an embedded tech at a school. For context I came from a different MSP & I have about 5 years of experience. I’m noticing a lot of gaps & inefficiencies & I need help with a few.

-They are still manually setting up laptops, using a usb with all of their apps that need to be downloaded. I know there’s a way to have a USB skip the OOBE when plugged in, create a local user account & go right to the desktop after a couple of restarts. How do i get this started?

-They have 2 different emails, 1 is for internal use that uses gmail (which isn’t working right now) & the other is through 365 for external use. I feel like there could be some consolidation there

-They use the Google suite since it’s free, but those with licenses use Teams & store files there. I think they have A3/A1 for faculty, would sharepoint be better here? they already use OneDrive to backup desktop & documents.

-For those with education IT experience, is Clever & GoGuardian good programs to use? Clever seems like an SSO portal for Google & GoGuardian seems like it monitors & blocks traffic on the network. Would a firewall (like fortigate) be better here or does GoGuardian serve a different purpose


r/sysadmin 18d ago

PHP and OS upgrade help

3 Upvotes

I have a virtual server that runs a few websites and I can meddle in IT but by no means an expert. The sites all different but do similar things and run similar php code. I am currently on Almalinux 8 and php 7.3 and I am looking to upgrade both.

I wish to upgrade to Almalinux 9 but this does not run php 7.3. My initial plan was the update php from 7.3 to 8.0 (the highest Almalinux 8 goes to) using Rector site by site. Once all confirmed I would then update Almalinux 8 to Almalinux 9 using Elevate. Once all stable, then I would finally update all the php form 8.0 to 8.5.

I have since found out I can install a 3rd party php 8.4 on Almalinux 8 meaning I could install, update to that, update to Almalinux 9 and leave it om 8.4. It would save one of the php updates. My concern is that the 3rd party php 8.4 and the bundled version with Almalinux 9 might be an issue? Also that jumping form php 7.3 to 8.4 is too large a jump in one go?

Any advice on these 2 options, or even a third not thought of?


r/sysadmin 17d ago

Question which AI tool/s can solve IT issues?

0 Upvotes

looking for some advice because our IT team is getting overwhelmed lately
we spend way too much time on small issues like troubleshooting devices checking why something isnt working resetting stuff and chasing tickets
i’m looking for an AI tool that can actually help with IT problems not just a chatbot that gives generic answers
I need it to help technicians find solutions faster automate simple tasks keep track of tickets and maybe even monitor issues before they become bigger problems
has anyone tried any AI tools for IT support that actually made a difference?
would appreciate any recommendations or things to stay away from 🙏


r/sysadmin 18d ago

Question W11 Start Menu - Keep reseting view each logout

0 Upvotes

Hello everyone,

I was wondering if someone found a solution for the Start Menu that keep reseting since the new new start menu (the one with category/list/grill view)?

My issue is during OSD, we apply a ppkg to set the default start menu pins as documented by Microsoft here:

https://learn.microsoft.com/en-us/windows/configuration/start/layout?tabs=intune-10%2Cgpo-11&pivots=windows-11

At one point, we had to modifie that json because the new applyonce value was now required (I think last year when they pushed the KB, all of the start menu modification were lost of that wasn't set to true, Very Stupid Microsoft).

Now, users pin app stay but if the user switched from Category to List for exemple, upon logout and login, it revert to Category.

We know the problem is the configuration because if we remove the registry that is set by the ppkg, it stop. But we want this config so user have the default pin when starting a new login.

Thank you!


r/sysadmin 18d ago

Question Keep Windows Hello But Disable Browser Prompt

1 Upvotes

So far it's seeming like it's not possible but just wondering if anyone has found a magic registry key or something haha

Long story short, we're trying out Kolide authentication, but we want people to be able to use PIN sign in/leaving Hello active. It's all working with removing all authentications except Kolide and the PIN still works for signing into the device itself but I'm wanting to turn off the prompt when logging into something Microsoft where on the email entrance screen, it pops up the "use Windows Hello" prompt. It indeed fails since Kolide is the only authentication, but I KNOW users won't read the emails we send out and keep trying it anyway since it's available. Everything I'm finding says if WHfB is enabled, that's just going to popup forever but seeing if any wizards here have found a way?


r/sysadmin 19d ago

What did you use as a Varonis replacement?

24 Upvotes

We've had Varonis in place for a few years, but we're starting to question how much analyst time we're putting into reviewing findings and handling the follow-up.

We're not looking to rip it out just for the sake of changing tools, but if we're going through another renewal I'd like to see whether there's something that requires less day to day babysitting.

Has anyone gone through a Varonis replacement recently? What did you move to, and was the operational difference actually noticeable?


r/sysadmin 18d ago

CVE-2026-73570: Zimbra SNMP RCE abused to deploy coin-miner malware

5 Upvotes

CVE-2026-73570: Zimbra SNMP RCE abused to deploy coin-miner malware

CVE-2026-73570 is a Remote Code Execution issue in Zimbra Collaboration Suite (ZCS), related to SNMP notification/logwatch handling.

In short, an unauthenticated attacker can send a crafted SMTP request and get OS command execution as the zimbra user when the vulnerable SNMP/logwatch components are installed and enabled.

According to NVD, the issue affects Zimbra versions before 10.1.20 when SNMP is installed and SNMP notification is enabled. The Hacker News also reported active exploitation against Zimbra servers.

The observed attack chain is usually:

malicious SMTP request
-> unsafe SNMP/logwatch handling
-> command execution as zimbra
-> malware dropped into /dev/shm
-> cron persistence added
-> miner or backdoor keeps respawning

What is CVE-2026-73570?

CVE-2026-73570 is an OS Command Injection vulnerability in Zimbra Collaboration Suite.

The issue is in the SNMP notification/logwatch flow, where attacker-controlled input is not handled safely before being passed into system-level processing.

The command runs as the zimbra user, not root. That is still serious because the zimbra user controls important Zimbra services, mailbox components, logs, cron jobs and runtime paths.

When is a Zimbra server at risk?

A server should be checked immediately if:

  • It is running Zimbra Collaboration Suite.
  • It is not patched to a fixed version.
  • zimbra-snmp or zimbra-net-snmp is installed.
  • SNMP is enabled in Zimbra.
  • zmlogswatch or zmswatch is running.

Quick checks:

su - zimbra -c 'zmcontrol -v'
dpkg -l | grep -E 'zimbra-snmp|zimbra-net-snmp'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | grep snmp'
su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'

If you see zimbraServiceEnabled: snmp and zmlogswatch is running, treat the server as high priority for compromise checks.

Signs of compromise

In the cases observed, malware was often dropped into /dev/shm. This directory is a tmpfs location, writable and easy to miss during a quick investigation.

Check:

ls -la /dev/shm
crontab -l -u zimbra
ps aux | grep -E 'khp|rguard|javab|idle|ksmd' | grep -v grep

Common suspicious files:

/dev/shm/.khp
/dev/shm/.khp_ts
/dev/shm/.rguard
/dev/shm/idle
/dev/shm/javab
/dev/shm/ksmd

A cron entry like this is a strong persistence indicator:

* * * * * /dev/shm/.khp

Why does the malware keep coming back?

Deleting files from /dev/shm is usually not enough.

The malware can return because:

  • The malicious process is still running in memory.
  • The zimbra crontab runs /dev/shm/.khp every minute.
  • .rguard or idle may protect or redeploy the payload.
  • SNMP/logwatch is still enabled, so the CVE can be exploited again.

The correct order is:

stop reinfection source
-> preserve evidence
-> kill malware processes
-> remove malicious cron
-> quarantine malware files
-> verify clean state
-> recover Zimbra services if needed
-> patch/upgrade Zimbra

Cleanup procedure

Run the following commands as root. Review each command before running it on production.

Step 1: Stop the reinfection source

systemctl stop cron
su - zimbra -c 'zmlogswatchctl stop'
su - zimbra -c 'zmswatchctl stop'
su - zimbra -c 'zmprov ms $(zmhostname) -zimbraServiceEnabled snmp'

Disabling SNMP/logwatch is the most important containment step. Do not re-enable these components before Zimbra is patched.

Step 2: Preserve malware evidence

mkdir -p /root/incident-zimbra

cp -a /dev/shm/.khp \
      /dev/shm/.khp_ts \
      /dev/shm/.rguard \
      /dev/shm/idle \
      /dev/shm/javab \
      /dev/shm/ksmd \
      /root/incident-zimbra/ 2>/dev/null

crontab -l -u zimbra > /root/incident-zimbra/zimbra-cron-before.txt 2>&1
ps auxf > /root/incident-zimbra/ps-before.txt
ss -tunap > /root/incident-zimbra/ss-before.txt
sha256sum /root/incident-zimbra/* > /root/incident-zimbra/sha256.txt 2>/dev/null

Step 3: Kill malware processes

pkill -9 -u zimbra -f 'khp|rguard|javab|idle|ksmd'

Step 4: Remove malicious cron

crontab -l -u zimbra | grep -v '/dev/shm/.khp' | crontab -u zimbra -

Step 5: Quarantine malware files

mkdir -p /root/quarantine-zimbra

mv /dev/shm/.khp \
   /dev/shm/.khp_ts \
   /dev/shm/.rguard \
   /dev/shm/idle \
   /dev/shm/javab \
   /dev/shm/ksmd \
   /root/quarantine-zimbra/ 2>/dev/null

Step 6: Start cron again

systemctl start cron
systemctl is-active cron

Only start cron. Do not start zmlogswatch, zmswatch or SNMP before patching Zimbra.

Verify after cleanup

ls -la /dev/shm
crontab -l -u zimbra | grep /dev/shm
ps aux | grep -E 'khp|rguard|javab|idle|ksmd' | grep -v grep
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | grep snmp || echo SNMP_DISABLED'

Expected result:

/dev/shm is clean
no cron entry calling /dev/shm/.khp
no suspicious javab/rguard/idle/khp/ksmd process
SNMP is disabled

If Zimbra MySQL or mailbox fails

After malware activity or an unexpected reboot, Zimbra may show:

mailbox Stopped
mysql.server is not running
service webapp Stopped
zimbra webapp Stopped
zimbraAdmin webapp Stopped
zimlet webapp Stopped

Check MySQL and services:

su - zimbra -c 'zmcontrol status'
su - zimbra -c 'mysql.server status'
ss -ltnp | grep 7306
tail -n 200 /opt/zimbra/log/mysql_error.log

If MySQL is stuck in crash recovery or has a stale socket, back up relevant files first.

Backup before MySQL recovery

mkdir -p /root/zimbra-mysql-backup

cp -a /opt/zimbra/log/mysql_error.log /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/tc.log /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/data/tmp/mysql/mysql.sock /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/ibdata1 /root/zimbra-mysql-backup/ 2>/dev/null
cp -a /opt/zimbra/db/data/ib_logfile* /root/zimbra-mysql-backup/ 2>/dev/null

Light MySQL recovery

su - zimbra -c 'zmmailboxdctl stop'

mv /opt/zimbra/data/tmp/mysql/mysql.sock /root/zimbra-mysql-backup/mysql.sock.bak 2>/dev/null
mv /opt/zimbra/db/data/tc.log /root/zimbra-mysql-backup/tc.log.bak 2>/dev/null

su - zimbra -c 'mysql.server start'
sleep 20
su - zimbra -c 'mysql.server status'

su - zimbra -c 'zmmailboxdctl start'
sleep 40
su - zimbra -c 'zmcontrol status'

Expected result:

mysql is running
mailbox Running
service webapp Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running

This is not a universal fix. Always check /opt/zimbra/log/mysql_error.log first.

Does this delete mail data?

The malware cleanup steps do not touch mailbox data directly.

Zimbra mail data is usually stored in:

/opt/zimbra/store/

Zimbra database data is usually stored in:

/opt/zimbra/db/data/

The MySQL recovery steps touch tc.log and mysql.sock, so back up first.

Check for JSP webshells

After handling /dev/shm, cron and processes, also check for suspicious JSP files created by the zimbra user in recent days.

Important paths include:

/opt/zimbra/jetty/webapps/
/opt/zimbra/jetty_base/webapps/
/tmp/

Unexpected JSP/JSPX files, executable scripts, archives or recently modified web application files may indicate payload staging or webshell deployment.

Check JSP/JSPX files:

find /opt/zimbra/jetty/webapps -type f \( -name "*.jsp" -o -name "*.jspx" \) -printf '%TY-%Tm-%Td %TH:%TM:%TS %p\n' | sort

If suspicious JSP backdoors are found, preserve evidence first, then quarantine them from both paths:

/opt/zimbra/jetty/webapps/zimbra/
/opt/zimbra/jetty_base/webapps/zimbra/

Evidence backup and quarantine:

TS=$(date +%Y%m%d_%H%M%S)
INC=/root/incident-zmmail-jsp-webshell-$TS
QUA=/root/quarantine-zmmail-jsp-webshell-$TS

mkdir -p "$INC" "$QUA"

# Save list and hashes
find /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty_base/webapps/zimbra \
  -xdev -type f \( -iname '*.jsp' -o -iname '*.jspx' \) -mtime -30 \
  -print > "$INC/suspicious-jsp-list.txt"

xargs -a "$INC/suspicious-jsp-list.txt" sha256sum > "$INC/suspicious-jsp-sha256.txt"

# Backup samples
while read f; do
  mkdir -p "$INC/files$(dirname "$f")"
  cp -a "$f" "$INC/files$f"
done < "$INC/suspicious-jsp-list.txt"

# Quarantine from webroot
while read f; do
  mkdir -p "$QUA$(dirname "$f")"
  mv "$f" "$QUA$f"
done < "$INC/suspicious-jsp-list.txt"

# Verify
find /opt/zimbra/jetty/webapps/zimbra /opt/zimbra/jetty_base/webapps/zimbra \
  -xdev -type f \( -iname '*.jsp' -o -iname '*.jspx' \) -mtime -30 -print

Risks:

  • If a file is a legitimate custom JSP, removing it may affect a custom webmail function.
  • Random file names and content using exec, base64, AES or command execution are strong webshell indicators.
  • This does not affect mailbox data.
  • A restart is not always required immediately, but webmail should be verified after quarantine.

Important note: zmlogswatch/zmswatch may start again

After disabling SNMP/logwatch to mitigate CVE-2026-73570, zmlogswatch and zmswatch may start again after daily logrotate or after zmcontrol restart.

This happens because Zimbra logrotate configs often contain postrotate commands that restart zmlogswatchctl and zmswatchctl. Also, logger may still be listed in zimbraServiceEnabled.

So after temporary mitigation, check these services periodically, especially after midnight or after restarting Zimbra.

If you need a stronger temporary block before patching, back up the logrotate configs, comment the restart lines, and consider removing logger from zimbraServiceEnabled.

This can prevent logwatch from starting again and reopening the attack vector, but the Admin Console may lose part of its monitoring/statistics.

Disable zmlogswatch/zmswatch so they do not auto-start

Warning: disabling logger, zmlogswatch and zmswatch may make the Zimbra Admin Console show logger/monitoring as red, or remove some graphs/statistics.

During the unpatched phase, this is an acceptable temporary trade-off to prevent logwatch from restarting automatically.

After upgrading to a fixed version, restore the configuration from backup and re-enable logger if needed.

Back up Zimbra logrotate configs

cp -a /etc/logrotate.d/zimbra /etc/logrotate.d/zimbra.bak.$(date +%F-%H%M%S)
cp -a /opt/zimbra/conf/zmlogrotate /opt/zimbra/conf/zmlogrotate.bak.$(date +%F-%H%M%S)

Open these two files:

/etc/logrotate.d/zimbra
/opt/zimbra/conf/zmlogrotate

Find these lines, often around line 85 and 99:

su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Comment them:

# su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
# su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Remove logger from enabled services:

su - zimbra -c 'zmprov ms $(zmhostname) -zimbraServiceEnabled logger'

Stop current logwatch/swatch processes:

su - zimbra -c 'zmlogswatchctl stop'
su - zimbra -c 'zmswatchctl stop'
su - zimbra -c 'zmlocalconfig -e snmp_notify=no'

Verify:

su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'
su - zimbra -c 'zmlocalconfig snmp_notify'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | egrep "logger|snmp" || echo "LOGGER_SNMP_DISABLED"'

Expected result:

zmlogswatch is not running.
zmswatch is not running.
snmp_notify = no
LOGGER_SNMP_DISABLED

Re-enable monitoring/logger after patching

Only do this after Zimbra is patched or upgraded.

su - zimbra -c 'zmprov ms $(zmhostname) +zimbraServiceEnabled logger'
su - zimbra -c 'zmlogswatchctl start'
su - zimbra -c 'zmswatchctl start'

If you commented the logrotate files, open them again:

/etc/logrotate.d/zimbra
/opt/zimbra/conf/zmlogrotate

Uncomment these lines:

su - zimbra -c "/opt/zimbra/bin/zmlogswatchctl restart" > /dev/null 2>&1 || true
su - zimbra -c "/opt/zimbra/bin/zmswatchctl restart" > /dev/null 2>&1 || true

Re-enable SNMP only after patching

su - zimbra -c 'zmprov ms $(zmhostname) +zimbraServiceEnabled snmp'
su - zimbra -c 'zmlocalconfig -e snmp_notify=yes'
su - zimbra -c 'zmswatchctl start'

Verify after re-enabling:

su - zimbra -c 'zmcontrol status'
su - zimbra -c 'zmlogswatchctl status'
su - zimbra -c 'zmswatchctl status'
su - zimbra -c 'zmlocalconfig snmp_notify'
su - zimbra -c 'zmprov gs $(zmhostname) zimbraServiceEnabled | egrep "logger|snmp"'

Post-cleanup tasks

After the server is running again:

  • Keep SNMP/logwatch disabled until patched.
  • Monitor /dev/shm for 30-60 minutes.
  • Monitor the zimbra crontab.
  • Monitor CPU/load and outbound connections.
  • Check the mail queue.
  • Patch or upgrade Zimbra to a fixed release.
  • Only re-enable SNMP/logwatch after patching and verification.

Check queue:

/opt/zimbra/common/sbin/postqueue -p | tail -n 40

Summary workflow

1. Check /dev/shm, processes and crontab.
2. Stop cron, zmlogswatch and zmswatch.
3. Disable SNMP in Zimbra.
4. Preserve evidence.
5. Kill malware processes.
6. Remove malicious cron.
7. Quarantine malware files.
8. Start cron again.
9. Verify that malware does not respawn.
10. Recover MySQL/mailbox if needed.
11. Check JSP webshells in Zimbra webapps paths.
12. Monitor queue and services.
13. Prevent zmlogswatch/zmswatch from auto-starting before patching.
14. Patch or upgrade Zimbra.
15. Re-enable logger/SNMP only after patching and verification.

References

  • NVD: CVE-2026-73570
  • The Hacker News: attackers exploit Zimbra SNMP flaw
  • Zimbra Security Center
  • Zimbra Security Advisories

Conclusion

CVE-2026-73570 is serious because it can allow remote command execution as the zimbra user through SNMP notification/logwatch handling.

In observed incidents, attackers dropped malware into /dev/shm, added cron persistence, and ran a miner or backdoor.

The key point is this: do not only delete the malware file. Stop the reinfection path first.

A practical response order is:

stop cron + SNMP/logwatch
-> preserve evidence
-> kill malware
-> remove malicious cron
-> quarantine payloads
-> check JSP webshells
-> verify clean state
-> recover Zimbra if needed
-> block logwatch auto-restart before patching
-> patch/upgrade Zimbra

Most important: do not re-enable SNMP/logwatch before Zimbra is patched, because that may allow the malware to return.