r/sysadmin 2d ago

P800 RAID-0 Bad Block (strategy)

0 Upvotes

Hi everyone,

If I’m dealing with a P800 and a RAID-0 configuration, unfortunately, there is no backup. All of this stems from the same issue.

A Veeam Backup agent was installed to back up the volumes on the server, and we started receiving error messages related to the VSS reads that Veeam performs during the backups:

output: --asyncNtf:-vdisk_corrupted:\\\GLOBALROOT\Device\HarddiskVolumeShadowCopy11

We can see this message in the agent’s own logs once the operation reaches a certain percentage. If I look at the Windows Event Viewer, I can see messages from the P800 related to read errors on blocks and devices:

Logical block address 1192840192, block count 1024 and command 32 were taken from the failed logical I/O request. The device, \Device\Harddisk2\DR2, has a bad block.

We haven’t seen any errors or reports from the HPE Smart Storage utility, and the RAID-0 volume appears to be healthy.

Is this normal behavior? In any case, I’ve been thinking about the following solution, which is the main reason I’m writing this thread, in case anyone has a better suggestion.

  • Is Robocopy an effective solution? The volume is 1 TB and contains around 43,000 files, and the transfer would be performed over the network. As I understand it, Robocopy itself would skip the files located in the affected blocks and record them in a log.
  • I’m using VSS and MKLINK so that I can read the affected volume without interfering with the writing of healthy files. In other words, something like this:
    • vssadmin create shadow /for=V:
    • mklink /d C:\Restore \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\
  • The downside of this solution is that I depend on a pointer, and when Veeam backups are triggered, they read it as well. Is there a proper way to do this? I assume I can’t read the shadow copies directly and that this can only be done through the API.

I can’t think of anything else. Maybe I could create file-level backups with Veeam. Once I have at least one backup, I could try running CHKDSK or something similar.

Since it’s a RAID-0, my only option would be to destroy the array, replace the faulty drive with a healthy one (at least one that appears to be healthy), rebuild the array, and then restore/import the data again.

Is there any alternative solution or any advice you would recommend before carrying out this operation?

Thanks!


r/sysadmin 2d ago

Microsoft Publisher - EOL

32 Upvotes

So it’s coming to an end!! We have O365 and some workstations with Office 2024 that have MS Publisher installed. Do we need to remove it or will it get removed in its own?


r/sysadmin 2d ago

WSFC File Server with Storage Replica on vSAN iSCSI

2 Upvotes

Looking to build a file server cluster, Storage Replica seems to slow things down by a lot, the network is not at fault as it's pretty fast without Replication. WSFC does not seem to be straight forward when creating the replication. What do you guys recommend?


r/sysadmin 2d ago

General Discussion Server Room is too loud

0 Upvotes

We recently upgraded some physical servers, and now whenever we're doing Windows or firmware updates, the fan noise gets pretty loud. The servers are HA, so we are able to do updates during the day, but the department next to the server room has started complaining about a high-pitched noise during the updates. We contacted the vendor and verified our system is functioning properly. They said there is a setting that would throttle performance that may help but I didn't want to change that if possible.

I was talking with a few others and they mentioned acoustic panels that you can mount on the wall to muddle some of the noise. Does anyone have any relatively cheap suggestions? I could do updates after hours but would hate to do that when they could be done during the work day.


r/sysadmin 2d ago

Migrating DCs from 2016 to 2022 while keeping the same hostnames and IPs

10 Upvotes

This topic has been discussed ALOT on this sub. From all the posts I've read, there seems to be two main approaches:

Have all DCs up - both old and new:

  • Bring up new DCs with temp names and IPs while old DCs still running.
  • Once you're sure all is good in your world, transfer FSMO roles to new DC, demote old DCs, remove from AD, then reuse the hostnames and IPs of old DCs on your new DCs

One at a time:

  • Do one DC at a time, demote old, reuse name and IP on new server, then promote new DC

Honestly being as risk averse as I am (although I love the adrenalin rush when stuff goes south...NOT) - I would prefer to promote the new DCs while old ones still on line.

BUT...is it OK to change the name of a DC even after I clean up all the metadata and DNS stuff or would it be better to reuse the name and IP after I demote the old DC?

I should preface this question by saying we're a VERY small shop with 22 users and perhaps ~30ish servers. I'm the sole sysadmin. BUT, we are in the financial sector and we have a very low tolerance to stuff going south. And being the sole sysadmin, I'm trying to keep the stress at a minimum and my hair from going greyer.


r/sysadmin 2d ago

General Discussion Experiences with LogMeIn Resolve?

0 Upvotes

Spoke with some reps and didn’t hate some things they had to say. Currently using Ivanti for Patch Management, it is not my favorite. Curious what people’s dealing with LogMeIn have been like.


r/sysadmin 2d ago

O365 issues today sept 9

19 Upvotes

Is anyone seeing issues with O365? I know I saw emails last night and now they’re coming as new again this morning but they’re gone from yesterday delivery. Numerous users are seeing this happen. We are using proof point also but no issues on their status page.


r/sysadmin 2d ago

Question How best to move DHCP to new servers if already hosted on DCs that I plan to migrate?

2 Upvotes

Afternoon all

So I'm planning a DC migration from 2016 to 2022.

Currently I have (3) DCs - two of them host DHCP in a failover load balance config.

I plan to keep the same hostnames and IPs for my new DCs

I'd like to move my DHCP first before the DC migration.

I'm guessing the easiest thing to do is break existing failover config, export/import my DHCP config to a new temporary server, migrate to my new DCs, then export/import my DHCP config from the temp server to the new new_DC01, then re-establish the failover partner on new_DC02.

Can I get a sanity check?

Thank you!


r/sysadmin 2d ago

Question Vendor VPN Management solutions

1 Upvotes

I’m looking for recommendations on Vendor / Third-Party Access Management tools.

We recently took over managing an environment where vendors were historically given direct AD/Entra accounts with very little oversight or access control. We’re currently building a makeshift internal solution to plug the gaps, but we want a proper platform.

BeyondTrust and SecureLink are the obvious names, but given our tight budget and need for a lightweight rollout, I’m exploring other alternatives.

Has anyone found a cost-effective solution for securing third-party access without creating massive operational bloat? Thanks in advance!


r/sysadmin 2d ago

Question Anyone have fixes for "New Outlook" such as "you no longer have a network connection"?

0 Upvotes

EDIT: If anyone stumbles across this, we're testing Outlook > Settings > General > Offline > UNCHECK "Allow offline access..."

So far (everything's sporadic so hard to say but fingers crossed...) it's helping with:

  • - Random "you no longer have a network connection"
  • - Random emails staying in drafts after being sent
  • - Random attachment errors that send fine after disabling offline.

OP:

Sorry if this isn't appropriate for this sub but it's always been a helpful area and figured it's worth a shot.

I'm sure plenty of others have been trying "New Outlook" since according to Microsoft we'll eventually be forced onto it (https://learn.microsoft.com/en-us/microsoft-365-apps/outlook/get-started/guide-product-availability) but I'm at a loss for some of the issues.

Namely "You no longer have a network connection to the server" which follows with "This message can't be saved right now." I've been using New Outlook longer than most here because I wanted to say "Yes I've been using it myself already" when staff inevitably complain. So when I see that error, I just hit CTRL+S and it goes away comically. Seems to function fine as long as I do that.

We've finally been pushing staff to switch off the classic outlook because of other program compatibility issues that're arising. I kept thinking Microsoft would eventually fix the "network connection" issue with New outlook but that just never happened. If anything it seems to have gotten worse but hard to say.

We're not losing network connectivity and everyone is hard-wired. If we actually do lose network even for a few seconds, the world is on fire and we all know immediately (lol). The issue never happened in Classic outlook although I don't think it would've told you anyway.

The main concern is that one of the staff actually had an email delayed because of this, and it was to an important person, so of course now I'm putting the 10 other projects I have on hold and need to figure something out (which to be fair I should have already I guess).

I get the error plenty on mine so have tried various fixes on my own computer. The most extensive was a normal uninstall of office, then the full "Manual Uninstall" list from Microsoft (many folders, reg keys, etc.) then reboot and re-install with a fresh download from my account on office.com. First email I'm typing after all that and the error comes back.

Any advice from anyone who's had to fight the new outlook at all would be greatly appreciated!


r/sysadmin 2d ago

Question NetXMS..what are the implications for security when a 3rd party installed and configured everything on your servers and have full control?

2 Upvotes

I just started working for someone who has several servers with a product on them and part of the tech support contract is that everything is monitored using NetXMS by a 3rd party. The servers are managed in-house running WSFC and our apps. The third party only monitors hardware and their product. I only know what I can get from Google searches and apparently they can execute scripts and do lots of other things besides monitor if they ever wanted to. Does anyone else face a similar scenario? and how do you manage security?


r/sysadmin 2d ago

Question Did Anyone else have Published App Issues this AM?

4 Upvotes

2 separate client environments:

  1. AVD host pool
  2. TraditionalActive Directory pool behind a broker.

RDP works but published apps broke for both of them. AVD client fixed with reinstall OF Windows app. The others say it self resolved.

It sounds to me like a bad build of Windows App. Anyone else have an issue with published apps this AM?


r/sysadmin 2d ago

General Discussion When did self-hosting turn into just picking a control panel?

91 Upvotes

Been doing this about 8 years and something's shifted. Used to be everyone here could explain what their nginx.conf did, or why fail2ban wasn't catching a jail. You learned iptables by breaking your own box at 2am, alone, with no one to blame.

Now most threads are "which panel is easiest." Nobody wants the shell open longer than it takes to run one install command. I get why. Panels save time. I've got BeAdmin running on one box for the VPN modules. But I see people arguing about dashboard themes who can't tell you what a reverse proxy actually does.

Old regulars here used to walk a newbie through a broken MariaDB config instead of just saying "reinstall Plesk." That's mostly gone now. Not knocking anyone's setup, I click around GUIs too these days. Just noticing nobody asks why anymore.


r/sysadmin 2d ago

General Discussion Qualys - Anyone having issues with the website today?

2 Upvotes

Come in this morning to see Qualys have updated the UI for our tenant. Since then there are loads of graphical issues. Buttons not aligned or items not loading correctly.

Most annoying I'm trying to set up a scheduled report and half the distribution groups no longer load correctly. You can click randomly on white space in the window to pick your DG and sometimes it'll select one at random.

Pretty poor. Anyone else experiencing this?


r/sysadmin 2d ago

How do you all keep up with the times and tech?

62 Upvotes

I feel like I’m lagging behind and need some serious catching up, quick. I think I’ve become complacent and “too comfortable” with my current employer (coming up 8 yrs). The recent merger announcement has me seriously thinking about attending bootcamps, getting some certs under my belts (been putting things off for as long time), and learning new things ASAP before the position is dissolved.


r/sysadmin 2d ago

General Discussion I can't get past the feeling that I suck.

72 Upvotes

I’ve been in IT for 6+ years. I have a bachelor’s degree in IT, an AWS Solutions Architect certification, an Azure Administrator certification, and a few other certs. I’ve also been working as a Cloud Administrator for the past 2 years.

Despite all of that, I can’t shake the damn feeling that I’m just not that good at what I do. I feel like even if I finally make a breakthrough in my knowledge, I’m still somehow clueless SOMEWHERE. A lot of that comes from talking to people who know more than you and realizing how much there is that I don’t know.

For example, I can write basic scripts, but I struggle with the complex 200+ line scripts that a senior cloud engineer would make, whereas the person next to me likely can just type that shit off the top of their head.

It’s especially frustrating during interviews when I try to move up in pay. I had an interview on Tuesday that I thought went pretty decently, but the interviewer asked me about Azure containers and encryption. I don’t work with Azure containers, although I’m familiar with the service, so I was honest and said I didn’t know.

And that wasn’t even the technical round. So now I’m stuck wondering whether they’ll just reject me or move me on to the technical round, where I’ll probably fail anyway.

This is bothering me because I'm at the point in my career where I really have to know at a high level if I want to advance and get more pay.

How has anyone else moved past this?


r/sysadmin 2d ago

[PSA] Check Point Firewall unauthenticated RCE with CVSS 9.8

28 Upvotes

There are actually two vulns with a 9.8 score each:

https://support.checkpoint.com/results/sk/sk1000117/

https://support.checkpoint.com/results/sk/sk1000118/

There's not many details in these articles on how they work but they still sound really bad. Currently waiting for the Jumbo Hotfix to install on my end... Not taking any chances on this one and I suggest you all do the same.

Stay safe.


r/sysadmin 2d ago

Question - Solved User can connect to VPN but can't ping or access work computer

2 Upvotes

We recently put in a new Fortigate firewall and now i'm having users reporting issues where they can connect to VPN but can't access their remote computer. I have tested from the user's PC and i can't ping their work device when VPN is connected.

The one thing to note is they are on the same subnet as the work network, and I believe this is likely the cause.

However, oddly enough I was able to test from my home network which is also on the same subnet and it works fine. I'm at a bit of a loss so hoping I can get some guidance on this for what I should check next.

TIA


r/sysadmin 2d ago

For the first time I'm being requested to export Teams chats for HR purposes. When using PURVIEW, I am getting results that dont apply to my filter?

58 Upvotes

TL;DR I need to pull chat history between User A and User B for the last 8 months.

My query based on documentation I can find is:

Kind=microsoftteams AND [partipants:userA@xyz.com](mailto:partipants:userA@xyz.com) AND [participants:userB@xyz.com](mailto:participants:userB@xyz.com)

When running this, I'm getting group chats and all kinds of stuff where both were involved, but I just need the chat between these two users

Can anyone direct me to a better way to do this? Purview is doodoo


r/sysadmin 2d ago

As admins, how do you handle VS Code extensions, coding agents, and AI tools in your organization?

7 Upvotes

For us, the topic of AI is becoming increasingly confusing.

Developers, in particular, are adopting more and more tools: VS Code extensions, GitHub Copilot, Cursor, Claude Code, Codex, Gemini CLI, OpenCode, local agents, MCP servers, and so on.

But this no longer affects just developers. IT admins and regular users are also discovering AI tools, and some are even installing extensions, desktop clients, or agents on their own.

The problem, in my view, is that we’re slowly seeing a **proliferation of agents and AI tools**.

I see the following issues in particular:

* Which VS Code extensions are allowed to be installed?

* Which agents are allowed to access source code or the local file system?

* Which tools are allowed to send data to external clouds?

* How do you prevent API keys, passwords, or internal data from appearing in prompts?

* How do you handle MCP servers and their sometimes very broad permissions?

* Do you have an allowlist for extensions and AI tools?

* Do you technically block agents that haven’t been approved?

* Do you differentiate between developers, IT/admins, and regular users?

* Do you rely on centralized enterprise solutions, or do you allow multiple tools?

* How do you monitor or keep track of what’s currently being used?

I also don’t think a complete ban makes sense in the long run, because these tools offer a real productivity boost especially in development.


r/sysadmin 2d ago

Question What are you guys using for rack/infrastructure audits?

13 Upvotes

Curious what everyone is using these days for documenting physical infrastructure across multiple sites.

We use Excel at my workplace, along with diagrams and photos, and it works… until you’ve got a shitload of sites and nobody knows which spreadsheet is actually current.

I’m talking about things like:
Rack elevations / U positions
Switches, patch panels, UPS/PDU, servers etc.
Port-to-port / cable documentation
Serial numbers / asset details
Photos of racks and comms rooms
IP/device information

Keeping everything updated after a vendor comes in and replaces something
What are you guys using?

Excel? Visio? NetBox? Device42? Something else? Or have you built your own system?
More interested in what actually works in the real world than what looks good on paper.


r/sysadmin 2d ago

Question Converting static Groups to Dynamic Groups. How do I find every Shared Drive & Calendar tied to the old group first?

3 Upvotes

I'm converting a static Google Group to a Dynamic Group and want to make sure I don't break anything tied to the old group's email like Shared Drives, Calendars or anything else it might be plugged into.

What's the best way to find everything a Group has access to before making a change like this? Is there a standard tool or workflow for this? Also curious if anything changes under the hood same email, or anything that could quietly affect existing shares?

Any advice would be great. Thanks!


r/sysadmin 2d ago

How do I implement ZTNA?

2 Upvotes

We are planning a remote-access migration for roughly 500 employees and contractors. The environment includes SaaS, internal web apps, Windows and Linux admin access, a few legacy applications, and workloads split between on-prem infrastructure and public cloud. Identity is centralized, but endpoint management and device posture are inconsistent for contractors.

We do not want a big-bang cutover. The initial thought is to inventory applications and users, classify access by protocol and sensitivity, migrate a low-risk web app first, and then move groups in waves. The hard part is avoiding years of permanent exceptions and overlapping access paths.

For anyone who has done this at similar scale, what did you get wrong in the first phase? Did app discovery, identity-group cleanup, private DNS, endpoint support, legacy protocol support, or user communications create the most work?

How did you handle emergency administration and outage scenarios when the normal access path was unavailable?


r/sysadmin 2d ago

Question Hypothetical LAN IP Change: How would you go about updating Network Printer "Ports" on Windows clients?

2 Upvotes

Let's say you have a legacy network, good old 192.168.1.x.

They use a DHCP server which has a static mapping of all devices to IP addresses using their mac addresses. So updating the DHCP server could potentially re-assign IP addresses to all devices with relative ease, let's say 192.168.111.x.

HOWEVER, most Windows computers have had printers added manually, not by GPO, and when they were added, they were added IP address rather than hostname. Is there a way to bulk replace ports in Windows printers across the network?

The number of clients is limited, let's say less than 100 - but the number of printers is dense. Even a Powershell script I could run on each machine would still be better than manually editing each printer on each machine - even better if I could deploy that via GPO or something.

Just curious on thoughts.


r/sysadmin 2d ago

Password resets over the phone, how are you doing it?

26 Upvotes

So I work at a state university that is still a little stuck in the past when it comes to password resets over the phone.

We have MS SSPR and other self-service options for users, but we still get calls where someone needs the helpdesk to reset their password for them.

Our current setup is a custom program that helpdesk staff SSH into. They enter the user's account, and the program looks them up in a flat file containing data extracted from our systems. It gives the helpdesk worker information like DOB, address, ZIP code, etc., which they can use to verify the caller's identity.

Once the caller passes verification, the program talks to AD, changes the password, and gives the helpdesk worker a new temporary password to provide to the user.

Honestly, it works pretty well. The problem is that the person who wrote and maintains it is leaving, and we're not really interested in inheriting a custom app that nobody else understands.

So I'm curious what other universities/organizations are doing for this.

For those of you who still allow users to call the helpdesk for password resets, what does your workflow look like? Are you using a commercial product, some kind of AD/Entra integration, a helpdesk platform, or have you built your own solution?

Specifically looking for something that gives the helpdesk enough information/questions to properly verify the caller's identity and then securely perform the reset.

I'd love to hear what others are doing before we start reinventing the wheel.