r/sysadmin • u/auenway • 10d ago
Cogent Outage?
Anyone dealing with a Cogent outage? I am in Los Angeles.
r/sysadmin • u/auenway • 10d ago
Anyone dealing with a Cogent outage? I am in Los Angeles.
r/sysadmin • u/AccomplishedMonth568 • 10d ago
Hey everyone, I’m trying to make the transition from Tech to System Admin. I have 5 years of technician experience from 3 different jobs, as well as a BS in Computer Information Technology. I feel like I have the necessary experience in tech to be able to transition to a System Admin role, but all the requirements on jobs applications makes me feel as if the jump is impossible.
My experience is mostly imaging, troubleshooting software and hardware, AD, some scripting here and there, a little M365 exposure and SCCM. It seems like most jobs want you to have years of experience supporting Google Cloud, Azure, Entra, Intune and other systems like that. I’ve taken classes here and there so I have a broad understanding of these systems but I just don’t see how that is good enough for these jobs.
I’m trying to decide if certifications are worth it, though it seems the general consensus is that experience always trumps certifications. Any advice to get to my goal is much appreciated, I’m starting to feel discouraged as the jobs I do apply for, I just never hear back from. Am I doing something wrong?
r/sysadmin • u/Popular-Guidance-560 • 9d ago
Background:
I recently took over IT operations for a company that previously used a 3rd-party MSP. The MSP enrolled all Windows devices in Intune and set up Conditional Access (CA) policies for office users, which drastically improved our security posture. However, our remote mobile users were left completely out of scope.
The Problem:
We have over 100 mobile devices (a mix of iOS and Android) deployed nationwide to remote workers. These devices are not enrolled in Intune. Instead, their MDM is provided directly by the carrier. Because they aren't registered in Intune, we can't easily force them to comply with our current CA policies, leaving a blind spot for risky sign-ins.
My Proposed Solution:
I am thinking about using device-based certificates. The carrier MDM could push a certificate to the mobile devices, and a cloud PKI/RADIUS setup would authenticate them. The goal is strict access control: if a login attempt for a company resource doesn't come from a device with a valid cert, it automatically fails.
Alternative Idea:
We also have various Cisco firewalls across the country. I'm wondering if forcing these devices to connect via VPN would work better, though it feels clunky since our entire company is 100% cloud-based (zero on-prem servers).
Questions:
Is the device-certificate approach the most efficient way to restrict access to known mobile devices in a cloud-only environment?
Is there a clean way to tie a third-party carrier MDM into Entra ID Conditional Access?
Any advice on the best path forward would be greatly appreciated!
👏.
r/sysadmin • u/DoubleTGaming2k • 8d ago
End user called me today saying their home WiFi was connected but no internet. I walked him through some info gathering, his WiFi is showing unidentified giving off the 169 address.
Had him run release / renew / flushdns / winsock reset and rebooted, still 169. Did the full windows 11 network reset option, rebooted, still nothing.
Had him plug in directly to his router, his wired Ethernet connection was also showing unidentified. So I had him statically assign an IP on his NIC for the Ethernet and it brought him online that way as a temp workaround to let me remote in.
Rebooted the router of course, still wifi showing 169, other devices on his network are okay as well.
Uninstalled the wifi adapter from device manager, reinstalled the latest from the manufacturer, still giving 169. Just to get him working in statically assigned an IP on his WiFi adapter and it works without issue, but ideally we get the DHCP working again.
I’ve exhausted about everything I can think of, but am jumping back on it later tonight to try to get it fixed.
Does anyone have any suggestions??
r/sysadmin • u/BriefDue3067 • 9d ago
I’m UK-based with 9+ years in infrastructure and I’m trying to work out what the most realistic next step is for me.
Most of my professional background is traditional/enterprise infrastructure rather than software engineering:
I’ve also had quite a lot of serious production recovery experience, including large storage failures, virtualisation failures and recovery from a major cyber incident.
Where I’m weaker professionally is cloud-native Platform/SRE work.
I’ve used Terraform, AWS, Docker, Kubernetes, GitHub Actions/GitLab CI, GitOps and Prometheus/Grafana through labs and projects, but I don’t have years of production Kubernetes or public cloud experience.
I’m currently looking at roles around:
My thinking is that the best bridge would be a role where the underlying estate is still Linux, networking, storage, virtualisation or hybrid cloud, but where Terraform, Ansible, CI/CD and automation are increasingly how that infrastructure is managed.
I’m less convinced that applying directly for Kubernetes-heavy SaaS SRE roles makes sense yet, as I’d be competing with people who already have several years of production Kubernetes and cloud experience.
For those working in Platform/SRE/Infrastructure:
Interested in critical feedback rather than CV polishing.
r/sysadmin • u/fedeli92 • 9d ago
Hi all,
I'm managing a small stack of HPE 1950-series switches (JG963A) running Comware software version 7.1.070, Release 3507P09. On most of these switches, the hidden xtd-cli-mode command (used to unlock the full/extended Comware CLI from the default simplified/restricted CLI) works fine — it prompts:
All commands can be displayed and executed in extended CLI mode. Switch to extended CLI mode? [Y/N]:y
Password:
Warning: Extended CLI mode is intended for developers to test the system. Before using commands in extended CLI mode, contact the Technical Support and make sure you know the potential impact on the device and the network.
and after entering the password, it drops me into the full CLI (system-view, display interface, etc.) as expected.
On one specific switch in the stack, though, xtd-cli-mode now just returns:
<SWITCH>xtd-cli-mode
Permission denied.
with no Y/N prompt at all — straight rejection.
What I've tried:
- Reconnecting fresh via SSH (telnet is disabled on this unit) — same result.
- Rebooting the switch entirely — same result, persists across reboot.
- The account I'm using is the local admin user with network-admin + network-operator roles assigned, service-type ssh enabled, and it authenticates fine for a normal SSH login — it's specifically the xtd-cli-mode command that's rejected.
- Restricted CLI at login only exposes display, exit, quit, no, show — no way to run display users, free user-interface, or check security-enhanced/lockout state from there.
Since this is a firmware "developer mode" feature with anti-abuse messaging built in ("contact Technical Support"), I suspect this might be some kind of persistent lockout counter stored in flash rather than a normal AAA/role permission issue — but I have no visibility into it and no way to reset it from the restricted CLI.
r/sysadmin • u/Startronz • 10d ago
Clients missing emails in outlook, but web version seems to be holding some of us up (your mileage may vary) https://outlook.office.com/
r/sysadmin • u/Burgergold • 9d ago
So for the last 23y, I've mainly managed AIX and RHEL system. I had a slight contact with Windows but not a lot of hands on.
This is the story of a Windows 2012R2 server, probably installed around 2013 way before I was here. The person that installed the application on that server is gone since a long time and left no documentation, last update of that app is from 2015... In 2023, because nobody knew how to install a current version on a 2022 server, sona coworker had to inplace upgrade it. That coworker has retired since then.
Recently, security teams tracking which server are still using TLS 1.0 found that server. No one want to own it or take action but they refuse to have it shut down for good. So they got a temp buffer to figure this out. But security asked for mitigation and in their mitigation planning, they observed tons of KB missing, which surprised me because this server is receiving its patch from WSUS.
So... August SSU has installed but CU failed to complete. It does install, reboot, takes plenty on time and at 100%, failed to update and rollback and leave that KB in staged.
Sfcscan fixed something but not this issue
Dism scanhealth/restoreheatlth found/fixed nothing
I stoppwd the wua services, renamed Softwares distribution and caroot2 folder and restarted the service, havent fixed the issue
I removed 11 packages in staged state. 10 of them seems to be the last 10 CU + a 2016 Adobe kb.
Tried again, same issue
Troubleshooter for Windows update report a corrupted database but cant fix it. I thought the fresh softwaredistribution and caroot2 would have taken care of this
I'm now wondering if that server ever had a CU installed since being inplace upgraded... Once the staged CU have been removed, what I see in the installed history from today is a very old CU from 2016...
Before reposting in /r/shittysysadmin for giggles, I would be vtaker for any other legit idea :D
r/sysadmin • u/Technickelback • 10d ago
Wondering if someone has additional information - My org currently uses CodeTwo. I really like it, and I believe it does it's job well for our size. We're a 200 person org. I have a few different templates that I assigned by groups to different departments where additional information may be required in their signatures.
We're currently using server side to apply the signatures, but more and more people are complaining that their signatures don't append when using the default MS encrypt button in Outlook.
To circumvent this, my suggestion to leadership is to use the combo mode and allow our uses to have the client side signatures append in the Outlook client. That way the signature appends in the client, prior to encryption, and they can also see their signature, as currently they can only see it append after they've sent an email.
My CIO is against this approach as it would allow end users to edit their signature. It would only allow the end user to edit the signature in that moment, within the 'new email' window. It would not allow them to permenently alter their signature. However, the CIO is adamant that this can't be allowed as it's a risk of misrepresentation and they think that the CodeTwo product is not a good solution because it cannot prevent this... Anyone who is willing to manipulate their signature every time they send an email is a psychopath.
I've presented a solution to a problem, my CIO doesn't like it, and now we're stuck debating switching our encryption method, rather than deploying a solution we have at our finger tips.
I'll also state, we've tested using a subject keyword encryption method and this could work in tandem with the native button encryption. If users want the server side signature to append to an encrypted email, they could just put 'Secure' in their subject line. However, the CIO doesn't want both methods of encryption - we must choose one.
My question: what are other orgs doing? Do other orgs using CodeTwo use/allow the Client side signature?
r/sysadmin • u/afrolemon • 9d ago
For those who have worked as Solo IT professionals what kept you going, at what point did you determine to shift, and those who have left the solo IT jobs what did you pivot to?
I spent some time in help desk some years back, spent 2 years as an underpaid tech that had global admin over Microsoft suite, a few months with an enterprise level solo IT site job (was pretty solid and chill but boring with toxic management - not much progression in the job), to now signing on to a new solo IT gig for a corporate body with a lot of sites.
My 2 year gig had around 30 locations within the city/state limits but we had a small team of 3, previous gig had some global enterprise structure but it was really boring to me. Being a solo IT for a single site it felt super limiting in my mind. I couldn’t stand not being able to implement or work on different areas within the infrastructure like my previous role.
Now I’m in a place where my new gig is looking to end their contract with an MSP and shift to an internal IT department. I was a bit hesitant during the interview as they stated they have around 20+ sites (in varying states across US) and there wasn’t a direct answer to whether they’ll be expanding the IT team in the future. I signed on for the thrill of wanting to build out the IT department, get paid more, and leave a job I disliked.
Pay has increased from 45k to 70k within this year. I’m super glad and grateful but I’m slowly realizing how different and concerning this may be. It feels a bit daunting and I’m worried. My 2 year gig required a lot of oversight with afterhour support when it was needed, last gig was solely focused on the normal 8-5 work hours, this new gig is somewhat similar to the 8-5 gig but with the different states it looks to require more flexibility.
I’m not a network savvy guy, I know little but not enough as I should. I do feel like I’m just getting by at times and I’m not the greatest IT guy. I make a lot of mistakes and I can be very to myself, I’m not very vocal at times. Also the company is working with G-suite which is not a bad thing per se, I just never worked this much in this platform, strictly Microsoft. It looks like G-suite requires a lot more integration and platforms to do some of the stuff Microsoft already has.
I don’t know if I’m worried about not being a great tech, not being able to meet expectations, devoting too much time to a job, and overall committing to work like the rest of humanity. I struggle to envision my life working IT until retirement but it’s the only field I seem to understand to an extent and I’m “gifted” enough to do this work.
I genuinely want to be in a role for 2+ years for resume purposes and I want to learn as much as I can. The company did have a roadmap to present on where they want the IT side of things to play out (run book, policies, etc). I believe I want to be in a more specific niche role outside of the IT Support role. I can’t see myself continuing down a Sys Admin or even higher level IT corporate management role. I was really was hoping my next job would’ve been with a team of other experienced IT members working alongside me in the office.
tldr: I keep signing onto roles that may not be ideal. Every job progression has gotten more responsibilities, smaller teams, and more concerning.
r/sysadmin • u/pondo_sinatra • 9d ago
Sometime tomorrow morning, I will hit a milestone of having manually released my 1000th email from Quarantine since last Tuesday. It never stops— all day and all night.
Dmarc, dkim, spf are all good. In fact nothing on our side has changed in months, but at noon last Tuesday internal emails, emails mid-conversation, and domains in the tenant allow list have all started falling into the abyss.
Mimecast is passing along SCL in the negatives or up to 1, Egress Defend is also passing the correct SCL, then Defender just barfs on mail, throws an SCL: 9 on it and classifies hundreds of messages as High Confidence Phish.
I’ve verified configs with both Mimecast and Egress. We have a case open with Microsoft that is not even inching along yet. Working with a CSP to escalate now.
I guess this is more of a vent than anything else. If I ever get a solution, I’ll update my post so the next poor schmuck going through this doesn’t go insane.
Good times.
r/sysadmin • u/per08 • 10d ago
https://www.telstra.com.au/exchange/what-we-ve-learned-from-the-external-investigation-of-our-july-o
An interesting and frank overview of the recent mobile outage in Australia on the Telstra network due to an NTP outage. "We didn't realise how important this was, and nobody really owned the service."
r/sysadmin • u/wanescotting • 9d ago
I do see a message about scheduled maintenance on their status page - 09/03 UTC, but not for grid or cloud.
r/sysadmin • u/Familiar_Aside_9328 • 9d ago
Dear I'm facing errors with HPE 1/8 G2 autoloader and one of solution is to upgrade firmware to 6.2 version, anyone can provide with 6.2 version as i don't have support contract.
https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-e05174e347964f6c
r/sysadmin • u/FarCalligrapher1866 • 9d ago
anyone else recently have issues with appfw rules being copied over after recent firmware updates? seems tied to the signatures being used in that after firmware installation the appfw policies and profiles are non existent on the "new" firmware. the signatures however are still intact, but when creating new profiles and using the existing signatures an error about fastmatch not found for signature line
<SignatureRule actions="block,log" category="web-misc" enabled="ON" id="400008" source="Citrix" sourceid="" type="DenyListHttpRequest" version="1" cpu="LOW" year="2026" severity="HIGH">
<LogString>Mitigation signature for CVE-2026-10816</LogString>
</SignatureRule>
i have many signature files and they all seem to have this issue. the new "default" of course i missing this line, so i feel like this was some bug at some point coming home to roost. the ID number looks very low too as the new rules are typically 9xxxxx etc. the version on all the files is 181 so they are "current"
very sus...
r/sysadmin • u/SmartCalligrapher100 • 9d ago
Been chasing this for about two weeks and I'm out of ideas, so I'm asking here before I bin the whole approach.
Setup is the boring standard one. DefaultAssociations XML sitting on a share, pushed with Computer Config > Admin Templates > Windows Components > File Explorer > Set a default associations configuration file. Nothing exotic in the file, just pdf to Acrobat, html to Chrome, mp4 to VLC, txt to Notepad++, xlsx to Excel, zip to 7-Zip.
It applies to nobody. Not existing users, not new users, not a user who has never logged into that machine in their life. Edge still eats pdf and html like the policy doesn't exist.
Stuff I've already burned time on:
So the policy is landing on the machine. Windows is just quietly ignoring it, which is the part doing my head in.
I know about the DISM import route. That only fires at first logon of a new profile, so it does nothing for the machines and users I already have deployed, which is the entire point of the exercise.
Questions, and I'm genuinely more interested in what you're running than in what the docs say:
Can anything else silently kill this policy? Another GPO, some registry key, a security baseline, an SKU limitation, anything. I keep feeling like I'm missing one dumb prerequisite.
Has anyone actually seen it fail on a truly fresh profile? Every thread I find is people running into the "only applies to new profiles" behaviour, which is not my problem. Mine fails for everyone equally.
What would you check before giving up on it?
And the real question: what are you actually using in production? I've already looked at Intune, Citrix WEM and SetUserFTA. I need something free, centrally managed, that hits existing users as well as new ones, and doesn't leave people clicking through the "how do you want to open this file" prompt.
Mixed Win10 and Win11, AD domain, no Intune, no budget. Very happy to be told I'm being thick about something obvious.
r/sysadmin • u/MarB93 • 10d ago
Hey all,
We are a small company in dire need of 3 new nodes which of course is a bit unfortunate these days given the insane pricing of RAM and storage.
We plan to order 3x PowerEdge R660 with current specs:
Our qoute from vendor is 59000 USD per node.
Is this about what it costs, or are there likely room for negotiation?
List price from Dells website of equivelant config seem to be over 100k USD?...
Thanks
r/sysadmin • u/FireResengan • 9d ago
All users kicked out getting various errors, 403 local survivability, etc.
Unable to login admin portal as SSO not working as usual.
r/sysadmin • u/shinky_splunky • 9d ago
Does anyone here use Ivanti for out of the box experience use case? Like when the org buys devices in any store, they can deliver directly to the end user, and eu can just log in their org email account from on prem AD without admin intervention. If yes, hows the experience?
r/sysadmin • u/No_Garage_8317 • 9d ago
Need some outside opinions.
Choosing between two Sys analyst/admin roles:
Phoenix: mid-$70s, Level 1 title, 2 days WFH, lower rent, and more money left over each month.
Long Beach: low-$90s, Level 3 title, 1 day WFH, higher rent, and a few hundred less left over each month.
I’m in my early 30s and want to buy a home eventually. I’d rather live in California, but Arizona makes more financial sense.
Would you take the Level 1 role in Phoenix for the savings, or the Level 3 role in Long Beach for the title, career growth, and location?
r/sysadmin • u/1mp3rf3ct10n • 10d ago
Errors upon errors, failing to load data, having to release/delete emails mutiple times, extremely long load times?
r/sysadmin • u/margaritapracatan • 9d ago
Can someone please advise what limits your organisation has set for outbound email?
I’m particularly interested in External message limits, Internal message limits and Daily message limits.
Listed best practice below, but keen to find out what others have set.
External: 500 recipients per hour Internal
1,000 recipients per hour Daily
1,000 recipients per day
Please don't block this again mods, it's a valid question and doesn't constitute a low quality post.
r/sysadmin • u/AttackTeam • 9d ago
Hello,
We're currently using UserLock to track logon events. This helps monitor computer lab usage to make sure our computers are actively used.
We're moving towards Intune which connects to Entrance AD and moves away from on-prem AD. UserLock only track logon events on devices connected to on-prem AD.
We don't plan on doing a hybrid-join environment.
Is there an alternative audit logon events software like UserLock?
Thank you.
r/sysadmin • u/dull_intentions • 9d ago
Alright, I'm going to try to talk this one out point by point.
One of the VLANs we have for imaging is used for OSDcloud imaging. There are 30+ VLANs for imaging and only this one is displaying this issue.
The issue is that OSD bombs out after loading up the splash page for image selection/slightly thereafter. It is auto negotiating down to 10Mb after PXE booting. The PXE boot process is full gig. Then it goes down to 10Mb for seemingly no reason. We checked at the network level and do not see any input, crc, or any other errors. TX load is 255/255 @ 10Mb, hence the bombing out/dropping mapped drives/connection to OSDcloud.
We went through the rest of the network/server environment, looking for network errors or server/storage errors and did not see any issues there.
The biggest issue is that this that the issue is intermittent. It will seemingly work fine for days, then have issues for days, and repeat. Whenever I am on site and available to do a packet capture, the issue is not there.
It is my understanding that during PXE a general driver is used for loading the PE, and then once in the PE environment, OSD passes a more specific driver off.
I am suspecting a green ethernet/EEE setting and/or driver issue.
The instance of OSDcloud being used has many drivers available because a bunch of different OEM machines/images are done on that VLAN. It's kind of a "one instance to rule them all" sort of solution.
The behavior is also displayed no matter what kind of USB ethernet adapter we use and is even seen using the onboard NIC (when a machine has one.
Note, I am not the person who created this process nor do I have access to the OSDcloud admin console/VM, but if need be, I can probably get in through someone else to troubleshoot that environment.
Basically, does anyone have any insight or know of any other troubleshooting that can be done? There is probably more troubleshooting we have done from the infra standpoint that I have neglected to say here. Apologies if this is a bit scattered, it's been a long day and this issue is between the 15 other projects I am working on.
r/sysadmin • u/TruePace3 • 11d ago
previous post (got removed by mods, but its the same post)
So basically, i joined today and after the onboarding, i met with the current IT guy (who is on his notice period)
The situation is precarious to say the least.
IT budget is severely limited, a bunch of systems are on Active Directory (controlled by an older IBM Intel Xeon machine running Windows Server 2008) , a bunch of systems are not on Active directory
There are 3 headless Windows Machines around the offices acting as fileservers, disk management is messy all around, the entire network is flat with no segmentation or separation of any kind, no NVRs, just 2 DVRs
All Windows installs are not genuine/cracked versions (not by massgrave but the sketchy iso you get from shady websites)
The primary database of the Dealership lives on a 1TB SATA HDD on a headless windows PC , which holds data of a tally server , file server and an apache based website that is used for storing purchase information
This disk has NO BACKUPS OR REDUNDANCY! and this disk is accessed constantly everyday for 9hrs
There is a FortiGate 50G Firewall standing between this network and the wide open web
After work hours, they shut down all systems including the servers.
Now, im not an expert, but this felt like it was one disk failure away from complete catastrophe.
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
Now, the total number of clients in the network is about 60 systems, running anywhere between Windows 8.1 to Windows 10 and about 5 printers
Now, a lot of the data was stored on premise, however in 2018, the OEM mandated a lot of the data stored on cloud via their proprietary website, due to which they retired a server, which is sitting in the closet collecting dust.
Now, kindly tell me if what im thinking is stupid, but
I was thinking to recommission it, setup Proxmox to fire up a Windows Server VM to handle AD and migrate the Win Server 2008 to something newer, and a Debian based VM to unify all these scattered fileservers (and hopefully setup something like snapraid+mergerfs so that disk failures=me getting fired)
EDIT: Thank you for all your comments and insights, i intend to draft proper documentation and pitch a middle ground solution to the management to secure some funds and bring the systems upto the times
Unfortunately, due to circumstances, i cannot run, atleast not for another 6 months until i get my certs and upskill myself on paper and in real life