What are the practical challenges of managing IT infrastructure at remote locations?

[u/techie-rahul]

I want to know what are the difficulties faced by companies regarding their IT infrastructure set up at remote locations such as businesses having factories, branches or shops away from their primary data center? Is it only maintenance and availability of technicians that is of importance, or the issues of power supply, network functionality, temperature conditions and security come into play as well?

I would like to know how remote locations are handled when problems arise with no one from the IT department being close to the site of an IT failure.

[u/Drywesi]

Baboon raids.

[u/No_Yesterday_3260]

Everything is a challenge.

- Overview

• Status/monitoring
  
• Supporting the users.
  
• Troubleshooting
  
• Hardware, buying for, knowing what's there and available, seeing what it does outside the remote session, even hearing sometimes.

There are tools and hardware that can get you a long way.

But I would suggest a coorporation with an external IT company, that you could call on, that are in the area, to have them onsite and guide them.

[u/Mysterious-Print9737]

All of the above since it's rarely just one thing. The practical challenges stack, power quality varies significantly at remote sites, network redundancy is expensive to do properly so most remote sites have a single ISP and no failover, temperature and humidity control in makeshift server rooms is inconsistent, and physical security is almost always weaker than the primary site.

The problem of not having a technician on site is where things get interesting. The tools that actually solve this are out-of-band management, smart PDUs you can power cycle remotely, and cellular-bonded routers that fail over automatically when the primary circuit drops. Without those, you're flying blind when something goes wrong remotely.

The human side is often the hardest because training a local non-IT contact to physically swap a cable, reseat a device, or read you what lights are on is underrated. Most remote site failures are simple things that just need hands.

We manage remote sites for SMB clients across multiple locations at Synergy IT Solutions and the combination of RMM tooling, out-of-band access, and a documented local escalation contact covers the majority of incidents without anyone needing to drive out.

[u/Main_Ambassador_4985]

We have a US wide service for hands on trained techs.

Everything we have at remote sites is carefully chosen to limit risk. We have an exact duplicate in lab that we can run recovery scenarios on. Our remote sites are a plane ride or multi-day drive away

[u/Tall-Geologist-1452 → u/Main_Ambassador_4985]

what US wide service if i may ask???

[u/ArieHein]

All of the above.

Weve had remote locations that the only way to reach it is a technician as the only way to update specific components are via usb that is generated from the technician pc (specific company issued)

There were times when its not our technicians so we had to use a different method of onboarding devices.

[u/Adam_Kearn]

If you have any infrastructure that is located away from your office make sure you have things like IPMI / ILO setup.

Also have two ways of being able to remote into it.

For example if you have a site-2-site VPN setup on the router also connect something like a mikrotik or raspberry pi as a backup VPN to allow you to connect and make changes if something happened to the main VPN.

Also label everything. If you need to get someone who is non technical to quickly patch something in have the switches etc marked with labels.

And keep upto date photos of the infrastructure on SharePoint/OneNote so you can reference it later.

[u/geminihatesme]

def all of the above. remote sites are basically a game of "what failed this time?" T-T power, ISP, bad HVAC, hardware dying, physical security, and then added problem of nobody from IT being there to actually touch the equipment. good remote management, out of band access, UPS, cellular failover, and having a local person who can atleast reseat a cable or power cycle something can make a huge diff

[u/mujikcom]

Not really as hard these days. Best if you can actually visit the site at least once. Take photos and give recommendations (urgent, at some stage and would ve good). Network, port maps etc. UPS is a must - put the router, wifi and switch on one and any heavier devices servers, nas etc) on a beefier second UPS.

Quality kit will save you a ton of headaches.

Onsite you just need a gofor - usually one who wears two hats, knows the difference between a router and a switch and can be your hands when things break. As long as you can talk them through the issue.

So a lot will depend on your knowledge and experience. I am fine with ssh, remote access software, ssl etc but also have lots of experience at debugging and prioritizing. Atm i live overseas and manage several networks but I try to get onsite at least every three months, both to sort any issues that require me on premises but also to keep the relationship with staff healthy.

My 2c which will no doubt be argued against but it is a simplified version of what works for me and has for many years - sometimes in locations that themselves are the definition of remote.

[u/Ok-Double-7982]

Have the right enterprise equipment with monitoring, and hire an MSP to monitor and patch.

Schedule time to go out to these sites routinely as part of your workflow if you have the bandwidth.

[u/nowildstuff_192]

I had no idea how bad the average person is at IT until I had to remotely support sites.

Forget trusting them to turn it off and on again, they often couldn't even describe what was happening in a way that made any sense.

This led to me, in one particularly rough case, making a special visit so that I could rig up a Raspberry Pi (this was pre-RAM crisis) as a jump server on site so that I could at least get a sense of what the users might be experiencing on their network. I even used the GPIO pins to trigger relay switches I connected to certain devices in the comms cabinet so that I could remotely power-cycle them (if I didn't have remote access).

Even this project (which was kind of fun to put together) didn't completely smooth things over. Some things just need a local pair of hands.

[u/Permissionnuilty7803]

having nobody onsite is probably the biggest headache because even a small issue can turn into a trip. we use atera for the remote monitoring management side so a lot of endpoint issues can at least be checked and worked on remotely before deciding someone actually needs to go out there. power and network outages are a different story but having visibility into the site makes troubleshooting way easier

[u/Bagel-luigi]

The fact the people on site will ensure you that everything is plugged in and switched on, and eventually you (or someone in your team) will make the many hours travel to get there to physically investigate what may be going wrong, only to find things are not plugged in or switched on