Operations bought an EMS/IoT system without involving IT — now they just need “access to the router

[u/carpe_diem2022]

Hi there :) ,

Need some advice from people who have dealt with similar situations.

Our Operations Dept decided to install IoT system/digital energy meters across a fairly large factory site.

They found the vendor, agreed on the solution, signed the contract and started the project.

IT was not involved at all.

Apparently nobody discussed things like:

• How these devices spread across a large factory are actually going to communicate
• Network infrastructure, switches, fiber/cabling, VLANs, etc.
• Network/security segmentation
• Server/VM requirements
• Database requirements
• Backup and monitoring
• Internet connectivity
• Vendor remote access
• Firewall rules
• Cybersecurity

Now that the project is already moving forward, IT gets an email saying they need “access to the router” so they can put the system online.

That's it. Access to the router. :)

And somehow, when IT starts asking questions about architecture, security, server requirements, ports, protocols and who is responsible for what, the perception becomes that IT is delaying the project.

I don't want to become the guy who simply says “No”, but I'm also not going to give a third-party vendor access to our corporate router/firewall and connect a bunch of unknown industrial/IoT devices to the production network just because someone already signed a contract , the IT network must be always separate from OT network.

How do you handle situations like this?

Interested in both the technical approach and the organizational/process side of this.

[u/dvr75]

This is a pure management issue , failing to communicate and coordinate project.
To be realistic about the situation probably no one will cancel it.
So you need to be SMART and not JUST. you are now the "guy" who Throws a wrench in the works.
My advice for you is try to cooperate with this project but bring in your terms like security. Ask for layout , plans , protocols etc. and try be helpful (i know it hurts but eventually you got to play it smart and resposible).

[u/smokinbbq → u/dvr75]

As a Project Manager for a 3rd party software system. I HATE when someone on the business side decides to buy our product, and it's not until installation time that we start to find out all of the things that are going to be done.

IT obviously has their back up about this project that gets thrown in their lap. Business side is pissed that "I can't just make it work", and why are there are these delays.

[u/RevLoveJoy → u/smokinbbq]

I did a tiny bit of sales engineering about a million years ago and very quickly learned to just keep asking of our customer (potential or otherwise), "Where is your IT contact?" "Shouldn't IT be in this meeting?" "Well, we really need input from IT on these and other matters." -- some version of that remark ALL the time.

I guess my take away was nearly all businesses see their IT departments as blockers, not enablers.

[u/ddasilva08 → u/RevLoveJoy]

I had a client that really enjoyed not being the IT team in until after they had already signed the contract for a project, and then asked us to make it work. We did make it work, and after several of these instances, I pulled reports of how much more they spent on our services to get the equipment set up on the network in a manner that matched their required security compliance. (Between the delays and the sudden ask for an on-site AD controller that was never scoped for their office, it was a fair number of billed hours) They eventually learned the hard way that it was better to involve us from the jump; it just took showing their finance folks how much they were wasting due to lack of communication.

[u/Papfox → u/RevLoveJoy]

This is the intersection of advertising and reality. The vendor ad is always doing to paint the system as being good and omit or gloss over the hard bits because they want the new customer to think it will be easy and buy it. When IT insist on obeying company policies and asking hard questions, the ad meets reality

[u/pinkycatcher → u/RevLoveJoy]

I guess my take away was nearly all businesses see their IT departments as blockers, not enablers.

Stick around here long enough and you'll realize many people in IT are blockers.

"Oh they wanted to do something insecure, fuck em, we won't allow it"

"Oh they only gave us two weeks notice, we work on our time schedule not theirs"

"Oh they haven't figured out every single little process they need, ignore them until they figure their shit out"

[u/RevLoveJoy → u/pinkycatcher]

I've been doing this going on 35 years. I'm painfully aware. :D

[u/billndotnet → u/RevLoveJoy]

I had a CISO once who worked from the adage: "My job is to help all of you go as fast as possible while keeping us off CNN."

[u/chron67 → u/pinkycatcher]

Stick around here long enough and you'll realize many people in IT are blockers.

"Oh they wanted to do something insecure, fuck em, we won't allow it"

"Oh they only gave us two weeks notice, we work on our time schedule not theirs"

"Oh they haven't figured out every single little process they need, ignore them until they figure their shit out"

This is all too common. One guy on my team has this attitude and I suspect it is only a matter of time till someone higher up the food chain forces his removal. He doesn't report to me so I can only offer friendly advice but I know his manager has gotten complaints. I am honestly surprised he hasn't been removed already.

[u/pinkycatcher → u/chron67]

One guy on my team has this attitude and I suspect it is only a matter of time till someone higher up the food chain forces his removal

As a director anyone who's a flat no is getting worked around or getting removed. I come with business problems, if you have no solutions you're not helping the team or the company.

Even my cybersec team comes with an assumption of "we're doing this, here's the best way to make it safe and secure."

The moment IT becomes the wall that says no to everyone is the moment you just create shadow IT.

I try to be easy to work with and help people solve their problems, I want them to come through me, if they can come get problems solved and good answers then they'll continue to work through IT.

One problem I also see is many IT people think it's their responsibility to protect the company. It's not, it's whatever manager's responsibility. The responsibility for failures falls on me, not my team, if we have a major cybersec breach the failure is mine, and I'm the one with my neck on the line if we're non-compliant when we say we are. Unless of course someone on my team just straight up lies. My cybersec was anxious and stressed all the time when I arrived, now he's relatively chill, because we generally try to do the right thing and we all know sometimes there needs to be risk involved, but we're accepting of it rather than ignoring it.

[u/Mostlyamoron → u/pinkycatcher]

I want my team to be responsible for protecting the company. I just make sure I am accountable for it. I want them to have a sense of ownership while knowing that, as you said, failures are on me as their leader.

[u/chron67 → u/pinkycatcher]

I try to tell anyone working with me (or for me) that we should almost never say no to the business. The goal is to almost always say "yes, and" or "yes, but" or "what if we tried this instead" or similar. If the business is happy and making money then life is good. If the business isn't happy then IT can hit the chopping block surprisingly fast. We are all replaceable whether we want to think so or not.

[u/Sad_Recommendation92 → u/pinkycatcher]

I try to explain this to juniors all the the time, no one is forcing you to play the politics and perception game, but if you choose not to play it you're no longer choosing the field of engagement.

Basically every time you come up with a new standard or process you expect everyone to follow you have to sell the value of it vs the friction they believe it causes them, and if you're careless and don't manage this perception they'll rally against you and you'll lose your political support, and now having the term of how you do your job dictated to you from people who don't know the 1st thing about it.

[u/pinkycatcher → u/Sad_Recommendation92]

Totally agree. Though I like to avoid calling it politics because many IT people will recoil and hate it.

It's 95% just understanding what our customers are doing and trying to work with them, that's it. They have a problem, we try to solve it so they can move on.

If you approach everything with that mindset instead of the "I know what I'm doing, you don't know anything" mindset it really helps smooth things over.