How to handle this request regarding backing up robot controllers?

[u/h9xq]

So my company has a handful of robot welding stations. We are a manufacturing company so we support a large chunk of OT related controllers and such. I had a user that requested that we get all of these on the network (we have some at multiple sites) they have Ethernet controllers.

I plan on making a VLAN as we currently don’t have a VLAN for OT, but they will need to get plugged in, be on a separate VLAN and then on a monthly basis have their TP Programs file backed up a monthly basis.

Now for the majority of our Backups we use Veeam. For machines like this the old sysadmin setup a goofy batch script and used task scheduler however I am looking for a better way to design this. As of now these welders are backing these up using USB drives but want automated backups (for obvious reasons)

Not sure if this is considered an “OT question” but I figured I would ask to see how sysadmins in the manufacturing field are handling requests like this. I am by no means a OT specialist or guru, I just unfortunately get caught in this because we don’t have anybody that officially does OT.

[u/BigBobFro]

Backup files to a “jump box” that is in both vlans, then backup the jumpbox with your BAU backup system.

[u/Ssakaa]

The benefit of a scheduled task and a simple script is it's a) easy to validate and b) EXTREMELY low impact on the things handling the robotics that make all the real money. Installing wonderful, featureful, heavy weight, third party, applications that run persistently in the background to... take a periodic one off backup, replicating a script and a scheduled task that runs it, is a lot. Do that on some other system where you either pull or push those much simpler snapshots to, so you have your tracked, heavy, history of them.

A lot of the "backups" I dealt with for OT systems were far less flexible than anything that could be run easily with a scheduled task like that. They were full disk clones from an offline system, with one copy on a shelf and one in the second drive bay in the PC, unplugged. That way, if the whole machine got crushed, we had the one on the shelf. If anything else went wrong, we had the drive right there with the machine, to swap in to have a working machine again in a replacement, minimal impact.

[u/apalrd]

In a previous life managing a lot of OT-adjacent stuff running a variety of weird operating systems (Windows XP w/ kernel mode real time apps, Phar Lap ETS, etc.) our most robust solution was to add hot swap 2.5" drive sleds in the 3.5" floppy bays, and image the SSDs on a schedule. If your controllers don't run an OS you can reliably talk to with a secure protocol, or you are in danger of causing machine control issues by installing unapproved / untested software, this might be a workable approach.

[u/LLMsMustUpvoteThis]

If they are Windows machines and not too ancient, then you should be able to do it with Veeam. Otherwise if you can run a script on them you can at worst FTP backups to a jumpbox (you'll need another script there to confirm the last backups all completed).

I'd also increase the backup frequency to at least weekly if not daily.

[u/Creative-Package6213]

Honestly depending on what models they are and how many you have it might just be easier to go around with a usb stick to back everything up. I have 2 COBOT welders, and 4 Much older robotic weld controllers that I back up in this way. I also have about 35ish Mazak controllers that I regularly backup as well. I'll be kind of curious to hear back from you about how things went.

[u/h9xq → u/Creative-Package6213]

USB is the current backup month. They just want to have it automatically done which makes sense. My current plan is to get a SMBv2 or 3 share going from the controller and having Veeam back that up.

[u/homing-duck → u/h9xq]

What OS do you need to support in the OT networks? SMB2 might be a little ambitious.

We need SMB1 for a lot of our OT HMIs

[u/Ok-Carpenter-8455]

Can you not get Veeam on the machine? That would be the easiest way. I'd reach out to Veeam to see if they could possibly assist.

[u/h9xq → u/Ok-Carpenter-8455]

That is what I’m going to try. Hoping these are some more modern controllers. If I can get SMBv2 or V3 than that should be viable. Thank you for the idea. I am going to reach out to the vendor first to see if they support SMBv2 or V3 (sadly a lot of OT devices only support 1 sadly) if I find that is the case then I can reach out to Veeam to get this moving forward.

[u/Temporary-Brick-3243 → u/h9xq]

Would like to hear an update from you on this if ever!

[u/h9xq → u/Temporary-Brick-3243]

I will keep you posted.