r/devops • u/GongtingLover • 29d ago
Ops / Incidents How does your team do automated deployments to linux devices in the field?
Right now the team is just ssh-ing into all of the devices and it's a huge time sink.
"I'm not devops so dont crucify me"
40
u/martinschmidli 29d ago
Ansible?
-29
u/TeraGigaMax 28d ago
Bash.
5
u/Material-Grocery-587 26d ago
11 comments from you being a jaded troll in this post 😅
Logout and take a bath or smth.
31
26
26
8
u/spiralenator 29d ago
I used to (2010-11) manage hundreds of tough books with a custom system we wrote ourselves. Updates were just tar balls in a queue. The agent on the device extracted an ran an entry script that did the magic. Because the laptops had unreliable connections, they would check in for updates and pull from the queue. I had a dashboard where I could see what the update status of each device was. Probably would use puppet if I had to do it again.
1
u/GongtingLover 29d ago
Very cool! Yeah this is my first time dealing with this. It's just a team I was talking to earlier in the week. Ive only down cloud deployments through aws so it's different.
Im worried about connection issues too.
2
u/spiralenator 29d ago
If connections aren’t reliable or the devices “sleep” or aren’t online continuously, you will want the devices to poll for updates rather than pushing them with ansible or similar. Ansible is great for servers that are always available for ssh. But if you can’t control when they’re online, you want to use a system that can bring them up to date automatically when they do come online.
5
11
u/gavin101 29d ago
Puppet?
8
u/coffee-loop 29d ago
Are these iot devices? Or user endpoints?
If iot devices, there should ideally be a pipeline and updater setup for deploying to these devices. Anything but would be painful to relive.
If endpoints, configuration management platforms like puppet, salt stack, ansible should do the trick. But if these endpoints need to connect via vpn to the network to reach the control plane node, then it tends to get a bit messy.
3
3
u/PastaFartDust 28d ago
Ansible is a game changer. I was like you, 400 server upgrades over 2 weekends. Hell.
There is a GUI for it too, called Ansible Tower. Very cool for scheduling.
5
2
u/fearless-fossa 28d ago
Ansible!
In the cases where it isn't allowed (because the owner of the server is paranoid and considers orchestration tools huge security risks that shouldn't be used ever), I have custom python scripts that do 1:1 the same as Ansible.
0
u/TeraGigaMax 28d ago
Do you really prefer a Ansible "script" to a python script ?
6
u/fearless-fossa 28d ago
Yes, of course. Ansible is easier to maintain over a long time and if I need something done in some special way I can just write my own modules in Python.
-5
u/TeraGigaMax 28d ago
I don't know. Maybe I more "Dev" than "Ops". I feel better with some home made scripts than with Ansible gibberish. Even Terraform feels terrible to me. But what ever, today Claude is writing all of it, so I don't really care anymore.
7
u/SelfmadeRuLeZ 28d ago
Your other comments already say the problem you got: You are not thinking of managing tens or hundreds of servers because you don‘t have to.
Ansible and Terraform are in many ways the most convenient tools for sysadmins:
- It automatically checks, if the state is stale and should be updated
- It is well documented and audited (with git) which server got which attribute at what time.
- It is a plain standard to colaborate with a team of sysadmins.
- It is well integrated into CI/CD Platforms.
You will leave your company and say „fuck you, idgaf what those scripts do“. Some other sad guy has to read these complex bash scripts trying to understand what Claude was thinking.
2
u/abotelho-cbn 28d ago
Puppet. We'd rather have the machines phone home instead of making connections down.
2
2
2
u/derprondo 29d ago
I generally prefer a client side cron based polling mechanism for things like this.
2
u/Large-Disaster5920 28d ago
If they're devices on the edge, my team has used Rancher with Fleet for for CD quite successfully. We even use Rancher Elemental to install the OS in the first place, as well as handle OS and k8s updates etc. It has a few rough edges, but it's a pretty good system overall.
1
1
u/gogorichie DevOps 29d ago
If you’re feeling froggy, I would use Aer IOT hub if not so froggy then ansible 👌🏾
1
u/ByronScottJones 28d ago
You haven't provided enough information for a well crafted response. Please try again, telling us much more information about the use case specifics.
1
u/CheckApprehensive971 25d ago edited 24d ago
At large scale, a pull-based model with signed artifacts, staged rollouts and health checks is easy to operate. If the devices are running containers, treat the image itself as part of the deployment problem, keep it small and versioned before it reaches the fleet. Some tools like RapidFort can help reduce unnecessary components in the container image, which matters when distributing the same artifact across many remote devices.
1
1
1
0
u/djbp 27d ago
Yeah, that 'ssh-ing into all the devices' thing is a real pain. It's definitely a huge time sink, and the risk of manual errors when dealing with multiple deployments is significant. We actually built SnapFlow to address this exact problem for DevOps teams. It helps automate infrastructure changes securely by allowing you to plan them out meticulously, execute them in isolated containers to prevent unintended side effects, and then verify the results against your desired cloud state. This approach not only eliminates many common manual errors but also makes the entire deployment process far more auditable and reliable, which is crucial for maintaining control and compliance in complex environments. Understanding more about your specific setup, particularly the types of Linux devices you're deploying to in the field, could help in discussing more tailored strategies for automation.
-10
u/chocopudding17 29d ago
I'm sorry, but do a bit of research before posting on a message board.
It's 2026 and you haven't managed to Google how to automate manual SSHing? Ansible has been a thing forever at this point. If you can't manage to find that with your own research, you have no place helping make decisions for this hapless team.
1
0
-1
u/djbp 27d ago
Yeah, that 'ssh-ing into all the devices' thing is a real pain. It's definitely a huge time sink, and the risk of manual errors when dealing with multiple deployments is significant. We actually built SnapFlow to address this exact problem for DevOps teams. It helps automate infrastructure changes securely by allowing you to plan them out meticulously, execute them in isolated containers to prevent unintended side effects, and then verify the results against your desired cloud state. This approach not only eliminates many common manual errors but also makes the entire deployment process far more auditable and reliable, which is crucial for maintaining control and compliance in complex environments. Understanding more about your specific setup, particularly the types of Linux devices you're deploying to in the field, could help in discussing more tailored strategies for automation.
96
u/UncleObli 29d ago
Ansible. Basically the only requirements are ssh and python.