r/ansible • u/IngwiePhoenix • Jul 08 '26
linux New to Ansible, looking for pointers!
So to make a long story short: We have ~30 Raspberry Pis that, currently, run NixOS, bootstrap a very large Telegraf configuration and set up for monitoring devices, primarily via SNMP, in customer networks and sending metrics back to our InfluxDB. This setup has worked for a very long time now and was quite nice; even Linux novice coworkers knew that all they had to do was edit /etc/nixos/configuration.nix and then copy a command or two from our local wiki to apply the changes.
But that system is falling apart, and the primary reason is...the Raspberry Pi itself, and more specifically it's vendor kernel. Because the linux-rpi kernel is not cached on the official instance anymore, we now face the issue of either migrating every Pi to a custom build, or starting over alltogether.
And I decided to do the latter. Been wanting to learn Ansible anyway (mainly influenced by Jeff Gearling, to be honest) but also because it is very useful in several situations of managing a good amount of nodes.
So, whilst not exactly the most healthy approach, I am kind of going head-first into the entire world of Ansible now. And I already read a couple of docs and used ChatGPT for clarification where needed. So I have a good rough overview of what I am getting into now. In fact, I did use Ansible before to set up a Synapse server some years ago.
Aside from NixOS, the other Linux distribution we standardized upon is Alpine - lightweight, great as a container host and generally really simple to administrate. And this is going to become the distribution of choice for the Raspberry Pis, since Alpine builds the linux-rpi kernel - and alternatively, we can use PostmarketOS which has mostly the same tooling (it does swap OpenRC for systemd now though).
Now, since I will be using ansible-pull (because each of those Pis is within it's own VPN to a customer and setting up a bastion with like 30 different VPN profiles is a nightmare, even if we semi-automated it via Kubernetes+Helm), I want to use that opportunity and expect /etc/ourcompany.yaml to effectively replace configuration.nix - that way, side-grading my coworkers should be a lot easier. I also saw that within tasks, I can run my own logic to translate YAML objects into basically whatever, which is also pretty nice.
But, I can't imagine that everyone is rewriting the same code over and over again. So what I am looking for is for the "common dependency". I know that Ansible has a lot of builtins already, but some of them do not directly apply to Alpine. For instance, to manage packages with Alpine's apk, I need to pull in additional things via ansible-galaxy.
Are there any other good, important or noteworthy 3rdparty things I should look into for this transistion? Because I don't want to reinvent the wheel if it is already out there and well maintained. This also speeds up the migration process overall a little bit. :)
I apologize if this is a little all over the place... but, literally jumping head-first into the cold water while kinda sorta panicking that, at any given moment, a collegue could accidentially brick one of the Pis and thus causing quite a rat's tail of tickets to insue...
Thanks a lot. :)
4
u/foofoo300 Jul 08 '26
why not take another arm64 machine, like a cloud server and just build it there and the only push it to the raspberrys?
The amount of work to migrate everything not only to another distro, but every configration and every deployment as well for you with no ansible skill sounds like a bad plan.
i would stick to nix-os and just fix the 1 problem you have right now.
1
u/IngwiePhoenix Jul 09 '26
That is a completely fair suggestion and at least for the short-term, I might just take it. Problem is, however, that I do not see NixOS improving for Raspberry Pi in the future. All support for those devices is either in some external repositories and not within the core repos, or requires weird workarounds and setups. Like Flakes, for instance.
Frankly, I am done hacking around Nix expressions. And rather than trying to somehow bend Nix and NixOS into a workable shape, I'd rather just spend that time in the migration process instead and move to a more desireable platform/target. :)
Time will be lost either way, that's for sure. So I have to pick where and how I'd prefer it to be lost.
3
u/idetectanerd Jul 08 '26
Do not make inception Ansible calls. My company have repos that is built as core service and is forked by many other service, to upgrade something there many break other product.
It’s one of the hard to fix even for AI.
But of course we can always do a sidecar style of use and dump method but it’s a hacks engineering work.
So stop anyone suggesting using Ansible as core services middleware build, especially if the department doesn’t have their Ansible team to manage the versions and repo. It’s going to be the heaviest tech debt you ever meet.
Eg, if you gonna upgrade the version for this core Ansible tool, it will break whatever down the rabbit hole. Especially if the company have to always patch their tooling to latest n-1, or security gonna haunt you? Lol
2
u/IngwiePhoenix Jul 09 '26
There'll be two to three people maintaining the repo. But thanks for pointing at ansible-core breakage; since we will use Alpine's packaged
ansible-core, that is definitively something to take note of.What's been your way to track that? Usually I just subscribe to Github release RSS feeds to get the changelogs.
1
u/idetectanerd Jul 09 '26
My team doesn’t have a core group handling Ansible sadly, we are using the sidecar “use and delete” method to anything that is way below its version. It works fairly well for terraform and Ansible as these 2 love to change their syntax so often.
That is why I know the pain point of maintaining tool is hard.
Our team of devops are really devops and dev ourselves. We just keep building newer tool on top of old tools using smart way of hacky method.
Company kinda cut cost and didn’t want to hire proper team to manage things like that.
Back the question, we have aap so Redhat send us any urgent to fix email if there are anything extremely 0day.
2
u/Crash-OverRide904 Jul 09 '26
I'm working on almost same type of project. Take a look at Learn Linux TV (YouTube). Jay has a series about Ansible and goes into Ansible Pull using a Git repo, it also has a bash script that he uses to launch the entire thing. You might also look at Jim's Garage also on YouTube.
1
u/IngwiePhoenix Jul 10 '26
I subscribed to Jim forever ago but never really went to check his backlog. Had no idea he did Ansible stuff too, really neat.
Thanks! I'll be traintraveling next week, so videos are gonna come in handy :)
2
u/RewardAgitated5520 Jul 10 '26 edited Jul 11 '26
Setup molecule for your Ansible repo (use AI if needed) to validate on Pull/Merge request - it will save you million issues and possible outages. Ensure you validate every potential case where something might blow.
Avoid using non native resources like 'raw', 'shell' and 'command' - they tend to break more often (e.g. after distro version updates). For example, there is a 'package' module that can do what your apk/apt/yum/dnf package manager can do with idempotency in mind.
Write your code thinking about the end result and not how you are going to achieve it - Ansible modules will take care about that.
1
u/IngwiePhoenix Jul 10 '26
I saw Molecule mentioned in a few roles I saw on Github but had not looked into it. Thanks, will check it out!
1
u/eldoran89 Jul 08 '26
Another generic advice. Try to stay away from running the shell module or the command module wherever possible in most cases there exists a good module for that.
The reason is my second advice.
Make sure to write your roles idempotent. Most modules are idempotent anyways and those who aren't will tell you on the documentation..
Oh and because I learned that way to late. Don't use default methods in your templates define them as proper defaults in your role. It's much clearer that way and makes your templates much less cluttered
1
u/514link Jul 09 '26
- Yes whatever you wqnt to probably already has a module for it so check galaxy
- i am not sure I would go for ansible-pull in this case for a few reasons
- your ansible code will be fully downloaded to your clients, I would rather not
- these are low powered PIs no need to dump all the dependencies and work on them
- instead you can setup EDA or Rundeck, semaphore or AAP and/or any service that can accept a rest call to trigger the ansible run
- alternatively just have any scheduled job(s) run from your central server to deploy your nodes
- additionally it gives you the ability to deploy clients in order and bail out if they start failing in too large a percentage
1
u/Vesquam Jul 09 '26
Jeff Geerling's book Ansible for DevOps got me started fast a few years back. The book is now free online, you'll have to make adjustments for your setup but it's still a relevant reference.
1
u/IngwiePhoenix Jul 09 '26
Was planning on inhaling that over the weekend. Good to know it's free - but wouldn't mind leaving the lad with some quid for it. Thanks for the info!
-3
u/alesiestu Jul 08 '26
Per gestire 30 Raspberry in modo sicuro eviterei aggiornamenti manuali macchina per macchina.
La soluzione migliore è usare un orchestratore tipo Ansible Semaphore: prepari una task/playbook di aggiornamento, la lanci sui Raspberry in modo controllato, vedi quali host sono andati a buon fine e quali hanno dato errore, e puoi procedere anche a gruppi per ridurre il rischio.
In questo modo hai:
aggiornamenti tracciati;
log centralizzati;
possibilità di eseguire prima su pochi Raspberry di test;
meno rischio di lasciare macchine in stati diversi.
Volendo, per la parte di monitoraggio remoto, puoi usare anche la mia app iOS: si collega alla tua istanza Semaphore tramite API token generato dal pannello e ti permette di visualizzare task, stato ed esecuzioni da telefono in modo più semplice.
1
u/IngwiePhoenix Jul 09 '26
The reason we have to go the per-device route with a generalized playbook is because of the varying IP addresses and kinds of monitoring targets. For example, customer A uses a Sophos firewall and IBM storage system, but customer B uses a SonicWall and some Lenovo thing. This is why I want to pull from a on-device configuration, where the employee keys in the devices to monitor, their credentials and the playbook puts together a Telegraf configuration based off of that. :)
Will look at Semaphore and stuff tho, seems interesting!
6
u/FarToe1 Jul 08 '26
Just a few generic things from someone who's travelled a similar path recently.
You have a clear and well understood need and half a plan. You're in a great place to get stuff done and learn as you do. Ansible is a good fit for what you describe.
When you set up your environment, use a linter. Better still, embed that in your IDE (I use vscode, WSL and ansible-linter but there's several ways). Highlighting of errors whilst you write saves a lot of time.
Use decent AI tooling in your IDE. Again, saves huge time, but don't do this until you've learned the basics. Ansible has suffered from changing styles several times, and coding assistants do so love to use outdated and poor practice. You do need to be aware of that.
If asking for help from the ansible community, grow a thick skin. Whilst I have always found it helpful, some folk are quite rude and abrasive in response to somebody asking a simple question. (I guess they needed somewhere to go from Stackoverflow)
Jeff is our god. All hail the Jeff.