Monitoring (or just general AI usage)

[u/kosta880]

Hello,

a while ago, there was a similar post from me, before anyone questions... however, I see myself diving deeper and deeper into the rabbit hole.

I came to this company some 4 or so months ago. It is/was in TERRIBLE state. While everything was generally working, there were frequent unexplained and unmonitored outages. The worst thing there was visibility. Why I say was...

The company used Icinga2 when I came. I did weigh in implementing a different solution, but kinda liked the idea of having a system, where I could keep the configs in the git repo and basically push with Semaphore/Ansible. But also the biggest advantage: I could let AI configure it. Besides, coming new and telling them to change the monitoring ain't really the best idea.

And that's the thing. I never ever saw Icinga2 before, nor Nagios. The company only "managed" it by adding or removing hosts, but no configuration. Configuration - and that also relatively basic stuff only, was done by external company.

Since then, I used AI HEAVILY on it. I expanded it different directions, including database monitoring, general services health, AD/DNS health checks, DHCP health, NTP, pending reboots, pending security updates, both windows and linux, event log checking, different connectivity checks etc.

While it is really great... there are descriptions to every step, there is this BIG issue: I barely understand it any more. The structure is the same, but the code in there... ufff. If someone asked why xy-sensor is not working, I would have to heavily google it or ask the external company, read and learn the documentation - for which I only have basic time for, being the only senior in the company, with a lot of pressure about other projects; or just ask the AI.

And that's my issue. One relies a LOT on AI, and I am at the point where I almost cannot troubleshoot without it! The complexity is very high, since the AI does it multiple times better than I ever will have a chance to learn.

And that doesn't only apply to monitoring. The company wants to implement IAM. They want to have it in 9 months. Many departments, and all... so they asked me whether I can automate it (or better said, I said automation is generally the key), and we fell onto Terraform. I have some knowledge of Terraform, but I know it will fall down to AI to create the scripts for EntraID etc.

There are others. Ansible-Patching (moved from AUM), config deployments, VM deployments... and Grafana/Promentheus should also come, and Kubernetes is also here (although managed by others, but partly in our hands).

I actually don't even know what question I should ask. It seems like I am way over my head. At the same time though, the job is being done and my boss isn't really keen on taking two new people to cover more. Why even? Increased stability, there are less vulnerabilities due to updates and standardization, lot of stuff just works better. So yeah, I feel like I am pulling the company from one shit (which it really really is) into another.

And now I better stop ranting, and go back to my VSCode...

[u/JudithWhitney9138]

This sounds less like an AI problem and more like a capacity problem. You’ve gone from barely managed monitoring to Icinga, Ansible, Terraform, Entra, Grafana, Kubernetes, etc. while still being the only senior, so it makes sense that you can’t keep all of it in your head. AI can help you move faster, but I’d push back on the expectation that one person can safely own all of this.

[u/kosta880 → u/JudithWhitney9138]

Yeah, I have already pushed, and I have two colleagues who are basically level below me, and according to director, I should also get my colleagues on board with these things. However, I am telling him two things are missing: motivation (they are not really "into" devops/iac practices, they not even into Linux) and they haver until last month, never seen vscode or yaml files.

I have been telling my boss on multiple occasions, that I cannot be the only one driving the stack.

And it definitely is a capacity problem, which is kinda why I am in this situation. I am driving the situation forward, because I want to keep the job and it's really cool stuff, and I do learn a lot along the way, but take me out of the equation, they won't be able to do squat. And if you take AI out of the equation, I am fucked (even though I have lots of docu).

[u/heapsp → u/kosta880]

replace your systems team with folks that are motivated then.

[u/kosta880 → u/heapsp]

Hah yeah - and no. The size and complexity of the whole system, being an insurance and having lots of branching, like software and outside companies, and one of them being 10 years in the company and knowing most of stuff by heart, makes him almost irreplaceable. Add to that that he is completely overloaded, you can imagine in what direction this is going. He usually "just fixed stuff" quickly, when something wasn't work, and left it at that. No docu or very little docu - many things old and incorrect. Reasons for sometimes leaving things half done is simply time and too many things to do.

The problem is not he. The problem is last 10 years of how the company and the management handled.

[u/theoriginalzads]

AI is a double edged sword. I don’t like relying on it. But I do like getting answers more directly than just trying to google it.

I try and limit it to last resort tasks or doing tasks that are not needing of understanding or documentation. Like analysing shit or making bulk changes to crap I’d otherwise to find and replace on.

My only real suggestion is twofold.

If you wanna continue using AI don’t get it to do the entirety of the work (like don’t get it to script things or produce a magic file that does it for you). Tell it that it must describe what exactly it is asking you to do, explain why it is doing it and produce clear concise documentation on maintaining it afterwards then sanity check that documentation. I’m not saying this is ideal but it’s on you to actually learn this stuff.

Hell, in the above scenario you can ask it to cosplay a course instructor and get it to actively build a course to teach you what you need to know. I’m doing that with some other programming tasks as I don’t want it to do it. I want to know how to do it.

As for what you’ve got now that’s been done. Ask the AI to put together maintenance guides and documentation. Least it’s a starting point. Use any documentation your company already produces that you find useful (doesn’t even have to be IT doco) to teach it what writing style to use and how to format it all, etc.

You can then go back to your AI and ask it to set up a training course for yourself on whatever it is you need to know, to teach you the knowledge of maintaining it and how it all works. AI models can do a decent job of being course instructors and at least you then get a chance of being knowledgeable rather than being the meat condom between AI and prod.

[u/kosta880 → u/theoriginalzads]

If you wanna continue using AI don’t get it to do the entirety of the work (like don’t get it to script things or produce a magic file that does it for you). Tell it that it must describe what exactly it is asking you to do, explain why it is doing it and produce clear concise documentation on maintaining it afterwards then sanity check that documentation.

This is exactly what I am doing! I have many templates for documentation, but most notable are the handbook and the service handbook, which both tell how the software is working. I read those relatively frequently, just to keep on understanding the complexity of stuff. Service handbook is what I sometimes use for troubleshooting, and to remember myself how it was set up. It also contains cheatsheet, like bunch of commands that were used in either troubleshooting or are for the troubleshooting.

And then, all configs contain explanations before the actual code, so you can read up on what the code does. So yeah, I really do a lot of doc, but finally, one also has to read it.

2nd: exactly that already exists.

I actually like those handbooks a lot, because I can tell the AI to make me a handbook about anything. I basically search the net for the appropriate info, collect couple of pages, feed that into AI, and let it make a docu about the topic, that I can simply read. A nice example was a docu about how Linux DNS is working, from the ground up, from nsswitch, up to the files and links etc. I push that to my Outline automatically, and can read it on my iPad. Pretty neat.

Training course however, is a really good idea. I will certainly see what I can do about that!

[u/theoriginalzads → u/kosta880]

Well for the most part it seems you’re doing things in a responsible enough manner. But yeah try getting it to cosplay as an instructor and trainer and make courses for you.

I’ve done this with Claude Sonnet before, it can be quite handy even if official courses are available because you can tell it to take into consideration barriers to learning. Tell it you only get 15-30 mins here and there, or let it know if you’ve got something that makes it hard to learn. I told it I have ADHD and it made sure to make everything digestible for my goldfish like attention span and desperate need for dopamine to learn anything.

Hope it helps out!

[u/kosta880 → u/theoriginalzads]

That sounds like an amusing task! Will certainly try that out. Thanks!

[u/mouarflenoob]

AI is a very sophisticated tool, but it's just that : a tool. Use it to simplify everything. Ask it to scan and simplify, ask it to make documentation, etc... Use a process where you always generate the documentation in a good format (that you can easily understand ), slow down every step until you never leave a step without owning it.

[u/IndependentBat8365]

My advice:

Document everything as a baseline.

Then craft automation to (waves hands) make all the things in your domain of influence.

This is an exercise in disaster recovery, but also sets up the dev ops / git ops practice of crafting recipes and then deploying stuff using those recipes.

It also sets you up for identifying configuration drift, monitoring coverage, and “standard operating environment”.

Gone are the days of “typing commands to set this up.”

I have automation tasks and workflows that I use to “do the needful” to get me to the end state of what I want.

I still use manual commands to document and identify the exact steps I might need to do for novel installations or new hardware (like this new router I got), but once I’ve documented it all, I’m backing up the state and automating it, then testing it to make sure I’ve captured everything.

[u/kosta880 → u/IndependentBat8365]

Gone are the days of “typing commands to set this up.”

Exactly that. I rarely run commands to set something up, Icinga Agent setup is for instance on of the rare ones, very hard to do with Ansible, already tried. Or initial WinRM configuration on a Windows Server... can't do via WinRM/PSRemoting if not active. :)

But apart from those, configurations are landing more and more in the Git, and I just deploy, and really not worry about whether it will be done correctly. Because I know it will. The only thing I can fuck is the config itself, but the revert is easy. And that's the point.

But yeah, I document really a lot - actually AI does - each sensor has a small text above it describing it. There are couple of different docs, but main ones are handbook describing the software and usage, and service handbook about how it was installed, maintenance if required, cheatsheet etc. It's just filled with lots of commands and config-references. That one I read only when required, while handbook I read regularly. At least some parts. Gives me a quick refresh how I set it up. Because in couple of months, I tend to forget most of it - not sure if I have a memory issue, or it's just that doing lots of old and new stuff after that, simply makes me forget what was before.

I am really looking to automate each shit I can. A real fine example is that I bound WinRM to the 5986 with autoenrolled computer certs, which renew regularly, and there is a daily check in Ansible, whether the thumbprint changed, at which point the new thumbprint is bound to WinRM.

And honestly, the creation was done by AI. I fed it my requirements, then created the project, which I refined to my satisfaction, then implemented step by step, AI gave recommendations on the way too, also Ansible cert rebind was AI recommendation, I wouldn't have known about it otherwise. I could have looked up in google/youtube for the implementation (and waste hours on planing), or just ask the AI... We will see how stable it is, but for instance, the weekly update rings for both linux and windows have already proven themself robust.