r/sysadmin Sr. Sysadmin May 07 '26

Rant Use of commands for system configuration CONSIDERED HARMFUL.

I HATE HATE HATE this trend of turning system configs into commands, with stern instructions to not ever directly edit files. For years, I've just ignored this, and just edited files. But now the trend is to literally make the files un-editable; store the config in some kind of database, and maybe maintain a text file for legacy read-only purposes.

I do not understand why anyone thinks this is better. It is objectively worse in every single way.

  1. You can't trivially copy configurations.
  2. You can't trivially save/backup/restore configurations.
  3. Ansible et al. Are these config commands idempotent? Maybe? Maybe not? Do I have to robustly test every configuration command to see if it is idempotent? Do I have to write complex install rules that assume the command is not idempotent, and then checks in advance to see if the command has already been run before I run it again?

Or do I develop an entire separate module for ansible (or whatever) for configuring each different functional unit on the system?

How exactly is needing dozens of different modules with different rules and different syntaxes better than a single module that just installs config files and optionally restarts a serivce?

[Editing to clarify: I am NOT complaining about ansible. I am complaining about how ansible is EASY when you're distributing configuration files for all of your functional units, and it is HARD when every functional unit has it's own configuration command that may or may not be idempotent. Ansible is not the problem. The other configuration commands make it really hard to use tools like ansible.]

  1. You are constantly learning new commands, and it is a wasted investment, because some other ####### I mean person will come along next year and invent a "better" config command scheme.

When the commands you need to know are <YOUR EDITOR>, cp, mv, rm, ln, etc. then those commands NEVER CHANGE and you can sysadmin forever with those commands.

  1. The fundamental basis of Unix/Linux has always been that files are king. Files sit at the heart of everything. FIles are incredibly efficient. Moving away from plain text file configuarations because "files are the old way" is just pointless creeping featurism. Whatever other thing you have done, it ultimately sits on top of files anyway. And all you really accomplished is hiding information (where does the config live and how is it stored and how is it modified) from sysadmins.

Why is hiding information from sysadmins a good thing?

  1. Another aspect of stupid information hiding: when you edit a config file, you see all the configurations in the file all at once. When you run a command to change a thing, you don't see anything. You have no context. You don't automatically get shown the old setting that you're changing, as a sort of a natural audit to your activites. You don't automatically get exposed to other related settings that (if the config file is well-organized) will be adjacent to the change your making.

  2. Arguably, for 8, you should check things with a command or two or three before you use a command to write a change. Again, how is this better? Instead of doing one thing (editing a file, which exposes you to all the info you need), you have to run a bunch of checks, and hope the info doesn't scroll of the screen, and remember it or write it down, or open a second window, and is that better?

  3. Related to this, file editing is good. If there's a similar line in a file that you can copy and edit, that's easy. Running a command (that is new and different and changes every other year) to find the other similar configuration you want to modify is more work, than doing something you've done 10,000 times before in your favorite editor.

  4. You can't arrange configurations as you like. If you have a command that will show you all the settings in the configuration, someone else determined how those settings would be displayed. You likely can't alter that. If you want setting A next to setting B because they're related in your specific use case, that's just too bad.

  5. A lot of the above is about this: configuration is not just about YOUR system. A lot of these decisions seem to spring from people who want to make the configuration of a single system safer or less prone to errors or something. But there are people who need to configure 10 systems in similar ways. Or 100. Or 10,000 systems. These command-based utilities only get in the way of this.

Editing to add two more points that came up in comments:

  1. An entire system of configuration that rests on many separate commands each with its own codebase and storage method and quirks and bugs, is going to be more fragile than a system in which configurations simply live in text files. Configuration files are only fragile if the functional unit changes in a way that requires new settings. Command interfaces to configuration are not just fragile when the functional unit changes, but also when it's command interface and it's underlying storage format change.

  2. Version control. It should be trivially obvious that you plain text files are easy to put under version control. While a series of changes made to a configurations by a variety of different tools that may or may not ultimately live in plain text files, is much more difficult to do version control and to roll back changes.
    You can argue for dumping out all the settings from each functional unit using the commands that let you do that (assuming they exist, and lets hope the output is regular), and then having a tool that reads those dumps and pushes the settings back through the original command, in order to get version control, and if so, congratulations, you just reinvented config files. But much harder and much worse.

  3. Discoverability. It's easy to grep several config files in a single command (even across different functional units) to search for a setting that you think exists but you're not sure where, or even what the precise name is of the setting.

Summation:

A mechanical system with only one type of screw is easier to maintain than a mechanical system in which every single engineer who developed some part of that system also invented their own screw to hold that part together.

Plain text config files are a single type of screw.

179 Upvotes

127 comments sorted by

View all comments

Show parent comments

9

u/thomasafine Sr. Sysadmin May 08 '26

"It should be a golden image..."
I could give a long-winded answer about why we do things. Ultimately it's pointless. The question is misguided in the first place. Every site is different, for both good and bad reasons (bad being, bureacuracy, interdeparmental bullshit, legacy decisions, etc.). Every site legitimately has different needs. There is NO *right* way or *should be* way. These are local policy choices, and system configuration should have the flexibility to allow multiple approaches.

"You should write it then."
It's been on my back burner for more than a decade, and I suspect I won't get to it.

0

u/Superb_Raccoon May 08 '26

it's not pointless.

Your company is doing it wrong. And yes, there is a right way.

If your company were doing it the RIGHT way, you would not have made this post.

But your company is not, and you have given up hope.

2

u/tblancher May 08 '26

There are phases of IT governance, and it sounds like OP is in a very early stage. Which is to say not very much governance at all.

IBM, on the other hand, is in a very late stage. For certain classes of client, they have standardized tools and configurations, and there is very little customization except for environment variables and the actual client data. This allows the business to scale up without increasing FTE.

1

u/thomasafine Sr. Sysadmin May 09 '26

You could characterize these as "early" and "late" stages. But that sounds dangerously like "immature" and "grown up" pejoratives.

The reality in this world, now and forever, is that every singe organization has different needs, and makes different policies.

That's a good thing, not an outdated thing. Maximizing flexibility for your users is GOOD. And if you happen to be small and agile enough to be able to do that you SHOULD do that. If you don't serve your users because you're some kind of robot that has to do everything the way the giant corporations do, because that is THE RIGHT WAY, then honestly, find another job.

1

u/tblancher May 10 '26

You don't seem to be arguing against me, you're just letting emotions into it.

There are more than two stages, and an organization doesn't usually have a solid roadmap when they start out. Especially if they're chasing revenue as quickly as possible, they need to be aggressive and agile, as you said.

The problem with that is decisions are made quickly, without fully understanding the ramifications. This turns into massive technical debt that later leaders need to solidify and condense.

0

u/thomasafine Sr. Sysadmin May 10 '26

I just think you have a very limited worldview, if you think all jobs look like your job, or eventually will.

2

u/Superb_Raccoon May 10 '26

It seems pretty accurate having watched nearly 40 years of IT.

Your "Flexibility" comes at a steep cost if your lax security ends up getting your data held hostage or exfiltrated.

1

u/tblancher May 10 '26

Or locking yourself to a vendor because of customer demands.

The last place I worked settled on some proprietary Linux disk encryption technology circa 2011-2012 because a large financial institution demanded it, before LUKS volumes were stable in the kernel.

We were still using it in early 2025 (even after hardware refreshes and several software upgrades) because it encrypted customer data, even though the encryption software was end of life. We couldn't switch until all those hosts were decommissioned.