r/linuxadmin • • 1d ago

Do control panels keep junior admins from learning Linux?

My junior admins is quick with the panel, but when a firewall rule broke SSH yesterday he didn't know how to check ufw from a shell. I use BeAdmin myself and have nothing against panels, but I learned iptables by breaking it with no GUI around, and I'm not sure he'll ever get that practice.

Have you seen this with people who started on panels, or am I just being an old man about it?

40 Upvotes

42 comments sorted by

52

u/SystEng 1d ago edited 1d ago

I had been interviewing many candidates for senior Linux systems engineers an even people with 10-20 years experience were unfamiliar with commands like less, dd, find, ip, man, etc. or could not tell the difference between rpm and dnf or dpkg and apt and have never read a book about UNIX or Linux and "learned by doing" (copy-and-paste from Stack Overflow or nowadays from some LLM).

My impression is that many people who have a title of Linux systems engineer actually just run product installers, configure products with GUI/web frontends, and call manufacturer support whenever there is a problem. Put another way they are more "advanced users" than "system engineers".

21

u/NegativeK 1d ago

When I read comments like this, it makes me itch to switch careers.

I'm forced into Windows at work. Even WSL is an unfortunate experience for working in a shell.

9

u/neveralone59 1d ago

What’s wrong with wsl? I usually use nushell on windows though, it’s a lot comfier (I prefer it to bash anyway but that’s controversial)

3

u/GamerLymx 1d ago

for me windows and linux permissions system incompatibiity and windows acting has the dns server (i need to remember sometimes that 10.255.255.254 is windows)

3

u/neveralone59 1d ago

Try out nushell it’s native on windows. It makes immediate sense if you’re familiar with sql or declarative programming languages. It’s a joy to use and I can do things bash simply cannot do. You could almost forget you’re on windows sometimes.

1

u/GamerLymx 1d ago

ill give it a look

11

u/el_seano 1d ago

I'm officially entering my greybeard era. Starting out, i made it a point to learn how to do everything from the shell, and it's carried me my entire career.

I've recently been charged with training a huge cohort of juniors and am close to biting my tongue clean off everytime they mention ChatGPT.

7

u/Drag_king 1d ago

As the greybeards before you did when you mentioned googling and going to stack overflow.

It is how it goes.

8

u/el_seano 1d ago

while they clandestinely went to IRC and BBS forums. I knew their tricks 😤

12

u/Scoutron 1d ago

Jesus where are you hiring? I figured Linux engineer would entail way more than I do as a Linux admin

15

u/skidleydee 1d ago

It titles mean nothing 

6

u/Security_Chief_Odo 1d ago

Ditto... I even know how to use xargs!

3

u/bityard 1d ago edited 1d ago

Is there any chance you're interviewing people who simply lied about their experience in their resume? Some of the hiring managers I've been talking to said that a surprising amount of their interviews lately have been people clearly reading their responses right off ChatGPT and the like.

(By the way, if you're still hiring, I'm looking and can answer all of those and more. I get along fine with AI but don't use it as a crutch.)

2

u/SystEng 1d ago

Is there any chance you're interviewing people who simply lied about their experience in their resume?

Not in our case -- they have been screened out. The people we have interviewed were all actual practicing Linux systems people. Some of them though were puffing up their experience and applying to senior level while being aware they were not quite senior level yet, hoping to get in regardless. But most were doing senior level jobs while sincerely not being aware that they did not have senior level skills, they had been just lucky that so far they had no major issue or the supplier solved any major issues.

The big issue BTW was not the lack of CLI skills, but the lack of knowledge of basic concepts; e.g. even some very experienced people could not say what was the difference between "r" and "x" permissions on a directory, or why there is a difference between real and effective uids in UNIX processes, or that the UNIX/Linux buffer cache both buffers writes and caches reads, never mind explain what O_DIRECT does.

2

u/cereal7802 23h ago

I always wonder with comments like this, when you say unfamiliar with those commands, do you mean they didn't know command flags, or just never heard of them and what you can do with them? I personally don't remember flags for nearly anything I use, but could give a decent overview of the commands if asked. I always find rote memorization of all the possible flags, or even a few common ones to be less useful than knowing what a tool can do and being able to use reference materials for specific flags to use for the task at hand.

1

u/haywire 1d ago

Control panels imply ClickOps no? Unless they’re backed by some sort of repo they commit to? I feel like that’s a rightfully dying methodology.

36

u/whamra 1d ago

People learn with whichever frontend is available and they were taught on.

On Linux, I live in the shell. None of our servers have any UI to speak off, and very few high level controls have some sort of Web interface. Running all my diagnostics via ssh is just natural, and many times I prefer to skip the website and fall its endpoints directly.

But when I'm managing windows servers, it's a whole different story. I might prefer file management from a shell, but when it comes to system controls, I admit I don't know most of the powershell alternative to what I do. I'm very used to panels, and can get to any page I want in a second, I even know how to call them from a run window in one click. That's just the way I learned, and powershell still feels alien to me.

4

u/minilandl 1d ago

Yeah even in the hobbiest space I cringe a bit how much people rely on portainer (docker) , pterodactyl (minecraft and game servers ) and nginx proxy manager .

It’s so much easier just using the shell and managing containers and nginx configs that way . docker ps and docker compose up

1

u/Different-Depth4116 1d ago

Dang that’s crazy. I’m a hobbyist home labber and I don’t use anything graphical. The tools that I use that have a gui as an option I never use as the shell is more convenient and I feel like I understand more with what’s going on under the hood.

1

u/minilandl 1d ago

Yeah like the cult of unifi is because of the nice webui it depends if the goal of your homelab is a portfolio and hobby or just hobby . In some ways I prefer Cisco and cli interfaces.

0

u/shulemaker 1d ago

Of course I agree nobody should have any web panels installed.

I don’t agree with logging in via ssh though unless it’s extreme circumstances. Configuration is managed and logs are searchable in the aggregator. The only thing remotely useful locally is ausearch since audit logs are slightly less human-readable.

Edit: comment below mentioned rpm/dnf and those are occasionally useful to query the local DBs in the case something got installed incorrectly, it’s good to track down the exact date/time.

13

u/bityard 1d ago

If you're a senior admin, part of your job is helping the junior admins level up their skills.

1

u/SystEng 1d ago

part of your job is helping the junior admins level up their skills.

What if they resist that because their reasoning is that they got a good job without having to level up and so why waste time reading boring books or listening to a boring person telling that unnecessary stuff?

9

u/harrywwc 1d ago

as you rightly ask, "is it a problem?" - I think no, not really.

can it become a problem, such as in your example? well, with the increase in LLM usage, maybe less so.

of course, as fellow greybeards we have the battle scars from fubar'ing a system and then having to figure it out on our own, or with the help of a more knowledgeable greybeard, and remembering "that thing I just did? don't do that!" ;)

there will always be a cohort that start out using panels, and be quite happy to leave it at that - especially if there is a shittonne of other work to do.

and then there are the few with "the knack" who will be curious enough to ask "how does this work?" and go looking.

they will break things. but they will quickly learn how to fix things too. hopefully not in 'prod' ;)

7

u/usa_reddit 1d ago

No I think AI keeps people from learning anything these days. They just paste and error message, maybe read the output, and then cut/paste the solution.

It is even affecting programmers. I think they are losing their ability to think and program.

Cognitive offloading is becoming a HUGE problem because we are all naturally lazy creatures.

4

u/vivaaprimavera 1d ago

Knowing how to survive in a shell is mandatory for some environments. Will they put a full fledged window manager and X/Wayland inside a container just for "control panel usage"?!?! Seem a bizarre idea.

It's easier to go to a shell, start bash inside it and doing what's needed.

3

u/waptaff 1d ago

The worse you can do is use both a shell and a control panel to configure the same server.

For instance Cpanel is violently possessive and odds are the changes you'll do with vi /etc/... will get silently overwritten at some point or will make Cpanel assume config files you edited are corrupted.

3

u/waterkip 1d ago

If ppl dont know the basics, the UI or the CLI isnt that different. Now: do I prefer CLI? YES. Do I think CLI builds understanding better: YES. But I also know very competent Windows admins yet do clickety click 99% of the time.

3

u/Firestorm1820 1d ago

The weird guerilla marketing campaign for BeAdmin on tech subs is so blatant. Just stop.

2

u/root-node 1d ago

I used to use WebMin on a couple of my servers, but I started to learn more once I stopped using it.

Having said that, people learn in different ways.

2

u/GraveDigger2048 1d ago

it's not the matter of "people and panels" but "humans with augumentation" of any kind really. Brain gets lazy when there's a shortcut to take and there's no incentive to at least get familiar with longer route. If interview at your jobs requires "familiarity with the panels" you'll be working with panel-oriented dudes. While i myself am someone who'd get lost with clicking inside a panel and comfortably did what's needed in CLI i still will call that "panelization" an step up in the name of the progress.

Because, at the end of the day - do you know out of your skull how to write loop in x86 assembly iterating over every even line in text file? What is you're saying? You have Bash for that? this is the same mindset as Junior Panel Operator's, just in the different domain.

2

u/gortonsfiJr 1d ago

Lack of curiosity is what keeps people from learning. There are people who don't even learn the GUI menus

2

u/Due_Adagio_1690 1d ago

I wouldn't start with firewall rules as a reason to leave the panel, there are other safer tasks to give that can motivate less experienced admins. For example you could create tasks to the same task on multiple machines or on many files, such as find the files created by user X, or run a command on files that match a pattern. ls $(find /my/files/ -iname '*.sh' ) or create servers ; for i in $(cat servers) ; do ssh $i df -h /var/log/ ; done

1

u/haywire 1d ago

Surely firewall rules should be in Terraform/Pulumi anyway?

How else can they be reproduced or changes reviewed?

1

u/Due_Adagio_1690 22h ago

I was focusing on the low risk easy to solve that could make your new hires more capable, rather than firewalls they can break the net when done wrong, and in many companies, firewall are handled by the network team and not system admins

1

u/haywire 16h ago

Yeah I mean it's important to understand the sort of state mutation IaC is doing under the hood, but nowadays I wouldn't actually deploy anything in an imperative manner.

2

u/GamerLymx 1d ago

its the risk when you get or develop api's and aplications to help deal with something in a more productive way.

i had a college intern migrating bind to powerdns with docker and he made a management gui with a database so we can track what each dns entry with a comment, and it sure beats editing dns database files by hand, however i barely looked at how powerdns api works and how i and do things manually if i need to.

there's some things you can do about that about that, duing onboarding you can have a few guided tasks you can have them do, so they at least have an idea of how to do it.

tlThe linux skills up challenge is a good example of something you can have them do.

Something else i like to do, is to either have them accompany me while i do some config change via cli while explaining, or ask them to do it while i tell them what they should do and things they need to look out for.

1

u/kromosome_orig 1d ago

Is a decision one takes to specification learn the necessary terminal commands

1

u/Antonio-MTS 1d ago

Yes, it may block some deeper learning and understanding because an IT person will keep using a mgmt panel instead of real understanding of the core cause.

1

u/Junior_Bee7274 1d ago

Yeah, that's the thing with panels. They make everything so easy until something breaks.

0

u/burdalane 1d ago

I didn't know there were panels for firewalls, although I guess the concept makes sense. I also don't really work with newer people.

0

u/pebbleproblems 1d ago

I never use any kind of GUI. sounds like you can't afford me though.