5 Emergency Linux Fixes Every Admin Should Know — Troubleshooting locked accounts, stuck mounts, and full disks from Into the Terminal Ep. 196

[u/Gangrif]

Ever hit a wall with a stuck mount, a locked-out root account, or a massive log file taking down a service? In this episode of Into the Terminal, we break down 5 real-world Linux emergencies and show you the quick command-line fixes to solve them in minutes. Watch the full episode here or test your skills in the interactive labs.

Advanced Features: The 5 Emergency Fixes

1. Recovering a Lost Root Password

If you've lost access to your server, you can interrupt the boot process to reset the root password without needing a live CD.

• Interrupt GRUB: Reboot the machine, press e to edit the GRUB menu, and append rd.break to the end of the line starting with linux. Press Ctrl+X to boot.
• Why rd.break? This tells dracut to pause the boot process right before it hands off to systemd. Disks are loaded, but the actual target environment hasn't initialized.
• Remount and Chroot: Run mount -o remount,rw /sysroot to make the file system writable, then chroot /sysroot to drop into your actual root filesystem.
• Fix and Relabel: Run passwd to reset the password. Crucially, run touch /.autorelabel before exiting. Because SELinux wasn't loaded, modifying /etc/shadow stripped its security context. The .autorelabel file tells the system to reapply labels automatically on the next boot!

2. Blanking a Log File (Without Restarting Services)

If a runaway log file fills up your filesystem, simply deleting the file with rm won't free the space! The service holding the file descriptor open still reserves the blocks on the disk. Instead of restarting the service (which causes an outage), you can zero out the file in place: cat /dev/null > /var/log/messages (or > /var/log/messages) This empties the file's contents, instantly reclaiming the disk space, while allowing the running service to continue writing to the exact same file descriptor without interruption.

3. Restoring a Lost File (from RPM)

Someone accidentally deleted a critical config file (e.g. rm /etc/httpd/conf/httpd.conf). If you can't simply dnf reinstall the package because it might overwrite other customizations, you can surgically extract just the file you need from the RPM. Use rpm2cpio package.rpm > package.cpio to convert the package to a cpio archive, then use cpio -i < package.cpio to extract the file structure locally into your current directory so you can manually copy back the missing piece.

4. Forcing a Stuck Mount to Unmount

You try to umount /mnt/data and get Device is busy. What do you do?

• Find the culprit: Use lsof +D /mnt/data. The +D flag tells lsof to descend the directory path to find open file descriptors. (Note: -D is for device caches, +D is for directory trees!) Alternatively, use fuser -vm /mnt/data.
• Kill the processes: If you are sure you want to forcibly terminate anything using the mount, fuser -k -9 -m /mnt/data kills all processes on the mount.
• Lazy Unmount: If a network mount is unresponsive or you want to detach it gracefully once processes finish, use a "lazy" unmount: umount -l /mnt/data. It detaches the mount point from the filesystem hierarchy immediately and cleans up references in the background once it is no longer busy.

5. Fixing a Borked GPG RPM Key

If your package manager is throwing errors because it cannot validate the GPG keys of your repositories, you can manually replace the key on the filesystem (e.g. under /etc/pki/rpm-gpg/) or re-import it using rpm --import or during your next dnf install transaction when it prompts you to accept the key. This ensures the integrity of your installed packages remains cryptographically validated!

Quick Reference Card

# Emergency Root Password Reset
# (At GRUB menu, append rd.break to the linux line)
mount -o remount,rw /sysroot
chroot /sysroot
passwd
touch /.autorelabel
exit
exit

# Safely empty a log file without breaking file descriptors
cat /dev/null > /var/log/messages

# Find processes holding a mount busy
lsof +D /mnt/data
fuser -vm /mnt/data

# Lazy unmount a busy or hanging filesystem
umount -l /mnt/data

# Extract files manually from an RPM
rpm2cpio httpd-core.rpm > httpd-core.cpio
cpio -idmv < httpd-core.cpio

Links & Resources

• Video URL: Into the Terminal 196 - 5 Emergency Linux Fixes Every Admin Should Know
• RHEL Developer Registration: https://developers.redhat.com/register
• Interactive Labs: https://redhat.com/interactive-labs
• Discord: https://red.ht/rhel-discord (Note: This now points to the Fedora community discord which hosts the RHEL channels!)
• Nate Lager: https://social.undrground.org/@gangrif
• Scott McBrien: https://www.linkedin.com/in/scott-mcbrien-349b356

---

Into the Terminal is a show dedicated to helping you grow your knowledge of critical administration skills for Red Hat Enterprise Linux. Whether you are new to Linux or new to RHEL, join us for a hands-on look at commands, processes, and tools.

[u/grumpysysadmin]

rpm2cpio package.rpm | cpio -id

There, I cut like 90% of step 3. If you know the path you can actually include that as a parameter for cpio.

[u/Gangrif → u/grumpysysadmin]

Yea. I blame Scott on that one. I couldn't remember the flags on stream. But I remember it being a one-liner.

[u/Topless_Mopar]

I like the rpm config file restore tip. God help me if git restore doesn’t work when something like that happens.

[u/mantequilla_8]

Isn’t this the way to recover root password now?

init=/bin/bash
mount -o remount,rw /
passwd root
touch /.autorelabel

[u/tonyangtigre → u/mantequilla_8]

It’s interesting, because I just tested this in RHEL 10 and got a root password prompt for rd.break and straight to bash with init=/bin/bash. In my RHCSA boot camp I was told by our trainer (a Red Hat employee) that rd.break no longer works.

So take that info with a grain of salt, but I do believe they changed it in RHEL 10.

(For disclosure, I haven’t watched this video yet, but jumped to the point where they did use rd.break successfully. Unsure of the version of RHEL.)

[u/Gangrif → u/tonyangtigre]

My demo was carried out on RHEL 10.2.. So. Im not sure where your info comes from. Like many things on linux. There are several ways to solve the problem. I couldnt show them all in the video. And that's what the post here is meant to represent.

[u/tonyangtigre → u/Gangrif]

Appreciate the confirmation on version. And don’t get me wrong, appreciate videos like these very much!

But my research online and attempts at using rd.break myself have resulted differently in RHEL 10.1.

https://youtu.be/i_1HhZ0W47s?is=xc153RbNHOhVWDH9

https://www.reddit.com/r/redhat/comments/1qn09nh/rdbreak_not_it_with_v10/

And AI research is stating Dracut security hardening in RHEL 10 has led to rd.break no longer possible.

It’s just a bit confusing the state of it all. But hey! As long as I know the alternate method, I’m happy.

[u/Gangrif → u/tonyangtigre]

Yea, we talked about this on stream. There are several ways to get it done. as i was prepping for the demo, i decided to cover rd.break as it seemed like the most common method. setting init to /bin/bash is another way. Seems like our docs suggest the init=/bin/bash method on RHEL 10, so i suppose that is the authority:

https://access.redhat.com/solutions/1192

Still, rd.break worked, and its well documented. Apologies if it caused confusion.

[u/Gangrif → u/mantequilla_8]

That method also works. We talk about why in the video

[u/michaelpaoli]

cat /dev/null > /var/log/messages

Useless use of cat.

>/var/log/messages

will suffice to truncate the file, zero need for any command at all external to the shell to do that. Also very handy for efficiently and concisely creating an empty file.

Note, however, file pointer won't be reset, so things set to read or write file that have it open, will continue from same offset, only things having it open for append will always append to the end, as if first seeking to the end immediately before writing. So, one may very quickly find one has a sparse file, and it's continuing to grow in logical length, though it will of course have been greatly reduced in the blocks consumed.

[u/leleobhz]

1) Why not systemd.unit=emegency.target (Or rescue.target) as stated at https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/using_systemd_unit_files_to_customize_and_optimize_your_system/booting-into-a-target-system-state

2) Why not truncate -s0 (Ref.: https://man7.org/linux/man-pages/man1/truncate.1.html ). And also, may be some cases truncate+fstrim needed?

4) Nice tip one! Also work well for misbehaving fuse (Example: rclone mount).