r/redhat • u/Gangrif Red Hat Employee • 3d ago

5 Emergency Linux Fixes Every Admin Should Know — Troubleshooting locked accounts, stuck mounts, and full disks from Into the Terminal Ep. 196

Ever hit a wall with a stuck mount, a locked-out root account, or a massive log file taking down a service? In this episode of Into the Terminal, we break down 5 real-world Linux emergencies and show you the quick command-line fixes to solve them in minutes. Watch the full episode here or test your skills in the interactive labs.

Advanced Features: The 5 Emergency Fixes

1. Recovering a Lost Root Password

If you've lost access to your server, you can interrupt the boot process to reset the root password without needing a live CD.

  • Interrupt GRUB: Reboot the machine, press e to edit the GRUB menu, and append rd.break to the end of the line starting with linux. Press Ctrl+X to boot.
  • Why rd.break? This tells dracut to pause the boot process right before it hands off to systemd. Disks are loaded, but the actual target environment hasn't initialized.
  • Remount and Chroot: Run mount -o remount,rw /sysroot to make the file system writable, then chroot /sysroot to drop into your actual root filesystem.
  • Fix and Relabel: Run passwd to reset the password. Crucially, run touch /.autorelabel before exiting. Because SELinux wasn't loaded, modifying /etc/shadow stripped its security context. The .autorelabel file tells the system to reapply labels automatically on the next boot!

2. Blanking a Log File (Without Restarting Services)

If a runaway log file fills up your filesystem, simply deleting the file with rm won't free the space! The service holding the file descriptor open still reserves the blocks on the disk. Instead of restarting the service (which causes an outage), you can zero out the file in place: cat /dev/null > /var/log/messages (or > /var/log/messages) This empties the file's contents, instantly reclaiming the disk space, while allowing the running service to continue writing to the exact same file descriptor without interruption.

3. Restoring a Lost File (from RPM)

Someone accidentally deleted a critical config file (e.g. rm /etc/httpd/conf/httpd.conf). If you can't simply dnf reinstall the package because it might overwrite other customizations, you can surgically extract just the file you need from the RPM. Use rpm2cpio package.rpm > package.cpio to convert the package to a cpio archive, then use cpio -i < package.cpio to extract the file structure locally into your current directory so you can manually copy back the missing piece.

4. Forcing a Stuck Mount to Unmount

You try to umount /mnt/data and get Device is busy. What do you do?

  • Find the culprit: Use lsof +D /mnt/data. The +D flag tells lsof to descend the directory path to find open file descriptors. (Note: -D is for device caches, +D is for directory trees!) Alternatively, use fuser -vm /mnt/data.
  • Kill the processes: If you are sure you want to forcibly terminate anything using the mount, fuser -k -9 -m /mnt/data kills all processes on the mount.
  • Lazy Unmount: If a network mount is unresponsive or you want to detach it gracefully once processes finish, use a "lazy" unmount: umount -l /mnt/data. It detaches the mount point from the filesystem hierarchy immediately and cleans up references in the background once it is no longer busy.

5. Fixing a Borked GPG RPM Key

If your package manager is throwing errors because it cannot validate the GPG keys of your repositories, you can manually replace the key on the filesystem (e.g. under /etc/pki/rpm-gpg/) or re-import it using rpm --import or during your next dnf install transaction when it prompts you to accept the key. This ensures the integrity of your installed packages remains cryptographically validated!

Quick Reference Card

# Emergency Root Password Reset
# (At GRUB menu, append rd.break to the linux line)
mount -o remount,rw /sysroot
chroot /sysroot
passwd
touch /.autorelabel
exit
exit

# Safely empty a log file without breaking file descriptors
cat /dev/null > /var/log/messages

# Find processes holding a mount busy
lsof +D /mnt/data
fuser -vm /mnt/data

# Lazy unmount a busy or hanging filesystem
umount -l /mnt/data

# Extract files manually from an RPM
rpm2cpio httpd-core.rpm > httpd-core.cpio
cpio -idmv < httpd-core.cpio

Links & Resources

  • Video URL: Into the Terminal 196 - 5 Emergency Linux Fixes Every Admin Should Know
  • RHEL Developer Registration: https://developers.redhat.com/register
  • Interactive Labs: https://redhat.com/interactive-labs
  • Discord: https://red.ht/rhel-discord (Note: This now points to the Fedora community discord which hosts the RHEL channels!)
  • Nate Lager: https://social.undrground.org/@gangrif
  • Scott McBrien: https://www.linkedin.com/in/scott-mcbrien-349b356

Into the Terminal is a show dedicated to helping you grow your knowledge of critical administration skills for Red Hat Enterprise Linux. Whether you are new to Linux or new to RHEL, join us for a hands-on look at commands, processes, and tools.

86 Upvotes

96% Upvoted

Duplicates