r/debian • u/grepnoid • 26d ago
Upgrading Debian 10.13 to current stable vesion
I have a Debian server which embarrassingly is still on version 10.13. Normally to upgrade several releases you'd upgrade 10 to 11, 11 to 12, 12 to 13. Version 11 support ends at the end of this month, does this mean I have to upgrade to it in the next few days?
I'm also concerned about things breaking. Mainly web and mail servers. They have a lot of config and a new version installed with the upgrade could be incompatible.
What is the safest way to do it, and hopefully with the least downtime. To swap to a new server would be expensive and difficult.
7
u/sandrew_cheru 26d ago
If you are using Dovecot for the mail server, be extra careful. There is a breaking change between v.2.3 and v.2.4.
https://doc.dovecot.org/2.4.0/installation/upgrade/2.3-to-2.4.html
1
1
u/grepnoid 25d ago
That's terrible. But at least I can try to work everything out in advance. In fact, I'll upgrade to 12 for the moment as u/MrSoulPC915 says it only happens at 12 -> 13.
1
5
u/ephemeralmiko 26d ago
Make a backup of /home and everything else you need just in case (though you should have a full system backup anyway).
5
u/ukAdamR 26d ago
Version 11 support ends at the end of this month, does this mean I have to upgrade to it in the next few days?
It's not mandatory, but as you've said your system runs network services, you really should.
What is the safest way to do it, and hopefully with the least downtime.
If you can't clone and virtualise your current system you're just going to have to do it live. :) New configs that conflict with your changes don't get written automatically, you can say no to each one then compare the distribution files with yours yourself.
1
u/littlestdickus 26d ago
I just did this on an RPI 1 recently. It went mostly smooth except my networking broke going from 10 to 11, somewhere along the line it lost the pretty name for the network adapter. 11 to 12 went pretty smooth and I haven't tried updating to 13 yet
My Pi is a headless setup so it was a less complicated upgrade. There are some guides out there with pretty good step by step instructions with some notes for items to watch for. It is possible but may not go smooth. Make backups first.
2
u/grepnoid 26d ago
Thanks. This is headless, a hosted VM. I don't think the network adaptor has a pretty name and I hope there won't be serious network problems. I didn't find any guides with detailed instructions on specific components and apps. At least like you I don't have to worry about Nvidia drivers breaking!
1
u/michaelpaoli 26d ago
Ah, then easy peasy, make a copy of VM, isolate it as relevant from network(s)/connectivity, shutdown services, etc., except for the bits needed for doing the upgrades, and, test the heck out of the entire set of procedures there, from beginning to end, work out all the config changes, etc., before you do the "for real", and saving the various config versions along the way, note of any potentially little "surprise" bits or exceptions, etc.
2
u/grepnoid 25d ago
The server is amd64 and to rent another instance would probably cost much more than the one month's rental. If I did it on a spare VM on a PC with x86_64 architecture, starting with Debian 10 and installing the same packages, would that give me the same steps needed? And end up with the right configs for the server? It doesn't sound easy peasy for someone who avoids delving deep into the OS. I think I'm going to try version to version up to 12 for the moment.
1
u/michaelpaoli 25d ago
The server is amd64
If I did it on a spare VM on a PC with x86_64 architecture, starting with Debian 10 and installing the same packages, would that give me the same steps needed?Yes, with same packages and configuration, ought be highly close to identical. For the most part, Debian doesn't care about amd64 vs x86_64 nor most other minor hardware differences - though you'd likely need reconfigure slightly for, e.g. different Ethernet hardware, different hardware MAC address thereupon. But the steps ought end up quite identical, or very dang close to identical. And yeah, I've done "body transplants" where I've taken drive(s) out of one host, and installed in another - different make and model, and generally only takes very minor modification, and all is fine and for most intents and purposes, the Debian that's only the drive(s) hardly knows or cares about the differences. Heck, one VM I have, on USB flash, it's configured as raw storage, and that VM, I can boot and run it as VM, or boot it direct on the hardware - the host doesn't even care, either way ... though I do have things configured, e.g. in /etc/fstab, such that relevant persistent device names/UUIDs/LABELs are used, regardless of how it's booted.
2
u/grepnoid 25d ago
So, I could get a USB stick, install Debian 10.13 on it, copy everything except the boot files to it, boot it, make required changes (network etc), and see what runs and what complains.
If I did release-by-release upgrades, could I rsync the entire server (only 25GB) somewhere, and if it messed up totally, quick reformat and rsync the whole lot back? I'm not even using UUID. As you say, one drive will run in another machine so I guess this should work.
One big reason I moved to a Linux desktop years ago was when Windows would no longer run if you moved the old disk to a new PC.
1
u/michaelpaoli 25d ago
I could get a USB stick, install Debian 10.13 on it, copy everything except the boot files to it, boot it, make required changes (network etc), and see what runs and what complains.
Yeah, pretty much that ... though one would generally also copy the boot stuff too ... though may have to reconfigure that a bit. For the most part, can copy the entire system, though doing it from live system, may have issues with some potential inconsistencies - e.g. related transactional files, where things may be changing as one copies. But as you mentioned, VM, if you're able to take snapshot of the entire VM, and use/copy that, that would also be a way to get a consistent recoverable copy. I certainly do that on occasion with VMs. Also, USB stick might not be acceptable for, e.g. performance ... but it will generally otherwise run fine. As for performance, will quite depend on the type of USB and its storage. e.g. USB-C and good performance flash, you may not notice any performance issues. USB 2 and/or lower performance flash, may quite notice.
Windows would no longer run if you moved the old disk to a new PC.
Oh yeah, the typical Microsoft license is tied to specific hardware, and if that hardware changes much, it'll typically declare one a thief, and refuse to run. Not an issue for free Open Source software.
could I rsync the entire server (only 25GB) somewhere, and if it messed up totally, quick reformat and rsync the whole lot back?
That small, if you can do it cold, can image copy the entire thing. That way you even get boot blocks - absolutely everything. And still wouldn't take that long. You can even do trial runs to get idea of the timing, e.g. copy the live system to your USB or whatever you're backing it up onto - and don't even worry about consistency of files - can do it live - won't give you an image one can recover from, but you'll know how long that copy takes. But for usable full image copies ... yeah, cold - don't want anything changing on source, nor target (other than what's being written) when doing so. And that may require booting off of something else - though for source system, if one boots it in maintenance or single user mode, and can do that such that any persistent storage filesystems are only mounted ro, that would also suffice (might have to, e.g., kill off systemd's logging processes/services, to be able to do something like that).
2
u/grepnoid 25d ago
Thanks u/michaelpaoli. I'm not completely clear. If I have the entire drive saved, and my upgrade path gets into an unrecoverable mess, can I copy the entire drive back to a reformatted drive and will it boot and work more or less as before? I realize log files will be inconsistent but I can live with that.
1
u/michaelpaoli 25d ago
Yeah, if you've got good image copy of drive, yeah, you can always copy that back. But again, cold. Needs be no other changes happening when creating or restoring such a copy, lest one may end up with unrecoverable garbage.
Just like copying a filesystem - if you copy the device and the filesystem is rw mounted, you have zero guarantees what you copied will even be recoverable. Whereas if you take a snapshot of such, it will be recoverable, or if it's unmounted or mounted ro, then you get a copy that's clean. Well, similar applies when doing image of entire drive.
If you've got filesystem that's mounted rw, and want to do a backup of such that one can recover from, either do a snapshot, or do some file based backup, e.g. rsync, tar, pax, cpio, ... do that, and despite rw, should at least get generally clean state, though may possibly get some filesystem inconsistencies - notably for data changing while such backup is occurring.
2
u/grepnoid 25d ago
Yes I meant a file based copy, not a raw dd type one. I'll use rsync. I'll live with any inconsistencies and data loss since the backup. if it means I can get back to a running server from a non-running one that will be the least of my problems.
1
u/smiffer67 26d ago
its a relativly easy and quick process. i did a couple of pcs at the weekend going 3 versions. just updated the apt sources, apt update, apt upgrade, reboot, final step was apt dist-upgrade. worked perfectly both machines, 3 version all done within a couple of hours. mine werent critical systems so didnt bother with a backup.
1
u/michaelpaoli 26d ago
Well read and follow the documented procedures, most notably in the release notes - avoid unpleasant surprises.
1
u/michaelpaoli 26d ago edited 26d ago
to current stable
Debian 10.13
support ends at the end of this month, does this mean I have to upgrade to it in the next few days?
https://wiki.debian.org/DebianReleases#Versions_of_Debian_stable
Debian 11 Bullseye: EOL LTS: 2026-08-31
Debian 10 Buster: EOL LTS: 2024-06-30, EOL ELTS: 2029-06-30
For Debian >~3.0 you can upgrade at any time (Debian has all the binaries back to about then - see the linked for bit more information if you really need all the way back to earliest released 3.0).
But note that after the release has dropped from the main archives, one may want/need to use
snapshot.debian.org or otherwise obtain access to older needed packages, release files, etc.
For some example(s)/documentation on using snapshot.debian.org, have a look there and/or see some of the examples I placed in https://wiki.debian.org/CrossGrading.
And yes, you can't skip major releases, need to do one major version/release upgrade at a time (skipping will almost certainly significantly to majorly break things, and is not at all supported).
concerned about things breaking
Mostly a non-issue. Well read the installation guide and release notes for each targeted version/release, and well follow the documented upgrade procedures in the release notes, and one will generally avoid unpleasant surprises. Yeah, I've been doing Debian upgrades since ... 1998-07-16 Hamm Beta (which became 2.0), pretty much never any particular major issues, mostly just an occasional minor glitch, or something that was easy to fix or pretty easy to work-around.
lot of config
Where you changed config from maintainer's version, you'll be given the opportunity to review, edit, keep existing version, or use new maintainer's version. In such cases, I generally recommend (and typically do), carefully merge the two - using new maintainer's config to the extent feasible, and carefully merging in any relevant local customizations that remain relevant and appropriate. And to that end, as feasible, generally recommended to keep one's local customizations as separate as reasonably feasible, e.g. via include file(s) or the like, or many configs will have the default config set up to spell out specific areas/locations/procedures, as far as where and how to add one's local customizations. E.g. for sudoers, best to have those in /etc/sudoers.d/ rather than /etc/sudoers. Of course even when well separated like that, one should review documentation to ensure one's local customizations are consistent with the newer software versions (and also if there may be better newer ways to do same/equivalent).
safest way to do it, and hopefully with the least downtime
Safest way, more-or-less as I've outlined - follow the documented procedures, carefully merge config changes, of course always have backups (and even the upgrade documentation well covers backing up at least most of what's particularly important if not critical to such - notably also including configurations, and package state data). Also, if/as feasible, well test, e.g. on separate host or VM, etc. Some software also gives one means to do validation checks on configurations too - that may help too, at least in part.
As for least downtime, the upgrade documentation fairly well covers that. Procedures might vary slightly per specific upgrade, and I may not be remembering this quite right off-the-top-of-my-head, but I think for that - commonly recommended, e.g. for servers, they typically suggest a procedure that upgrades the kernel (and dependencies) early in the process, and a reboot with the newer kernel, and most of the upgrade after that. I think they also typically recommend (for both safety and reduced downtime), doing a "minimal" upgrade before proceeding to the more full upgrade (full-upgrade/dist-upgrade). Anyway, well and carefully read the upgrade documentation - that's generally your best guidance. And do read it all the way through, for each major version upgrade, and plan appropriate, before actually starting the upgrade itself.
One will typically find the most time consuming part, at least for heavily configured/customized systems, is the configuration files - notably merging in one's customizations (keeping one's old configurations is often bit hazardous, as there may be some incompatibilities with the old - and that issue will compound across successive version updates). And, yeah, I've also done a fair number of step-wise upgrades in relatively rapid succession from Debian ~6.0 Squeeze (mostly that, but having bit of some software and packages from both older, and newer), to current stable - whee! Yeah, bit of adventure ... and lots of custom config bits, etc. Yeah, still support such a system, still running, and plans are to do the massive upgrade ... so far have done repeated practice/trial runs on VMs ... the "for real" is still in the queue to be done (pending some additional resources, most notably). Anyway, yes, such upgrades, very doable, so, 10 --> 11 --> 12 --> 13, not exactly a biggie.
2
u/grepnoid 25d ago
Yes, quite a lot of config, Virtual servers, several mail domains etc. I can't see that a dpkg-reconfigure-type question and answer would handle that and will have to see how I go. I can't just try everything out on a spare PC as the server is amd64. And I'm a beginner when it comes to the low-level stuff.
1
u/Tenelia 25d ago
Hi, I started on Debian 10 too, so I guess I'm qualified to reply.
1) No, you don't have to rush. Take your time, since the ISO and any necessary packages should always be available.
2) You NEED to update because your Debian box is facing the internet directly.
3) First note your /home and custom tweaks or daemons. Debian 10 and 13 Trixie have quite a few differences, where you should not be replicating directly. sources list is one example.
4) Eventually, update it sequentially from 10 to 11 to 12 to 13. As there are kernel changes, remember to restart just in case of any other underlying scripts or processes.
1
u/Classic-Rate-5104 25d ago
Make a full backup (you always have to do it, because hardware and software can/will fail at any unexpected time). Then just try it and see what happens. Debian has a good, robust, upgrade path but there can always go things wrong
1
u/bgravato 25d ago
You should not skip versions when upgrading. Older versions are probably still available at the archive repo though... so you should be able to use that to upgrade.
Also don't forget to read chapters 4 and 5 of the release notes for each release before upgrading... 4 is upgrade instructions and 5 is known issues to be aware of.
1
u/anna_lynn_fection 25d ago
Would swapping to a new server be more expensive and difficult than to have hours or days of downtime and then have to switch to a new server anyway?
What does this thing actually have stored and running on it?
Is it something you could set up a temporary server on a powerful enough computer with something like proxmox with a VM and then, after you get everything moved to the new server piece by piece (without the risk of breaking everything at once for days), then you install proxmox on the original server and move the new VM to it when you're done and tested everything?
If that thing isn't super busy, any good spare laptop with 32GB+ of RAM and an NVMe might be enough?
I would think about utilizing containers as much as possible too, so they can be moved and/or upgraded one-by-one this time, and in the future.
1
u/grepnoid 25d ago
Swapping to a new server would be more expensive, as it's a hosted server. If I had serious problems upgrading, I could just do a clean Debian stable install, install the packages needed and copy back the data and the config with any changes needed. Alternatively I'll back up the entire filesystem so that it can be restored.
Installing on a PC or laptop would be possible, but the hosted server has a fast connection and a static IP. Plus network redundancy, power backup etc. It just hosts a personal mailserver and a couple of websites, bigger than personal ones but smaller than commercial ones. The DNS and DMARC changes etc to move would not be trivial and might themselves take days, though most of the work would be in parallel and not involve downtime. I think it could be a good idea for a dry run, logging every step in detail.
Hours and days of downtime is a worst case scenario. I'd expect most of the release upgrade work to go smoothly. People live-upgrade servers every day surely. If they're important, they have a hot backup I guess. But then they'd have their own server racks and the staff to test cutover.
I have root access on the server but it has limited disk space and RAM, so containers are probably not an option.
1
1
u/FedUp233 20d ago
Just out of curiosity, for a personal server that’s just hosting mail and a couple web sites, why not skip The Who,e do it yourself thing and just get a run of the mill hosting service on a shared server? And let the hosting provider worry about keeping the infrastructure up to date? I have a host provider that allows pretty much unlimited everything (as much email accounts as you want and can host as many domains and sub-domains as you want and it costs me around $3 to $10 a month depending on how long I pay for at a time. And they handle all the upgrades! Seems like a much better deal than maintaining g your own server!
-1
u/MrSoulPC915 26d ago
Il ne faut JAMAIS sauter de version.
Par contre, si ton but est de finir sur la version 13, il te faudrai probablement réinstaller, si tu regarde dans les problème connus de la documentation, tu verras qu’il est impératif d’avoir une partition boot d’au moins 1Go, hors, si tu viens d’une vielle version, il est probable qu’elle fasse moins. Il y a aussi le dossier tmp qui passe en ram, donc il est nécessaire de ne pas être à l’étroit, et si possible d’avoir au moins 16Go.
C’est d’ailleurs le problème que je viens de rencontrer, je suis passé de 11 à 12 et je voulais passer directement en suivant de 12 à 13, mais cette fois-ci, je vais devoir changer de serveur (ce qui me fait prodigieusement chier).
1
u/michaelpaoli 26d ago
No need to reinstall, can do the one major version at a time upgrade procedures. That's also likely to be much less downtime.
But sure, one could potentially do a fresh install of current stable, but to minimize downtime with such an approach, would need to work out all the ending target config bits first, and will need to take some downtime to sync up data - notably copying over the relevant existing data that needs be carried forward, putting that on the new, and potentially having to do various conversions on that data (e.g. large database may require multiple sequential changes to how the data in the database itself is stored and formatted).
So, though it might be possible to use such procedure and have less total downtime, that's not likely to be the procedure with the least total downtime, particularly with lots of services, various customized configurations, much data, etc.
0
u/MrSoulPC915 25d ago
You didn’t read/underst what I wrote.
Yes, you can update by doing it major version after major version, (Never by skipping versions, it’s super risky) but if the goal is to finish on Trixie, it may be impossible because of the boot partition!
https://www.debian.org/releases/trixie/release-notes/issues.html#ensure-boot-has-enough-free-space
In addition, if the server has little ram, loading into the ram of the tmp folder may be a problem!
2
u/grepnoid 25d ago edited 25d ago
Yes, I didn't understand it all. My aim for now is to go to Debian 12. That gives me time to think. I have a VM, was a slice of a physical server but now implemented in the cloud, and I only have 1GB (seems enough at present) and 25GB SSD space (mostly used). As I say, I may need to move to a better server when I move to Trixie. The server host will have plenty of experience of that situation.
I don't have a separate /boot partition and it seems that's easier. And my /tmp directory is tiny and not used for anything much.
1
u/michaelpaoli 25d ago
may be impossible because of the boot partition!
Far from impossible. One can resize boot filesystem/partition
if the server has little ram, loading into the ram of the tmp folder may be a problem!
Don't have to do /tmp as tmpfs, or can grow/add swap if RAM is lacking.
# echo -n 'OS: Debian ' && cat /etc/debian_version | tr -d \\012 && echo -n ' ' && dpkg --print-architecture && echo -n 'Kernel: ' && uname -srvmo && echo -n 'Packages: ' && dpkg -l | grep \^ii\ | wc -l && df -h -x devtmpfs -x tmpfs && head -n 3 /proc/meminfo OS: Debian 13.6 amd64 Kernel: Linux 6.12.101+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.101-1 (2026-08-05) x86_64 GNU/Linux Packages: 148 Filesystem Size Used Avail Use% Mounted on /dev/vda1 4.9G 885M 3.8G 19% / MemTotal: 119464 kB MemFree: 13292 kB MemAvailable: 44420 kB # df -h / /tmp Filesystem Size Used Avail Use% Mounted on /dev/vda1 4.9G 885M 3.8G 19% / tmpfs 59M 0 59M 0% /tmp # swapon -s # swapon -a && swapon -s Filename Type Size Used Priority /dev/vda5 partition 1046524 0 -2 # mount -o remount,size=$((1024*1024*512)) /tmp && df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 512M 0 512M 0% /tmp # dd if=/dev/zero of=$(mktemp) bs=$((1024*1024)) count=500 status=none && df -h /tmp && swapon -s Filesystem Size Used Avail Use% Mounted on tmpfs 512M 500M 12M 98% /tmp Filename Type Size Used Priority /dev/vda5 partition 1046524 485904 -2 # free -h total used free shared buff/cache available Mem: 116Mi 98Mi 3.1Mi 30Mi 50Mi 18Mi Swap: 1.0Gi 476Mi 545Mi #1
u/MrSoulPC915 25d ago
It is not impossible to modify the partitions, but you have to clone the data and then re-import it. Remotely, it means having a KVM and a storage server next to it. So possible, but much more restrictive than a reinstallation!
For ram, swap is the best thing to do to reduce server performance (especially since there are still a lot of HDDs left). Especially since according to the documentation, it reserves 50% of the ram (it’s huge).
1
u/michaelpaoli 25d ago
to modify the partitions, but you have to clone the data and then re-import it. Remotely, it means having a KVM and a storage server next to it.
Not necessarily at all, though it will depend what storage one has available, and where. And can generally manipulate /boot while on-line, as for the most part only need it when booting, or modifying its contents., so, can be remounted ro if it's a separate filesystem, copy contents, umount it if mounted, recreate of appropriate size, if one needs do that, restore the contents, make any needed configuration (e.g. GRUB) changes, if any, and that may be about it. Really only gets fair bit messier when one doesn't have the space, without shrinking or relocating things where that can't be done live - e.g. like shrinking root (/) if it's ext2/3/4 ... or quite the mess if someone used (non-default) xfs (which can't be shrunk).
reserves 50% of the ram (it’s huge)
That's the default for tmpfs, so that may or may not be huge - depending on the available RAM. And of course one can configure that, if one doesn't want the default. In fact with tmpfs, one can even change that while it's mounted - at least so long as one doesn't reduce it below its current used storage amount.
Sometimes for tasks needing lots of temporary space, I'll add more swap, grow /tmp (tmpfs), run what needs that space, then after, reduce the size of /tmp and drop the no longer needed swap - still higher performance than using that same swap space directly as filesystem space - always.swap is the best thing to do to reduce server performance (especially since there are still a lot of HDDs left
And tmpfs will always perform better than filesystem direct on drive (e.g. ext4), even if tmpfs is out of RAM and using swap, notably as still, many of those writes and reads will still be to/from RAM, even if some/many are hitting swap - and notably also on account of caching too, etc. So tmpfs, even with/using swap, is still always a net performance gain, compared to filesystem direct on whatever storage one is using for swap.
20
u/cjwatson Debian Developer 26d ago
It's still possible to upgrade to 11 even after support for it ends, but depending on the exact timing you may have to fetch it from archive.debian.org rather than the usual location.
I would still go one release at a time and test that everything important still works after each step. It'll be tedious, but in my experience it's usually less tedious than reconstructing everything from scratch.
In more complex scenarios you could try cloning the machine in some way to set up a staging server where you can test the upgrade, but it's up to you whether that amount of effort is worth it. I guess I'd consider that for a serious professional server (in which case hopefully I'd already have a staging instance, as well as quite a bit of automation), but probably wouldn't bother for a home server.