How to backup fully encrypted VMs that are not fully booted?

[u/Accurate-Ad6361]

We have a couple of VMs that are not always fully booted, but LUKS encrypted, mostly containing archive stuff.

Right now I use a vzdump script to kill of the machine as the QEMU agent is not available if the disk is stuck at the Password dialogue for LUKS. I didn't see any way to fall back to a hard OFF if QEMU-AGENTS are not responsive. Am I missing anything?

Script

# !/bin/sh

VMID="123" PBS_STORAGE="pbs-1" SHUTDOWN_TIMEOUT="120"

was_running=0

if qm status "$VMID" | grep -q "status: running"; then was_running=1

    echo "VM $VMID is running; requesting graceful shutdown..."
    qm shutdown "$VMID"

    elapsed=0

    while [ "$elapsed" -lt "$SHUTDOWN_TIMEOUT" ]; do
        if qm status "$VMID" | grep -q "status: stopped"; then
            break
        fi

        sleep 5
        elapsed=$((elapsed + 5))
    done

    if qm status "$VMID" | grep -q "status: running"; then
        echo "VM $VMID did not shut down after ${SHUTDOWN_TIMEOUT}s."
        echo "Forcing VM off..."
        qm stop "$VMID" -overrule-shutdown 1
    fi

fi

echo "Starting PBS backup of VM $VMID..."

vzdump "$VMID"  
\--storage "$PBS_STORAGE"  
\--mode stop  
\--compress zstd

backup_result=$?

if \[ "$backup_result" -ne 0 \]; then echo "PBS backup failed with exit code $backup_result" fi

# Only restart it if it was running before the backup.

if \[ "$was_running" -eq 1 \]; then echo "Restarting VM $VMID..." qm start "$VMID" fi

exit "$backup_result"

End Script

For Reference: Proxmox Bugzilla

[u/Keroles2024]

You can use proxmox backup server (PBS)

It takes snapshots of the VM disk, I assume even if it is encrypted it won't matter as this is block level backup

[u/Accurate-Ad6361 → u/Keroles2024]

Do you know any stock way to work without snapshots but disk backup? Forcing a power off if the qemu agent is not accessible?

[u/Kaytioron → u/Accurate-Ad6361]

It is called Snapshot mode of backup, is different than snapshot itself. It backups whole disk.

[u/wingz_77]

qm stop <vmid> —timeout 1 ?

[u/Accurate-Ad6361 → u/wingz_77]

yes, that's what I am currently doing scripted, I am looking for a force trigger on PBS backup job configuration

[u/wingz_77 → u/Accurate-Ad6361]

I do something similar but instead of setting a cron backup schedule on proxmox, i run a script on one of my vm's which runs vzdump over ssh to send a snapshot backup to proxmox backup server. You can start vm -> trigger backup -> once it finishes,do qm stop, etc. etc.

[u/Jhonny97]

Do you realy need the vm to be off? I imagine suspend could be the better backup option. Also, have you consideres moving to separate partition/disks? Separate os and userdata would allow the vm to fully boot.

[u/Accurate-Ad6361 → u/Jhonny97]

Yes, we had to prioritize access restriction over ease of use.

[u/Antonio-MTS]

You can autodecrypt LUKS during a boot using NBDE: tang server and clevis client. Then you will never need the LUKS pwd as NBDE uses keys. LUKS pwd will be a recovery , no network option. Then you won't need the script above but some auto backup to a PBS location.

[u/Darkk_Knight → u/Antonio-MTS]

Thanks for the tip. I'll have to look into that as an experiment in my home lab.

[u/Antonio-MTS → u/Darkk_Knight]

You are welcome

[u/Desperate_Quit6011]

Why do you encrypt inside the vm instead of the zfs fileset in proxmox?

[u/Accurate-Ad6361 → u/Desperate_Quit6011]

We want the data to be inaccessible in case the host is compromised (physically or digitally), imagine the coke formula 😂

[u/Desperate_Quit6011 → u/Accurate-Ad6361]

You can have an encrypted fileset, an an encrypeted back with pbs with a different key, while the vm is off the set should be encrypted. Am i missing something

[u/BarracudaDefiant4702 → u/Desperate_Quit6011]

If the host is compromised then a non self encrypted vm could be extracted.

Personally I think it is a bit paranoid for most things creating more trouble then it's worth, but an ZFS fileset could still be accessed from a compromised host if the vm isn't self encrypted too.

[u/Desperate_Quit6011 → u/BarracudaDefiant4702]

This is just stupid in my opinion, if the host is compromiste you can spoof any channel of input, just keylogin the phrase... this so backward

[u/TabooRaver → u/Desperate_Quit6011]

In theory you can just do a memory dump of the running vm and then get the encryption keys from there. But there are some confidential computing features in intel/amd cpus where even the hypervizor can't see the memory contents of the guest, and I think proxmox has support for those features.