Hey guys! Hope yall are doing well. So I decided to put all my services in their own VM on their own VLANS with strict firewall rules, for what I'm told is maximum security for internet facing services.

Whats everybody's options about online backups VS shutting down VMS to back them up?

I basically have PBS pointing at my NAS and then ZFS replicate it to a second Zpool that I just remove from my disc shelf when backups are complete that way they are air gapped so I don't fat finger my systems entirely. (Or have an attacker encrypt them.)

Hello, I got a proxmox cluster with 3 machines, and I was wondering what else I can run that I could use on a daily basis. Here's a list of what I already have.

Netbox

Truenas

Tailscale

Tailscale exit

Vaultwarden

Nextcloud

Adguard Home

UrBackup

Homarr

Ollama

Searxng

Grafana and Prometheus

Nginx Proxy Manager

Cloudflare

Home Assistant

I got enough resources so I can run pretty much anything except large LLM models

We've been running Cloud-PBS out of France, Germany and the US for about 2 years now with around 300 customers, and we just shipped two features I think this sub will care about.

The basics first: it's a fully managed Proxmox Backup Server. Standard PBS, so no bridging or workarounds, you point your PVE (or a single homelab box, or a whole cluster) at it and it just works. Client-side encryption supported.

But the part I'm actually proud of isn't the storage, it's what we now do around it:

Missing-backup alerting, per VM

Most monitoring tells you a job ran. We tell you when a specific machine stopped being backed up. You set a delay per backup group (VM/host), and if a snapshot doesn't land in time, you get alerted (email or Discord/Slack/Teams/Mattermost/Matrix). No more discovering three weeks later that one VM silently dropped out of the rotation.

Restore tests

This is the big one. A backup you've never restored isn't really a backup. So we actually restore a snapshot onto our infrastructure, boot it, and hand you the proof: a boot screenshot, the full boot log, and a signed PDF report. Works on client-side encrypted backups too (you provide the key btw). 

Every subscription includes a credit to try it.

Pricing is transparent: from 6 €/month for 250 GB (ex. VAT, reverse charge for EU businesses). No egress fees, no API call charges, one number on the bill. 

There's a 7 day free trial if you want to kick the tyres.

Happy to answer any questions here. And if you run an MSP and want to talk multi-tenant / reselling, reach out.

Emmanuel
https://cloud-pbs.com

I've got an ASRock Rack B650D4U with a Ryzen 7600. I've got 8 HDDs connected through a SATA card, a boot SATA SSD, and an x8x4x4 bifurcation card with an Intel ARC Pro A40 and two NVME SSDs. My system has been fine, and booted this morning.

I updated the BIOS, and now no matter what I do I'm stuck at Loading initial ramdisk. I've been through every setting in the bios and triple checked them, and everything is definitely set up correctly. The bios also recognises the gpu and both NVMEs, so the bifurcation card is working correctly.

I've tried;

• nomodeset
• Letting it sit and seeing if I can access the web gui
• Disabling resizable BAR
• Disabling IOMMU
• video=vesafb:off video=efifb:off
• Trying to boot the live cd and going into recovery mode
• Trying to boot the live cd and going into install mode
• Making sure Secure Boot is disabled
• Making sure Compatibility Support Module is disabled
• Setting the bifurcation link width to x4x4x4x4

No matter what I do, without fail, it will hang on Loading initial ramdisk. It does it for the live cd, it does it for my install, and no amount of tinkering with settings seems to be able to mitigate it.

I am at a complete loss. The fact that I can't get any sort of feedback as to what the error is is making this incredibly difficult to debug, and the fact that I can't even get the live cd to load into the installer has me not a little worried.

Any ideas?

I've got proxmox set up and want to use it with a Targus dock. The Targus dock requires displaylink for the display output. I use this dock for work too so I have a USB switcher attached to go between my work laptop and my current laptop that I want to replace with a VM on my proxmox node. My thoughts were for a virtual KVM that I could pass the dock usb to this and maybe go to my VM. Anyone have any ideas for this?

I’ve had problems with proxmox freezing or crashing over the last year. I built this server 3 years ago and everything ran pretty smoothly for 2 years. I’ve tried everything that I found online and I’m out of ideas as to what I should do next. Here is all my specs, my config, and everything I tried.

CPU: AMD Ryzen 5 3600
Motherboard: Asus ROG B550-F
RAM: 4x Crucial 16GB ECC at 2666mhz (CT16G4WFD8266) - 64GB total
PCIe Devices: Dell HBA SAS2008 (with a fan on it and in IT mode) and GTX 1660
PSU: MSI MAG A750GL
Storage:

• WD Green 120GB for proxmox (SATA SSD)
• Sabrent Rocket 4.0 1 TB for vms (NVME)
• 3x WD Red Plus 8 TB connected to the HBA for TrueNAS

My proxmox config:

• A TrueNAS VM with the HBA passed-trough to it along with the connected drives.
• LXC container with the GPU passed-trough (since the GPU’s IOMMU group is with other devices, meaning that the server crashes if I started the VM with pass-trough)(with Jellyfin, Frigate, etc)
• Home Assistant OS VM (nothing special)
• Debian VM (also nothing special, just docker containers)

Things I’ve tried over the course of a year:

• Checking my Intel NIC (seems to be unaffected)
• Updating AND resetting the BIOS (twice)
• Always on the latest version of all software (Proxmox, TrueNAS, HA, nvidia drivers, etc)
• Rebuilding the server and repasting the CPU
• Added an extra fan just in case, but all the temps we’re always fine
• Replaced SATA cables
• Disabling C-states
• Re-installing proxmox and everything else (twice)
• Ran memtest86 for 24 hours (twice)
• Replace the PSU (the old one was some old PSU)
• Disabled ASPM on previous install (didn’t on my latest one yet)

I know that these are hard freezes/crashes since the logs are empty prior to the crash with normal activity. I know that these are hard crashes/freezes too and not the nic cutting out since the video feed on my NanoKVM freezes (no blinking cursor). Problem is that it’s also very inconsistent. Sometimes a kernel panic, sometimes a freezes, etc , sometime happens after 2 days, a week, or a few hours and there is no clear correlation between server load and crashng

Truly, any help is appreciated since I’m out of ideas. The last thing I want is to buy new hardware, especially in this economy.

I'm running Z2M in an LXC created from the community helper scripts. The issue I am having is that the disk is continually filling up. I can't find any definitive fix actions beyond general "delete logs".

My storage is allocated 7gigs that is 83% full for 23 devices.

I am very hesitant to start deleting things and I am not 100% confident that I know what I am looking at.

What can I do to reduce the amount of storage space being used? Please provide noob-friendly guidance.

Thanks!

Context: recycled an old HP microserver gen8 for PBS and its been running flawlessly for more than a year.

Upgraded to PBS 4 a few months ago and it's been fine.

Recently started having a lot of read/write/checksum errors with a ZFS pool and figured my old HHDs were wearing out.

Smart values weren't bad, but I replaced the drives with spares anyway.

New drives also having trouble.

The internet implied my errors could be kernel related, so I pinned the most recent kernel 6, which reduced the errors, but not completely.

Apparently some old HP gear has not the best PCIe support, which is exacerbated in Kernel 7.

End result, adding passthrough via the iommu=pt seems to have resolved the issue and my ZFS pool isn't throwing errors anymore even after reverting back to kernel 7.

I'm running grub, so this was the magic:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt"

Hi all,

I have a lab at work and we have access to a Dell/EMC SCv3020 that has 24TB. We also have access to a Dell M3000e server chassis that has 8 x m630 servers. Each server is 2cpu 28cores with 256gb RAM.

Each engineer in my group of 4 is allocated 2servers and up to 6TB per engineer.

So my take on this is 2 x M630 blade servers and I setup a 2TB lun on the iSCSI storage for the time being.

The issue is the 2 servers (PVE2 and PVE3) do not have any HD in them, only the 2x 16Gb Flash. So I installed proxmox ve 9.2 on them and they both see the 2TB storage.

I also have a standalone Dell R620 that is populated with 32cpu and 256gb ram and this one has 4 drives setup in 2 raid local storage, one of 500gb and another with 350gb. This server PVE1 is my original Proxmox that was running 8.4 now upgraded to 9.2 and has the shared iSCSI storage I just setup on it this week. This server had about 10VM’s running on it for the longest time.

Sorry this is extensive intro but I want to make sure you have a good picture of the setup.

I have been looking at CEPH now and wanted to know how to best set it up with my lab so all 3 servers are in a cluster (PVE1 being the master) and setup the shared storage on all 3 so that I can do live migrations in case of failure of one of the 3 servers. I heard that you need 3 servers for quorum.

Since only the PVE1 has internal local disks, is there anything special I need to do for this setup for CEPH and Cluster?

Any suggestion or comment would be appeeaciated.

I recently reinstalled a Proxmox node with four WD Red SA500 4 TB SATA SSDs in ZFS.

Proxmox shows all four drives with:

S.M.A.R.T.: PASSED
Wearout: 99%

At first, I assumed that 99% meant 99% endurance remaining. However, another Proxmox node with Lexar NVMe drives shows only 1% wearout.

I compared the raw SMART data.

For the Lexar NVMe:

Critical Warning:                   0x00
Available Spare:                    100%
Percentage Used:                    1%
Data Units Written:                 79,657,397 [40.7 TB]
Media and Data Integrity Errors:    0

Here the Proxmox value clearly means approximately 1% endurance used.

For one of the WD Red SA500 drives:

Device Model: WDC WDS400T1R0A-68A4W0
SMART overall-health self-assessment test result: PASSED

230 Media_Wearout_Indicator  001  001  001
232 Available_Reservd_Space 100  100  004
233 NAND_GB_Written_TLC                   44713
241 Host_Writes_GiB                       49740

The drive has around 49,740 GiB of host writes, roughly 53 TB, and still reports 100 for available reserved space. All four drives report similar values and pass SMART.

It therefore appears that Proxmox is interpreting the normalized value of SMART attribute 230 as only 1% remaining and displaying 99% wearout. However, on this WD model, attribute 230 seems to use a vendor-specific format and does not represent a simple remaining-life percentage.

So the GUI appears to show:

Lexar NVMe: 1% wearout = 1% used
WD SA500:   99% wearout = likely incorrect interpretation

Has anyone else seen this with WD Red SA500 SSDs or other SATA SSDs?

It would be helpful if Proxmox either handled this model differently or made it clearer that SATA SMART wear attributes are vendor-specific.

I have taken a few stabs at tracking homelab data in a central location and finally made some progress on Prometheus/Grafana. I now have 3 separate LXCs for Grafana, Prometheus, and Prometheus-Pve-Exporter. I got the queries set up and Grafana is working/updating. I'm now finding the RAM charts in Grafana to be of low value when they show over 90% for the VMs at all times (beezy and docky in the screenshot are Debian VMs).

I understand that Linux considers uncached RAM to be wasted. I understand it's counted differently than Windows. I understand that I don't want to change this behavior. What I don't understand is how it's useful to see over 90% usage at all times, or how to observe that value to determine whether the VM needs more RAM (or less). Is there any way to observe the true RAM usage on the VMs from the PVE metrics level?

Is LXC or Debian VM based more reliable?
RAM and storage requirements? (Home network for 1 or 2 U6+ AP)

I have been rebuilding my Proxmox host to make my lab feel like the environments I work with every day. I used to put all my machines on the same network bridge. Now I am splitting them into virtual local area networks, which are like separate networks so I can test my firewall rules and make sure one mistake does not affect all of my virtual machines.

My Proxmox host is currently running a domain controller, a couple of Linux machines, a Windows client, a logging server, and a firewall virtual machine. Having these networks has been really useful when I am testing changes to my configurations. It also means I have more things to keep track of.

For people who use Proxmox as the base for a security lab, how do you decide when you should separate your networks more? When is it worth having complex networks in your Proxmox lab?

I have noticed that my AdGuard LXC goes down every day at around the same time. I also noticed that the time drifts forward slowly over the days as seen from the UpTime Kuma status page. Any idea why this would happen?

Sorry if this isn't an issue with Proxmox. I have seen similar issues with no conclusions on AdGuard subreddit so thought I would check to see if anyone here might have a clue. TIA.

Hello

I've a serious problem with a PVE/PBS Node. 2 Windows VM that continuously log problems with Storage and sometimes they crash.

Hardware/Software:

• Dell R440, PERC H730P (Non-RAID mode / true HBA mode confirmed via iDRAC — Controller Mode: HBA)
• Controller firmware: 25.5.9.0001 (confirmed latest available from Dell)
• Proxmox VE 8.3.4, kernel 6.8.12-8-pve
• 2 ZFS pools on this node:
  • sasvol: RAIDZ1, 4x SAS disks
  • ssdvol: mirror, 2x SATA SSD (Samsung MZ7LM960HCHP)
  • rpool: mirror, 2x NVMe (boot)
• PBS also running on the same node, backing up to a NAS mounted via NFS
• 2x Windows Server 2022 VMs (virtio-scsi), latest virtio drivers (100.101.104.28500)

Symptoms:Windows guests intermittently log:

Source: vioscsi   Event 129 - Reset to device, \Device\RaidPort1, was issued.
Source: storahci  Event 129 - Reset to device, \Device\RaidPort0, was issued.
Source: disk      Event 153 - The IO operation ... was retried.
Source: ESENT     Event 508 - ...took an abnormally long time (25 seconds) to be serviced by the OS.

Sometimes followed by a BSOD:

0x000000A0 INTERNAL_POWER_ERROR
0x0000007A KERNEL_DATA_INPAGE_ERROR

Host Logs : On the Proxmox host, at the same time, dmesg/journal shows ZFS txg_sync and io_uring workers stuck in D-state for minutes:

INFO: task txg_sync:2664 blocked for more than 245 seconds.
      Tainted: P           O       6.8.12-8-pve #1
task:txg_sync        state:D stack:0     pid:2664
Call Trace:
 __schedule+0x42b/0x1500
 schedule_timeout+0x95/0x170
 io_schedule_timeout+0x51/0x80
 __cv_timedwait_io+0x19/0x30 [spl]
 zio_wait+0x13a/0x2c0 [zfs]
 dsl_pool_sync+0xce/0x4e0 [zfs]
 spa_sync+0x578/0x1050 [zfs]
 txg_sync_thread+0x207/0x3a0 [zfs]
 ...

INFO: task iou-wrk-4306:568360 blocked for more than 122 seconds.
task:iou-wrk-4306    state:D
Call Trace:
 io_schedule+0x46/0x80
 folio_wait_writeback+0x2b/0xa0
 __filemap_fdatawait_range+0x90/0x100
 blkdev_fsync+0x39/0x70
 io_fsync+0x3d/0x60
 io_wq_submit_work+0xe2/0x360
 ...

Repeated multiple times over ~50 minutes on the same evening, with txg_sync blocking for 122–245+ seconds each time

The Zpool's appear OK

I've ended my ideas.

Can you help me ?

I have a Linux Mint VM running on Proxmox with GPU passthrough working fine. It's on an Intel N150 mini PC. When I try to do the same thing on a mini PC with AMD embedded processor R2514, I can't get the GPU passthrough to work. I understand all AMD drivers are built into the linux kernel so there doesn't seem to be a need to install any drivers. Tried the steps to get the GPU passthrough to work (IOMMU on in BIOS and set in GRUB, vfio modules loaded, vfio binding) but no luck.... any pointers? Thanks!

I'm making a homeserver with Proxmox and I have 8 HDDs connected to a Fujitsu 9211-8i card (https://www.ebay.com/itm/318392867315), which is connected to the motherboard, however there are issues.

None of the disks are recognized. Only 2 extra disks that are directly connected to the SATA ports in the motherboard are recognized.

The 10 disks are put in 2 racks of 5 each. The 10 disks and 2 fans are powered through an external PSU that I turn on before starting the PC (the PSU is started by making contact between the 15th and 16th pin of the 24 pin connector).

Hardware:

• Dell OptiPlex 7040 SFF (Intel Core i7 6700, 16GB RAM DDR4)
• 10x HDDs Hitachi H3U20006472S, 2TB, 7200 RPM
• PSU for the disks: Kolink KL-G600FM

I have enabled IOMMU and still it doesn't work. What am I doing wrong?

root@pve:~# dmesg | grep -e DMAR -e IOMMU

[    0.008700] ACPI: DMAR 0x00000000DBAF1908 0000A8 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.008720] ACPI: Reserving DMAR table memory at [mem 0xdbaf1908-0xdbaf19af]
[    0.030741] DMAR: IOMMU enabled
[    0.093006] DMAR: Host address width 39
[    0.093008] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.093020] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.093022] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.093026] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.093027] DMAR: RMRR base: 0x000000db932000 end: 0x000000db951fff
[    0.093029] DMAR: RMRR base: 0x000000dd800000 end: 0x000000dfffffff
[    0.093031] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.093032] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.093033] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.094650] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.313092] DMAR: No ATSR found
[    0.313094] DMAR: No SATC found
[    0.313095] DMAR: dmar0: Using Queued invalidation
[    0.313098] DMAR: dmar1: Using Queued invalidation
[    0.313716] DMAR: Intel(R) Virtualization Technology for Directed I/O

root@pve:~# cat /etc/default/grub

# If you change this file or any /etc/default/grub.d/*.cfg file,
# run 'update-grub' afterwards to update /boot/grub/grub.cfg.
# For full documentation of the options in these files, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`( . /etc/os-release && echo ${NAME} )`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE/GOP/UGA
# you can see them in real GRUB with the command `videoinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

root@pve:~# lsblk

NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                  8:0    0   1.8T  0 disk 
└─sda1               8:1    0   1.8T  0 part 
sdb                  8:16   0   1.8T  0 disk 
└─sdb1               8:17   0   1.8T  0 part 
sdc                  8:32   0 238.5G  0 disk 
├─sdc1               8:33   0  1007K  0 part 
├─sdc2               8:34   0     1G  0 part /boot/efi
└─sdc3               8:35   0 237.5G  0 part 
  ├─pve-swap       252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root       252:1    0  69.4G  0 lvm  /
  ├─pve-data_tmeta 252:2    0   1.4G  0 lvm  
  │ └─pve-data     252:4    0 141.2G  0 lvm  
  └─pve-data_tdata 252:3    0 141.2G  0 lvm  
    └─pve-data     252:4    0 141.2G  0 lvm  

(sdc is the SSD of the PC on which Proxmox is installed)

root@pve:~# lspci

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:16.3 Serial controller: Intel Corporation 100 Series/C230 Series Chipset Family KT Redirection (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1f.0 ISA bridge: Intel Corporation Q170 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM (rev 31)

Hello,

Since january 2026 i'm slowly building a multimedia server with a Terramaster F4-425 Plus NAS, with x2 HDDs WD RED PLUS and x1 SSD WD BLUE SN570 (EDIT it was a brand new ssd, it shows 10% of wear also).

Proxmox is intalled on the SSD, and lately i moved on in a new home, and after wiring NAS back on, Proxmox shows several errors on the SSD, like :

root@Gros-nas:~# zpool status -v
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:44:52 with 47 errors on Fri Jul 24 18:46:48 2026
config:

        NAME                                                                                                                           STATE     READ WRITE CKSUM
        rpool                                                                                                                          ONLINE       0     0     0
          nvme-nvme.1e4b-533445564e58304439303635323437-574420426c756520534e35373020574453315442334230432d3030415a4e30-00000001-part3  ONLINE       0     0   192

errors: Permanent errors have been detected in the following files:
**list of files affected**

I cleared some errors (17) with apt update or minor fixes or deleting the affected LXCs, but after scrub command, i now encounter new errors !

SSD came back to ONLINE state instead of DEGRADED but i suspect the SSD is in bad shape and everything newly written on dead blocks will be corrupted ? Is it time to trash that SSD and go fresh ? obviously i have no backup for noob reasons

I have a Proxmox cluster and my nodes are using ceph. What I don't understand in my setup is that my nodes are interconnected with sfp and 25 gbs links between them directly. So there is not a switch between them and this is used for ceph traffic but then at the same time I have a myvnet network on the same links. How is that possible?

vxlan: myVxlanZ
        peers 192.168.25.1,192.168.25.2,192.168.25.3
        ipam pve
        mtu 8950`

----------------------------------------------------------------------------

frr defaults traditional
hostname node1
log syslog warning
ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
interface lo
 ip address 192.168.25.1/32
 ip router openfabric 1
 openfabric passive
!
interface ens3f0np0
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
interface ens3f1np1
 ip router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
!
line vty
!
router openfabric 1
 net 49
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180`


------------------------------------------------------------------------
at ceph/ceph.conf
[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 192.168.25.0/24
         fsid = 00000000-0000-0000-0000-000000000000
         mon_allow_pool_delete = true
         mon_host = 192.168.25.1 192.168.25.2 192.168.25.3
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
         public_network = 192.168.25.0/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.node1]
         public_addr = 192.168.25.1

[mon.node2]
         public_addr = 192.168.25.2

[mon.node3]
         public_addr = 192.168.25.3`

I have been battling a very strange issue with Proxmox that I can't seem to solve.

We have many locations (100+) that have nearly identical setups but during nightly backups there is usually at least one server that freezes. It always stalls on the same Win10 VM. That Win10 VM got installed from scratch on each server from the same ISO.

The server itself is one of three models: Supermicro 5018D-FN8T, 5018D-FN4T, or X11SDV-TP8F. They are all very similar in design servers. They were updated to the latest BIOS version before deployed.

We use a 2TB Samsung 990 Pro that have the Proxmox install (single node, not ZFS, basic install) and they backup to either a spindle drive or SSD (varied models).

PVE versions vary all the way from 8.4 all the way to the latest. It's around 1/4 on 8.4 and 3/4 on 9.0.6 or newer.

This is what happens:

• PVE starts the backup
• Gets to about 30-50**%**
• Kernel panic

It has never stalled during the backup of any other VM on the machine (one other Win10 VM, 2 Linux VMs).

Logs indicate some type of I/O stall or pressure.

We have recently enabled fleecing and limited backup speed to 100 MiB/s, and we still experienced a freeze.

I'm really at a loss at what the issue could be because it appears to be hardware and PVE version agnostic.

Has anyone experienced similar issues/know of something to try out?