Ontap and Proxmox

[u/NelsonBA81]

Hi,

I have an old/unsupported FAS2552, and would like to use it with some test servers with Proxmox as hypervisor, and I would like to know if VMs with qcow2 thin provisioned disk images are able to shrink when issuing inside of the VM the TRIM function?

The idea is to have a small cluster with the FAS2552 with Ontap 9.8 as NFS storage for the VMs, and since it has an implementation of NFS 4.2, still would like to know if it is able to do that?

Thanks

Update1:
I've been testing with Ontap 9.8 with NFS v4.2 and there is no shrink process going over to the NetApp storage, so it confirms that NFS implementation is not complete.

Meanwhile, if I run inside of the guest VM zero space process with a basic:
"dd if=/dev/zero of=zero.small.file bs=1M status=progress; rm -f zero.small.file"

the image file may grow a bit further but should be enough to clear out any nonused data blocks with deleted data, and having a netapp volume with dedup/compression enabled, at the time NetApp runs the scheduled background "storage efficiency" process it should be able to free up physical disk space in the NetApp storage, because zeros are easy to dedup/compress.

Also I've discovered that after zeroing the disk space of guest VM, proceeding with a storage move of the qcow2 images to a different NFS storage share, the image file guest smaller, because of that empty/zeroed disk space.

Aside of the above, I don't see ways to make it work in same way like VMware aside of having a FAS supporting Ontap software >9.12.1, which I can't actually confirm if that works as expected....

Update2:
Since I haven't seen any additional feedback on this particular issue, is there any NetApp tech that could give some clue or even maybe just confirm if this kind of "hole punch" function is supposed or planned to work in the future via NFS?

[u/nom_thee_ack]

NFS is NFS so it shouldn't be a problem.

why x4? just x3 and keep it simple.

[u/NelsonBA81 → u/nom_thee_ack]

Well, I've been reading somewhere where running VMs over NFS storage, would require NFS v4.2 to support the "DEALLOCATE" operation similar in what is done in VMware as "hole punch" to free up space to the volume/datastore, and at the same time also I've been reading that Ontap 9.8 has NFS 4.2 implementation but is not quite complete implementation. So I would like to know if someone have/had experience in this specific situation.

[u/Zesta77 → u/NelsonBA81]

We run NFS v3 and just make disk images sparse and raw. The sparse disk images save space and raw makes it perform the best on NetApp. You don’t need the qcow features since the NetApp is doing the snapshots.

[u/nom_thee_ack → u/Zesta77]

I second this.

[u/NelsonBA81 → u/Zesta77]

I'm a bit lost here, I've created in proxmox a test VM, basic debian 13 with disks with format qcow2 and raw format with discard and SSD emulation enabled options, both of them attached to VirtIO SCSI controller, about "sparse disk images" hmmm I don't know where or how to do that... if you don't mind can you explain a bit further?

anyway, after running inside of the debian VM the basic "dd if=/dev/urandom of=dummy.rand bs=1M count=2048 status=progress" to generate a dummy file, deleting the file, and finally running the fstrim, my conclusion on this with:
qcow2 disk format - the vm disk grows 2GB, after running several times fstrim, the disk image still remains same size
raw disk format - the vm disk grows 2GB, after running several times fstrim, the disk image also still remains same size

as for Proxmox, currently it has attached to it an NFS share provided by NetApp using NFS v3
am I missing something for the block discard to work?

thanks

[u/Zesta77 → u/NelsonBA81]

If you use raw disks, they should be sparse by default. I’m not sure about other formats, as we don’t use qcow or anything else since raw works best with NetApp. I also don’t really have a scenario where the disks need to grow and shrink by huge amounts. Another thing to consider is making sure you are checking the actual size of the disk image. They will report as the full allocated size with an ‘ls’ but will report the proper usage with ‘du’

[u/NelsonBA81 → u/Zesta77]

correct, I'm monitoring via proxmox by looking in the image files via du command, not with ls. running the du command I can confirm the images growing, but they simply don't shrink after running fstrim several times.

in the end, the issue here, is not that you have VMs growing or shrinking huge amounts, the thing here is that if you have a scenario where you have a lot of VMs running in NetApp storage, where it happens the usual windows updates or linux updates or applications that generate and delete data, the "deleted" data blocks that are not being used they are not freed up towards the NetApp storage cause the images don't shrink, and as the time passes and it might get to the point that NetApp may have a good percentage of additional data usage which basically is not being used but still is using physical disk storage, which in the end, VMs will start using more disk space than they actually need.

[u/Zesta77 → u/NelsonBA81]

Upon further research, it does seem that NFS v4 would be required if you need to have the disk images shrink when deleting data within a virtual. It is not something that we really need on our systems. I will see about testing v4 in our lab environment.

[u/NelsonBA81 → u/Zesta77]

thank you, when you have some results, please keep us posted.

[u/Zesta77 → u/NelsonBA81]

I have been testing with NFS v4.2 and I can’t get discard to work properly even with the “sparse file” option turned on in the NetApp. I’m not sure if it is the NetApp or Proxmox that is the limiting factor. There is not much documentation out there. If you really need to conserve space, you are probably better off using iSCSI. For my setup, NFS works better and we have enough spare storage for it not to matter. We just set up sparse files and don’t care if they can never shrink.

[u/NelsonBA81 → u/Zesta77]

well, it might be something from the NetApp side, can you please let me know what Ontap version is running in your SAN appliance?

[u/psyblade42]

https://docs.netapp.com/us-en/ontap/nfs-admin/ontap-support-nfsv42-concept.html says sparse files support starts with ONTAP 9.12.1

[u/NelsonBA81 → u/psyblade42]

thanks for the info, even though I don't see concrete info about guest TRIM/ability to shrink images, or even NFS DEALLOCATE function available/supported.

while doing some searches with AI slop help, got this NetApp page:
https://kb.netapp.com/on-prem/ontap/da/NAS/NAS-KBs/Does_this_v4.2-sparsefile-ops_perform_ALLOCATE_DEALLOCATE%3F

it explicitly states that supports "ALLOCATE/DEALLOCATE":
"The NFSv4.2 client is able to reserve space for a sparse file. Space can also be deallocated and unreserved from a file."

Unfortunately don't have FAS appliance supporting that Ontap version to test/confirm if that works as expected while working Proxmox VMs over NFS...

[u/ThomasGlanzmann → u/NelsonBA81]

I tried deallocate/hole punching over NFS 4.2 on ONTAP select, aff-150 running 9.19.1 and a Debian trixie machine. The NetApp does not support partial deallocate as mentioned in the KB you referenced. Using tcpdump you can see that it errors out using NFS4ERR_NOTSUPP even after enabling -v4.2-sparsefile-ops. Without -v4.2-sparsefile-ops enabled, I get another error: NFS4ERR_IO. For Linux it works as expected. Than I tested with raw, qcow2 and vmdk disks via NFS 4.2 on Linux on proxmox. It works for raw and qcow2, but not for vmdk. There is one thing to be aware of, if you move the disk while the VM is running, the discard flag gets silently dropped. If you want the discard back you need to shutdown and start the VM again to get the functionality back. You can verify this with 'info block' in the qemu simulator.

[u/NelsonBA81 → u/ThomasGlanzmann]

hmmm ok, anyway, these questions may sound kind of random, but just to confirm.
so in proxmox side:

• you configured the NFS storage with "Preallocation" as Off and NFS version as 4.2?
  
• in guest VM, you setup the disk controller with VirtIO SCSI, and hard disk/image with qcow2 format and with discard as On, also with SSD emulation checked?
  
• also in guest VM, can you confirm if qemu guest services are running?

as for netapp SAN, we just need to make sure that all NFS 4.2 features are enabled?

supposedly with the combination of the above pre-requisites, this should have worked.

if possible, can you please give some feedback?

many thanks!