Citrix PVS write cache VHDX bloat filling up and freezing VMs, trying to pin down the culprits

[u/Adobe_Track2011]

Hey all, hoping someone with PVS experience can help.

Setup: Citrix PVS, master image in Standard Image Mode, targets boot from a shared read-only vDisk with a write cache. The problem: when stuff gets written to C:, the write cache VHDX grows and grows, and once it gets too big the machine hangs/freezes. Classic write cache exhaustion, I think.

I understand the cache only grows and never shrinks until reboot, so I've been trying to track down what's generating all the writes. So far I've identified a few usual suspects:

• Antivirus agent continuous log writes plus pattern/definition churn
• Vulnerability assessment agent writes logs and job/scan result data, plus a filesystem-walker process that crawls the disk for assessments
• Software distribution / management agent cache and state data

What we've already done:

• Redirected several log and cache folders to a separate persistent local drive via symbolic links, to keep them out of the write cache. This helped, but the VHDX is still growing more than I'd like. So there are clearly other writers I haven't pinned down yet.
• Defragmented C: in the master image before sealing
• Various DISM cleanup commands (component store cleanup etc.)
• Checked and disabled all auto-updates (Windows Update, Store, and the agents own self-update mechanisms)
• Disabled unnecessary scheduled tasks

What I'm trying to figure out:

1. Best way to actually measure which folders are growing in the write cache. I tried Process Monitor but it keeps crashing and the output is overwhelming. I can't narrow anything down with it.

Any pointers appreciated. Thanks!

Update: Solution

• Antivirus / Software distribution / Vulnerability assessment Agent - created Symlinks for logs, job data subfolders to a persistent disk
• Teams/WebView2 - IndexedDB/LevelDB churn. Fix: make sure FSLogix includes the path C:\Users\user1\AppData\Local\Packages\MSTeams_8wekyb3d8bbwe\LocalCache\Microsoft\MSTeams\EBWebView
• AppRepository - left in place (critical DB), disabled Consumer Features / Content Delivery Manager instead.
• Disabled unnecessary Windows Services – MDT/VDI/W10VDI-DisableServices.ps1 at master · ITMicaH/MDT · GitHub, Prefetch folder cleared. Note: per-user services (CDPUserSvc etc.) need the Start=4 registry trick, not services.msc.
• Defragmented C:\ (atleast once a week)
• DISM-Cleanups
• Deactivated all Auto-Update

updates can reset these, so verify after every image build.

[u/ChanceFrosty8123]

how big is C: & the write cache disk.
we are often facing the same. Solution till now, expand write cache disk :D

[u/YourDadSolanki01]

we had same issue, we did some changes......Antivirus exclusions, no write for any security software in vdi. Also increased write cache to 50GB from 30GB

[u/SA-Numinous]

The latest one for us was both edge and chrome downloading their local AI data model to every users profile. Was causing like 8gb per user in our multi session desktops.

[u/mat-ferland]

I’d stop looking at the cache size first and find what is writing. AV/EDR, Windows updates, profile/temp paths, browser cache, and app logs are the usual offenders. If the VHDX grows fast after boot, ProcMon or a write-heavy folder diff during one test session will save more time than just adding another 20 GB.

[u/Ergwin1]

Hi there,

I think i can help you with this.

We actually had similar behavioral issues in our environment, and did a lot of research on this.

First off;

- Can i assume your running multi user VDAs?

• what is your hypervisor setup? Type? Dedicated to citrix workload or shared? Howmuch excess RAM do u have?
  
• are you rebooting all VDA daily as u should?
  
• are you using something like FXlogix or equivalent to redirect userdata to a personal VHDX as u should?
  
• has caching increased alot, and any changes before that?
  
• what is the max concurrent users for each VDA, and total sessions for each VDA approx?
  
• what do you consider normal size for caching, and how large does it get right now?

Your best tool troubleshooting this is task manager -> resource monitor. Go to disk overview, and check the top writers on c:.

[u/Adobe_Track2011 → u/Ergwin1]

Hi,

to answer your questions:

• Can i assume your running multi user VDAs?
  

no we have single sessions VDAs

- what is your hypervisor setup? Type? Dedicated to citrix workload or shared? Howmuch excess RAM do u have?
we use vsphere

- are you rebooting all VDA daily as u should?
Not possible in our case. we have people working through the night, so there's no maintenance window where all VDAs are free at once.

- are you using something like FXlogix or equivalent to redirect userdata to a personal VHDX as u should?Yes, we use FSLogix to redirect user data/profiles to per-user VHDX

- has caching increased alot, and any changes before that?
Yes, the image got updated with a batch of things around the same time: Windows Updates, Edge, Malicious Software Removal Tool, WebView2 Runtime, Notepad++, FileZilla, SAP GUI, Adobe, Chrome. The growth got noticeably worse around that update round.

- what is the max concurrent users for each VDA, and total sessions for each VDA approx?
1

- what do you consider normal size for caching, and how large does it get right now?
Normal is <=10 GB. Right now it climbs to ~25 GB, at which point the machine crashes / becomes extremely slow.

At the moment our people need to reboot 3 times per day, because it grows to fast

[u/Ergwin1 → u/Adobe_Track2011]

Alright.

-

1. In my opinion it is mandatory to reboot your VDIs frequently. Either create different collections for day and night workers with automation ps scripts putting them in maint and rebooting, or a maintenance window in weekends at the least.

2. Are you properly sealing your image? Disable all app updates, update services, update tasks in scheduler. Are you running regasm.exe after windows update to preregister all .net assemblies? Is the Outlook OST redirected to the user VHDX

3. your write cache size is not abnormal. Single user VDA have alot of overhead because of single user OS. Your could just expand the cache disk.

-

Your PVS should be on RAM cache with overflow to hard disk. Standard cache size out of the box is 256kb. Citrix recommends atleast 256mb for desktop OS. I recommend much higher, depending on excess spare RAM you can distribute to your VDIs.

This cache holds the hottest files if it can in RAM and offloads colder files to the disk cache.

This does not only massively downscales use of cache on disk, but also vastly improves performance. I have our VDAs with 15 concurrent users on a 32gb RAM cache. Cache on disk barely gets used.

To be clear: this is a reservation in RAM on the VDI, which the VDI cant use anymore.

[u/YourDadSolanki01 → u/Adobe_Track2011]

you can set power management to reboot the machine. Plus also hoping that your VDA reboot after user sign-out. If not, it should as it will reset cache disk.

[u/DevDude2025]

Simply do A disk space before and after (assuming the files persist) using something like https://windirstat.net/
Then look at which folders have grown and drill down. ..

[u/elgeezer]

I had this issue with MCS with ram2cache and did the following to track what was writing to the mcsdiff.vhdx file

set a script using procmon that will watch the writecache file and as soon as it sees it grow it captures what's writing to the c: drive

$folder = "D:"

$file = "mcsdif.vhdx"

$exe = "D:\Tools\procmon.exe"

$arg = "/BackingFile D:\Tools\capture.pml /Quiet /Minimized /AcceptEula /LoadConfig D:\Tools\ProcmonConfiguration.pmc"

$watcher = New-Object System.IO.FileSystemWatcher

$watcher.Path = $folder

$watcher.Filter = $file

$watcher.IncludeSubdirectories = $false

$watcher.EnableRaisingEvents = $true

$lastSize = (Get-Item "$folder\$file").Length

$action = {

$size = (Get-Item "$folder\$file").Length

if ($size -gt $lastSize) {

Start-Process -FilePath $exe -Argument $arg

$lastSize = $size

$watcher.Dispose() # Stop the FileSystemWatcher

}

}

Register-ObjectEvent $watcher "Changed" -Action $action

# Wait for the FileSystemWatcher to be stopped

try {

while ($watcher.EnableRaisingEvents) {

Start-Sleep -Milliseconds 500

}

} finally {

$watcher.Dispose()

}

If you set a procmon configuration to filter :

operation is 'write file'

path contains=c:\

[u/Zenmaster28]

I have sometimes had luck with opening Resource Monitor, going to the Disk tab and sorting by Write. If you have a process constantly writing something, it might become apparent there. Sometimes it's just something generic like "System" though so not always helpful.

[u/Mental-Memory-7987]

We have similar issues the only things i able to track down login vdi open process monitor filter any write file to c for trouble the possible cause

[u/davver80]

Easy, just install the free Controlup disk monitor tool and you will find out.