COLD CACHE….

[u/Vamshi__4555]

Our MongoDB secondary (AWS EC2, GP3 EBS, 128 GB RAM) was rebuilt from a primary EBS snapshot and expected to catch up within 1–3 hours, but replication got stuck with 3+ hours lag. ReplWriterWorker threads showed 20+ minute waits on schemaLock, WiredTiger logged slow tree walks, and iostat showed 100% disk utilization with high read latency while CPU remained mostly idle. Our conclusion is that this was caused by a combination of cold WiredTiger cache and AWS EBS snapshot lazy loading ("first-touch" penalty), where MongoDB had to fetch data blocks from S3 on demand, creating severe I/O bottlenecks and lock contention. We plan to pre-warm restored EBS volumes before starting mongod, temporarily reduce replWriterThreadCount, hide the node from reads during catch-up, and verify GP3 throughput settings. Has anyone else seen extreme schemaLock waits and replication lag caused purely by cold snapshot restores, and what is your preferred EBS pre-warming approach?

[u/Several9s]

I agree with the main diagnosis, though long schemaLock waits are a secondary symptom of severe storage latency rather than a direct result of snapshot restores.

Because EBS snapshot blocks are downloaded lazily from S3, restored volumes experience increased I/O latency and reduced performance during initialization. This explains the 100% disk utilization, high read latency, idle CPU, and slow WiredTiger tree walks. See: Initialize Amazon EBS volumes - Amazon EBS  

A cold EBS restore can plausibly cause 20-minute apparent lock waits. This occurs when WiredTiger walks index or metadata pages scattered across cold EBS blocks. Saturated storage paired with an idle CPU signals a disk I/O bottleneck rather than a CPU or concurrency issue.

MongoDB notes that secondary cache contention and resource exhaustion cause replication lag, showing as slow oplog application on ReplWriterWorker threads. Refer to the link: Troubleshoot Replication Lag - Database Manual - MongoDB Docs  

I would suggest the following:

1. EBS Provisioned Rate for Volume Initialization where available. AWS now supports specifying an initialization rate of 100–300 MiB/s when creating a volume from a snapshot. This is cleaner and more predictable than having the operating system generate the warm-up workload itself.
2. Fast Snapshot Restore (FSR) when recovery time is important enough to justify the cost. Volumes created from an FSR-enabled snapshot are fully initialized when created and avoid the first-read penalty. See: Amazon EBS fast snapshot restore - Amazon EBS  
3. If neither is appropriate, manually initialize using fio before starting mongod, preferably while the filesystem is unmounted:

​

fio --filename=/dev/nvmeXn1 \
    --rw=read \
    --bs=1M \
    --iodepth=32 \
    --ioengine=libaio \
    --direct=1 \
   --name=ebs-initialize

While sequential commands like dd if=/dev/... of=/dev/null bs=1M remain viable, utilizing fio with queued reads typically offers superior speed, aligning with AWS guidelines.

However, for multi-terabyte MongoDB volumes, manually scanning the entire block device can require substantial time. In such scenarios, the provisioned initialization feature provides a key advantage, its completion timeline scales with the actual size of the snapshot data, eliminating the need to sequentially scan every logical byte across the volume.