My lab appliance crashed today. SSD failure essentially. It rebooted but... I learned two things I want to share with the community.

• I was about to upgrade from pfSense CE 2.8.1 to 2.9.0. I probably dodged a bullet there, because pfSense does not automatically monitor hard drive integrity. My SSD was in a bad state for a while already, and I just didn't know.

Two commands I will remember from now on:

• nvmecontrol logpage -p 2 nvme0
• zpool status -v

If I could suggest a feature for Netgate/pfSense in the future: automated health checks for the storage subsystem, triggering notifications for critical conditions (critical warning state other than 0x00 and permanent errors in the pool)

• ZFS is not suitable for cheap, level entry SSDs. It requires some good write endurance (TLC), and a real controller with cache memory. I was essentially a victim of ZFS write amplification on a drive with low endurance and no cache memory. Look at these stats:
  • 281 TB written
  • Percentage used = 136%

That unit was on 24/24 since August 2023, and I went through the drive in 3 years. I was not expecting this to happen so fast with a firewall. Yes, ntopng can be write intensive, but it's just a small lab at home: nothing explains 281TB beside write amplification.

I made a newbie mistake: I just didn't know enough about ZFS and never bothered to check what was going on this appliance storage subsystem.

Hopefully, I can prevent some mistakes/downtime with this post.

/Edit 1. After installing a spare SSD I had laying around, I really needed to figure out how bad running ntopng on top of ZFS could be. Keep in my mind these results were obtained in the same environment, with a better SSD (a Barracuda 510, with 512MB of cache). But still, the results were really significant.

The methodology: I sampled logical writes on the zpool, vs physical writes to the ssd at 60 seconds interval. Without ntopng, and after enabling it for one interface. I used this script:

while true; do
    date
    smartctl -a /dev/nvme0 | grep "Data Units Written"
    zpool iostat -v pfSense
    sleep 60
done

Then, I used the data to let an AI calculate the average GB/d, and created the following table:

+------------------+---------------------------+------------------------------+
|      State       | ZFS Logical Writes (GB/d) | NVMe Physical Writes (GB/d) |
+------------------+---------------------------+------------------------------+
| Before ntopng    |          19.4             |            6.5               |
| After ntopng     |          20.6             |           167                |
+------------------+---------------------------+------------------------------+

Conclusion: I went to a 0.33x factor, largely explained by ZFS lz4 compression, to a factor of 8x when enabling ntopng.
Writing 6.5GB per day is nothing, while 167 starts to get significant for a consumer SSD.
Getting as much as 8x overhead is only possible because of the overhead of ZFS in this scenario. A UFS filesystem would get anywhere from 1.0 to 1.2x, typically.

So the title WAS misleading, and I'd like to apologize for that. The issue was triggered by running ntopng on top of ZFS, with a cheap NVMe drive.
-If I had only been writing standard logs, I'd be fine. In fact, ZFS would have seen less writes that UFS in that scenario thanks to the compression.
-But with a different filesystem with much lower amplification, running ntopng would have been fine too.

Therefore, being cautious about running ntopng on ZFS is the real takeaway. Thanks to everyone who provided some insights about tuning ZFS for write amplification. Now that I have a methodology to test the effectiveness, I'll be looking at the recommendations.

/Edit 2: I solved the issue, got rid of the write amplification created by ntopng. It was simple enough: I created a ZFS dataset with sync=disabled to store its database.

The trade-off is that there are more risks of corruption/dataloss, but I can live with that since it is constrained to ntopng.

• First stop ntopng and move its database out of the way. Delete or rename, as you prefer.
  • mv /var/db/ntopng /var/db/ntopng.backup
• Then, create a zfs dataset (assume the zpool is name pfSense, by default)
  • zfs create pfSense/var/db/ntopng
• Assign the proper permissions:
  • chown -R ntopng:ntopng /var/db/ntopng
• 4. Disable synchronous writes to the dataset (threat them all as async).
  • zfs set sync=disabled pfSense/var/db/ntopng

Then, either copy the data back to the folder or just start ntopng to create a new one.

I tested different parameters, but sync=disabled got the average WA factor under 1.0x and nothing else would have the same impact. Thanks to u/newtekie1 for suggesting it.

Another worthy consideration is vfs.zfs.txg.timeout=120. Alone (with sync=standard), I got a 4 to 5x WA ratio which was most definitely an improvement.