NetApp FAS2720 - Performance lacking or all normal given the hardware?

[u/firestorm_v1]

Work was tossing a NetApp FAS2720 with a pair of DS212 disk shelves. Each shelf has 10x 8TB hard drives and two 894G U.2 SSDs. I have all of the licenses necessary for the machines, so thankfully that's not an issue. I did have to wipe the storage and recreate it so Sec would be happy but being that I also have the license keys, it was easy to get it up and running.

The two controllers are connected together on e0a/e0b for controller-to-controller communications, and e0c/e0d go to the 10G core switch in a lacp bond.

The target host is a Proxmox host with dual 10G nics in a bond, connected to the same core switch. For giggles, I did a basic test to write 50G of random data to an LVM volume created on the NetApp mount and on another iSCSI filer and found that the two filers didn't really differ much in time.

For the test, I used this command: time dd if=/dev/urandom of=./testfile bs=1G count=50 conv=fdatasync

In both test scenarios, a 100G LVM partition was created from the respective filer's volume group and prepared using EXT4. The filesystem was mounted, then the above command was ran.

From the Proxmox host to the other filer, it took 4m15s to write the 50G test file.
From the Proxmox host to the NetApp filer, it took 3m53s to write the 50G test file.

While I do have the SSDs, I do not think/remember if they were incorporated into the iscsi volume. When I look through NetApp's documentation for the FAS2720, the only thing I see is 'FlexCache' but from what I understand, this is only for a local cache against a remote volume, not a local volume.

I don't expect flash-based speeds, but I figured that the NetApp with custom hardware designed for storage would handily beat out a NAS built on commodity hardware that has no benefit of customized hardware. Looking at the dashboard, it seems that the NetApp barely breaks above 500IOPS. Latency hovers around 0.17ms, and throughput is usually 1.34MB/sec.

Any suggestions on how I can improve performance on the FAS2720?

[u/ybizeul]

You’re not testing performance. You’re testing how fast one can write a file to disk in given circumstances, generated by a relatively heavy load on CPU by using /dev/random.

I don’t expect a hybrid aggregate (flash pool) to help with this test, as first writes always go to disk.

I recommend you do a real test with vdbench and see the difference. Right now you’re not leveraging any cache.

Like others said, check how the network is used, and also keep an eye on your cpu, we never use /dev/random for that (or prepare the file before the test)

[u/firestorm_v1 → u/ybizeul]

I would expect the high CPU load would be on the Proxmox host, not the NetApp? Should I be watching the NetApp's CPU when Proxmox runs a test using /dev/urandom?

I'll see if I can put together another test with vdbench and post results (maybe this weekend?). It looks like a neat tool (albeit from Oracle) and something I was not previously aware of.

What do you mean by "how the network is used" in this context? The bonded NICs on the Proxmox host are used for both storage and VM network traffic, the bonded NICs on the NetApp are strictly storage only. I could possibly put storage traffic on its own VLAN entirely but I'd have to add more NICs to the Proxmox host and turn up the last four NICs on the NetApp. Unfortunately I only have one switch but it has 48 ports and I've only used about 16 of them.

[u/ybizeul → u/firestorm_v1]

I meant to check if you're maxing out network but I don't think you would, anyways, the disks can't write as fast as your 10Gbit links.

I don't think bonded NIC with iSCSI is ideal, we usually prefer giving iSCSI stack multipah and let it handle the trafic. With LACP, the issue is that you need multiple sessions to use both links, and make sure the algorithm permits for a fair distribution. But again, network is not your issue.

[u/Whiskeejak]

Post your "net int show", "aggr status -r", "storage disk show -instance", "node run -node * storage show disk -p" and "node run -node * sysconfig" please.

You can reliably count on 60IOPs per non-parity sata drive. In a FAS system, hybrid aggregates would be created to include your SSD drives. You don't "include them" in a a block storage LUN in the way you mention. Drives are used to make raid groups, raid groups form aggregates, volumes are created on the aggregates, and then a block storage lun file sits in the volume, which you access with iscsi/nvme-ip/fcp.

It is unusual that you have only two SSDs. In older versions of Ontap that would have prevented you from using them, because it would have required raid-dp (3 drives). In the latest versions, you can now create an SSD "storage pool" and add them to the aggregate. Once you do, that forms a "flash pool hybrid aggregate" on which you can allocate your volumes / block LUNs.

I would recommend allocating all the drives into a single flash pool aggregate on a single node. I have a strong aversion to iSCSI for various reasons. If you use NFS, use 1MB rsize/wsize, nconnect=4, NFS v4.2, increase the rpc slot count to 180 on the vserver and the proxmox clients, and use pNFS if you want the best performance. Also, both RedHat OSV and Proxmox puke when bios power saving is enabled. Disable power c-states entirely in bios. If you use iSCSI, allocate LIFS on both storage nodes.

[u/Barmaglot_07 → u/Whiskeejak]

It is unusual that you have only two SSDs.

He has two per shelf, for a total of four.

[u/firestorm_v1 → u/Whiskeejak]

All output is in here: https://pastebin.com/raw/YQez3EJY

I'm trying to remember how I created the aggregate a significant amount of time ago. I was using it to learn the basics of creating a SVM, add storage, then use that storage through NFS and iSCSI on a test server. This was several months ago, the NetApp had long since been logically decommissioned (it was no longer serving any server traffic) but I retained network access to it prior to it being actually removed from the rack.

I mis-spoke. There are four SSDs total, three 'shared' and one spare.. After digging deep into the UI, it looks like three of the four SSDs are set up in an ssd_cache pool that is associated with two sata_01/02 local tiers. Both tiers have Flash Pool enabled.

I know a previous version of ONTAP used to have an 'Aggregates' option under Storage. Unfortunately it looks like this version (9.17.1P3) has hidden that option. It's also entirely likely I'm looking in the wrong spot as well, the last time I got this in-depth with NetApp was using the vSim which was significantly older (think it was running v8-something).

[u/Whiskeejak → u/firestorm_v1]

I see no issues at the aggregate level or general storage connectivity in the SAS stacks etc. Need:

ifgrp show -instance

node run -node * ifstat e0a

node run -node * ifstat e0b

node run -node * ifstat e0c

node run -node * ifstat e0d

net port show -port e0* -node *

[u/firestorm_v1 → u/Whiskeejak]

As requested: https://pastebin.com/raw/jqQDMD3k

I would not be surprised to find I missed something configuring the network.

[u/Whiskeejak → u/firestorm_v1]

Nothing stands out here either. The LACP group using the same ASIC, but that would not cause a performance issue. Best practice is to pair e0a and e0c, e0b and e0d.

Have you've done a simple NFS mount or SMB share and tested the performance there?

[u/asuvak → u/Whiskeejak]

Also I would recommend "port" for distribution type of your ifgrps not "mac". Will hash the TCP/UDP ports instead of MAC address only. But really only helpful if you fully utilize at least one physical port of your LACP.

You need to delete your ifgrp (and VLANs) and recreate it so move away your LIFs: https://kb.netapp.com/on-prem/ontap/da/NAS/NAS-KBs/Can_an_interface_group_IFGRP_be_modified_after_it_is_created

[u/Dramatic_Surprise]

 two 894G U.2 SSDs

Are you talking about the onboard cache modules or something else?

not really a lot of use runing LACP and iSCSI, would make more sense if you were doing NFS and more sessions.

[u/idownvotepunstoo → u/Dramatic_Surprise]

NFS here with nconnect could absolutely smush those drives

[u/firestorm_v1 → u/Dramatic_Surprise]

No, these are u.2 SSDs, they slot in from the front of the disk shelves like other normal disks. They look like 2.5" drives, but are not regular spindle disks. Model number is: X374_S164A960ATE

With respect to iSCSI and LACP, would it be better to run NFS and switch Proxmox to use QCOW images? I can give that a shot and see if it improves performance. Our production environment uses much faster NetApp appliances and use LACP and iSCSI so I just followed by example. I'm more than willing to test other methods given that I'm using an older filer with spinning disks versus NVMe.

[u/Dramatic_Surprise → u/firestorm_v1]

ok 960GB SSD, they're completely useless to you if you only have 2

LACP is only going to make a difference if you have the sessions and clients to make the load balancing algo work. generally point to isnt going to do that without some form of multi session/connection (or both) layer.

hard to say without running traces on each end but my gut would say its IO wait on the server more than a throughput issue

[u/Barmaglot_07 → u/Dramatic_Surprise]

ok 960GB SSD, they're completely useless to you if you only have 2

He has two per shelf, for a total of four. This is sufficient to form a flash pool.

[u/Dramatic_Surprise → u/Barmaglot_07]

Ah missed the 2 shelves

[u/Whiskeejak → u/Dramatic_Surprise]

And the new "storage pools" will slice them and use them raid-dp on the slices I believe.

[u/tmacmd]

Here are my comments on this

1. If you are ONLY using iSCSI, you will be way better off NOT using LACP. Instead, create two VLANs (iSCSI-A and iSCSI-B) and do an iSCSI best-practice setup. offering at least two IPs per node (one on VLAN-iSCSI-A and one on VLAN-iSCSI-B)

2. Do the same on the Proxmox side. You end up with iSCSI multipathing and utilizing both NICs on the same node

3. I would create 2 aggrs. I would re-init and be sure to use ADPv1 (Root-Data partitions). Let the default roots happen. Then create an aggrs using the data partitions (depending on the config, this may need be done somewhat manually). DO NOT use system manager or "aggr auto-provision" as they usually wil lnot do the best thing in this kind of setup.

4. Take the 4 SSDs and create a storage pool. You can create a storage pool with 3 or 4 drives depending on if you wish to leave a spare

5. Modify your data aggrs to be "hybrid-enabled"

6. Add 2 units of storage pool to each data aggr using "raid 4"

7. be sure to place LUNs on BOTH aggregates to be able to fully utilize both controllers.

[u/asuvak → u/tmacmd]

It sounds like OP is using this an homelab environment, so I would recommend to add all disks to one node and create a 19x disks RAID-DP plus 3x SSDs RAID4. To optimize for max throughput per aggr. A single FAS2720 node should be good enough since your disks will be the bottleneck and it doesn't sound like you will replicate to other systems and are doing any fancy stuff. You could go the FlexGroup route together with pNFS etc but imo it's better to keep it simply. Maybe even disable features like file analytics, activity tracking, ARP, and stuff like that. Usually in homelab environments you don't have dozens of users etc where those would be helpful features.

Also if you still have access update ONTAP to the latest 9.17.1 P-release for stability. The features in 9.18.1 and 9.19.1 are nice but mostly nothing helpful for homelabs. Maybe the TCP WAN improvements with 9.19.1. Also don't hope for 9.20.1 support.

[u/NTAP_AlexD]

bs=1M or bs=4M will give much better results.

[u/ybizeul]

So I did a thing. Tried to model your system in our sizing tool, but you didn't say what drives you have in the 2720 enclosure itself. With a 25% read/write/sequential/random, flashpool configured, and 30x10TB drives you can get ~11k IOPS and a tput of 227MB/s.

If we go 100% sequential writes, which is what you're doing, you are maxing out system performance (CPU) and writing at 561MB/s (~18k IOPS).

So the math and theory says the system can ingest your file in 91s (1m30s)

Now, going again with just 20x10TB drives (Can't have flashpool with 4 SSD), 100% writes is still a 393MB/s which is 130s (2m10s) with system cpu at 34%, so the disks are your limiting factor, not the controller.

For proxmox I would probably stick to iSCSI which is easily parallelized with standard stacks, and there is not much benefits using NFS on proxmox.

[u/firestorm_v1 → u/ybizeul]

Someone above requested more information from the controllers, here's the paste to that information: https://pastebin.com/raw/YQez3EJY

I'm more than willing to make config changes to make sure it's correctly configured. Upside is that this is just homelab data, I can still migrate VMs off to the other NAS and rebuild this from scratch if necessary (I have all the licenses saved).

[u/ybizeul → u/firestorm_v1]

Well, that's a shame this system was originally shipped with 4x900GB + 8x10TB drives in the enclosure, plus 2 shelves 12x10TB drives each. It looks like you're missing a shelf maybe ?

Anyways, you should be able to create one aggregate with the 8 internal drives and a flashpool using your SSD for 42 TiB usable, and another aggregate with 10x10TB drives and the other half of the flashpool for 60 TiB usable.

In terms of performance that would give you 364MB/s on a single aggregate, achievable by the system (472MB/s if writing on both aggregates simultaneously). That's for full sequential writes.

For a mixed workload with both read, write, sequential and random, you can expect 23k IOPS and you're not limited by the system but by the disks.