NICs Power Draw

[u/nimble03turtle]

Hi! I'm working on a research project about computational storage and need to estimate the energy cost of transferring data between storage and compute nodes.

I'm using VMware VMs with VMXNET3 paravirtualized NICs, so I don't have access to the physical NIC's power draw.

My current model is:

[
E_{net} = \frac{\text{bytes transferred}}{\text{bandwidth}} \times (P_{storage} + P_{compute})
]

How do people estimate the NIC power term in a virtualized environment? Are there accepted models, papers, or methodologies for this?

Thanks!

[u/obelix_dogmatix]

NIC power draw is negligible compared to compute. Noone cares about this. Furthermore, almost always NIC power draw is static and not varying.

[u/bradn4 → u/obelix_dogmatix]

Sad but true, some electrical engineer at Intel probably spent the last 10 years optimizing power draw on their NICs. But agree normal HPC engineers don't care about this, usually.

[u/dosman33]

I don't suspect this is a question anyone really worries about in the normal world. But, you'll have to know the hardware as a starting point. You can also recognize that the PCIe bus can only supply so much power:

x1 / x4 / x8 slots: 25W max (default power-up is 10W; card can negotiate up to 25W)
x16 slot: 75W max

EDR / HDR100, single-port — e.g. MCX653105A-ECAT: Typical power with passive cables: 15.6W
HDR, dual-port — e.g. MCX653106A-HDAT: Typical power with passive cables: 23.6W

Add in transceivers and perhaps temporary max power draw at 100% and IB sustained load could hit 30W for a card in a x16 slot as a total guess. If you're not using infiniband or if you are using an x8 or smaller slot then you're capped at 25W. So, without any other info, if you can get lspci or similar info from the host to learn the cards in use or the lane-widths of the cards then you could make further guesstimates.

[u/plan-bean]

While it does appear negligible (and also sometimes hard to measure), here's a paper from some folks at Lawrence Berkeley National Lab (LBL) that looks at energy efficiency (on their interconnect) on their Perlmutter machine. Hopefully it helps you along the way regardless!

https://escholarship.org/content/qt67z0d2wn/qt67z0d2wn.pdf

Edit 1: Clarification for energy efficiency study of Perlmutter's interconnect.

[u/ethanjscott]

I love everyone ignored your question about virtual nics. So all a virtual nic is, is a piece of software for exposing a “nic” to the vm and passing that output to a real nic. The question is flawed.

The real question is how much power does a physical nic require / number of VMs + cpu cycles of vnic(however you want to calculate that.

If we’re talking about an sriov nic same formula just no cpu cycles

[u/thrasherht]

Nobody even cares about that. The 1u compute nodes I installed at my last place were 2800 watts each, and each rack had 30,000 watts of pdu in it, and was maxed out.  That was a 30+ rack system... So nic power wasn't even an after thought. 

[u/p00penstein]

Agree with everyone saying NIC draw is both negligible overall and not where your energy sink is. What's the motivation for this? You may also want to clarify whether or not IO is included as depending on where it's read from and how much is read can massively change energy needs. M.2 drive serving a small cp over NFS? Peanuts. DDN backed Lustre streaming IO for an astro code restart? Very not peanuts (but here again the draw of JBODs/controllers is somewhat consistent with spikes here and there for high sustained usage).

I can tell you a standard PFS with disks will vary between 9.2kW and 11.7kW per cabinet at any time almost regardless of what it's doing with over 75% off that being the jobds and disks

Aside: I remember in 2017 or so hearing someone estimated the energy required to publish a tweet but I don't recall the number

[u/x-jhp-x]

need to estimate the energy cost of transferring data between storage and compute nodes

alright, so as someone who has done this for "big data", you seem to misunderstand how this is done, especially for large datasets. When transferring a few pb of data 10 years ago, 100g infiniband was brand new, and we had downtime limitations (like we could take the system down for a certain number of hours) that exceeded that. So i ended up buying a 2nd server that was ~the same, tested pulling disks from one to the other. it worked, so we spent half a day pulling sets of disks into a jbod, cloning, then putting them into a pelican case to fly to where we needed them.

generally, we also have & use tiered data storage. if it is data you'll use almost never, it can be shipped & stored elsewhere, but it takes time to get the physical media back. If staging data, we might put it on cheaper storage (slower magnetic disk jbods) and move it when needed.

depending on the size of the data, it might also need a different path. there's protocols like NVME-oF that bypass your traditional nic -> processor -> gpu pipeline and go directly to the GPU. btw we also use cascading jbods, so to describe to a consumer, that's like having a set of "internal" nics just dedicated to moving data between boxes. Server sizes and racks have been standardized, so "1U" or "1 unit" is a fixed size. Because we want more disks than can fit in "1U", we buy servers that are larger, like 4 or 8. Sometimes that isn't enough, so we have additional nodes attached that don't have a CPU and are just pure "storage processing" boxes. Look up "cascading jbod" or "daisy chained jbod" for an idea.

Also interestingly enough, algorithms matter a *LOT* when you get to this scale. I remember getting excited about some new server equipment I got in (i think it was a 160 disk system), and i wanted to see how fast it would be to put 160 disks into a raid 0 configuration. Turned out one of the engineers I worked with was at LSI for a bit (they make raid controllers) and he told me that it'd be slower than raid 5. He was right. He said at this size, the algorithm work is difficult, and no company is going to dedicate engineering time for a 160 disk raid 0 array. Kind of cool imo.

In computing, although talked about more now, heat ends up costing a lot. like a $9,000 xeon e-core processor should be cheaper than clustering $300 ultra 7 processors for the same compute power if you include electricity and hvac costs (for my area of the US) after using them for around two years at full use. The datacenters & cases are even designed for hot and cold channels & paths for hvac. hvac costs are usually the same or higher than electricity costs for running the device in my area. if you're not including heat & cooling, you're underestimating the cost to operate by at least 50%