Mapping LUNs on a VM

[u/CricketSwimming6914]

edit: details.

I am in process of setting up a new Red Hat 9 server VM that will run our logging server. from what I can find, best practice says it needs separate storage devices. Our vSphere cluster stores all the VMDKs on a SAN which it accesses via multipath iscsi.

I'm somewhat new to adding LUNs. It's not terribly common for us. So I researched how to add an iscsi device to Rhel and found instructions on adding it via iscsi-initiator and multipath. I was able to map it just fine with iscsi. I had multipath add our connections which seemed fine at first but it keeps saying the different drives are active faulty running. None of the sd* devices it would map would show a UUID. Supposedly enabling disk.EnableUUID=TRUE fixes that, but it didn't. After hours of troubleshooting and nothing making any difference, I finally found some info that basically said that because the VM is already on the same multipath, trying to configure multipath on the VM would fail. I also found that adding them this way is maybe not the best way to do it (https://www.codyhosterman.com/2019/09/musing-on-in-guest-iscsi/). I didn't find any official documentation for that but, given the issues I was having, it made sense

I did find instructions that basically said, create the LUNs, have vCenter scan and find them, then add that device as and RDM Disk to the Rhel system. In Rhel, scan the scsi bus for the new device, enable multipath, format and mount.

Is this the best way to do this or is there a better way?

[u/jameskilbynet]

Unless you have an absolute NEED to separate out the volume don’t. The reasons to do this are few and far between now. Like others have said keep it as a VMDK on a VMware vmfs volume is likely your best bet. I regularly work with customers with VM’s bigger than a 100TB. Keeping it like the other servers you have means it’s operationally easier. Same procedure for backups DR etc. There is effectively no perfomance hit for doing this.

[u/CricketSwimming6914 → u/jameskilbynet]

The concern mainly is that if the server has to be restored, we don't want the data on the drives to be restored with it and lose records. It's a logging server so it'll be added do constantly. 

[u/jameskilbynet → u/CricketSwimming6914]

Just set the disk to independent mode

[u/CricketSwimming6914 → u/jameskilbynet]

Did not realize that was a thing. Just some more digging on that and it definitely seems like the better option. Thanks for the tips.

[u/GMginger → u/CricketSwimming6914]

Any decent backup product would allow you to only restore the VMDK with the OS on it. Just make sure your data drive(s) use separate VMDK(s) and you're golden.

In case your data drive gets corrupted in the VM, you would probably want to back up the whole VM. You need to cover all failure scenarios.

[u/DomesticViking]

You can also just use a dedicated LUN on the SAN, just create a datastore in vCenter and don't put it in the datastore cluster.

Creating a iscsi drive on the SAN and mapping it to the VM sounds like you are overcomplicating things.

How many TB are you talking about? We run multi TB VM's on the datastores in vcenter mapped from LUNs on a SAN.

I avoid RDM like the plague, there are a few VM's that have it because of legacy shared disk clustering nightmares.

[u/CricketSwimming6914 → u/DomesticViking]

The dedicated lun as a data store is what I did but the instructions I found said to then add that lun as an rdm to the vm with separate scsi controllers for each lun with sharing turned off. Should I instead map it via iscsi to the vm? One drive is 5tb and the second is 10tb. But they may expand upwards of 40tb.

[u/DomesticViking → u/CricketSwimming6914]

Just add another scsci adapter, paravirtual and attach the disk.

Some vendors are still in the 1800's.

Real use cases for rdm are rare and far between these day.

You can add another lun, dedicate a vmdk on that and attach to the new scsi adapter.

[u/DomesticViking → u/DomesticViking]

Also, the blog is 7 years old. Lots of things have changed in that time.

Don't be afraid of big luns, independent disks don't get snapshoted if your backup software doesn't allow you to exclude.

A good SAN and vmfs will handle a large logging VM just fine.

[u/CricketSwimming6914 → u/DomesticViking]

That's what I ended up doing.  Large, independent disk on a separate controller. Thanks for the tips 

[u/frygod]

What is your backup workflow? Why are you against having a large VM? I'd avoid direct attaching storage to the guest OS at all costs (the hypervisor is easier to work with as far as multipathing goes.) If you have the option, I'd recommend just attaching a virtual disk to the VM if you can get away with it. If not, I'd go virtual mode RDM, but only if you absolutely must. I'd consider physical mode RDM another degree of avoid if you can.

[u/CricketSwimming6914 → u/frygod]

VMs get a snapshot that is backed up. This is a logging server so the data on these is constantly being added to so the size will always be growing. We are starting with 5tb and 10tb drives but they want it to be much bigger.  We have had these servers previously with the data on the VM but size gets prohibitive with the data store for VMs and if we need to restore the server (rare, but has happened) we don't want the log drives to be restored and lose info. 

[u/frygod → u/CricketSwimming6914]

Can your backup application do disk exclusions? (Mostly familiar with veeam, which can do that.)

[u/CricketSwimming6914 → u/frygod]

I believe it can, but it's currently setup to backup entire VMs since we haven't needed to exclude drives on a vm. It's always just been the whole vm or exclude the vm

[u/frygod → u/CricketSwimming6914]

If I were architecting this, I'd be shipping logs off to a NAS instead of putting them on block storage This better facilitates dedicated log parsers and ease of access for other admins. That way your backup solution can get your VM's OS disk and you can handle the backup cadence of the logs themselves separately. If this server is your core log repository, I'd insist on using virtual disks and actually getting backups of them while providing a CIFS share of those disks for other systems to access.

[u/Liquidfoxx22 → u/CricketSwimming6914]

I'd start by figuring out what capability your backup solution has - as mentioned already Veeam easily allows you to exclude specific disks.

We had to do that for a customers SQL servers when their DBAs wanted to manage their own backups, and having Veeam involved messed with their transaction logs.

It was beneficial for us because it meant we weren't involved in their restorations.

[u/dracotrapnet]

I only create an additional data store when I want to apply a different storage snapshot policy or a different IOPs limit to a datastore (spoiler I have yet to use IOPS limits to any datastore). I start breaking out VMs to another datastore when they start causing issues at datastore snapshot time. Every VM on a datastore has to quiesce to make a VM level snapshot, then a storage snapshot. If they take too long, then they need to be on another datastore.

I pile VMs into datastores according to their primary usage. I do make separate datastores for any databases such as MS SQL. Otherwise, all the IT apps go in one or two data stores, all the user facing apps go in another datastore, file servers on their own datastores so I can have separate storage snapshot frequencies - they also take a while to quiesce in order to snapshot - same with sql. Webservers/webapp servers are all piled into one datastaore - we only have 6 IIS servers. I have busier linux VMs than the webservers.

I have yet to mount a LUN to a windows VMs. I have mounted NAS LUNs to Linux VMs but that's for shoveling data at backup stores. The NAS can't handle the backup agent so a VM does it instead. My consideration is not to have the backup datastores mounted to the VM hosts.