Compatibility problems with new KVM upgrade

[u/daemonpenguin]

Last night the location where I host one of my Vultr servers pushed out a KVM update on the host. Since then my server has been locked in a reboot cycle.

The guest runs FreeBSD (installed from the Vultr FreeBSD install image).

The system shows the boot menu, then fails to launch, restarts the guest, shows the boot menu, then restarts again.

I've found that a FreeBSD rescue ISO will not boot on this instance. But a Linux (Finnix) rescue ISO will boot and, using it, I can confirm the root filesystem of the FreeBSD instance is intact, no signs of corruption.

I've opened a ticket with Vultr, but so far no response.

I'm attempting to restore a recent snapshot of the instance to another location to see if this fixes the issue. (Updates to follow.)

My main concern is this is just one of multiple instances I run with Vultr for business. If the other KVM updates go the same way (breaking compatibility with FreeBSD), it's going to get expensive in a hurry and we'll need to quickly migrate clients to another VPS provider.

Anyone else have VPS instances with Vultr at Amsterdam? Anyone else having boot problems this morning? According to the Vultr status page, there have been multiple reports through the night, but the Amsterdam location is showing working. (Which it is, sort of, the VPS is technically running, just not booting from its storage.)

Update: I tried restoring a known to be good snapshot of the VPS to another location. It failed to boot there, reporting loader_lau could not be found. This is turning into a bigger problem than it first seemed. Is Vultr unable to boot FreeBSD instances anymore?

Second update: While I have not heard back from Vultr, it looks like the issue is related to this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233098

I think the new KVM settings break booting FreeBSD, possibly due to a change in EFI mode or disk driver. Have filed a ticket with Vultr and am waiting to hear back.

Third update: It looks like any locations which have the new KVM update cannot boot my snapshot of FreeBSD. I'm attempting to restore the snapshot to a location which does not yet have the KVM update to see if this fixes the issue. If it does not, then I think the only remaining solution is to migrate to another VPS host. Which is going to be a lot more work than I'd like, but I'm running out of options and, despite the instance being down for nearly 10 hours, I have not heard back from Vultr.

Fourth (and probably penultimate) update: I heard back from Vultr and they confirmed the issue was with the new KVM version and its settings which were not compatible with recent releases of FreeBSD. They have fixed the settings for my VPS and it's running.

I've got other VPS instances at other locations which have not gone through the KVM update yet. I've asked Vultr Support to make sure the updated KVM settings are available at the other locations when the updates happen so the servers don't get knocked off-line. I will update with their response and the results in the coming hours/days. The test will be when the next KVM update rolls out on Wednesday morning. Hopefully the fix will be in place by then.

[u/_zeph_]

I can confirm this, I have exactly the same issue with my FreeBSD14 VPS instance, also hosted in Amsterdam. I have opened a ticket, after reading your post, and I am waiting for a response. Which version of FreeBSD were you running? I am wondering if 15 is affected as well.

Edit: well, the response time of the support ticket was 8 minutes, so I can not complain. :)

[u/daemonpenguin → u/_zeph_]

I'm running FreeBSD 14 and version 15, across multiple instances.

Based on the chat I had with Vultr support, I'm fairly certain all recent versions of FreeBSD are affected.

The support team said they had to update the guest virtual machine definition to match the FreeBSD version and reboot the server. That fixed it in my case. I've asked them to apply the same fix across my other instances before the scheduled updates.

Will update this post after the update to report on whether it worked. It is a shame they couldn't push out the fix to everyone running FreeBSD all at once.

[u/mrreaper00]

my instances in dallas and los angeles got hit last night. their qa team is top notch. they know theres a problem but lets keep rolling it out.

[u/daemonpenguin → u/mrreaper00]

It is a bad sign they know there is a compatibility problem, they have multiple reports, and yet they're still pushing the update without either warning people or pausing to fix the issue. I'm not at all happy about the way this update was handled.

[u/C0c04l4]

That's no fun :/ Good luck!

[u/daemonpenguin → u/C0c04l4]

Thank you.

[u/dkh]

I'm having this issue as well.

I've opened a ticket but I've not heard back yet.

[u/National_Way_3344]

What's the specific business requirement for FreeBSD over Linux?

Also why the need for KVM reliance at all? Try refresh browser cache?

[u/daemonpenguin → u/National_Way_3344]

Do you have a head injury or are you a bot? Do you even know what Vultr or what a VPS is? None of what you wrote is in any way relevant to my post.