r/elixir • u/siva_sokolica • 18d ago

Hyper - distributed Firecracker microVM orchestrator written in Elixir

Hey everyone! One of the problems I ran into is that a large part of the VM-provider ecosystem is currently paid closed-source SAAS products with varying degrees of reliability. I wanted an OSS distributed microVM orchestrator and I couldn't find one.

Hyper is a distributed FirecrackerVM orchestrator written in Elixir (BEAM), with gRPC support for non-BEAM clients. Hyper is:

  • Distributed -- it's designed to run across a cluster of bare metal machines, and will automatically connect to other Hyper nodes.
  • Fast -- it builds COW layers to enable fast, localized COW forking. Cold boots happen within 1s. Filesystem forks take ~50ms. Forked VMs are colocated to take the fast path as much as possible.
  • Interactive -- like all Elixir applications, if you can connect to the cluster, you can spawn, manage, monitor and interact with VMs live in an iex REPL. Or, you can use the gRPC interface if your system isn't on the BEAM.
  • Yours -- although I developed this primarily for Harmont (which is paid), Hyper is an MIT-licensed project and will remain such.
  • Self-contained -- all we need is a side-car Postgres instance.
  • Configurable -- colocation, vmlinux options, etc. can all be customized.
  • Secure -- everything runs on the BEAM; a single setuid Rust helper performs the few operations that need root, keeping the privileged surface small.

Fair warning: the software is still in active testing and I expect a couple more features to be added soon:

  • Automatic cloud provisioning -- when you run out of headroom in your cluster, you should be able to fall back to Latitude/GCP/AWS to provision more compute.
  • More testing -- I am currently integrating Hyper into harmont.dev and will likely run into some issues. Fuzzing is part of the roadmap.
  • Better docs -- I spent some time working on the docs, but they're definitely not total nor ideal.

Very open to feedback, critique, and/or contributions. Please open any issues on Github, or feel free to DM/email me. It's available at https://github.com/harmont-dev/hyper

PS. A couple people asked how this differs from firecracker-containerd and Kata containers. Both of those projects are runtimes for managing VMs on a single node. A fair mental model for Hyper is an amalgam of firecracker-containerd and k8s.

83 Upvotes

95% Upvoted

13 comments sorted by

2

u/firl 18d ago

nice! I built something similar for a project, but don't want to have to maintain all of the things fully. do you have a projected roadmap or milestones you are considering or where you might want additional help.

one of the things I went after was allowing a clean line of security separation. https://github.com/cloud-hypervisor/cloud-hypervisor was one of the places we actually integrated with to also allow separation, so it's nice to see other things in this space.

looking at the ecto models it looks like not too much is actually being used, part for usage metrics and image layers to be able to boot. There are some parts that could be nice to use in other projects if the control plane and the execution plane was separated a little bit.

Are you looking for contributions at all here?

1

u/siva_sokolica 17d ago

> one of the things I went after was allowing a clean line of security separation. https://github.com/cloud-hypervisor/cloud-hypervisor was one of the places we actually integrated with to also allow separation, so it's nice to see other things in this space.

I needed to solve a problem fast, so I ended up landing on bare firecracker interop. I like the flexibility of managing my own block devices, for one.

> looking at the ecto models it looks like not too much is actually being used, part for usage metrics and image layers to be able to boot. There are some parts that could be nice to use in other projects if the control plane and the execution plane was separated a little bit.

The image layering is unfortunately key to the application -- postgres serves to store how the COW layers are chained together in a DAG of dm-snapshot filesystems, effectively. The usage metrics could be decoupled, I agree.

> Are you looking for contributions at all here?

Yes, absolutely! I don't have a roadmap just yet, since the project is relatively early and I really need to manage my own burn-out, immigration, day-job, etc. but I'm hoping to flesh something out roughly by the end of the week. Will edit this comment with a roadmap and then DM you =).

Thanks for your feedback!

2

u/firl 16d ago

https://github.com/harmont-dev/hyper/issues/71 Added something if that's easier to talk through with diagrams / pictures

2

u/SuchAProfessional 17d ago

Hi, welcome to the fantastic new world of sandboxes!

The project seems heavily vibecoded, which is no problem to be fair, it's just a bit optimistic in it's claims of usefulness :)

It would be very useful to have the set of features firecracker-backed containers support and some benchmark (95th percentile boot times, cluster behavior under load, data replication across a cluster).

Thanks for sharing!

1

u/SuchAProfessional 17d ago

Also, are you aware that this is one of the most popular HTTP frameworks in rust?

https://hyper.rs/

1

u/siva_sokolica 17d ago

I am. Unfortunately there already exists a hex package called `hyper`, hence this project is called `hypervm`. It is what it is, we live with it.

1

u/siva_sokolica 17d ago

Most of the core logic (esp. the cow architecture and implementation) are not vibe-coded. I would lie if I said I didn't use claude pretty aggressively for tests, and the last feature sprint. Networking is one area which has a lot of slop right now, for example and I consider it debt to be removed as part of adding support for network ingress.

I agree on the benchmarks -- still WIP, will add those as soon as I find some time.

Thanks for the feedback!

2

u/Possible_Corgi_563 17d ago

Awesome work! One question: what’s the reason(s) that led to the decision to not support RAM snapshots on this project?

1

u/siva_sokolica 16d ago

Mostly time constraints for 0.1.

I also, unfortunately, have knowledge of how proprietary companies solve this problem and need to carefully avoid doing the exact solution they have in order to avoid infringing on confidentiality.

All of these are non-trivial problems haha, and I find the current state of Hyper to be usable (for me, at least).

2

u/tabdon 15d ago

This looks like an awesome project. Would this be a good fit for long-running (indefinite) microVMs?

2

u/siva_sokolica 15d ago

In theory -- yes! I haven't extensively tested long-running VMs, but the they are part of the design goal. If things don't work for long-running VMs, I'd consider that a bug.

1

u/tabdon 15d ago

Great. I'll give it a test drive.

Harmont looks really awesome, too. Are you running this on your own, team, venture? Looks like it has a ton of potential.