Sylve looks like a decent project with a promising future but this article really doesn't explain why they picked it over Proxmox at all. They explain a lot of things but I can't see the advantage over prox other than they wanted to use it.
OP here. One thing we mentioned in the blog but probably didn’t emphasize enough is how deeply ZFS is integrated into the UI.
With Sylve, you rarely need to touch the CLI. Snapshots, datasets, ZVOLs, even flashing images directly to ZVOLs, it’s all handled from the UI in a straightforward way.
That tight ZFS integration also lets us build more flexible backup workflows. You can back up VMs, jails, or entire datasets to any remote machine that supports SSH + ZFS. This is powered by Zelta (https://zelta.space) (which is embedded directly into the Go backend), so it’s built-in rather than something you bolt on.
In Proxmox, you can achieve similar things, but it’s less intuitive and usually involves setting up additional components like Proxmox Backup Server.
I did actually notice the ZFS gui which is indeed something lacking in proxmox which doesn't default to ZFS in the installer. However once you do install it using ZFS it actually makes use of it pretty well and the user does not need to mess with the zfs cli tools much. Obviously it would be nice to have a GUI for all zfs operations too. Then again even TrueNAS refers you back to the cli for SOME operations.
On proxmox ZFS syncs do not require proxmox backup server, which actually has its own format which is very efficient in speed and disk space, but you do either need something like sanoid/syncoid or use of the shell.
Do you have any opinions on how this works vs doing iSCSI to some other storage system using ZFS? That's how I've been handling Proxmox on the backend, and have mixed feelings. The GUI leaves a very great deal to be desired in honestly curious ways, have to touch the CLI a lot even for super basic networking or auth stuff, and of course neither side has the same insight to the data structures in question. Either you've got to do ZVOL instances and thus manual effort or scripting, or you give Proxmox a single big blob then let it manage that with LVM but that means the storage side can't give any granular help on snapshots and the like. It still can deal with data integrity and backups and storage redundancy and all that but no further, and some increased overhead. But on the other hand, I do feel like a really firm separation of concerns isn't without value. Having native support though is an interesting alternative I hadn't really considered.
> They explain a lot of things but I can't see the advantage over prox other than they wanted to use it.
A huge, totally obvious, advantage is that FreeBSD isn't using systemd. I'm now nearly systemd-free, if not for Proxmox. But my VMs are systemd free. And, by definition, my containers too (where basically the entire point is that there's a PID 1 for the service and that PID 1, in a container is not systemd).
So the last piece missing for me is getting rid of Proxmox because Proxmox is using systemd.
I was thinking about going straight to FreeBSD+bhyve (the hypervisor) but that felt a bit raw. FreeBSD+Sylve (using bhyve under the hood) seems to be, at long last, my way out of systemd.
I've got several servers at home with Proxmox but I never, on purpose, relied too much on Proxmox: I kept it to the bare minimum. I create VMs and use cloudinit and tried to have most of it automated and always made it with the idea of getting rid of Promox.
I've got nothing against Proxmox but fuck systemd. Just fuck that system.
Whether an appliance OS uses SystemD or not is as silly of a concern as “does the lead developer prefer cheddar or brie”
What about performance characteristics? Recoverability of workloads?
I’m interested in a FreeBSD base OS because it seems ZFS is better integrated and ZFS has a lot of incredibly useful tools that come with it. If Bhyve is at least nearly as performant as KVM, I’d be hard pressed not to give it a whirl.
I run Proxmox at home, but now that I have been drinking the NixOS koolaid over the past 2 years, all of my homelab problems suddenly look like Nix-shaped nails.
Well it looks like we might soon be able to have the benefits of NixOS while also having bhyve (and presumably Sylve): https://github.com/nixos-bsd/nixbsd
I have the same thing with proxmox especially after I realized how well it integrates with proxmox backup server. And I haven't even gotten into clustering yet. It really is a very solid product.
This is really interesting. I've played with bhyve before but I didn't realise anyone actually used it in anger. And that people had written such great tooling around it.
My home lab still uses ESXi 8. But it needs something new and I was looking at proxmox. However I may give this a try first.
That’s true, bhyve doesn’t support nested virtualization right now.
In practice though, most setups don’t actually need it if you’re running workloads directly on the host.
Also, if your goal is testing or simulating clusters, you can already run Sylve inside jails. That gives you multiple isolated “nodes” on a single machine without needing nested virt. We have a guide for it here:
https://sylve.io/guides/advanced-topics/jailing-sylve/
So you can still experiment with things like clustering, networking, failure scenarios, etc., just using jails instead of spinning up hypervisors inside VMs.
Nested virt is still useful for specific cases like testing other hypervisors or running Firecracker inside VMs, but for most Sylve-style setups it hasn’t really been a blocker.
We run Proxmox VMs that are running Hashicorp's Nomad orchestration at $DAYJOB. The Nomad clients are then turning around and running the docker containers (Proxmox -> Nomad VM -> Docker). For us it's easier to manage and segregate duties on the initial metal this way.
Nested virtualization can be very handy in both the lab and in production. In the lab, you can try out a new hosting platform by running one atop the other. IE: Proxmox on VMWare, Hyper-V on KVM. This lets you try things out without needing fresh bare metal hardware.
In prod, let's say you run workloads in Firecracker VMs. You have plenty of headroom on your existing hardware. Nested virtualization would allow you to set up Firecracker hosts on your existing hardware.
Perhaps I'm misunderstanding, but wouldn't that case be covered by simply putting some vms under a vnet and others on another vnet and make them talk to each other? I can't also understand what you mean by "fresh bare metal hardware". In either case you don't need bare metal, being a top level vm or a nested one.
If you're evaluating VM hosts (proxmox, hyper-V, vmware, etc...) You need to have support for nested virtualization all the way down. Otherwise, if you want to evaluate a VM infrastructure, you need to start with bare-metal. Really, you just need to make sure that your top level support nested virtualization, but I understand their point.
However, the point about firecracker VMs in place of containers I think is really a good use-case. Firecracker can provide a better isolation environment, so it would be great to be able to run Firecracker VMs for workloads, which would require that the host (and the VM host above) support nested virtualization.
One example: when learning Proxmox itself. I was able to set up a multi-node cluster with more complicated networking than I was normally comfortable with and experiment with failures of all sorts (killing a node, disabling NICs, etc.) without needing more hardware or affecting my existing things.
Outside of learning and testing I am not sure of what uses there might be but I'm curious to know if there are.
OP here. It’s less about Sylve doing something Proxmox can’t do, and more about a bunch of QoL improvements that come from us being heavy Proxmox users and building what we felt was missing.
A few concrete things:
ZFS-first UX: Not just "ZFS as storage”, but everything built around it. Snapshots, clones, ZVOLs, replication, all cleanly exposed in the UI without dropping to CLI.
Simple backups without extra infra: Any remote box with SSH + ZFS works. No need to deploy something like PBS just to get decent backups.
Built-in Samba shares: You can spin up and manage shares directly from the UI without having to manually configure services.
Magnet / torrent downloader baked in: Sounds small, but for homelab use it removes a whole extra container/VM people usually end up running.
Clustering: but not all-or-nothing, You can cluster nodes when you need it, and also disable/unwind it later. Proxmox clusters are much more rigid once set up.
Templates done right: Create a base VM/jail once and spin up N instances from it in one go, straight from the UI.
FreeBSD base: It's not really a benefit of Sylve, but rather the ecosystem that FreeBSD provides.. Tighter system integration, smaller surface area, no systemd, etc. (depending on what you care about)
None of this is to say Proxmox is bad, it’s great. This is more "we used it a lot, hit some friction points, and built something that feels smoother for our workflows."
Without looking at the Sylve docs, I'll conjecture that it has deeper integration with ZFS. With a foundation on FreeBSD, there is a likelihood Sylve can support ZFS-on-root rollbacks better than hacking it into Proxmox. A rollback capability is why I'm looking for Proxmox alternatives. In the Linux world, Talos Linux and IncusOS provide A/B updates which achieve a similar rollback capability. With something based on FreeBSD, your "immutable" OS and all of it's data can be treated equally as ZFS datasets. There's also a higher risk that a Linux kernel update will break ZFS.
Regardless of the number of drives available, you gain an advantage when your file system can leverage snapshots to roll backwards or forwards. There are other Linux-native filesystems that can provide this capability too, but many admins prefer ZFS because the full range of capabilities is unparelleled.
Perhaps I'm missing your point, but proxmox+lxc on zfs storage works fine in proxmox? If just looks like any other storage in proxmox and on commandline you've got all the usual zfs tools
I think it comes down to the standard argument against ZFS on linux -- uncertainty. It works *now*. Will it continue to work? Will any upstream changes in the Linux kernel cause issues with the ZFS modules bolted on top?
It is unlikely for there to be issues with ZFS and Linux. It's too common now, but it's not included in the main Linux tree, so it's not explicitly tested.
So, it's a low risk, but not zero risk.
More to the point here, when working with FreeBSD, ZFS is a first-class citizen (moreso even), so working with it *should* be more integrated with a FreeBSD solution than Proxmox, but how much more (and is that meaningful) is probably a qualitative feel than quantitative fact.
Sylve appears to be a FreeBSD/BSD exclusive implementation of managing vms, etc.
Proxmox is Debian/Ubuntu based.
Both will have their advantages. It might not be about better or worse, the particular things you use may in some cases run better on BSD, or the security management could more fit what you are after.
I love FreeBSD but Linux just provides every feature under the sun when it comes to virtualization. Do you find any missing features on bhyve ? Is bhyve reliable ? I can't imagine its been tested as thoroughly as KVM ...
Bhyve is quite cool but no nested virt which means you cannot nest vm_enter/exit calls with EPT pages so you cannot virtualize within those guests. I found this crucial. For instance Qubes OS won't run in Bhyve by any means.
Anecdotally, Bhyve has worked in FreeBSD for a decade now. Eventually it got ported to Illumos because it was better than their implementation of QEMU.
If you are unsure of bhyve's abilities then why not test yourself? Speculation and guessing about stability or testing is useless without seeing if it works in your application.
> If you are unsure of bhyve's abilities then why not test yourself?
It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time.
That is why you ask for recommendations of hotels, restaurants, travel destinations, good computer brands, software and so on from friends, relatives or other trusted parties/groups. This does not mean your don't form your opinions. You use the opinions of others as a sort of bootstrap or prior which you can always refine.
HN is actually the perfect place to ask for opinions. Someone just said bhyve does not support nested virtualization (useful input !). Someone else might chime in and say they have run bhyve for a long time and they trust it (and so on...)
I agree with you and do not understand the “I read every manual” and “I test all software” crowd. I play around with A LOT of software but I cannot test it all.
Speculation is not useless if you are saying “the answer I got makes it 99% likely that this solution will not work for me”. Curation has immense value in the world today. I investigate only the options most likely to be useful. And that still takes all my time.
The phrasing of your questions is the problem. They are uninformed, too general, and assuming. The last sentence reads as if you outright dismiss bhyve because YOU can't imagine it was tested thoroughly.
> It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time.
It's totally possible when you know what your application requires but you didn't state anything.
> Someone just said bhyve does not support nested virtualization (useful input !).
Ok you have a problem with the way I framed my questions and my (unintentional) tonality. Fair enough. Let's move from critique of the way I asked my questions to what your experience with bhyve has been, if you're willing to share that.
Have you used bhyve ? What has your experience been with it ? Have you used KVM+QEMU -- can you compare your experience between both of them ?
With Sylve, you rarely need to touch the CLI. Snapshots, datasets, ZVOLs, even flashing images directly to ZVOLs, it’s all handled from the UI in a straightforward way.
That tight ZFS integration also lets us build more flexible backup workflows. You can back up VMs, jails, or entire datasets to any remote machine that supports SSH + ZFS. This is powered by Zelta (https://zelta.space) (which is embedded directly into the Go backend), so it’s built-in rather than something you bolt on.
In Proxmox, you can achieve similar things, but it’s less intuitive and usually involves setting up additional components like Proxmox Backup Server.
On proxmox ZFS syncs do not require proxmox backup server, which actually has its own format which is very efficient in speed and disk space, but you do either need something like sanoid/syncoid or use of the shell.
A huge, totally obvious, advantage is that FreeBSD isn't using systemd. I'm now nearly systemd-free, if not for Proxmox. But my VMs are systemd free. And, by definition, my containers too (where basically the entire point is that there's a PID 1 for the service and that PID 1, in a container is not systemd).
So the last piece missing for me is getting rid of Proxmox because Proxmox is using systemd.
I was thinking about going straight to FreeBSD+bhyve (the hypervisor) but that felt a bit raw. FreeBSD+Sylve (using bhyve under the hood) seems to be, at long last, my way out of systemd.
I've got several servers at home with Proxmox but I never, on purpose, relied too much on Proxmox: I kept it to the bare minimum. I create VMs and use cloudinit and tried to have most of it automated and always made it with the idea of getting rid of Promox.
I've got nothing against Proxmox but fuck systemd. Just fuck that system.
What about performance characteristics? Recoverability of workloads?
I’m interested in a FreeBSD base OS because it seems ZFS is better integrated and ZFS has a lot of incredibly useful tools that come with it. If Bhyve is at least nearly as performant as KVM, I’d be hard pressed not to give it a whirl.
I run Proxmox at home, but now that I have been drinking the NixOS koolaid over the past 2 years, all of my homelab problems suddenly look like Nix-shaped nails.
Looks like Nix will eat the world soon. :)
[1] https://github.com/EnigmaCurry/nixos-vm-template
I actually have a few hosts that only run docker. I might be able to test with those.
Likewise for disk i/o -some people swear by 9P as a backing mechanism, some by ZVOL.
My home lab still uses ESXi 8. But it needs something new and I was looking at proxmox. However I may give this a try first.
In practice though, most setups don’t actually need it if you’re running workloads directly on the host.
Also, if your goal is testing or simulating clusters, you can already run Sylve inside jails. That gives you multiple isolated “nodes” on a single machine without needing nested virt. We have a guide for it here: https://sylve.io/guides/advanced-topics/jailing-sylve/
So you can still experiment with things like clustering, networking, failure scenarios, etc., just using jails instead of spinning up hypervisors inside VMs.
Nested virt is still useful for specific cases like testing other hypervisors or running Firecracker inside VMs, but for most Sylve-style setups it hasn’t really been a blocker.
In prod, let's say you run workloads in Firecracker VMs. You have plenty of headroom on your existing hardware. Nested virtualization would allow you to set up Firecracker hosts on your existing hardware.
However, the point about firecracker VMs in place of containers I think is really a good use-case. Firecracker can provide a better isolation environment, so it would be great to be able to run Firecracker VMs for workloads, which would require that the host (and the VM host above) support nested virtualization.
Outside of learning and testing I am not sure of what uses there might be but I'm curious to know if there are.
Or better, how does it do it better than proxmox?
This isn't to say that proxmox is the best thing since sliced bread, I'm curious as to what makes sylve better, is it the API?
A few concrete things:
ZFS-first UX: Not just "ZFS as storage”, but everything built around it. Snapshots, clones, ZVOLs, replication, all cleanly exposed in the UI without dropping to CLI.
Simple backups without extra infra: Any remote box with SSH + ZFS works. No need to deploy something like PBS just to get decent backups.
Built-in Samba shares: You can spin up and manage shares directly from the UI without having to manually configure services.
Magnet / torrent downloader baked in: Sounds small, but for homelab use it removes a whole extra container/VM people usually end up running.
Clustering: but not all-or-nothing, You can cluster nodes when you need it, and also disable/unwind it later. Proxmox clusters are much more rigid once set up.
Templates done right: Create a base VM/jail once and spin up N instances from it in one go, straight from the UI.
FreeBSD base: It's not really a benefit of Sylve, but rather the ecosystem that FreeBSD provides.. Tighter system integration, smaller surface area, no systemd, etc. (depending on what you care about)
None of this is to say Proxmox is bad, it’s great. This is more "we used it a lot, hit some friction points, and built something that feels smoother for our workflows."
Can you explain your use case when you absolutely can't provide a separate M.2 drive solely for the OS?
It is unlikely for there to be issues with ZFS and Linux. It's too common now, but it's not included in the main Linux tree, so it's not explicitly tested.
So, it's a low risk, but not zero risk.
More to the point here, when working with FreeBSD, ZFS is a first-class citizen (moreso even), so working with it *should* be more integrated with a FreeBSD solution than Proxmox, but how much more (and is that meaningful) is probably a qualitative feel than quantitative fact.
A Un*x system that doesn't use systemd as an init system.
Proxmox is Debian/Ubuntu based.
Both will have their advantages. It might not be about better or worse, the particular things you use may in some cases run better on BSD, or the security management could more fit what you are after.
I wonder why not run both :).
Proxmox is due for it's viral moment though.
Also: https://www.youtube.com/watch?v=wo4oD5UON30
It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time.
That is why you ask for recommendations of hotels, restaurants, travel destinations, good computer brands, software and so on from friends, relatives or other trusted parties/groups. This does not mean your don't form your opinions. You use the opinions of others as a sort of bootstrap or prior which you can always refine.
HN is actually the perfect place to ask for opinions. Someone just said bhyve does not support nested virtualization (useful input !). Someone else might chime in and say they have run bhyve for a long time and they trust it (and so on...)
So I can't agree with your viewpoint.
Speculation is not useless if you are saying “the answer I got makes it 99% likely that this solution will not work for me”. Curation has immense value in the world today. I investigate only the options most likely to be useful. And that still takes all my time.
> It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time.
It's totally possible when you know what your application requires but you didn't state anything.
> Someone just said bhyve does not support nested virtualization (useful input !).
What nested applications are you planning to run?
Have you used bhyve ? What has your experience been with it ? Have you used KVM+QEMU -- can you compare your experience between both of them ?