I tried this in my homelab, and I wanted to convert my Proxmox with it’s LXC’s over to it…but the problem I ran into was that it was a little too obtuse for what I wanted to use it for.

Long story:

I wanted to get away from Proxmox as a platform and LXCs in general over to another hypervisor and using standard containers. I was tired of having a full fat web GUI to manage things with. Yes you can use pct and other CLIs tools in Proxmox to get some speed, but you are still dealing with an opinionated, dare I say proprietary stack.

Incus is really cool in that it’s CLI driven first, and you can add a GUI later if you want. I thought that was nice, and I was able to get some VMs started in my lab cluster and get clustering kinda working. I thought I was in a good place to switch.

But then, my main file server died (thankfully just the mobo, not the ZFS disks). And I was scrambling to figure out how to get all of my functionality back without having to go back to adding the storage back to Proxmox. So I took it as an opportunity to go to incus.

Well it turns out, it’s not so turn key depending on your method of deployment. For one, the actual “IncusOS” deployments are really particular about having a newer version of TPM (not something the old boxes I purchase typically have). So you are still likely to install something like Debian + incus-cli tools.

Once you have your distro + incus package setup though, there are a few things you have to tackle. For one, if you actually want separate IP addresses per VM (or LXC) you’ll likely need to solve the chicken and egg problem with macvlan, otherwise you’ll have to figure out something for a reverse proxy with the network bridge.

So because I was pressed against the wall (my family was irritated about no jellfyin or immich), and I knew that I didn’t want to do LXCs again and simply wanted to do most things I can in containers, I went the podman route instead.

Every app gets it’s own local user. You can install podman-compose, and use the same commands as docker compose basically to get your app up. All rootless, and you can also setup podman compose apps with daemons so that they start on boot: https://www.it-hure.de/2024/02/podman-compose-and-systemd/. If you don’t care about rootless isolation, I think Docker is still fine. Everyone seems to make documentation and build their apps with it in mind still.

I still had a need for VMs, so now I just use the standard Qemu+KVM+libvirtd combo for the one or two apps that need that need a full on separation from other apps.

All of it is in Ansible playbooks now, and I’m pretty happy with the setup I think.

TL; DR

Incus is really cool. I probably wouldn’t bother with IncusOS on old hardware, I’d just install a standard Linux distro + incus packages. Spend your time evaluating it (and making it infrastructure code for easy deployment maintenance). Maybe try out the other “standard” ways to do it instead to compare (Docker, Podman, Qemu+KVM, etc).

I decided to go with IncusOS when I got a new HP EliteDesk i5 8th gen as a server a while back.

I’ve had some issues with it mostly regarding my setup with iSCSI. Of course me running iSCSI on my NAS is probably not the best way to do it but the whole incus install froze up as soon as it got disconnected in any way. Which includes the NAS updating or the router updating.

There’s also some limitations as Incus is rootless that makes a few things difficult like setting up an SMB share in a container since that’ll require some kernel permissions. If you have the RAM or storage to do this in a VM instead it wouldn’t be much of an issue as they’re not limited to the rootlessness of the instances. Different workaround for file shares would be fuse like sshfs since it happily runs as a normal user.

I do love the experience of manually managing the servers in Incus very very much. On my machine I can just run incus shell instance-name and I’m in. Inside of my instances I often run Debian with Podman quadlets or Docker if needed and it works great. With podman updating the containers is also a breeze.

On top of that if you read into cloud-init a little but you basically get a Hetzner like experience. Fully configured and updated distro with your favourite packages and your own certificates installed, if you use that, directly and ready to go. It’s amazing and a great time server when starting a new project.

IncusOS has gotten some nice updates in recent months like Netbird support and Wireguard as two examples. They seem to have a keen ear on community wishes. The forums are great too and project maintainers are nice and respectful from what I’ve seen.

The installation procedure with the secure boot keys where a bit of a hassle but once running it is nice. Updates are automated and shows you if a reboot is required. The web UI makes handling most tasks and doing maintenance a breeze.

I can wholeheartedly recommend it from my experience with a small “but”. If you want to automate or use kubernetes or learn more enterprise “what is mostly used” stuff it can be difficult. I wanted to see if I could get K3s running as a bit of a test and learn but it doesn’t seem to want to run in a rootless container. Giving the explicit required permissions also seem difficult to do or find and isn’t straight forward. There is some plugin stuffs for kubernetes to use Incus directly as well but I couldn’t wrap my head around it or feel like it would give me anything for my small homelab.

I tried automating the instance updates with Ansible but it failed 90% of the time with context cancellations. Having to continuously rerun the Ansible playbook until I gave up and did it manually. Eventually I just wrote a bash script using bash arrays and for loops to do what I wanted.

In short, it’s great if you just want to manage some stuff in your homelab and separating things. I’m sure it’s great in larger enterprise-esque setups too if you can take the time and brain power to learn everything around it since the availability and support is there but the instructions and documentation for those things requires existing knowledge around it. Unlike getting Incus up and running in a more basic setup which is greatly documented and easy to follow along most of the time.

Use Incus on Nixos, runs my Blocky local DNS server i pass into tailscale for specifically my domain and subdomains. Doesnt give me any trouble

I didn’t follow the instructions carefully enough and the installer bricked a motherboard by incorrectly enrolling secure boot keys. The board doesn’t have dual bios or a way to flash without POST, the SPI has to be removed, flashed, and resoldered, which surprised me that gigabyte would not have a recovery plan in place across all their boards but that’s my fault for not checking what I bought.

TLDR: read the instructions in their guide carefully. You can brick a board with this.

Did consider it about a year ago but seemed too early at the time compared to the stability of Proxmox. It dles however look like it has come quite far. Will add it to my to-checkout-list 😊

I use Incus on Debian 13. Debian 13 packages Incus, and that version only does security updates and focuses on stability. To be clear, Debian’s model is to not do bugfixes or feature updates, to prevent any update from being breaking in any way. This ensures users can automatically get security updates.

I get it from zabbly’s repos: https://github.com/zabbly/incus , in order to get newer versions of Incus, and to get, however. There is a risk of breaking updates but I’ll live and fixing the occasional thing on my one server is okay. The Zabbly incus repos also come with the UI, although the UI is just a static site you can put in the right spot. I like that design, because it means that only the API can do anything, which lessens the attack surface.

I also use Incus personally on my laptop on Arch Linux. It’s nice to have the same UI/UX for vm management, and being able to download the premade images from their docker like repository is pretty nice as well.

I want to upgrade to it from Proxmox when I eventually get a new processor. It seems perfect for my use case; proxmox is far too complex for what little I do with it.

I’ve been using Incus for awhile now, but TIL there’s an IncusOS.

Sounds neat but I would be worried about the no shell stuff, I run my things headless for the most part and like SSH and occasionally VNC access.

Running Incus on a headless Debian 13 host using a Ryzen 2700 CPU with an uptime of at least 200 days, no issues. Really like the ability to seamlessly pass through a GPU on any container running the same kernel. Used to run ESXi, never looking back.

I’ve used Incus (but not incusos) for over two years now. Works well.

Stability is great, flexibility to do what I need is also great.

Also there are quite a few options for deployment, I use ansible.

I’ve been running IncusOS on a 10 year old Dell Inspiron 22-3263 with no issues for around a year now. The install process was straightforward, even for this older hardware. Administration is via an intuitive web GUI, and if you’ve ever used Ubuntu Micro Cloud, you’d already be quite familiar with the interface, as Incus is a fork of that, IIRC.

Is incus-compose a part of the base system, or an add-on? How does it work on IncusOS if it’s a command-line tool and there’s neither a remote nor a local shell, only a web API? Does it provide a built-in way to automate container updates?

Haven’t heard about it. I probably should give it a try.