Closing the loop: my homelab, from a YAML file to a running server
Regular readers (hi, both of you) know my long-standing habit of memory-based chaos when it comes to my home infrastructure. ("Hi, my name's Dan, and I have a self-hosting click-ops problem."..."Hi, Dan!"). I'm in active recovery: first configuring my daily driver with Ansible, then moving my domains and this site into Terraform. Although I've not written about it much, there's a related effort on which I've been working off and on for over a year.
Today was a banner day: a new container and a new VM went from a few lines of YAML to fully configured, SSH-able servers running real services
without me touching either of them by hand.
That's the last link in a chain I've been building for a while, and it feels like a good moment to step back and brag describe the whole thing.
No code walkthroughs today, just the overall shape and the tools I chose. And yes, I'm already second-guessing myself, but that's okay...
Four verbs
The whole system boils down to four steps, each with one job:
- Define what should exist (don't forget to validate it, too!)
- Render that definition into the inputs each tool expects.
- Provision the systems that don't exist yet.
- Configure everything that does.
Each step is easy to run on its own, inspect, and debug. That turned out to matter more than any individual tool choice.
Define: one source of truth per site
Everything starts with a plain YAML description of a site, meaning a physical location with its own network. The heart of it is a device list: every server, container, VM, switch, and gadget, with its name, address, DNS names, what it's for, and which Ansible roles it should get. Next to that sit the site's VLANs and a handful of site-wide settings.
Two decisions shaped this step more than anything else:
- Code and data live in separate repos. The framework (roles, scripts, Terraform modules) knows nothing about my network. My network's description lives in a private repo of its own. The framework never says "Dan's Immich server"; it says "this is what an Immich server should look like".
- No secrets in files. Ever. Wherever a password or token is needed, the data holds a reference to an item in 1Password, and the tooling looks it up at the moment it's used. Nothing sensitive is ever written to disk.
Alternatives: I considered using secrets managers other than 1Password, and I suspect that any of them which support secrets access via CLI could easily drop in place. (I did throw a thin shim layer in my scripts to support just that, because YAGNI is for purists). That said, I already live in 1Password all day, and its CLI made the "reference now, look up later" pattern easy.
Render: turning the definition into tool inputs
Every downstream tool wants its input in its own shape. Ansible wants an inventory, the DHCP server wants reservations, DNS wants records, and Terraform wants variables. Rendering is the step that produces all of them from the one site definition, as Plain Old Text Files I can read before anything acts on them.
It's implemented in Bash plus yq, which is a sentence I expected to regret. Mostly I don't.
Rendering is deterministic, it's safe to run any time, and it fails loudly on bad input (a device with no DNS name, an instance pointing at a host that
doesn't exist).
I did learn that yq comes in more than one flavour, and each flavor(lol!) has varying opinions on how to handle edge cases.
Alternatives: Python with Jinja2 is the obvious choice and probably what most people would reach for. (Heck, I'll probably do so when I rewrite this for no good reason)... I wanted something with no runtime to install beyond two binaries, and the render step is small enough that Bash hasn't hurt yet. (Famous last words.)
Provision: making systems exist
This is the link that was missing until today. My existing servers mostly live as containers and VMs on Proxmox, created with some helper scripts and a fair number of manual steps. The new approach uses Incus, the community fork of LXD, which runs both system containers and full VMs behind one clean API, on a stock Debian box. Is this better than Proxmox? Unsure yet, but it is definitely more API-first and easier to grok for my poor CLI-oriented brain. Yes, Proxmox has it's own suite of perfectly valid CLI tools, but I like the remote CLI model Incus uses, and post-provisioning bootstrapping scripts were far easier to arrange there.
Terraform drives it. One generic module, which knows nothing about any particular site, creates whatever instances the rendered input asks for. Each site gets its own remote state, stored in GitLab like my Cloudflare stacks, so a fresh clone on another machine picks up exactly where the last one left off. Instances join the LAN like any other device, with the hardware address the site definition gave them, and the render step has already produced the DHCP reservation and DNS record they need. (Pushing those to the router is still a manual step. It's on the list...I'm looking at you, Unifi API...or maybe I'll get truly cray-cray and spin up my own virtual network appliance for DNS and DHCP..)
Then comes the part I'm happiest with. As soon as an instance exists, the provisioning step reaches into it through Incus's authenticated API,
runs my standard bootstrap script (create the automation user, install SSH keys, lock down sshd), and reads back the new machine's SSH host keys.
Those go straight into my known_hosts. Because they came over a channel I already trust, there's no "are you sure you want to continue
connecting?" prompt, and no habit of blindly typing yes. Rebuild an instance from scratch and its new keys quietly replace the old ones.
Alternatives: OpenTofu is the open-source fork of Terraform, and should be a drop-in replacement for the way I use it (I haven't tried it yet). Proxmox has community Terraform providers too; I went with Incus for its single API across containers and VMs.
Configure: making systems do stuff
Configuration is Ansible, which has grown a long way past the workstation playbooks from that 2024 post. There are now about three dozen roles: a baseline every server gets (users, SSH hardening, time sync, and the like), plus one role per service (media, photos, home automation, calendars, feeds, identity, backups, and so on). Every role is idempotent, pins its versions, and gets its secrets from 1Password at runtime, and (nearly!) every service's backups have been through one or more live restore drills in production.
Because the inventory is rendered from the site definition, giving a machine a new job means adding a group to its entry and running one command.
Alternatives: To be honest, I haven't really explored alternatives to Ansible. Most required client daemons or some such; I like just doing magic over the same SSH channel I'll use for ad-hoc admin tasks. That said, I'm curious to test drive Salt, Puppet, and Chef. They all solve this class of problem, with agents and a more server-centric model. NixOS is the most interesting option: describe the entire machine declaratively and rebuild it atomically. I'm Nix-curious, but Ansible's "it's just SSH" model is hard to beat for a mixed fleet of Debian boxes, an Unraid server, and a repurposed Surface tablet.
Today's milestone
Here's what it looks like end to end now. I add a short entry to a site's device list: name, address, "container, one core, half a gig of RAM, Debian 13". I run the provisioning script. A minute later the machine exists, is on the network, is bootstrapped, and its host keys are trusted. I run the configuration script, and it comes up with the full baseline. Today I built a container and a VM that way, threw both away, rebuilt them from scratch just to watch it work, and configured them without a single error.
It wasn't all smooth sailing. A few highlights from the blooper reel:
- The Incus host fell off the network, completely and silently, right after downloading its very first image. I never reproduced it, including during a download three times larger. If anyone has a theory, my inbox is open; for now it's filed under "haunted".
- The first VM refused to boot because, in an effort to avoid installing a desktop environment's worth of dependencies on a headless server, I'd also skipped the one package QEMU needs for its virtual console. Minimalism has a price.
- The bootstrap step quietly handled only the first new instance, because the command that reaches into an instance helpfully read its input from the same place my loop was reading the list of instances. Oops. (I *would* blame Claude for this, but, well, it's ultimately me claiming credit, so I need to take the blame too.)
What's next
The obvious next step is retiring Proxmox, once I've decided to set up permanent residence in the Land of Incus: rebuilding each existing server on Incus through the same pipeline, restoring its data from backup rather than copying disks around. The backup and restore drills I've been working through recently were, in hindsight, rehearsals for exactly this. Before that, a small new control-plane site gets to be the production-use guinea pig, hosting its own Git server and file sync, so that eventually my lab can self-manage most of the tools it depends on.
And one more thing. The framework half of all this has always been written with other people in mind: no hardcoded names, sample data alongside the real data, documentation for every role. At some point it might just escape into the wild, along with a walkthrough that starts with a spare PC and a Unifi gateway and ends with a basic "piss off, Big Cloud!" homelab. No promises on timing, though...I am a busy boy, so don't hold your breath.
Summing up
- Separate what (your data) from how (your code). Everything else gets easier.
- Give each step one job and an inspectable output. Debugging becomes a matter of simple inspection "what's wrong in this file?" instead of "what is happening?!?"
- Secrets are references, never values.
- Rebuild things on purpose, often. It's the only way to know you can.
-
As with the Cloudflare move, I did a lot of this with Claude Code. It wrote much of the plumbing and caught (and, in fairness, introduced) a few
of the bugs above, and I still read every plan before typing
yes.