Skip to content

Home

The owner's homelab at two sites, folly and offsite, managed as code: its apps, platform, hosts and runbooks.

This wiki explains how to use, operate and change the owner's homelab. The infra repository declares its configuration.

The lab has two sites: folly at home and offsite at a remote site. Each site has a UniFi network and a Kubernetes cluster. Kubernetes nodes run Talos Linux. NixOS hosts run supporting services. OpenTofu declares network and cloud resources.

Learn the lab#

Read How changes ship to learn which controller applies each kind of change.
Read Network to understand how the sites connect.
Read Folly design to understand node installation, identity and trust limits.
Read fleetd to learn how the node lifecycle controller works.
Use a runbook when you need to perform a task.

The glossary defines lab-specific terms. The wiki is public. Each merge of docs/ to main publishes it.

Start here#

Network: the two sites and how they connect
Hosts: each machine, its job and how to reach it
Runbooks: the procedures, grouped by area
Test a change: the local check for each kind of change, before you open a pull request
Glossary: the lab's own terms
Connect an agent to the wiki or to kthx: give an MCP client these pages, or the kthx tools

Apps #

kthxliveThe lab's hosting product on the offsite cluster, which the owner and agents use to serve a directory at <name>.kthx.dev or to build a repository and deploy it to a cluster or a cloud.RowbuttliveA chat bot, code name mate, that answers the owner in Discord and Slack threads with a coding agent whose tools run in a sandbox on the offsite cluster.clankerbankerliveA public pay-per-call API at clankerbanker.ca, a kthx built app on offsite, that sells fortunes, dice rolls, storage and model answers to bots for USDC over x402.Flame Boss exporterliveA Go exporter on folly that reads barbecue cooks from Flame Boss's cloud into Prometheus, for a Grafana dashboard and cook alerts in Discord.PBXliveTwo Asterisk phone switches, one on folly for the four lines of the office phone and one on offsite that hands its callers to an ElevenLabs troll agent.ElevenLabsliveThe voice agents and the phone number that git declares for the PBX and Switchboard, and the offsite CronJob that makes the ElevenLabs account match them.SwitchboardliveA Bun service with two roles, a live board of the folly PBX's calls and a ringer on offsite that phones the owner through an ElevenLabs voice agent.pulsifer.caliveThe owner's personal website and CV at pulsifer.ca, a Hugo site that GitHub Actions builds and GitHub Pages serves.SlingshotliveA public web tool on Vercel that captures, inspects, diffs and replays HTTP webhooks, with one feed for each project.Smiirl counterliveA Go service on the folly Kubernetes cluster that replaces Smiirl's cloud, so the owner sets what the Smiirl counter shows from a web page.Tidbyt appsliveSix Pixlet apps, rackstat, tempest, wishin, callerid, burnsafe and flameboss, that the Tronbyt server on folly renders for the lab's Tidbyt pixel displays.Weather HubliveA web dashboard on offsite that shows the owner's WeatherFlow Tempest weather stations, on the kiosk displays and on phones.Weather APIliveAn API on offsite that answers Canadian weather questions from Environment Canada and the family's Tempest stations, for apps and for Rowbutt.WikiliveThis documentation site at wiki.lolwtf.ca, which a Bun renderer builds from docs/ and Cloudflare Pages serves, with a read-only MCP endpoint for agents.LookoutliveA Windows app, installed from an MSI, whose service keeps the monitoring agents on a Windows desktop at the versions git declares and reports its own health to folly. tallboy runs it.

Internal tools#

netbenchliveAn internal web tool on the folly cluster that runs iperf3 tests from a pod to Kubernetes nodes, NixOS lab hosts outside Kubernetes and the offsite cluster.fleetdliveThe service on spore that checks machine identity and controls folly node installation.KeylimeexperimentThe remote attestation verifier for folly's running nodes, being introduced beside fleetd's own check.

Parked#

BosunparkedA Go daemon that keeps a warm pool of one-job GitHub Actions runners in Cloud Hypervisor microVMs. No host runs it.

Platform #

How changes shipThe path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.WorkloadsWhat runs on the folly and offsite Kubernetes clusters, the files that list it, and the workloads that have their own page.NetworkThe two sites, their UniFi gateways and networks, and the tunnel, routes, ingress and tailnet that connect them.SecretsWhere secrets are stored, what decrypts them, and how they reach hosts and workloads.BackupsHow Garage, Velero, CloudNativePG and etcd snapshots back up the lab's databases, volumes and etcd, and how each site copies them to GCS.NetBoxThe record of every machine in the lab, on capsule, reachable only over the tailnet.PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.IdentitySPIRE on spore gives each folly node with a TPM 2.0 an identity rooted in its TPM, by LDevID or EK hash, gives its pods SPIFFE IDs, and lets a pod reach GCP with no key file.Node lifecycleHow a folly node goes from a NetBox device to an attested Talos node with a SPIRE identity, and back out, with the system that owns each step.OpenTofu and AtlantisThe OpenTofu roots that declare the network, cloud accounts, identity and cluster bootstrap, and the Atlantis server that plans and applies them from pull requests.Build and releaseHow first-party code builds, how its images and charts publish and deploy, how Renovate updates dependencies, and which CI checks run on a pull request.Cloud accountsThe Google Cloud projects, Cloudflare zones and Google Workspace domain that OpenTofu declares, and what each one is for.ObservabilityThe metrics, logs, traces, dashboards and alerts on each Kubernetes cluster, the hosts and Windows desktops they watch, and the daily k6 check.