infra_wiki

How changes ship

The path each kind of change takes from a pull request to the running system, and the GitOps rule that no one changes live state by hand.

Every change to the lab ships from git. A pull request (PR) declares the change, and a controller or a workflow applies it. The GitOps rule says that nobody changes live state by hand. The next apply overwrites a hand change or fails on it.

Paths#

ChangeApplied byWhen
An OpenTofu root under terraform/ or clusters/<site>/bootstrap/Atlantis, the server that applies OpenTofu from PRsWhen someone comments atlantis apply on the PR. Atlantis then merges the PR.
The Talos Linux machine configuration of a cluster, an OpenTofu root under clusters/<site>/talos/Atlantis, over the Talos API of each nodeWhen someone comments atlantis apply on the PR. A node whose installer image changes reboots.
A Kubernetes manifest under clusters/Flux, the GitOps controller in each clusterAt the next sync of main after the merge
A NixOS host under nix/The host's auto-upgrade, or nixos-rebuildAt the next daily auto-upgrade after the merge
A first-party imagecontainers.yml, then FluxAfter the merge, the workflow builds the image and opens a PR that pins its digest. See Build and release.
A wiki page under docs/wiki.ymlOn merge, to Cloudflare Pages. See Wiki.
A folly node's lifecycle: its NetBox device's desired_action or decommissioning statusfleetd, through TinkerbellWhen the owner sets the field in NetBox, within two minutes. See Provision a node.

Kubernetes#

The bootstrap root of each cluster is the OpenTofu root in clusters/<site>/bootstrap/. It installs flux-operator and a FluxInstance, the object that configures Flux. The FluxInstance syncs the infra GitRepository, Flux's copy of this repository, at refs/heads/main. It starts from the Flux Kustomizations in clusters/<site>/flux-system/, which name the paths that Flux applies.

folly syncs every hour, and offsite every five minutes. To sync a merge sooner, reconcile the infra GitRepository. A reconcile of a Flux Kustomization alone applies the revision that Flux already has.

Flux applies the manifests under clusters/. kthx owns the resources of each App it deploys, and who owns what states that boundary.

NixOS#

Auto-upgrade rebuilds a host from main each day. NixOS lists the hosts that have none. Those change only when someone deploys them with nixos-rebuild or the nixos-deploy workflow.

Rules#

  • Do not run kubectl apply to change state. Use kubectl, flux get and flux reconcile to inspect or to force a sync.
  • Do not run tofu apply on your machine. It takes the state lock from Atlantis and causes drift. The one exception is the owner's: talosctl bootstrap and the first tofu apply of the clusters/<site>/talos/ and clusters/<site>/bootstrap/ roots, once per rebuilt cluster, until Flux syncs it. Rebuild a cluster on Talos has the steps. Atlantis owns both roots from the next PR.
  • Apply an OpenTofu PR before you merge it, as OpenTofu and Atlantis says.
  • Merge a host change on the day you deploy it from a branch. The next auto-upgrade rebuilds the host from main and removes the change.

Where it lives#

  • clusters/<site>/bootstrap/flux-values.yaml: instance.sync, the source, path and interval of the Flux sync. A change to it gets no autoplan, as OpenTofu and Atlantis says.
  • clusters/<site>/flux-system/: the Flux Kustomizations of each cluster
  • nix/system/nixos.nix: system.autoUpgrade
  • .github/workflows/nixos-deploy.yaml: the deploy workflow for the Pi hosts

Edit on GitHub

Linked from 10
  • HomeThe owner's homelab at two sites, folly and offsite, managed as code: its apps, platform, hosts and runbooks.
  • Build and releaseHow first-party code builds, how its images and charts publish and deploy, how Renovate updates dependencies, and which CI checks run on a pull request.
  • PlatformThe shared systems that the apps and hosts depend on, and what applies each layer from git.
  • KubernetesThe folly and offsite Kubernetes clusters, how Flux applies clusters/ from git, and the sandbox runtimes and storage the clusters offer.
  • OpenTofu and AtlantisThe OpenTofu roots that declare the network, cloud accounts, identity and cluster bootstrap, and the Atlantis server that plans and applies them from pull requests.
  • Folly node lifecycle and trust designThe technical design, tradeoffs, evidence and threat boundaries for node provisioning, trusted boot and workload identity at folly.
  • GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
  • Style guideThe rules for writing and reviewing wiki pages, runbooks and code comments in this repository.
  • Apply a Kubernetes changeChange the manifests under clusters/, share a resource between the two clusters, validate the change, and make sure Flux applies it after the merge.
  • Apply an OpenTofu changeChange an OpenTofu root, check it on your machine, and apply it through Atlantis on the pull request.

↑↓ move ↵ open esc close