🚒Runbooks

• Operational procedures for when things misbehave. These pages are served from Cloudflare, off the infrastructure they describe, so they stay readable during an outage.

• The runbooks

  • Runbooks/Deploy a NixOS Host — build, deploy, verify, and roll back NixOS hosts

  • Runbooks/Terraform Change — Atlantis-first OpenTofu workflow and local validation

  • Runbooks/Kubernetes GitOps Change — inspect Flux, reconcile resources, and handle SOPS safely

  • Runbooks/Managed Postgres — reach a CloudNativePG database through kubectl cnpg, and check whether it has a backup

  • Runbooks/OpenBao Bootstrap — initialize and verify the folly OpenBao instance

  • Runbooks/Add Shared Kubernetes Resource — use the clusters/base/ pattern for both clusters

  • Runbooks/Adopt Folly Monitoring CRDs — stamp Helm ownership metadata onto folly's pre-existing Prometheus Operator CRDs before wiring monitoring-crds

  • Runbooks/Validate Infra Changes — validation commands by change area

  • Runbooks/Inspect UniFi Network — read-only UniFi discovery before making changes

  • Runbooks/SOPS Secrets and Age Keys — operator age key, harmonia keypairs, two-stage sops-nix recipient setup, decryption-failure triage

  • Runbooks/Cluster Admin Access — JIT tokens for day-to-day kubectl, the break-glass certificate, and how to withdraw access

  • Runbooks/Kiosk — the Raspberry Pi kiosk hosts: Cage/Wayland, Firefox, container-backed apps

  • Runbooks/Adopt Folly Prometheus Operator CRDs — the one-time live ownership stamp and Kustomization wiring that lets folly join monitoring-crds

  • Runbooks/Install Spindrift — from nothing to an enrolled, Target-connected Spindrift installation: Terraform bootstrap, chart declaration, first-operator enrolment

  • Runbooks/Connect an Agent to the Wiki — point Claude Desktop or any MCP client at wiki.lolwtf.ca/mcp so an agent can read the homelab docs

  • Runbooks/Connect an Agent to Spindrift — mint an agent token and point an MCP client at spindrift-control.lolwtf.dev/mcp to drive the platform

  • Runbooks/Developer Connect GitHub OAuth — one-time browser authorization that moves the trusted-builds GitHub connection from PENDING_USER_OAUTH to COMPLETE

• Conventions

  • Tag runbook pages #runbook, lead with quick checks, then symptom-shaped sections ("If X…"), each with copy-pasteable commands and expected output.

  • Prefer mise run <task> where a task exists; it encodes the correct binary and flags. Give a raw invocation only where mise has no task — deploying to a live host, sops, flux reconcile.

  • Commands assume the reader is on the LAN or the tailnet. Note when a host needs a special path.

  • A runbook that changes desired state still ships through git — see Architecture/GitOps.

  • Where an agent skill covers the same workflow, this page is the durable source. The skill carries a runbook: pointer in its frontmatter and holds only agent-specific guidance; it does not restate the procedure.

  • Do not symlink SKILL.md files into docs/pages. Skills are agent instructions, not Logseq outline pages, and this site is public.

Edit this page on GitHub