Runbooks

Runbooks are the step-by-step procedures that the owner and agents follow to change, check or repair the lab. The wiki is on Cloudflare Pages, outside the lab, so the runbooks stay readable when the lab is down.

Hosts#

• Deploy a NixOS host: deploy a host's configuration or restore its previous generation
• Install Talos on a node: boot a NixOS node into the Talos installer with kexec, and check its volumes
• Prepare a machine: set a new folly machine's firmware and enroll the lab's Secure Boot keys over PXE
• Provision a node: declare a new folly machine in NetBox and install it as a Talos worker through its desired_action
• Reprovision a node: reinstall, attest again or wipe an installed folly node by its desired_action
• Retire a node: drain a folly worker, set its NetBox device to decommissioning, and drop it from the exports
• Build the signed Talos images: build folly's Secure Boot installer, ISO, iPXE and HookOS, and check what the build publishes
• Change the forge EEPROM: change forge's boot settings and keep its HTTP boot fallback
• Install a Windows desktop: install and update the owner's Windows configuration
• Install Windows monitoring: add a desktop to folly's Prometheus and repair its agents

Kubernetes#

• Apply a Kubernetes change: change clusters/, share a resource between the clusters, and make sure Flux applies it
• Get cluster admin access: get kubectl access and withdraw access
• Rebuild a cluster on Talos: replace a cluster's NixOS nodes with Talos, bootstrap etcd, Cilium and Flux, and restore its backups
• Make a Talos secrets bundle: generate a cluster's Talos secrets with its imported signing key, and pin its new CA
• Install Cilium and Flux on Talos: write a new cluster's kubeconfig context, install Cilium with its BGP peering, and bootstrap Flux
• Move a cluster's data through a Talos rebuild: freeze the writers, take the last backups, and restore the databases, volumes and kthx Apps
• Verify a Talos cluster: check the nodes, network, identity and workloads of a cluster that runs Talos
• Issue a talosconfig: mint the owner's and Rowbutt's Talos API credentials and store them in 1Password
• Rotate the cluster bootstrap token: replace folly's cluster-wide bootstrap token in the Talos bundle, apply it, and delete the old one
• Add a TPM node to SPIRE: read a node's TPM endorsement key hash, admit it on spore, and make sure its agent serves SVIDs
• Operate Postgres: connect to, inspect and restart the CloudNativePG databases
• Restore a database: restore a Postgres database from its barman backups, and run the restore drill
• Recover a re-created database: bring a database back when its Cluster is created again under the same name
• Restore a volume: restore the files of a volume from a Velero backup
• Restore from the GCS copy: copy a Garage bucket back from the encrypted GCS copy
• Restore a site: copy a kthx quick site's files back from the noncurrent versions the sites bucket keeps
• Migrate Garage to three nodes: move folly's backup store from spore alone to a replication factor 3 cluster on the anchors
• Migrate a local-path volume: move a hostPath volume to a local volume so that Velero can back it up

Cloud and network#

• Apply an OpenTofu change: change a root, check it, and apply it through Atlantis
• Release a stale state lock: confirm a state lock is stale, then release it and plan the root again
• Authorize Developer Connect: authorize the GitHub connection of the trusted-builds project
• Inspect the UniFi network: read the live state of a UniFi console and compare it with git

Secrets#

• Manage SOPS secrets: restore the operator age key, edit a SOPS file, add a host, and rotate the key
• Switch a NetBox token to its own user: mint a token for the terraform and fleetd users, move each onto it, and revoke the admin tokens
• Initialize OpenBao: initialize a new OpenBao instance and store its recovery key
• Make a YubiKey a DevID CA: generate a DevID CA key on an anchor's YubiKey, certify it with Atlantis, and write the certificate into the card

Apps#

• Install kthx: install the kthx engine on the offsite cluster
• Connect an agent to kthx: mint an agent token and connect an MCP client, Rowbutt or the workstation
• Repair the Rowbutt GitHub credential: check, rotate and replace the GitHub App credential of Rowbutt
• Rotate the Rowbutt Slack token: replace Rowbutt's Slack App-Level Token, and revoke tokens that other clients hold
• Sign Rowbutt in to ChatGPT: sign mate in to the owner's ChatGPT subscription from chat, and sign it out
• Operate the Rowbutt model fallback: check which model answers Rowbutt, keep turns off ChatGPT, and drill the fallback
• Evaluate the Rowbutt models: read what the recorded turns say about each model, and replay the recorded tool decisions against a candidate
• Operate the Smiirl counter: check, calibrate and repair the Smiirl counter
• Operate the Flame Boss exporter: check a cook, act on its alerts, and replace the Flame Boss token
• Repair a kiosk: restart or start a Weather Hub kiosk
• Connect an agent to the wiki: add the wiki's MCP endpoint to an MCP client
• Operate the office phone: check, change and debug the office phone and the folly PBX
• Screen callers on the office phone: add a contact who skips the press-5 screen, use the star codes, and render the prompts
• Operate the offsite PBX: check the offsite PBX that hands calls to the troll agent, and find why a call failed

Every change#

• Test a change: run the local check for each kind of change before you open a pull request