Secrets
Where secrets are stored, what decrypts them, and how they reach hosts and workloads.
Hosts and workloads get secrets from SOPS files in git, the homelab 1Password vault and GCP Secret Manager.
Parts#
| Part | Job | Where it runs |
|---|---|---|
| SOPS | Encrypts the values in *.sops.yaml files | Your machine |
| Flux | Decrypts clusters/**/*.sops.yaml when it applies them | Each cluster |
| sops-nix | Decrypts nix/secrets/<host>.sops.yaml at activation | NixOS hosts |
| 1Password | Holds long-lived secrets and recovery copies | The homelab vault |
| External Secrets Operator (ESO) | Copies values from a store into Kubernetes Secrets | Each cluster |
| OpenBao | A secrets server with no consumers | folly |
Keys and recipients#
SOPS encrypts with age, a file-encryption tool. An age recipient is a public key that SOPS encrypts a file to, and the matching private key decrypts the file.
- Every SOPS file lists the operator key, the owner's age key. Its private half is in
~/.config/age/keys.txtand the 1Password itemsops homelab age key. - Flux reads the operator key from the Secret
sops-ageinflux-system. Git does not declare that Secret. - A host recipient is the age key that
ssh-to-agederives from the SSH host key.nix/system/sops.nixmakes sops-nix decrypt with that key, so a host reads only the files encrypted to it. nix/secrets/bosun.sops.yamlhas oldschool, a Talos node, as its host recipient. No host configuration reads it.- In
clusters/, SOPS encrypts onlydataandstringData.
Stores#
Both clusters have three ESO ClusterSecretStores.
| Store | Reads from | Authenticates with |
|---|---|---|
onepassword-connect | The homelab vault, through 1Password Connect | A Connect token from a SOPS file |
gcp-secret-manager | kthx App config in Secret Manager in bluenose | Workload identity |
kthx-datastores | The credentials of kthx Datastores | A ServiceAccount that reads Secrets in kthx-datastores |
OpenBao#
Git declares no auth method, mount, policy or client for OpenBao. Flux deploys the HelmRelease vault from clusters/folly/apps/vault/. The release runs one pod, vault-openbao-0, with Raft storage and a UI at https://vault.lolwtf.ca/ui/.
A sealed OpenBao cannot read its storage. The pod unseals itself with the GCP KMS key openbao through workload identity. If oidc.lolwtf.ca or the folly token signer fails, a restarted pod stays sealed.
Velero backs up the volume data-vault-openbao-0 each night, because the pod has the annotation backup.velero.io/backup-volumes. The copy is best effort: it reads the Raft files while the server writes them, so a backup can be inconsistent.
Rules#
- Keep a recovery copy of each host secret in a
homelabitem titled<host> <thing>. If the operator key and the host key are lost, SOPS cannot decrypt the file. - A workload that reads the
onepassword-connectstore uses the vault item as its live value. An edit reaches the Secret within an hour. - Declare no secret from a new host file before the file lists the host recipient, or the deploy fails.
- Replace the operator key in
.sops.yaml, every SOPS file, bothsops-ageSecrets and 1Password, or Flux cannot decrypt.
Where it lives#
.sops.yaml: the recipients for each pathnix/secrets/: the host SOPS filesclusters/base/platform/: ESO, 1Password Connect and the storesterraform/gcp/projects/homelab-ng/kms.tf: the OpenBao unseal key
Related#
Linked from 11
- Built appsA kthx built app is a repository or archive that kthx builds into a signed artifact and deploys to Targets on the clusters, GCP, Vercel or Cloudflare.
- Cloud accountsThe Google Cloud projects, Cloudflare zones and Google Workspace domain that OpenTofu declares, and what each one is for.
- NetBoxThe record of every machine in the lab, on capsule, reachable only over the tailnet.
- NixOSThe flake that builds each Linux host and OS image, the fleet baseline each host shares, and the daily upgrade from main.
- PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
- WorkloadsWhat runs on the folly and offsite Kubernetes clusters, the files that list it, and the workloads that have their own page.
- GlossaryThe lab-specific terms that the wiki pages use, what each means, and the page that defines it.
- Initialize OpenBaoInitialize a new OpenBao instance on folly, store its recovery key and root token, and make sure it unseals itself.
- Manage SOPS secretsRestore the operator age key, edit or add a SOPS file of encrypted secrets, give a NixOS host its own SOPS file, add a harmonia cache key, and rotate the operator key.
- Repair the Rowbutt GitHub credentialCheck, resync, rotate and replace the GitHub App credential that Rowbutt's sandboxes push with.
- Rotate the Rowbutt Slack tokenReplace the Slack App-Level Token that Rowbutt's Socket Mode connection uses, and revoke the tokens of clients that share its events.