infra_wiki

Issue a talosconfig

Mint the owner's os:admin talosconfig for a Talos cluster and Rowbutt's os:reader one, and store each in 1Password.

Use this runbook when a cluster moves to Talos, and before a talosconfig certificate expires. A talosconfig is the client certificate and endpoints that talosctl uses to reach the Talos API. Its certificate lasts one year.

1Password itemType and file nameHolds
talos-<site>-secretsSecure NoteThe cluster's secrets bundle, which Atlantis reads
talos-<site>-adminDocument, talosconfigThe owner's os:admin talosconfig, with one context named <site>
talos-rowbutt-readerDocument, talosconfigRowbutt's os:reader talosconfig, with a context for each Talos cluster

<site> is folly or offsite, and <checkout> is a checkout of this repository.

Before you start#

  • You need talosctl, jq, yq and op, signed in to the homelab vault.
  • The item talos-<site>-secrets exists.

Issue the admin talosconfig#

warning

The Talos API cannot revoke a client certificate. Keep the files out of the repository, and delete them after the upload.

  1. Go to a new directory under /dev/shm.

  2. Mint the os:admin file from the cluster's secrets bundle.

    topology=<checkout>/clusters/<site>/config/cluster-topology.json
op read "op://homelab/talos-<site>-secrets/notesPlain" > secrets.yaml
talosctl gen config --with-secrets secrets.yaml --output-types talosconfig -o admin \
  <site> "https://$(jq -r .data.API_SERVER_HOSTNAME "$topology"):6443"

    Result: Created admin.

  3. Set its endpoint and node to API_SERVER_IP.

    ip=$(jq -r .data.API_SERVER_IP "$topology")
talosctl --talosconfig admin --context <site> config endpoint "$ip"
talosctl --talosconfig admin --context <site> config node "$ip"

  4. If the item talos-<site>-admin exists, delete it.

  5. Store the file as the Document talos-<site>-admin.

    op document create admin --title talos-<site>-admin --file-name talosconfig --vault homelab

  6. Delete the directory.

  7. Add the context to your workstation, as Get cluster admin access describes.

Issue the Rowbutt talosconfig#

  1. Go to a new directory under /dev/shm.

  2. For each cluster that runs Talos, mint an os:reader file with your admin context, and merge it into reader. config new names its context reader@<site>, so rename it to <site> before the merge.

    ip=$(jq -r .data.API_SERVER_IP <checkout>/clusters/<site>/config/cluster-topology.json)
talosctl --context <site> config new --roles os:reader --crt-ttl 8760h reader-<site>
yq -i '.context = "<site>" | .contexts = {"<site>": .contexts["reader@<site>"]}' reader-<site>
talosctl --talosconfig reader config merge reader-<site>
talosctl --talosconfig reader --context <site> config endpoint "$ip"

  3. Make sure that reader has one context for each cluster that runs Talos.

    talosctl --talosconfig reader config contexts

    Result: One row for each such cluster, named folly or offsite.

  4. If the item talos-rowbutt-reader exists, delete it.

  5. Store reader as the Document talos-rowbutt-reader.

    op document create reader --title talos-rowbutt-reader --file-name talosconfig --vault homelab

  6. Delete the directory.

  7. Make sure that the ExternalSecret has synced. It refreshes every hour.

    kubectl --context offsite -n mate get externalsecret mate-sandbox-talos

    Result: STATUS is SecretSynced.

  8. Restart mate, which reads the file when it starts.

    kubectl --context offsite -n mate rollout restart deploy/mate

    Result: deployment.apps/mate restarted.

If something goes wrong#

SymptomCauseAction
mate logs the sandbox talosconfig could not be read, with grants os:admin.The Document holds a certificate with a role other than os:reader.Issue the Rowbutt talosconfig again. Then rotate the OS CA with talosctl rotate-ca --kubernetes=false, because the stored certificate cannot be revoked. A rotation of the Kubernetes CA too strands the workers fleetd installed, as PKI says.
mate logs no sandbox talosconfig yet.The Secret mate-sandbox-talos is absent.Make sure that the item exists and the ExternalSecret has synced.
config contexts shows <site>-1.The merge renamed a context that was already in the file.Start again in a new directory.

Edit on GitHub

Linked from 7
  • optiplexA Dell OptiPlex 3050 micro that is the folly cluster's only control-plane node, on Talos Linux.
  • retrofitAn HP EliteDesk 800 G2 mini PC that is the offsite cluster's only control-plane node, on Talos Linux.
  • PKIThe FML certificate chain, the Kubernetes cluster CAs and token signers, the cluster OIDC issuers, workload identity federation and the hardware DevID CAs.
  • Get cluster admin accessGet kubectl access to folly and offsite with the admin certificate from talosctl, and withdraw access.
  • RunbooksStep-by-step procedures for the hosts, clusters, network, secrets and apps, grouped by area.
  • Install Talos on a nodeBoot a running NixOS node into the Talos installer with kexec, check it in maintenance mode, and check its volumes after the install.
  • Make a Talos secrets bundleGenerate a cluster's Talos secrets bundle with its imported service-account key, store it in 1Password, and pin its new Kubernetes CA in the cutover pull request.

↑↓ move ↵ open esc close