Kubernetes GitOps Change

#runbook #kubernetes #gitops

  • Use this when changing manifests under clusters/ or inspecting Flux deployment state. Architecture lives in Architecture/Kubernetes; the apply model is on Architecture/GitOps.

  • Rule

    • Author desired state in git and let Flux reconcile it after merge. Do not use kubectl apply to author state.

    • Use explicit contexts:

    • kubectl --context folly get nodes
kubectl --context offsite get nodes

  • Inspect reconciliation

    • flux --context folly get kustomizations -A
flux --context folly get helmreleases -A
flux --context offsite get kustomizations -A
flux --context offsite get helmreleases -A

    • For a specific object:

    • kubectl --context <cluster> -n <namespace> describe <kind> <name>
kubectl --context <cluster> -n <namespace> get events --sort-by=.lastTimestamp

  • Force a reconcile

    • Use this for inspection or to speed up a merged change:

    • flux --context <cluster> reconcile kustomization <name> -n flux-system --with-source

    • If a HelmRelease is stuck after its source reconciles:

    • flux --context <cluster> reconcile helmrelease <name> -n <namespace>

  • SOPS secrets

    • SOPS-encrypted files match clusters/**/*.sops.yaml; only encrypted data and stringData belong there.

    • Edit with SOPS:

    • sops clusters/<cluster>/<path>/<secret>.sops.yaml

    • Encrypt a new matching file:

    • sops -e -i clusters/<cluster>/<path>/<secret>.sops.yaml

    • Never paste decrypted values into this wiki, issues, PR comments, or logs.

  • Managed Postgres

    • Databases are CloudNativePG Cluster objects and have their own tooling — reach one with kubectl cnpg psql, not kubectl exec against an instance pod. See Runbooks/Managed Postgres.

  • HelmRelease source pattern

    • Keep HelmRepository, GitRepository, or OCIRepository sources colocated with the resource that consumes them.

    • Do not centralize sources unless the local pattern changes across the repo.

  • Atlantis and ArgoCD auth

    • Atlantis runs in offsite and may need to authenticate to ArgoCD in folly for Terraform/Argo checks.

    • Symptom of an expired or rotated ArgoCD token:

      • GitHub PR status for Atlantis plan fails.

      • Atlantis logs show authentication or signature errors talking to ArgoCD.

    • Rotation shape:

      • Generate a fresh token for the Atlantis ArgoCD account using an authenticated ArgoCD admin path.

      • Store it only in the SOPS-encrypted Atlantis secret.

      • Reconcile or wait for Flux to deploy the updated secret.

      • Re-run the Atlantis plan.

    • Do not record generated tokens or admin credentials in plaintext.

  • Validate before PR

    • Build the kustomization root that includes the change:

    • kubectl kustomize clusters/<cluster>/<category>

    • For shared clusters/base/ changes, validate both clusters. See Runbooks/Add Shared Kubernetes Resource.

Linked references 9

Runbooks/Adopt Folly Prometheus Operator CRDs

Use this to wire folly onto the monitoring-crds Kustomization the same way Runbooks/Kubernetes GitOps Change's GitOps rule normally requires — except the first step is a live, by-hand mutation. That is deliberate: see "The one sanctioned exception" below before running anything here.

Runbooks/Validate Infra Changes

See Runbooks/Kubernetes GitOps Change and Runbooks/Add Shared Kubernetes Resource.

Runbooks/Add Shared Kubernetes Resource

Use this when adding or changing resources shared by both clusters through clusters/base/. General GitOps flow is in Runbooks/Kubernetes GitOps Change.

Architecture/GitOps

Step-by-step: Runbooks/Kubernetes GitOps Change. Cluster internals: Architecture/Kubernetes.

Runbooks/Install Spindrift

Author it like any cluster secret: clusters/**/*.sops.yaml matches the first creation rule in .sops.yaml and encrypts data/stringData to the operator key. The mechanics are on Runbooks/Kubernetes GitOps Change; key handling is on Runbooks/SOPS Secrets and Age Keys. Generate both values fresh — never reuse another installation's. A keyring key is exactly 32 bytes, base64url-encoded: bun -e &quot;console.log(require('crypto').randomBytes(32).toString('base64url'))&quot; mints one (openssl rand -base64 emits +/= characters the keyring refuses).

Runbooks/Install Spindrift

Merge the PR. Flux applies clusters/ on main; forcing a sync and chasing a stale revision are covered by Runbooks/Kubernetes GitOps Change.

Runbooks/Managed Postgres

Every Postgres in the fleet is a CloudNativePG Cluster. Use the operator's own kubectl cnpg plugin rather than reconstructing what it does out of kubectl exec. The GitOps rules on Runbooks/Kubernetes GitOps Change apply here too: desired state is authored in git, and everything below is inspection.

Runbooks/Managed Postgres

Changing instance count, storage, or Postgres version is a manifest change that ships through git — see Runbooks/Kubernetes GitOps Change.

Runbooks

Runbooks/Kubernetes GitOps Change — inspect Flux, reconcile resources, and handle SOPS safely

Edit this page on GitHub