Install kthx

Use this runbook to install the kthx engine on the offsite cluster, or to rebuild it. The engine is the web and reconciler processes of apps/kthx-engine/ and their database. kthx lists the live addresses.

Before you start#

• Get Atlantis access (Apply an OpenTofu change), the operator age key (Manage SOPS secrets), and offsite access (Get cluster admin access).

• The console hostname, <hostname>, is kthx's control.host in clusters/offsite/apps/kthx/helm-release.yaml. The first passkey is registered to it, so browsers must reach it.

• <public-hostname> is the KTHX_ENGINE_PUBLIC_HOSTNAME value in helm-release.yaml. The Cloudflare tunnel forwards the GitHub webhook and /mcp to it.

• For another namespace, change spindrift in namespace.yaml and on each line that this command prints:

  git grep -n -e 'spindrift:spindrift' -e 'namespace: spindrift$' -e 'spindrift\.spindrift\.svc'

  Then change the spindrift subject in the KubeAuthenticationConfig of clusters/folly/talos/patches.tf and clusters/offsite/talos/patches.tf, which Atlantis applies.

Declare the installation#

clusters/offsite/apps/kthx-engine/ is the installation, and packages/charts/kthx-engine/values.yaml describes its values. A Target is where kthx deploys built apps.

1. In helm-release.yaml, set hostname to <hostname> and ingress.enabled to false.

   kthx's kthx-control route is the console's edge. control.console in clusters/offsite/apps/kthx/helm-release.yaml names this Service, and kthx-route-grant.yaml admits the route.

2. Make sure each Kubernetes Target cluster declares an Apps Gateway and binds the Target ClusterRoles.

   offsite declares both in gateway.yaml and target-rbac.yaml, and folly in clusters/folly/apps/kthx-engine-target/.

Write the installation Secret#

secret.sops.yaml is the Secret kthx-engine-env. The engine reads each key below under its KTHX_ENGINE_ name only, and secret.sops.yaml keys each value that way.

1. Find out if the database exists.

   kubectl --context offsite -n spindrift get cluster.postgresql.cnpg.io spindrift-db

   Result: NotFound for a new installation.

2. If the database does not exist, make a keyring.

   bun -e "const k=require('crypto').randomBytes(32).toString('base64url');console.log(JSON.stringify({active:'k1',keys:{k1:k}}))"

   If it exists, keep the keyring. To rotate it, add a new active key and keep the others.

   Result: A JSON line with active and keys.

3. Make an enrolment token.

   bun -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"

   Result: A 43-character base64url string.

4. Open the Secret in your editor.

   SOPS_AGE_KEY_FILE=~/.config/age/keys.txt sops clusters/offsite/apps/kthx-engine/secret.sops.yaml

5. Set each key under stringData. Save the file and close the editor.

   Result: sops encrypts the file again.

6. Open a pull request with these changes.

Apply the OpenTofu roots#

The roots terraform/gcp/projects/bluenose/ and terraform/gcp/projects/trusted-builds/ need resources from each other. Apply them in three phases.

1. Comment atlantis plan -d terraform/gcp/projects/trusted-builds on the pull request.

   Result: Atlantis comments the plan.

2. Comment atlantis apply -d terraform/gcp/projects/trusted-builds.

   Result: The controller grants fail. The apply creates the attestor and imports the KMS keys.

3. Do steps 1 and 2 for terraform/gcp/projects/bluenose.

   Result: The apply creates kthx-engine-controller.

4. Do steps 1 and 2 for terraform/gcp/projects/trusted-builds again.

   Result: Atlantis merges the pull request.

Make sure the installation runs#

1. Make sure the installation is ready.

   flux --context offsite get sources oci -n spindrift
   flux --context offsite get helmreleases -n spindrift
   kubectl --context offsite -n spindrift get cluster.postgresql.cnpg.io spindrift-db
   kubectl --context offsite -n spindrift get deploy,job

   Result: READY is True, the Cluster is healthy, the spindrift-migrate-* Job is Complete, and then both Deployments are 1/1.

Enrol the first passkey#

1. Open https://<hostname>.

   Result: "Claim this installation", or "Sign in" if the database existed.

2. If the page shows "Sign in", sign in with your passkey. Skip steps 3 to 8.

3. Enter the value of KTHX_ENGINE_ENROLMENT_TOKEN.

4. Select "Enrol a passkey".

5. Register the passkey when the browser asks.

   Result: The onboarding wizard opens.

6. If you have a file from "Download this installation" in Settings, select "Restore from a file".

7. Complete the wizard.

   For "Where artifacts are published", use registry from the supply_chain_manifest_block output in the last trusted-builds apply comment.

8. Select "Configure this installation".

   Result: The page shows "This installation is configured."

Connect GitHub and Targets#

1. If Repositories shows "Create the App on GitHub", select it. Create the App on GitHub.

2. If you adopt an App, select "Active" under Webhook in its GitHub settings.

   Use https://<public-hostname>/internal/github/webhook and KTHX_ENGINE_GITHUB_WEBHOOK_SECRET.

3. If a repository belongs to another account, select "Make public" in the App's Advanced settings.
4. Select "Install on GitHub". Connect each repository in Repositories and each Target in Targets.

If something goes wrong#

Related#

• Built apps
• Ownership and security
• Operate Postgres
• Connect an agent to kthx