Identity

SPIRE gives folly nodes and their pods identities in the trust domain lolwtf.ca. Its server runs on spore.

A node's agent proves that it holds a trusted TPM. SPIRE accepts either a device certificate or an approved TPM endorsement key. The agent then gives credentials to authorized pods.

Terms#

Read Node lifecycle first for the installation sequence. The technical design explains the trust limits.

Server#

The server has no k8s_psat node attestor, so only a TPM can attest a node.

The hash is the SHA-256 of the EK public key in DER form, in lowercase hex. The fingerprint is the SHA-1 of the LDevID certificate. The attest service sets the LDevID's subject to CN=<name>, serialNumber=<serial>, OU=<class>, O=<site>: the node's name, its SMBIOS serial, NetBox's class and the site. Every JWT-SVID carries the issuer https://spire.lolwtf.ca, set by homelab.spire.jwtIssuer.

SPIRE 1.15.5's tpm_devid gives the selectors in the table and no others: no selector names the subject's OU or O. So an alias cannot select a class or a site, and each machine gets its own alias by subject CN.

• Each node alias, by LDevID CN or EK hash, is a fleet-* entry for a node fleetd lists in spire-nodes.json, so a node gets it with no deploy and loses it when fleetd withdraws it. spore declares none.
• Change the EK allow list in ekHashes and the workload entries in workloads of nix/hosts/spore.nix. The allow list admits an agent; its alias by EK hash comes from fleetd's file, as the LDevID alias does. spore declares no entry, and the sync deletes any entry whose ID starts with git- and leaves entries made by hand.
• An agent matches a node entry only when it has every selector, so a node has one alias for each attestor: by EK hash, and by LDevID CN. A node with no entry in ekHashes attests only by LDevID.
• Add a node's EK hash before its agent starts. The plugin refuses an EK that is not on the list.
• homelab.spire.devid.caChains takes one file for each issuing DevID CA, folly-devid-ca-<name>.pem in terraform/pki/certs/. The server reads the first certificate of each file, so list the CA files and not the bundle folly-devid-ca.pem. The fml intermediate or root in that pool would admit any certificate it signs.
• Run the CLI as root with -socketPath /run/spire/server/private/api.sock. Its default socket path is wrong for this server.
• The firewall admits the agent port and the exporter only from folly's K8S_NODE_CIDR and CILIUM_POD_CIDR. Cilium does not masquerade traffic to spore, so pods arrive from their own addresses.

fleetd entries#

Two owners write registration entries, and the entry ID says which.

Registration and withdrawal#

fleetd writes spire-nodes.json from NetBox. It has no SPIRE credential. The applier runs as root through the server's socket.

The applier accepts these values:

• Node names that match [a-z0-9-].
• EK hashes with 64 lowercase hexadecimal characters.
• LDevID fingerprints with 40 lowercase hexadecimal characters.

A malformed file changes nothing. The same applies if a node or EK hash appears as both registered and revoked.

A revoked record includes the node name and EK hashes. It also includes the stored LDevID's SHA-1 fingerprint until the certificate's notAfter time.

The applier matches revoked agents by agent ID, EK hash or tpm_devid:subject:cn:<name>. The CN match covers renewed LDevIDs too. An EK hash held by a registered node is never revoked. A TPM can therefore move to a new node name.

Bans and readmission#

A banned agent cannot attest with the same agent ID. An unexpired LDevID or an allowed EK does not override the ban.

SPIRE has no unban operation. When the node registers again, the applier removes its banned agent record if nothing still revokes it. The agent can then attest again.

A new LDevID produces a new agent ID. An EK agent retains its hash-based ID, so it needs the removal step.

The applier also removes manual bans on registered nodes. To keep a node out, set its NetBox device to failed.

Credential expiry#

SPIRE checks no LDevID revocation beyond the ban. Setting decommissioning does not immediately end access:

1. fleetd updates its export on the next pass.
2. The path unit starts the applier. A 10-minute timer retries if the path unit misses the update.
3. The applier bans the agent. Existing workload SVIDs remain valid until their TTL expires, normally one hour.

BanAgent needs an attested-node record. If an agent first attests after an applier run, the next run bans it.

Protection against bulk withdrawal#

One run can withdraw or ban agents for at most homelab.spire.fleetEntries.maxWithdrawals nodes. The default is one.

A node counts toward the limit if:

• It has a fleet-* alias but is absent from nodes.
• The run bans one of its agents. This includes revoked nodes and listed nodes that lose an EK alias.

A bulk edit that clears ek_hash pins can exceed the limit. The applier then changes nothing and exits with an error. SporeSpireFleetEntriesFailed fires after 15 minutes.

The applier refuses further runs until the export is below the limit or the owner authorizes the withdrawal. The owner records authorized node names in /var/lib/spire-fleet-entries/withdraw on spore, as root, one name per line. The next run acts on those names and removes the file.

The applier bans agents before deleting entries. If it fails between these operations, the next run plans the bans again.

Source: nix/services/spire/fleet-entries.sh. mise run spire:test tests it against a stub server.

Duplicate registrations#

SPIRE rejects entries with identical parents, SPIFFE IDs and selectors.

The fleet applier skips an entry already supplied by a git-* or manual entry. The Nix sync deletes a duplicate fleet-* entry when its declaration supplies the same identity.

The declared entry wins in either order. The node keeps its alias while registration ownership changes.

The chain#

A node's identity starts before Talos runs. Each step relies on the one before it.

Technical design separates these proofs from installation authority and lists the threats that remain. A claimed serial and signed LLDP report do not prove chassis identity or physical location.

Provision a node runs the first five steps.

Lifecycle#

A folly node's identity follows its device's status in NetBox. fleetd and SPIRE read it.

Only the owner moves a device out of failed or into decommissioning. fleetd refuses a failed device.

Quarantine#

Quarantine changes the attestation verdict, not the device status.

A quarantined worker gets no new identity until the owner clears the verdict. Its next hourly check or renewal must prove both keys and pass policy. fleetd then restores its record and LDevID files. The owner uncordons the node.

Use Provision a node, Reprovision a node or Retire a node for the procedures.

Agents#

The DaemonSet spire-agent in the spire namespace runs on each folly node labelled lolwtf.ca/tpm=true. Its image, ghcr.io/jonpulsifer/spire-agent-tpm, adds the agent half of the EK plugin and spire-devid-init to SPIRE's agent.

Agent initialization#

The init container devid-init reads the SMBIOS serial. It fetches <ATTEST_URL>/devid/<serial> from fleetd on spore. The spire-attest ConfigMap supplies ATTEST_URL.

It selects tpm_devid only if the LDevID:

• Names this node and serial.
• Is current.
• Matches its key blob.

Otherwise, or if ATTEST_URL is empty, it selects the EK attestor.

The tpm_devid agent stores state in /var/lib/spire-agent/tpm_devid. The TPM's storage root key wraps the blobs. The agent derives that key from the TCG H-1 or H-2 template. The blobs cannot load on another TPM.

The container is privileged, since the device cgroup refuses /dev/tpmrm0 to any other container. It shares the host's PID namespace and network, so the k8s workload attestor can ask the kubelet for a caller's pod. Its key and SVID stay in /var/lib/spire-agent on the node, so a new pod resumes with no second attestation.

Pod access#

A pod reaches the Workload API at /run/spire/sockets/agent.sock through a hostPath volume, which needs a privileged namespace. A node with no label runs no agent, so its pods get no SVID. The Job spire-smoke prints the SPIFFE ID that the agent serves it.

• An agent's first attestation trusts the spire-bundle ConfigMap. spore's notifier pushes the bundle into it, and Flux declares no data, so its server-side apply never overwrites the push. If the ConfigMap is empty or older than the server's CA, a new agent fails.
• The notifier reads folly's API server from cluster-topology.json, the CA from clusters/folly/config/kubernetes-ca.pem, and the token from spire/folly-bundle-writer-token in nix/secrets/spore.sops.yaml. If it cannot write the bundle at startup, the server exits and retries every 30 seconds. A push that fails at a rotation waits for the next rotation or restart.
• Reloader restarts the agents when spire-agent or spire-attest changes, not when the bundle changes. An attested agent follows the rotations itself.
• A workload gets an SVID only after spore registers its entry under its node's alias.
• A node pulls the agent image only while its GHCR package is public.

Add a TPM node to SPIRE admits a node by LDevID or EK hash, turns on tpm_devid, and turns on the bundle notifier.

Renewal#

The DaemonSet tpm-reattest in the spire namespace runs tink-attest renew -every 1h on each node labelled lolwtf.ca/tpm=true, as clusters/folly/apps/spire/tpm-reattest.yaml declares. It reads the LDevID and LAK certificates from NV 0x01c90000 and 0x01c90001. When the LDevID expires within 30 days, or NV holds the LAK and no LDevID, it renews the LDevID on fleetd's DevID listener, at the same ATTEST_URL as devid-init. fleetd issues the renewal from spore's YubiKey CA, whatever CA issued the LDevID on file. Otherwise it runs the hourly check, which the next section describes.

1. It fetches /devid/<serial> and checks that the key blobs are the keys the certificates in NV name.
2. POST /v1/renew with the serial answers a session and a nonce.
3. It loads the LAK and the LDevID key under the H-2 storage root key. With the LAK it quotes PCRs 0 to 7 and certifies the LDevID key, both qualified by the nonce.
4. POST /v1/renew/<session>/evidence carries the quote, the PCR values, the event log from securityfs, and the certification. The service replays the log as at intake and answers a new certificate for the same key.
5. It checks that the certificate chains to a DevID CA in the ConfigMap spire-devid-ca, certifies the same key, and names the node and its serial. The routes are plain HTTP, so this check is what keeps a forged answer out of NV.
6. It writes a renewed LAK certificate first, then the LDevID certificate, so a failed write leaves the renewal due for the next run. It then deletes the spire-agent pod on its node, and fails when no pod matches. The new pod's devid-init fetches the new certificate.

scripts/render-spire-devid-ca.sh writes spire-devid-ca from terraform/pki/certs/folly-devid-ca.pem, and the kustomize workflow fails when the two differ. A run that fails runs again an hour later. A node with no LDevID in NV has nothing to renew. The ServiceAccount tpm-reattest may list and delete pods in spire. The ValidatingAdmissionPolicy tpm-reattest allows it to delete only a spire-agent pod on the node that its token names. TalosAttestLDevIDExpiring fires when an LDevID expires within 23 days, which is a week of failed renewals.

Hourly check#

On a day with no renewal due, the run takes steps 1 to 4 with check set in the request and the BIOS version from /sys/class/dmi/id/bios_version in the evidence. The service judges the boot by the renewal policy, records the verdict, last_attested and PCR 0 in NetBox, and answers no certificate, so NV and the spire-agent pod stay as they are.

A boot that misses the policy after the LAK's proof quarantines a worker: its attest_verdict reads quarantined: <reason>, and fleetd serves it no LDevID and cordons it. A controlplane node stays in place and alerts at critical, so the cluster's API stays up. fleetd has the rules.

A run opens /dev/tpmrm0 and reads the event log before it sends anything. Without the TPM it logs NO TPM, sends nothing, and tries again an hour later; a node whose TPM is off has no device to mount, so its pod does not start. A service that predates the check answers without check, and the run sends it no evidence. TalosAttestNotChecked fires when fleetd has recorded no passing quote from a node for 3 hours.

GCP access#

A workload gets GCP access in four steps.

1. spore registers its SPIFFE ID.
2. A grant in terraform/gcp/projects/homelab-ng/ names the principal principal://iam.googleapis.com/<fml-pool>/subject/spiffe://lolwtf.ca/ns/<ns>/sa/<sa>.
3. The pod mounts the agent socket and runs spire-agent api fetch jwt with the audience //iam.googleapis.com/<fml-pool>/providers/spire.
4. The pod writes the SVID to a file, and an external_account credential config whose credential_source is that file. With no service_account_impersonation_url, GCP sees the SPIFFE ID itself.

<fml-pool> is projects/<project-number>/locations/global/workloadIdentityPools/fml-pool. A JWT-SVID is short-lived, so a long-running workload rewrites the file before it expires.

The test Job in spire-test runs as uploader, which can only create objects in the bucket homelab-ng-spire-test. Its log names the SPIFFE ID and the object it wrote. Bump the suffix of the Job's name to run it again.

• The provider spire admits every SPIFFE ID in the trust domain. Grant on one subject, never on the pool or on an attribute.spiffe_id prefix.
• The server's jwt_issuer, the discovery provider's jwt_issuer and the provider's issuer_uri are one value. GCP rejects an SVID whose iss differs.
• The discovery provider reads its keys through the Workload API as spire-oidc/oidc-discovery-provider. Without that entry, spire.lolwtf.ca serves no keys and every exchange fails.
• iam.workloadIdentityPoolProviders in terraform/gcp/projects/homelab-ng/policy.tf lists the issuer. A new issuer needs an entry there first.

Live state#

riptide, surge and swell carry the lolwtf.ca/tpm=true label. riptide's and surge's agents attest by their LDevIDs, and fleetd lists both nodes. swell is down. optiplex has no SPIRE identity: its node has no /dev/tpmrm0, so tpm_off in clusters/folly/talos/nodes.tf keeps the label off it, its EK hash is off the allow list, and fleetd does not list it. The allow list holds riptide's EK hash. shale has a TPM 1.2 and runs no agent. The smoke Job is suspended.

Where it lives#

• nix/services/spire-server.nix: the server, both node attestors, the bundle notifier, the entry sync, the agent count and the firewall
• nix/services/spire/roots/: the Infineon and Nuvoton EK CAs that tpm_devid trusts, the same files as apps/fleetd/attest/roots/
• apps/fleetd/ and nix/services/fleetd.nix: fleetd and its deployment on spore
• nix/hosts/spore.nix: the EK allow list ekHashes and the workloads
• nix/lib/spire.nix: the SPIRE and plugin builds, with nix/lib/spire-tpm-plugin-sdk.patch
• clusters/folly/apps/: the agent, its bundle, the bundle writer, tpm-reattest and the smoke Job in spire/, the discovery provider in spire-oidc/, and the GCP test in spire-test/
• images/: the agent image in spire-agent-tpm/, with spire-devid-init in spire-agent-tpm/devid-init/
• terraform/gcp/projects/homelab-ng/workload-identity.tf: the provider spire in fml-pool, with the test bucket and its grant in spire-test.tf beside it
• terraform/network/cloudflare/lolwtf.ca.tf: the tunnel rule for spire.lolwtf.ca
• clusters/folly/monitoring/spore.yaml: the scrape and the SPIRE alerts
• clusters/folly/config/cluster-topology.json: K8S_NODE_CIDR, CILIUM_POD_CIDR, API_SERVER_HOSTNAME, API_SERVER_PORT

Related#

• PKI: the cluster ServiceAccount issuers in the same pool
• Cloud
• spore