Operate the office phone

The office Cisco SPA504G has four lines. Each registers to an Asterisk PBX on folly, and the PBX carries each line to its own voip.ms sub-account over TLS. The phone talks only to the PBX and holds no voip.ms credential.

How it fits together#

• Handset → PBX. Each line registers as lineN to the PBX's LoadBalancer address. The PBX identifies a line by that username and admits it only from the handset's reserved address, so the lines carry no password. Both addresses are CATHY_IP and PBX_SIP_VIP in clusters/folly/config/cluster-settings.yaml.
• PBX → voip.ms. One registration per sub-account to a numbered server over TLS on 5061, with the certificate verified and SRTP mandatory. Every call is G.722 end to end. The passwords live in 1Password item voip.ms sub accounts in the homelab vault, one concealed field per sub-account, labelled with the username.
• Everything declared in git. Asterisk config is under clusters/folly/apps/pbx/config/ and clusters/base/apps/pbx/config/; the handset's profile is clusters/folly/apps/pbx/provision/cathy.xml; the image is nix/images/asterisk.nix, built by .github/workflows/nix-images.yml.

Every line depends on folly being up — dial tone, and 911. A folly outage leaves the desk phone with no lines at all; a cell phone is the emergency backup.

A merge under clusters/folly/apps/pbx/ or clusters/base/apps/pbx/ rolls the pod, and Asterisk hangs up every call within 15 seconds of the stop. Merge only when core show channels shows no active call, and not within an hour of a 911 call from the handset: a roll forgets GLOBAL(LAST911), and a callback to a screened line meets the screen. Line 4 screens unknown callers; Screen callers on the office phone covers the screen, the contacts and the star codes.

Check its health#

• Grafana → PBX. Trunk and line registration, calls, and four log panels: call flow, errors, SIP messages, and the handset's profile fetches.

• The Switchboard board at https://switchboard.lolwtf.ca. Each call in progress with its IVR stage, each line's handset and trunk registration, and the calls that ended since the board started. See Switchboard board.

• Alerts in clusters/folly/monitoring/pbx-rules.yaml go to Discord: PBXDown, PBXTrunkNotRegistered per sub-account, and PBXHandsetOffline per line.

• The Asterisk CLI. Pass -C, or the CLI looks for its control socket in the wrong directory:

  kubectl --context folly -n pbx exec deploy/pbx -c asterisk -- \
    /bin/asterisk -C /etc/asterisk/asterisk.conf -rx 'pjsip show registrations'

  pjsip show contacts, pjsip show endpoints and — during a call — pjsip show channelstats (per-leg packet counts, loss and jitter) are the others worth knowing. The image has no grep; filter outside the exec.

Change the phone#

Edit clusters/folly/apps/pbx/provision/cathy.xml and merge. The profile is partial: the phone changes only the settings the file names and leaves everything else alone. Never change a setting by hand on the phone — its next resync puts the file's value back.

The phone resyncs hourly. To apply a change now, have it fetch the profile from a host that can reach the Management network — the UDM, through Inspect the UniFi network:

curl "http://<CATHY_IP>/admin/resync?http://<PBX_SIP_VIP>/cathy.xml"

The fetch shows up in the provision container's log and on the dashboard. http://<CATHY_IP>/admin/spacfg.xml returns the phone's whole running config. It includes every setting the phone has, so read it through a filter rather than printing it.

Rotate the board's ARI password#

The Switchboard board reads the PBX as the ARI user in clusters/folly/apps/pbx/config/ari.conf. The password is the SOPS Secret pbx-ari, in clusters/folly/apps/pbx/ari-secret.sops.yaml. The board restarts when that Secret changes. The PBX reads it only when its pod starts, because Reloader watches only pbx-secrets on the PBX. Until the PBX restarts, the board shows ARI disconnected (unauthorized), and a restart of the PBX is the fix for that state whatever caused it.

1. From the repo root, write a new password straight into the encrypted file. Nothing prints it:

   SOPS_AGE_KEY_FILE=~/.config/age/keys.txt bash -c 'set -euo pipefail
   f=clusters/folly/apps/pbx/ari-secret.sops.yaml
   pw=$(head -c 24 /dev/urandom | od -An -vtx1 | tr -d " \n")
   printf "apiVersion: v1\nkind: Secret\nmetadata:\n  name: pbx-ari\n  namespace: pbx\ntype: Opaque\nstringData:\n  PBX_ARI_PASSWORD: \"%s\"\n" "$pw" |
     sops encrypt --filename-override "$f" /dev/stdin >"$f"'

2. Merge, and wait for Flux to apply it.

3. When core show channels shows no active call, restart the PBX:

   kubectl --context folly -n pbx rollout restart deploy/pbx

Read the SIP#

PJSIP logs every SIP message the PBX sends and receives. In VictoriaLogs:

{namespace="pbx", container="asterisk"}

Vector removes the SRTP key from every a=crypto line and every Authorization header before storing them (clusters/base/monitoring/vector.yaml). The pod's own stdout is not redacted — read kubectl logs output through a filter that drops those lines.

Diagnose a failed call#

Techniques that settle it#

• Start captures before the test call. Hubble's per-node buffer on folly covers under a minute. Run hubble observe --namespace pbx --follow in the node's cilium-agent, then place the call.
• Isolate one variable at a time with a throwaway Asterisk. A local Asterisk with no registration can send the same INVITE with one thing changed — transport, account, caller ID — and read voip.ms's verdict. That is how the fraud hold above is told apart from a config fault.
• Prove TLS without credentials. Qualify-only OPTIONS to the server exercise the handshake and certificate check and carry no auth. Never trial a REGISTER with an unproven password: failed authentication can get the shared public address blocked, and that address carries every line.
• Validate what the cluster runs, not what you wrote. flux build piped into kubectl apply --server-side --dry-run=server catches both Flux's variable substitution and schema typing, which kubectl kustomize never sees. mise run pbx:check boots the image's Asterisk against each site's rendered config with no route off the machine. It fails when a PJSIP object does not load, when 911 stops reaching a line's trunk, when an inbound call can reach a trunk, when a trunk identifies by anything but header or a folly trunk stops taking voip.ms's DID-form call, when a prompt is missing from pbx-sounds, or when the open, contact, 911-callback, press-5 or spam route changes. On offsite, it fails when a caller stops reaching the agent with its headers, when a miss or a cap answers the caller, or when the held count changes. .github/workflows/pbx.yml runs it on every PBX change.