How Rowbutt works

mate is the process behind Rowbutt. It runs the agent loop for each thread, runs the agent's tools in the thread's sandbox, and gives each turn short-lived credentials.

Parts#

The model is MATE_MODEL, on the owner's ChatGPT subscription through pi-ai's openai-codex provider, at the reasoning level in MATE_THINKING. When ChatGPT cannot answer, MATE_FALLBACK_MODEL answers through pi-ai's opencode-go provider, at MATE_FALLBACK_THINKING. Model routing says when. The system prompt is the persona in apps/mate/profile/persona.md, the profile's note about the surface and the sandbox, the owner's global apps/mate/profile/owner-agents.md, the profile's overrides of those rules, AGENTS.md, the memory in apps/mate/profile/memory.md, then an index of the skills in apps/mate/profile/skills/ and .agents/skills/.

The persona, the owner's file and the skills are copies of files in the owner's dotfiles repository. mise run mate:profile refreshes them from its main, and the mate-profile workflow fails when a copy has drifted. The investigator reads only the Protect and Communicate sections of the global file.

profiles.ts builds the overrides from the profile's grants that mate's config backs: the GitHub App, 1Password Connect and the sandbox ServiceAccounts. They claim no access that the sandbox does not hold. The owner's local pi reads the same persona through the mate pi package in dotfiles/pi/mate/, and the memory from an infra checkout when the machine has one.

The mate image bakes these files, and the agent reads a skill from the sandbox's checkout.

Profiles#

apps/mate/src/profiles.ts declares each profile's mode, system prompt note, model, tools, credentials, sandbox network and budget. Rowbutt lists them.

Each row in mate_threads names its profile in column profile. mate writes it when the row is born and never changes it; a row with no profile runs as operator. A message that starts with +<profile> asks for that profile: on a new thread it sets the row's, and in a known thread it must match.

Only Threads.start, the trusted trigger the custodian, the responder and the dream use, opens an automation or job thread. A row naming a profile this mate does not declare opens nothing, and its thread says so.

Interactive profiles share MATE_MAX_CONCURRENT turns and the in-memory window of MATE_MAX_TURNS_PER_DAY. Automation and job profiles share a second lane of one turn, and each counts its own daily cap in mate_profile_turns, by UTC day. An owner reply in an automation or job thread runs in that lane.

mate_turns_running, mate_queue_depth and mate_sandbox_waiters carry a lane label, and mate_turns_total carries profile and mode. In the sandbox cap, an interactive waiter goes ahead of an automation one, and an automation waiter never takes an interactive thread's sandbox. A job thread releases its sandbox with reason finished and archives itself when its turn ends.

Every sandbox carries lolwtf.ca/profile, and serves one profile: a thread condemns a sandbox labelled for another profile and mints a new one. A sandbox on the mate-sandbox-reader network also carries lolwtf.ca/minted-by=mate-reader, and its name ends in -r, so an image from before profiles never lists or reuses it. Spares are operator pods, and only operator adopts one.

When MATE_SESSION_RETENTION_DAYS is set above 0, retention deletes the closed rows of profiles whose grants equal operator's, as sweepable in profiles.ts lists them. A deleted row is born again as operator at the next mention in its thread, so mate keeps every investigator row.

Daily custodian#

apps/mate/src/custodian.ts checks the local date and hour in America/Halifax every minute. When it is 18:00 or later, mate_custodian_runs claims the calendar day, posts a Slack root in the configured channel, saves its timestamp and calls Threads.start with the custodian profile and the owner as the asker.

start writes the thread's row, posts the assignment as 📋 assignment so a replayed session still has it, and queues the turn. The row is the attempt: once it exists, start does nothing, so a refused or failed first turn is said once in the thread, and a reply runs it again.

An open job row that a restart finds with no turn run says so in its thread. The turn runs in the automation lane with operator's credentials, and the thread releases its sandbox when the report is done. The daily assignment can fix clear issues through branches and PRs, and merge understood PRs after required checks and reviews pass.

It follows AGENTS.md for GitOps and escalates risky or unclear changes. A failed database or Slack call before the row exists retries while the same Atlantic day remains. A crash between Slack's post and the timestamp write may leave a duplicate root; investigate a missing or duplicate report in mate's logs and Slack.

Alert responder#

apps/mate/src/responder.ts answers POST /alertmanager on mate's port, behind the bearer in the file MATE_ALERTMANAGER_TOKEN_FILE names, which mate reads at each request. The Secret mate-alertmanager mounts it, and Alertmanager's alertmanager-mate reads the same 1Password item, mate alertmanager token, so a Secret that appears or rotates after mate started needs no restart.

The route exists whenever MATE_RESPONDER_CHANNEL is set: it answers 503 while mate starts, while the token file is missing, or when Slack or the database is unavailable, and Alertmanager retries a 503 for about its groupInterval, as it does a 500 from a store or Slack failure and a 503 from a draining mate.

A request that is not a POST, carries another bearer or is not JSON is refused with a 4xx, which Alertmanager counts as a failed notification and does not retry. A JSON body that is not the version 4 payload mate reads is answered 200 and logged as an error, because Discord still carries the alert and a payload drift should not page.

mate_alert_groups holds one row per Alertmanager group key. A firing notification claims the row, replacing one that resolved, posts the root in MATE_RESPONDER_CHANNEL if the row has no thread_ts yet, saves the timestamp and calls Threads.start with the responder profile and the owner as the asker.

start is idempotent, so a start that failed after the root was saved runs at Alertmanager's retry. When the thread exists, the row's fingerprints say which alerts it knows: a notification with new ones updates the row and calls Threads.wake with their details, and one without them does nothing.

A resolved notification marks the row and posts one line, and a group mate never saw is ignored. The prompt lists at most 12 alerts and clips each label list and annotation at 600 characters, and the webhook takes at most 20 alerts a notification.

Dreams#

apps/mate/src/dreams.ts checks the local date and hour in America/Halifax every minute, as the custodian does. When it is 03:00 or later, mate_dreams claims the calendar day with the window it reads: from the previous row's until to now, or the last 24 hours when there is no previous row, and at most 7 days back.

It posts a Slack root in MATE_DREAMS_CHANNEL, saves its timestamp and calls Threads.start with the dreamer profile and the owner as the asker. A retry on the same day reads the same window, and a clock that stepped back still gets a window of at least a millisecond.

The row is the attempt, as the custodian's is: a dream whose turn fails is said once in its thread, a reply there runs the assignment again over the same window, and the next night's window starts where this one's ended, so a failed night is not read again on its own.

The assignment names the window in both clocks and asks for the recall tools, the three kinds of lesson, the edit to apps/mate/profile/memory.md, one pull request from dreams/<day> and the public-repository screen; dreamPrompt in dreams.ts is its text.

The dreamer profile holds a GitHub token and no other grant, lists the base tools and the recall tools, runs on the mate-sandbox network because the reader network has no GitHub egress, and adopts no spare because its pod differs from operator's. Retention keeps its rows, as its grants differ from operator's.

That network reaches the internet on ports 80 and 443, so a transcript the dreamer reads is untrusted text in a sandbox that can push and post; the review of its pull request is the control, and memory.md is read into every profile's prompt once merged.

apps/mate/src/recall.ts backs the recall_threads and recall_thread tools, which run in mate's process like wake. recall_threads reads mate_threads for the rows whose updated_at falls in the window, newest first, at most 50, leaves out the dreamer profile's rows, and takes each thread's opening ask from the first user message in the session's first eight pi_entries records, parsed in mate rather than cast in SQL so one record with a \u0000 escape fails no listing.

recall_thread reads every pi_entries record of the row's session, every conversation included, in order and renders the model messages: the user's text as owner:, the assistant's text as you:, a tool call as → name args and its result as ← name: text. Each text is passed through redact and then clipped, so a clipped secret leaves no prefix, and its later lines are indented, so a message forges no line of the transcript.

The pages are 24,000 characters, split at line ends. Thinking blocks and system messages are left out. A window is at most 30 days.

Wakes#

The brain gives each thread whose profile lists wake its own copy of the tool, which records the running turn's asker. Wakes.schedule upserts the thread's row in mate_wakes, keyed by the thread, and posts the ⏰ line. Every minute, Wakes.tick lists the rows. A timed row fires once due_at passes.

A pull request row reads /pulls/<n> and the Actions runs on its head through GithubApp.read, which mints once more on a 401, since a turn's end revokes the token it shares. The row fires when the PR settles, and at due_at whatever its state. Firing deletes the row only if it is still the one listed, then calls Threads.wake, which posts the line, recalls the thread like a reply, and queues the line as its prompt.

A thread with no row, or a stranger's asker, drops the wake with a log line. A draining mate refuses the wake, and the tick writes the row back for the next process. A restart after the thread accepts the wake and before its turn starts loses the wake.

A PR with over 100 runs on its head, or a workflow that another workflow's end starts, can fire the wake before every run is done.

A Stop click, or a reply of stop, stops a running turn, then calls Wakes.cancel. A stopped turn's wake call is refused, so it cannot leave a new wake behind. A typed stop that finds neither says ✋ nothing to stop here.

Inbound messages#

mate reads only the allowlist's messages. A mention in an allowed channel opens a thread. A reply in a thread with a row in mate_threads continues it, whether the row is open, closed for quiet or finished, and across restarts. It runs under the row's profile with its stored session, so a reply to the custodian's report runs as custodian in the automation lane.

Slack retries an undelivered event for about six minutes. mate claims the event id of an allowlisted human's message or Stop in mate_slack_events before it answers, so a retry after a restart is answered once, and an event older than an hour is not. Other events skip the claim. While the store is down, or a claim takes more than three seconds, the cutoff is the process start.

mate keeps its gateway session in mate_gateway_session and closes the gateway keeping it, so a restarted mate resumes and Discord replays what arrived in between. A resume delivers no READY, so mate asks the API who it is before it joins the Discord surface and registers the guild commands, and the voice side asks the API where the owner is when the gateway has not said.

A session that Discord does not hold (an outage past the time Discord keeps one, or a store that is down), or one that identified with other intents than the build runs with, identifies fresh, so a message sent in that gap gets no answer and mate_inbound_dropped_total does not count it.

mate downloads no file on a message. The prompt ends with a line naming each file, with its type and size.

mate_inbound_dropped_total counts an allowlisted message that runs nothing and says nothing, by surface and reason: no-mention, channel, thread-create, stale or subtype. The log line an inbound message was ignored carries its ids, never its text.

Voice#

apps/mate-voice/ is the bridge that holds Rowbutt's Discord voice connection, and packages/mate-voice-protocol/ is the JSON protocol the two speak. mate stays the only gateway session: the bridge dials mate's /voice WebSocket on its port, asks mate to send each voice state update on the gateway, and mate relays the bot's own VOICE_STATE_UPDATE and the VOICE_SERVER_UPDATE back. Both sides heartbeat every five seconds, and three missed beats end the link. A second bridge that dials in replaces the first.

/join, /leave, and a mention that says only join or leave, are the voice commands. mate registers the two slash commands on its guild at every READY, and the guild's Discord settings decide who sees them. Every path checks MATE_ALLOWED_USER_IDS: another user is refused. mate joins whichever voice channel the owner is in; MATE_ALLOWED_CHANNEL_IDS gates the text channels, not the voice ones.

A slash command is answered where only its user sees it; a mention is answered in its channel, and that channel gets the call's status lines. mate holds one call. It leaves when the owner leaves the channel, when the channel has nobody else in it, when asked, and when the bridge stalls.

If the bridge's link drops mid-call mate sends the leave itself, and if mate restarts the bridge keeps the call until the next mate adopts it or thirty seconds pass.

While no mate-voice pod is linked, the commands answer that the bridge is not connected.

The bridge can hear and speak itself, when MATE_VOICE_ELEVENLABS_VOICE_ID names a voice. scribe.ts streams each person's decoded voice to ElevenLabs Scribe as 48 kHz mono in 100 ms chunks, over one realtime WebSocket per speaker with the key in MATE_VOICE_ELEVENLABS_KEY_FILE, the read key, which holds speech_to_text and text_to_speech; the owner's session lives with the call and anyone else's closes after a minute of quiet and reopens at their next word; Scribe's VAD closes each utterance, and the bridge sends every partial and committed transcript to mate over the link, the speaker being the owner.

mouth.ts speaks mate's reply through the text-to-speech multi-context WebSocket in that voice, on MATE_VOICE_ELEVENLABS_TTS_MODEL, one context per utterance: mate's speak deltas go up a clause or a sentence at a time, since the server voices each piece as it comes, and the audio comes back as 24 kHz mono in many small chunks.

playback.ts plays each utterance as one stream through the same player as the join tone, in its turn in the queue: the stream starts once it holds JITTER_MS of audio, the chunks after it join the same resource, the player fills an underrun of up to MAX_MISSED_FRAMES frames with silence, and the context's final message ends the stream.

native.ts holds the floor between them: the owner speaking while mate speaks is a barge-in, which closes the context, stops the player and tells mate how many characters were voiced; a transcript made mostly of mate's own words from the last eight seconds is echo through the owner's speakers and is dropped.

A Scribe session that drops is redialled up to MATE_VOICE_SCRIBE_REDIAL_CAP times, then the call is deaf; the speech socket is reopened on the next word. On mate, voice-talk.ts keeps the call's turns, answers each committed transcript through the voice lane, streams the reply as speak frames, and on a barge-in or a newer transcript aborts the reply and keeps the part that was voiced.

The reply after a failed or absent lane is one spoken apology.

The other mode is ElevenLabs Speech Engine, when MATE_VOICE_ELEVENLABS_ENGINE_ID names an engine instead; it speaks a library voice, since it refuses a cloned one. It dials mate's /speech-engine path over the offsite tunnel, on the same port as everything else, with a JWT signed by the SHA-256 of the API key that owns the engine; mate holds that hash alone, from Secret mate-speech-engine as MATE_SPEECH_ENGINE_KEY_SHA256, and refuses every connection while it is unset.

Each spoken turn arrives as the whole transcript so far. speech-engine.ts streams the voice lane's reply back in chunks tagged with the turn's id, and a newer turn aborts the reply under way. The voice lane in voice-lane.ts answers each turn on MATE_VOICE_MODEL at MATE_VOICE_THINKING, with a spoken-style prompt, within MATE_VOICE_TIMEOUT_MS for the whole turn, tool rounds included.

A turn with no word from the model after MATE_VOICE_FILLER_MS opens with one short flat filler such as "One sec.", and a tool round the model starts before saying anything opens with "On it."; each is said at most once a turn, is part of the reply that the transcript records, and 0 turns the filler off.

Its tools are in voice-tools.ts and run in mate's own process. A call has one thread in the text channel the join came from, opened at the call's first spoken line and registered with the threads; the owner's committed words and mate's replies are posted there as the call goes.

start_task posts the task in that thread and runs it there as a typed message would, with the thread's sandbox, cards and Stop button, one task after another; task_status says whether each task started from the call still runs; channel_members lists who is in the voice channel by display name, username and roles, and who is talking, from the speaking events the bridge relays; recall_threads and recall_thread read mate's other threads.

When the owner is not alone, each reply also gets a line naming who else is in the channel and who is talking. Anyone in the channel may talk to mate, and their lines reach the lane as "Name: words"; the tools are offered on the owner's turns alone.

A task that changes something is recorded on its first call and read back, starting on a second call after the owner's next words were a yes; a read-only task starts at once. When a task's thread turn ends, mate sums up the thread's last message in a sentence and the bridge voices it as an announcement, between replies; on the Speech Engine path, which speaks only in reply, the result is folded into the next turn instead.

The tools and gates are the same in both modes. A model pi's catalog lacks leaves the lane off and mate booting; the engine then apologises. The engine itself is declared beside the agents that ElevenLabs reconciles.

On the bridge in that mode, engine.ts is the conversation with ElevenLabs Speech Engine: it fetches a signed URL for the engine named in MATE_VOICE_ELEVENLABS_ENGINE_ID with the key in MATE_VOICE_ELEVENLABS_KEY_FILE, which in that mode has to be the write key, since ElevenLabs signs a conversation URL only for a key with convai_write, opens the conversation WebSocket, sends the owner's decoded voice as 48 kHz mono in 100 ms chunks, plays the engine's 24 kHz voice back through the same player as the join tone, flushes it on an interruption, and answers pings.

The conversation starts when the voice connection is ready and ends with the call. Logs carry counts, never audio or words.

A turn#

1. The model streams its answer. A turn that calls no tool ends here, and no sandbox exists for it.
2. The first tool call leases the thread's sandbox: its own, else the ready spare, else a new one. mate moves the Sandbox's spec.shutdownTime two hours ahead.
3. mate opens one pods/exec stream to mate-hands with a new epoch, which kills any command an earlier stream left running. It writes the turn's credentials as files with mode 0600. Every later tool call in the turn uses the same stream.
4. At turn end, mate empties the credential files, and revokes the GitHub token. It then shuts mate-hands down, which ends every process the turn started, and moves shutdownTime two hours ahead.

If the sandbox dies mid-turn, the tool call fails, and the next call starts a new sandbox.

A new sandbox runs the image named in ConfigMap mate-sandbox-image, which CD rewrites at each mate-sandbox build. mate reads the mounted file at each mint, so a new sandbox image restarts nothing: the next new sandbox runs it, the sweep replaces a spare on an older image, and a thread's existing sandbox keeps its image until it goes. If the file cannot be read, mate keeps the last image it read.

Model routing#

A router in apps/mate/src/route.ts sends each model request to ChatGPT first. If ChatGPT fails before its first text, reasoning or tool call, the router sends the same request to the fallback in the same step, so pi records no failure. pi retries an error after content. Stop, the turn timeout and a context overflow never go to the fallback.

A breaker that all threads share keeps requests off ChatGPT while it is down:

After the wait, one request tries ChatGPT over SSE, and the other requests stay on the fallback. The breaker is in memory, so a restarted mate tries ChatGPT first. While the fallback answers, the turn's status line starts with ↪️ and names the reason. The first turn of a limit, auth, unconfigured or rejected outage ends with one notice.

A request leaves out the other model's reasoning where it sits beside an answer or a tool call. ChatGPT's requests count at $0 in mate_turn_cost_usd, so the metric is the fallback's list price. mate_model_routes_total counts each request by route and reason, and mate_model_primary_failures_total counts each request that ChatGPT itself failed, which MateModelPrimaryFailing reads.

Credentials#

The model key, the kthx agent token, the database role and the App key stay in mate's pod. mate reads and refreshes the ChatGPT sign-in only while MATE_MODEL is a ChatGPT model and mate-db has a URL and a CA. The ChatGPT sign-in stays in mate-db, which an operator sandbox reaches through Secret mate-db-app, so the agent can use the owner's ChatGPT account.

Sign Rowbutt in to ChatGPT has the kill switch. The files in the sandbox exist from the turn's first tool call to its end. An operator sandbox is cluster-admin, so it can still read every Secret in mate.

MATE_CONNECT_SECRET is unset, so a sandbox has no 1Password token. A read-only sandbox gets no GitHub token, SSH key, talosconfig, kthx deployer identity, ring token or Connect token.

The token's audience is api. offsite admits it for mate-sandbox-admin and mate-sandbox-reader, and folly admits the same token as federated:system:serviceaccount:mate:<account> through the allow-list in clusters/<site>/talos/patches.tf. clusters/folly/apps/mate-sandbox/ binds the admin user to cluster-admin and the reader user to view and mate-sandbox-reader, and clusters/folly/monitoring/mate-sandbox-reader.yaml gives the reader user its monitoring proxy access. mate writes a context for each cluster in MATE_SANDBOX_KUBE_PEERS from the sandbox's checkout: the API server in clusters/<cluster>/config/cluster-topology.json and the CA in clusters/<cluster>/config/kubernetes-ca.pem.

The 1Password item SSH: rowbutt holds the key as PKCS#8, and apps/mate/src/ssh-key.ts rewrites it in OpenSSH format when mate starts. The SSH client config, sshClientConfig in apps/mate/src/credentials.ts, completes a short host name with lolwtf.ca and reaches folly's Anchor network hosts through MATE_SANDBOX_LAB_JUMP. The Deployment leaves it unset, so the sandbox jumps through mate's default, capsule. folly's firewall policy nest_k8s_to_capsule_ssh admits the sandbox on capsule's port 22 and nowhere else on the Anchor network. See Routing and firewall.

An operator sandbox runs as ServiceAccount mate-sandbox-kthx, declared in clusters/offsite/apps/mate/kthx-deployer.yaml, through serviceAccountName in apps/mate/src/sandboxes.ts. The MATE_KTHX_GCP_* variables on the Deployment name the token's audience, the account and the ConfigMap. A sandbox without the kthx grant runs as the namespace default and gets none of this. The sandbox network reaches sts.googleapis.com, iamcredentials.googleapis.com, oauth2.googleapis.com and storage.googleapis.com through its web-port rule.

clanky-bot[bot] is in atlantis_users in clusters/offsite/apps/atlantis/policies/only-me.rego and in atlantis_appliers in clusters/offsite/apps/atlantis/policies/appliers.rego, so its comments can plan and apply.

Grafana accepts the cluster identity through the query-only sidecar in clusters/base/monitoring/grafana-mate/. It supplies a Viewer identity to loopback-only auth proxy, without a password or session cookie. Query Grafana as mate gives the API commands and limits.

Network#

sandbox-network-policy.yaml declares the sandbox policy and mate-sandbox-baseline, which selects every pod with lolwtf.ca/hands and allows DNS for cluster.local alone. A sandbox whose own policy is missing gets that egress and no more. sandbox-reader-network-policy.yaml declares the read-only policy and the checkout window. Before the agent's first command, mate sets the pod's lolwtf.ca/checkout to closed and waits for its CiliumEndpoint to carry the new label.

The microVM isolates the kernel. mate's ingress admits the node it runs on, for the readiness probe, Alertmanager's pods in monitoring, for the alert responder, the bridge's pods, for the voice link, and cloudflared's pods in cloudflare, through which ElevenLabs Speech Engine reaches /speech-engine; the edge admits ElevenLabs' addresses alone and mate verifies the JWT.

The sandbox's policy keeps it away from mate and from Alertmanager, whose API takes an alert from anyone, but as cluster-admin and root on the nodes it can reach both on purpose.

No policy selects the mate-db instance or the mate-db-backup Job's pod. The CloudNativePG controller, Prometheus and the kubelet reach the instance, and that Job reaches Google Cloud. A sandbox reaches neither, because its egress leaves out every pod in mate.

Session store#

clusters/offsite/apps/mate/database.yaml declares mate-db, one Postgres instance on a local-path volume. Its database and owner role are both mate. CloudNativePG writes the role's credentials to Secret mate-db-app and its CA to Secret mate-db-ca, and mate connects with sslmode=verify-full against that CA. Flux never prunes the Cluster, because CloudNativePG deletes the volume with it; removing the store is a deliberate delete.

packages/pi-store-postgres/ keeps pi's sessions in tables prefixed pi_. mate keeps a row per thread in mate_threads: its session id, its sandbox and the turn in flight, so a restarted mate finds every open thread and continues a closed one at its next reply. mate_slack_events holds the Slack event ids answered in the last day.

mate_wakes holds each thread's pending wake. mate_alert_groups holds each Alertmanager group the responder has seen, with its thread and the alerts the thread knows. mate_dreams holds each night's dream, with its thread and the window it read. mate_gateway_session holds the Discord gateway session. mate_profile_turns counts each automation and job profile's turns per UTC day.

mate_credentials holds the ChatGPT sign-in, and mate is its only writer. A credential that mate cannot save stays in memory, and mate retries the write every 30 seconds. mate keeps every session. When MATE_SESSION_RETENTION_DAYS is set above 0, it deletes the session of a thread closed for more than that many days.

If the store is down, mate stays connected and tells each thread that it cannot reach its memory.

CronJob mate-db-backup, in database-backup.yaml beside it, writes a gzipped pg_dump to gs://homelab-ng-mate/backups/pg/ at 04:43 UTC. The dump leaves out the rows of mate_credentials, so a restore needs a new ChatGPT sign-in. It signs in to Google Cloud as mate-db-backup@homelab-ng through workload identity federation, with no key. terraform/gcp/projects/homelab-ng/mate.tf declares that account and the bucket, which deletes a dump after 30 days. mate-db has no barman-cloud ObjectStore, because a base backup cannot leave out the credential rows.

Fence#

A ValidatingAdmissionPolicy in clusters/offsite/apps/mate/fence/ denies mate-sandbox-admin any pods/exec, pods/attach or pods/portforward into mate, which holds mate's credentials and every thread's sandbox. The sandbox's network policy also excludes mate from its in-cluster egress. Both guard against accidents, not intent: an operator sandbox is cluster-admin, so it can read mate's Secrets or delete the policy. mate-sandbox-reader holds no pods/exec, pods/attach or pods/portforward anywhere.

Rules#

• Keep one replica with strategy: Recreate. Two pods on Discord both reply, two on Slack each get half the events, and both write each thread's session.
• Run no second mate against mate-db, such as a local run with its DATABASE_URL. Both would refresh the one ChatGPT token, and OpenAI refuses a refresh token spent twice. bun run smoke in apps/mate is the exception: it answers on the fallback model, and opens no thread but its own.
• Keep the probe readiness-only. /healthz returns 503 while mate waits for Discord's session start limit, so a liveness probe restarts mate and spends more of it.
• Keep MATE_TURN_MINUTES under 55, or mate does not start.
• Keep MATE_MAX_SANDBOXES plus MATE_SPARES sandboxes within hotrod's free CPU. Each sandbox requests 500m, and one that does not fit stays Pending.
• Keep terminationGracePeriodSeconds at 45 or more. On SIGTERM mate lets running turns finish, then closes the rest, and the next pod resumes each one in a new message. A resumed command returns as interrupted, with its outcome unknown.
• Expect background processes to end with the turn, because mate shuts mate-hands down at turn end.
• Bump @earendil-works/pi-durable, @earendil-works/pi-ai and @earendil-works/chord together, while no turn runs: mate-db holds sessions in pi's format, and the new version resumes any open run. pi marks pi-durable experimental, so read its changelog before a bump.

Where it lives#

• apps/mate/src/mate.ts: the process, built from its config, its clock, its cluster, its database and its chat surfaces, with its timers and the order it stops in
• apps/mate/src/brain.ts: the agent loop, one pi-durable harness per open thread
• apps/mate/src/sandbox-lease.ts: a turn's lease on its sandbox, and the sandbox cap
• apps/mate/src/hands-env.ts: pi's execution environment over the mate-hands stream
• apps/mate/src/credentials.ts: the credential files of a turn
• apps/mate/src/sandboxes.ts: the Sandbox, its manifest and the spare pool
• apps/mate/src/sandbox-image.ts: the reader of the sandbox image file
• clusters/offsite/apps/mate/sandbox-image.yaml: the sandbox image
• apps/mate/src/responder.ts: the Alertmanager webhook and the mate_alert_groups table
• apps/mate/src/dreams.ts: the nightly dream, its assignment and the mate_dreams table
• apps/mate/src/recall.ts: the recall_threads and recall_thread tools over mate_threads and pi_entries
• apps/mate/profile/memory.md: what Rowbutt remembers, read into every system prompt
• apps/mate/src/evals.ts and apps/mate/src/evals-cli.ts: bun run evals, the ledger of recorded steps by model and the decision replay over mate_threads and pi_entries
• apps/mate/src/store.ts: the connection to mate-db and the mate_threads table
• apps/mate/src/credential-store.ts: the mate_credentials table
• apps/mate/src/wakes.ts: the wake tool, the mate_wakes table and the minute tick that fires wakes
• apps/mate/src/chatgpt.ts: the ChatGPT sign-in, the chatgpt commands and the keeper that refreshes the token
• apps/mate/src/route.ts: the router between ChatGPT and the fallback, and its breaker
• apps/mate/src/profiles.ts: the profiles, their grants, budgets and overrides
• apps/mate/src/profile.ts: the system prompt, its persona, the owner's instructions and the memory
• apps/mate/src/mcp.ts: the kthx MCP bridge
• packages/pi-store-postgres/: pi's session storage on Postgres
• packages/mate-hands/: the mate-hands daemon, and the protocol that apps/mate/src/hands.ts speaks to it
• images/mate-sandbox/Dockerfile: the sandbox image
• clusters/offsite/apps/mate/database.yaml: the session store
• clusters/offsite/monitoring/mate-rules.yaml: alerts, tested by mise run k8s:check-rules
• .github/containers.json: the CD entries for both images