Evaluate the Rowbutt models

Rowbutt stores each thread's session in mate-db. Each recorded step names the model that answered it. Use this runbook before changing MATE_MODEL or MATE_FALLBACK_MODEL in clusters/offsite/apps/mate/deployment.yaml. It also helps investigate slow or costly responses.

bun run evals in apps/mate provides two reports:

• A ledger of the recorded steps for each model.
• With --model, a replay of recorded tool decisions against a candidate model.

For each replay, the candidate receives the recorded context at one tool call. The score measures agreement on the tool and action. It does not measure correctness.

Each decision requests one call and executes nothing after it. If the candidate chooses another path, the recorded tool results cannot describe that path. Read the thread behind each disagreement before accepting either model's decision.

Before you start#

• Get kubectl access to offsite (Get cluster admin access).
• For a replay, have the key of the candidate's provider. An OpenCode Go candidate reads OPENCODE_API_KEY, which is the password of the opencode api key item in the homelab vault. Any other provider reads a pi-ai credential from the JSON file MATE_EVALS_CREDENTIALS_FILE, keyed by provider id; the replay refreshes an OAuth credential in place. Never use Rowbutt's own ChatGPT sign-in: mate refreshes that token, and a second refresher breaks it.
• A replay spends the candidate's tokens: each decision sends the thread's whole context so far.

Read the store#

1. Reach mate-db from the workstation.

   kubectl --context offsite -n mate port-forward svc/mate-db-rw 15433:5432 &
   sleep 2

2. Build the URL from the owner role's password, without printing it.

   password="$(kubectl --context offsite -n mate get secret mate-db-app -o jsonpath='{.data.password}' | base64 -d)"
   export MATE_EVALS_DATABASE_URL="postgres://mate:${password}@127.0.0.1:15433/mate?sslmode=require"
   unset password

   sslmode=require asks for TLS without checking the certificate, which a port-forward to the CloudNativePG pod needs. The owner role can write, so the evals open their one connection read-only with SET default_transaction_read_only = on and run only SELECTs; the role's password is letters and digits, so it needs no URL encoding. The nightly dump in gs://homelab-ng-mate/backups/pg/ restored into a local Postgres works the same way, with that database's URL, and reads nothing live.

Read the ledger#

1. Run the evals over the last week.

   cd apps/mate && bun run evals -- --since 7d

   Result: one row per provider/model that answered a step, with its steps, tool calls, tool results that failed, steps that ended in an error or were aborted, the median time from the message before a step to the step, and the cost pi-ai priced. --profile custodian keeps one profile's threads, and --json prints the same as JSON.

Replay the decisions#

1. Export the candidate's key, then name the candidate.

   export OPENCODE_API_KEY="$(op read 'op://homelab/opencode api key/password')"
   bun run evals -- --since 7d --model opencode-go/qwen3.8-max --decisions 40

   Result: a table for each candidate with these measures:

   • Decisions replayed and decisions answered.
   • Share of answers with the same tool and the same action.
   • Answers with text but no tool, errors, and cost.

   Each disagreement includes the thread key and step for review.

   Option	Purpose
   Repeated --model	Compare several candidates
   --thinking	Select the candidate's reasoning level
   --decisions	Limit the sample; select each thread's newest decision in turn

   A step with several tool calls is scored by its first call. The candidate sees the context pi sent for that step. After compaction, this means the summary and retained context, not the full history.

2. Decide. A candidate that agrees on most actions, answers with no tool rarely and costs less is a fair default; one that errors or answers with text where the recorded model called a tool is not ready. Change MATE_MODEL, MATE_THINKING, MATE_FALLBACK_MODEL or MATE_FALLBACK_THINKING in clusters/offsite/apps/mate/deployment.yaml through a pull request, and read Operate the Rowbutt model fallback for what each pair means.

Where it lives#

• apps/mate/src/evals.ts: the ledger, the decisions, the sample, the replay and the tables
• apps/mate/src/evals-cli.ts: bun run evals, its flags and the credential file
• docs/apps/mate/how-it-works.md#session-store: the tables the evals read