Overview
Goldens and personas, what each one catches, and the rings a change goes through.
Testing an agent means writing down what it must do, running that against a real model, and keeping it as the gate before every deploy. Pinecall gives you two kinds of test case and runs both for you.
What is a test?#
Start with both. Your first ten cases suggests six goldens and four personas, and walks the loop once.
How it works#
A test is a loop of four steps. You write the cases; Pinecall does the rest.
Write the cases#
Goldens are JSON files in test/<agent>/goldens/. Personas are kept by the gateway, written with
pinecall personas add or the console's Harness ▸ Personas.
Run them#
pinecall test plays every golden against a real model. pinecall simulate --persona <name> puts a
persona on the line, in writing or out loud.
Read what broke#
Every verdict carries its reason. A golden that broke is written out whole, with the exact requests the model answered.
Keep them#
The goldens run in CI before every deploy. After the deploy, every real call is judged at hang-up, and one that breaks a judge is kept as a case: reproduced, fixed, and played every night once you approve it.
The five kinds of test#
Each kind of test answers one question. They are not levels of thoroughness: green unit tests say nothing about the goldens, and green goldens say nothing about a real phone line.
| kind | the question | what runs it | costs |
|---|---|---|---|
| unit tests | does the class behave? | vitest, in your own repository |
nothing |
| goldens | does the agent hold its goldens? | pinecall test |
real model calls |
| spoken tests | does it hold on a spoken line? | pinecall simulate --voice, pinecall test --voice |
a call |
| a replay | does one real call pass the code checks? | pinecall eval <call-id> |
a replay, no model |
| judges | how did every call go? | the gateway, at hang-up | one eval per judge that answered; none on your own judge model |
Each has its page: Unit tests, Goldens, Spoken tests, Replaying a real call and the Judges at hang-up.
Where each half runs#
pinecall test and pinecall simulate start your agent in a process on your machine, exactly as
pinecall chat does. Your @tool methods run there, against your own database, and a breakpoint in one is
reachable with --inspect. The gateway drives the conversations and scores them, because the judges, the
provider keys and the call logs are its. So a test needs a key and a gateway like any other verb.
The gateway runs one suite per agent at a time. A second pinecall test while one is running is refused
with a 409 that names the run already going.
What a broken golden leaves behind#
Every golden that did not hold is written to .pinecall/evals/<run>/<golden>.json, in the directory you ran
from:
| key | holds |
|---|---|
| the golden | as you wrote it |
| the verdicts | every judge, with its reason |
| the log | the call's whole log |
asked |
every request the model answered, with its system blocks, tools and messages |
asked is the only place the prompt is kept. The call log stores a hash of each block on purpose. A spoken
run builds its requests on the worker, and the file says so instead of listing them.
A golden is fixed and the agent is the variable. Never soften a golden so a change can pass. When a golden breaks, the fix is in the class: a tool's docstring, a stage, a sentence of
render(). Change a golden only when what the business wants changed.
Golden judges and hang-up judges are not the same#
A golden is scored by the checks its expect sets, plus consent and heard. A real call is scored at
hang-up by the panel: the library's judges switched on for the agent, then the org's and the agent's own.
consent belongs to both; grounded and any other judge belong to a golden when its expect names them. A
call opened by a test run is scored by the run, never by the hang-up panel.