Goldens · Pinecall Docs

A conversation written down, with what must be true at the end, run under every model you name.

A golden is one conversation written down: the state it opens in, what the caller says, and what must be true when it ends. pinecall test plays it against a real model, under every model you name, and tells you which expectations held.

What is a golden?#

A golden is a JSON file in test/<agent>/goldens/, named after what it is about. A file holds one golden or a list of them. A golden with no name takes the file's name.

test/clinica-norte/goldens/books-after-yes.json

{
  "state":  { "stage": "book", "patient": { "name": "Marta Ruiz" } },
  "input":  ["That one works for me.", "Yes, please confirm it."],
  "expect": { "tools": ["book"], "not": ["free of charge"] }
}

This golden opens the call already at the booking stage, says two lines, and expects the agent to call book and never to say "free of charge".

The fields#

field	what it is
`name`	the golden's name in the report; the file's name when absent
`state`	the state the call opens in, written over the class's own state before the first render
`input`	the caller's lines, in order; 40 at most
`memory`	facts memory already holds about the caller. They answer `recall` for this call only, and nothing is written to memory
`events`	facts your backend sends mid-call: `{ "after_turn", "name", "data" }`. `after_turn: 0` is before the first line
`today`	`YYYY-MM-DD`, the day the model is told it is, so a golden that names a weekday reads the same next year
`expect`	what must be true at the end. Each field you set is one judge. Expectations reference has every one

The two other files in that folder, docs.json and memory.json, are goldens of another kind, for the index and for memory. pinecall test skips them; Testing knowledge & memory covers them.

How a golden is played#

Each golden is one written call on the same session a real caller gets, on your org's provider keys:

the call opens with no greeting: a golden starts in the middle of a conversation;
its state rides the call's call.started, and your agent opens in it;
each line is said in turn, and the run waits for the log to go quiet before the next one;
each events step is sent as your app's own call.event, after the line it names. An event your agent never declared is refused by name, and the golden ends there;
memory answers recall. search is the real index, because that is what the golden is asking about.

Run them#

Terminal

pinecall test                                            # every agent's goldens
pinecall test --agent sales                              # one agent's
pinecall test test/clinica-norte/goldens/books-*.json    # some files
pinecall test --grep books --watch                       # one golden, again on every save
pinecall test --model haiku --model openai/gpt-4.1-mini  # a column per model
pinecall test --json                                     # the run as one JSON document

The report is a matrix: one line per golden, the latency it was answered with, and under a golden that broke, one line per judge that did not hold, with its reason and the call's id:

  ✗ books-after-yes
        silence  broken   the golden forbids these and the agent said 'free of charge' in agent turn 2
        log               call_29d7c7b6cdd643de9c659984a0125c8c

The last line counts what held, the judge-model calls, the cost of the calls and how long it took. The verb exits 1 when any golden did not hold, so the same command gates CI.

With two models or more, the report adds the goldens the models disagree about. A disagreement is a finding: the answer is a fix in the class, never a softer golden. Comparing models covers it.

Said out loud#

pinecall test --voice plays the same goldens on a real spoken line: each caller line is synthesized in a voice the agent does not use, and the agent is dispatched to a worker as on a phone call. --background-noise <dB> puts a television behind the caller, and --packet-loss drops that share of the caller's packets, from 0 to 1. A spoken run builds the session from your agent's settings, so it refuses --model and a golden that sets today. Spoken tests has more.

Limits#

	at most
goldens in one run	200
models in one run	8
calls in one run (goldens × models)	200
lines in one golden	40
one run	15 minutes

A run stopped early (the agent's process went away, the org ran out of a quota, the deadline passed) keeps the goldens judged so far and says why.

Where goldens come from#

Three places, and each has its reason:

Source	Lives in	Use it for
written by hand	`test/<agent>/goldens/`	what the business wants, before a caller ever asks
a case	your org, on Pinecall	a real call a judge broke on, kept at hang-up; the nightly plays it once approved
a case pulled into the repository	`test/<agent>/goldens/`	a case whose fix is code: `pinecall cases pull <name>` puts it in the pull request

A golden that names your code (a stage, a tool, an event) belongs in the repository, so it changes in the same commit as the code it names. A case that only names words can stay a case.

To write any finished call down as a file without keeping it on Pinecall:

Terminal

pinecall runs promote call_29d7c7b6cdd643de9c659984a0125c8c

It writes the call's opening state, its caller lines and an expect derived from its broken verdicts to test/candidates/. Promote the call that broke: one that held gives an empty expect, and what it must keep doing is yours to write. Read the file and edit it before moving it into goldens/: where it came from is not approval.

Next steps#

Expectations referenceEvery expect field and how it is judged. PersonasCallers who improvise, for what goldens cannot foresee. CasesReal calls a judge broke on, kept, fixed and played every night. Testing in CIThe goldens as the gate before a deploy. pinecall testEvery flag of the verb.