The harness

Write who calls#

A persona is a caller a model plays: a goal, a manner, the facts they know, and their own rule for hanging up satisfied. Four to start — the one who books, the one who cancels, the one who asks what you do not do, the one who will not say their name.

pinecall personas add el-que-cancela --goal "cancelar la cita del jueves" --style "apurado, al grano"

Put one on the line#

A simulation is that caller against the agent, in writing or out loud, improvised turn by turn. Pinecall plays the caller and calls the agent as anybody calls it, so whatever answers it answers the simulation: the agent you deployed, in production or the sandbox, or the one pinecall start holds in your directory. The console's Harness ▸ Simulations starts one and shows it as it happens; a spoken one is heard live, both sides. Its tools run for real: an agent that books, books.

pinecall simulate --persona el-que-cancela --voice

Say what every call must have done#

The judges read every call at hang-up: the runtime's panel — consent, grounded, promises — and the ones you write, one question each, for the org or for one agent. A simulation meets them too, so a persona's run is a verdict, not a transcript to read.

pinecall judges add offers-next-slot "The agent offered the next free slot before ending." --agent front-desk

Replay the goldens#

A golden is a conversation written down with what must be true at the end. pinecall test replays every one under every model you name and answers a matrix; CI runs it before a deploy and fails on a golden that broke. Every run is kept: the console's Goldens tab lists them with what changed since the run before, and pinecall runs diff says it in a terminal.

pinecall test --models haiku,sonnet
pinecall runs diff

A broken call comes back as a case#

Every real call is judged at hang-up. One that breaks a judge is kept whole as a case: the caller's words, the state it opened in, what memory recalled, and what the broken judges forbid. You play it again against the agent you hold, fix it where it belongs — the settings, or the code — and approve it into the nightly, so the same mistake cannot come back unseen.

A number that slips fires a monitor#

A monitor watches one number of the Observability series — end-to-end or first-token latency at the median, the judges' held rate, the share a person took over, the share of tools that failed, spend, calls — over 1, 7 or 30 days, above or below a line, for every agent or one. The first seal of a call that finds it on the wrong side fires monitor.fired on the agent's log, once a day, with the value that crossed; Harness ▸ Monitors shows the last day each fired, the bell tells whoever chose it, and the org's webhook gets the post (Alerts).

pinecall monitors add "slow answers" --metric e2e_median_s --above 2 --days 7
pinecall monitors add "judges slipping" --metric held_rate --below 0.9 --agent front-desk

Drift says it in points#

Drift compares a judge's held-rate this week against the month before and exits 1 past a threshold, for a scheduled job. A monitor is the same idea over any number, set once in the console and fired by the runtime.