Replaying a real call · Pinecall Docs

pinecall eval: one finished call checked again by code, with an exit code for a pipeline.

pinecall eval checks one finished call again, by code alone, against rules you pass in: words the agent must never say, and how fast it must answer. Nothing is re-run and no model is asked, so it is fast, free and gives the same answer every time. It exits 1 when a check broke, so it fits in a pipeline.

How it works#

Pinecall rebuilds the call from its log and runs six checks over it. You choose the words and the latency budget; the rest comes from the log and from the tools the agent declares.

Check	Holds when	Skipped when
`consent`	every irreversible tool ran after its own `confirm.granted`	no tool declares a side effect, or the agent is not connected to the gateway
`register`	the agent said none of the `banned` words	no words were sent
`errors`	the call logged no error it did not recover from	never
`latency`	each budgeted measure's worst turn is within its limit	no turn carried a budgeted measure
`talk`	the agent's share of the talking is within `talk_share`	no `talk_share` in the budget
`interruptions`	each time the caller cut in with words, the agent answered them instead of starting its reply over	the caller never cut the agent off

Each check answers held, broken, deferred or skipped. Only broken fails the call. consent is deferred when irreversible tools ran and the log carries no confirmation at all, which today is every live call; the hang-up panel holds the same call and says why in its reason (Built-in judges).

Run it#

Find the call id in the console's Calls, or with pinecall sessions:

Terminal

$ pinecall eval call_29d7c7b6cdd643de9c659984a0125c8c --policy policy.json
call_29d7c7b6cdd643de9c659984a0125c8c  clinica-norte
  consent        held      no irreversible tool ran in this call; 1 tool call(s) did
  register       held      none of the 2 declared word(s) was said in 4 agent turns
  errors         held      the call logged no error
  latency        broken    llm_node_ttft 1.209s > 1.000s at its worst of 4 turns; tts_node_ttfb 0.412s <= 0.600s at its worst of 4 turns; e2e_latency 2.638s > 2.500s at its worst of 4 turns
  talk           skipped   no talk_share in the budget: send one, the most of the talking the agent may do (0 to 1)
  interruptions  skipped   no reply of the agent's was cut off in this call

--json prints the answer as the gateway wrote it: call, agent, passed, and verdicts, one per check with check, status and detail.

The policy file#

policy.json

{
  "banned": ["flat rate", "guarantee"],
  "budget": { "llm_node_ttft": 1.0, "e2e_latency": 2.5, "tts_node_ttfb": 0.6 }
}

A single word in banned matches whole words only (guarantee does not match inside guaranteed); a phrase matches anywhere. Case is ignored.

budget uses the names the call's metrics carry, in seconds, except talk_share, which is a share from 0 to 1:

Key	Measures	Default
`e2e_latency`	from the caller's end of turn to the agent's first audio	2.0 s
`llm_node_ttft`	the model's first token	1.0 s
`tts_node_ttfb`	the voice's first byte	0.6 s
`transcription_delay`, `end_of_turn_delay`	the ears, and deciding the caller is done	none
`dead_air`	the silence before a reply	none
`interruption_delay`	how long the agent kept talking after the caller cut in	none
`talk_share`	the agent's share of the talking time, checked by `talk`	none

A budget replaces the defaults whole. {"budget": {"talk_share": 0.6}} checks talk share and skips latency. A key that is not one of the names above matches nothing, so latency is skipped with no error: llm_ttft is not llm_node_ttft. Read the latency line after you change the file.

With no --policy, the banned words (register) and talk are skipped and latency uses the three defaults.

In the console#

A finished call's page has Re-check by code at the foot of its log: the same six checks, on the same call. Promote to a golden beside it writes the call as a golden candidate, like pinecall runs promote <call-id>.

When to use it#

After a change to the latency budget you promise a customer: replay last week's calls against it.
In a pipeline: a script that replays the newest calls and fails on the first broken.
On a call someone complained about: the six lines say what code can say before anyone reads the transcript.

It answers what code can decide. Whether the agent stated a fact it was not given, or kept a promise, is for the judges at hang-up (Judges).

Next steps#

Reading a callThe whole call: turns, tools, state, latency, verdicts. Call metricsWhat each latency measures, and where it is written. DriftEvery call's verdicts, counted over weeks.