Overview
Every call judged at hang-up, each judge a question a model answers: what it answers, when it runs, and what an eval costs.
Every call your agent finishes is judged at hang-up, with nobody watching. Each judge is one question a model answers about the finished call, and its answers are the last entry of the call's log.
What is a judge?#
A judge is a name and a question about a finished call. The judge model reads the whole call — both sides' turns and every tool call between them, each line under its position in the log — answers the question, gives one sentence of reason, and cites the lines it rests on. Nothing is run again.
How a judge answers#
A judge answers one of three ways:
| answer | what comes back |
|---|---|
verdict |
held or broken. The only answer that can fail a call |
choice |
one of the two or more choices it lists, written classified with the choice |
score |
1 to 5, written classified with the score |
Any judge may also answer na: the question did not apply to this call. An outbound-only question on a call
that came in is na, and so is a question about booking on a call where nothing was booked.
A judge runs on one of three kinds of call:
| when | runs on |
|---|---|
always |
every call the org judges. The default |
simulations |
only a call a simulated caller played |
trigger |
a call where a short yes-or-no, asked first, says it applies. A no is na, and the question is never asked |
Beside the call, a judge may read more. prompt is the agent's prompt as the call ran it; evidence is what
the call carried (the knowledge, the documents searched, the facts recalled, the tool answers, the states);
facts is what Pinecall knows about the call (your org's name, the direction, which tools are irreversible
and their confirmations, an opt-out, how the call ended, what a simulated caller expected).
How it works#
When a call ends, the gateway writes its summary, then asks the panel, in this order:
| order | judges |
|---|---|
| 1 | Pinecall's library, the judges switched on for this agent |
| 2 | the org's own judges, by name |
| 3 | the agent's own judges, by name |
Each answer is written into one call.score entry, then the log is sealed. A judge whose model fails is
skipped, saying why, and the call seals anyway.
The answers#
| answer | means |
|---|---|
held |
the call met the question |
broken |
it did not; the reason says where, with the log positions |
classified |
a choice or score judge answered; it neither holds nor breaks |
na |
the question did not apply. Never billed, never counted for or against the call |
deferred |
the judge model could not settle it |
skipped |
nobody asked it: no judge model, the call's ceiling was reached, or the model failed. The reason says which |
A call passed when no verdict judge said broken among those that said held or broken. A call where
no verdict judge held or broke carries no passed at all, and not_judged says why when nothing answered:
it is neither green nor red.
Evals: what judging costs#
One eval is one judge that answered one call: held, broken or classified. na, deferred and
skipped are never counted. call.score carries the call's evals, and they are billed to your org.
The judge model is chosen in this order:
- the agent's own:
@judgein its class, or thejudgesetting; - the org's:
pinecall judging --model, or the console's Judging card; - Pinecall's judge model.
A model you name that runs on your org's own key for its vendor — a local model through base_url among
them — is yours to pay: its evals are never billed, call.score says own_key, and no ceiling of
Pinecall's applies. On Pinecall's model, or on a key Pinecall lends, the evals are billed, and what judging
one call may spend is capped by a ceiling. Once a call reaches it, every judge after that is skipped,
saying so. GET /v1/org/judging reports the model and the ceiling.
The judge is asked for a forced tool call, so a model you name must support tool calling.
When a call is judged#
A call is judged at hang-up when your org judges its calls and a judge model is at hand. Judging is on unless
someone turned it off (pinecall judging off, or PUT /v1/org/judging with a usage key); the console's
Overview offers to turn it back on.
A call opened by a test run is judged by that run, not by this panel. A call that was not judged can be judged later:
curl -X POST "$PINECALL_URL/v1/evals/judge/call_29d7c7b6cdd643de9c659984a0125c8c" \
-H "Authorization: Bearer $PINECALL_KEY"A call judged already is refused unless you add ?again=true, which replaces its answers. The judges are
read as they are when this runs, not as they were when the call ended.
Read the answers#
pinecall sessions <call-id> # the score of one call, one line per judge
pinecall simulate --persona hurried --judge # a simulated call and its score
pinecall runs drift --agent clinica-norte # each judge's held-rate this week against the month beforeThe console's Quality shows the share of judged calls that held, day by day, and what broke; Judges lists every judge a call meets, with how each has held. Drift explains how to fail a scheduled job when a judge starts sliding.
A golden is scored with checks of its own beside the judges:
heardand one check perexpectfield (Expectations reference).consentis asked of every golden,groundedwhen a golden asks for it, and any other judge when the golden names it.