pinecall judges

judges#

pinecall judges [list] [--org | --agent <name>] [--json]
pinecall judges add <name> --asks '…' [--answer verdict|score|choice:a,b,c] [--when always|simulations|trigger:'…']
                    [--reads prompt,evidence,facts] [--org | --agent <name>] [--json]
pinecall judges on|off <name> [--org | --agent <name>] [--json]
pinecall judges rm <name> [--org | --agent <name>] [--json]
pinecall judges try <name> (--last <n> | --calls <id,id…>) [--asks '…' and add's flags] [--agent <name>] [--json]

Every judge is one question a model answers about a finished call, with the whole call in front of it — both sides' turns, the tool calls between them, each line under its log position, which the answer cites. A call meets three lists, in this order: Pinecall's library — consent, grounded, promises, disclosed, identified, honoured-stop, ended-well and expected-outcome on by default, relevance, repetition and sentiment off — switched on or off for the org (--org) or for one agent, the agent's switch winning; the org's own (--org), asked of every agent's calls; and an agent's own, about its job alone. The gateway keeps all of it for both environments: the console's Judges shows the same, and a change needs no deploy.

$ pinecall judges add offers-next-slot --asks "The agent offered the next free slot before the caller asked twice."
offers-next-slot written · 12 judge(s)

$ pinecall judges add call-reason --asks "Why did the person call?" --answer choice:book,cancel,question
call-reason written · 13 judge(s)

$ pinecall judges off relevance --org
relevance off · 11 judge(s)

$ pinecall judges
consent           pinecall       on  verdict                       always  Every irreversible tool ran only after the caller agreed to that action.
…
offers-next-slot  clinica-norte  on  verdict                       always  The agent offered the next free slot before the caller asked twice.
call-reason       clinica-norte  on  choice: book|cancel|question  always  Why did the person call?

$ pinecall judges try offers-next-slot --last 3
CA_8f4a2c  ✓ held  Offered Thursday at 09:30 unasked.  [seq 14]
CA_2b91d0  – n/a  The caller only asked the address.
CA_77e1aa  ✗ broken  Asked twice before a slot came.  [seq 9, 17]
3 evals · $0.0031 · nothing was written

add writes one of your own whole, and the same name again replaces it. --answer is how it answers: verdict (held or broken, the default, and the only answer that can fail a call), score (1 to 5) or choice:a,b,c (one of two or more). --when is which calls it reads: always (the default), simulations (only a call a simulated caller played) or trigger:'…' — a short yes-or-no asked first, and a no is N/A without the question. --reads adds what the judge reads beside the call: prompt (the agent's prompt as the call ran it), evidence (what the call carried: the text searched, the tool answers, the states) and facts (the org, the direction, the irreversible tools and their confirmations, an opt-out, how the call ended). Any judge may answer N/A — the question did not apply — and N/A is never billed. Each judge that answers is one eval; on a judge model of your org's own key (judging) evals are not billed.

on and off switch one of Pinecall's; one of your own runs while it is written, and the gateway refuses a switch of it with exit 1, the hint naming rm. rm drops one of your own; Pinecall's are never dropped, and the hint names off. try asks one judge of the agent's newest finished calls (--last, 1 to 50) or of the calls named (--calls), and writes nothing: a judge already written, or Pinecall's, by its name alone; one not yet saved, written whole with --asks and add's flags. Each line is a call and what the judge answered; the last, the evals and what they cost.

The agent is the project's one, or the one --agent names by its folder's name; --org names the org's list instead, needs no project, and beside --agent is refused; try is always an agent's. A name is lower-case letters and digits joined by hyphens, as a verdict names it; anything else, an add without --asks, an --answer, --when or --reads that is none of its words, or a try naming neither or both of --last and --calls, is refused here with exit 2, before it travels. A name the org and an agent would share is the gateway's refusal. --json prints what the gateway answered — {"judges": […]}, or the try's {rows, evals, cost_usd} — for every verb.

judging#

pinecall judging                                             whether calls are judged, and on what
pinecall judging [on|off] [--model <vendor/model> [--option key=value …] | --platform] [--json]

Every finished call of the org is judged at hang-up unless judging is off. The judges run on Pinecall's judge model, and each judge that answers is one eval on the org's bill. --model names a model of your own instead, for every agent's calls (an agent's own pinecall agent set --judge wins over it); the gateway tries it on the org's keys before it keeps it. On the org's own key for its vendor (pinecall providers add) the model's bill is yours and its evals are never billed; a local model is your vendor's plugin pointed at your server — --option base_url=…, repeated key=value like agent set's, run on your own key alone. On a key Pinecall lends the evals are billed as on Pinecall's, under the platform's ceiling per call. --platform goes back to Pinecall's model. The judge is asked for a tool call, so the model must call tools.

$ pinecall judging
judging on · Pinecall's judge model · evals billed · ceiling $0.05 a call

$ pinecall judging --model openai/qwen3-32b --option base_url=http://gpu.internal:8000/v1
judging on · openai/qwen3-32b · base_url="http://gpu.internal:8000/v1" · your own openai key: evals not billed

on and off keep the model; --model keeps on or off. A model with no vendor (qwen3), --model beside --platform, or --option with no --model is refused with exit 2 before anything is asked; a model the org's keys cannot run is the gateway's refusal, exit 1.