Judging the judges
Label what a judge should have said; below 80% agreement it is untrusted.
A judge is only worth its verdicts if it agrees with what a person who knows would have said. Calibration is how you check: label what a judge should have answered on real calls, and the gateway tells you how often it agreed.
What is calibration?#
A label is one person's answer for one judge on one finished call: it should have held, or it should
not have. The gateway puts each label beside the verdict that judge gave on the same call and counts how
often they agree.
| rule | value |
|---|---|
| labels a judge needs before it is judged | 10 compared |
| agreement it needs to be trusted | 80% |
Below 10 compared labels, trusted is null: there is not enough to say. From 10 on, a judge that agrees
with fewer than 80% of them is trusted: false. It is reported as untrusted, never hidden and never dropped.
This works for every judge that answers held or broken: the library's and your own. A choice or
score judge classifies a call instead, and has nothing to compare a label with.
How it works#
Pick calls the judge has answered#
Read a call's score with pinecall sessions <call-id>, or on the console's Calls screen. Choose calls where
you know the right answer, including some the judge got right, some it got wrong, and some close ones.
Label them#
Send what the judge should have said, with a key that has the evals scope and may read the call:
curl -X POST "$PINECALL_URL/v1/evals/calibration" \
-H "Authorization: Bearer $PINECALL_KEY" \
-H "Content-Type: application/json" \
-d '{"call": "call_29d7c7b6cdd643de9c659984a0125c8c", "judge": "offers-next-slot", "held": false,
"note": "it offered Thursday, but only after the caller asked twice"}'A second label for the same call and judge replaces the first. The label is kept with the call, in sandbox or production, under the person whose key sent it.
Read the agreement#
curl "$PINECALL_URL/v1/evals/calibration?agent=clinica-norte" \
-H "Authorization: Bearer $PINECALL_KEY"{"judges": [{"judge": "offers-next-slot", "labelled": 12, "compared": 11, "agreed": 8,
"rate": 0.727, "trusted": false}],
"labels_to_judge": 10, "agrees_at_least": 0.8}Without ?agent=, the answer covers every agent of the org. A person's key reads sandbox unless the
request sends the pinecall-env: production header; a server token reads sandbox or production, whichever it was made in.
Reading the answer#
| field | means |
|---|---|
labelled |
labels given for this judge |
compared |
labels whose call has a held or broken verdict from this judge to compare with |
agreed |
of those, how many matched |
rate |
agreed / compared, or null before any were compared |
trusted |
null under 10 compared, then whether rate is at least 0.8 |
A label is labelled but not compared when the judge said na, skipped or deferred on that call, or the
call was never judged.
There is no CLI verb or console screen for calibration yet. The two requests above are the whole of it.
When a judge is untrusted#
Read the labels it disagreed with, and the reason it gave on each call.
- Your own judge. The question usually has two readings. Rewrite it so only one is left, then label new calls: the old verdicts answered the old question. Your own judges has the rules.
- A built-in judge. Its question is fixed. The rate tells you how much to rely on its verdicts for your
agent; the reasons tell you what it reads differently from you. When it does not fit your business, switch
it off for the agent (
pinecall judges off <name>) and write a judge of your own in its place.
Labels are tenant data like the call they are on: erasing the call erases its labels.