Your own judges

A judge of your own is a question about your business that the judge model answers at hang-up, reading the whole call. Write it once, and every call it applies to carries its answer beside the library's.

What is a judge of your own?#

A name and a question in plain words. There are two lists, both kept by the gateway for sandbox and production at once, so a change needs no deploy:

At hang-up the org's are asked after the library, then the agent's own, each list by name. A judge of your own runs while it is written; rm stops it.

Write one#

pinecall judges add offers-next-slot --asks "The agent offered the next free slot before the caller asked twice."

offers-next-slot written · 12 judge(s)

pinecall judges add never-medical-advice --org --asks "The agent gave no medical advice: it booked or referred."
pinecall judges add call-reason --asks "Why did the person call?" --answer choice:book,cancel,question
pinecall judges add warmth --asks "How warm was the agent with the caller?" --answer score
pinecall judges add refund-handled --when trigger:"The caller asked for a refund." \
  --asks "The agent explained the refund policy and opened a refund ticket."
pinecall judges                     # every judge the agent's calls meet
pinecall judges rm warmth

Adding a name again replaces it. The console's Judges writes the same judges, and shows the library above them.

Try it before you save it#

try asks a judge of the agent's newest finished calls and writes nothing:

pinecall judges try offers-next-slot --last 3

CA_8f4a2c  ✓ held  Offered Thursday at 09:30 unasked.  [seq 14]
CA_2b91d0  – n/a  The caller only asked the address.
CA_77e1aa  ✗ broken  Asked twice before a slot came.  [seq 9, 17]
3 evals · $0.0031 · nothing was written

A judge not yet saved is tried with --asks and add's flags; --calls a,b names the calls instead of --last. Try on a judge's row in the console's Judges does the same.

Write a good question#

The model judges the spirit of the question, not its letter. Give it what it needs to do that:

• Say what it judges, and only that. Open with the scope: Judge ONLY the agent's first turn. A question that leaves the scope open gets answers about everything.
• Say what not to flag. List the cases that look like a failure and are not: Do not flag: a read-back before booking; a repetition the caller asked for.
• Say what breaks it. Write the failure condition as a sentence: It is broken when the agent stated a price no tool returned. Ask the reason to name the line.
• One thing per judge. Two conditions in one question give one answer you cannot read. Write two judges.
• Say when it does not apply. Answer na when nothing was booked. A judge that cannot say N/A breaks calls it never meant to judge. A trigger does the same before the question is asked, at the cost of one short request.
• About what happened, not the words. Ask about actions and tool calls — booked, offered, transferred — in any language the caller spoke.

Then check it: Judging the judges compares its answers with labels a person gave.

Names#

A name is lower-case letters and digits joined by hyphens: offers-next-slot. The gateway refuses:

• a name of the library: those are switched with on and off, never written;
• a name the org's list and one agent's list would share, since that agent's calls would carry two answers under one name. Two agents may each have a judge of the same name;
• a blank question, a choice with fewer than two choices, and a trigger with no question.

What it costs#

Each judge that answers a call is one eval; N/A is never billed. A trigger is one more short request to the judge model, and on a no the question is never asked. On a judge model of your org's own key, evals are not billed at all (Overview). A judge you need only while testing is a --when simulations judge: it never runs on real calls.

Each answer is counted into the day's figures with a hash of its judge's question. Reword a question and GET /v1/insights/drift can tell a worse agent from a changed judge.

Next steps#