Drift
A judge's held-rate this week against the month before, and failing a job before a customer notices.
Drift is a judge's held-rate this week against the weeks before. One bad call is noise; a judge that falls
from 91% to 80% over a week is a change, and nobody sees it in any single call. pinecall runs drift counts
it and exits 1 past a threshold, so a scheduled job can fail before a customer notices.
How it works#
Every finished call is judged at hang-up, and its call.score carries each judge's verdict. Drift judges
nothing again: it counts the verdicts already written.
| Period | Calls counted, by default |
|---|---|
| The window | the calls that ended in the last 7 days |
| The baseline | the calls that ended before the window, back to 30 days ago |
For each judge it computes the share of held among held and broken in each period, and the difference
in percentage points. deferred and skipped verdicts are left out of both counts, and a call nobody judged
is counted on a line of its own, never as a failure.
Run it#
$ pinecall runs drift --agent clinica-norte
clinica-norte drift the last 7d against the 30d before it
consent 100.0% → 100.0% +0.0 points (40/40 held, was 110/110 held)
grounded 90.9% → 80.0% -10.9 points (32/40 held, was 100/110 held)
promises 99.1% → 100.0% +0.9 points (40/40 held, was 109/110 held)
2 not judged in the window, 0 in the baseline
the 3 most recent broken call(s):
…Each row reads: the baseline's rate, the window's, the change, and the counts behind them. Under the table,
the most recent broken calls in the window (up to three calls), each with the judge, the seqs it cites and
its reason. Here grounded fell 10.9 points, past the default threshold of 10, so the command exits 1.
| Flag | Default | Means |
|---|---|---|
--agent <slug> |
required | whose calls |
--window |
7d |
the recent period: a number and s, m, h or d |
--baseline |
30d |
how far back the comparison reaches; must be longer than the window |
--threshold |
10 |
the fall in points that fails the command |
--limit |
200 |
how many of the agent's newest calls are read |
--json |
the result as data: every judge's rates, the not-judged counts, the broken calls | |
--prod |
production's calls instead of the sandbox's |
The baseline is the time before the window, not including it. With the defaults the window is the last 7 days and the baseline the 23 days before them. A baseline no longer than the window is refused.
Enough calls to count#
Drift reads the agent's newest 200 calls at most, the most the sessions list returns. An agent that takes 200 calls in a week has nothing left for the baseline, and every judge shows a dash where its baseline rate should be. A judge with no settled verdict in either period has no change, and cannot fail the command.
Pick the window by traffic: a busy agent compares --window 24h --baseline 7d, a quiet one keeps the
defaults.
Reading a fall#
A judge's rate can fall for three reasons, and only one of them is the agent:
- The agent changed. A deploy, a new model, a new prompt. Compare the dates of the broken calls with
your releases, then turn one of them into a golden:
pinecall runs promote <call-id>. - The callers changed. A campaign, a new number, a season. The broken calls say what they asked for.
- The judge changed. Someone edited a judge's question, or the judge model moved. The
call.scoreof each call records which judge model gave it and a hash of every question asked, so a changed judge shows in the calls themselves.
A judge you wrote yourself drifts the same way as the built-in ones. Watch the one that matters most to the business, not only the defaults.
In the console and in CI#
An agent's Quality shows the same counts for its judges: the last 7 days against the 30 before, and the
points between, a fall past the threshold in red. Quality asks the pinecall start holding the agent for
those counts; with no such process, the table shows each judge's rate over the calls the page read.
In CI, put drift in the nightly job with if: always(), so a red test run does not hide a sliding judge
(Testing in CI).