Cases
A real call a judge broke on, kept at hang-up: reproduce it, fix it where it belongs, approve it into the nightly.
A case is a real call — or a simulation — one of your judges broke on, kept so the same mistake cannot come back unseen. Nobody writes it: the call does, at hang-up. Cases are grouped by the judge that broke them, a group is diagnosed into a cause and a fix, and you apply the fix where it belongs and approve the group. From then on the nightly plays every case in it.
How it works#
A judge breaks#
Every real call is judged at hang-up. When one does not pass, Pinecall keeps it as a pending case of your org: the caller's lines, the state the call opened in, the facts memory recalled, what your backend sent mid-call, and what the broken judges forbid from now on.
It joins its group#
Every case joins its agent's open group for the judge that broke it: fifty calls one judge broke on are one thing to read. Diagnose asks the judge model to read the group's calls, the prompt, the settings and the tools, and to name the cause they share and the smallest fix — a setting, code, or knowledge.
You play it again#
pinecall test --case <name> plays the same caller against the agent you hold with pinecall start, in
the sandbox. Red means you reproduced it. The console's Run it does the same.
You fix it where it belongs#
A setting the diagnosis proposes is applied in one click (pinecall cases apply <group>): a new version
of your settings, no deploy. Code — a tool or a prompt block — is your assistant's to make: the console
copies the brief, and the pinecall mcp cases tool tells it what to change and how to prove it. The
table below says which is which when you fix it by hand.
You approve it, or dismiss it#
Approved, it joins the nightly. Dismissed because the judge was wrong, it becomes a label in that judge's calibration.
The console's Harness ▸ Cases is the inbox: what waits for you, what plays every night, what you
dismissed. pinecall cases is the same list in a terminal.
What a case expects#
A verdict knows what broke, never what was right. So a case's expect says what must not happen again:
| The judge that broke | The case expects |
|---|---|
consent |
not_tools: the tool that ran before its yes is never called |
grounded |
grounded: true: no concrete fact the call did not carry |
| any other judge of the library, the org's own, the agent's own | judges: [name]: that judge does not break when it is asked again |
expected-outcome |
nothing: its expectations are a simulated caller's, and a case plays written lines |
consent, grounded and judges are asked of the judge model; the other fields are settled by code. See
the expectations reference.
A simulation a judge broke on is kept too, with the persona that played it (the case says
simulated by interrupts). A call whose caller said nothing, or one already kept, is not.
Groups#
pinecall cases groups # one line a group: the judge, how many calls, the cause, the fix
pinecall cases propose grp_4c81e2a07d1b # the cause and the smallest fix, from the judge model (one eval)
pinecall cases apply grp_4c81e2a07d1b # a proposed setting, written as a new version of your settings
pinecall cases persona grp_4c81e2a07d1b # a caller like these, kept as the agent's own persona
pinecall cases approve grp_4c81e2a07d1b # every case in it, at once
pinecall cases dismiss grp_4c81e2a07d1b --judge-was-wrong promisesA diagnosis is {cause, kind, where, change, confidence}. setting names a setting and its new value;
code names the tool or the prompt block and what it must do; knowledge what the agent must be told;
none says the calls show no fault of the agent's — dismiss the group as the judge's mistake. The
console's Harness ▸ Cases shows the groups above the loop, each with Diagnose, Apply the
setting, Open in your assistant (the brief copied for a Claude with the Pinecall MCP), Make a
persona, Approve all and the judge's mistake.
Reproduce it#
pinecall cases # what waits, what plays every night, what you dismissed
pinecall cases show promises-me-llaman-manana-29d7c7
pinecall test --case promises-me-llaman-manana-29d7c7--case plays a case whatever its status, so you can reproduce one before you decide it. A golden that did
not hold is written to .pinecall/evals/<run>/<name>.json, and its asked holds every request the model
answered: the four static blocks and the view, turn by turn. That is where you read which block is wrong.
Fix it where it belongs#
| The reason points at | The block | Fix it in |
|---|---|---|
| a price, an hour, a policy, what to say when asked | knowledge | the settings: Configure ▸ Knowledge, or pinecall agent knowledge edit |
| how the call opens or ends | greeting, hang-up | the settings: pinecall agent set --greeting … |
| a fact kept that should not be, or one forgotten | the memory policy | the settings: --remember, --forget |
| the model guessed, or cut the caller off | the model, turn taking | the settings: --llm, --endpointing-ms |
| the wrong thing in one stage | render(), the view |
the code |
| a tool used for the wrong thing, or before a yes | the tool's docstring | the code |
| the same slip in every turn | the class docstring | the code |
A fix in the settings#
Save a new version of your own settings#
In the sandbox, pinecall agent set and Configure write your corner. A colleague's calls do not hear it.
Run the case on that version#
pinecall test --case promises-me-llaman-manana-29d7c7 --version 15The console's Run it has the same picker: your versions, each with its note.
Hand it to the team, then to production#
pinecall agent pull > mine.json
pinecall agent push mine.json --team
pinecall agent push mine.json --prod
pinecall testfrom your laptop runs on your corner; CI's sandbox token runs on the team's. A fix that is only yours is green at your desk and red in CI until you push it to the team.
A fix in the code#
pinecall test --case promises-me-llaman-manana-29d7c7 --watch # again on every save, until it holds
pinecall cases pull promises-me-llaman-manana-29d7c7 # into test/<agent>/goldens/pull writes the case as a golden in your repository, so the pull request carries the test of its own fix:
a golden that names a stage, a tool or an event changes with the code that names them. The case is marked
kept in the repository, and the nightly leaves it to the file.
Decide it#
| Console | CLI | |
|---|---|---|
| keep it, and play it every night | Approve | pinecall cases approve <name> |
| the judge was wrong | Dismiss ▸ |
pinecall cases dismiss <name> --judge-was-wrong <judge> |
| not worth keeping | Dismiss ▸ Not worth keeping | pinecall cases dismiss <name> |
| look at it again | Open it again | pinecall cases reopen <name> |
| a whole group at once | the group's buttons | pinecall cases approve <group> · dismiss <group> |
A case is never softened so a change can pass. If the business wants something else now, dismiss the case and write the new rule as a golden.
Keep a call by hand#
A call that went well is worth keeping too. On a finished call, Keep as a case keeps it approved, or:
pinecall cases keep call_29d7c7b6cdd643de9c659984a0125c8c --name thursday-afternoonA call that held gives an empty expect: pull it and write what the agent must keep doing.
Limits#
| at most | |
|---|---|
| cases waiting per agent | 50 |
Past 50 a broken call is judged as ever and not kept, until you decide some. Cases are played in the sandbox only, as written calls, never through production's tools. A case is erased with the call it came from.