Your first ten cases

You do not need a test plan to start. Write ten cases, run them, read the ones that broke, and fix what they found. One pass of that loop gives you a suite you can rely on before every deploy.

How it works#

This page walks the loop once on the agent pinecall new writes: a receptionist that answers from the business's documents and otherwise takes a message with one tool, takeMessage.

A golden checks the agent's decisions. A persona finds what you did not think to write. Start with both:

{
  "input": ["Hi, this is Ana Lopez. Please tell the manager my last order arrived broken."],
  "expect": { "tools": ["takeMessage"] }
}

{
  "input": ["I'd rather not say who I am.", "Just tell them the order was broken."],
  "expect": { "not_tools": ["takeMessage"] }
}

{
  "input": ["This is Tom Baker, about the refund. When will they call me back?"],
  "expect": { "tools": ["takeMessage"], "not": ["tomorrow", "within", "today"] }
}

pinecall personas add hurried --goal "leave a message about a broken order" \
  --style "short sentences, interrupts, gives one fact at a time" \
  --fact "their name=Ana Lopez" \
  --expect "The agent should confirm the message reached the team." --interrupts 0.3

pinecall personas add distrustful --goal "find out whether a refund was really issued" \
  --style "doubts every answer, asks how the agent knows, repeats the question in other words" \
  --fact "their name=Marta Ruiz" \
  --expect "The agent should say what the documents say, without inventing a date." \
  --expect "The agent should offer to pass the question on."

pinecall personas add confused --goal "report that an order arrived broken" \
  --style "rambles, mixes up the order and the invoice, gives the facts out of order" \
  --fact "their name=Luis Prado" \
  --expect "The agent should ask one question at a time." \
  --expect "The agent should read the message back correctly." --pause 1.5

pinecall personas add wants-a-person --goal "speak to a human about a refund" \
  --style "polite, asks twice whether this is a machine" \
  --fact "their name=Tom Baker" \
  --expect "The agent should say it is an assistant when asked." \
  --expect "The agent should take a message for a person."

pinecall test                                     # the six goldens, against a real model
for p in hurried distrustful confused wants-a-person; do
  pinecall simulate --persona "$p" --judge        # each caller, judged at hang-up
done

Two kinds of judges#

The two verbs above are judged by different judges. Read each report knowing which is which.

Only consent is in both, and grounded when a golden asks for it. A golden never runs promises or your own judges: write those as a question on every call instead.

Read what broke#

Every verdict comes with its reason. A golden that broke is also written out whole to .pinecall/evals/<run>/<golden>.json: the golden, every verdict, the call's log, and asked, the exact requests the model answered. When a case breaks, one of three things is wrong:

• The agent. It did what the case says it must not. Read asked, or run pinecall prompt to see the prompt a state produces. The fix is in the class: a tool's docstring, a stage, a sentence of render().
• The case. It asks for something the business does not want. Change it, and make it specific: "takes the message and says it reached the team", not "handles it well".
• The caller or the judge. A persona that did not pursue its goal needs a sharper --goal or --style. A judge question that can be read two ways needs rewriting.

pinecall test --grep <name> --watch runs one golden again every time a golden file changes. A change to the class needs a restart of the command.

Look at the cases that held too. A pass the agent got by luck is a case that will break later.

Keep them#

These ten cases are your regression suite. Run pinecall test before every deploy, in CI, on a sandbox server token kept in the pipeline's secrets as PINECALL_KEY. A red run stops the deploy.

Once the agent takes calls, grow the suite from what really happened:

• A question about this agent's job becomes a judge on every call: pinecall judges add offers-a-callback --asks "The agent offered to have someone call back."
• A call worth keeping becomes a golden candidate: pinecall runs promote <call-id> writes it to test/candidates/. Read it and move it into goldens/ yourself.
• A judge that slides over a week shows up in pinecall runs drift.

Next steps#