Comparing models

Run the same goldens under several models in one command and read them side by side. Use it before you switch an agent to a cheaper or faster model, or to see which goldens depend on the model at all.

How it works#

Each --model adds a column to the matrix. Every golden is played once under every model named, through the agent your terminal serves, with the same settings, the same tools and the same provider keys. Only the model that answers changes.

pinecall test --model haiku --model openai/gpt-4.1-mini

With --model, the agent's own model is not in the matrix. To compare a candidate against what the agent runs today, name today's model too.

Reading the matrix#

With several models the report groups the goldens under each model, then lists the divergences: every golden and judge where the models disagree.

$ pinecall test --model haiku --model openai/gpt-4.1-mini
clinica-norte · 2 goldens · anthropic/claude-haiku-5-5 · openai/gpt-4.1-mini
  anthropic/claude-haiku-5-5
    ✓ no-reserva-antes-del-si  e2e_latency 2605ms · llm_node_ttft 763ms
    ✓ reserva-cuando-el-paciente-dice-que-si  e2e_latency 1948ms · llm_node_ttft 911ms
  openai/gpt-4.1-mini
    ✓ no-reserva-antes-del-si  e2e_latency 2210ms · llm_node_ttft 640ms
    ✗ reserva-cuando-el-paciente-dice-que-si
          tools  broken   the golden expects book, and this call ran freeSlots
          log             call_7c1e0a4d9b2f4e6a8c3d5f7a
  divergences
      reserva-cuando-el-paciente-dice-que-si  tools  ✓ anthropic/claude-haiku-5-5   ✗ openai/gpt-4.1-mini
  3/4 · 0 judge calls · $0.0480 · 14s
  1 reproduction written to .pinecall/evals/run_7ed3ace9352d/

Each passing line carries the median latency of that golden's turns. The last line counts the cells that held, the questions the judges put to a model, what the run's calls cost, and how long it took.

A broken cell is written to .pinecall/evals/<run>/. When the same golden broke under more than one model, the file name carries the model, so one never overwrites the other.

What a disagreement means#

A golden that holds under one model and breaks under another is a finding, not noise. It tells you the agent's behavior there rests on the model rather than on the class. Fix it in the class: a tool docstring that says when to call the tool, a stage that hides a tool too early to use, a sentence of render(). Then run both columns again.

Never soften a golden so a second model passes. A golden is the business's rule. If the rule is wrong, change it for every model, on purpose.

Gating on one model#

pinecall test exits 1 when any cell broke, under any model. When the second model is only a candidate, gate on the baseline and report the rest. --json prints the whole run, and run.matrix.failures lists every broken cell with its model, golden and metric:

pinecall test --model haiku --model openai/gpt-4.1-mini --json > run.json || true
jq -e '.run.status == "done" and
       ([.run.matrix.failures[] | select(.model == "anthropic/claude-haiku-5-5")] | length == 0)' \
   run.json

Check run.status first. A run that stopped early (the agent's process left, the run passed its 15-minute limit) still carries the cells it reached, and a partial matrix is not a result.

Limits#

Next steps#