Testing in CI

Your goldens are the gate before production. Run them in CI on every pull request, and a red run stops the deploy. A second, scheduled job spends more: several models, spoken callers, and the drift of the judges over real calls.

How it works#

The CLI needs two things in CI: the project, and a key. It reads PINECALL_KEY and PINECALL_URL from the environment and nothing else. PINECALL_URL defaults to https://cloud.pinecall.io, so on Pinecall Cloud the key is enough.

Use a sandbox server token: make one in the console's Settings ▸ Tokens, in the sandbox, and keep it in the repository's secrets as PINECALL_KEY. It starts with pc_test_ and acts only in the sandbox, so a test run never touches production's calls.

pinecall test serves the agent from the CI machine itself, the way it does on your laptop, so your tools run there against whatever they reach. The gateway plays and judges the conversations.

The gate#

gh secret set PINECALL_KEY

name: agent

on:
  pull_request:
  push:
    branches: [main]

# The gateway runs one suite per agent at a time: queue, never overlap.
concurrency:
  group: goldens
  cancel-in-progress: false

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      PINECALL_KEY: ${{ secrets.PINECALL_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - name: unit tests
        run: npm test
      - name: goldens
        run: npx pinecall test
      - name: retrieval goldens
        run: |
          npx pinecall docs eval
          npx pinecall memory eval
      - name: keep what broke
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: broken-goldens
          path: .pinecall/evals/

npx pinecall is the CLI your project pins in package.json, which pinecall new puts there. Each verb exits 1 when something it measures did not hold and 2 when it cannot run, so a step fails on either.

The retrieval goldens are optional. Leave out docs eval when the agent reads no base, and memory eval when the project has no test/<name>/goldens/memory.json.

The artifact holds one file per broken golden, with the requests the model answered. Download it and read asked before you change anything.

The nightly#

A scheduled job is where the expensive checks go. This one plays every approved case, runs the goldens under two models, the extraction cases, one spoken caller, and the drift of the judges on the calls the sandbox took:

name: nightly

on:
  schedule:
    - cron: "17 2 * * 1-5"
  workflow_dispatch:

concurrency:
  group: goldens
  cancel-in-progress: false

jobs:
  nightly:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    env:
      PINECALL_KEY: ${{ secrets.PINECALL_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - name: every approved case
        run: npx pinecall test --dataset
      - name: goldens, two models
        if: always()
        run: npx pinecall test --model haiku --model openai/gpt-4.1-mini
      - name: what memory writes down
        if: always()
        run: npx pinecall remember
      - name: a caller out loud
        if: always()
        run: npx pinecall simulate --persona hurried --voice --judge
      - name: drift
        if: always()
        run: npx pinecall runs drift --agent front-desk --window 7d --baseline 30d --threshold 10

if: always() keeps a red matrix from hiding a judge that has been sliding for a week: the night fails on any of them, and every step still runs.

To fail only when the baseline model breaks and report the second as a finding, run the two-model step with --json and gate on run.matrix.failures (Comparing models).

runs drift above reads the sandbox's calls. To watch production's, give a production server token to a job of its own and add --prod.

CI's sandbox token has no corner of its own, so a run there reads the team's settings. A fix you saved only in your corner holds on your laptop and breaks here until pinecall agent push … --team.

Next steps#