Spoken tests

A written test proves the agent decides right. A spoken test proves it still decides right when every word goes through a voice, a phone-quality line and speech recognition. Add background noise and lost packets, and you test the line your callers actually have.

How it works#

--voice puts a real call on a real line. A Pinecall worker opens a room, dispatches the agent into it, and a simulated caller speaks each line out loud. The agent hears it through its own speech-to-text, answers with its own voice, and the same judges read the log the call leaves.

The caller speaks the agent's language in a voice the agent does not have, so the two sides are told apart by ear. A persona that names its own --tts and --voice is read in those instead.

Goldens, out loud#

pinecall test --voice --grep reserva

The same goldens, the same judges, the same matrix. Each line of input is said in turn, and the heard judge checks that every one of them reached the agent: a line the speech-to-text lost is a broken golden, not a silent pass.

Two things a written run allows are refused out loud, because the worker builds the spoken session itself:

A broken spoken golden is still written to .pinecall/evals/<run>/, but its asked says the requests were not recorded: they were built in the worker. The log in the same file is the whole record.

Personas, out loud#

pinecall simulate --persona hurried --voice --judge
pinecall simulate --persona hurried --listen

--listen puts the call on this machine's speakers while it happens, both sides mixed, through ffplay, play, aplay or pw-play, whichever you have. It turns --voice on and says so.

$ pinecall simulate --persona hurried --listen --turns 2
--listen is a call with audio in it: --voice is on
hurried · move the appointment to Tuesday afternoon, giving no more than needed

The caller waits for the agent's greeting before its first line, as a person does. On a spoken call the caller's turns printed are what the agent's speech-to-text heard, not what the caller meant to say.

A spoiled line#

pinecall test --voice --background-noise 12 --packet-loss 0.05
pinecall simulate --persona hurried --voice --background-noise 12 --packet-loss 5

The two verbs read --packet-loss differently. pinecall test --voice --packet-loss 5 is refused, because a share is at most 1. In simulate, setting only --packet-loss also puts the television behind the caller at 15 dB; set --background-noise too when you want loss alone.

Both flags are about audio, so simulate refuses them without --voice.

What it costs#

A spoken test is a real call: the agent's speech-to-text, model and voice, and the caller's model and voice, all billed like any call. A spoken simulate takes 40 caller turns at most. Run spoken tests on the goldens that matter for audio (names, numbers, dates said aloud) rather than the whole suite on every push.

In the console, an agent's Harness ▸ Simulations runs the same spoken simulation and lets you hear it live in the browser. Its Noisy line settings start at 15 dB and 0% loss.

Next steps#