Expectations reference

A golden's expect says what must be true when its conversation ends. Each field you set becomes one check, and the report names the check, not the field. Most are settled by code; consent, grounded and judges ask the judge model.

How it works#

{
  "input":  ["Hi, this is Tom Baker, about the refund. When will they call me back?"],
  "expect": { "tools": ["takeMessage"], "not": ["tomorrow", "today"], "register": "usted" }
}

This golden is scored by five judges: consent and heard, which every golden gets, then tools, silence and register, one per field set. The golden holds when every one of them holds.

Every field#

state, memory, events and today sit beside expect, at the top of the golden. Goldens describes them.

The phrases: says, says_any, not#

All three match a phrase anywhere inside the agent's turns, ignoring case. Only the agent's turns are read, never the caller's.

• says needs every phrase. Use it for words that must appear.
• says_any needs one of them. Use it when there are several right answers: ["9 to 2", "nine to two"]. The reason names the phrase that was said.
• not forbids every phrase, and is reported as silence. The reason names each phrase and the agent turn it was said in.

A phrase is matched as text, so not: ["free"] also catches "freedom". Write the shortest phrase that can only mean the thing you forbid.

Tools: tools and not_tools#

tools reads which tools were called, not their order or arguments. not_tools names the log position of each forbidden call, so you can find it in the call's log.

Use not_tools, not not, when an action must not happen. "Did not say booked" passes when the agent books silently; "did not call book" does not.

consent and heard#

Every golden gets consent, the library's judge asked of the golden's call: it fails when an irreversible tool ran before the caller agreed to that action, after they declined, or on an agreement given to something else. A golden whose call ran no irreversible tool holds it without asking any model.

Every golden with lines gets heard. It fails when the agent heard fewer lines than the golden says, because whatever else that call did, it was not this golden. Without it, a caller nobody heard would pass every "never did X".

grounded#

"grounded": true asks the library's grounded of the golden's call: every concrete fact the agent stated — a price, an hour, a date, a name, an address, a policy, an availability — must appear, in any wording or language, in what the agent could read: its knowledge, the documents it searched, the facts recalled, its tool answers, or what the caller said. A call that states no such fact holds.

consent, grounded and judges ask the judge model. In a run with no judge model at hand they break the golden, saying so, rather than pass a call nobody read.

register#

For Spanish agents. "register": "usted" breaks on any whole word of the informal register (tú, ti, te, contigo, tu, tus, tuyo…). "register": "tu" breaks on usted, ustedes, consigo, suyo and its forms. le, les, su and sus are never counted: they are as often the third person.

replies#

replies is about the facts your backend injects with events, not about whether the agent answered the caller. It needs at least one event; a golden that sets replies and injects none breaks, because a check that could not look must not pass.

{
  "input":  ["Do you have anything on Thursday?"],
  "events": [{ "after_turn": 1, "name": "slot_freed", "data": { "at": "10:15" } }],
  "expect": { "replies": true }
}

With true, the agent's first turn after each event must contain a value from its data (here 10:15). With false, it must not. When does not matter, only whether.

judges#

judges names any judge — one of the library such as promises or disclosed, or one of your own — and asks each of the golden's call, as it would be asked at hang-up, whether or not your org switched it on. The golden breaks when one answers broken; na and a choice or score answer never break it.

{
  "input":  ["So you'll call me tomorrow to confirm?"],
  "expect": { "judges": ["promises"] }
}

A case a judge broke on expects exactly this. A name nobody wrote breaks the golden and names the judges there are. expected-outcome is never useful here: a golden plays written lines, so its caller expects nothing and the judge says na.

Next steps#