Testing knowledge & memory

A conversation golden cannot tell you whether the index missed a better passage, or whether a call taught memory the right facts. Three goldens of their own answer those questions, and two of them cost no model call at all.

How it works#

Knowledge and memory each have a read side and a write side. A conversation only ever sees what it was handed, so each side gets a golden that looks at the step directly.

docs eval and memory eval score by code, with no model in the loop: two runs over the same data give the same numbers. Each exits 1 when a question missed.

The index: docs.json#

A list of questions, each with the chunk that should answer it. expects is the heading path a chunk carries: a file alone accepts any chunk of it, a heading accepts that section and what is under it.

[
  { "asks": "how much is my co-payment?",
    "expects": "insurance.md › Insurance and billing › Co-payments" },
  { "asks": "do I have to fast before the blood test?",
    "expects": "test-preparation.md › Preparing for a test › Blood tests" }
]

$ pinecall docs eval
clinica-norte · pplx-embed-context-v1-0.6b · 7 questions · recall@4 1.00 · nDCG@10 0.89 · 918 ms

With no --k, it asks with the k the agent reads that base with, because the golden asks what a turn gets. --k 1 asks whether the right chunk comes first.

Every missed question is printed with what came back instead. Fifty to a hundred questions per base is the size that stops being noise.

Recall: memory.json#

Each question brings the facts its contact holds and what the caller just said. The facts go to a scratch contact and are deleted after the question, so no real contact is read or written.

[
  { "holds": ["Prefers mornings", "Dr. Vidal's patient since 2024", "Allergic to penicillin"],
    "asks": "does Tuesday work for you?",
    "expects": ["Prefers mornings"] }
]

pinecall memory eval            # with the k a turn reads: six facts
pinecall memory eval --k 1      # is the right fact first?

A fact counts as found when what came back contains it, ignoring case, accents and spacing.

Give each question more facts than a turn reads. A contact with four facts gets all four back whatever the ranking did, and recall@6 1.00 then says nothing. A smaller --k makes recall bite.

Extraction: test/<name>/memory/#

At hang-up, one model call reads the whole call and decides what memory adds, replaces and drops. A case is one call already held, both speakers, and what must come of it:

{
  "name": "keeps the allergy and never the card",
  "said": [["caller", "I'm Marta, I'm allergic to penicillin"],
           ["agent",  "Noted. Does Tuesday work for you?"],
           ["caller", "Yes. And here's my Visa, 4242 4242 4242 4242"]],
  "holds": [],
  "expect": { "writes": ["allergies"], "never": ["payments"], "never_says": ["4242 4242 4242 4242"] }
}

pinecall remember               # every case
pinecall remember --grep tarjeta

Every answer is judged by code: a category is a word, a value is a literal, a superseded fact is an id. No model compares two sentences.

Use: a golden with memory#

A conversation golden can open the call already knowing facts about the caller. They are answered to the agent's recall for that call alone, and nothing is written to memory.

{
  "memory": ["Prefers to be called in the morning"],
  "input": ["Do you have anything on Tuesday?"],
  "expect": { "tools": ["freeSlots"], "not": ["what time suits you"] }
}

What the agent knows by heart (the agent's Knowledge setting) has no golden: it reaches the model whole on every call, so nothing ranks it. The conversation goldens and the grounded judge cover it.

Next steps#