A knowledge base that answers

A golden of real questions, documents written to be found, the base's language, the questions each section answers, and k: each step measured.

An agent answers from what its search hands it. When the right section is not among the chunks a turn reads, the model never sees it and answers from something close, or from nothing. This guide is the order of work that gets the right section into the turn, with one command to measure each step.

What makes a base find the answer?#

Three things, in this order of weight:

A golden of real questionsThe questions callers ask, each with the section that answers it. Without it every change is a guess. Documents written to be foundOne answer under one heading, a section that stands on its own, nothing in the folder that answers nothing. A search told what it readsThe base's language, the questions each section answers, and how many chunks a turn reads.

The figure that matters is recall@k: of the golden's questions, the share whose section came back among the k chunks a turn reads. A section that did not come back cannot be used, however good the model is.

Write the golden first#

Write agents/<agent>/test/goldens/docs.json before you change anything, and run it once to have a number to beat:

Terminal

pinecall docs eval

The best questions are the ones your callers already asked. Every search a call ran is in its log, with the caller's words as the query:

Terminal

pinecall sessions log <call> --types turn.user,docs.sources

A line is cut at 160 characters, which is the query and a little more. What each search found (the base, path, heading and score of every chunk) is in the entry's data, and --json prints it whole:

Terminal

pinecall sessions log <call> --types docs.sources --json

With no calls yet, write the questions a person would ask, in their words rather than your headings', and replace them as calls come in. Fifty to a hundred is the size that stops being noise. Testing knowledge & memory has the file's shape.

Put in what answers, and nothing else#

Strip what is not content. A page saved from a website brings its menu, its footer and its "related articles". Each becomes a chunk that competes for the turn's places and answers nothing.
One answer under one heading. ## Cancelling an appointment is found by a caller who wants to cancel; ## Other is found by nobody. A long page with no headings is cut at paragraphs and loses its subject.
A section stands on its own. A chunk reaches the model without the page around it, with only its heading path. Name the subject again in the text rather than writing "as above".
One page per subject. Two pages that say nearly the same thing push each other down the ranking. Keep one, or let the golden accept both.
Cut by measuring. Before taking a folder out because it looks like noise, push without it and run the golden. If recall drops, your callers ask about it.

Say the base's language#

Terminal

pinecall docs push --language en

Every chunk is found twice, by its meaning and by its words, and the words are stemmed in the base's language so that supervise finds supervising. Left out, the language is counted from the files, and pinecall docs list shows what it decided. Documents in two languages are two bases, each pushed with its own.

Write the questions each section answers#

An asks comment at the top of a section lists, one a line, the questions it answers. No site shows it; the search indexes each question in the words a caller would use, and a caller who asks it is handed the section.

agents/front-desk/docs/insurance.md

## Co-payments

<!-- asks
how much is my co-payment?
do I pay anything at the visit?
-->

A visit is fifteen euros at the desk; the rest is billed to the insurer.

Three to six per section, in the caller's words rather than the heading's. Write them by hand for the sections that matter most, or have a small model write them from each section and keep the result beside the documents, so that only a changed section is asked again. Each question counts as a chunk.

Choose how many chunks a turn reads#

Terminal

pinecall docs eval --k 4
pinecall docs eval --k 8
pinecall docs attach front-desk --k 8

A turn is handed whole sections, and every chunk costs prompt tokens on every turn. Take the smallest k after which recall stops rising.

Read every miss#

docs eval prints each question it missed with the first thing that came back instead. What came back says what to fix:

What came back	What it means	The fix
another section that answers as well	the golden is too narrow	add it to the question's `expects` list
a section on the same subject that does not answer	two sections overlap	merge them, sharpen the headings, or add `asks` to the right one
the right file, another section	the answer sits under a heading that does not name it	give it a heading of its own
nothing related	the caller's words are not in the documents	add `asks` in their words, or write the answer: it may be missing

Never reword a question so that a change passes. The golden measures what callers ask; a question made easier measures nothing.

What a good number is#

recall@k at the k you attach. Around 0.9, the agent usually has the answer in front of it; at 0.95 and above the misses are worth reading one by one; under 0.8 it often answers from the wrong page.
nDCG@10 says how high the answer ranked. The golden reads only the k sections a turn would get, so below k 10 it ranks within those k and a section past them is a miss.
A number compares only with itself: the same golden, the same embedder. A golden of easy questions scores high and says little.
On real calls, the grounded judge checks every answer against the chunks it was handed. Its held rate over calls that searched is what callers actually got.

Follow-up questions#

With --mode retrieved, each turn is searched on its own words. That is what makes it fast, and it means "and how much is that?" is searched with no idea what that is. The model already holds the search tool whenever a base is attached, and writes its query with the whole conversation in front of it. Say in its instructions to search again, with the whole question, when the chunks it was handed do not answer. Search from a tool is for a tool of your own that searches too.

What these steps did for our own agent#

The agent on pinecall.io answers from all of docs.pinecall.io: 186 pages, held to 61 questions.

	recall@4	recall@8	nDCG@10
pushed as it was	0.62	0.82	0.55
the language said, the golden's equal pages accepted	0.72	0.89	0.61
the questions each section answers	0.77	0.90	0.64

With four questions taken from real calls added (65), and the index read as it should be, the base answers at recall@8 0.91, and a search takes 270 ms end to end where it took 900.

Cutting the base to the pages a visitor reads first scored 0.80 at k 8. A fifth of the questions wanted a reference page, so the cut was undone: cut by measuring, above.

Next steps#

Knowledge basesPush, attach, and every flag of a base. Testing knowledge & memoryThe golden's file, its figures, and memory's goldens. How retrieval worksChunking, the two indexes, fusion, and what the log keeps. Testing in CIpinecall docs eval costs no model: run it before every push.