The runtime CLI

The operator's terminal: the two processes, the database, the tenants and the fleet. One verb per group; pinecall-runtime --help prints them all, <group> --help that group's verbs. The tenant's terminal is pinecall, the CLI repo's (pinecall/cli) docs/the-cli.md, and they never overlap: nothing here writes an agent, nothing there issues a key.

What each group speaks to#

gateway · worker start · worker overflow · runner start#

gateway serves both environments: in a pod on the address PINECALL_GATEWAY_LISTEN names, behind the cluster's load balancer; elsewhere on the loopback address PINECALL_GATEWAY_URL names, and a URL that is not loopback is refused in one sentence. worker start is a worker of the fleet PINECALL_FLEET names, until told to stop or cordoned; worker overflow the one that answers when the fleet is full. runner start keeps the hosted apps of the environment its PINECALL_RUNNER_KEY opens running, one gVisor pod each: the runner is itself a pod of the hosting cluster (charts/hosting), outside the runtime's own cluster (On Kubernetes, "Hosting"). Every variable they read is Environment variables.

local#

pinecall-runtime local [--dir <path>] up [--services-only]
pinecall-runtime local [--dir <path>] down [--volumes]
pinecall-runtime local [--dir <path>] init --email <address> --person "<name>" [--org <slug>] [--name "…"]
pinecall-runtime local [--dir <path>] env

The runtime whole on one machine, for developing an agent against it: no cloud account, no key from anywhere. up writes the Docker Compose files the package ships into --dir (~/.pinecall-runtime/local unless named), brings up Postgres (17, with pgvector and pg_textsearch, built once), Redis and LiveKit on loopback ports of their own, migrates the schema, draws the machine's secrets once into <dir>/env (0600: the vault key, the ops key, the token key, the sandbox fleet's key), then runs the gateway on http://127.0.0.1:8080 — the console included, the sandbox at /sandbox/ — and a worker of the sandbox fleet, their lines prefixed gateway | and worker |. Ctrl-C stops the two; the services stay. --services-only stops after the schema, for a checkout that runs the two from its sources (make local-gateway, make local-worker). Run up again any time: it keeps the secrets and the database.

down stops the services and keeps the database; --volumes deletes it. init is init below, run on the local settings, so the first org and person need no shell line. env prints that line, set -a; . <dir>/env; set +a, for any other verb against the local runtime. It needs Docker with the compose plugin: Docker Desktop on macOS and Windows, Docker Engine on Linux; without it up says so in one sentence. A phone call needs the SIP bridge on the host's network, which Docker on macOS does not have: docker compose -f <dir>/compose.yaml --profile phone up -d sip, on Linux.

init#

pinecall-runtime init [--org <slug>] --email <address> --person "<name>" [--name "…"] [--role admin]

The first org and the first person on a runtime nobody has used yet: the org made (or found, run twice), its first admin invited, that person made an operator of this deployment, and the invitation link printed once with the two lines to type next.

orgs#

orgs list · orgs add <slug> [--name] · orgs rm <org>
orgs invite <org> <email> --name "…" [--role] · orgs operator <org> <email> [--revoke] · orgs remove-member <org> <email>
orgs move <agent> <org>
orgs quota <org> --env production|sandbox [--minutes n] [--messages n] [--agents n] [--concurrent-calls n]
           [--memory-facts n] [--knowledge-chunks n] [--numbers n] [--seats n] [--llm-tokens n]
           [--budget-usd n] [--lends vendor[/model],… | --lends none]
orgs dialling <org> [--dial-anywhere | --no-dial-anywhere] [--per-minute n] [--per-day n] [--max-duration-s n]
orgs sso <org> [--off]
orgs provider-key set|rm|list <org> <vendor>        # set reads the key from stdin

<org> is an id or a slug. quota replaces the whole set for one environment; dialling the whole set of guards; sso --off is the break-glass; provider-key set reads from stdin because argv is what ps shows.

keys#

keys issue [--org <org>] --env production|sandbox [--label "…"] [--scope <scope>]… [--subject <member>] [--name "…"]
keys list [--org <org>] · keys revoke <fingerprint> · keys fleet production|sandbox
keys runner production|sandbox

issue prints the key once; the table keeps the fingerprint. fleet mints an environment's fleet key on the database and prints it once: in a cluster nobody types it, since the chart's install Job runs it for both environments into the secret pinecall-fleet-keys, and on a laptop make local writes the sandbox's into .local/env. runner mints the key of the environment's runner the same way, put once in Secret Manager for the hosting cluster (Hosted apps, On Kubernetes, "Hosting").

routes#

routes list [--org] [--env] · routes add <number> <agent> [--channel phone|whatsapp] [--org] [--env]
routes rm <number> [--org] · routes seed [--file infra/seed/routes.json]

seed reads a JSON list of routes, each the body routes add sends (number, agent, channel, org, env), and adds them one by one. Its default, infra/seed/routes.json, is not in the repository: name your own file with --file.

fence#

fence export

The networks 5060 opens to, Twilio's first, then the carriers the operator admits and the addresses he approved (Telephony, "The fence, twice"), as Terraform reads them: {"sip_sources": [...]}. Written to infra/terraform/environments/<env>/sip_sources.auto.tfvars.json (the orgs' addresses, never committed), make tf-plan and make tf-apply bring the cloud's firewall level with the platform.

sip#

sip repoint [--from <world>=<name>]…

Every trunk of the platform's Twilio account and of each org's that still sends an environment's calls to the platform's name, or to a name --from says the environment was reached at before (--from production=sip.old.example.com), sent on to its SIP name (PINECALL_SIP_DOMAIN, Environment variables), a line each: the account, the trunk, the environment, where it pointed and where it points now. Once, when the platform's SIP moves off its names or a name is retired; run again, it sends nothing on.

fleet#

fleet list [--fleet <name>] · fleet cordon <worker> · fleet uncordon <worker>

list is the roster the gateway hears: each worker, what it holds, its seats, load, standing (accepting, failing, full, draining, cordoned, gone) and when it was heard, then each fleet summed, with its rooms waiting for a worker when there are any (a call LiveKit or a worker dropped, offered again by the gateway: Scaling, "Who takes a call"). cordon tells a worker, on its next heartbeat, to take no new call, finish what it holds and leave; uncordon takes that back while it has not left. How many workers a fleet runs is the cluster's: Scaling, "The burst"

load#

pinecall-runtime load --org <id> --agent <slug> --script <call-log.json> --calls <n> --ramp <seconds> --minutes <m>

Synthetic calls held against the gateway at PINECALL_GATEWAY_URL exactly as workers hold them, so the control plane is measured without LiveKit, audio or a vendor. Each call knocks the endpoints a worker knocks, through the worker's own client: it opens a call in the sandbox environment (there is no other), queues the script's entries a worker writes at the script's own gaps on the worker's own writer, which sends what is queued as one batch while the last is out, then its call.ended, and seals it once all were answered; the entries the gateway writes itself (call.ringing, call.attached, tool.call, tool.result, call.summary, call.score, memory and sources) are left out, and so are tool round-trips, which need an app socket. A call that ends before the run does is followed by a fresh one, so the number held stays at the top; a refusal ends that call only, and its slot opens another a second later. --script is a call's log as JSON, the shape of tests/wire/golden/call-log.json (125 entries over 53.5 s); --calls is how many at once at the top, reached linearly over --ramp seconds and held --minutes.

PINECALL_GATEWAY_URL=https://<staging's sandbox name> PINECALL_WORKER_KEY=<its sandbox fleet key> \
  pinecall-runtime load --org org_load --agent load --script tests/wire/golden/call-log.json \
  --calls 500 --ramp 60 --minutes 5

It runs against a cluster made for it (staging, make deploy ENV=staging), never production's: the calls are real calls of the org, logged, counted against its quotas and sealed. The org is one made for the run, with its sandbox quotas (orgs quota --env sandbox) above the run and its hang-up judging off (PUT /v1/org/judging), or every seal asks the platform's judge model. Each call at once has a client and a connection of its own, as a worker's job has, so ulimit -n must be above --calls. At the end it prints, a line each:

sessions · memory · retention · traceback · facts · usage · drift · migrate · providers · doctor#

sessions list [--agent] [--limit], sessions show <call> [--json], sessions tail [<call>], sessions recording <call>: the log read back off Postgres, every tenant's; each read of a call is a row of its org's access log (reader: operator), and so is each org a traceback showed. memory reembed embeds every fact another model wrote under the platform's embedder. retention due lists the sealed calls past their org's retention_days (PUT /v1/org/policy), oldest first; retention run erases them, each through the erasure path with retention as who asked, 5 000 a run at most, then forgets the detail records of erased phone calls, and the dials, older than 24 months, and the WhatsApp message ids claimed more than 7 days ago (WhatsApp); then makes the log's days a week ahead (call_log is partitioned by UTC day; a row no day holds lands in call_log_default, and a day the default already holds rows of is named and not made) and drops each past day nothing is left in; the CronJob pinecall-retention runs it at 04:00 UTC every night. doctor's days line names rows in the default and fewer than two days made ahead. traceback <number> [--since YYYY-MM-DD] answers a carrier's traceback: every phone call with the number, still kept or erased with its record, and every dial to it placed or refused, with the org, the environment, the number shown and who asked — 24 months back unless --since says a day. migrate up applies what the database lacks, migrate status says what it lacks (exit 1 while behind), migrate plan names every migration on the disk. providers list [--does llm|stt|tts] lists every vendor this build runs and whether the platform holds its key; providers seed <file> writes the providers row a deployment starts from, once: after it, the console edits it at /v1/ops/providers. providers prices <file.csv> [--apply] says what a prices file changes in the row's rates (new, changed, the same, and the models only the platform holds, which it keeps) and writes nothing until --apply; the repository ships infra/seed/prices.csv. facts rebuild [--call <call>] [--org <org id>] [--since YYYY-MM-DD] folds each call's facts row (what the lists, the inbox and the insights read) again from its log, every call's when no flag is given, and writes the row where it differs: one call at a time, each in a transaction of its own under the lock its appends take, so a live call waits milliseconds and nothing holds a long transaction; it prints how many calls it read, how many rows it rewrote, and how many it left as they were: a log holding an entry this release's wire refuses is never refolded, because the fold would pass the entry by and lose what the release that wrote it folded from it (doctor's sample passes those by too). usage rebuild folds every org's usage totals (what admission counts, a row per org, environment and month) again from the summaries in the log, as the usage feed folds each one, and rewrites the table in one transaction that holds it: a summary written meanwhile waits and is counted after; it prints how many summaries and rows. doctor asks each thing the platform needs one question, a line each, and exits 1 when one is missing. Its facts line examines the 20 newest calls and 20 heads from a random point of the call ids (never every head: on a deployment that is a probe per call), names each whose head gave out fewer seqs than its rows hold and how many it examined, and refolds 20 sealed calls from a random point of the call ids, naming each whose stored facts differ and the columns that do: facts rebuild --call mends one. Its offers line names the rooms a caller joined over a minute ago that are still kept: three offers 12 s apart let every room go before then, so one still kept is a room no gateway is sweeping. drift rebuild [--org <org id>] [--since YYYY-MM-DD] forgets the drift of the days the flags name (every org's and every day's when none), each stage's histogram and each judge's count that /v1/insights and /v1/insights/drift read, and counts every sealed call of them again from its log, each in a transaction of its own; it prints how many sealed calls it read and how many it counted. A seal whose count broke says so in the gateway's journal, and this is what mends it.

vault rotate#

Every secret the platform keeps is sealed under the first key of PINECALL_VAULT_KEY, and any key the list holds opens it. To turn the vault: put a new key in front of the list (in a cluster, a new version of the secret pinecall-<world>-vault-key in Secret Manager holding <new>,<old>, which External Secrets brings to the pods within the hour) and restart the gateways, so every secret written from then on is sealed under the new one; then vault rotate re-seals every sealed value of the schema under the first key, one line per column:

box_settings.ciphertext: 3 re-sealed, 0 under the first key already, 0 opened by no key listed

The columns are the vendors' keys (the org's and the platform's), the carriers, the mailboxes, the identity providers, the hosted apps' tokens, the org's secrets, the private values of calls' logs, the recordings' keys and the one-use words of sign-in (a terminal's key while it waits). Each row is its own write, guarded by the value it still holds, so a secret the gateway rewrites meanwhile is left to it; a run cut short is run again, and finds what it did under the first key. It exits 1 while a value opens under no key listed: that value was sealed under a key the list no longer holds, and the old key comes out of the list only once every line says 0 there.