Your own GPU

The runtime names no vendor in code: every livekit plugin it carries is a vendor, and the providers row says which ones a call runs on. So open models on your own NVIDIA GPU can be the runtime's vendors with nothing but data: three model servers, and a row that points at them. No cloud vendor hears, thinks or speaks in the call, and no vendor bills for it.

How it works#

Every server speaks the OpenAI API, so the runtime reaches all of them through the openai plugin it already has, with a base_url per stage.

The servers run with Docker Compose on the machine that holds the card, outside the cluster. The cluster's pods reach them over a private network: the workers for a call's ears, model and voice, and the gateways for the embedder and the hang-up judges. Both files are in the runtime's infra/models/.

What you need#

• A runtime, on Kubernetes or on your laptop, and its operator key (PINECALL_OPS_KEY; on a cluster the secret pinecall-<env>-ops-key in Secret Manager).
• An NVIDIA card with 12 GB of memory or more. Everything loaded takes about 10 GB: Gemma about 8.5, the ears and Kokoro about 1 each. It was measured on 24 GB (an RTX 3090).
• About 25 GB of disk for the images and the models.
• The NVIDIA driver, Docker with the compose plugin and NVIDIA's container toolkit on that machine. nvidia-smi shows the card, and docker run --rm --gpus all ubuntu nvidia-smi shows it from a container.
• A private route from the cluster to the machine on ports 11434, 8880 and 8000. The repository's Terraform makes neither the GPU machine nor a firewall rule for it: make both the way your network is run, and admit the cluster's pod and node ranges alone. Never a public address: the servers ask for no key, so anyone who reaches them uses your GPU.

Set it up#

mkdir ~/pinecall-models && cd ~/pinecall-models
curl -fsSLO https://raw.githubusercontent.com/pinecall/runtime/main/infra/models/compose.yaml
curl -fsSLO https://raw.githubusercontent.com/pinecall/runtime/main/infra/models/providers.json
MODELS_BIND=10.0.0.5 docker compose up -d

H=10.0.0.5
curl -s http://$H:11434/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"gemma4:12b","reasoning_effort":"none","messages":[{"role":"user","content":"Di hola"}]}'
curl -s http://$H:8880/v1/audio/speech -H 'Content-Type: application/json' \
  -d '{"model":"kokoro","voice":"ef_dora","input":"Hola, ¿en qué te puedo ayudar?","response_format":"wav"}' -o hola.wav
curl -s http://$H:8000/v1/audio/transcriptions -F file=@hola.wav \
  -F model=deepdml/faster-whisper-large-v3-turbo-ct2 -F language=es
curl -s http://$H:11434/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"bge-m3","input":"hola"}' | head -c 120

curl -fsS -X PUT https://<your name>/v1/ops/providers \
  -H "Authorization: Bearer $PINECALL_OPS_KEY" -H 'Content-Type: application/json' --data @row.json

curl -fsS -X PUT https://<your name>/v1/ops/provider-keys/openai \
  -H "Authorization: Bearer $PINECALL_OPS_KEY" -H 'Content-Type: application/json' -d '{"key": "local"}'

The row's tuning sends every openai stage to your servers, and the placeholder replaces the runtime's own OpenAI key. A runtime that also runs OpenAI's models stops reaching them. And a new embedding leaves the facts and chunks written under the old one findable by their words only, until they are written again (pinecall-runtime memory reembed does it for memory).

A call#

The smallest agent that uses all three stages is one class with its prompt in the docstring:

import { Agent } from "@pinecall/agents";

/**
 * Eres la recepción de la peluquería Rulos, en Montevideo. Hablas de vos, con frases cortas.
 * Todo lo que dices se lee en voz alta: sin listas, sin markdown, los números como se dicen.
 * Abrimos de martes a sábado, de diez de la mañana a siete de la tarde. El corte cuesta ochocientos pesos.
 */
export default class Recepcion extends Agent {}

Run pinecall start in the project, open the agent in the console's sandbox and talk to it. pinecall sessions show <call> then shows what the ears heard, what the model said and how long each stage took.

What the row says#

When something does not come up#

What it measured#

An RTX 3090 holding the three servers, with the runtime on the same machine; a caller speaking five lines in Spanish, Smart Turn reading the end of each turn. Across a network, add its round trips.

Of those ~1.6 s, the ears close the turn about 0.55 s after the caller's last word, Gemma 4's first token takes about 0.7 s, and the voice's first audio 0.1–0.3 s.

What it does not do yet#

• A tool the model must call. Ollama does not honour tool_choice: "required". The spoken caller of pinecall simulate --voice demands one and fails on Gemma; so does each model judge, which then gives no verdict. An agent's own tools are called when the model chooses to. A server that constrains its output to the call (llama.cpp's llama-server --jinja, or vLLM) fixes it; only the row's base_url changes.
• Two calls at once share one Ollama, which answers one request at a time per model by default. For more than a demo, serve the model with vLLM, which also speaks the OpenAI API.

Next steps#