Pipecat Evals

name: capital_question

turns:
  - user: "What is the capital of Germany?"
    expect:
      - event: response
        eval: "the response says the capital of Germany is Berlin"

name: book_table

persona: |
  Jamie, booking dinner for two tonight at 6 PM. Gives a name and phone number
  when asked (Jamie Lee, 555-0142). Polite, answers one question at a time.
goal: "Book a table for two at 6 PM, then end the call."
success: "the bot confirmed a reservation for two at 6 PM"

metrics:
  - name: politeness
    criterion: "the reply is courteous, never curt or dismissive"
    min_score: 1

pipecat eval run capital_question.yaml book_table.yaml

​

Why evals matter

• Regression safety: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do.
• Fast iteration: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost.
• Semantic assertions: an LLM judge checks meaning (“the response says the capital is Berlin”), not exact strings, so tests don’t break when wording changes.
• Whole-flow coverage: a simulated caller pursues a goal in its own words, so you learn whether the agent finishes the job, not just whether one scripted path still works.
• A feedback signal for AI coding assistants: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See The Eval Loop.

​

Two kinds of scenario

​

How it works

1. The eval transport. Your agent runs unchanged with the eval transport. If your agent uses create_transport() and the development runner, this is already built in: start it with -t eval and it hosts a local WebSocket server speaking RTVI, instead of connecting to Daily, WebRTC, or telephony.
2. The eval harness. The harness connects to that transport as an RTVI client and plays the user’s side of the conversation: a scripted scenario’s turns, or a persona LLM’s replies. It collects the events your agent emits (transcriptions, LLM responses, spoken output, function calls, and timing) and either asserts on them in order, for a script, or hands the whole transcript to a judge, for a simulation.

​

Text and audio modes

​

What you can test

• Response content: substring checks (text_contains) or semantic judging (eval) of the agent’s replies.
• Multi-turn context: verify the agent remembers earlier turns.
• Function calling: assert that specific tools were called, with specific arguments, in a scripted turn or across a whole simulated conversation.
• Interruptions: barge in mid-response and verify the agent recovers (send_after).
• Latency: per-event budgets with within_ms, or the slowest reply of a simulation with the latency measure.
• Goals and outcomes: whether a simulated caller got what they came for, or was turned down correctly when they shouldn’t.
• Conversation quality: per-reply criteria such as politeness or brevity, scored across every reply of a simulation.
• Vision: serve an image when the agent requests one and judge its description.

​

YAML or Python

​

Requirements

• Pipecat CLI: the pipecat eval commands ship with the CLI extra: uv tool install "pipecat-ai[cli]". If you’ve added pipecat-ai[cli] to your project instead, run them with uv run pipecat eval (just like uv run bot.py). The same commands are also available as python -m pipecat.evals.
• A judge LLM (for eval: assertions, and for every simulation): Ollama by default (ollama pull gemma4:12b), or point the scenario’s judge: block at OpenAI or any OpenAI-compatible endpoint.
• A persona LLM (simulations only): by default the same local Ollama model as the judge, so a simulation needs no API key. A simulator: block can name another Ollama model, or any OpenAI-compatible model that supports function calling through a factory:.
• Audio services (audio mode only): the harness needs a TTS to synthesize the user’s voice and an STT to transcribe the agent’s speech. Both can be local models or HTTP-based services; the defaults are local (Kokoro and Moonshine or Whisper, installed with uv add "pipecat-ai[kokoro,moonshine]" or uv add "pipecat-ai[kokoro,whisper]"), which download once on first use and run with no keys and no per-run cost. WebSocket-streaming services aren’t supported here, which keeps the harness simple.
• Your agent’s own credentials: the agent under test is your real agent, so it needs the same service API keys it normally would.

​

Production evaluation

• Testing over the deployed path: test calls over API, WebSocket, or telephony, with varied caller populations, real phone-network conditions, and load, exercising the transport your users actually hit.
• Observability: continuous evaluation of live traffic, with automated quality scoring of calls and transcripts, audio-signal metrics, and trends tracked over time to catch quality drift.

​

Next steps