Skip to main content
Run scenario-based behavioral evals. pipecat eval run tests scenarios, scripted or simulated, against an already-running agent; pipecat eval suite spawns the agents listed in a manifest and runs their scenarios concurrently. Both exit 0 when everything passes and 1 otherwise. If pipecat-ai[cli] is a dependency of your project, run these commands with uv run pipecat eval. They’re also available as python -m pipecat.evals. Before running, pipecat eval loads the nearest .env file, walking up from the working directory, so a simulation’s persona LLM and a hosted judge find their credentials the way your agent does. Variables already set in the shell win. See the Pipecat Evals guide for concepts, the scripted and simulated scenario formats, and manifests.

​
eval run

Run one or more scenarios against an already-running agent (started with -t eval). A simulation runs once here, whatever its runs: field says; a success rate over several runs is the suite’s job, since each run needs a fresh agent. Usage:
Arguments:
​
path
required
One or more scenario YAML files, scripted or simulated, or directories of them. A directory expands to its .yaml and .yml files in filename order, non-recursively, skipping !include fragments (files with no name:).
Options:
​
string
default:"ws://localhost:7860"
WebSocket URL of the agent’s eval transport.
​
flag
Print a line for each turn and expectation as it resolves. For a simulation, print the conversation as it happens and the judge’s reasons at the end.
​
flag
Record each scenario’s conversation audio (audio-mode scenarios).
​
string
default:"recordings"
Directory for --audio recordings: <record-dir>/<scenario>.wav.
​
string
Directory for cached synthesized user audio. Defaults to <user-cache-dir>/pipecat/tts.
​
flag
Disable the user-audio cache: re-synthesize every turn (no reads or writes).
​
integer
default:"60"
Default per-expectation timeout in seconds, for scripted expectations without their own within_ms. A simulation is bounded by its own max_turns, max_duration_s, and max_silence_s instead.
​
string
default:"."
Directory for each scenario’s logs: <logs-dir>/<scenario>.eval.log (plus .debug.log under --debug).
​
flag
Also save <scenario>.debug.log with the harness’s full per-pipeline logs.
​
flag
Cancel the agent’s pipeline (exit it) after the run. By default the agent is left running so it can serve more scenarios.
​
flag
Fire the bot’s on_client_disconnect callback when the eval client disconnects. Bots often cancel their pipeline there, so it’s off by default. A scenario’s trigger_disconnect: field opts in on its own.

​
eval suite

Spawn the agents in a manifest and run their scenarios concurrently. A manifest lists scripted and simulated scenarios side by side, and the file says which it is. A simulation runs as many times as its runs: field says, and every run must pass. Everything except the suite: list can be set in the manifest or overridden on the command line (the command line wins). Usage:
Arguments:
​
path
required
Manifest YAML listing agents and their scenarios.
Options:
​
string
Only run bots whose path contains this substring.
​
string
Only run this scenario name.
​
string
Only run scenarios of this kind: script or simulation.
​
string
Run subdirectory name under runs_dir. Defaults to a timestamp.
​
path
Output base, overriding the manifest’s runs_dir. A <name>/ subdirectory with logs/ and recordings/ is created under it. Defaults to eval-runs.
​
path
Override the manifest’s bots_dir (bot paths are relative to it).
​
path
Override the manifest’s scenarios_dir.
​
integer
Override the manifest’s concurrency (how many runs execute at once).
​
integer
Override the manifest’s repeat, how many times to run each (bot, scenario) pair, to measure flakiness. Must be at least 1. A repeat set here or in the manifest also runs each simulation this many times instead of its own runs: (1 included). A repeated sweep reports a pass rate per pair and always exits 0, since a rate is neither a pass nor a fail.
​
integer
Override the manifest’s base_port (default 7900). Each run gets `base_port
  • index`.
​
string
Override the manifest’s cache_dir for cached synthesized user audio.
​
flag
Disable the user-audio cache: re-synthesize every turn (no reads or writes).
​
integer
default:"60"
Default per-expectation timeout in seconds, for scripted expectations without their own within_ms.
​
string
Override the manifest’s spawn template. Default: "{python} {bot} -t eval --port {port}".
​
string
Override the Python interpreter used to spawn each agent.
​
flag
Record conversation audio.
​
flag
Also save <run>.debug.log with the harness’s full per-pipeline logs.

​
Examples