Skip to content
Voice Agent FleetRequest a free fleet review

Changed a voice agent prompt? Test it against real call scenarios before it goes live

This free guide is for teams running voice agents on Vapi, Retell or ElevenLabs. It shows how to build a test gate from each platform's own tests and where a dedicated testing tool helps.

Checked against the Vapi, Retell and ElevenLabs docs on 18 November 2026.

Request a free fleet reviewOr read how a safe rollout works

A test gate in five parts

A gate is a saved set of call scenarios, a pass rule written down in advance, and a rule that nothing is published until the rule is met.

  1. Build the scenario set from real calls

    The Vapi monitoring docs say to turn confirmed production failures into regression tests. Every call that went wrong becomes a scenario.

  2. Write the pass rule

    For example: every scenario passes, or a named share passes on repeated runs. ElevenLabs can repeat a test and report a pass rate. Decide the number before you see the results.

  3. Run it before the change reaches callers

    Use a draft, a branch or a separate org where the platform allows it. The next section shows what each platform offers.

  4. Publish only on a pass

    The docs of the three platforms describe running tests in CI. We found no built-in gate that blocks a publish, so the gate is a step in your own pipeline.

  5. Keep the result with the version

    Store the run next to the version it tested. When a call goes wrong later, you can see what was and was not covered.

Vapi: Simulations, Evals and Scorecards

Vapi Simulations test the latest published version, not the draft. That is the catch: on a live number, a version you publish to test also takes live calls.

  • Simulations run an AI tester through a real voice or chat conversation with an assistant or squad and score the outcome. Scenarios and suites are available through the API too.
  • The path to production page says a published version takes live calls right away, before you rerun your simulations. It also says that to test changes before they reach live calls, use config as code.
  • With GitOps, simulation suites can run on every pull request against the branch's own files. Nothing is deployed, tool calls get mocked results, and each run uses simulation minutes.
  • Evals run on mock conversations at the text and model layer, with exact match, regex or AI judging, and tool-call checks. The docs say they do not test speech recognition, audio quality or turn-taking.
  • Scorecards grade completed calls after the fact. The docs say they are available through the API only for now.

Retell: simulation tests and batch tests

Retell saves test cases and reruns them as a batch, with pass or fail and a written explanation for each. Simulation tests run as text conversations.

  • A test case holds a prompt for an AI-simulated caller, success criteria, dynamic variables and mocks for custom functions. The docs say saved cases become a regression suite to rerun after every prompt or flow change.
  • Simulation testing works with single-prompt, multi-prompt and Conversation Flow agents. Agents on a custom LLM are not supported.
  • A batch runs many cases together. The docs say the list shows batches against one agent version at a time, and that batches can be created and read through the API to fit CI.
  • A tool call that matches no mock falls through to the real tool, so mock every tool that writes data.
  • The docs we read do not say whether a batch can run against a draft, so create a draft and confirm that on one agent before you build the gate around it.

ElevenLabs: simulation, next-reply and tool-call tests

ElevenLabs tests run through the API or the command line against a branch, and the default is Main. That makes a branch the natural place to run the gate.

  • Simulation tests run a multi-turn conversation with a simulated user. Next-reply tests have a model judge the agent's next message. Tool-call tests check the tool and its parameters.
  • A test can be repeated 2 to 20 times, and the results are grouped as a pass rate.
  • The run-tests API takes an optional branch ID, and tests run on the main branch if you give none. The command line tool has a test command that, according to its README, exits non-zero on failure.

When a dedicated testing tool is worth adding

Native tests can be enough for a first gate on one platform. Dedicated voice-agent testing tools exist, for example Cekura, Hamming and Coval, and are worth a look in the cases below.

  • You run agents on more than one platform and want one scenario set that is not tied to a single platform's test format.
  • The native tests do not cover a scenario you care about, and you have a list of the gaps.
  • You have outgrown running the gate as a script in your own pipeline.

We name these tools for orientation only. This guide does not compare or rank them.

Sources

Official documentation and forum pages this guide relies on, as read on 18 November 2026. Platforms change fast, so check the page before you act.

Want someone to review how your fleet ships changes?

The free 30-minute fleet review call covers how your agents are versioned, how a change reaches live calls and how you would roll it back. We ask for no data and no access before the call.

Request a free fleet reviewOr read how a safe rollout works