Skip to main content
Evaluations for voice simulation define the calls you want to test—a simulated persona speaks with your voice agent over a real audio channel while Netra records, transcribes, and evaluates the conversation.

​
Why Voice Simulation Evaluations Matter

Voice evaluations test qualities that text simulations cannot reach:

​
Creating a Voice Evaluation

Open Evaluate → Evaluations, click New Evaluation, and choose: Voice evaluation creation wizard
  • Turn Type: Multi Turn Simulation
  • Simulation Type: Voice

Add Manually

Build voice simulation scenarios from scratch, one scenario at a time

Upload CSV

Import a spreadsheet of voice simulation scenarios in bulk

Create using AI

Generate test cases using AI based on your description
1

Configure Basics

Provide an evaluation name and optional tags, then select Multi Turn Simulation as the turn type and Voice as the simulation type.
2

Set Up Voice Scenario

Describe each call you want to simulate:Then configure the voice pipeline your agent uses during the call:
Choose an STT provider that matches your callers’ accents and languages. You can mix providers across stages—each stage is configured independently.
3

Set Caller Behaviour

Behaviour Instructions — Describe how the simulated caller should behave: what they open with, how they react, and what they do if the agent struggles.
Information Available to the Simulated User — Enter the context data the caller can reference as a Table, as JSON, or as Plain Text.
Background Noise — Choose a background noise clip (cafe, street, office, and more) and set its intensity on a 0-1 scale. Use this to test how well your pipeline copes with realistic environments.
JSON Validation: When using JSON format, ensure there are no duplicate keys. The system validates JSON structure before allowing you to proceed.
4

Select a Persona

Pick the caller persona from your Personas library. Every scenario requires exactly one persona—the persona defines the caller’s voice, speech traits, accent, and language.
Create and manage personas under Library → Personas before building voice evaluations.
5

Select Evaluators

Select evaluators from the library. Alongside the TTS, STT, and Conversational evaluators, voice evaluations can score:
  • TTS quality: Humanness, Pronunciation Accuracy, Language Accuracy, Speaking Rate / Pace
  • STT accuracy: Word Error Rate, Transcription Correctness, Time to First Transcript
  • Conversational delivery: Backchannelling Naturalness, Expressiveness, Speaking Percentage
For each evaluator, rename it, choose the judge provider and model, adjust pass criteria, and fill any configurable variables—such as expected languages, words-per-second range, TTFT limit, or speaking-percentage range.Review all configurations, then click Create Evaluation to finalize.

​
Running a Voice Simulation

Open your evaluation, go to the Test Runs tab, and click Run Test Suite. Select a registered voice agent—your LiveKit-connected application—and start the run. Netra places simulated calls to your agent for every scenario. Run voice simulation modal You can also trigger simulations from the SDK. See the SDK reference for Python and TypeScript. Results appear in Test Runs with full transcripts and synced audio playback.

​
Managing an Evaluation

Open any voice evaluation to manage it: Voice evaluation detail page

​
Best Practices

​
Designing Scenarios

  • One goal per scenario: Keep each call focused so results are easy to interpret
  • Vary difficulty: Mix straightforward requests with ambiguous or multi-part ones
  • Test boundaries: Include scenarios your agent should decline or escalate

​
Tuning Realism

  • Start quiet: Run first passes without background noise, then add it once the basics work
  • Match your users: Choose personas whose accent, pace, and language resemble real callers
  • Bound the cost: Cap max duration and turns to keep runs predictable
Last modified on August 28, 2026