Simulations

Evaluations for voice simulation define the calls you want to test—a simulated persona speaks with your voice agent over a real audio channel while Netra records, transcribes, and evaluates the conversation.

Why Voice Simulation Evaluations Matter

Voice evaluations test qualities that text simulations cannot reach:

Benefit	Description
Real Audio Pipeline	Exercise your actual STT, reasoning LLM, and TTS stack under realistic conditions
Natural Callers	Personas speak aloud with configurable pace, accent, and disfluencies
Noise Robustness	Inject background noise at a controlled level to test real-world resilience
Bounded Calls	Enforce max duration, silence timeouts, and turn limits per scenario

Voice Simulation Dashboard

Navigate to Evaluate → Evaluations from the left navigation panel to access your evaluations. Filter by Multi turn type to focus on simulations, and switch between card and list views from the page toolbar. Each voice simulation shows a voice type icon with a Multi-turn badge at the bottom-right. Voice evaluations always use Multi Turn Simulation.

In card view, each card shows the evaluation name, type and turn type (Voice · Multi-turn), tags, created date, a Run action, and a menu with Duplicate, Edit, and Delete.

In list view, columns are:

Column	Description
Name	Simulation name, with the voice icon and Multi-turn badge
Turn Type	Always Multi-turn for voice simulations
Tags	Metadata labels for filtering and organization
Created	Timestamp for version tracking
Actions	Run, Edit, Clone, and Delete

Creating a Voice Evaluation

Open Evaluate → Evaluations, click New Evaluation, and choose Voice as the evaluation type. Voice automatically sets the conversation style to Multi Turn Simulation.

The creation wizard has three steps: Setup → Scenario → Evaluators. The evaluation name and tags are always editable in the header bar at the top.

Setup

Click New Evaluation and configure the basics:

Field	Description
Evaluation name	A descriptive identifier for your voice test suite (required)
Type of evaluation	Select Voice (Beta). This automatically sets the conversation style to Multi Turn Simulation

Tags can be added at any point via the header bar at the top of the wizard.

Set Up Voice Scenario

On the Scenario step, add one or more voice scenario cards. Each card has three sections:01 Setup

Field	Description
Scenario name	A short label for the scenario (required)
Max conversation turns	Maximum number of conversation turns (1–50)
Max call duration	Maximum call duration in minutes (0.5–30 minutes)

02 Your agent

Field	Description
Scenario description	Describe what the caller is trying to accomplish — this becomes the goal that drives the call (required)

03 The callerVoice goal — Describe what should happen on this call (required). This tells the simulated caller what outcome to work towards.

Call about a damaged order and ask for a refund. If asked for
details you were not given, say you do not remember.

Information available to the simulated user — Enter the context data the caller can reference as a Table, as JSON, or as Plain Text.

{
  "order_number": "ORD-123456",
  "purchase_date": "2024-01-15",
  "issue": "arrived with a cracked screen"
}

Background noise — Optionally choose a background noise clip (cafe, street, office, and more) and set its intensity. Use this to test how well your pipeline copes with realistic environments.Persona — Pick the caller persona from your Personas library. Every scenario requires exactly one persona — the persona defines the caller’s voice, speech traits, accent, and language. You can search for an existing persona or create a new one inline.

Create and manage personas under Library → Personas before building voice evaluations, or create one inline from the scenario card.

Voice pipeline — Configure the providers and models for each stage of the call:

Stage	Required	Description
STT	Yes	Speech-to-text provider and model that transcribes the caller
Reasoning	Yes	The LLM provider and model powering your agent’s responses
TTS	No	Text-to-speech provider, model, and voice that speak your agent’s replies

Choose an STT provider that matches your callers’ accents and languages. You can mix providers across stages — each stage is configured independently.

JSON Validation: When using JSON format for known facts, ensure there are no duplicate keys. The system validates JSON structure before allowing you to proceed.

Click + Add Scenario to add more voice scenarios to the same evaluation.

Configure Evaluators

Click Next to reach the Evaluators step. This is a split view with the evaluator library on the left and your Scoring Pipeline on the right.

Click on any evaluator card in the library, or drag it into the pipeline, to add it
Expand an evaluator in the pipeline to configure its judge model, pass criteria, and any configurable variables

Voice evaluations can score:

TTS quality: Humanness, Pronunciation Accuracy, Language Accuracy, Speaking Rate / Pace
STT accuracy: Word Error Rate, Transcription Correctness, Time to First Transcript
Conversational delivery: Backchannelling Naturalness, Expressiveness, Speaking Percentage

For each evaluator, rename it, choose the judge provider and model, adjust pass criteria, and fill any configurable variables — such as expected languages, words-per-second range, TTFT limit, or speaking-percentage range.

Variable mappings are not shown for multi-turn evaluations — the library defaults handle mapping automatically.

Create Evaluation

On the final step two buttons appear in the footer:

Create Evaluation — saves the evaluation and opens the detail page
Save and Run — saves the evaluation and immediately opens agent selection to trigger a test run. Only voice-modality agents appear in the agent selector

Running a Voice Simulation

Open your evaluation, go to the Test Runs tab, and click Run Test Suite. Select a registered voice agent—your LiveKit-connected application—and start the run. Netra places simulated calls to your agent for every scenario.

Results appear in Test Runs with full transcripts and synced audio playback.

Managing an Evaluation

Open any voice evaluation to manage it:

Tab	Description
Details	Browse scenarios as cards showing goal, resolved persona name, behavior, turns, and duration. Add more scenarios at any time.
Analytics	Evaluation analytics for this evaluation—overall health, average score over time, and per-evaluator pass/fail rates
Test Runs	Runs triggered from this evaluation, with the Run Test Suite action

Best Practices

Designing Scenarios

One goal per scenario: Keep each call focused so results are easy to interpret
Vary difficulty: Mix straightforward requests with ambiguous or multi-part ones
Test boundaries: Include scenarios your agent should decline or escalate

Tuning Realism

Start quiet: Run first passes without background noise, then add it once the basics work
Match your users: Choose personas whose accent, pace, and language resemble real callers
Bound the cost: Cap max duration and turns to keep runs predictable

Simulation Overview - Understand the full simulation framework
Personas - Create reusable caller personas
Evaluators - Score audio quality, transcription, and delivery
Test Runs - View transcripts and synced call audio

Get Started

Observability

Insights

Evaluations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Simulations

Why Voice Simulation Evaluations Matter

Voice Simulation Dashboard

Creating a Voice Evaluation

Running a Voice Simulation

Managing an Evaluation

Best Practices

Designing Scenarios

Tuning Realism

​Why Voice Simulation Evaluations Matter

​Voice Simulation Dashboard

​Creating a Voice Evaluation

​Running a Voice Simulation

​Managing an Evaluation

​Best Practices

​Designing Scenarios

​Tuning Realism

​Related

Why Voice Simulation Evaluations Matter

Voice Simulation Dashboard

Creating a Voice Evaluation

Running a Voice Simulation

Managing an Evaluation

Best Practices

Designing Scenarios

Tuning Realism

Related