Simulations

Simulations define the scenarios you want to test—multi-turn conversations with specific goals, user personas, and success criteria. Unlike single-turn evaluations, simulations create dynamic, goal-oriented conversations that reflect real-world user interactions.

Why Simulations Matter

Simulations transform simple Q&A testing into realistic conversation testing:

Benefit	Description
Goal-Oriented Testing	Test whether your agent achieves specific objectives, not just individual responses
Persona-Based Scenarios	Simulate different user types—frustrated, confused, friendly, or neutral
Multi-Turn Conversations	Test how your agent handles back-and-forth dialogue (1-50 turns)
Context Simulation	Provide user data and context for realistic scenario execution

Simulation Dashboard

Navigate to Evaluate → Evaluations from the left navigation panel to access your evaluations. Filter by Multi turn type to focus on simulations, and switch between card and list views from the page toolbar. Each text simulation shows a text type icon with a Multi-turn badge at the bottom-right. Text simulations always use Multi Turn Simulation.

In card view, each card shows the evaluation name, type and turn type (Text · Multi-turn), tags, created date, a Run action, and a menu with Duplicate, Edit, and Delete.

In list view, columns are:

Column	Description
Name	Simulation name, with the text icon and Multi-turn badge
Turn Type	Always Multi-turn for text simulations
Tags	Metadata labels for filtering and organization
Created	Timestamp for version tracking
Actions	Run, Edit, Clone, and Delete

Creating a Multi-Turn Simulation

Click New Evaluation in the top right corner of the Evaluations page.

Add Manually

Build simulation scenarios from scratch, one scenario at a time

Generate with AI

Draft scenarios from a prompt or an existing agent

The creation wizard has three steps: Setup → Scenario → Evaluators. The evaluation name and tags are always editable in the header bar at the top.

Setup

Field	Description
Evaluation name	A descriptive identifier for your simulation suite (e.g., “Customer Refund Scenarios”)
Type of evaluation	Select Text from the type selector
Conversation style	Select Multi Turn Simulation for full conversation flow

Tags can be added at any point via the header bar at the top of the wizard.

To simulate phone or voice calls instead, select the Voice type. See Voice Simulations.

Set Up Scenario

On the Scenario step, add scenarios in either way:

Type them in — fill each scenario card yourself
Generate with AI — click Generate scenarios with AI under the cards (see Generate Scenarios with AI)

Each scenario card has three sections:01 Setup

Field	Description
Scenario name	A short label for the scenario (e.g., “Customer refund request for defective product”) (required)
Max conversation turns	Maximum number of conversation turns (1–50)

Lower (1-10): Quick interactions and single-question support
Medium (11-25): Standard support conversations
Higher (26-50): Complex, multi-step problem resolution

The simulation stops when the goal is achieved, the max turns limit is reached, or the scenario is abandoned.02 Your agentDescribe the scenario you’d like to test — Describe what the simulated user is trying to achieve. This becomes the goal that drives the conversation.

A customer contacts support to return a refund for a defective product.
The agent should verify the order, confirm eligibility, and process the
refund while maintaining a helpful tone.

03 The simulated userBehaviour instructions — Describe how the simulated user should behave during the conversation.

The user starts by asking to get a refund, asks clarifying questions if
confused, expresses frustration if the issue is not resolved quickly.

Persona — Choose how the simulated user behaves emotionally:

Persona	Description
Neutral	Straightforward and factual, sticks to the point
Friendly	Polite and cooperative, patient with the agent
Frustrated	Impatient, wants quick resolution, may be curt
Confused	Needs extra clarification, asks follow-up questions
Custom	Define your own persona behavior in free text

Information available to the simulated user — Provide the context data the simulated user can reference during the conversation. Enter it as a Table, as JSON, or as Plain Text.Example (Table):

Key	Value
order_number	ORD-123456
purchase_date	2024-01-15
product_name	Wireless Headphones
order_total	$129.99
shipping_address	123 Main St, New York, NY

Example (JSON):

{
  "order_number": "ORD-123456",
  "purchase_date": "2024-01-15",
  "product_name": "Wireless Headphones",
  "order_total": "$129.99",
  "shipping_address": "123 Main St, New York, NY"
}

File Attachments — Attach up to five files per scenario for the simulated user to reference. Supported formats are PDF, DOCX, PNG, JPG/JPEG, and CSV, with a maximum size of 5 MB per file. Each attachment requires a short description.Simulation provider and model — Select the provider and model that will play the simulated user (e.g., OpenAI / GPT-4o). This is required for each scenario. AI-generated drafts leave this blank — you must set it before creating.

Use consistent, capable models for realistic user simulation.

JSON Validation: When using JSON format, ensure there are no duplicate keys. The system validates JSON structure before allowing you to proceed.

Click + Add Scenario to add more scenarios to the same evaluation.

Configure Evaluators

Click Next to reach the Evaluators step. This is a split view with the evaluator library on the left and your Scoring Pipeline on the right.

Click on any evaluator card in the library, or drag it into the pipeline, to add it
Expand an evaluator in the pipeline to configure its judge model, pass criteria, and any configurable variables

If you used Generate scenarios with AI with Auto evaluation enabled, suggested evaluators may already be pre-filled. Review them before continuing.

Select evaluators from Netra’s library of session-level evaluators:Quality Evaluators:

Conversational Flow — Evaluates the natural flow of conversation
Conversation Completeness — Checks if the conversation reaches a complete resolution
Conversation Memory — Tests if the agent remembers context across turns
Factual Accuracy — Verifies the agent communicates correct information
Guideline Adherence — Checks if the agent follows specified guidelines
Speakability — Evaluates how natural the agent’s responses sound

Conversational Evaluators:

Backchannelling Naturalness — Tests natural backchanneling responses
Expressiveness — Evaluates emotional expressiveness in responses

For each evaluator, rename it, choose the judge provider and model, adjust pass criteria, and fill any configurable variables:

Guideline Adherence: your agent’s instructions and constraints
Factual Accuracy: the reference facts your agent must communicate correctly

Create Evaluation

Two buttons appear in the footer:

Create Evaluation — saves the evaluation and opens the detail page
Save and Run — saves the evaluation and immediately opens agent selection to trigger a test run

Generate Scenarios with AI

Use AI to draft scenarios from a free-text description or an existing agent. Drafts appear as scenario cards for review — nothing is saved until you create the evaluation.

Open the generate panel

On the Scenario step, click Generate scenarios with AI below the scenario cards.

Choose context

Pick one context mode:

Mode	When to use	What you provide
Prompt	One-off suites or agents not yet registered	A description of at least 30 characters. Optionally attach up to 5 context files (PDF, DOCX, or CSV, max 5 MB each)
Agent	Agent already exists in Netra	Select the agent (it must have a main purpose set). Optionally add short extra instructions for this run

Set count and Auto evaluation

Choose 1–10 scenarios. Leave Auto evaluation on to suggest evaluators for the generated batch, or turn it off if you only want scenarios.

Generate and review

From the success toast you can:

Retry — run the same request again
Add more with AI — open the panel to generate additional scenarios
Dismiss — return to the dock

Each generated card pre-fills: scenario name, goal, persona, behaviour instructions, and max turns. The provider and model fields are left blank — you must fill those in before creating.Edit any field before continuing.

Confirm evaluators and create

If you left Auto evaluation on, the evaluators step arrives with suggested evaluators already selected — review and adjust as needed.Click Create Evaluation to save or Save and Run to immediately trigger a test run.

Running a Simulation

Once your simulation is configured, you can run simulations from the dashboard or the SDK.

From the Dashboard

Open Run Test Suite

Open your simulation, go to the Test Runs tab, and click Run Test Suite.

Select an Agent and Name the Run

Pick a registered agent to converse with the simulated users, give the run a name, and start it. Netra runs every scenario and evaluates each conversation.

View Results

Monitor progress and results in Test Runs.

From the SDK

Use the SDK’s simulation methods with your Simulation ID to trigger simulations from your own code. See the SDK reference for Python and TypeScript.

Managing a Simulation

Open any simulation to manage it:

Tab	Description
Details	Browse scenarios as cards showing goal, persona, behavior, and turn limits. Add more scenarios at any time.
Analytics	Evaluation analytics for this simulation—overall health, average score over time, and per-evaluator pass/fail rates
Test Runs	Runs triggered from this simulation, with the Run Test Suite action

You can also clone scenarios or the entire simulation to create variants.

Best Practices

Crafting Effective Scenarios

Be specific: “Get a refund for a damaged product” is better than “Ask about returns”
Include context: Provide enough detail for realistic simulation (order details, timeline, issue description)
Include edge cases: Create scenarios that challenge your agent’s boundaries
Use AI as a draft, then edit: Generate with AI to bootstrap coverage, then refine goals, personas, and provider settings before saving

Choosing User Personas

Neutral: Best for baseline performance testing
Friendly: Tests whether your agent maintains professionalism even when not challenged
Frustrated: Critical for customer support agents—tests patience and de-escalation
Confused: Tests clarity and explanation quality
Custom: Use for industry-specific personas (technical users, non-native speakers, etc.)

Defining User Data

Provide realistic data: Use representative order numbers, dates, and values
Include edge cases: Test with missing fields, unusual values, or conflicting data
Keep it relevant: Only include data that matters for the scenario
Use consistent formats: Standardize date formats, currency, and naming

Verifying Facts

To check that your agent communicates critical information correctly:

Attach the Factual Accuracy evaluator and provide the must-communicate facts as its Reference Facts variable
Focus on critical facts: What MUST the agent get right?
Be precise: “5-7 business days” is better than “about a week”

Simulation Overview - Understand the full simulation framework
Evaluators - Configure scoring logic for simulations
Test Runs - View simulation results and conversation transcripts
Traces - Debug simulation turns with execution traces

Get Started

Observability

Insights

Evaluations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Simulations

Why Simulations Matter

Simulation Dashboard

Creating a Multi-Turn Simulation

Add Manually

Generate with AI

Generate Scenarios with AI

Running a Simulation

From the Dashboard

From the SDK

Managing a Simulation

Best Practices

Crafting Effective Scenarios

Choosing User Personas

Defining User Data

Verifying Facts

​Why Simulations Matter

​Simulation Dashboard

​Creating a Multi-Turn Simulation

Add Manually

Generate with AI

​Generate Scenarios with AI

​Running a Simulation

​From the Dashboard

​From the SDK

​Managing a Simulation

​Best Practices

​Crafting Effective Scenarios

​Choosing User Personas

​Defining User Data

​Verifying Facts

​Related

Why Simulations Matter

Simulation Dashboard

Creating a Multi-Turn Simulation

Generate Scenarios with AI

Running a Simulation

From the Dashboard

From the SDK

Managing a Simulation

Best Practices

Crafting Effective Scenarios

Choosing User Personas

Defining User Data

Verifying Facts

Related