Quick Start: Simulation
New to simulations? Get your first simulation running in minutes.
Why Simulation Matters
Traditional testing falls short for conversational agents. Simulations provide a comprehensive way to test multi-turn interactions with realistic user behaviors:| Question | What Netra Simulates |
|---|---|
| Does my agent handle multi-turn conversations correctly? | Full conversation flows with simulated user responses |
| Can my agent achieve specific goals? | Goal-oriented scenarios with success/failure tracking |
| How does my agent perform with different user personas? | Frustrated, confused, friendly, or neutral users |
Core Building Blocks
The Simulation suite is built on three interconnected pillars:Evaluators
Evaluators assess the entire conversation after it completes. Netra provides 8 preconfigured library evaluators in two categories:| Category | Evaluators |
|---|---|
| Quality | Guideline Adherence, Conversation Completeness, Profile Utilization, Conversational Flow, Conversation Memory, Factual Accuracy |
| Agentic | Goal Fulfillment, Information Elicitation |
Datasets
Datasets are collections of simulation scenarios that define multi-turn conversation goals.| Feature | Description |
|---|---|
| Multi-Turn Scenarios | Define conversation goals with simulated user interactions |
| User Personas | Choose from neutral, friendly, frustrated, confused, or custom personas |
| User Data & Facts | Provide context data and facts the agent must communicate correctly |
| Variable Mapping | Map evaluator inputs to scenario fields, agent responses, or conversation metadata |
Test Runs
Test Runs execute your simulation scenarios, providing detailed conversation transcripts and evaluation results.| Feature | Description |
|---|---|
| Conversation Transcript | Full multi-turn dialogue between simulated user and agent |
| Scenario Details | View goal, persona, user data, and fact checker configuration |
| Trace Integration | Link directly to execution traces for each turn to debug issues |
| Aggregated Metrics | View total cost, average latency, and pass/fail rates across the simulation |
Use Cases
Goal Achievement Testing
Validate that your agent can successfully complete user objectives:- Create scenarios with specific goals (e.g., “Get a refund from customer support”)
- Define what facts the agent must communicate
- Run simulations and verify goal achievement across different personas
- Analyze conversation transcripts to understand failure points
Persona-Based Testing
Test agent performance with different user types:- Create datasets with various personas (frustrated, confused, friendly)
- Run the same scenario across all personas
- Compare results to identify which personas your agent handles poorly
- Refine your agent based on insights
Getting Started
Configure Evaluators
Set up evaluators to define your scoring criteria — choose from the library or create custom ones.
Create a Dataset
Build a multi-turn dataset with simulation scenarios, user personas, and facts to verify.
Run Simulations
Execute your dataset and view conversation transcripts and results in Test Runs.
Related
- Evaluators - Configure scoring logic and criteria
- Datasets - Create multi-turn simulation scenarios
- Test Runs - Analyze simulation results and conversation transcripts
- Traces - Understand how simulations connect to trace data