Quick Start: Simulation
New to simulations? Get your first simulation running in minutes.
Why Simulation Matters
Traditional testing falls short for conversational agents. Simulations provide a comprehensive way to test multi-turn interactions with realistic user behaviors:| Question | What Netra Simulates |
|---|---|
| Does my agent handle multi-turn conversations correctly? | Full conversation flows with simulated user responses |
| Can my agent achieve specific goals? | Goal-oriented scenarios with success/failure tracking |
| Does my agent respect constraints and limitations? | guideline adherence across conversation turns |
| How does my agent perform with different user personas? | Frustrated, confused, friendly, or neutral users |
Core Building Blocks
The Simulation suite is built on four interconnected pillars:Agents
Agents define the AI system you want to test, including its capabilities and limitations.| Feature | Description |
|---|---|
| Name | Unique identifier for your agent |
| Abilities | What your agent can do—roles, capabilities, tools, and knowledge base access |
| Constraints | What your agent should avoid—security restrictions, escalation criteria, and boundaries |
Datasets
Datasets are collections of simulation scenarios that define multi-turn conversation goals.| Feature | Description |
|---|---|
| Multi-Turn Scenarios | Define conversation goals with simulated user interactions |
| User Personas | Choose from neutral, friendly, frustrated, confused, or custom personas |
| User Data & Facts | Provide context data and facts the agent must communicate correctly |
| Variable Mapping | Map evaluator inputs to scenario fields, agent responses, or conversation metadata |
Evaluators
Evaluators assess your agent’s performance during simulations. Netra offers two evaluation scopes: Session-Level Evaluation Evaluate the entire conversation for goal achievement, fact accuracy, and overall performance.Netra provides a Library of preconfigured evaluators covering Quality, Performance, Agentic behavior, and Guardrails. Customize any evaluator and save it to My Evaluators for reuse across datasets.
Test Runs
Test Runs execute your simulation scenarios, providing detailed conversation transcripts and evaluation results.| Feature | Description |
|---|---|
| Conversation Transcript | Full multi-turn dialogue between simulated user and agent |
| Scenario Details | View goal, persona, user data, and fact checker configuration |
| Trace Integration | Link directly to execution traces for each turn to debug issues |
| Aggregated Metrics | View total cost, average latency, and pass/fail rates across the simulation |
Use Cases
Goal Achievement Testing
Validate that your agent can successfully complete user objectives:- Create scenarios with specific goals (e.g., “Get a refund from customer support”)
- Define what facts the agent must communicate
- Run simulations and verify goal achievement across different personas
- Analyze conversation transcripts to understand failure points
Persona-Based Testing
Test agent performance with different user types:- Create datasets with various personas (frustrated, confused, friendly)
- Run the same scenario across all personas
- Compare results to identify which personas your agent handles poorly
- Refine agent abilities and constraints based on insights
Constraint Validation
Ensure your agent respects boundaries and limitations:- Define agent constraints (e.g., “Don’t provide medical advice”)
- Create scenarios that test these boundaries
- Use evaluators to verify guideline adherence
- Track violations across multiple conversation turns
Getting Started
Create an Agent
Start by creating an agent with its abilities and constraints defined.
Create a Dataset
Build a multi-turn dataset with simulation scenarios, user personas, and facts to verify.
Configure Evaluators
Add evaluators to define your scoring criteria—choose from the library or create custom ones.
Run Simulations
Execute your dataset and view conversation transcripts and results in Test Runs.
Related
- Agents - Define agents with abilities and constraints
- Datasets - Create multi-turn simulation scenarios
- Evaluators - Configure scoring logic and criteria
- Test Runs - Analyze simulation results and conversation transcripts
- Traces - Understand how simulations connect to trace data