Test Runs for simulation show the execution results of your multi-turn datasets. Each run provides a complete conversation transcript between the simulated user and your agent, along with evaluation results, scenario details, and performance metrics.
Why Simulation Test Runs Matter
Simulation test runs provide deep insights into conversational agent performance:
| Capability | Benefit |
|---|
| Conversation Transcripts | See the full multi-turn dialogue to understand how your agent performed |
| Scenario Details | View goal, persona, user data, and fact checker configuration |
| Turn-by-Turn Tracing | Jump directly to execution traces for each conversation turn |
| Evaluation Results | Review turn-level and session-level evaluator scores |
| Exit Reason Tracking | Understand why conversations ended (goal achieved, failed, abandoned, max turns) |
| Aggregated Metrics | Monitor cost, latency, and success rates across simulations |
Test Runs Dashboard
Navigate to Evaluation → Test Runs from the left navigation panel to see simulation test runs.
| Column | Description |
|---|
| Name | Name of the test run |
| Type | Multi-turn for simulation test runs |
| Started At | Timestamp when the simulation began |
| Status | Current state: Completed, In Progress, or Failed |
| Dataset | The dataset used for this simulation |
Filtering and Search
- Date Range: Filter runs by time period to compare performance over time
- Search: Find specific test runs by agent or dataset name
- Sort: Order by date, status, or dataset
Viewing Test Run Details
Click on any simulation test run to access detailed results.
Summary Metrics
The top of the detail view shows aggregated performance data:
| Metric | Description |
|---|
| Total Items | Number of scenarios run in this test |
| Passed Items | Count of scenarios that achieved their goals |
| Failed Items | Count of scenarios that did not achieve goals |
| Total Cost | Aggregate token/API cost for all scenarios |
| Total Duration | End-to-end time for the simulation run |
| Average Latency | Mean response time across all turns |
Viewing Scenario Details
Click on any test run item to view the detailed scenario results. This opens a modal with three tabs.
Tab 1: Conversation
The Conversation tab shows the full multi-turn dialogue between the simulated user and your agent.
Features:
- Turn-by-Turn Display: Each conversation turn is clearly separated
- User Messages: Shows what the simulated user said
- Agent Responses: Shows what your agent replied
- Trace Links: Click View Trace on any turn to see detailed execution traces
- Turn Index: Track which turn number you’re viewing (Turn 1, Turn 2, etc.)
- Exit Reason: Shows why the conversation ended
Exit Reasons:
| Exit Reason | Description |
|---|
| Goal Achieved | The scenario objective was successfully completed |
| Goal Failed | The objective could not be achieved |
| Abandoned | The simulated user gave up or stopped engaging |
| Max Turns Reached | Hit the turn limit before goal completion |
Use the View Trace link to debug specific turns where the agent’s response
was unexpected or incorrect. Traces show the full LLM call, tool usage, and
latency breakdown.
Tab 2: Evaluation Results
The Evaluation Results tab shows scores from all configured evaluators.
Each evaluator produces a normalized score between 0 and 1. Scores at or above 0.6 pass; scores below 0.6 fail.
Example Results:
| Evaluator | Score | Pass/Fail |
|---|
| Goal Fulfillment | 1 | Pass |
| Factual Accuracy | 1 | Pass |
| Conversation Completeness | 1 | Pass |
| Profile Utilization | 0.75 | Pass |
| Guideline Adherence | 0.5 | Fail |
Tab 3: Scenario Details
The Scenario Details tab shows the complete configuration used for this simulation.
Scenario Section:
| Field | Value |
|---|
| Goal | The scenario objective (e.g., “Get a refund for a damaged product”) |
| Max Turns | Maximum turns allowed (e.g., 5) |
| User Persona | The persona used (e.g., Frustrated 😤) |
{
"order id": "3",
"product name": "laptop stand"
}
{
"item usage": "unused",
"refund window": "7 days",
"days since delivery": "28"
}
| Field | Value |
|---|
| Provider | The LLM provider used (e.g., openai) |
| Model | The model used for simulation (e.g., gpt-4.1) |
The Scenario Details tab is crucial for understanding the context of each
simulation. It shows exactly what data the simulated user had access to and
what facts the agent was expected to communicate.
Analyzing Simulation Results
Identifying Patterns
When reviewing simulation test runs, look for:
- Goal achievement rates: What percentage of simulations achieved their goals?
- Persona differences: Does your agent perform better with certain personas? Run the same scenarios with all persona types and compare results.
- Turn efficiency: Are conversations longer than necessary? Compare turn counts for successful vs failed scenarios.
- Common failure points: Which turns typically cause issues?
- Fact accuracy: Are specific facts consistently missed?
- Cost trends: Monitor total cost across test runs and identify scenarios that consume excessive turns.
Debugging Failed Simulations
For each failed scenario:
- Review the Conversation tab: Identify where the conversation went wrong
- Check the Evaluation Results tab: See which evaluators failed and why
- Examine the Scenario Details tab: Verify the user data and facts were correct
- Click View Trace: Inspect the full execution flow for problematic turns — check LLM inputs, tool calls, and latency breakdowns
Comparing Across Runs
To track improvement or regression:
- Run simulations after each agent update
- Compare goal achievement rates and evaluator scores across runs
- Investigate scenarios that changed from pass to fail
- Track turn efficiency and cost trends over time
Best Practices
- Test after every agent change: Run simulations when updating your agent to catch regressions early
- Create baseline runs: Establish performance benchmarks before making changes
- Always check traces for failures: Don’t just read the conversation — inspect the execution flow, LLM context, and tool calls
- Review latency: Identify slow turns that might frustrate real users
- Simulation Overview - Understand the full simulation framework
- Datasets - Create scenarios that generate test runs
- Evaluators - Configure scoring logic for simulations
- Traces - Debug simulation turns with execution traces
