Evaluators
Online Evaluations work alongside Evaluators. Learn how to build custom evaluators for your use cases.
Why Online Evaluations Matter
Without continuous production monitoring, issues compound silently. Online Evaluations help you answer critical questions in real-time:| Question | What Online Evaluations Measure |
|---|---|
| Is my agent producing quality outputs in production? | Real-time pass rates across all configured evaluators |
| Are there regressions after deploying a new agent version? | Continuous score tracking and trend analysis over time |
| Which evaluators are flagging the most failures? | Per-evaluator breakdown with pass/fail rates |
| Am I meeting quality SLAs for my AI product? | Configurable alert thresholds with automatic notifications |
| What does my evaluation cost look like over time? | Total evaluation cost tracking across all configured evaluators |
How It Works
Online Evaluations run as a background process that continuously analyses all traces coming into your project. When configured for a specific agent (or all agents), the system:- Intercepts incoming traces — Every trace logged to the project is checked against the online evaluation’s scope and filters
- Applies sampling — Based on your configured sampling rate (1–100%), traces are selected for evaluation
- Runs evaluators — Each selected trace is scored against all configured evaluators (LLM-as-Judge, latency, cost, regex, etc.)
- Records results — Evaluation scores, pass/fail status, and reasoning are stored per trace
- Triggers alerts — If configured thresholds are breached, alerts fire to your contact points
Online Evaluations operate at the trace scope — each incoming trace is treated as a single evaluation unit scored against your configured evaluators.
Online Evaluations List Page
When you navigate to Online Evaluations, you see a table listing all configured online evaluations for the current project.| Column | Description |
|---|---|
| Name | The name assigned to the online evaluation |
| Total Evaluations | Total number of trace evaluation results recorded |
| No. of Evaluators | Number of evaluators configured for this online eval |
| Status | Active (running) or Inactive (paused) |
| Actions | Edit and Delete buttons |
- Search — Filter evaluations by name
- Refresh — Reload the list to see the latest data
- Create button — Opens the configuration modal to create a new online evaluation
Online Evaluation Detail View
On clicking into a specific online evaluation, you see its detail page with three tabs and an active/inactive toggle in the top right.Active / Inactive Toggle
The toggle at the top right controls whether the online evaluation is currently running. When Active, it continuously evaluates incoming traces. When Inactive, no new traces are evaluated but historical results remain accessible.Overview Tab
The Overview tab is a time-filtered analytics dashboard that provides a high-level picture of how your evaluation is performing.
| Metric | Description |
|---|---|
| Traces Evaluated | Total number of traces that have been evaluated in the selected time range |
| Overall Pass Rate | Percentage of pass judgments across all evaluators, pooled together |
| Average Evaluator Score | Mean numeric score across all evaluators for the selected period |
| Total Evaluation Cost | Estimated LLM cost incurred for running the evaluators |
- Evaluator Scores — A combined line chart showing per-evaluator pass rates over time, with a white average score line overlaid
- Traces Evaluated — A bar chart showing evaluation volume over time
- Online Evaluation Cost — A line chart tracking cost over time
Results Tab
The Results tab provides granular, trace-level evaluation data split into two sections:
| Column | Data |
|---|---|
| Evaluator | Evaluator name |
| Type | Evaluator type (e.g., LLM-as-Judge) |
| Traces Evaluated | Number of traces scored by this evaluator |
| Pass Rate | Percentage of traces that passed |
| Fail Rate | Percentage of traces that failed |
- Timestamp — When the trace was evaluated
- Trace ID — Unique identifier (clickable to view full trace)
- Evaluator columns — One column per configured evaluator showing score, pass/fail status, and reasoning
Alerts Tab
The Alerts tab displays a history of all triggered alerts for this online evaluation.
| Column | Description |
|---|---|
| Triggered Time | When the alert was fired |
| Alert Details | Source (overall pass rate vs. per-evaluator) and detail message |
| Severity | CRITICAL or WARNING |
| Status | TRIGGERED or RESOLVED |
Configuring an Online Evaluation
Creating an online evaluation is a 4-step process:Rule Details (Step 1)
Define the scope and identity of your online evaluation:
| Field | Required | Description |
|---|---|---|
| Agent | No | Select a specific agent to evaluate, or leave as “All agents” to evaluate every trace in the project |
| Filters | No | Add trace filters to narrow which traces get evaluated (e.g., by metadata, tags, or attributes). A matching trace count preview shows how many traces matched in the last 24 hours |
| Name | Yes | A descriptive name for this online evaluation (e.g., “Production Quality Monitor”) |
| Description | No | Optional description explaining the purpose of this evaluation |
Evaluator Selection (Step 2)
Choose which evaluators will score your traces. You can select from two sources:
- My Evaluators — Custom evaluators you’ve created in your workspace
- Library — Pre-built evaluator templates grouped by category
| Category | What It Covers |
|---|---|
| Quality | Response accuracy, relevance, faithfulness to context, and hallucination detection |
| Multimodal | Image generation quality, format validation, and text-image alignment |
| Guardrails | Safety checks — bias detection, topic boundaries, and harmful content |
| Performance | Response time, cost, and token consumption thresholds |
| Semantic | Meaning-level comparison between outputs and references |
| Agentic | Whether the agent achieved the user’s stated goal |
| JSON Evaluator | Validates output JSON structure and field values |
| Regex Evaluator | Validates output format against a regex pattern |
| Tool Use | Verifies correct tool calls against expected tools |
You must select at least one evaluator. Only single-turn evaluators are available for online evaluations.
Evaluator Configuration (Step 3)
Configure each selected evaluator with its specific settings:\
- Select the provider and model (e.g., OpenAI / gpt-4o-mini)
- Define evaluation criteria and pass criteria
- Map evaluator variables to trace data using JMESPath expressions
- Variables can source from trace attributes, span data, or literal values
- Numerical evaluators: Set a threshold with an operator (
>=,<=,>,<,=) - Boolean evaluators: Set pass condition (
is_true/is_false)
Sampling & Alerts (Step 4)
Configure how many traces to evaluate and optionally set up alert rules.
Use the slider or presets to set any value from 1–100%.Alerts (optional):Toggle alerts on to receive notifications when quality degrades. When enabled, configure:
Each alert rule supports:
| Preset | Description |
|---|---|
| 10% | Light sampling for high-volume production |
| 25% | Balanced sampling for moderate traffic |
| 50% | Substantial coverage |
| 75% | High coverage |
| 100% | Evaluate every matching trace (default) |
| Setting | Description |
|---|---|
| Contact Points | Select notification channels (email, Slack, etc.) |
| Re-evaluation Interval | How often to check alert conditions (15 min to 24 hours) |
| Overall Pass Rate Alerts | Trigger when overall pass rate crosses a threshold for a duration |
| Per-Evaluator Alerts | Set individual thresholds per evaluator |
- Severity — CRITICAL or WARNING
- Comparison type — is_above or is_below
- Threshold — The value that triggers the alert
- Duration — How long the condition must persist before firing
Editing an Online Evaluation
To edit an existing online evaluation:- Navigate to the Online Evaluations list page
- Click the Edit (pencil icon) button in the Actions column of the evaluation you want to modify
- The configuration modal opens pre-filled with the current settings
- Modify any settings across the 4 steps (Rule Details, Evaluator Selection, Evaluator Configuration, Sampling & Alerts)
- Click Update to save your changes
You can also toggle an online evaluation between Active and Inactive from the detail page without opening the edit modal.
Deleting an Online Evaluation
To delete an online evaluation:- Navigate to the Online Evaluations list page
- Click the Delete (trash icon) button in the Actions column
- A confirmation modal appears: “Are you sure you want to delete [name]? This action cannot be undone.”
- Click Delete to confirm
Best Practices
| Practice | Why It Matters |
|---|---|
| Start with 100% sampling, reduce as volume grows | Ensures full coverage during setup, then optimize for cost |
| Configure alerts with appropriate durations | Avoids noisy alerts from transient spikes |
| Use per-evaluator alerts for critical quality dimensions | Pinpoints exactly which quality aspect is degrading |
| Combine multiple evaluator types | Catches different failure modes (quality, latency, structure) |
| Filter by specific agents for targeted monitoring | Focuses evaluation budget on the most critical agents |
| Review the Results tab regularly | Identifies patterns that summary metrics might miss |
| Set meaningful pass criteria per evaluator | Defines what “good” means for your specific use case |
FAQ
What traces does an online evaluation analyse?
What traces does an online evaluation analyse?
An online evaluation analyses all incoming traces for the project that match the configured scope. If you select a specific agent, only traces from that agent are evaluated. If you add filters, only traces matching those filter conditions are evaluated. The sampling rate then determines what percentage of matching traces actually get scored.
Can I run multiple online evaluations on the same project?
Can I run multiple online evaluations on the same project?
Yes. You can configure multiple online evaluations, each with different agents, filters, evaluators, and sampling rates. Each operates independently.
What happens when I toggle an online evaluation to Inactive?
What happens when I toggle an online evaluation to Inactive?
When inactive, no new traces are evaluated. Historical results remain accessible in the Overview, Results, and Alerts tabs. Toggling back to Active resumes evaluation of new incoming traces.
How is the Overall Pass Rate calculated?
How is the Overall Pass Rate calculated?
The overall pass rate pools all pass/fail judgments across all evaluators for all evaluated traces in the selected time range. If a trace has 3 evaluators and passes 2, that contributes 2 passes and 1 failure to the aggregate.
What does sampling rate control?
What does sampling rate control?
Sampling rate determines the percentage of matching traces that are actually evaluated. At 100%, every trace is scored. At 25%, roughly one in four matching traces is selected. This helps manage evaluation costs at high traffic volumes.
How do alerts work?
How do alerts work?
Alerts check conditions at the configured re-evaluation interval (e.g., every 15 minutes). If the overall pass rate or a per-evaluator metric crosses the threshold for the specified duration, the alert fires and notifications are sent to your configured contact points. Alerts auto-resolve when conditions return to normal.
Can I use library evaluators?
Can I use library evaluators?
Yes. In Step 2 (Evaluator Selection), the Library tab shows pre-built evaluator templates. When you select a library evaluator, it is automatically saved to your workspace and linked to the online evaluation.
What is the evaluation cost?
What is the evaluation cost?
Evaluation cost reflects the estimated LLM cost for running evaluators (primarily LLM-as-Judge type). Non-LLM evaluators (latency, regex, JSON validation) have no cost. The Total Evaluation Cost metric in the Overview tab tracks this over time.
Related
- Evaluation Agents — Create custom evaluators for online and offline evaluation
- Traces — View and debug individual trace executions