Skip to main content

​
Online Evaluations

Continuously evaluate production traces in real-time by running configurable evaluators against incoming agent data — automatically and at scale.
AI agents in production don’t fail in predictable ways — they degrade silently, drift over time, and exhibit issues that only surface under real-world conditions. Netra’s Online Evaluations framework brings automated, continuous quality assessment directly into your production pipeline. Instead of manually reviewing traces, you define evaluation rules once, and every incoming trace that matches your criteria gets scored automatically.

Evaluators

Online Evaluations work alongside Evaluators. Learn how to build custom evaluators for your use cases.

​
Why Online Evaluations Matter

Without continuous production monitoring, issues compound silently. Online Evaluations help you answer critical questions in real-time:

​
How It Works

Online Evaluations run as a background process that continuously analyses all traces coming into your project. When configured for a specific agent (or all agents), the system:
  1. Intercepts incoming traces — Every trace logged to the project is checked against the online evaluation’s scope and filters
  2. Applies sampling — Based on your configured sampling rate (1–100%), traces are selected for evaluation
  3. Runs evaluators — Each selected trace is scored against all configured evaluators (LLM-as-Judge, latency, cost, regex, etc.)
  4. Records results — Evaluation scores, pass/fail status, and reasoning are stored per trace
  5. Triggers alerts — If configured thresholds are breached, alerts fire to your contact points
Online Evaluations operate at the trace scope — each incoming trace is treated as a single evaluation unit scored against your configured evaluators.

​
Online Evaluations List Page

When you navigate to Online Evaluations, you see a table listing all configured online evaluations for the current project. The page also provides:
  • Search — Filter evaluations by name
  • Refresh — Reload the list to see the latest data
  • Create button — Opens the configuration modal to create a new online evaluation
Clicking any row navigates to the detailed view for that online evaluation.

​
Online Evaluation Detail View

On clicking into a specific online evaluation, you see its detail page with three tabs and an active/inactive toggle in the top right.

​
Active / Inactive Toggle

The toggle at the top right controls whether the online evaluation is currently running. When Active, it continuously evaluates incoming traces. When Inactive, no new traces are evaluated but historical results remain accessible.

​
Overview Tab

The Overview tab is a time-filtered analytics dashboard that provides a high-level picture of how your evaluation is performing. Summary Cards: Charts:
  • Evaluator Scores — A combined line chart showing per-evaluator pass rates over time, with a white average score line overlaid
  • Traces Evaluated — A bar chart showing evaluation volume over time
  • Online Evaluation Cost — A line chart tracking cost over time
All charts respect the selected time range and automatically group data by minute, hour, or day depending on the range.

​
Results Tab

The Results tab provides granular, trace-level evaluation data split into two sections: Evaluator Breakdown Table: Evaluation Results Table: A detailed per-trace results table with:
  • Timestamp — When the trace was evaluated
  • Trace ID — Unique identifier (clickable to view full trace)
  • Evaluator columns — One column per configured evaluator showing score, pass/fail status, and reasoning
Clicking a row opens a sidebar with full evaluation details for that trace, including per-evaluator scores, reasoning, pass criteria, and a link to view the complete trace.

​
Alerts Tab

The Alerts tab displays a history of all triggered alerts for this online evaluation. The table is sortable by Triggered Time, Severity, and Status.

​
Configuring an Online Evaluation

Creating an online evaluation is a 4-step process:
1

Rule Details (Step 1)

Define the scope and identity of your online evaluation:
If you add filters, all filter rows must be complete (field, operator, and value) before proceeding.
2

Evaluator Selection (Step 2)

Choose which evaluators will score your traces. You can select from two sources:
  • My Evaluators — Custom evaluators you’ve created in your workspace
  • Library — Pre-built evaluator templates grouped by category
Browse evaluators using the search bar or filter tabs (All / My Evaluators / Library). Select at least one evaluator to proceed.The Library provides pre-built evaluators across the following categories:
You must select at least one evaluator. Only single-turn evaluators are available for online evaluations.
3

Evaluator Configuration (Step 3)

Configure each selected evaluator with its specific settings:For LLM-based evaluators:
  • Select the provider and model (e.g., OpenAI / gpt-4o-mini)
  • Define evaluation criteria and pass criteria
Variable mapping:
  • Map evaluator variables to trace data using JMESPath expressions
  • Variables can source from trace attributes, span data, or literal values
Pass criteria:
  • Numerical evaluators: Set a threshold with an operator (>=, <=, >, <, =)
  • Boolean evaluators: Set pass condition (is_true / is_false)
Each evaluator card shows its configuration status. All required fields must be filled before proceeding.
4

Sampling & Alerts (Step 4)

Configure how many traces to evaluate and optionally set up alert rules.Sampling Rate:Use the slider or presets to set any value from 1–100%.Alerts (optional):Toggle alerts on to receive notifications when quality degrades. When enabled, configure:Each alert rule supports:
  • Severity — CRITICAL or WARNING
  • Comparison type — is_above or is_below
  • Threshold — The value that triggers the alert
  • Duration — How long the condition must persist before firing
Click Create (or Update in edit mode) to save. The online evaluation begins running immediately in Active status.

​
Editing an Online Evaluation

To edit an existing online evaluation:
  1. Navigate to the Online Evaluations list page
  2. Click the Edit (pencil icon) button in the Actions column of the evaluation you want to modify
  3. The configuration modal opens pre-filled with the current settings
  4. Modify any settings across the 4 steps (Rule Details, Evaluator Selection, Evaluator Configuration, Sampling & Alerts)
  5. Click Update to save your changes
Changes take effect immediately — the next incoming trace will be evaluated using the updated configuration.
You can also toggle an online evaluation between Active and Inactive from the detail page without opening the edit modal.

​
Deleting an Online Evaluation

To delete an online evaluation:
  1. Navigate to the Online Evaluations list page
  2. Click the Delete (trash icon) button in the Actions column
  3. A confirmation modal appears: “Are you sure you want to delete [name]? This action cannot be undone.”
  4. Click Delete to confirm
Deleting an online evaluation permanently removes the configuration and all associated evaluation history. This action cannot be undone.

​
Best Practices

​
FAQ

An online evaluation analyses all incoming traces for the project that match the configured scope. If you select a specific agent, only traces from that agent are evaluated. If you add filters, only traces matching those filter conditions are evaluated. The sampling rate then determines what percentage of matching traces actually get scored.
Yes. You can configure multiple online evaluations, each with different agents, filters, evaluators, and sampling rates. Each operates independently.
When inactive, no new traces are evaluated. Historical results remain accessible in the Overview, Results, and Alerts tabs. Toggling back to Active resumes evaluation of new incoming traces.
The overall pass rate pools all pass/fail judgments across all evaluators for all evaluated traces in the selected time range. If a trace has 3 evaluators and passes 2, that contributes 2 passes and 1 failure to the aggregate.
Sampling rate determines the percentage of matching traces that are actually evaluated. At 100%, every trace is scored. At 25%, roughly one in four matching traces is selected. This helps manage evaluation costs at high traffic volumes.
Alerts check conditions at the configured re-evaluation interval (e.g., every 15 minutes). If the overall pass rate or a per-evaluator metric crosses the threshold for the specified duration, the alert fires and notifications are sent to your configured contact points. Alerts auto-resolve when conditions return to normal.
Yes. In Step 2 (Evaluator Selection), the Library tab shows pre-built evaluator templates. When you select a library evaluator, it is automatically saved to your workspace and linked to the online evaluation.
Evaluation cost reflects the estimated LLM cost for running evaluators (primarily LLM-as-Judge type). Non-LLM evaluators (latency, regex, JSON validation) have no cost. The Total Evaluation Cost metric in the Overview tab tracks this over time.
  • Evaluation Agents — Create custom evaluators for online and offline evaluation
  • Traces — View and debug individual trace executions
Last modified on July 31, 2026