> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getnetra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Online Evaluations

> Continuously evaluate production traces in real-time by running configurable evaluators against incoming agent data — automatically and at scale.

AI agents in production don't fail in predictable ways — they degrade silently, drift over time, and exhibit issues that only surface under real-world conditions. Netra's Online Evaluations framework brings automated, continuous quality assessment directly into your production pipeline. Instead of manually reviewing traces, you define evaluation rules once, and every incoming trace that matches your criteria gets scored automatically.

<Card title="Evaluators" icon="robot" href="/Evaluation/Evaluators">
  Online Evaluations work alongside Evaluators. Learn how to build custom evaluators for your use cases.
</Card>

## Why Online Evaluations Matter

Without continuous production monitoring, issues compound silently. Online Evaluations help you answer critical questions in real-time:

| Question                                                   | What Online Evaluations Measure                                 |
| ---------------------------------------------------------- | --------------------------------------------------------------- |
| Is my agent producing quality outputs in production?       | Real-time pass rates across all configured evaluators           |
| Are there regressions after deploying a new agent version? | Continuous score tracking and trend analysis over time          |
| Which evaluators are flagging the most failures?           | Per-evaluator breakdown with pass/fail rates                    |
| Am I meeting quality SLAs for my AI product?               | Configurable alert thresholds with automatic notifications      |
| What does my evaluation cost look like over time?          | Total evaluation cost tracking across all configured evaluators |

## How It Works

Online Evaluations run as a background process that continuously analyses all traces coming into your project. When configured for a specific agent (or all agents), the system:

1. **Intercepts incoming traces** — Every trace logged to the project is checked against the online evaluation's scope and filters
2. **Applies sampling** — Based on your configured sampling rate (1–100%), traces are selected for evaluation
3. **Runs evaluators** — Each selected trace is scored against all configured evaluators (LLM-as-Judge, latency, cost, regex, etc.)
4. **Records results** — Evaluation scores, pass/fail status, and reasoning are stored per trace
5. **Triggers alerts** — If configured thresholds are breached, alerts fire to your contact points

<Info>
  Online Evaluations operate at the **trace** scope — each incoming trace is treated as a single evaluation unit scored against your configured evaluators.
</Info>

***

## Online Evaluations List Page

When you navigate to **Online Evaluations**, you see a table listing all configured online evaluations for the current project.

| Column                | Description                                          |
| --------------------- | ---------------------------------------------------- |
| **Name**              | The name assigned to the online evaluation           |
| **Total Evaluations** | Total number of trace evaluation results recorded    |
| **No. of Evaluators** | Number of evaluators configured for this online eval |
| **Status**            | Active (running) or Inactive (paused)                |
| **Actions**           | Edit and Delete buttons                              |

The page also provides:

* **Search** — Filter evaluations by name
* **Refresh** — Reload the list to see the latest data
* **Create button** — Opens the configuration modal to create a new online evaluation

Clicking any row navigates to the detailed view for that online evaluation.

***

## Online Evaluation Detail View

On clicking into a specific online evaluation, you see its detail page with three tabs and an active/inactive toggle in the top right.

### Active / Inactive Toggle

The toggle at the top right controls whether the online evaluation is currently running. When **Active**, it continuously evaluates incoming traces. When **Inactive**, no new traces are evaluated but historical results remain accessible.

### Overview Tab

The Overview tab is a time-filtered analytics dashboard that provides a high-level picture of how your evaluation is performing.

<Frame>
  <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=a76dd08fdce4732eebe7aed7af903fe0" alt="Image" width="1700" height="983" data-path="images/image.png" />
</Frame>

**Summary Cards:**

| Metric                      | Description                                                                |
| --------------------------- | -------------------------------------------------------------------------- |
| **Traces Evaluated**        | Total number of traces that have been evaluated in the selected time range |
| **Overall Pass Rate**       | Percentage of pass judgments across all evaluators, pooled together        |
| **Average Evaluator Score** | Mean numeric score across all evaluators for the selected period           |
| **Total Evaluation Cost**   | Estimated LLM cost incurred for running the evaluators                     |

**Charts:**

* **Evaluator Scores** — A combined line chart showing per-evaluator pass rates over time, with a white average score line overlaid
* **Traces Evaluated** — A bar chart showing evaluation volume over time
* **Online Evaluation Cost** — A line chart tracking cost over time

All charts respect the selected time range and automatically group data by minute, hour, or day depending on the range.

### Results Tab

The Results tab provides granular, trace-level evaluation data split into two sections:

<Frame>
  <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-1.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=438e19f488aea5e25ba8048b24c88078" alt="Image" width="1693" height="974" data-path="images/image-1.png" />
</Frame>

**Evaluator Breakdown Table:**

| Column           | Data                                      |
| ---------------- | ----------------------------------------- |
| Evaluator        | Evaluator name                            |
| Type             | Evaluator type (e.g., LLM-as-Judge)       |
| Traces Evaluated | Number of traces scored by this evaluator |
| Pass Rate        | Percentage of traces that passed          |
| Fail Rate        | Percentage of traces that failed          |

**Evaluation Results Table:**

A detailed per-trace results table with:

* **Timestamp** — When the trace was evaluated
* **Trace ID** — Unique identifier (clickable to view full trace)
* **Evaluator columns** — One column per configured evaluator showing score, pass/fail status, and reasoning

Clicking a row opens a sidebar with full evaluation details for that trace, including per-evaluator scores, reasoning, pass criteria, and a link to view the complete trace.

### Alerts Tab

The Alerts tab displays a history of all triggered alerts for this online evaluation.

<Frame>
  <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-2.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=efa102bcdf05cdc8c1ecb318567085a2" alt="Image" width="1688" height="975" data-path="images/image-2.png" />
</Frame>

| Column         | Description                                                     |
| -------------- | --------------------------------------------------------------- |
| Triggered Time | When the alert was fired                                        |
| Alert Details  | Source (overall pass rate vs. per-evaluator) and detail message |
| Severity       | CRITICAL or WARNING                                             |
| Status         | TRIGGERED or RESOLVED                                           |

The table is sortable by Triggered Time, Severity, and Status.

***

## Configuring an Online Evaluation

Creating an online evaluation is a 4-step process:

<Steps>
  <Step title="Rule Details (Step 1)">
    Define the scope and identity of your online evaluation:

    <Frame>
      <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-3.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=72d0482b1caa34747b56674000628c97" alt="Image" width="788" height="916" data-path="images/image-3.png" />
    </Frame>

    | Field           | Required | Description                                                                                                                                                                        |
    | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | **Agent**       | No       | Select a specific agent to evaluate, or leave as "All agents" to evaluate every trace in the project                                                                               |
    | **Filters**     | No       | Add trace filters to narrow which traces get evaluated (e.g., by metadata, tags, or attributes). A matching trace count preview shows how many traces matched in the last 24 hours |
    | **Name**        | Yes      | A descriptive name for this online evaluation (e.g., "Production Quality Monitor")                                                                                                 |
    | **Description** | No       | Optional description explaining the purpose of this evaluation                                                                                                                     |

    <Warning>
      If you add filters, all filter rows must be complete (field, operator, and value) before proceeding.
    </Warning>
  </Step>

  <Step title="Evaluator Selection (Step 2)">
    Choose which evaluators will score your traces. You can select from two sources:

    * **My Evaluators** — Custom evaluators you've created in your workspace
    * **Library** — Pre-built evaluator templates grouped by category

    Browse evaluators using the search bar or filter tabs (All / My Evaluators / Library). Select at least one evaluator to proceed.

    <Frame>
      <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-4.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=076e367a5be169519dfa7bc2523f67b9" alt="Image" width="789" height="923" data-path="images/image-4.png" />
    </Frame>

    The Library provides pre-built evaluators across the following categories:

    | Category            | What It Covers                                                                     |
    | ------------------- | ---------------------------------------------------------------------------------- |
    | **Quality**         | Response accuracy, relevance, faithfulness to context, and hallucination detection |
    | **Multimodal**      | Image generation quality, format validation, and text-image alignment              |
    | **Guardrails**      | Safety checks — bias detection, topic boundaries, and harmful content              |
    | **Performance**     | Response time, cost, and token consumption thresholds                              |
    | **Semantic**        | Meaning-level comparison between outputs and references                            |
    | **Agentic**         | Whether the agent achieved the user's stated goal                                  |
    | **JSON Evaluator**  | Validates output JSON structure and field values                                   |
    | **Regex Evaluator** | Validates output format against a regex pattern                                    |
    | **Tool Use**        | Verifies correct tool calls against expected tools                                 |

    <Info>
      You must select at least one evaluator. Only single-turn evaluators are available for online evaluations.
    </Info>
  </Step>

  <Step title="Evaluator Configuration (Step 3)">
    Configure each selected evaluator with its specific settings:\\

    <Frame>
      <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-5.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=407cf2eaef5c78484b7a738c8980a73a" alt="Image" width="789" height="920" data-path="images/image-5.png" />
    </Frame>

    **For LLM-based evaluators:**

    * Select the provider and model (e.g., OpenAI / gpt-4o-mini)
    * Define evaluation criteria and pass criteria

    **Variable mapping:**

    * Map evaluator variables to trace data using JMESPath expressions
    * Variables can source from trace attributes, span data, or literal values

    **Pass criteria:**

    * Numerical evaluators: Set a threshold with an operator (`>=`, `<=`, `>`, `<`, `=`)
    * Boolean evaluators: Set pass condition (`is_true` / `is_false`)

    Each evaluator card shows its configuration status. All required fields must be filled before proceeding.
  </Step>

  <Step title="Sampling & Alerts (Step 4)">
    Configure how many traces to evaluate and optionally set up alert rules.

    <Frame>
      <img src="https://mintcdn.com/netra/fnyvvQXplMfXTzuZ/images/image-6.png?fit=max&auto=format&n=fnyvvQXplMfXTzuZ&q=85&s=5406cc684cbfb64a9a1f315c21e9426d" alt="Image" width="781" height="923" data-path="images/image-6.png" />
    </Frame>

    **Sampling Rate:**

    | Preset | Description                               |
    | ------ | ----------------------------------------- |
    | 10%    | Light sampling for high-volume production |
    | 25%    | Balanced sampling for moderate traffic    |
    | 50%    | Substantial coverage                      |
    | 75%    | High coverage                             |
    | 100%   | Evaluate every matching trace (default)   |

    Use the slider or presets to set any value from 1–100%.

    **Alerts (optional):**

    Toggle alerts on to receive notifications when quality degrades. When enabled, configure:

    | Setting                      | Description                                                       |
    | ---------------------------- | ----------------------------------------------------------------- |
    | **Contact Points**           | Select notification channels (email, Slack, etc.)                 |
    | **Re-evaluation Interval**   | How often to check alert conditions (15 min to 24 hours)          |
    | **Overall Pass Rate Alerts** | Trigger when overall pass rate crosses a threshold for a duration |
    | **Per-Evaluator Alerts**     | Set individual thresholds per evaluator                           |

    Each alert rule supports:

    * **Severity** — CRITICAL or WARNING
    * **Comparison type** — is\_above or is\_below
    * **Threshold** — The value that triggers the alert
    * **Duration** — How long the condition must persist before firing

    Click **Create** (or **Update** in edit mode) to save. The online evaluation begins running immediately in Active status.
  </Step>
</Steps>

***

## Editing an Online Evaluation

To edit an existing online evaluation:

1. Navigate to the **Online Evaluations** list page
2. Click the **Edit** (pencil icon) button in the Actions column of the evaluation you want to modify
3. The configuration modal opens pre-filled with the current settings
4. Modify any settings across the 4 steps (Rule Details, Evaluator Selection, Evaluator Configuration, Sampling & Alerts)
5. Click **Update** to save your changes

Changes take effect immediately — the next incoming trace will be evaluated using the updated configuration.

<Info>
  You can also toggle an online evaluation between Active and Inactive from the detail page without opening the edit modal.
</Info>

***

## Deleting an Online Evaluation

To delete an online evaluation:

1. Navigate to the **Online Evaluations** list page
2. Click the **Delete** (trash icon) button in the Actions column
3. A confirmation modal appears: *"Are you sure you want to delete \[name]? This action cannot be undone."*
4. Click **Delete** to confirm

<Warning>
  Deleting an online evaluation permanently removes the configuration and all associated evaluation history. This action cannot be undone.
</Warning>

***

## Best Practices

| Practice                                                 | Why It Matters                                                |
| -------------------------------------------------------- | ------------------------------------------------------------- |
| Start with 100% sampling, reduce as volume grows         | Ensures full coverage during setup, then optimize for cost    |
| Configure alerts with appropriate durations              | Avoids noisy alerts from transient spikes                     |
| Use per-evaluator alerts for critical quality dimensions | Pinpoints exactly which quality aspect is degrading           |
| Combine multiple evaluator types                         | Catches different failure modes (quality, latency, structure) |
| Filter by specific agents for targeted monitoring        | Focuses evaluation budget on the most critical agents         |
| Review the Results tab regularly                         | Identifies patterns that summary metrics might miss           |
| Set meaningful pass criteria per evaluator               | Defines what "good" means for your specific use case          |

***

## FAQ

<AccordionGroup>
  <Accordion title="What traces does an online evaluation analyse?">
    An online evaluation analyses all incoming traces for the project that match the configured scope. If you select a specific agent, only traces from that agent are evaluated. If you add filters, only traces matching those filter conditions are evaluated. The sampling rate then determines what percentage of matching traces actually get scored.
  </Accordion>

  <Accordion title="Can I run multiple online evaluations on the same project?">
    Yes. You can configure multiple online evaluations, each with different agents, filters, evaluators, and sampling rates. Each operates independently.
  </Accordion>

  <Accordion title="What happens when I toggle an online evaluation to Inactive?">
    When inactive, no new traces are evaluated. Historical results remain accessible in the Overview, Results, and Alerts tabs. Toggling back to Active resumes evaluation of new incoming traces.
  </Accordion>

  <Accordion title="How is the Overall Pass Rate calculated?">
    The overall pass rate pools all pass/fail judgments across all evaluators for all evaluated traces in the selected time range. If a trace has 3 evaluators and passes 2, that contributes 2 passes and 1 failure to the aggregate.
  </Accordion>

  <Accordion title="What does sampling rate control?">
    Sampling rate determines the percentage of matching traces that are actually evaluated. At 100%, every trace is scored. At 25%, roughly one in four matching traces is selected. This helps manage evaluation costs at high traffic volumes.
  </Accordion>

  <Accordion title="How do alerts work?">
    Alerts check conditions at the configured re-evaluation interval (e.g., every 15 minutes). If the overall pass rate or a per-evaluator metric crosses the threshold for the specified duration, the alert fires and notifications are sent to your configured contact points. Alerts auto-resolve when conditions return to normal.
  </Accordion>

  <Accordion title="Can I use library evaluators?">
    Yes. In Step 2 (Evaluator Selection), the Library tab shows pre-built evaluator templates. When you select a library evaluator, it is automatically saved to your workspace and linked to the online evaluation.
  </Accordion>

  <Accordion title="What is the evaluation cost?">
    Evaluation cost reflects the estimated LLM cost for running evaluators (primarily LLM-as-Judge type). Non-LLM evaluators (latency, regex, JSON validation) have no cost. The Total Evaluation Cost metric in the Overview tab tracks this over time.
  </Accordion>
</AccordionGroup>

***

## Related

* [Evaluation Agents](/Evaluation/Evaluators) — Create custom evaluators for online and offline evaluation
* [Traces](/Observability/Traces/overview) — View and debug individual trace executions