> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getnetra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation Overview

> Measure and improve AI quality with Netra's evaluation framework. Run automated test suites, track scores over time, and catch regressions before production.

AI systems don't fail loudly. They drift, regress, and quietly degrade over time. Netra's Evaluation framework makes that invisible failure visible, giving you a structured, repeatable way to measure how your AI behaves—not just once, but continuously across releases, prompts, models, and environments.

<Card title="Quick Start: Evaluation" icon="rocket" href="/quick-start/QuickStart_Evals">
  New to evaluations? Get your first evaluation running in minutes.
</Card>

## Why Evaluation Matters

Without systematic evaluation, you're flying blind. Netra helps you answer critical questions with confidence:

| Question                                | What Netra Measures                                       |
| --------------------------------------- | --------------------------------------------------------- |
| Is my system producing correct answers? | Answer correctness, semantic similarity, factual accuracy |
| Did this update introduce a regression? | Side-by-side comparison across test runs                  |
| Are costs creeping up unnoticed?        | Token usage, latency, and cost per evaluation             |
| Are my agents executing correctly?      | Tool call sequences, decision paths, guardrail compliance |

## Core Building Blocks

The Evaluation suite is built on three interconnected pillars:

### Evaluators

[Evaluators](/Evaluation/Evaluators) are the scoring logic that assesses your AI's performance. Netra offers two approaches:

**LLM as Judge**
Best for subjective quality, semantic correctness, and nuanced criteria. Use prebuilt templates or write custom prompts with providers like OpenAI, Anthropic, and Google.

**Code Evaluators**
Best for deterministic checks using JavaScript or Python—JSON schema validation, regex matching, mathematical calculations, and custom business logic.

<Info>
  Netra provides a [Library](/Evaluation/Evaluators#library) of preconfigured evaluators covering Quality, Performance, Agentic behavior, and Guardrails. Customize any evaluator and save it to **My Evaluators** for reuse across datasets.
</Info>

**Playground Testing**

Before deploying an evaluator, test it in the integrated Playground:

* Input sample data and run evaluations in real-time
* Refine prompt templates and adjust pass/fail thresholds
* Verify edge case handling before adding to your pipeline

### Datasets

[Datasets](/Evaluation/Datasets) are collections of test cases that define what you want to evaluate.

| Feature                | Description                                                                |
| ---------------------- | -------------------------------------------------------------------------- |
| **Create from Traces** | Convert real production interactions into test cases with one click        |
| **Manual Creation**    | Build test suites from scratch in the dashboard                            |
| **Variable Mapping**   | Map evaluator inputs to dataset fields, agent responses, or trace metadata |
| **Metadata & Tags**    | Organize datasets by feature, model, or release version                    |

### Test Runs

[Test Runs](/Evaluation/TestRuns) execute your datasets through the evaluation pipeline, providing point-in-time snapshots of system health.

| Feature                | Description                                                          |
| ---------------------- | -------------------------------------------------------------------- |
| **Deep Diagnostics**   | Compare expected output vs. actual output side-by-side               |
| **Trace Integration**  | Link directly to execution traces to debug the "why" behind failures |
| **Aggregated Metrics** | View total cost, average latency, and pass/fail rates across the run |

## Use Cases

### Regression Testing

Catch quality degradation before it reaches production:

1. Create a dataset from your golden test cases
2. Run evaluations after each model or prompt change
3. Compare results across test runs to identify regressions

### Continuous Quality Monitoring

Track quality metrics over time:

1. Convert production traces into datasets
2. Schedule regular evaluation runs
3. Set up [alerts](/Alert-rules/Alert-rules) when pass rates drop below thresholds

### Model Comparison

Evaluate different models or prompts objectively:

1. Create a standardized dataset
2. Run the same inputs through different model configurations
3. Compare scores across test runs to make data-driven decisions

## Getting Started

<Steps>
  <Step title="Configure Evaluators">
    Define your scoring criteria by adding [evaluators](/Evaluation/Evaluators)—choose from the library or create custom ones.
  </Step>

  <Step title="Create a Dataset">
    [Create a dataset](/Evaluation/Datasets) from traces or manually in the dashboard, and attach your evaluators.
  </Step>

  <Step title="Run Evaluations">
    Execute your dataset and view results in [Test Runs](/Evaluation/TestRuns).
  </Step>

  <Step title="Iterate and Improve">
    Use insights from test runs to refine your prompts, models, and evaluation criteria.
  </Step>
</Steps>

## Related

* [Quick Start: Evaluation](/quick-start/QuickStart_Evals) - Get started with evaluations in minutes
* [Datasets](/Evaluation/Datasets) - Create and manage test case collections
* [Evaluators](/Evaluation/Evaluators) - Configure scoring logic and criteria
* [Test Runs](/Evaluation/TestRuns) - Analyze evaluation results and track regressions
* [Traces](/Observability/Traces/overview) - Understand how evaluations connect to trace data