Evaluations

Evaluations are the foundation of your evaluation pipeline. They define what you’re testing—the inputs, expected outputs, and metadata that Evaluators use to score your AI system’s performance. Create them manually, generate them with AI, or import them from a CSV for comprehensive coverage.

Why Evaluations Matter

Evaluations transform ad-hoc testing into systematic quality assurance:

Benefit	Description
Reproducibility	Run the same tests across model updates, prompt changes, and code releases
Regression Detection	Compare results over time to catch quality degradation early
Objective Benchmarking	Measure performance against defined criteria, not gut feeling
Flexible Inputs	Support text prompts and image inputs with per-item metadata

Evaluation Dashboard

Navigate to Evaluate → Evaluations from the left navigation panel to access your evaluations. The page is shared by single-turn evaluations and multi-turn simulations — use the Turn type filter to distinguish them, and switch between card and list views from the page toolbar. Each text evaluation shows a text type icon with a Single-turn badge at the bottom-right.

In card view, each card shows the evaluation name, type and turn type (Text · Single-turn), tags, created date, a Run action, and a menu with Duplicate, Edit, and Delete.

In list view, columns are:

Column	Description
Name	Evaluation name, with the text icon and Single-turn badge
Turn Type	Single-turn for text evaluations
Tags	Metadata labels for filtering and organization
Created	Timestamp for version tracking
Actions	Run, Edit, Clone, and Delete

Creating an Evaluation

Click New Evaluation in the top right corner of the Evaluations page.

Add Manually

Build test suites from scratch, one test case at a time

Generate with AI

Draft test cases from a prompt or an existing agent

Upload CSV

Import a spreadsheet of test cases in bulk

Multi-turn simulation scenarios are created separately. See Simulation Evaluations.

The creation wizard has three steps: Setup → Test Cases → Evaluators. The evaluation name and tags are always editable in the header bar at the top.

Creating Items Manually

Setup

Click New Evaluation and configure the basics:

Field	Description
Evaluation name	A descriptive identifier for your test suite (required)
Type of evaluation	Select Text
Conversation style	Select Single Turn for request/response pairs

Tags can be added at any point via the header bar at the top of the wizard.

Add Test Cases

On the Test Cases step, build your dataset in any of these ways:

Type them in — fill each row directly in the table
Generate with AI — click Generate test cases with AI under the table (see Generate Test Cases with AI)
Import CSV — click Import CSV to upload a spreadsheet (see Uploading a CSV)

For each test case, provide:

Field	Required	Description
Input	Yes	The prompt or query sent to your AI system
Expected Output	Yes	The reference answer used by evaluators
Metadata	No	Additional context such as expected tools, thresholds, or reference data

AI test case generation is only available for Text evaluations. It does not appear for Image or Voice types.

Configure Evaluators

Click Next to reach the Evaluators step. This is a split view with the evaluator library on the left and your Scoring Pipeline on the right.

Click on any evaluator card in the library, or drag it into the pipeline, to add it
Expand an evaluator in the pipeline to configure its judge model, pass threshold, and variable mappings

Variable mappings connect evaluator prompt placeholders to your data:

Source	Use Case
Test Case	Input, expected output, or metadata from the test case
Agent Response	The actual LLM output generated at evaluation time
Execution Data	Metadata from logged traces (latency, tokens, cost)
Value	A fixed literal value

If you used Generate test cases with AI with Auto-attach evaluators on, suggested evaluators are already pre-filled here. Review and adjust as needed.

Create Evaluation

On the final step two buttons appear in the footer:

Create Evaluation — saves the evaluation and opens the detail page
Save and Run — saves the evaluation and immediately opens agent selection to trigger a test run

Generate Test Cases with AI

Use AI to draft test cases from a free-text description, attached context files, and/or an existing agent. Drafts appear in the table for review — nothing is saved until you create the evaluation.

Open the generate panel

On the Test Cases step, click Generate test cases with AI under the table. The generate panel slides up over the step.

Provide context

Provide at least one of the following:

Input	When to use	What you provide
Instructions	Describe the coverage you need	Free-text description (at least 30 characters if you include instructions)
Context files	Ground cases in real data	Up to 5 files — PDF, DOCX, or CSV, max 5 MB each
Agent	Reuse an existing agent as context	Select a registered agent, optionally with extra instructions

You can combine instructions, files, and an agent in the same run.

Set case types, count, and Auto-attach evaluators

Configure these options in the panel before you generate:

Case types — Choose which areas to cover: Happy flows, Edge cases, Error handling, Policies & compliance, and/or Adversarial
No of test cases — Choose 1–10 test cases
Auto-attach evaluators — Leave on to suggest evaluators for the generated batch, or turn it off if you only want test cases

Generate and review

Click Generate. A toast shows progress while the LLM runs.When complete, the drafts merge into the test cases table. From the success toast you can:

Retry — run the same request again with the same settings
Add more with AI — open the panel to generate additional cases
Dismiss — return to the dock

Edit any generated input, expected output, or metadata before continuing.

Confirm evaluators and create

If you left Auto-attach evaluators on, the Evaluators step arrives with suggested evaluators already selected — review and adjust as needed.Expand any evaluator to configure its judge model, pass threshold, and variable mappings, then click Create Evaluation to save or Save and Run to immediately trigger a test run.

Uploading a CSV

For bulk creation, prepare a CSV with columns for input, expected output, and optional metadata.

input,expected_output,metadata
"What is Netra?","An observability platform","{""category"":""product""}"

Upload Your File

On the Test Cases step, click Import CSV and select your file. A sample template is available for download.

Map Columns

Assign each CSV column to a target field — input, expected_output, metadata — or skip it.

Review and Confirm

Row-level errors are flagged before import. Fix any issues and re-upload if needed. Click Add & Continue in the footer to commit the rows and move to the Evaluators step.

Configure Evaluators

Select evaluators from the library, configure variable mappings per evaluator, then click Create Evaluation or Save and Run.

Managing an Evaluation

Open any evaluation to manage it. The detail page has three tabs:

Tab	Description
Details	Browse and edit test cases; add new items individually
Analytics	Embedded evaluation analytics—overall health, average score over time, and per-evaluator pass/fail rates
Test Runs	Runs triggered from this evaluation, with a Run Test Suite action

From the detail page you can also:

Clone the evaluation or individual test cases to create variants
Bulk edit items across the evaluation
Manage default evaluators — add evaluators that apply to all items, or override them for specific items

Running an Evaluation

Once your evaluation is configured with evaluators, trigger a test run via the SDK.

Copy Evaluation ID

Open your evaluation and copy the Evaluation ID displayed at the top of the page.

Trigger Evaluation

Use the Evaluation ID to fetch the evaluation and run a test suite. The task function receives the input field from each evaluation item and should return the generated output as a string.

from netra import Netra

Netra.init(app_name="my-app")

def my_task(input_data):
    # Your AI logic — takes the input from each evaluation item
    # and returns the generated output as a string
    return response

dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")

result = Netra.evaluation.run_test_suite(
    name="My Evaluation",
    data=dataset,
    task=my_task,
)

import { Netra } from "netra-sdk";

await Netra.init({ appName: "my-app" });

async function myTask(inputData: string): Promise<string> {
  // Your AI logic — takes the input from each evaluation item
  // and returns the generated output as a string
  return response;
}

const dataset = await Netra.evaluation.getDataset("your-dataset-id");

const result = await Netra.evaluation.runTestSuite(
  "My Evaluation",
  dataset,
  myTask,
);

The output is compared against expectedOutput by the evaluators attached to the evaluation. See the SDK reference for Python and TypeScript for the full API.

View Results

Monitor progress and results in Test Runs.

Best Practices

Organizing Evaluations

Use descriptive names: “Customer Support - Refund Requests” is better than “Evaluation 1”
Tag consistently: Create a tagging convention (e.g., by feature, model version, or test type)
Version your evaluations: Include version numbers in tags when testing prompt iterations

Building Effective Test Cases

Cover edge cases: Include unusual inputs, long prompts, and potential failure scenarios
Balance quantity and quality: A smaller evaluation of high-quality test cases beats a large evaluation of weak ones
Include negative tests: Add cases where the expected behavior is to refuse or ask for clarification
Use AI as a draft, then edit: Generate with AI to bootstrap coverage, then refine rows and confirm evaluator mappings before saving

Maintaining Evaluations

Update regularly: Add new test cases as you discover new patterns
Remove outdated cases: Delete test cases that no longer reflect current requirements
Review failed cases: Investigate failures to determine if the AI is wrong or the expected output needs updating
Clone before major changes: Duplicate an evaluation when testing significant prompt or model revisions

Evaluation Overview - Understand the full evaluation framework
Evaluators - Configure scoring logic for your evaluations
Test Runs - Analyze evaluation results
Simulation Evaluations - Create multi-turn scenarios for simulations

Get Started

Observability

Insights

Simulations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Evaluations

Why Evaluations Matter

Evaluation Dashboard

Creating an Evaluation

Add Manually

Generate with AI

Upload CSV

Creating Items Manually

Generate Test Cases with AI

Uploading a CSV

Managing an Evaluation

Running an Evaluation

Best Practices

Organizing Evaluations

Building Effective Test Cases

Maintaining Evaluations

​Why Evaluations Matter

​Evaluation Dashboard

​Creating an Evaluation

Add Manually

Generate with AI

Upload CSV

​Creating Items Manually

​Generate Test Cases with AI

​Uploading a CSV

​Managing an Evaluation

​Running an Evaluation

​Best Practices

​Organizing Evaluations

​Building Effective Test Cases

​Maintaining Evaluations

​Related

Why Evaluations Matter

Evaluation Dashboard

Creating an Evaluation

Creating Items Manually

Generate Test Cases with AI

Uploading a CSV

Managing an Evaluation

Running an Evaluation

Best Practices

Organizing Evaluations

Building Effective Test Cases

Maintaining Evaluations

Related