Evaluations

Image evaluations are collections of test cases designed to evaluate image generation, editing, and analysis systems. They define the visual inputs, expected outputs, and metadata that Image Evaluators use to score your AI’s performance. Create them manually or import from CSV for comprehensive visual quality coverage.

Why Image Evaluations Matter

Image evaluations transform ad-hoc visual testing into systematic quality assurance:

Benefit	Description
Reproducibility	Run the same visual tests across model updates and prompt changes
Regression Detection	Compare visual quality over time to catch degradation early
Objective Benchmarking	Measure image quality against defined criteria, not subjective impressions
URL-based Inputs	All images referenced via URLs in metadata fields

Image Evaluation Dashboard

Navigate to Evaluate → Evaluations from the left navigation panel to access your evaluations. Switch between card and list views from the page toolbar. Each image evaluation shows an image type icon with a Single Turn badge at the bottom-right. Image evaluations are always Single Turn.

In card view, each card shows the evaluation name, type and turn type (Image · Single Turn), tags, created date, a Run action, and a menu with Duplicate, Edit, and Delete.

In list view, columns are:

Column	Description
Name	Evaluation name, with the image icon and Single Turn badge
Turn Type	Always Single Turn for image evaluations
Tags	Metadata labels for filtering and organization
Created	Timestamp for version tracking
Actions	Run, Edit, Clone, and Delete

Creating an Image Evaluation

Click New Evaluation in the top right corner of the Evaluations page.

Add Manually

Build image test suites from scratch, one test case at a time

Upload CSV

Import a spreadsheet of image test cases in bulk

Image evaluations always use Single Turn. When you select Image as the evaluation type, the conversation style is automatically set to Single Turn.

The creation wizard has three steps: Setup → Test Cases → Evaluators. The evaluation name and tags are always editable in the header bar at the top.

Creating Items Manually

Setup

Click New Evaluation and configure the basics:

Field	Description
Evaluation name	A descriptive identifier for your image test suite (required)
Type of evaluation	Select Image. This automatically sets the conversation style to Single Turn

Tags can be added at any point via the header bar at the top of the wizard.

Add Test Cases

On the Test Cases step, build your dataset by typing test cases directly in the table or importing a CSV.

For each test case, provide:

Field	Required	Description
Input	Yes	The text prompt describing what to generate or edit (e.g., “A red sports car in a showroom”)

Click + Add Column to add optional metadata columns. Choose a preset, or enter a custom column name and click Add:

Optional Column	Description
Edit Instruction	Editing instruction to apply to the input image
Expected Images	Reference image URLs for comparison
Expected Tools	Expected tool calls
Expected Aspect Ratio	Expected aspect ratio (e.g., “16:9”)
Expected Dimensions	Expected pixel dimensions (e.g., “1024x1024”)
Expected Format	Expected image format (e.g., “PNG”)
Custom	Any name you enter — stored as a metadata field on each test case

AI test case generation is not available for Image evaluations.

Configure Evaluators

Click Next to reach the Evaluators step. This is a split view with the evaluator library on the left and your Scoring Pipeline on the right.

Click on any evaluator card in the library, or drag it into the pipeline, to add it
Expand an evaluator in the pipeline to configure its judge model, pass threshold, and variable mappings

Only image evaluators (multimodal evaluators) appear in the library for image evaluations.Variable mappings connect evaluator prompt placeholders to your data. Each variable can use one of these sources:

Source	Use Case
Test Case	A field from the evaluation item. For image evaluations this is Input or Metadata (for example `reference_images`, `expected_dimensions`, `input_edit_instruction`). Expected Output is not available for image evaluations
Agent Response	The task output at evaluation time — for image evaluators this is typically the generated or edited image URL (`output_image`)
Execution Data	Values from logged traces: summary metrics (cost, tokens, latency, tools, and similar) or span-level attributes
Value	A fixed literal value. Shown only when that variable allows an exact value as a source

Image library evaluators ship with defaults — for example output_image maps to Agent Response, and fields like reference_images or expected_aspect_ratio map to Test Case metadata. Adjust mappings only when your columns or task output differ from those defaults.

Create Evaluation

On the final step two buttons appear in the footer:

Create Evaluation — saves the evaluation and opens the detail page
Save and Run — saves the evaluation and immediately opens agent selection to trigger a test run

Uploading a CSV

For bulk creation, prepare a CSV with columns for input, expected output, and optional metadata.

input,expected_output,metadata
"A photorealistic portrait with natural lighting","","{""reference_images"":[""https://example.com/ref1.jpg""],""expected_dimensions"":""1024x1024""}"
"Remove the background from this product photo","","{""input_image"":""https://example.com/product.jpg"",""input_edit_instruction"":""Remove background""}"

Upload Your File

On the Test Cases step, click Import CSV and select your file. A sample template is available for download.

Map Columns

Assign each CSV column to a target field—input, expected_output, metadata—or skip it.

Review and Confirm

Row-level errors are flagged before import. Common issues include invalid image URLs in metadata, unsupported formats, or malformed metadata JSON. Fix any issues and re-upload if needed. Click Add & Continue in the footer to commit the rows and move to the Evaluators step.

Configure Evaluators

Select evaluators from the library, configure variable mappings per evaluator, then click Create Evaluation or Save and Run.

Metadata Fields

Image evaluations use metadata fields to store image URLs and expected properties. These fields map to image evaluator variables:

Metadata Field	Description	Example Value
`reference_images`	Array of reference image URLs for comparison	`["https://example.com/ref.jpg"]`
`text_caption`	Description of the image for text-image alignment evaluation	`"A sunset over mountains"`
`expected_dimensions`	Expected pixel dimensions in WxH format	`"1024x1024"`
`expected_aspect_ratio`	Expected aspect ratio in W:H format	`"16:9"`
`expected_format`	Expected image file format	`"PNG"`
`expected_max_file_size`	Maximum allowed file size with unit	`"2MB"`
`input_image`	URL of the original image (for editing tasks)	`"https://example.com/original.jpg"`
`input_edit_instruction`	Editing instruction applied to the image	`"Remove the background"`

The output_image variable is automatically populated from your task function’s return value. You do not need to include it in metadata.

Managing an Image Evaluation

Open any evaluation to manage it. The detail page has three tabs:

Tab	Description
Details	Browse and edit image test cases; preview images inline
Analytics	Embedded evaluation analytics—overall health, average score over time, and per-evaluator pass/fail rates
Test Runs	Runs triggered from this evaluation, with a Run Test Suite action

From the detail page you can also:

Preview images directly in the test case list
Clone the evaluation or individual test cases to create variants
Bulk edit items across the evaluation
Manage default evaluators — add evaluators that apply to all items, or override them for specific items

Running an Image Evaluation

Once your evaluation is configured with evaluators, trigger a test run via the SDK.

Copy Evaluation ID

Open your evaluation and copy the Evaluation ID displayed at the top of the page.

Trigger Evaluation

Use the Evaluation ID to fetch the evaluation and run a test suite. The task function receives the image input from each evaluation item and should return the generated output.

from netra import Netra

Netra.init(app_name="my-app")

def my_image_task(input_data):
    # Your AI logic — generates/edits image based on input_data text prompt
    # and returns the image URL (must be publicly accessible)
    return generated_image_url

dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")

result = Netra.evaluation.run_test_suite(
    name="Image Evaluation",
    data=dataset,
    task=my_image_task,
)

import { Netra } from "netra-sdk";

await Netra.init({ appName: "my-app" });

async function myImageTask(inputData: string): Promise<string> {
  // Your AI logic — generates/edits image based on inputData text prompt
  // and returns the image URL (must be publicly accessible)
  return generatedImageUrl;
}

const dataset = await Netra.evaluation.getDataset("your-dataset-id");

const result = await Netra.evaluation.runTestSuite(
  "Image Evaluation",
  dataset,
  myImageTask,
);

The returned image URL is compared against metadata fields by the evaluators attached to the evaluation. See the SDK reference for Python and TypeScript for the full API.

View Results

Monitor progress and results in Image Test Runs.

Best Practices

Organizing Image Evaluations

Use descriptive names: “Product Photos - Lighting Variants” is better than “Image Evaluation 1”
Tag by use case: Create tags for different evaluation scenarios (e.g., “portrait”, “landscape”, “product”)
Version your evaluations: Include version numbers in tags when testing model iterations

Building Effective Image Test Cases

Cover diverse scenarios: Include various image types, styles, and complexity levels
Include edge cases: Low resolution, unusual formats, complex compositions
Balance quantity and quality: A smaller evaluation of diverse, high-quality test cases beats a large evaluation of similar images
Add negative tests: Include cases where the expected behavior is to reject or fail gracefully

Maintaining Image Evaluations

Update regularly: Add new test cases as you discover new patterns
Remove outdated cases: Delete test cases that no longer reflect current requirements
Review failed cases: Investigate failures to determine if the AI is wrong or the expected output needs updating
Clone before major changes: Duplicate an evaluation when testing significant model or prompt revisions

Evaluation Overview - Understand the full evaluation framework
Image Evaluators - Configure scoring logic for image evaluations
Image Test Runs - Analyze image evaluation results
Text Evaluations - Compare with text-based evaluation evaluations

Get Started

Observability

Insights

Simulations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Evaluations

Why Image Evaluations Matter

Image Evaluation Dashboard

Creating an Image Evaluation

Add Manually

Upload CSV

Creating Items Manually

Uploading a CSV

Metadata Fields

Managing an Image Evaluation

Running an Image Evaluation

Best Practices

Organizing Image Evaluations

Building Effective Image Test Cases

Maintaining Image Evaluations

​Why Image Evaluations Matter

​Image Evaluation Dashboard

​Creating an Image Evaluation

Add Manually

Upload CSV

​Creating Items Manually

​Uploading a CSV

​Metadata Fields

​Managing an Image Evaluation

​Running an Image Evaluation

​Best Practices

​Organizing Image Evaluations

​Building Effective Image Test Cases

​Maintaining Image Evaluations

​Related

Why Image Evaluations Matter

Image Evaluation Dashboard

Creating an Image Evaluation

Creating Items Manually

Uploading a CSV

Metadata Fields

Managing an Image Evaluation

Running an Image Evaluation

Best Practices

Organizing Image Evaluations

Building Effective Image Test Cases

Maintaining Image Evaluations

Related