Evaluations

Image evaluations are collections of test cases designed to evaluate image generation, editing, and analysis systems. They define the visual inputs, expected outputs, and metadata that Image Evaluators use to score your AI’s performance. Create them manually or import from CSV for comprehensive visual quality coverage.

Why Image Evaluations Matter

Image evaluations transform ad-hoc visual testing into systematic quality assurance:

Benefit	Description
Reproducibility	Run the same visual tests across model updates and prompt changes
Regression Detection	Compare visual quality over time to catch degradation early
Objective Benchmarking	Measure image quality against defined criteria, not subjective impressions
URL-based Inputs	All images referenced via URLs in metadata fields

Image Evaluation Dashboard

Navigate to Evaluate → Evaluations from the left navigation panel to access your evaluations.

Column	Description
Evaluation Name	Unique identifier for the image test suite
Tags	Metadata labels for filtering and organization
Created At	Timestamp for version tracking
Actions	Quick access to edit or delete evaluations

Creating an Image Evaluation

Click New Evaluation in the top right corner of the Evaluations page.

Add Manually

Build image test suites from scratch, one test case at a time

Upload CSV

Import a spreadsheet of image test cases in bulk

Create using AI

Generate test cases using AI based on your description

Configure Basics

Both creation methods start with the same configuration:

Field	Description
Name	A descriptive identifier for your image test suite
Tags	Labels for filtering (e.g., “image-generation”, “quality-testing”, “v2-model”)
Turn Type	Single Turn for image input/output pairs
Data Type	Select Image for image-based evaluations
Creation Method	Add manually or Upload CSV

Multi-turn image scenarios are not currently supported. Image evaluations use single-turn test cases.

Creating Items Manually

Open Creation Form

Click the New Evaluation button and select Image as the data type.

Add Image Test Cases

For each test case, provide:

Field	Required	Description
Input	Yes	Text prompt describing what to generate or edit (e.g., “A red sports car in a showroom”)
Metadata	No	Image URLs and expected properties (see Metadata Fields below)

Example metadata for text-to-image generation:

{
  "reference_images": ["https://example.com/reference1.jpg"],
  "text_caption": "A photorealistic portrait with natural lighting",
  "expected_dimensions": "1024x1024",
  "expected_format": "PNG"
}

Example metadata for image editing:

{
  "input_image": "https://example.com/original.jpg",
  "input_edit_instruction": "Remove the background"
}

Select Evaluators

Click Next and select image evaluators from the library or your saved configurations. Choose evaluators appropriate for your visual quality criteria.

Map Variables

Configure variable mappings to connect evaluator inputs to your image data:

Source	Use Case
Input	Text prompt describing what to generate or edit
Task Output	Generated or edited image URL (returned by your task function)
Metadata Fields	Image URLs and expected properties (reference_images, text_caption, etc.)
Value	Fixed literal values for comparison

Finalize

Click Create Evaluation to complete the process.

Uploading a CSV

For bulk creation, prepare a CSV with columns for input, expected output, and optional metadata.

input,expected_output,metadata
"A photorealistic portrait with natural lighting","","{""reference_images"":[""https://example.com/ref1.jpg""],""expected_dimensions"":""1024x1024""}"
"Remove the background from this product photo","","{""input_image"":""https://example.com/product.jpg"",""input_edit_instruction"":""Remove background""}"

Upload Your File

Choose Upload CSV as the creation method and select your file. A sample template is available for download.

Map Columns

Assign each CSV column to a target field—input, expected_output, metadata—or skip it.

Review Validation

Row-level errors are flagged before import. Common issues include invalid image URLs in metadata, unsupported formats, or malformed metadata JSON. Fix any issues and re-upload if needed.

Select Evaluators

Continue through evaluator selection and variable mapping as with manual creation.

Metadata Fields

Image evaluations use metadata fields to store image URLs and expected properties. These fields map to image evaluator variables:

Metadata Field	Description	Example Value
`reference_images`	Array of reference image URLs for comparison	`["https://example.com/ref.jpg"]`
`text_caption`	Description of the image for text-image alignment evaluation	`"A sunset over mountains"`
`expected_dimensions`	Expected pixel dimensions in WxH format	`"1024x1024"`
`expected_aspect_ratio`	Expected aspect ratio in W:H format	`"16:9"`
`expected_format`	Expected image file format	`"PNG"`
`expected_max_file_size`	Maximum allowed file size with unit	`"2MB"`
`input_image`	URL of the original image (for editing tasks)	`"https://example.com/original.jpg"`
`input_edit_instruction`	Editing instruction applied to the image	`"Remove the background"`

The output_image variable is automatically populated from your task function’s return value. You do not need to include it in metadata.

Managing an Image Evaluation

Open any evaluation to manage it. The detail page has three tabs:

Tab	Description
Details	Browse and edit image test cases; preview images inline
Analytics	Embedded evaluation analytics—overall health, average score over time, and per-evaluator pass/fail rates
Test Runs	Runs triggered from this evaluation, with a Run Test Suite action

From the detail page you can also:

Preview images directly in the test case list
Clone the evaluation or individual test cases to create variants
Bulk edit items across the evaluation
Manage default evaluators — add evaluators that apply to all items, or override them for specific items

Running an Image Evaluation

Once your evaluation is configured with evaluators, trigger a test run via the SDK.

Copy Evaluation ID

Open your evaluation and copy the Evaluation ID displayed at the top of the page.

Trigger Evaluation

Use the Evaluation ID to fetch the evaluation and run a test suite. The task function receives the image input from each evaluation item and should return the generated output.

from netra import Netra

Netra.init(app_name="my-app")

def my_image_task(input_data):
    # Your AI logic — generates/edits image based on input_data text prompt
    # and returns the image URL (must be publicly accessible)
    return generated_image_url

dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")

result = Netra.evaluation.run_test_suite(
    name="Image Evaluation",
    data=dataset,
    task=my_image_task,
)

import { Netra } from "netra-sdk";

await Netra.init({ appName: "my-app" });

async function myImageTask(inputData: string): Promise<string> {
  // Your AI logic — generates/edits image based on inputData text prompt
  // and returns the image URL (must be publicly accessible)
  return generatedImageUrl;
}

const dataset = await Netra.evaluation.getDataset("your-dataset-id");

const result = await Netra.evaluation.runTestSuite(
  "Image Evaluation",
  dataset,
  myImageTask,
);

The returned image URL is compared against metadata fields by the evaluators attached to the evaluation. See the SDK reference for Python and TypeScript for the full API.

View Results

Monitor progress and results in Image Test Runs.

Best Practices

Organizing Image Evaluations

Use descriptive names: “Product Photos - Lighting Variants” is better than “Image Evaluation 1”
Tag by use case: Create tags for different evaluation scenarios (e.g., “portrait”, “landscape”, “product”)
Version your evaluations: Include version numbers in tags when testing model iterations

Building Effective Image Test Cases

Cover diverse scenarios: Include various image types, styles, and complexity levels
Include edge cases: Low resolution, unusual formats, complex compositions
Balance quantity and quality: A smaller evaluation of diverse, high-quality test cases beats a large evaluation of similar images
Add negative tests: Include cases where the expected behavior is to reject or fail gracefully

Maintaining Image Evaluations

Update regularly: Add new test cases as you discover new patterns
Remove outdated cases: Delete test cases that no longer reflect current requirements
Review failed cases: Investigate failures to determine if the AI is wrong or the expected output needs updating
Clone before major changes: Duplicate an evaluation when testing significant model or prompt revisions

Evaluation Overview - Understand the full evaluation framework
Image Evaluators - Configure scoring logic for image evaluations
Image Test Runs - Analyze image evaluation results
Text Evaluations - Compare with text-based evaluation evaluations

Get Started

Observability

Insights

Simulations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Evaluations

Why Image Evaluations Matter

Image Evaluation Dashboard

Creating an Image Evaluation

Add Manually

Upload CSV

Create using AI

Configure Basics

Creating Items Manually

Uploading a CSV

Metadata Fields

Managing an Image Evaluation

Running an Image Evaluation

Best Practices

Organizing Image Evaluations

Building Effective Image Test Cases

Maintaining Image Evaluations

​Why Image Evaluations Matter

​Image Evaluation Dashboard

​Creating an Image Evaluation

Add Manually

Upload CSV

Create using AI

​Configure Basics

​Creating Items Manually

​Uploading a CSV

​Metadata Fields

​Managing an Image Evaluation

​Running an Image Evaluation

​Best Practices

​Organizing Image Evaluations

​Building Effective Image Test Cases

​Maintaining Image Evaluations

​Related

Why Image Evaluations Matter

Image Evaluation Dashboard

Creating an Image Evaluation

Configure Basics

Creating Items Manually

Uploading a CSV

Metadata Fields

Managing an Image Evaluation

Running an Image Evaluation

Best Practices

Organizing Image Evaluations

Building Effective Image Test Cases

Maintaining Image Evaluations

Related