Image Evaluators

Image evaluators assess the quality, accuracy, and relevance of AI-generated or edited images. They transform subjective visual assessments into measurable metrics—from composition quality and style consistency to format compliance and text-image alignment. Use them with Image Evaluations to build automated visual quality pipelines.

Why Image Evaluators Matter

Without systematic visual assessment, you cannot measure improvement or catch regressions in image generation and editing systems:

Challenge	How Image Evaluators Help
Subjective quality	Multimodal LLM judges provide consistent, scalable visual assessment
Format validation	Code evaluators enforce dimensions, file types, and size constraints
Text-image alignment	Specialized evaluators verify images match prompts or descriptions
Editing accuracy	Comparison evaluators measure before/after quality and fidelity

Image Evaluator Types

Netra offers several approaches to image evaluation, each suited to different use cases:

Multimodal LLM as Judge

Vision-capable models evaluate subjective criteria like composition, style, and semantic accuracy. Best for nuanced visual quality assessment.

Image Analysis

Rule-based evaluators that assess technical properties—dimensions, file size, format, and aspect ratio—without AI judgment.

Image-Text Coherence

Measures alignment between image content and text descriptions. Verifies generated images match prompts or expected visual output.

Code Evaluator

Custom JavaScript or Python logic for specialized visual checks—pixel analysis, color distribution, or domain-specific validation.

Library Evaluators

Netra’s evaluator library includes pre-built image evaluators organized by category. Navigate to Library → Evaluators and browse the Multimodal category.

Evaluator	Type	What It Measures
Image Analysis	multimodal	Evaluates how faithfully an AI-generated image represents the text prompt
Image Aspect Ratio Check	multimodal-rule	Validates aspect ratio matches target (e.g., 16:9, 4:3)
Image Dimensions Check	multimodal-rule	Validates exact pixel dimensions (e.g., 800x800)
Image Editing Quality	multimodal	Evaluates how well an edited image reflects intended changes
Image Helpfulness	multimodal-llm	Measures how much images help users understand text
Image Output Format Validation	multimodal-rule	Validates file format (JPEG, PNG, GIF, WEBP, BMP, TIFF)
Image Size Check	multimodal-rule	Verifies file size within threshold
Image-Text Coherence	multimodal-llm	Evaluates how well images and text work together
Image-Text Reference	multimodal-llm	Checks whether text accurately describes what’s shown in images

These evaluators are available in the Library under the Multimodal category. Click Add on any evaluator card to customize it and save to My Evaluators. The multimodal-rule evaluators perform deterministic checks without AI judgment, while multimodal and multimodal-llm evaluators use vision-capable models for subjective assessment.

Creating Custom Image Evaluators

Click Add Custom in the top right corner to open the creation wizard.

Choose Evaluator Type

Select LLM as Judge for vision-based assessment or Code Evaluator for deterministic checks.

Select Turn Scope

Choose Image as the turn scope. This configures the evaluator to accept image inputs alongside text.

Configure the Evaluator

For LLM as Judge:

Write a prompt that includes {{image}} and {{expected_output}} placeholders
The {{image}} variable will be populated with the image from your evaluation item
Select a vision-capable model (e.g., GPT-4o, Claude 3.5 Sonnet)

For Code Evaluator:

Write a handler function that receives the image data and expected output
Return a score or boolean based on your validation logic

Example LLM prompt:

Analyze this image and evaluate its quality based on the following criteria:

Image: {{image}}
Expected Description: {{expected_output}}

Rate the image on:
1. Composition (0-10)
2. Clarity (0-10)
3. Style consistency (0-10)

Set Output and Pass Criteria

Output Type	Configuration
Numerical	Set a threshold (e.g., average score >= 7.0 to pass)
Boolean	Pass if all criteria meet minimum requirements

Test in Playground

Upload a sample image or provide an image URL
Run the evaluator in real-time
Refine your prompt until results are consistent

Code Evaluator Example

For deterministic image validation, use a Code Evaluator: JavaScript example:

function handler(input, output, expectedOutput) {
  // output contains the image metadata
  const minWidth = 512;
  const minHeight = 512;
  const allowedFormats = ['image/jpeg', 'image/png', 'image/webp'];

  return (
    output.width >= minWidth &&
    output.height >= minHeight &&
    allowedFormats.includes(output.format)
  );
}

Python example:

def handler(input, output, expectedOutput):
    min_width = 512
    min_height = 512
    allowed_formats = ['image/jpeg', 'image/png', 'image/webp']

    return (
        output['width'] >= min_width and
        output['height'] >= min_height and
        output['format'] in allowed_formats
    )

Variable Mapping for Image Evaluators

When configuring evaluators on evaluations, map variables to connect evaluator inputs to your image data:

Variable	Description	Source
`{{input}}`	Text prompt describing what to generate or edit	Evaluation item input
`{{output_image}}`	Generated or edited image URL	Task function return value
`{{input_image}}`	Original image (for editing tasks)	Metadata `input_image` field
`{{reference_images}}`	Array of reference images for comparison	Metadata `reference_images` field
`{{text_caption}}`	Description of the image	Metadata `text_caption` field
`{{expected_dimensions}}`	Expected pixel dimensions	Metadata `expected_dimensions` field
`{{expected_aspect_ratio}}`	Expected aspect ratio	Metadata `expected_aspect_ratio` field
`{{expected_format}}`	Expected file format	Metadata `expected_format` field
`{{expected_max_file_size}}`	Maximum allowed file size	Metadata `expected_max_file_size` field
`{{input_edit_instruction}}`	Editing instruction applied to the image	Metadata `input_edit_instruction` field

The output_image variable is automatically populated from your task function’s return value. All other image-related variables are sourced from the evaluation item’s metadata fields.

Best Practices

Writing Effective Image Evaluation Prompts

Be specific about criteria: Define exactly what constitutes good composition, clarity, or style
Provide reference descriptions: Include detailed expected output descriptions
Use structured scales: Rate specific aspects (composition, color, style) rather than overall quality
Test with diverse images: Validate with various image types, styles, and edge cases

Choosing the Right Evaluator Type

Use Case	Recommended Evaluator
”Does this image look professional?”	Image Analysis (multimodal)
“Is the image at least 1024x1024?”	Image Dimensions Check
”Does the aspect ratio match 16:9?”	Image Aspect Ratio Check
”Does the image match this description?”	Image-Text Coherence
”Is the file size under 5MB?”	Image Size Check
”Is this a valid PNG file?”	Image Output Format Validation
”Did the edit preserve the original style?”	Image Editing Quality
”Do the images help explain the text?”	Image Helpfulness
”Does the text describe what’s in the images?”	Image-Text Reference

Testing Before Deployment

Always use the Playground before adding evaluators to production evaluations:

Test with representative sample images from your use case
Include edge cases (low resolution, unusual formats, complex scenes)
Verify pass/fail thresholds produce expected results
Test with both successful and failure scenarios

Evaluators Overview - Understand the full evaluator framework
Text Evaluators - LLM-as-Judge, code, and rule-based text evaluators
Voice Evaluators - TTS, STT, and conversational evaluators
Evaluation Overview - Evaluations, test runs, and the evaluation framework
Image Evaluations - Create image test cases for your evaluators
Image Test Runs - View evaluation results for image evaluations
Quick Start: Evaluation - Get started with evaluations

Get Started

Observability

Insights

Evaluations

Simulations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Why Image Evaluators Matter

Image Evaluator Types

Multimodal LLM as Judge

Image Analysis

Image-Text Coherence

Code Evaluator

Library Evaluators

Creating Custom Image Evaluators

Code Evaluator Example

Variable Mapping for Image Evaluators

Best Practices

Writing Effective Image Evaluation Prompts

Choosing the Right Evaluator Type

Testing Before Deployment

​Why Image Evaluators Matter

​Image Evaluator Types

Multimodal LLM as Judge

Image Analysis

Image-Text Coherence

Code Evaluator

​Library Evaluators

​Creating Custom Image Evaluators

​Code Evaluator Example

​Variable Mapping for Image Evaluators

​Best Practices

​Writing Effective Image Evaluation Prompts

​Choosing the Right Evaluator Type

​Testing Before Deployment

​Related

Why Image Evaluators Matter

Image Evaluator Types

Library Evaluators

Creating Custom Image Evaluators

Code Evaluator Example

Variable Mapping for Image Evaluators

Best Practices

Writing Effective Image Evaluation Prompts

Choosing the Right Evaluator Type

Testing Before Deployment

Related