Overview

Evaluators are the scoring logic that determines whether your AI system meets quality standards. They transform subjective assessments into measurable metrics — from semantic correctness and tool execution accuracy to audio naturalness and visual composition. Use them with evaluations to build automated quality pipelines.

Quick Start: Evaluation

New to evaluations? Get your first evaluation running in minutes.

Evaluator Categories

Netra organizes evaluators by the modality they assess:

Text Evaluators

LLM-as-Judge, code-based, and rule-based evaluators for text outputs. Assess answer correctness, relevance, safety, tool execution, and custom business logic.

Voice Evaluators

Audio-aware evaluators for TTS, STT, and conversational delivery. Score voice naturalness, transcription accuracy, pronunciation, and speaking behavior.

Image Evaluators

Multimodal and rule-based evaluators for image generation and editing. Assess composition, style, format compliance, and image-text alignment.

Evaluator Types

Across all categories, Netra provides several evaluation approaches:

Type	Best For	How It Works
LLM as Judge	Subjective quality, semantic correctness, nuanced criteria	Uses AI models to evaluate AI outputs
Audio LLM as Judge(LALM)	Voice naturalness, pronunciation, conversational delivery	Listens to call recordings to score audio quality
Code Evaluator	Deterministic checks, JSON validation, regex, custom logic	Runs JavaScript or Python handlers
Rule-Based Checks	Latency, cost, token usage, tool-call matching, format validation	Configured through forms without writing prompts or code
Multimodal Judge	Image quality, composition, style, text-image alignment	Uses vision-capable models to assess visual outputs

Evaluators are not limited to their primary modality. Text evaluators can evaluate transcript content from voice interactions, making them useful for scoring conversation quality even when the input is audio-derived.

Scenario	How to Evaluate
Voice agent conversation quality	Use voice evaluators for audio quality + text evaluators on the transcript for factual accuracy and topic adherence
Voice agent goal completion	Run text-based goal fulfillment evaluators against the conversation transcript
Voice agent safety	Apply text-based guardrail evaluators (toxicity, bias) to the transcript
Image generation with text prompts	Use text evaluators to assess prompt adherence alongside image evaluators for visual quality

When a voice simulation completes, Netra produces both audio and transcript data. You can attach text evaluators to the same evaluation to evaluate the conversation content, while voice evaluators assess the audio delivery — giving you full coverage of both what was said and how it sounded.

Evaluator Library

Netra provides a library of 49 preconfigured evaluators across 12 categories, ready to use or customize:

Category	Modality	Representative Evaluators
Quality	Text	Answer Correctness, Answer Relevance, Faithfulness, Context Precision
Agentic	Text	Goal Accuracy, Goal Fulfillment, Information Elicitation
Guardrails	Text	Toxicity, Bias, Topic Adherence
Performance	Text	Latency, Cost, Token Usage
Semantic	Text	Semantic Similarity, SQL Semantic Equivalence
Tool Use	Text	Tool Correctness (exact, partial, sequence)
JSON Evaluator	Text	JSON key/value validation
Regex Evaluator	Text	Pattern matching
Multimodal	Image	Image Analysis, Image Editing Quality, Image-Text Coherence
TTS	Voice	Humanness, Pronunciation Accuracy, Speaking Rate
STT	Voice	Word Error Rate, Transcription Correctness, Time to First Transcript
Conversational	Voice	Backchannelling Naturalness, Expressiveness, Speaking Percentage

Creating Evaluators

You can create evaluators in two ways:

From the Library — Browse prebuilt evaluators, click Add, customize the prompt or parameters, and save to My Evaluators
Custom — Click Add Custom, choose the evaluator type (LLM as Judge, Code, or rule-based), and configure from scratch

Every evaluator supports Playground testing so you can validate it against sample data before attaching it to evaluations.

Getting Started

Choose Your Modality

Decide whether you’re evaluating text, voice, or image outputs — or a combination.

Browse the Library

Start with prebuilt evaluators from the Library. Customize thresholds and prompts to match your quality standards.

Test in Playground

Validate evaluators against sample data before deploying to production evaluations.

Attach to Evaluations

Connect evaluators to evaluations and map variables to your data fields.

Run and Analyze

Execute evaluations and analyze results in Test Runs. Iterate on prompts and thresholds.

Text Evaluators - LLM-as-Judge, code, and rule-based text evaluators
Voice Evaluators - TTS, STT, and conversational evaluators
Image Evaluators - Multimodal and rule-based image evaluators
Evaluation Overview - Evaluations, test runs, and the evaluation framework
Quick Start: Evaluation - Get started with evaluations in minutes

Get Started

Observability

Insights

Evaluations

Simulations

Evaluators

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Quick Start: Evaluation

Evaluator Categories

Text Evaluators

Voice Evaluators

Image Evaluators

Evaluator Types

Evaluator Library

Creating Evaluators

Getting Started

Quick Start: Evaluation

​Evaluator Categories

Text Evaluators

Voice Evaluators

Image Evaluators

​Evaluator Types

​Cross-Modal Evaluation

​Evaluator Library

​Creating Evaluators

​Getting Started

​Related

Evaluator Categories

Evaluator Types

Cross-Modal Evaluation

Evaluator Library

Creating Evaluators

Getting Started

Related