Stress Testing

Prompts don’t fail in obvious ways — they drift, produce inconsistent outputs, and degrade quietly across different models and edge cases. Netra’s Stress Testing framework makes that invisible unreliability visible, giving you a structured, repeatable way to measure prompt performance across models, runs, and evaluators before every publish.

Prompt Studio

Stress Testing lives inside Prompt Studio. Learn how to build, test, and version prompts.

Why Stress Testing Matters

Without systematic reliability testing, a prompt that works once might fail unpredictably in production. Netra helps you answer critical questions with confidence:

Question	What Netra Measures
Will my prompt produce consistent outputs at scale?	Pass rates across 1–100 repeated executions per model
Which model delivers the best reliability for my use case?	Side-by-side scoring of up to 5 models on identical inputs
Does my prompt meet latency, cost, and quality requirements?	Per-run evaluations against configurable thresholds
Did my latest prompt edit introduce a regression?	Version-over-version reliability trends with score history
Is my prompt production-ready?	AI-generated health assessment (Healthy / Needs Optimization / Critical Issues)

Core Building Blocks

The Stress Testing framework is built on three interconnected pillars:

Models & Runs

Configure which models to test and how many executions to perform. Netra runs your prompt repeatedly against each selected model to surface inconsistencies that single runs miss.

Feature	Description
Multi-model selection	Test up to 5 models simultaneously in a single stress test
Run presets	Quick-select 1 (spot check), 25 (early signal), 50 (reliable), or 100 (conclusive) runs per model
Custom runs	Set any value from 1 to 100 runs per model
Model parameters	Override temperature, top P, max tokens, and penalties per model
Execution summary	See total executions before starting (models × runs per model)

Your currently configured model in Prompt Studio is pre-selected as the first model. Add additional models to compare performance across providers like OpenAI, Anthropic, and Google.

Evaluators

Evaluators score every individual run against your quality and performance criteria. Netra provides 6 evaluator types:

Evaluator	What It Measures	Configuration Required
LLM-as-Judge	Subjective quality assessed by another LLM	Provider, model, optional reference answer
Latency	Response time in milliseconds	Expected latency threshold (ms)
Cost	Per-run execution cost	Expected cost threshold
Token Count	Total tokens consumed	Expected token count threshold
JSON Validation	Output matches expected JSON structure	Expected JSON object, optional keys to ignore
Regex Match	Output matches a regular expression	Regex pattern with optional flags

Each evaluator supports pass criteria — numerical thresholds (e.g., latency < 2000ms) or boolean conditions (pass/fail) — so you define exactly what “good” means for your use case.

Results & Analysis

After all runs complete, Netra aggregates scores and generates actionable insights:

Feature	Description
AI Analysis	LLM-generated summary with health badge (Healthy / Needs Optimization / Critical Issues)
Per-model metrics	Average latency, cost, tokens, and evaluator score for each model
Pass rate breakdown	Visual distribution of all-pass, all-fail, partial, and error outcomes
Radar chart	Multi-axis comparison of models across all evaluators
Evaluator breakdown table	Detailed pass/fail/score per evaluator per model
Run logs	Paginated, filterable list of every individual execution with expandable output
Reliability trends	Score history chart across versions (available after 2+ completed tests)

Step-by-Step Guide

Prerequisites

A prompt with at least one message in Prompt Studio
All prompt variables filled with representative values
At least one model provider configured in your workspace

Run Your First Stress Test

Open Stress Test Configuration

In Prompt Studio, ensure your prompt variables are filled with values that represent real usage. Click the Stress Test button next to the Run button in the actions panel.

All prompt variables must be filled before starting a stress test. If any variables are empty, the button will trigger a validation error.

Select Models (Step 1 of 2)

The configuration modal opens on Step 1: Configure. Your current model is pre-selected. Add more models to compare:

Click Add Model to open the model picker
Select a provider and model from the dropdown
Add up to 5 models total
Remove models by clicking the × on each model chip

Set Runs Per Model

Choose how many times each model should execute the prompt. More runs produce more statistically reliable results.

Preset	Runs	Label	Best For
1	1	Spot check	Quick sanity check
25	25	Early signal	Initial quality assessment
50	50	Reliable	Production readiness (default)
100	100	Conclusive	High-confidence decisions

You can also enter a custom value (1–100). The footer shows total executions:

3 models × 50 runs = 150 total executions

Configure Evaluators (Step 2 of 2)

Click Next to move to Step 2: Evaluators. Select which evaluators to use and configure each one:

Toggle evaluators on/off using the checkbox on each card
Expand a card to configure its settings
Set pass criteria — either a numerical threshold with an operator or a boolean condition (is_true / is_false)
For LLM-as-Judge, select the provider and model that will act as the judge

You must select at least one evaluator. The counter at the top shows how many evaluators are currently selected.

Start the Stress Test

Click Run at the bottom of the modal. The test starts immediately in the background:

A progress banner appears above the action buttons showing real-time status
Executions are processed asynchronously — you can continue editing your prompt
Status polls automatically every 4 seconds

Also, you can click View Results button to open Results modal and can stop any running Stress Test

Monitor Progress

The banner updates in real-time:

Running — Shows Running · X/Y runs with an animated indicator
Multiple tests — Shows N tests running · X/Y runs when concurrent tests are active
Completion — A success toast appears with the overall score and a link to results

You can also access past and running tests anytime via the View past stress tests link.

Track History & Reliability Trends

Access Run History via the banner link or “View past stress tests.” The history modal shows:

A table of all past stress tests with version/draft, score, duration, status, and who ran it
A Reliability Chart (appears after 2+ completed tests) showing average score trends over time
Filter by version to compare performance across prompt iterations
Stop any running test directly from the history

Review Results — Overview Tab

On clicking each row in Run History table, a new modal with two tabs will open. The Overview tab shows:AI Analysis — A generated summary with a health badge:

🟢 Healthy — Prompt is reliable and production-ready
🟡 Needs Optimization — Some issues detected; review recommendations
🔴 Critical Issues — Significant reliability problems; do not publish

Per-Model Metrics — A grid showing each model’s average latency, total cost, total tokens, and average evaluator score.Pass Rate Breakdown — A stacked bar per model showing the distribution of all-pass, all-fail, error, and partial outcomes.Radar Chart — A visual comparison of all models across every evaluator axis.Evaluator Breakdown Table — Tabular view of exact scores per model per evaluator.

Review Results — Logs Tab

Switch to the Logs tab for granular inspection of every individual run:

Filter by model — Focus on a specific model’s runs
Filter by eval status — Show only Passed, Partial, Failed, or Error runs
Expand any row — See the full model output and per-evaluator scores with pass/fail reasons

Evaluators in Detail

LLM-as-Judge

Uses a separate LLM to evaluate the quality of each run’s output. The judge model reads the prompt, the output, and (optionally) a reference answer, then produces a score.

Setting	Description
Provider	The model provider for the judge (e.g., OpenAI)
Model	The specific judge model (e.g., gpt-4o-mini)
Reference Answer	Optional ground truth for more consistent scoring
Pass Criteria	Numerical score threshold (default: `>= 0.5` on a 0–1 scale)

Latency

Measures response time in milliseconds and checks against your expected threshold.

Setting	Description
Expected Latency	Maximum acceptable response time (ms)
Pass Criteria	Boolean — passes if actual is within expected

Cost

Measures per-run execution cost and checks against your budget threshold.

Setting	Description
Expected Cost	Maximum acceptable cost per run
Pass Criteria	Boolean — passes if actual is within expected

Token Count

Measures total tokens consumed (prompt + completion) and checks against your limit.

Setting	Description
Expected Tokens	Maximum acceptable token count
Pass Criteria	Boolean — passes if actual is within expected

JSON Validation

Deep-compares the model output against an expected JSON structure to validate structured outputs.

Setting	Description
Expected JSON	The JSON object the output should match (required)
Keys to Ignore	Optional list of keys to exclude from comparison
Pass Criteria	Boolean — passes if structure matches

Regex Match

Tests the model output against a regular expression pattern.

Setting	Description
Pattern	The regex pattern to match against the output
Flags	Optional regex flags (e.g., `i` for case-insensitive)
Pass Criteria	Boolean — passes if pattern matches output

Understanding Pass Criteria

Every evaluator uses pass criteria to determine if a run passes or fails:

Numerical Pass Criteria

For evaluators that produce a numeric score (like LLM-as-Judge):

Operator	Meaning	Example
`>=`	Greater than or equal to	`Score >= 0.8`
`<=`	Less than or equal to	`Score <= 0.3`
`>`	Greater than	`Score > 0.5`
`<`	Less than	`Score < 0.5`
`=`	Equal to	`Score = 1.0`

Boolean Pass Criteria

For evaluators that produce a true/false result (latency, cost, token, JSON, regex):

Condition	Meaning
`is_true`	Run passes if the evaluator returns true
`is_false`	Run passes if the evaluator returns false

Test Statuses

Status	Meaning	Actions Available
Running	Executions in progress; banner shows live count	Stop (cancel)
Completed	All runs finished; results and AI analysis available	View results
Failed	One or more runs could not complete execution	View results
Partial	Some models completed, others failed	View results
Cancelled	Manually stopped by the user	View completed results

Use Cases

Pre-Publish Validation

Ensure your prompt meets quality bars before releasing to production:

Finish editing your prompt in Prompt Studio
Run a stress test with 50+ runs across your target production model
Configure evaluators matching your quality criteria (e.g., LLM-as-Judge >= 0.8, latency < 3000ms)
Review the AI health analysis — check if status is “Healthy”

Model Selection & Comparison

Choose the best model for your use case with data, not guesses:

Add all candidate models (up to 5) to a single stress test
Set runs to 50+ for statistically meaningful results
Compare the radar chart and per-model scores across evaluators
Select the model with the best balance of quality, speed, and cost

Regression Detection

Catch quality degradation when iterating on prompts:

Run a stress test on your current published version as a baseline
Create a draft with your proposed changes
Run the same stress test configuration on the draft
Compare scores in Run History — the reliability chart shows trends across versions

Best Practices

Practice	Why It Matters
Run stress tests before every publish	Catches regressions before they reach production
Use at least 50 runs for production decisions	Small sample sizes produce misleading results
Test with multiple models	Reveals model-specific weaknesses you wouldn’t see otherwise
Configure evaluators relevant to your specific use case	Generic tests miss domain-specific failure modes
Track reliability across versions	Spots degradation trends before production impact
Provide reference answers for LLM-as-Judge when needed	Improves scoring consistency with a ground truth
Keep temperature low for deterministic tasks	Reduces output variance and produces cleaner test results

FAQ

Can I run a stress test on a draft?

Yes. Stress tests work on both published versions and drafts. This lets you validate changes before publishing a new version.

How many models can I test at once?

Up to 5 models in a single stress test. Each model runs independently with its own parameter configuration (temperature, top P, max tokens, etc.).

What is the maximum number of runs per model?

You can run up to 100 executions per model. The UI provides quick presets (1, 25, 50, 100), but you can enter any custom value within the range.

Can I stop a running stress test?

Yes. Open the Run History modal and click the Stop button on the running test.

What does the AI Analysis measure?

After all runs finish, an AI model analyzes the aggregated scores, pass rates, and per-model performance. It produces a natural-language summary explaining what went well and what needs attention, along with a health rating (Healthy, Needs Optimization, or Critical Issues).

Can I run multiple stress tests simultaneously?

Yes. The progress banner tracks multiple concurrent tests and shows aggregated progress. Each test runs independently in the background.

When does the reliability chart appear?

The reliability trend chart appears in Run History after you have 2 or more completed stress tests on the same prompt. It shows average score progression over time.

How is the overall score calculated?

Each run’s score is the average of all evaluator scores for that run (boolean results become 1 for pass, 0 for fail). The overall stress test score is the average across all runs and all models.

What happens if a run fails during execution?

If a model call fails, that run is marked as failed. The system retries up to 3 times with exponential backoff. If all retries fail, the run is recorded as failed and counted in the failure metrics. Other runs continue unaffected.

Prompt Studio — Build, test, and version prompts in a centralized workspace
Evaluation Overview — Broader evaluation framework for datasets and test runs
Simulation Overview — Test AI agents with multi-turn conversations
Traces — Debug individual executions with full trace visibility

Get Started

Observability

Insights

Evaluation

Simulation

Red Teaming

Prompt Management

Monitoring & Dashboard

Account settings

Migrations

Prompt Studio

Why Stress Testing Matters