Prompt Studio
Stress Testing lives inside Prompt Studio. Learn how to build, test, and version prompts.
Why Stress Testing Matters
Without systematic reliability testing, a prompt that works once might fail unpredictably in production. Netra helps you answer critical questions with confidence:| Question | What Netra Measures |
|---|---|
| Will my prompt produce consistent outputs at scale? | Pass rates across 1–100 repeated executions per model |
| Which model delivers the best reliability for my use case? | Side-by-side scoring of up to 5 models on identical inputs |
| Does my prompt meet latency, cost, and quality requirements? | Per-run evaluations against configurable thresholds |
| Did my latest prompt edit introduce a regression? | Version-over-version reliability trends with score history |
| Is my prompt production-ready? | AI-generated health assessment (Healthy / Needs Optimization / Critical Issues) |
Core Building Blocks
The Stress Testing framework is built on three interconnected pillars:Models & Runs
Configure which models to test and how many executions to perform. Netra runs your prompt repeatedly against each selected model to surface inconsistencies that single runs miss.| Feature | Description |
|---|---|
| Multi-model selection | Test up to 5 models simultaneously in a single stress test |
| Run presets | Quick-select 1 (spot check), 25 (early signal), 50 (reliable), or 100 (conclusive) runs per model |
| Custom runs | Set any value from 1 to 100 runs per model |
| Model parameters | Override temperature, top P, max tokens, and penalties per model |
| Execution summary | See total executions before starting (models × runs per model) |
Your currently configured model in Prompt Studio is pre-selected as the first model. Add additional models to compare performance across providers like OpenAI, Anthropic, and Google.
Evaluators
Evaluators score every individual run against your quality and performance criteria. Netra provides 6 evaluator types:| Evaluator | What It Measures | Configuration Required |
|---|---|---|
| LLM-as-Judge | Subjective quality assessed by another LLM | Provider, model, optional reference answer |
| Latency | Response time in milliseconds | Expected latency threshold (ms) |
| Cost | Per-run execution cost | Expected cost threshold |
| Token Count | Total tokens consumed | Expected token count threshold |
| JSON Validation | Output matches expected JSON structure | Expected JSON object, optional keys to ignore |
| Regex Match | Output matches a regular expression | Regex pattern with optional flags |
Results & Analysis
After all runs complete, Netra aggregates scores and generates actionable insights:| Feature | Description |
|---|---|
| AI Analysis | LLM-generated summary with health badge (Healthy / Needs Optimization / Critical Issues) |
| Per-model metrics | Average latency, cost, tokens, and evaluator score for each model |
| Pass rate breakdown | Visual distribution of all-pass, all-fail, partial, and error outcomes |
| Radar chart | Multi-axis comparison of models across all evaluators |
| Evaluator breakdown table | Detailed pass/fail/score per evaluator per model |
| Run logs | Paginated, filterable list of every individual execution with expandable output |
| Reliability trends | Score history chart across versions (available after 2+ completed tests) |
Step-by-Step Guide
Prerequisites
- A prompt with at least one message in Prompt Studio
- All prompt variables filled with representative values
- At least one model provider configured in your workspace
Run Your First Stress Test
Open Stress Test Configuration
In Prompt Studio, ensure your prompt variables are filled with values that represent real usage. Click the Stress Test button next to the Run button in the actions panel.
Select Models (Step 1 of 2)
The configuration modal opens on Step 1: Configure. Your current model is pre-selected. Add more models to compare:
- Click Add Model to open the model picker
- Select a provider and model from the dropdown
- Add up to 5 models total
- Remove models by clicking the × on each model chip
Set Runs Per Model
Choose how many times each model should execute the prompt. More runs produce more statistically reliable results.
You can also enter a custom value (1–100). The footer shows total executions:
| Preset | Runs | Label | Best For |
|---|---|---|---|
| 1 | 1 | Spot check | Quick sanity check |
| 25 | 25 | Early signal | Initial quality assessment |
| 50 | 50 | Reliable | Production readiness (default) |
| 100 | 100 | Conclusive | High-confidence decisions |
3 models × 50 runs = 150 total executions
Configure Evaluators (Step 2 of 2)
Click Next to move to Step 2: Evaluators. Select which evaluators to use and configure each one:
- Toggle evaluators on/off using the checkbox on each card
- Expand a card to configure its settings
- Set pass criteria — either a numerical threshold with an operator or a boolean condition (is_true / is_false)
- For LLM-as-Judge, select the provider and model that will act as the judge
You must select at least one evaluator. The counter at the top shows how many evaluators are currently selected.
Start the Stress Test
Click Run at the bottom of the modal. The test starts immediately in the background:
- A progress banner appears above the action buttons showing real-time status
- Executions are processed asynchronously — you can continue editing your prompt
- Status polls automatically every 4 seconds
Monitor Progress
The banner updates in real-time:
- Running — Shows
Running · X/Y runswith an animated indicator - Multiple tests — Shows
N tests running · X/Y runswhen concurrent tests are active - Completion — A success toast appears with the overall score and a link to results
Track History & Reliability Trends
Access Run History via the banner link or “View past stress tests.” The history modal shows:
- A table of all past stress tests with version/draft, score, duration, status, and who ran it
- A Reliability Chart (appears after 2+ completed tests) showing average score trends over time
- Filter by version to compare performance across prompt iterations
- Stop any running test directly from the history
Review Results — Overview Tab
On clicking each row in Run History table, a new modal with two tabs will open. The Overview tab shows:AI Analysis — A generated summary with a health badge:
- 🟢 Healthy — Prompt is reliable and production-ready
- 🟡 Needs Optimization — Some issues detected; review recommendations
- 🔴 Critical Issues — Significant reliability problems; do not publish
Review Results — Logs Tab
Switch to the Logs tab for granular inspection of every individual run:
- Filter by model — Focus on a specific model’s runs
- Filter by eval status — Show only Passed, Partial, Failed, or Error runs
- Expand any row — See the full model output and per-evaluator scores with pass/fail reasons
Evaluators in Detail
LLM-as-Judge
Uses a separate LLM to evaluate the quality of each run’s output. The judge model reads the prompt, the output, and (optionally) a reference answer, then produces a score.| Setting | Description |
|---|---|
| Provider | The model provider for the judge (e.g., OpenAI) |
| Model | The specific judge model (e.g., gpt-4o-mini) |
| Reference Answer | Optional ground truth for more consistent scoring |
| Pass Criteria | Numerical score threshold (default: >= 0.5 on a 0–1 scale) |
Latency
Measures response time in milliseconds and checks against your expected threshold.| Setting | Description |
|---|---|
| Expected Latency | Maximum acceptable response time (ms) |
| Pass Criteria | Boolean — passes if actual is within expected |
Cost
Measures per-run execution cost and checks against your budget threshold.| Setting | Description |
|---|---|
| Expected Cost | Maximum acceptable cost per run |
| Pass Criteria | Boolean — passes if actual is within expected |
Token Count
Measures total tokens consumed (prompt + completion) and checks against your limit.| Setting | Description |
|---|---|
| Expected Tokens | Maximum acceptable token count |
| Pass Criteria | Boolean — passes if actual is within expected |
JSON Validation
Deep-compares the model output against an expected JSON structure to validate structured outputs.| Setting | Description |
|---|---|
| Expected JSON | The JSON object the output should match (required) |
| Keys to Ignore | Optional list of keys to exclude from comparison |
| Pass Criteria | Boolean — passes if structure matches |
Regex Match
Tests the model output against a regular expression pattern.| Setting | Description |
|---|---|
| Pattern | The regex pattern to match against the output |
| Flags | Optional regex flags (e.g., i for case-insensitive) |
| Pass Criteria | Boolean — passes if pattern matches output |
Understanding Pass Criteria
Every evaluator uses pass criteria to determine if a run passes or fails:Numerical Pass Criteria
For evaluators that produce a numeric score (like LLM-as-Judge):| Operator | Meaning | Example |
|---|---|---|
>= | Greater than or equal to | Score >= 0.8 |
<= | Less than or equal to | Score <= 0.3 |
> | Greater than | Score > 0.5 |
< | Less than | Score < 0.5 |
= | Equal to | Score = 1.0 |
Boolean Pass Criteria
For evaluators that produce a true/false result (latency, cost, token, JSON, regex):| Condition | Meaning |
|---|---|
is_true | Run passes if the evaluator returns true |
is_false | Run passes if the evaluator returns false |
Test Statuses
| Status | Meaning | Actions Available |
|---|---|---|
| Running | Executions in progress; banner shows live count | Stop (cancel) |
| Completed | All runs finished; results and AI analysis available | View results |
| Failed | One or more runs could not complete execution | View results |
| Partial | Some models completed, others failed | View results |
| Cancelled | Manually stopped by the user | View completed results |
Use Cases
Pre-Publish Validation
Ensure your prompt meets quality bars before releasing to production:- Finish editing your prompt in Prompt Studio
- Run a stress test with 50+ runs across your target production model
- Configure evaluators matching your quality criteria (e.g., LLM-as-Judge
>= 0.8, latency< 3000ms) - Review the AI health analysis — check if status is “Healthy”
Model Selection & Comparison
Choose the best model for your use case with data, not guesses:- Add all candidate models (up to 5) to a single stress test
- Set runs to 50+ for statistically meaningful results
- Compare the radar chart and per-model scores across evaluators
- Select the model with the best balance of quality, speed, and cost
Regression Detection
Catch quality degradation when iterating on prompts:- Run a stress test on your current published version as a baseline
- Create a draft with your proposed changes
- Run the same stress test configuration on the draft
- Compare scores in Run History — the reliability chart shows trends across versions
Best Practices
| Practice | Why It Matters |
|---|---|
| Run stress tests before every publish | Catches regressions before they reach production |
| Use at least 50 runs for production decisions | Small sample sizes produce misleading results |
| Test with multiple models | Reveals model-specific weaknesses you wouldn’t see otherwise |
| Configure evaluators relevant to your specific use case | Generic tests miss domain-specific failure modes |
| Track reliability across versions | Spots degradation trends before production impact |
| Provide reference answers for LLM-as-Judge when needed | Improves scoring consistency with a ground truth |
| Keep temperature low for deterministic tasks | Reduces output variance and produces cleaner test results |
FAQ
Can I run a stress test on a draft?
Can I run a stress test on a draft?
Yes. Stress tests work on both published versions and drafts. This lets you validate changes before publishing a new version.
How many models can I test at once?
How many models can I test at once?
Up to 5 models in a single stress test. Each model runs independently with its own parameter configuration (temperature, top P, max tokens, etc.).
What is the maximum number of runs per model?
What is the maximum number of runs per model?
You can run up to 100 executions per model. The UI provides quick presets (1, 25, 50, 100), but you can enter any custom value within the range.
Can I stop a running stress test?
Can I stop a running stress test?
Yes. Open the Run History modal and click the Stop button on the running test.
What does the AI Analysis measure?
What does the AI Analysis measure?
After all runs finish, an AI model analyzes the aggregated scores, pass rates, and per-model performance. It produces a natural-language summary explaining what went well and what needs attention, along with a health rating (Healthy, Needs Optimization, or Critical Issues).
Can I run multiple stress tests simultaneously?
Can I run multiple stress tests simultaneously?
Yes. The progress banner tracks multiple concurrent tests and shows aggregated progress. Each test runs independently in the background.
When does the reliability chart appear?
When does the reliability chart appear?
The reliability trend chart appears in Run History after you have 2 or more completed stress tests on the same prompt. It shows average score progression over time.
How is the overall score calculated?
How is the overall score calculated?
Each run’s score is the average of all evaluator scores for that run (boolean results become 1 for pass, 0 for fail). The overall stress test score is the average across all runs and all models.
What happens if a run fails during execution?
What happens if a run fails during execution?
If a model call fails, that run is marked as failed. The system retries up to 3 times with exponential backoff. If all retries fail, the run is recorded as failed and counted in the failure metrics. Other runs continue unaffected.
Related
- Prompt Studio — Build, test, and version prompts in a centralized workspace
- Evaluation Overview — Broader evaluation framework for datasets and test runs
- Simulation Overview — Test AI agents with multi-turn conversations
- Traces — Debug individual executions with full trace visibility