Evaluation - Netra Docs

The Netra SDK exposes an evaluation client that lets you:

Manage datasets - Create datasets and add test items
Run test suites - Execute tasks against datasets with automatic tracing
Apply evaluators - Score outputs using built-in or custom evaluators
Fetch results - Retrieve detailed run results programmatically

This page shows how to use Netra.evaluation to manage datasets, run test suites, fetch results, and programmatically evaluate your AI applications.

Getting Started

The evaluation client is available on the main Netra entry point after initialization.

from netra import Netra

Netra.init(app_name="sample-app")

# Access the evaluation client
Netra.evaluation.create_dataset(...)
Netra.evaluation.add_dataset_item(...)
Netra.evaluation.get_all_datasets(...)
Netra.evaluation.get_dataset(...)
Netra.evaluation.run_test_suite(...)
Netra.evaluation.get_run_results(...)

create_dataset

Create an empty dataset that can hold test items for evaluation runs.

from netra import Netra
from netra.evaluation import TurnType

Netra.init(app_name="sample-app")

result = Netra.evaluation.create_dataset(
    name="Customer Support QA",
    tags=["support", "qa", "v1"],
    turn_type=TurnType.SINGLE,  # or TurnType.MULTI for multi-turn
)

print(f"Dataset created: {result.id}")
print(f"Name: {result.name}")
print(f"Tags: {result.tags}")

create_dataset(
    name: str,
    tags: Optional[List[str]] = None,
    turn_type: TurnType = TurnType.SINGLE,
) -> CreateDatasetResponse | None

Parameters

Parameter	Type	Description
`name`	`str`	Name of the dataset (required)
`tags`	`list[str]?`	Optional tags for categorization
`turn_type`	`TurnType`	`SINGLE` for single-turn or `MULTI` for multi-turn datasets

Response: CreateDatasetResponse

Field	Type	Description
`id`	`str`	Unique dataset identifier
`name`	`str`	Dataset name
`tags`	`list[str]`	Associated tags
`project_id`	`str`	Project identifier
`organization_id`	`str`	Organization identifier
`created_by`	`str`	Creator identifier
`updated_by`	`str`	Last updater identifier
`created_at`	`str`	Creation timestamp
`updated_at`	`str`	Last update timestamp
`deleted_at`	`str?`	Deletion timestamp (if soft-deleted)

TurnType

Value	Description
`TurnType.SINGLE`	Single-turn evaluation (one input → one output)
`TurnType.MULTI`	Multi-turn evaluation (conversation sequences)

add_dataset_item

Add a single test item to an existing dataset.

from netra import Netra
from netra.evaluation import DatasetItem

Netra.init(app_name="sample-app")

result = Netra.evaluation.add_dataset_item(
    dataset_id="dataset-123",
    item=DatasetItem(
        input="What is the return policy for electronics?",
        expected_output="Electronics can be returned within 30 days with original packaging.",
        tags=["policy", "returns"],
        metadata={"category": "electronics", "priority": "high"},
    ),
)

print(f"Item added: {result.id}")
print(f"Input: {result.input}")

add_dataset_item(
    dataset_id: str,
    item: DatasetItem,
) -> AddDatasetItemResponse | None

Parameters

Parameter	Type	Description
`dataset_id`	`str`	ID of the target dataset
`item`	`DatasetItem`	The test item to add

DatasetItem

Field	Type	Description
`input`	`Any`	The input to pass to your task (required)
`expected_output`	`Any?`	Expected output for comparison
`tags`	`list[str]?`	Optional tags for the item
`metadata`	`dict?`	Optional metadata for evaluators

Response: AddDatasetItemResponse

Field	Type	Description
`id`	`str`	Unique item identifier
`dataset_id`	`str`	Parent dataset ID
`project_id`	`str`	Project identifier
`organization_id`	`str`	Organization identifier
`source`	`str`	Source of the item
`source_id`	`str?`	Source reference ID
`input`	`Any`	The input value
`expected_output`	`Any`	The expected output
`is_active`	`bool`	Whether the item is active
`tags`	`list[str]`	Associated tags
`metadata`	`dict?`	Item metadata
`created_by`	`str`	Creator identifier
`updated_by`	`str`	Last updater identifier
`created_at`	`str`	Creation timestamp
`updated_at`	`str`	Last update timestamp
`deleted_at`	`str?`	Deletion timestamp (if soft-deleted)

get_all_datasets

Fetch all datasets, optionally filtered by a specific tag.

from netra import Netra

Netra.init(app_name="sample-app")

# Fetch all datasets
all_datasets = Netra.evaluation.get_all_datasets()

# Fetch only datasets that include a specific tag
tagged_datasets = Netra.evaluation.get_all_datasets(tag="production")

if tagged_datasets:
    print(f"Found {len(tagged_datasets.datasets)} dataset(s)")
    for dataset in tagged_datasets.datasets:
        print(f"{dataset.id} | {dataset.name} | {dataset.tags}")

get_all_datasets(
    tag: Optional[str] = None,
) -> GetAllDatasetsResponse | None

Parameters

Parameter	Type	Description
`tag`	`str?`	Optional tag to filter datasets. If omitted, all datasets are returned.

Response: GetAllDatasetsResponse

Field	Type	Description
`datasets`	`list[CreateDatasetResponse]`	List of datasets returned by the backend.

CreateDatasetResponse

Field	Type	Description
`id`	`str`	Unique dataset identifier
`name`	`str`	Dataset name
`tags`	`list[str]`	Associated tags
`turn_type`	`str?`	Turn type (`single` or `multi`)
`dataset_type`	`str?`	Dataset type value from backend
`project_id`	`str`	Project identifier
`organization_id`	`str`	Organization identifier
`created_by`	`str`	Creator identifier
`updated_by`	`str`	Last updater identifier
`created_at`	`str`	Creation timestamp
`updated_at`	`str`	Last update timestamp
`deleted_at`	`str?`	Deletion timestamp (if soft-deleted)

get_dataset

Retrieve a dataset and all its items by ID.

from netra import Netra

Netra.init(app_name="sample-app")

dataset = Netra.evaluation.get_dataset(dataset_id="dataset-123")

print(f"Total items: {len(dataset.items)}")

for item in dataset.items:
    print(f"ID: {item.id}")
    print(f"Input: {item.input}")
    print(f"Expected: {item.expected_output}")
    print("---")

get_dataset(
    dataset_id: str,
) -> GetDatasetItemsResponse | None

Parameters

Parameter	Type	Description
`dataset_id`	`str`	ID of the dataset to retrieve

Response: GetDatasetItemsResponse

Field	Type	Description
`items`	`list[DatasetRecord]`	List of dataset items

DatasetRecord

Field	Type	Description
`id`	`str`	Item identifier
`dataset_id`	`str`	Parent dataset ID
`input`	`Any`	The input value
`expected_output`	`Any`	The expected output

run_test_suite

Execute a test suite against a dataset, running your task function on each item and optionally applying evaluators.

from netra import Netra
from openai import OpenAI

Netra.init(app_name="sample-app")

client = OpenAI()

def my_task(input_data):
    """Task function that processes each dataset item."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": input_data},
        ],
    )
    return response.choices[0].message.content

# Get dataset
dataset = Netra.evaluation.get_dataset(dataset_id="dataset-123")

# Run test suite
result = Netra.evaluation.run_test_suite(
    name="GPT-4o Mini Evaluation",
    data=dataset,
    task=my_task,
    evaluators=[exact_match],  # Optional local evaluators, see Local Evaluators below
    max_concurrency=10,
)

print(f"Run ID: {result['runId']}")
print(f"Items processed: {len(result['items'])}")

run_test_suite(
    name: str,
    data: Dataset,
    task: Callable[[Any], Any],
    evaluators: Optional[List[Any]] = None,
    max_concurrency: int = 50,
) -> Optional[Dict[str, Any]]

Parameters

Parameter	Type	Description
`name`	`str`	Name for this test run (required)
`data`	`Dataset`	Dataset from `get_dataset()`
`task`	`Callable`	Function that takes input and returns output
`evaluators`	`list[BaseEvaluator]?`	Optional local evaluators that score each completed item in your process
`max_concurrency`	`int`	Max parallel task executions (default: 50)

Response

Field	Type	Description
`runId`	`str`	Unique run identifier
`items`	`list[dict]`	Results for each processed item

Item Result

Field	Type	Description
`index`	`int`	Item index in dataset
`status`	`str`	`"completed"` or `"failed"`
`traceId`	`str`	Trace ID for observability
`spanId`	`str`	Span ID for the task execution
`testRunItemId`	`str`	Backend item identifier

The task function receives the input field from each dataset item. Return the output that should be compared against expected_output by evaluators.

Local Evaluators

A local evaluator is a Python class that scores each item’s output in your own process. run_test_suite then sends the scores to Netra, and they appear in the test run next to the platform evaluator scores. Use local evaluators for checks you want to keep in code, such as exact matching, JSON parsing, or calls to your own scoring model. Subclass BaseEvaluator, implement evaluate, and create an instance with an EvaluatorConfig.

from netra import Netra
from netra.evaluation import (
    BaseEvaluator,
    EvaluatorConfig,
    EvaluatorContext,
    EvaluatorOutput,
    ScoreType,
)

Netra.init(app_name="sample-app")


class ExactMatchEvaluator(BaseEvaluator):
    def evaluate(self, context: EvaluatorContext) -> EvaluatorOutput:
        is_match = str(context.task_output).strip() == str(context.expected_output).strip()
        return EvaluatorOutput(
            evaluator_name=self.config.name,
            result=is_match,
            is_passed=is_match,
            reason="Output matches the expected output" if is_match else "Output differs from the expected output",
        )


exact_match = ExactMatchEvaluator(
    EvaluatorConfig(
        name="exact_match",
        label="Exact Match",
        score_type=ScoreType.BOOLEAN,
    )
)

result = Netra.evaluation.run_test_suite(
    name="FAQ Agent Exact Match",
    data=dataset,
    task=my_task,
    evaluators=[exact_match],
)

class BaseEvaluator(ABC):
    def __init__(self, config: EvaluatorConfig) -> None: ...

    @abstractmethod
    def evaluate(self, context: EvaluatorContext) -> EvaluatorOutput: ...

The SDK stores the config on the instance as self.config. evaluate can be a regular method or an async def method. The SDK awaits async evaluators automatically.

EvaluatorConfig

Field	Type	Description
`name`	`str`	Unique identifier for the evaluator (required). Must match `evaluator_name` in every `EvaluatorOutput` the evaluator returns.
`label`	`str`	Display name for the evaluator (required)
`score_type`	`ScoreType`	The kind of score the evaluator produces (required). Also accepted as `scoreType`.

ScoreType

Value	Description
`ScoreType.BOOLEAN`	Pass/fail result, for example `True` or `False`
`ScoreType.NUMERICAL`	Numeric score, for example `0.85`
`ScoreType.CATEGORICAL`	Category label, for example `"partially_correct"`

EvaluatorContext

The SDK passes an EvaluatorContext to evaluate for each item.

Field	Type	Description
`input`	`Any`	The item input passed to your `task` function
`task_output`	`Any`	The value your `task` function returned
`expected_output`	`Any`	The item’s expected output, or `None` if not set
`metadata`	`dict?`	The item’s metadata. Set only for items you build locally with `DatasetItem`, such as `LocalDataset(items=[...])`. Always `None` for items fetched with `get_dataset()`.

EvaluatorOutput

Field	Type	Description
`evaluator_name`	`str`	Must equal `config.name`. Use `self.config.name` so the two can’t drift apart.
`result`	`Any`	The score value. Return a value that matches the evaluator’s `score_type`.
`is_passed`	`bool`	Whether the item passed this evaluator
`reason`	`str?`	Optional explanation shown with the score

How local evaluators run

run_test_suite registers each evaluator’s config with the test run when it creates the run.
Evaluators run only on items whose task completed. Items whose task raised an exception are marked failed and are not evaluated.
For each item, the SDK waits up to 60 seconds for the item’s span to be ingested. It then runs the evaluators one after another, in list order. Different items are evaluated concurrently.
run_test_suite returns after every evaluator has finished.

Local evaluator failures are silent. If evaluate raises an exception, or returns an evaluator_name that doesn’t match config.name, the SDK drops that evaluator’s result for the item and continues the run. Check the test run in the Netra dashboard to confirm that each evaluator produced scores.

The evaluators parameter accepts evaluator instances only. The SDK ignores strings such as evaluator names or IDs. Evaluators configured in the Netra dashboard are attached to the dataset and run on the platform, so you don’t pass them to run_test_suite.

get_run_results

Fetch detailed results for a completed test run by its run ID. Use this after run_test_suite to retrieve evaluation scores, item-level outcomes, and other run metadata from the backend.

from netra import Netra

Netra.init(app_name="sample-app")

# After running a test suite
result = Netra.evaluation.run_test_suite(
    name="GPT-4o Mini Evaluation",
    data=dataset,
    task=my_task,
    evaluators=[exact_match],
)

run_id = result["runId"]

# Fetch the full run results
run_results = Netra.evaluation.get_run_results(run_id=run_id)

print(f"Run results: {run_results}")

get_run_results(
    run_id: str,
) -> Any | None

Parameters

Parameter	Type	Description
`run_id`	`str`	The unique identifier of the test run to fetch results for (required)

Response

Returns the JSON response from the backend containing the full run results, including evaluation scores and item-level details. Returns None if the run_id is empty or the request fails. The top-level response wraps the run data:

Field	Type	Description
`success`	`bool`	Whether the request succeeded
`data`	`RunResult`	The run result object (see below)
`error`	`Any?`	Error details, `null` on success
`meta`	`dict`	Request metadata (timestamp, path, version, status code)

RunResult

Field	Type	Description
`id`	`str`	Unique run identifier
`name`	`str`	Name of the test run
`projectId`	`str`	Project identifier
`organizationId`	`str`	Organization identifier
`status`	`str`	Run status
`evaluationStatus`	`str`	Evaluation status
`time`	`str`	Timestamp when the run was initiated
`turnType`	`str`	`"single"` or `"multi"`
`createdAt`	`str`	Creation timestamp
`updatedAt`	`str`	Last update timestamp
`deletedAt`	`str?`	Deletion timestamp (if soft-deleted)
`runContext`	`Any?`	Optional run context
`testRunSummary`	`TestRunSummary`	Aggregated summary of the run
`redirectUrl`	`str`	Direct link to the run in the Netra dashboard

TestRunSummary

Field	Type	Description
`totalItems`	`int`	Total number of items in the run
`passedItems`	`int`	Number of items that passed
`failedItems`	`int`	Number of items that failed
`durationMs`	`float`	Total run duration in milliseconds
`totalCostUsd`	`float`	Total cost of the run in USD
`averageLatencyMs`	`float`	Average latency per item in milliseconds

Example Response

{
    "success": true,
    "data": {
        "id": "4dd93b23-a769-401e-96c3-db42f408b65b",
        "name": "Image Fidelity Test",
        "projectId": "b0e6f0f3-b3fb-4d73-aea4-a75fbb2d72f3",
        "organizationId": "b4a00d6b-52ff-4f2b-be84-db8bd2d5f657",
        "status": "completed",
        "evaluationStatus": "completed",
        "time": "2026-04-02T10:45:36.751Z",
        "turnType": "single",
        "createdAt": "2026-04-02T10:45:36.761Z",
        "updatedAt": "2026-04-02T10:47:48.271Z",
        "deletedAt": null,
        "runContext": null,
        "testRunSummary": {
            "totalItems": 1,
            "passedItems": 1,
            "failedItems": 0,
            "durationMs": 7978,
            "totalCostUsd": 0.024752200000000002,
            "averageLatencyMs": 136562
        },
        "redirectUrl": "https://demo.getnetra.ai/test-runs/4dd93b23-a769-401e-96c3-db42f408b65b?currentOrg=b4a00d6b-52ff-4f2b-be84-db8bd2d5f657&currentProject=b0e6f0f3-b3fb-4d73-aea4-a75fbb2d72f3"
    },
    "error": null,
    "meta": {
        "timestamp": "2026-04-20T05:51:22.075Z",
        "path": "/evaluations/run/4dd93b23-a769-401e-96c3-db42f408b65b",
        "version": "1.0.0",
        "statusCode": 200
    }
}

Pair get_run_results with run_test_suite to programmatically inspect evaluation outcomes. The run_id is available in the runId field of the run_test_suite response. Use redirectUrl from the response to jump directly to the run in the Netra dashboard.

When to Use Which API

Dataset Management

create_dataset / add_dataset_item / get_all_datasets / get_datasetBuild and manage test datasets programmatically. Use for CI/CD pipelines or when generating test cases from production data.

Test Execution

run_test_suiteExecute your AI task against a dataset with automatic tracing and evaluation. Use for regression testing and model comparisons.

Results Retrieval

get_run_resultsFetch detailed results for a completed run, including evaluation scores and item-level outcomes. Use for post-run analysis and CI/CD assertions.

Advanced Workflows

create_runCreate runs without immediate execution. Use when you need custom orchestration or want to manage run lifecycle separately.

Local Evaluators

BaseEvaluator / EvaluatorConfigPass local evaluator instances to run_test_suite to score outputs in code. Configure platform evaluators in the Netra dashboard.

Complete Example

from netra import Netra
from netra.evaluation import (
    BaseEvaluator,
    DatasetItem,
    EvaluatorConfig,
    EvaluatorContext,
    EvaluatorOutput,
    ScoreType,
    TurnType,
)
from openai import OpenAI

# Initialize
Netra.init(
    app_name="evaluation-demo",
    headers="x-api-key=your-api-key",
)
client = OpenAI()

# 1. Create a dataset
dataset_response = Netra.evaluation.create_dataset(
    name="Product FAQ Evaluation",
    tags=["faq", "products", "v2"],
    turn_type=TurnType.SINGLE,
)
dataset_id = dataset_response.id
print(f"Created dataset: {dataset_id}")

# 2. Add test items
test_cases = [
    {
        "input": "What is your return policy?",
        "expected_output": "Items can be returned within 30 days.",
    },
    {
        "input": "How long does shipping take?",
        "expected_output": "Standard shipping takes 3-5 business days.",
    },
    {
        "input": "Do you offer international shipping?",
        "expected_output": "Yes, we ship to over 50 countries.",
    },
]

for case in test_cases:
    Netra.evaluation.add_dataset_item(
        dataset_id=dataset_id,
        item=DatasetItem(
            input=case["input"],
            expected_output=case["expected_output"],
        ),
    )
print(f"Added {len(test_cases)} test items")

# 3. Define the task
def faq_agent(query: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a customer support agent. Answer concisely."},
            {"role": "user", "content": query},
        ],
    )
    return response.choices[0].message.content

# 4. Define a local evaluator
class ConcisenessEvaluator(BaseEvaluator):
    def evaluate(self, context: EvaluatorContext) -> EvaluatorOutput:
        word_count = len(str(context.task_output).split())
        is_concise = word_count <= 50
        return EvaluatorOutput(
            evaluator_name=self.config.name,
            result=is_concise,
            is_passed=is_concise,
            reason=f"Answer has {word_count} words (limit 50)",
        )

conciseness = ConcisenessEvaluator(
    EvaluatorConfig(name="conciseness", label="Conciseness", score_type=ScoreType.BOOLEAN)
)

# 5. Run the test suite
dataset = Netra.evaluation.get_dataset(dataset_id=dataset_id)

result = Netra.evaluation.run_test_suite(
    name="FAQ Agent v2 Evaluation",
    data=dataset,
    task=faq_agent,
    evaluators=[conciseness],
    max_concurrency=5,
)

# 6. Review results
print(f"\nRun completed: {result['runId']}")
for item in result["items"]:
    print(f"  Item {item['index']}: {item['status']} (trace: {item['traceId']})")

# 7. Fetch detailed run results
run_results = Netra.evaluation.get_run_results(run_id=result["runId"])
print(f"\nDetailed run results: {run_results}")

print("\nView detailed results in Netra dashboard → Evaluation → Test Runs")

Next Steps

Dashboard Query - Query dashboard metrics
Usage Utilities - Query traces and spans
Evaluators - Configure custom evaluators
Test Runs - View and analyze test run results

​Getting Started

​create_dataset

​Parameters

​Response: CreateDatasetResponse

​add_dataset_item

​Parameters

​DatasetItem

​Response: AddDatasetItemResponse

​get_all_datasets

​Parameters

​Response: GetAllDatasetsResponse

​CreateDatasetResponse

​get_dataset

​Parameters

​Response: GetDatasetItemsResponse

​DatasetRecord

​run_test_suite

​Parameters

​Response

​Item Result

​Local Evaluators

​EvaluatorConfig

​ScoreType

​EvaluatorContext

​EvaluatorOutput

​How local evaluators run

​get_run_results

​Parameters

​Response

​RunResult

​TestRunSummary

​When to Use Which API

Dataset Management

Test Execution

Results Retrieval

Advanced Workflows

Local Evaluators

​Complete Example

​Next Steps

Getting Started

create_dataset

Parameters

Response: CreateDatasetResponse

add_dataset_item

Parameters

DatasetItem

Response: AddDatasetItemResponse

get_all_datasets

Parameters

Response: GetAllDatasetsResponse

CreateDatasetResponse

get_dataset

Parameters

Response: GetDatasetItemsResponse

DatasetRecord

run_test_suite

Parameters

Response

Item Result

Local Evaluators

EvaluatorConfig

ScoreType

EvaluatorContext

EvaluatorOutput

How local evaluators run

get_run_results

Parameters

Response

RunResult

TestRunSummary

When to Use Which API

Complete Example

Next Steps