Evaluation

The Netra SDK exposes an evaluation client that lets you:

Manage datasets - Create datasets and add test items
Run test suites - Execute tasks against datasets with automatic tracing
Apply evaluators - Score outputs using built-in or custom evaluators

This page shows how to use netra.evaluation to manage datasets, run test suites, and programmatically evaluate your AI applications.

Getting Started

The evaluation client is available on the main Netra entry point after initialization.

import { Netra } from "netra-sdk-js";

const client = new Netra({
  apiKey: "your-api-key",
});

// Access the evaluation client
await client.evaluation.createDataset(...);
await client.evaluation.addDatasetItem(...);
await client.evaluation.getDataset(...);
await client.evaluation.runTestSuite(...);

createDataset

Create an empty dataset that can hold test items for evaluation runs.

import { Netra } from "netra-sdk-js";

const client = new Netra({ apiKey: "..." });

const result = await client.evaluation.createDataset(
  "Customer Support QA",           // name
  ["support", "qa", "v1"]          // tags (optional)
);

if (result) {
  console.log(`Dataset created: ${result.id}`);
  console.log(`Name: ${result.name}`);
  console.log(`Tags: ${result.tags}`);
}

createDataset(
  name: string,
  tags?: string[]
): Promise<CreateDatasetResponse | null>

Parameters

Parameter	Type	Description
`name`	`string`	Name of the dataset (required)
`tags`	`string[]?`	Optional tags for categorization

Response: CreateDatasetResponse

Field	Type	Description
`id`	`string`	Unique dataset identifier
`name`	`string`	Dataset name
`tags`	`string[]`	Associated tags
`projectId`	`string`	Project identifier
`organizationId`	`string`	Organization identifier
`createdBy`	`string`	Creator identifier
`updatedBy`	`string`	Last updater identifier
`createdAt`	`string`	Creation timestamp
`updatedAt`	`string`	Last update timestamp
`deletedAt`	`string \| null`	Deletion timestamp (if soft-deleted)

addDatasetItem

Add a single test item to an existing dataset.

import { Netra } from "netra-sdk-js";

const client = new Netra({ apiKey: "..." });

const result = await client.evaluation.addDatasetItem(
  "dataset-123",  // datasetId
  {               // item
    input: "What is the return policy for electronics?",
    expectedOutput: "Electronics can be returned within 30 days with original packaging.",
    tags: ["policy", "returns"],
    metadata: { category: "electronics", priority: "high" },
  }
);

if (result) {
  console.log(`Item added: ${result.id}`);
  console.log(`Input: ${result.input}`);
}

addDatasetItem(
  datasetId: string,
  item: DatasetEntry
): Promise<AddDatasetItemResponse | null>

// DatasetEntry interface
interface DatasetEntry {
  input: any;
  expectedOutput?: any;
  tags?: string[];
  metadata?: Record<string, any>;
}

Parameters

Parameter	Type	Description
`datasetId`	`string`	ID of the target dataset
`item`	`DatasetEntry`	The test item to add

DatasetEntry

Field	Type	Description
`input`	`any`	The input to pass to your task (required)
`expectedOutput`	`any?`	Expected output for comparison
`tags`	`string[]?`	Optional tags for the item
`metadata`	`Record<string, any>?`	Optional metadata for evaluators

Response: AddDatasetItemResponse

Field	Type	Description
`id`	`string`	Unique item identifier
`datasetId`	`string`	Parent dataset ID
`projectId`	`string`	Project identifier
`organizationId`	`string`	Organization identifier
`source`	`string`	Source of the item
`sourceId`	`string?`	Source reference ID
`input`	`any`	The input value
`expectedOutput`	`any`	The expected output
`isActive`	`boolean`	Whether the item is active
`tags`	`string[]`	Associated tags
`metadata`	`Record<string, any>?`	Item metadata
`createdBy`	`string`	Creator identifier
`updatedBy`	`string`	Last updater identifier
`createdAt`	`string`	Creation timestamp
`updatedAt`	`string`	Last update timestamp
`deletedAt`	`string?`	Deletion timestamp (if soft-deleted)

getDataset

Retrieve a dataset and all its items by ID.

import { Netra } from "netra-sdk-js";

const client = new Netra({ apiKey: "..." });

const dataset = await client.evaluation.getDataset("dataset-123");

if (dataset) {
  console.log(`Total items: ${dataset.items.length}`);

  for (const item of dataset.items) {
    console.log(`ID: ${item.id}`);
    console.log(`Input: ${item.input}`);
    console.log(`Expected: ${item.expectedOutput}`);
    console.log("---");
  }
}

getDataset(
  datasetId: string
): Promise<GetDatasetItemsResponse | null>

Parameters

Parameter	Type	Description
`datasetId`	`string`	ID of the dataset to retrieve

Response: GetDatasetItemsResponse

Field	Type	Description
`items`	`DatasetRecord[]`	List of dataset items

DatasetRecord

Field	Type	Description
`id`	`string`	Item identifier
`datasetId`	`string`	Parent dataset ID
`input`	`any`	The input value
`expectedOutput`	`any`	The expected output

runTestSuite

Execute a test suite against a dataset, running your task function on each item and optionally applying evaluators.

import { Netra } from "netra-sdk-js";
import OpenAI from "openai";

const client = new Netra({ apiKey: "..." });
const openai = new OpenAI();

// Task function that processes each dataset item
async function myTask(inputData: any): Promise<string> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: inputData },
    ],
  });
  return response.choices[0].message.content || "";
}

// Get dataset
const dataset = await client.evaluation.getDataset("dataset-123");

if (dataset) {
  // Run test suite
  const result = await client.evaluation.runTestSuite(
    "GPT-4o Mini Evaluation",  // name
    dataset,                    // data
    myTask,                     // task
    [exactMatch],               // local evaluators (optional), see Local Evaluators below
    10                          // maxConcurrency
  );

  if (result) {
    console.log(`Run ID: ${result.runId}`);
    console.log(`Items processed: ${result.items.length}`);
  }
}

runTestSuite(
  name: string,
  data: Dataset,
  task: TaskFunction,
  evaluators?: any[],
  maxConcurrency?: number
): Promise<Record<string, any> | null>

// TaskFunction type
type TaskFunction = (input: any) => any | Promise<any>

Parameters

Parameter	Type	Description
`name`	`string`	Name for this test run (required)
`data`	`Dataset`	Dataset from `getDataset()`
`task`	`TaskFunction`	Function that takes input and returns output
`evaluators`	`LocalEvaluator[]?`	Optional local evaluators that score each completed item in your process
`maxConcurrency`	`number`	Max parallel task executions (default: 50)

Response

Field	Type	Description
`runId`	`string`	Unique run identifier
`items`	`object[]`	Results for each processed item

Item Result

Field	Type	Description
`index`	`number`	Item index in dataset
`status`	`string`	`"completed"` or `"failed"`
`traceId`	`string`	Trace ID for observability
`spanId`	`string`	Span ID for the task execution
`testRunItemId`	`string`	Backend item identifier

The task function receives the input field from each dataset item. Return the output that should be compared against expectedOutput by evaluators.

Local Evaluators

A local evaluator is an object that scores each item’s output in your own process. runTestSuite then sends the scores to Netra, and they appear in the test run next to the platform evaluator scores. Use local evaluators for checks you want to keep in code, such as exact matching, JSON parsing, or calls to your own scoring model. A local evaluator is any object with a config property and an evaluate method.

import { Netra } from "netra-sdk-js";

const client = new Netra({ apiKey: "..." });

const exactMatch = {
  config: {
    name: "exact_match",
    label: "Exact Match",
    scoreType: "boolean",
  },
  evaluate(context: { taskOutput: any; expectedOutput?: any }) {
    const isMatch =
      String(context.taskOutput).trim() === String(context.expectedOutput).trim();
    return {
      evaluatorName: this.config.name,
      result: isMatch,
      isPassed: isMatch,
      reason: isMatch
        ? "Output matches the expected output"
        : "Output differs from the expected output",
    };
  },
};

const result = await client.evaluation.runTestSuite(
  "FAQ Agent Exact Match",
  dataset,
  myTask,
  [exactMatch]
);

// Shape runTestSuite expects for each entry in `evaluators`
interface LocalEvaluator {
  config: EvaluatorConfig;
  evaluate(
    context: EvaluatorContext
  ): EvaluatorOutput | Promise<EvaluatorOutput>;
}

interface EvaluatorConfig {
  name: string;
  label: string;
  scoreType: "boolean" | "numerical" | "categorical";
}

interface EvaluatorContext {
  input: any;
  taskOutput: any;
  expectedOutput?: any;
  metadata?: Record<string, any>;
}

interface EvaluatorOutput {
  evaluatorName: string;
  result: any;
  isPassed: boolean;
  reason?: string;
}

The package doesn’t export these evaluator types or the ScoreType enum. Define the interfaces in your own code, as shown in the Signature tab, or use object literals. Pass scoreType as a string.

evaluate can return the output directly or return a Promise. The SDK awaits promises automatically.

EvaluatorConfig

Field	Type	Description
`name`	`string`	Unique identifier for the evaluator (required). Must match `evaluatorName` in every output the evaluator returns.
`label`	`string`	Display name for the evaluator (required)
`scoreType`	`string`	The kind of score the evaluator produces (required). One of the values below.

Score Types

Value	Description
`"boolean"`	Pass/fail result, for example `true` or `false`
`"numerical"`	Numeric score, for example `0.85`
`"categorical"`	Category label, for example `"partially_correct"`

EvaluatorContext

The SDK passes this context to evaluate for each item.

Field	Type	Description
`input`	`any`	The item input passed to your `task` function
`taskOutput`	`any`	The value your `task` function returned
`expectedOutput`	`any?`	The item’s expected output
`metadata`	`Record<string, any>?`	The item’s metadata. Set only for items you build locally as `DatasetItem` objects. Always `undefined` for items fetched with `getDataset()`.

EvaluatorOutput

Field	Type	Description
`evaluatorName`	`string`	Must equal `config.name`. Use `this.config.name` so the two can’t drift apart.
`result`	`any`	The score value. Return a value that matches the evaluator’s `scoreType`.
`isPassed`	`boolean`	Whether the item passed this evaluator
`reason`	`string?`	Optional explanation shown with the score

How local evaluators run

runTestSuite registers each evaluator’s config with the test run when it creates the run.
Evaluators run only on items whose task completed. Items whose task threw an error are marked failed and are not evaluated.
For each item, the SDK waits up to 60 seconds for the item’s span to be ingested. It then runs the evaluators one after another, in array order. Different items are evaluated concurrently.
runTestSuite resolves after every evaluator has finished.

Local evaluator failures are silent. If evaluate throws, or returns an evaluatorName that doesn’t match config.name, the SDK drops that evaluator’s result for the item and continues the run. Check the test run in the Netra dashboard to confirm that each evaluator produced scores.

The evaluators parameter accepts evaluator objects only. The SDK ignores strings such as evaluator names or IDs. Evaluators configured in the Netra dashboard are attached to the dataset and run on the platform, so you don’t pass them to runTestSuite.

When to Use Which API

Dataset Management

createDataset / addDatasetItem / getDatasetBuild and manage test datasets programmatically. Use for CI/CD pipelines or when generating test cases from production data.

Test Execution

runTestSuiteExecute your AI task against a dataset with automatic tracing and evaluation. Use for regression testing and model comparisons.

Advanced Workflows

createRunCreate runs without immediate execution. Use when you need custom orchestration or want to manage run lifecycle separately.

Local Evaluators

config + evaluatePass local evaluator objects to runTestSuite to score outputs in code. Configure platform evaluators in the Netra dashboard.

Complete Example

import { Netra } from "netra-sdk-js";
import OpenAI from "openai";

async function main() {
  // Initialize
  const client = new Netra({
    apiKey: "your-api-key",
  });
  const openai = new OpenAI();

  // 1. Create a dataset
  const datasetResponse = await client.evaluation.createDataset(
    "Product FAQ Evaluation",
    ["faq", "products", "v2"]
  );

  if (!datasetResponse) {
    console.error("Failed to create dataset");
    return;
  }

  const datasetId = datasetResponse.id;
  console.log(`Created dataset: ${datasetId}`);

  // 2. Add test items
  const testCases = [
    {
      input: "What is your return policy?",
      expectedOutput: "Items can be returned within 30 days.",
    },
    {
      input: "How long does shipping take?",
      expectedOutput: "Standard shipping takes 3-5 business days.",
    },
    {
      input: "Do you offer international shipping?",
      expectedOutput: "Yes, we ship to over 50 countries.",
    },
  ];

  for (const testCase of testCases) {
    await client.evaluation.addDatasetItem(datasetId, {
      input: testCase.input,
      expectedOutput: testCase.expectedOutput,
    });
  }
  console.log(`Added ${testCases.length} test items`);

  // 3. Define the task
  async function faqAgent(query: string): Promise<string> {
    const response = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        {
          role: "system",
          content: "You are a customer support agent. Answer concisely.",
        },
        { role: "user", content: query },
      ],
    });
    return response.choices[0].message.content || "";
  }

  // 4. Define a local evaluator
  const conciseness = {
    config: { name: "conciseness", label: "Conciseness", scoreType: "boolean" },
    evaluate(context: { taskOutput: any }) {
      const wordCount = String(context.taskOutput).split(/\s+/).length;
      const isConcise = wordCount <= 50;
      return {
        evaluatorName: this.config.name,
        result: isConcise,
        isPassed: isConcise,
        reason: `Answer has ${wordCount} words (limit 50)`,
      };
    },
  };

  // 5. Run the test suite
  const dataset = await client.evaluation.getDataset(datasetId);

  if (dataset) {
    const result = await client.evaluation.runTestSuite(
      "FAQ Agent v2 Evaluation",
      dataset,
      faqAgent,
      [conciseness],
      5
    );

    // 6. Review results
    if (result) {
      console.log(`\nRun completed: ${result.runId}`);
      for (const item of result.items) {
        console.log(
          `  Item ${item.index}: ${item.status} (trace: ${item.traceId})`
        );
      }

      console.log(
        "\nView detailed results in Netra dashboard → Evaluation → Test Runs"
      );
    }
  }
}

main();

Next Steps

Dashboard Query - Query dashboard metrics
Usage Utilities - Query traces and spans
Evaluators - Configure custom evaluators
Test Runs - View and analyze test run results

Python SDK

TypeScript SDK

Getting Started

createDataset

Parameters

Response: CreateDatasetResponse

addDatasetItem

Parameters

DatasetEntry

Response: AddDatasetItemResponse

getDataset

Parameters

Response: GetDatasetItemsResponse

DatasetRecord

runTestSuite

Parameters

Response

Item Result

Local Evaluators

EvaluatorConfig

Score Types

EvaluatorContext

EvaluatorOutput

How local evaluators run

When to Use Which API

Dataset Management

Test Execution

Advanced Workflows

Local Evaluators

Complete Example

Next Steps

​Getting Started

​createDataset

​Parameters

​Response: CreateDatasetResponse

​addDatasetItem

​Parameters

​DatasetEntry

​Response: AddDatasetItemResponse

​getDataset

​Parameters

​Response: GetDatasetItemsResponse

​DatasetRecord

​runTestSuite

​Parameters

​Response

​Item Result

​Local Evaluators

​EvaluatorConfig

​Score Types

​EvaluatorContext

​EvaluatorOutput

​How local evaluators run

​When to Use Which API

Dataset Management

Test Execution

Advanced Workflows

Local Evaluators

​Complete Example

​Next Steps

Getting Started

createDataset

Parameters

Response: CreateDatasetResponse

addDatasetItem

Parameters

DatasetEntry

Response: AddDatasetItemResponse

getDataset

Parameters

Response: GetDatasetItemsResponse

DatasetRecord

runTestSuite

Parameters

Response

Item Result

Local Evaluators

EvaluatorConfig

Score Types

EvaluatorContext

EvaluatorOutput

How local evaluators run

When to Use Which API

Complete Example

Next Steps