Simulation

The Netra SDK exposes a simulation client that lets you:

Run simulations - Execute multi-turn conversations against your AI agent
Define tasks - Create custom task implementations to wrap your agent
Use lifecycle hooks - Run setup/teardown scripts around scenarios (beforeAll, beforeEach, before, after, afterEach, afterAll)
Control concurrency - Manage parallel execution for throughput

This page shows how to use netra.simulation to run multi-turn simulations and test your AI agents programmatically.

Getting Started

The simulation client is available on the main Netra entry point after initialization.

import { Netra } from "netra-sdk-js";

const client = new Netra({
  apiKey: "your-api-key",
});

// Access the simulation client
await client.simulation.runSimulation(...);

runSimulation

Execute a multi-turn conversation simulation against a dataset. Your task function is called repeatedly for each turn until the conversation completes.

import { Netra } from "netra-sdk-js";
import {
  BaseTask,
  TaskResult,
  type SimulationHooks,
} from "netra-sdk-js/simulation";

const client = new Netra({ apiKey: "..." });

class MyAgentTask extends BaseTask {
  private agent: any;

  constructor(agent: any) {
    super();
    this.agent = agent;
  }

  async run(
    message: string,
    sessionId?: string | null,
    files?: any[] | null,
    setupContext?: Record<string, any> | null,
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await this.agent.chat(message, {
      sessionId,
      files,
      authToken: ctx.auth_token,
    });
    return {
      message: response.text,
      sessionId: sessionId || "default",
    };
  }
}

const hooks: SimulationHooks = {
  beforeAll: async () => ({ employee_id: "emp-1" }),
  beforeEach: async (shared) => ({
    auth_token: "tok-123",
  }),
};

const result = await client.simulation.runSimulation({
  name: "Customer Support Simulation",
  datasetId: "dataset-123",
  task: new MyAgentTask(myAgent),
  context: { environment: "staging" },
  maxConcurrency: 5,
  hooks,
});

if (result) {
  console.log(`Completed: ${result.completed.length}`);
  console.log(`Failed: ${result.failed.length}`);
}

runSimulation(options: SimulationOptions): Promise<SimulationResult | null>

// SimulationOptions interface
interface SimulationOptions {
  name: string;
  datasetId: string;
  task: BaseTask;
  context?: Record<string, any>;
  maxConcurrency?: number; // default: 5
  hooks?: SimulationHooks;
}

Parameters (SimulationOptions)

Parameter	Type	Description
`name`	`string`	Name of the simulation run (required)
`datasetId`	`string`	ID of the dataset containing conversation scenarios
`task`	`BaseTask`	Task instance that wraps your AI agent
`context`	`Record<string, any>?`	Optional context data passed to the simulation
`maxConcurrency`	`number`	Maximum parallel conversations (default: 5)
`hooks`	`SimulationHooks?`	Optional lifecycle hooks for setup/teardown (see Hooks)

Response: SimulationResult

Field	Type	Description
`success`	`boolean`	Overall success status
`totalItems`	`number`	Number of dataset items
`completed`	`ConversationResult[]`	Successfully completed items
`failed`	`ConversationResult[]`	Failed items with error details

Completed Item (ConversationResult)

Field	Type	Description
`runItemId`	`string`	Unique item identifier
`success`	`boolean`	Always `true` for completed items
`finalTurnId`	`string`	ID of the last turn in the conversation

Failed Item (ConversationResult)

Field	Type	Description
`runItemId`	`string`	Unique item identifier
`success`	`boolean`	Always `false` for failed items
`error`	`string`	Error message describing the failure
`turnId`	`string`	ID of the turn where failure occurred

BaseTask

Create a custom task by extending the BaseTask abstract class. Your implementation wraps your AI agent and handles the conversation flow.

import { BaseTask, TaskResult, type ProcessedFile } from "netra-sdk-js/simulation";

abstract class BaseTask {
  /**
   * Execute a single turn in the conversation
   * @param message - The input message from the simulation
   * @param sessionId - The session identifier
   * @param files - Optional list of base64-encoded file attachments from the
   *                dataset item. Will be `null` when no files are attached.
   * @param setupContext - Optional merged context from lifecycle hooks
   *                       (`beforeAll` + `beforeEach` + item `before`).
   *                       `null` when no hooks are configured. Existing tasks
   *                       that omit this parameter remain backwards compatible.
   * @returns The task result containing the response message and session ID
   */
  abstract run(
    message: string,
    sessionId?: string | null,
    files?: ProcessedFile[] | null,
    setupContext?: Record<string, any> | null,
  ): Promise<TaskResult> | TaskResult;
}

interface TaskResult {
  message: string;    // The response message from your agent
  sessionId: string;  // The session identifier for conversation continuity
}

Implementation Requirements

Requirement	Description
Extend `BaseTask`	Your class must extend the `BaseTask` abstract class
Implement `run()`	The `run()` method must return a `TaskResult`
Handle sessions	Maintain conversation context using the `sessionId`
Return message	Always return a response message in the `TaskResult`
Accept files	The `run()` method receives an optional `files` parameter containing base64-encoded attachments
Accept setup context	Optionally declare `setupContext` to receive merged hook context (tokens, accounts, etc.)

Hooks (Pre/Post Scripts)

Multi-turn datasets often share state across scenarios. Rather than merging scenarios or forcing sequential execution, use hooks to set up and tear down state around scenario runs. Hooks run entirely on the SDK side — script code never leaves your environment. Lightweight metadata (function name and description) is sent to the backend so the Netra UI can show which hooks are configured. In the Netra dashboard (multi-turn Scenario Run Details → Conversation tab), configured hooks appear in collapsible Pre-script / Post-script panels with each hook’s name and description. Item run status badges include Prescript Failed and Postscript Failed.

Hook points

Hook	When it runs	Failure effect
`beforeAll`	Once, before any scenario starts	Entire run is marked failed; no scenarios execute
`beforeEach`	Before every scenario	That scenario is marked `prescript_failed` (terminal; eval suppressed); others continue
`before`	Before specific scenarios only (keyed by `datasetItemId`)	That scenario is marked `prescript_failed` (terminal; eval suppressed); others continue
`after`	After specific scenarios only (keyed by `datasetItemId`)	If the scenario otherwise succeeded → `postscript_failed` (terminal; eval not suppressed). If already failed/`prescript_failed`, status is preserved
`afterEach`	After every scenario (after item-specific `after`)	Same as `after` — both always attempt to run; errors are combined when marking `postscript_failed`
`afterAll`	Once, after all scenarios complete	Successfully completed scenarios are marked `postscript_failed`; already-failed/`prescript_failed` items keep their status

Context passing

beforeAll()                            → sharedContext (Record | null)
beforeEach(sharedContext)              → merged into setupContext
before[datasetItemId](mergedContext)   → merged into setupContext (if registered)
BaseTask.run(..., setupContext)
after[datasetItemId](result, setupContext) (if registered)
afterEach(result, setupContext)
afterAll(results, sharedContext)

setupContext is the merge of beforeAll + beforeEach + item before. It is passed to BaseTask.run, item after, and afterEach. If a before hook fails mid-way, teardown still receives the furthest successfully built setupContext (so afterEach can clean up what beforeEach created). afterAll receives only the run-level sharedContext, but its results include setup/first-turn failures as well as conversation failures. Prefer beforeEach / afterEach when the same setup/teardown applies to every scenario. Use item-keyed before / after for scenario-specific logic.

SimulationHooks

import { type SimulationHooks } from "netra-sdk-js/simulation";

const hooks: SimulationHooks = {
  beforeAll: setupEnvironment,    // () => Record | null | void
  beforeEach: setupEach,          // (sharedContext) => Record | null | void
  before: {                       // Record<datasetItemId, (sharedContext) => Record | null | void>
    "refund-scenario-001": setupRefund,
  },
  after: {                        // Record<datasetItemId, (result, setupContext) => void>
    "refund-scenario-001": teardownRefund,
  },
  afterEach: teardownEach,        // (result, setupContext) => void
  afterAll: teardownEnvironment,  // (results, sharedContext) => void
};

All hooks can be sync or async. TypeScript has no runtime docstrings — set a .description property on each hook function so the Netra UI can display it (truncated to 200 characters).

Example

import { Netra } from "netra-sdk-js";
import {
  BaseTask,
  TaskResult,
  type SimulationHooks,
} from "netra-sdk-js/simulation";

async function beforeAll() {
  const employee = await api.createEmployee({ name: "Test User" });
  return { employee_id: employee.id };
}
beforeAll.description = "Create a test employee before any scenario runs.";

async function beforeEach(sharedContext: Record<string, any> | null) {
  const token = await api.login({
    employeeId: sharedContext?.employee_id,
  });
  return { auth_token: token };
}
beforeEach.description = "Obtain a fresh auth token before every scenario.";

async function setupRefund(sharedContext: Record<string, any> | null) {
  const account = await api.createRefundAccount();
  return { refund_account_id: account.id };
}
setupRefund.description = "Create a refund account for the refund scenario only.";

async function teardownRefund(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
) {
  await api.deleteRefundAccount(setupContext?.refund_account_id);
}
teardownRefund.description = "Delete the refund account after the refund scenario.";

async function afterEach(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
) {
  await api.logout({ token: setupContext?.auth_token });
}
afterEach.description = "Log out after every scenario.";

async function afterAll(
  results: Record<string, any>,
  sharedContext: Record<string, any> | null,
) {
  await api.deleteEmployee(sharedContext?.employee_id);
}
afterAll.description = "Delete the test employee once all scenarios finish.";

const hooks: SimulationHooks = {
  beforeAll,
  beforeEach,
  before: { "refund-scenario-001": setupRefund },
  after: { "refund-scenario-001": teardownRefund },
  afterEach,
  afterAll,
};

class MyAgentTask extends BaseTask {
  async run(
    message: string,
    sessionId?: string | null,
    files?: any[] | null,
    setupContext?: Record<string, any> | null,
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await myAgent.chat(message, {
      sessionId,
      authToken: ctx.auth_token,
    });
    return {
      message: response.text,
      sessionId: sessionId || "default",
    };
  }
}

const client = new Netra({ apiKey: "your-api-key" });

const result = await client.simulation.runSimulation({
  name: "Multi-Scenario Workflow Test",
  datasetId: "your-dataset-id",
  task: new MyAgentTask(),
  hooks,
});

Keys in before / after must be real datasetItemId values from your dataset. A prescript_failed item is terminal (eval suppressed); other scenarios continue. A postscript_failed item is also terminal, but eval is not suppressed — conversation evaluations remain valid. Wrap teardown in try/catch so cleanup errors do not mark otherwise-successful items postscript_failed.

Complete Example

import { Netra } from "netra-sdk-js";
import { BaseTask, TaskResult } from "netra-sdk-js/simulation";
import OpenAI from "openai";

async function main() {
  // Initialize
  const client = new Netra({
    apiKey: "your-api-key",
  });
  const openai = new OpenAI();

  // Store conversation history per session
  const conversations: Record<string, any[]> = {};

  class OpenAITask extends BaseTask {
    /**
     * Task that uses OpenAI for multi-turn conversations.
     */
    async run(message: string, sessionId?: string | null): Promise<TaskResult> {
      const session = sessionId || crypto.randomUUID();

      // Initialize conversation
      if (!conversations[session]) {
        conversations[session] = [
          {
            role: "system",
            content: "You are a helpful customer support agent.",
          },
        ];
      }

      // Add user message
      conversations[session].push({ role: "user", content: message });

      // Call OpenAI
      const response = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: conversations[session],
      });

      // Store and return response
      const content = response.choices[0].message.content || "";
      conversations[session].push({ role: "assistant", content });

      return {
        message: content,
        sessionId: session,
      };
    }
  }

  // Run simulation
  const result = await client.simulation.runSimulation({
    name: "Customer Support Test",
    datasetId: "your-dataset-id",
    task: new OpenAITask(),
    context: { model: "gpt-4o-mini", temperature: 0 },
    maxConcurrency: 5,
  });

  if (result) {
    // Analyze results
    console.log(`Total: ${result.totalItems}`);
    console.log(`Completed: ${result.completed.length}`);
    console.log(`Failed: ${result.failed.length}`);

    for (const failure of result.failed) {
      console.log(`  Failed ${failure.runItemId}: ${failure.error}`);
    }
  }
}

main();

Next Steps

Dashboard Query - Query dashboard metrics
Usage Utilities - Query traces and spans
Simulation Overview - Learn about simulation testing
Evaluation - Evaluate AI outputs

Python SDK

TypeScript SDK

Getting Started

runSimulation

Parameters (SimulationOptions)

Response: SimulationResult

BaseTask

Implementation Requirements

Hooks (Pre/Post Scripts)

Hook points

Context passing

SimulationHooks

Example

Complete Example

Next Steps

​Getting Started

​runSimulation

​Parameters (SimulationOptions)

​Response: SimulationResult

​BaseTask

​Implementation Requirements

​Hooks (Pre/Post Scripts)

​Hook points

​Context passing

​SimulationHooks

​Example

​Complete Example

​Next Steps

Getting Started

runSimulation

Parameters (SimulationOptions)

Response: SimulationResult

BaseTask

Implementation Requirements

Hooks (Pre/Post Scripts)

Hook points

Context passing

SimulationHooks

Example

Complete Example

Next Steps