# Organization Settings
Source: https://docs.getnetra.ai/Account-settings/Organisation
Manage your Netra organization settings, team members, projects, API keys, and billing. Control access and configure workspace-level preferences.
The Organization Settings page serves as the central hub for managing all aspects of your Netra organization. From here, you can control member access, project configurations, billing details, and provider integrations in one unified interface.
## General
Manage your organization's core identity and access:
* **Organization Name**: Edit your organization's display name
* **Organization ID**: View and copy your unique organization identifier
## Members
Manage organization members and their access levels:
* View all members with their details:
* Name and email
* Role within the organization
* Membership date
* Account status
* Project memberships
### Member Management
#### Editing Member Access
* To modify a member's permissions, click the edit icon next to their name in the members list. You can assign one of three roles:
* **Project Member**: Basic access to assigned projects with view and limited edit permissions
* **Project Admin**: Can manage project settings and members within their assigned projects
* **Organization Owner**: Full administrative access to all organization settings and projects
* You can also decide the projects the member has access to by selecting/deselecting the projects from the dropdown in the project section.
A member can be a part of multiple projects.
#### Inviting New Members
To add new team members:
Click the "Invite Members" button in the top-right corner.
In the invitation modal:
* Enter the email address of the person you want to invite. Enter multiple email addresses by pressing **Enter** after each address.
* Select their role from the dropdown menu
* Choose which projects they should have access to
Click "Send Invitation" to dispatch an email with joining instructions.
The invited member will receive an email with instructions to join your organization. You can track the status of pending invitations in the members list.
If you need to add multiple members to same project with same role, you can add all their email addresses together in the invitation modal.
## Projects
Manage all projects within your organization:
* View project details:
* Project name and ID
* Creation date
* Status (Active/Inactive)
### Project Management
* **Create New Project**:
* Enter project name
* Auto-generated project ID
Project names and IDs cannot be edited after creation. The project ID will be visible in the projects list after creation which can be copied for further use.
## Usage & Billing
Track and manage your organization's resource usage and subscription:
* **Current Plan**: View your active plan (Developer/Enterprise)
* **Span Usage**: View your organization's span usage:
* Included in plan
* Total organization usage
* Additional usage costs
* **Usage Analytics**: View your organization's span usage analytics:
* Spans by instrumentation
* Spans by project
* Spans over time
## Provider
Manage your AI model providers and their configurations in one place. This section allows you to connect and configure different AI providers that your organization will use for various AI tasks.
## Available Adapters
Netra supports the following provider adapters:
* **Mistral AI**
* **OpenAI**
* **Google GenAI**
* **Anthropic**
## Provider Management
### Viewing Provider Details
For each configured provider, you can view:
* **Provider Name**: Your reference name for this configuration
* **Adapter Type**: The AI service being used (Mistral AI, OpenAI, Google GenAI, or Anthropic)
* **Base URL**: The endpoint URL for API requests (if custom)
* **API Key**: Indicates if a valid API key is configured
* **Action items**: An edit and delete button
### Editing Provider Configuration
Click the **Edit Provider** button to modify advanced settings:
Click the edit icon next to the provider you want to configure.
* **Base URL**: The custom endpoint URL for API requests
* *Note*: Only modify if you're using a custom endpoint or proxy
* **Embedding Models** (Not available for Anthropic):
* Add custom embedding model names
* Format: `model-name` or `provider/model-name`
* **LLM Judge Models**:
* Add custom model names for evaluation
* Used for automated quality assessment
Click "Save" to apply your configuration.
### Adding a New Provider
Click the **Add Provider** button.
* **Provider Name**: Enter a descriptive name (e.g., "Production OpenAI")
* **Adapter**: Select from the dropdown (Mistral AI, OpenAI, Google GenAI, or Anthropic)
* **API Key**: Enter your API key
* **Base URL**: Only required for custom endpoints or proxies
* **Custom Models**:
* Add embedding models (except for Anthropic)
* Configure LLM judge models for evaluation
Click **Create Provider** to save your configuration.
## Configuration Details
### Base URL
* **What it is**: The endpoint URL where API requests are sent
* **When to set**: Only required if you're using a custom endpoint or proxy
* **Default**: Automatically uses the official provider endpoint if left blank
### Embedding Models
* **What it is**: Custom models for generating text embeddings
* **Availability**: Not available for Anthropic provider
* **Use case**: When you need to use custom or fine-tuned embedding models
### LLM Judge Custom Models
* **What it is**: Models used for automated evaluation and quality assessment
* **Use case**: Running automated tests and quality checks on model outputs
# Project Settings
Source: https://docs.getnetra.ai/Account-settings/project
Configure your Netra project settings, manage team members, set up data exports, and connect integrations. Control project-level access and API keys.
Your project's command center. Manage team access, API keys, model configurations, and data exports — all from one place.
## General
Set up the basics for your project:
* **Project Name**: Edit your project's display name
* **Project ID**: View and copy your unique project identifier (read-only)
**Want to switch projects?** Head to the **Analytics Dashboard** and click your current **Project Name** at the top of the left nav panel. A dropdown will reveal all your available projects — just pick one to switch instantly.
## Members
See who has access to your project at a glance. This section shows **Project Members** and **Project Admins** — the people actively collaborating within this project.
For each member, you can view:
| Detail | Description |
| ---------------- | -------------------------------------- |
| **Name & Email** | Member's identity and contact |
| **Role** | Project Member or Project Admin |
| **Access Level** | Permissions granted within the project |
| **Last Active** | When they were last seen |
Looking for **Organization Owners**? They won't appear here. Head over to [Organization Settings — Members](/Account-settings/Organisation#members) to view and manage them.
### Adding Members
Click **"Add Members"** to bring new collaborators into the project — search by name or email, select the members you need, and hit save. They'll be notified right away.
## API Keys
Your keys to programmatic access. Each API key listed here shows its name, prefix, creation date, last usage, and quick actions to edit or delete.
### Creating a New API Key
Click **"Create Secret Key"** in the API Keys section.
Give it a descriptive name so you can easily identify its purpose later.
Copy the generated key immediately and store it somewhere safe — the full key won't be shown again.
### Managing Existing Keys
* **Edit**: Rename a key for better identification
* **Delete**: Revoke access by removing keys you no longer need
## Models
Keep tabs on every language model in your project. At a glance, you can track token consumption and cost of usage across all configured models.
### Adding a New Model
Click **"Add Model"** to begin.
* **Model Name**: The name used in API calls and for tracking price changes.
* **Match Pattern**: Define how this model matches incoming data using Postgres regex syntax.
* Example: `(?i)^(modelname)$` for exact case-insensitive matching.
Add rate cards for the usage types that apply — input tokens, output tokens, caching, audio duration, or character count.
### Managing Existing Models
* **Duplicate**: Need a similar model? Duplicate an existing one and tweak the parameters and pricing to fit.
## Contact Points
Decide where your alerts land — email inboxes, Slack channels, or both. Contact points are the destinations that get notified when something needs your attention.
### Setting Up a Contact Point
Click **"Create Contact Point"** to get started.
Choose how you want to be notified:
* **Email** — alerts sent directly to specified addresses
* **Slack** — notifications posted to your team's workspace
* For **Email**: Enter the recipient addresses
* For **Slack**: Connect your workspace and pick a channel
Fire off a test notification to make sure everything works, then save your configuration.
## Usage
See exactly how your project is performing. The Usage section gives you a real-time view of resource consumption through **Span Analytics**:
| Metric | What it tells you |
| ---------------------- | ---------------------------------------------------- |
| **Total Spans** | Overall volume of spans generated |
| **Trends Over Time** | How usage is changing day to day |
| **By Instrumentation** | Which instrumentations are driving the most activity |
| **Performance** | Key performance indicators at a glance |
## Integration
Integrations allow you to connect external services to your project for exporting and storing observability data generated by Netra. Once configured, these integrations can be used as destinations for scheduled exports, enabling long-term storage, external analysis, or compliance workflows outside the Netra platform.
Netra supports major cloud platforms for seamless integration with your existing infrastructure.
***
### Supported Providers
Export data to Amazon S3 buckets with IAM-based authentication.
Export data to Google Cloud Storage buckets with service account authentication.
### Adding an Integration
Navigate to **Project Settings** → **Integration** and click **Add Integration**.
Choose your cloud provider:
| Provider | Description |
| -------- | ----------------------------------------------------- |
| **AWS** | Amazon S3 storage with access key authentication |
| **GCP** | Google Cloud Storage with service account credentials |
Enter your storage bucket information:
| Field | Description | Example |
| --------------- | ---------------------------------------------- | ---------------------------------------- |
| **Bucket Name** | The name of your cloud storage bucket | `my-company-netra-exports` |
| **Region** | Geographic region where your bucket is located | `us-east-1` (AWS) or `us-central1` (GCP) |
Enter your access credentials based on the selected provider:
**Your credentials are secure.** All access keys and secrets are encrypted using industry-standard encryption and stored securely. We never expose or log your credentials in plain text.
| Field | Description |
| --------------------- | ---------------------------------- |
| **Access Key ID** | Your AWS IAM access key identifier |
| **Secret Access Key** | Your AWS IAM secret access key |
**Minimum Permissions Required:**
| Permission | Resource |
| ----------------- | -------------- |
| `s3:ListBucket` | Bucket ARN |
| `s3:PutObject` | Bucket objects |
| `s3:GetObject` | Bucket objects |
| `s3:DeleteObject` | Bucket objects |
We recommend creating a dedicated IAM user with only these permissions scoped to the target bucket.
| Field | Description |
| ----------------------- | --------------------------------------------------- |
| **Service Account Key** | JSON key file contents for your GCP service account |
**Minimum Permissions Required:**
| Role | Resource |
| --------------------------- | ------------- |
| `roles/storage.objectAdmin` | Target bucket |
Create a service account with only the **Storage Object Admin** role scoped to the target bucket.
Click **Save Integration** to store your configuration. Netra will verify the connection by testing bucket access.
Ensure your bucket has the correct permissions before saving. Integration verification will fail if Netra cannot write to the specified bucket.
***
### Integration Listing
The Integration page displays all configured cloud storage connections:
| Column | Description |
| ----------------- | ------------------------------- |
| **Provider Type** | The cloud provider (AWS or GCP) |
| **Bucket Name** | The target storage bucket name |
***
## Export
Configure automated data exports to send your observability data to external storage at regular intervals. Exports allow you to maintain long-term archives, integrate with data warehouses, or comply with data retention requirements.
Export hourly or daily based on your needs.
Choose JSON or CSV format for compatibility with your tools.
Optionally export your complete historical data on first run.
***
### Creating an Export
Navigate to **Project Settings** → **Export** and click **Create Export**.
Configure the fundamental export settings:
| Field | Description |
| ---------------------- | -------------------------------------------------------- |
| **Export Name** | A descriptive name to identify this export configuration |
| **Destination Bucket** | Select from your configured integrations |
Choose the output format for your exported data:
| Format | Best For |
| -------- | --------------------------------------------------------------------- |
| **JSON** | Data pipelines, programmatic processing, preserving nested structures |
| **CSV** | Spreadsheet analysis, SQL imports, business intelligence tools |
Set how often data should be exported:
| Frequency | Description |
| ---------- | ------------------------ |
| **Hourly** | Export data every hour |
| **Daily** | Export data once per day |
Choose whether to include historical data:
| Option | Description |
| ------- | --------------------------------------------------------------------- |
| **Yes** | Include all historical data from project creation in the first export |
| **No** | Only export new data from the export creation time onwards |
Exporting full history may result in a large initial export depending on your project's data volume. Subsequent exports will only include incremental data.
Configure the folder structure for exported files:
| Field | Description | Example |
| --------------- | ------------------------------------------------------------ | --------------------------- |
| **Path Prefix** | Directory path within your bucket where files will be stored | `netra/exports/production/` |
**Resulting file path:** `s3://your-bucket/netra/exports/production/2026-01-29/traces_001.json`
Review all settings and click **Create Export** to activate the export schedule.
***
### Export Listing
The Export page displays all configured export jobs with the following information:
| Column | Description |
| ------------------ | -------------------------------------------------- |
| **Export Name** | The name you assigned to identify this export |
| **File Format** | Output format (JSON or CSV) |
| **Path Prefix** | Directory path within the destination bucket |
| **Frequency** | Export schedule (Hourly or Daily) |
| **Status** | Current state of the export (Active, Paused, etc.) |
| **Created At** | When the export configuration was created |
| **Last Synced At** | Timestamp of the most recent successful export |
# Alert Rules
Source: https://docs.getnetra.ai/Alert-rules/Alert-rules
Configure Netra alert rules to receive Slack or email notifications when your AI system exceeds cost, latency, error rate, or quality score thresholds.
Don't wait for users to report problems. Alert Rules proactively monitor your AI operations and notify you the moment something goes wrong—whether it's runaway costs, degraded performance, or elevated error rates. Catch issues before they impact users.
New to alerts? Set up your first alert rule in minutes.
## Why Alert Rules Matter
AI systems can fail silently. Costs creep up, latency degrades, and errors accumulate without obvious symptoms:
| Risk | How Alerts Help |
| --------------------------- | --------------------------------------------------------------------------- |
| **Cost Overruns** | Get notified when token spend exceeds thresholds per request or time period |
| **Performance Degradation** | Alert when latency spikes above acceptable levels |
| **Error Rate Spikes** | Detect when failures exceed normal baselines |
| **SLA Violations** | Monitor metrics that matter for your service commitments |
## Alert Rules Dashboard
Navigate to **Alert Rules** from the left navigation panel.
| Column | Description |
| ------------------ | ----------------------------------------- |
| **Alert Name** | Identifier for the alert rule |
| **Status** | Active or disabled |
| **Last Triggered** | When the alert most recently fired |
| **Actions** | Enable, disable, edit, or delete the rule |
## Creating Alert Rules
Click **Create Alert Rule** in the top right corner.
| Field | Description |
| --------------- | --------------------------------------------------------- |
| **Alert Name** | A descriptive name (e.g., "High Cost Alert - Production") |
| **Description** | Optional details about what the alert monitors |
Select where notifications should be sent when the alert triggers.
Contact points must be configured first in **Settings → Contact Points**. See [Configuring Contact Points](#configuring-contact-points) below.
Choose what level to monitor:
| Scope | Use Case |
| --------- | ---------------------------------------------------------- |
| **Trace** | Monitor entire requests end-to-end |
| **Span** | Monitor individual operations (LLM calls, tool executions) |
Choose what to measure:
* **Cost**: Token/API spend in USD
* **Latency**: Response time in milliseconds
* **Error Rate**: Percentage of failed requests
* **Token Count**: Input/output token usage
Narrow down which traces trigger the alert by filtering on:
* **Model**: Specific AI model (e.g., gpt-4, claude-3)
* **Tenant ID**: Monitor specific customers or organizations
* **Environment**: Production, staging, development
* **Service**: Particular microservice or component
Filters help reduce noise by limiting alerts to specific contexts. For example, monitor production costs separately from development.
Define when the alert should fire:
* **Threshold**: The value that triggers the alert
* **Operator**: Greater than, less than, equals
* **Time Window**: Evaluation period (optional)
Click **Create** to activate the alert rule.
## How Alerts Work
When trigger conditions are met, Netra:
1. **Evaluates** incoming [traces](/Observability/Traces/overview) against your alert rules
2. **Triggers** the alert when conditions match
3. **Sends notifications** to all configured contact points
4. **Updates** the alert's status and last triggered timestamp
Alerts evaluate in real-time as traces arrive. There's no polling delay—you're notified immediately when thresholds are breached.
## Configuring Contact Points
Before creating alerts, set up your notification channels.
Go to **Settings → Contact Points**.
Click **Create Contact Point** and provide:
* **Name**: A descriptive label (e.g., "Engineering Slack", "On-Call Email")
* **Integration**: Choose Email or Slack
Enter the email address(es) to receive notifications.
* Supports multiple recipients (comma-separated)
* Notifications include alert details, triggered values, and links to relevant traces
Choose your integration method:
**Option 1: Slack API**
* Provide the recipient channel or user
* Enter your Slack Bot token
**Option 2: Webhook URL**
* Create an [Incoming Webhook](https://api.slack.com/messaging/webhooks) in Slack
* Paste the webhook URL
Click **Create** to save the contact point. It's now available when creating alert rules.
## Use Cases
### Cost Monitoring
Prevent budget overruns:
1. Create an alert for **Cost > \$0.50** per trace
2. Set scope to **Trace** to monitor full request cost
3. Route to your finance or engineering Slack channel
### Latency SLAs
Ensure performance commitments:
1. Create an alert for **Latency > 3000ms**
2. Set scope to **Trace** for end-to-end latency
3. Notify your on-call team via email
### Error Detection
Catch failures early:
1. Create an alert for **Error Rate > 5%**
2. Set a time window to avoid false positives from single failures
3. Route to your incident management system
### Per-Tenant Monitoring
For [multi-tenant](/Observability/Tenants) applications:
1. Create alerts scoped to specific tenant IDs
2. Monitor per-tenant cost or error rates
3. Proactively reach out before customers report issues
## Managing Alerts
### Enable/Disable Alerts
Toggle alerts on or off without deleting them:
* Useful for maintenance windows
* Temporarily silence noisy alerts while investigating
### Edit Alert Rules
Modify existing alerts to:
* Adjust thresholds as your system scales
* Add or remove contact points
* Change scope or metrics
### Delete Alerts
Remove alerts that are no longer needed. This action is permanent.
## Best Practices
### Setting Thresholds
* **Start conservative**: Begin with higher thresholds and tighten over time
* **Use baseline data**: Review your [Dashboard](/Dashboard/Custom-dashboard) metrics to understand normal ranges
* **Account for variance**: Set thresholds above typical peaks to avoid alert fatigue
### Organizing Contact Points
* **Separate by severity**: Route critical alerts to on-call, informational alerts to Slack
* **Use descriptive names**: "Production Critical" vs "Dev Notifications"
* **Test before relying**: Send a test notification to verify delivery
### Avoiding Alert Fatigue
* **Be specific**: Narrow scope to reduce false positives
* **Set appropriate windows**: Use time-based aggregation for rate metrics
* **Review regularly**: Disable or tune alerts that fire too often without action
## Related
* [Quick Start: Alerts](/quick-start/QuickStart_Alerts) - Get started with alert rules
* [Traces](/Observability/Traces/overview) - Understand the data alerts monitor
* [Tenants](/Observability/Tenants) - Monitor per-customer metrics
* [Dashboard](/Dashboard/Custom-dashboard) - View metrics to inform threshold settings
# A/B Testing Model Configurations
Source: https://docs.getnetra.ai/Cookbooks/evaluation/ab-testing-configurations
A/B test AI model configurations with Netra's evaluation framework. Compare prompts, models, and parameters by running the same dataset against each setup.
In the [Multi-Tenant Cost Tracking](/Cookbooks/observability/multi-tenant-cost-tracking) cookbook, you set up tier-based configurations for a meeting summarization pipeline — Enterprise on GPT-4, Professional on GPT-4-turbo, and Starter on GPT-3.5-turbo. But how do you know whether the Enterprise tier's output is actually better enough to justify the cost? Without structured evaluation, you're guessing.
This cookbook walks you through the next step: using Netra's evaluation framework to A/B test those configurations. You'll run the same test cases against two tiers, score both with the same evaluators, and compare results side by side to make a data-driven decision.
**Prerequisite:** You need a Netra API key ([Get started here](/quick-start/Overview)) and the meeting summarization pipeline from the [Multi-Tenant Cost Tracking](/Cookbooks/observability/multi-tenant-cost-tracking) cookbook. The code below reuses the `MultiTenantMeetingSummarizer` class and tenant configurations from that cookbook.
## What You'll Learn
Create test cases that both configurations will be evaluated against
Set up evaluators for answer correctness and conciseness
Trigger separate evaluation runs for each configuration via the SDK
Interpret scores across runs to make data-driven configuration decisions
***
## Why A/B Test AI Configurations?
Different configurations serve different trade-offs. Systematic A/B testing answers these questions with data:
| Scenario | What to Compare | What You'll Learn |
| ----------------------- | ---------------------------------------- | -------------------------------------------------------- |
| **Model upgrade** | GPT-3.5-turbo vs GPT-4-turbo | Does the quality improvement justify the cost increase? |
| **Prompt optimization** | Original prompt vs revised prompt | Does the new prompt improve quality with the same model? |
| **Parameter tuning** | temperature=0.1 vs temperature=0.3 | Which setting produces more consistent results? |
| **Tier validation** | Enterprise config vs Professional config | Does the quality gap justify the price gap? |
Netra's evaluation framework makes this straightforward: create one dataset, run it against each configuration as a separate [Test Run](/Evaluation/TestRuns), and compare evaluator scores in the dashboard. See the [Evaluation Overview](/Evaluation/Evaluation-overview) for a deeper look at the framework.
***
Now, let's walk through the process of A/B testing two configurations:
## Step 1: Create Evaluators
You need two evaluators from the library.
### Answer Correctness (Library)
Go to **Evaluation → Evaluators**, switch to the **Library** tab, and add **Answer Correctness** from the Quality category.
### Conciseness (Library)
Add **Conciseness** from the Quality category.
| Evaluator | What It Measures |
| ---------------------- | -------------------------------------------------------------------------- |
| **Answer Correctness** | Is the generated output factually correct compared to the expected output? |
| **Conciseness** | Is the output appropriately brief without losing key information? |
You can test each evaluator in the **Playground** before using it in a dataset. See [Evaluators](/Evaluation/Evaluators) for the full reference.
***
## Step 2: Create a Dataset
Go to **Evaluation → Datasets** and click **Create Dataset**. Name it "A/B Test Dataset" and attach the two evaluators from Step 1.
You already have traces from running the meeting summarization pipeline in the [Multi-Tenant Cost Tracking](/Cookbooks/observability/multi-tenant-cost-tracking) cookbook. Add them to your dataset directly:
Go to **Observability → Traces** and select a trace from the [Multi-Tenant Cost Tracking](/Cookbooks/observability/multi-tenant-cost-tracking) cookbook. Choose traces with different meeting types (short standups, planning sessions, open-ended discussions) to get a diverse set of test cases.
Click on the trace, then click **Add to Dataset**. Select the "A/B Test Dataset" you just created. Fill in the **Expected Output** with the correct summary for that meeting transcript.
Add 3–5 traces covering different meeting types — short standups, complex planning sessions, and ambiguous discussions. More diverse test cases give you a clearer comparison between configurations.
You can also add test cases manually if you want to include specific edge cases that aren't in your traces.
For each evaluator, configure the variable mappings so the evaluator receives the correct inputs at runtime — map evaluator variables like `query` and `expected_output` to **Dataset item** fields, and `agent_response` to **Agent response**. See [Datasets](/Evaluation/Datasets) for the full mapping reference.
***
## Step 3: Trigger Test Runs
The key to A/B testing is running the **same dataset** against **different configurations** as separate test runs. Copy the **Dataset ID** from the dataset page and trigger one run per configuration.
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="ab-testing",
instruments={InstrumentSet.OPENAI},
)
# Reuse the summarizer from the Multi-Tenant Cost Tracking cookbook
summarizer = MultiTenantMeetingSummarizer()
dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")
# --- Configuration A: Enterprise tier (GPT-4) ---
def enterprise_pipeline(input_data):
result = summarizer.summarize_meeting(
tenant_id="apex-legal",
meeting_transcript=input_data,
)
return result["summary"]
Netra.evaluation.run_test_suite(
name="Meeting Summary — Enterprise (GPT-4)",
data=dataset,
task=enterprise_pipeline,
)
# --- Configuration B: Professional tier (GPT-4-turbo) ---
def professional_pipeline(input_data):
result = summarizer.summarize_meeting(
tenant_id="stratex-consulting",
meeting_transcript=input_data,
)
return result["summary"]
Netra.evaluation.run_test_suite(
name="Meeting Summary — Professional (GPT-4-turbo)",
data=dataset,
task=professional_pipeline,
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "ab-testing",
instruments: new Set([NetraInstruments.OPENAI]),
});
// Reuse the summarizer from the Multi-Tenant Cost Tracking cookbook
const summarizer = new MultiTenantMeetingSummarizer();
const dataset = await Netra.evaluation.getDataset("your-dataset-id");
// --- Configuration A: Enterprise tier (GPT-4) ---
async function enterprisePipeline(inputData: string): Promise {
const result = await summarizer.summarizeMeeting(
"apex-legal",
inputData,
);
return result.summary;
}
await Netra.evaluation.runTestSuite(
"Meeting Summary — Enterprise (GPT-4)",
dataset,
enterprisePipeline,
);
// --- Configuration B: Professional tier (GPT-4-turbo) ---
async function professionalPipeline(inputData: string): Promise {
const result = await summarizer.summarizeMeeting(
"stratex-consulting",
inputData,
);
return result.summary;
}
await Netra.evaluation.runTestSuite(
"Meeting Summary — Professional (GPT-4-turbo)",
dataset,
professionalPipeline,
);
```
You can test any number of configurations — model swaps, prompt variations, temperature changes — by adding more `run_test_suite` calls against the same dataset.
For more details on the evaluation API, refer to the [SDK documentation](/sdk-reference/evaluation/python).
***
## Step 4: Compare Results
Go to **Evaluation → Test Runs** to see both runs listed. Click into each run to see per-evaluator, per-item results.
### Build a Comparison Table
Pull the evaluator scores from each run and compare:
| Evaluator | Enterprise (GPT-4) | Professional (GPT-4-turbo) | Delta |
| ------------------ | ------------------ | -------------------------- | ----- |
| Answer Correctness | 0.95 | 0.89 | -0.06 |
| Conciseness | 0.80 | 0.88 | +0.08 |
You can also click **View Trace** on any result to inspect the exact LLM input, output, and token usage for that test case. This is useful for understanding why one configuration scored higher on a specific item.
***
## Interpreting Scores and Making Decisions
### Quality vs. Cost Analysis
Combine evaluator scores with cost data from your traces to see the full picture:
| Metric | Enterprise (GPT-4) | Professional (GPT-4-turbo) | Delta |
| ------------------ | ------------------ | -------------------------- | ----- |
| Answer Correctness | 0.95 | 0.89 | -6% |
| Conciseness | 0.80 | 0.88 | +10% |
| Avg Cost per Item | \$0.023 | \$0.008 | -65% |
| Avg Latency | 2.1s | 1.4s | -33% |
### Decision Framework
Use the comparison data to make an informed decision:
| Condition | Action |
| -------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Quality scores equivalent, one configuration is cheaper or faster | Switch to the cheaper/faster configuration |
| One configuration scores higher on your most important evaluator | Keep the higher-quality configuration if the cost difference is acceptable |
| Scores are mixed (one wins on correctness, the other on conciseness) | Prioritize the evaluator that matters most for your use case |
| Quality drops below your pass threshold (e.g., 0.7) | Do not switch — the cost savings aren't worth the quality loss |
After making a decision, re-run the evaluation periodically to confirm the quality gap hasn't changed — model behavior can shift with provider updates.
***
## Continuous A/B Testing Strategy
Run A/B tests regularly as part of your development workflow:
1. **Before model upgrades** — Compare the new model against your current one before switching in production
2. **After prompt changes** — Measure the impact of prompt modifications across all quality dimensions
3. **When optimizing cost** — Verify that a cheaper configuration maintains acceptable quality
4. **For tier validation** — Confirm that premium tiers deliver measurably better results than lower tiers
***
## See Also
Set up the tier-based meeting summarization pipeline this cookbook evaluates
Deep dive into Netra's evaluation framework: datasets, evaluators, and test runs
Evaluate tool selection, escalation, and workflow completion in agents
# Evaluating Agent Decisions
Source: https://docs.getnetra.ai/Cookbooks/evaluation/evaluating-agent-decisions
Evaluate AI agent decisions with Netra. Measure tool selection accuracy, escalation logic, and workflow completion using structured evaluation datasets.
A traced agent tells you what happened — which tools were called, how long each step took, and what the LLM generated. Evaluation tells you whether the agent made the right decisions. Without structured scoring, you can't tell if the agent is selecting the wrong tools, over-escalating simple requests, or stopping before the workflow is complete. These failures don't throw errors — they just produce worse outcomes.
This cookbook walks you through Netra's evaluation workflow: creating evaluators for agent-specific quality dimensions, building test datasets from your traces, running test suites, and interpreting results to improve your agent.
**Prerequisite:** You need a Netra API key ([Get started here](/quick-start/Overview)) and an AI agent to evaluate. The test cases below use the customer support agent from the [Tracing LangChain Agents](/Cookbooks/observability/tracing-langchain-agents) cookbook as a reference.
## What You'll Learn
Create structured test cases with inputs, expected outputs, and metadata for your evaluators
Set up evaluators for tool correctness, escalation accuracy, and workflow completion
Execute evaluations via the SDK and collect quality metrics
Interpret scores, debug failures using trace integration, and improve your agent
***
## Why Agent Decisions Need Evaluation
Agent evaluation differs from simple LLM evaluation. Agents make multi-step decisions that compound — a 95% accurate tool selection across 3 steps means only 86% of full workflows succeed (0.95^3):
| Failure Mode | What Goes Wrong | Why You Can't Spot-Check It |
| ------------------------ | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| **Wrong tool selection** | Agent uses `search_kb` when it should use `check_order_status` | The answer may still sound reasonable despite using the wrong data source |
| **Over-escalation** | Agent escalates a simple FAQ to a human operator | Each escalation looks cautious and safe in isolation — you need aggregate metrics to see the pattern |
| **Under-escalation** | Agent tries to handle a frustrated customer instead of escalating | Only visible when you compare the agent's decision against the expected action |
| **Incomplete workflow** | Agent looks up the ticket but never checks the related order | The partial answer addresses part of the question, so it looks acceptable on a quick read |
Netra's evaluation framework addresses this with [Datasets](/Evaluation/Datasets) (test cases with inputs, expected outputs, and metadata), [Evaluators](/Evaluation/Evaluators) (library and custom code-based scoring for tool usage, escalation, and completion), and [Test Runs](/Evaluation/TestRuns) (execution results with pass/fail rates, scores, and linked traces). The workflow is: create evaluators, build test cases, run, and review. See the [Evaluation Overview](/Evaluation/Evaluation-overview) for a deeper look at the framework.
***
Now, let's walk through the process of evaluating agent decisions:
## Step 1: Create Evaluators
You need three evaluators — one from the library and two custom LLM as Judge evaluators.
### Tool Correctness (Library)
Go to **Evaluation → Evaluators**, switch to the **Library** tab, and add **Tool Correctness** from the Tool Use category.
| Evaluator | What It Measures |
| -------------------- | ---------------------------------------------------------------------------------------- |
| **Tool Correctness** | Did the agent call the right tools, avoid forbidden tools, and use the correct sequence? |
### Escalation Accuracy (LLM as Judge)
Click **Add Evaluator** and create an LLM as Judge evaluator.
| Setting | Value |
| ----------------- | ------------ |
| **Type** | LLM as Judge |
| **Output Type** | Numerical |
| **Pass Criteria** | score >= 0.8 |
Use the following prompt template:
```
You are evaluating whether a customer support agent made the correct escalation decision.
Metadata for this test case:
{{metadata}}
Agent's response:
{{agent_response}}
Evaluate the escalation decision based on the "should_escalate" field in the metadata:
- If should_escalate is true, the agent must have escalated (e.g., transferred to a human, mentioned a specialist, offered to connect to a manager).
- If should_escalate is false, the agent must NOT have escalated.
Score:
- 1.0 if the agent's escalation decision is correct
- 0.5 if the agent escalated unnecessarily (false positive — less severe)
- 0.0 if the agent failed to escalate when required (false negative — severe)
```
### Workflow Completion (LLM as Judge)
Create another LLM as Judge evaluator that validates whether the agent completed all required steps.
| Setting | Value |
| ----------------- | ------------ |
| **Type** | LLM as Judge |
| **Output Type** | Boolean |
| **Pass Criteria** | true |
Use the following prompt template:
```
You are evaluating whether a customer support agent completed all required workflow steps.
Metadata for this test case:
{{metadata}}
Agent's response:
{{agent_response}}
Check the "required_steps" field in the metadata. If required_steps is empty or missing, return true.
Otherwise, verify that the agent's response addresses every step listed. Steps can be addressed using different wording — check for semantic equivalence, not exact keyword matches.
Return true if ALL required steps are covered, false otherwise.
```
You can test each evaluator in the **Playground** before using it in a dataset. See [Evaluators](/Evaluation/Evaluators) for the full reference.
***
## Step 2: Create a Dataset
Go to **Evaluation → Datasets** and click **Create Dataset**. Name it "Agent Decisions Dataset" and attach the three evaluators from Step 1.
### Configure Variable Mappings
For each evaluator, map the variables to their data source so the evaluator receives the correct inputs at runtime:
**Tool Correctness**
| Variable | Maps To |
| ---------------- | ---------------------------------------- |
| `expected_tools` | Dataset item → `metadata.expected_tools` |
| `actual_tools` | Execution data → summary metrics → tools |
**Escalation Accuracy**
| Variable | Maps To |
| ---------------- | ----------------------- |
| `agent_response` | Agent response |
| `metadata` | Dataset item → metadata |
**Workflow Completion**
| Variable | Maps To |
| ---------------- | ----------------------- |
| `agent_response` | Agent response |
| `metadata` | Dataset item → metadata |
### Add Test Cases
Add the following five test cases manually:
**1. Single-tool — Policy lookup**
| Field | Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Input** | What is your return policy? |
| **Expected Output** | Items can be returned within 30 days of purchase. The item must be unused and in its original packaging. Refunds are processed in 5-7 business days to your original payment method. |
| **Metadata** | `{"expected_tools": ["search_kb.tool"], "should_escalate": false}` |
**2. Multi-tool — Ticket with related order**
| Field | Value |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | Check ticket TKT-002 and tell me the status of the related order |
| **Expected Output** | Ticket TKT-002 is open regarding a damaged item. The related order ORD-12345 has been delivered. The order contains Headphones totaling \$79.99. |
| **Metadata** | `{"expected_tools": ["lookup_ticket.tool", "check_order_status.tool"], "should_escalate": false, "required_steps": ["ticket", "order", "delivered"]}` |
**3. Escalation — Angry customer**
| Field | Value |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | This is ridiculous! I've been waiting 3 weeks and nobody has helped me. I need to speak to a manager right now! |
| **Expected Output** | I understand your frustration, and I'm sorry for the delay. I'm transferring you to a specialist who can resolve this immediately. |
| **Metadata** | `{"should_escalate": true, "expected_tools": ["escalate_to_human.tool"]}` |
**4. No-tool — Simple thank you**
| Field | Value |
| ------------------- | ----------------------------------------------------------------------------------------- |
| **Input** | Thanks for your help, that's all I needed! |
| **Expected Output** | You're welcome! If you need anything else, don't hesitate to reach out. Have a great day! |
| **Metadata** | `{"expected_tools": [], "should_escalate": false}` |
**5. Edge case — Non-existent order**
| Field | Value |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Input** | I want a refund for order ORD-99999 |
| **Expected Output** | I wasn't able to find an order with ID ORD-99999. Could you double-check the order number? You can find it in your confirmation email. |
| **Metadata** | `{"expected_tools": ["check_order_status.tool"], "should_escalate": false, "required_steps": ["not found", "order"]}` |
Under **Evaluation → Datasets**, you should now see the "Agent Decisions Dataset" with five items and three evaluators under the **Evaluators** tab.
***
## Step 3: Trigger a Test Run
Copy the **Dataset ID** from the dataset page and use the code below.
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="agent-evaluation",
instruments={InstrumentSet.OPENAI, InstrumentSet.LANGCHAIN},
)
# Your agent logic — wrap your agent in a function that takes
# an input string and returns the generated response.
# Tip: if you followed the tracing cookbook, you can call agent.invoke() here.
def run_agent(input_data):
result = agent.invoke({"messages": [{"role": "user", "content": input_data}]})
return result["messages"][-1].content
dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")
result = Netra.evaluation.run_test_suite(
name="Agent Decision Evaluation",
data=dataset,
task=run_agent,
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "agent-evaluation",
instruments: new Set([NetraInstruments.OPENAI, NetraInstruments.LANGCHAIN]),
});
// Your agent logic — wrap your agent in a function that takes
// an input string and returns the generated response.
// Tip: if you followed the tracing cookbook, you can call agent.invoke() here.
async function runAgent(inputData: string): Promise {
const result = await agent.invoke({ messages: [{ role: "user", content: inputData }] });
return result.messages[result.messages.length - 1].content;
}
const dataset = await Netra.evaluation.getDataset("your-dataset-id");
const result = await Netra.evaluation.runTestSuite(
"Agent Decision Evaluation",
dataset,
runAgent,
);
```
For more details on the evaluation API, refer to the [SDK documentation](/sdk-reference/evaluation/python).
***
## Step 4: View Results
Go to **Evaluation → Test Runs** to see your test run with its status. Click on the test run to see the result for each evaluator, for each dataset item — whether it passed or failed.
You can also click **View Trace** on any result to see the exact reasoning steps (thought → action → observation), which tools were called and in what order, and where the agent deviated from expected behavior. See [Test Runs](/Evaluation/TestRuns) for the full reference.
***
## Interpreting Scores and Improving Quality
When evaluator scores are low, use this table to identify the likely cause and fix:
| Low Score In | Likely Cause | How to Fix |
| -------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Tool Correctness** | Ambiguous tool descriptions | Add clearer docstrings with explicit "use when" / "do not use for" guidance |
| **Escalation (false negatives)** | Agent misses urgency signals | Add more escalation triggers to the system prompt (e.g., specific keywords, wait times) |
| **Escalation (false positives)** | Agent is over-cautious | Narrow escalation criteria — list what should NOT be escalated |
| **Workflow Completion** | Agent stops before finishing all steps | Add explicit completion checks to the prompt (e.g., "verify all related records before responding") |
### Prompt Improvements Based on Evaluation
Use evaluation failures to refine your agent prompt:
```python theme={null}
# Before: Vague escalation guidance
system_prompt = """Escalate complex issues to human operators."""
# After: Specific criteria derived from evaluation failures
system_prompt = """
Escalate to human operators when ANY of these conditions are met:
- User expresses frustration ("ridiculous", "unacceptable", "furious")
- User has been waiting more than 2 weeks
- User explicitly asks to speak to a human
- The issue involves policy exceptions
Do NOT escalate for:
- Simple FAQ questions
- Routine order status checks
- Standard refund requests within policy
"""
```
After making changes, re-run the evaluation against the same dataset and compare results across test runs. Netra tracks all runs so you can see whether your changes improved quality.
***
## Continuous Evaluation Strategy
For production agents, run evaluations regularly:
1. **On every prompt change** — Re-run the full test suite to catch regressions
2. **After tool additions** — Ensure new tools don't disrupt existing tool selection patterns
3. **Weekly benchmarks** — Track quality trends over time to catch gradual degradation
4. **After model upgrades** — Verify that a new model version doesn't change escalation or tool selection behavior
***
## See Also
Set up comprehensive tracing for your agent before evaluating
Deep dive into Netra's evaluation framework: datasets, evaluators, and test runs
Test your agent through multi-turn simulated conversations
Compare different pipeline configurations systematically
# Evaluating RAG Quality
Source: https://docs.getnetra.ai/Cookbooks/evaluation/evaluating-rag-quality
Evaluate RAG pipeline quality with Netra. Measure retrieval relevance, answer correctness, and faithfulness using LLM-as-Judge and code evaluators.
A working RAG pipeline is only the starting point. Without systematic evaluation, you have no way to know whether your retriever is fetching the right chunks, whether the LLM is faithfully using the retrieved context, or whether the generated answer actually addresses the user's question. These failures are subtle — they don't throw errors, they just produce worse answers.
This cookbook walks you through Netra's evaluation workflow: creating test datasets from your traces, configuring evaluators that score RAG-specific quality dimensions, running test suites, and interpreting results to improve your pipeline.
**Prerequisite:** You need a RAG pipeline with Netra tracing configured and at least one trace visible in your dashboard. If you haven't set this up yet, follow the [Tracing a RAG Pipeline](/Cookbooks/observability/tracing-rag-pipeline) cookbook first.
## What You'll Learn
Turn real RAG interactions into reusable test cases directly from the dashboard
Set up LLM-as-Judge evaluators for retrieval quality, answer correctness, and faithfulness
Execute evaluations via the dashboard or SDK and collect quality metrics
Interpret scores, debug failures using trace integration, and improve your pipeline
***
## Why RAG Pipelines Need Evaluation
RAG systems have multiple failure modes that are invisible without structured evaluation:
| Failure Mode | What Goes Wrong | Why You Can't Spot-Check It |
| ------------------------ | ------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| **Irrelevant retrieval** | The retriever fetches chunks that don't contain the answer | Similarity scores look reasonable, but the content is off-topic |
| **Hallucination** | The LLM generates information not present in the retrieved context | The answer sounds fluent and confident, but fabricates details |
| **Missed intent** | The answer is factually correct but doesn't address the user's actual question | Only noticeable when you compare against a known-good response |
| **Inconsistency** | The same question gets different quality answers depending on retrieved chunks | Requires running the same inputs multiple times to detect |
Netra's evaluation framework addresses this with [Datasets](/Evaluation/Datasets) (test cases with inputs and expected outputs), [Evaluators](/Evaluation/Evaluators) (LLM-as-Judge or code-based scoring for relevance, correctness, and faithfulness), and [Test Runs](/Evaluation/TestRuns) (execution results with pass/fail rates, scores, and linked traces). The workflow is: create test cases, attach evaluators, run, and review. See the [Evaluation Overview](/Evaluation/Evaluation-overview) for a deeper look at the framework.
***
Now, let's walk through the process of evaluating a RAG pipeline:
## Step 1: Create Evaluators
Go to **Evaluation → Evaluators** and add the following three evaluators from the [library](/Evaluation/Evaluators#library):
| Evaluator | What It Measures |
| ---------------------- | -------------------------------------------------------------------------- |
| **Answer Correctness** | Is the generated answer factually correct compared to the expected output? |
| **Context Relevance** | Are the retrieved chunks relevant to the question being asked? |
| **Faithfulness** | Is the answer grounded in the retrieved context, without hallucination? |
You can tune the prompt for each evaluator and test it in the **Playground** to see how it scores before using it in a dataset. See [Evaluators](/Evaluation/Evaluators) for the full reference.
***
## Step 2: Create a Dataset from Traces
Go to **Observability → Traces** and select a trace from the [Tracing a RAG Pipeline](/Cookbooks/observability/tracing-rag-pipeline) cookbook, or any other RAG pipeline trace you have.
Click on the trace, then click **Add to Dataset**. Create a new dataset called "RAG Quality Dataset".
Add the three evaluators you created in Step 1 to the dataset.
For each evaluator, map the prompt variables to their source — **Dataset item** (input, expected output), **Agent response** (actual RAG output), or **Execution data** (trace metadata).
***
## Step 3: Add More Test Cases
Add one more trace with a different question to the same dataset by repeating the **Add to Dataset** flow.
Under **Evaluation → Datasets**, you should now see the "RAG Quality Dataset" with two dataset items and three evaluators under the **Evaluators** tab, as shown in the video above. You can add more evaluators or dataset items manually from this page.
***
## Step 4: Trigger a Test Run
Currently in Netra, test runs are triggered via code. Copy the **Dataset ID** from the dataset page and use the code below.
```python Python theme={null}
from netra import Netra
Netra.init(app_name="rag-evaluation")
# Your RAG logic — wrap your pipeline in a function that takes
# an input string and returns the generated answer.
# Tip: if you followed the tracing cookbook, you can call chatbot.chat() here.
def rag_pipeline(input_data):
response = chatbot.chat(input_data)
return response["answer"]
dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")
result = Netra.evaluation.run_test_suite(
name="RAG Quality Evaluation",
data=dataset,
task=rag_pipeline,
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({ appName: "rag-evaluation" });
// Your RAG logic — wrap your pipeline in a function that takes
// an input string and returns the generated answer.
// Tip: if you followed the tracing cookbook, you can call chatbot.chat() here.
async function ragPipeline(inputData: string): Promise {
const response = await chatbot.chat(inputData);
return response.answer;
}
const dataset = await Netra.evaluation.getDataset("your-dataset-id");
const result = await Netra.evaluation.runTestSuite(
"RAG Quality Evaluation",
dataset,
ragPipeline,
);
```
For more details on the evaluation API, refer to the [SDK documentation](/sdk-reference/evaluation/python).
***
## Step 5: View Results
Go to **Evaluation → Test Runs** to see your test run with its status. Click on the test run to see the result for each evaluator, for each dataset item — whether it passed or failed.
You can also click **View Trace** on any result to debug what went wrong or verify what was correct. See [Test Runs](/Evaluation/TestRuns) for the full reference.
***
## Interpreting Scores and Improving Quality
When evaluator scores are low, use this table to identify the likely cause and fix:
| Low Score In | Likely Cause | How to Fix |
| -------------------- | ---------------------------------- | ------------------------------------------------------------------------------------ |
| **Answer Relevance** | Wrong chunks retrieved | Increase `top_k`, reduce chunk size, add overlap between chunks |
| **Factual Accuracy** | LLM misinterprets context | Improve the system prompt, lower temperature, use a stronger model |
| **Coherence** | Disjointed or repetitive response | Refine the system prompt to request structured answers |
| **Faithfulness** | Model hallucinating beyond context | Add explicit grounding instructions (e.g., "Only answer using the provided context") |
After making changes to your pipeline, re-run the evaluation against the same dataset and compare results across test runs. Netra tracks all runs so you can see whether your changes improved quality.
***
## Continuous Evaluation Strategy
For production RAG systems, run evaluations regularly:
1. **On every deployment** — Run your test suite in CI/CD before releasing changes to retrieval logic or prompts
2. **Weekly benchmarks** — Track quality trends over time to catch gradual degradation
3. **After prompt changes** — Measure the impact of system prompt modifications on all quality dimensions
4. **After parameter tuning** — Validate that changes to chunk size, `top_k`, or overlap actually improve quality
***
## See Also
Set up comprehensive tracing for your RAG pipeline before evaluating
Deep dive into Netra's evaluation framework: datasets, evaluators, and test runs
Evaluate tool selection, escalation, and workflow completion in agents
Compare different pipeline configurations systematically
# Multi-Tenant Cost Tracking
Source: https://docs.getnetra.ai/Cookbooks/observability/multi-tenant-cost-tracking
Track AI costs per customer in B2B apps with Netra tenant observability. Monitor per-tenant spending, enforce SLAs, and generate usage reports.
This cookbook shows you how to build **comprehensive multi-tenant observability** for B2B AI platforms—tracking costs per customer, monitoring SLA compliance, and attributing usage across your entire customer base.
Run the complete notebook in your browser
All company names (MeetingMind, Apex Legal, Stratex Consulting, TechStart Inc) and scenarios in this cookbook are entirely fictional and used for demonstration purposes only.
## What You'll Learn
Use Netra's native tenant tracking to attribute all traces to specific customers
Query usage and cost data per tenant via API or dashboard
Set up tier-specific alerts that trigger on latency or error rate breaches
Understand session and user behavior within each tenant
**Prerequisites:**
* Python >=3.10, \< 3.14
* OpenAI API key
* Netra API key ([Get your key here](/quick-start/Overview))
***
## The MeetingMind Scenario
**MeetingMind** is a fictional B2B SaaS platform that provides AI-powered meeting summarization. The platform serves customers with different needs and budgets:
| Customer | Industry | Tier |
| ---------------------- | ------------ | ------------ |
| **Apex Legal** | Law Firm | Enterprise |
| **Stratex Consulting** | Consulting | Professional |
| **TechStart Inc** | Tech Startup | Starter |
Each tier uses a different configuration and has different SLA commitments:
| Tier | Model | Latency SLA | Rate Limit |
| ------------ | ----------- | ----------- | ------------ |
| Enterprise | GPT-4o-mini | P95 \< 2s | 60 calls/min |
| Professional | GPT-4o-mini | P95 \< 3s | 30 calls/min |
| Starter | GPT-4o-mini | Best effort | 10 calls/min |
***
## Step 1: Install Packages
```bash Python theme={null}
pip install netra-sdk openai
```
```bash TypeScript theme={null}
npm install netra-sdk openai
```
## Step 2: Set Environment Variables
```bash Python theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
```
```bash TypeScript theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
```
## Step 3: Initialize Netra for Multi-Tenant Tracking
Initialize Netra at application startup with auto-instrumentation for OpenAI:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Initialize Netra for multi-tenant observability
Netra.init(
app_name="meetingmind",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
environment="production",
trace_content=True,
instruments={InstrumentSet.OPENAI},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Initialize Netra for multi-tenant observability
await Netra.init({
appName: "meetingmind",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
environment: "production",
traceContent: true,
instruments: new Set([NetraInstruments.OPENAI]),
});
```
## Step 4: Define Tenant Configuration
Configure tier-specific settings for each customer:
```python Python theme={null}
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class TenantConfig:
"""Configuration for a tenant's service tier."""
tenant_id: str
tier: str
model: str
features: List[str]
latency_sla_ms: Optional[int]
max_calls_per_minute: int
# Tenant configurations
TENANT_CONFIGS = {
"apex-legal": TenantConfig(
tenant_id="apex-legal",
tier="enterprise",
model="gpt-4",
features=["summary", "action_items", "decisions", "custom_reports"],
latency_sla_ms=2000,
max_calls_per_minute=60
),
"stratex-consulting": TenantConfig(
tenant_id="stratex-consulting",
tier="professional",
model="gpt-4-turbo",
features=["summary", "action_items"],
latency_sla_ms=3000,
max_calls_per_minute=30
),
"techstart-inc": TenantConfig(
tenant_id="techstart-inc",
tier="starter",
model="gpt-3.5-turbo",
features=["summary"],
latency_sla_ms=None, # Best effort
max_calls_per_minute=10
),
}
```
```typescript TypeScript theme={null}
interface TenantConfig {
tenantId: string;
tier: string;
model: string;
features: string[];
latencySlaMs: number | null;
maxCallsPerMinute: number;
}
const TENANT_CONFIGS: Record = {
"apex-legal": {
tenantId: "apex-legal",
tier: "enterprise",
model: "gpt-4",
features: ["summary", "action_items", "decisions", "custom_reports"],
latencySlaMs: 2000,
maxCallsPerMinute: 60,
},
"stratex-consulting": {
tenantId: "stratex-consulting",
tier: "professional",
model: "gpt-4-turbo",
features: ["summary", "action_items"],
latencySlaMs: 3000,
maxCallsPerMinute: 30,
},
"techstart-inc": {
tenantId: "techstart-inc",
tier: "starter",
model: "gpt-3.5-turbo",
features: ["summary"],
latencySlaMs: null, // Best effort
maxCallsPerMinute: 10,
},
};
```
## Step 5: Create Multi-Tenant Meeting Summarizer
Build a service that tracks costs per tenant. This class handles tenant context setting, prompt building based on feature tiers, cost calculation, and SLA compliance checking — all within Netra spans.
```python Python theme={null}
from openai import OpenAI
import time
import uuid
import os
from netra import Netra, SpanType, UsageModel
class MultiTenantMeetingSummarizer:
"""Meeting summarization service with per-tenant cost tracking."""
def __init__(self):
self.openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
self.tenant_usage = {} # Track usage per tenant
def summarize_meeting(self, tenant_id: str, meeting_transcript: str, user_id: str = None) -> dict:
"""Summarize a meeting for a specific tenant with cost tracking."""
# Validate tenant
if tenant_id not in TENANT_CONFIGS:
return {"error": f"Unknown tenant: {tenant_id}"}
config = TENANT_CONFIGS[tenant_id]
# Set tenant context - this is the key for multi-tenant observability
Netra.set_tenant_id(tenant_id)
Netra.set_session_id(str(uuid.uuid4()))
if user_id:
Netra.set_user_id(user_id)
# Build the prompt
prompt = f"Summarize this meeting transcript into:\n"
if "summary" in config.features:
prompt += "- Executive Summary (2-3 paragraphs)\n"
if "action_items" in config.features:
prompt += "- Action Items (numbered list)\n"
if "decisions" in config.features:
prompt += "- Key Decisions Made\n"
if "custom_reports" in config.features:
prompt += "- Recommendations for Follow-up\n"
prompt += f"\nMeeting Transcript:\n{meeting_transcript}"
# Start a span for the summarization operation
with Netra.start_span("meeting-summarization") as span:
span.set_attribute("tenant_id", tenant_id)
span.set_attribute("tier", config.tier)
span.set_attribute("model", config.model)
start_time = time.time()
# Call the API (auto-traced)
response = self.openai_client.chat.completions.create(
model=config.model,
messages=[
{"role": "system", "content": "You are an expert meeting summarizer."},
{"role": "user", "content": prompt}
],
temperature=0.3
)
latency_ms = (time.time() - start_time) * 1000
summary = response.choices[0].message.content
# Calculate cost (simplified pricing model)
# GPT-4o-mini pricing (approx): $0.15/1M input, $0.60/1M output
input_price = 0.15 / 1_000_000
output_price = 0.60 / 1_000_000
prompt_tokens = response.usage.prompt_tokens
completion_tokens = response.usage.completion_tokens
total_tokens = response.usage.total_tokens
cost = (prompt_tokens * input_price) + (completion_tokens * output_price)
# Record detailed usage and cost in the span
span.set_usage([
UsageModel(
model=config.model,
cost_in_usd=cost,
usage_type="chat",
units_used=total_tokens
)
])
# Check SLA compliance
sla_compliant = True
if config.latency_sla_ms:
sla_compliant = latency_ms <= config.latency_sla_ms
span.set_attribute("sla_met", sla_compliant)
if not sla_compliant:
span.add_event("sla-breach", {
"actual_ms": latency_ms,
"sla_ms": config.latency_sla_ms
})
span.set_success()
# Local tracking
if tenant_id not in self.tenant_usage:
self.tenant_usage[tenant_id] = {"count": 0, "tokens": 0, "total_cost": 0.0, "total_latency": 0}
self.tenant_usage[tenant_id]["count"] += 1
self.tenant_usage[tenant_id]["tokens"] += total_tokens
self.tenant_usage[tenant_id]["total_cost"] += cost
self.tenant_usage[tenant_id]["total_latency"] += latency_ms
return {
"tenant_id": tenant_id,
"tier": config.tier,
"summary": summary,
"token_usage": {
"prompt": prompt_tokens,
"completion": completion_tokens,
"total": total_tokens
},
"latency_ms": latency_ms,
"sla_compliant": sla_compliant,
"cost": cost
}
def print_usage_summary(self):
"""Print usage summary by tenant."""
for tenant_id, usage in self.tenant_usage.items():
print(f"\n{tenant_id}:")
print(f" Calls: {usage['count']}")
print(f" Total Tokens: {usage['tokens']}")
print(f" Total Cost: ${usage['total_cost']:.4f}")
print(f" Avg Latency: {usage['total_latency']/usage['count']:.0f}ms")
```
```typescript TypeScript theme={null}
import { Netra, SpanType } from "netra-sdk";
import OpenAI from "openai";
import { v4 as uuidv4 } from "uuid";
class MultiTenantMeetingSummarizer {
private openaiClient: OpenAI;
private tenantUsage: Record<
string,
{ count: number; tokens: number; totalCost: number; totalLatency: number }
> = {};
constructor() {
this.openaiClient = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
}
async summarizeMeeting(
tenantId: string,
meetingTranscript: string,
userId?: string
) {
// Validate tenant
if (!TENANT_CONFIGS[tenantId]) {
return { error: `Unknown tenant: ${tenantId}` };
}
const config = TENANT_CONFIGS[tenantId];
// Set tenant context - this is the key for multi-tenant observability
Netra.setTenantId(tenantId);
Netra.setSessionId(uuidv4());
if (userId) {
Netra.setUserId(userId);
}
// Build the prompt
let prompt = "Summarize this meeting transcript into:\n";
if (config.features.includes("summary")) {
prompt += "- Executive Summary (2-3 paragraphs)\n";
}
if (config.features.includes("action_items")) {
prompt += "- Action Items (numbered list)\n";
}
if (config.features.includes("decisions")) {
prompt += "- Key Decisions Made\n";
}
if (config.features.includes("custom_reports")) {
prompt += "- Recommendations for Follow-up\n";
}
prompt += `\nMeeting Transcript:\n${meetingTranscript}`;
// Start a span for the summarization operation
const span = Netra.startSpan("meeting-summarization").start();
span.setAttribute("tenant_id", tenantId);
span.setAttribute("tier", config.tier);
span.setAttribute("model", config.model);
const startTime = Date.now();
try {
// Call the API (auto-traced)
const response = await this.openaiClient.chat.completions.create({
model: config.model,
messages: [
{ role: "system", content: "You are an expert meeting summarizer." },
{ role: "user", content: prompt },
],
temperature: 0.3,
});
const latencyMs = Date.now() - startTime;
const summary = response.choices[0].message.content;
// Calculate cost (GPT-4o-mini pricing: $0.15/1M input, $0.60/1M output)
const inputPrice = 0.15 / 1_000_000;
const outputPrice = 0.60 / 1_000_000;
const promptTokens = response.usage?.prompt_tokens || 0;
const completionTokens = response.usage?.completion_tokens || 0;
const totalTokens = promptTokens + completionTokens;
const cost = promptTokens * inputPrice + completionTokens * outputPrice;
// Record detailed usage and cost in the span
span.setUsage([
{
model: config.model,
costInUsd: cost,
usageType: "chat",
unitsUsed: totalTokens,
},
]);
// Check SLA compliance
let slaCompliant = true;
if (config.latencySlaMs) {
slaCompliant = latencyMs <= config.latencySlaMs;
span.setAttribute("sla_met", slaCompliant);
if (!slaCompliant) {
span.addEvent("sla-breach", {
actual_ms: latencyMs,
sla_ms: config.latencySlaMs,
});
}
}
span.setSuccess();
// Local tracking
if (!this.tenantUsage[tenantId]) {
this.tenantUsage[tenantId] = {
count: 0,
tokens: 0,
totalCost: 0,
totalLatency: 0,
};
}
this.tenantUsage[tenantId].count += 1;
this.tenantUsage[tenantId].tokens += totalTokens;
this.tenantUsage[tenantId].totalCost += cost;
this.tenantUsage[tenantId].totalLatency += latencyMs;
return {
tenantId,
tier: config.tier,
summary,
tokenUsage: {
prompt: promptTokens,
completion: completionTokens,
total: totalTokens,
},
latencyMs,
slaCompliant,
cost,
};
} finally {
span.end();
}
}
printUsageSummary() {
for (const [tenantId, usage] of Object.entries(this.tenantUsage)) {
console.log(`\n${tenantId}:`);
console.log(` Calls: ${usage.count}`);
console.log(` Total Tokens: ${usage.tokens}`);
console.log(` Total Cost: $${usage.totalCost.toFixed(4)}`);
console.log(
` Avg Latency: ${Math.round(usage.totalLatency / usage.count)}ms`
);
}
}
}
```
The key pattern here is calling `set_tenant_id()` early in the request lifecycle. This ensures all subsequent traces — including auto-instrumented OpenAI calls — are automatically attributed to the correct tenant.
## Step 6: Test with Sample Meetings
Simulate meeting summarization requests from different tenants:
```python Python theme={null}
# Initialize summarizer
summarizer = MultiTenantMeetingSummarizer()
# Enterprise tier (Apex Legal) - legal meeting
sample_meeting = """
Attendees: John (Partner), Sarah (Associate), Mike (Paralegal)
Duration: 45 minutes
Topic: Case Strategy for Smith v. Jones
John: Let's discuss our approach for the Smith case. The deposition is in 3 weeks.
Sarah: I've reviewed the discovery documents. The key issue is the contract's ambiguity around the liability clause.
Mike: I've created a timeline. The critical events are on pages 45-67 of the evidence log.
John: Good. Sarah, can you draft a summary of our position by Friday?
Sarah: I'll have it ready. Should I include recommendations for discovery?
John: Yes, especially around vendor communications. Mike, check if we have all related emails.
Mike: I'll pull those by tomorrow.
John: This looks solid. Let's reconvene next week after Sarah finishes the draft.
"""
result1 = summarizer.summarize_meeting(
tenant_id="apex-legal",
meeting_transcript=sample_meeting,
user_id="john.smith@apexlegal.com"
)
print(f"Tier: {result1['tier']}")
print(f"SLA Compliant: {result1['sla_compliant']}")
print(f"Latency: {result1['latency_ms']:.0f}ms")
print(f"Tokens Used: {result1['token_usage']['total']}")
# Professional tier (Stratex Consulting) - strategy meeting
meeting_transcript_2 = """
Team sync for Q2 strategy planning.
Attendees: CEO, CFO, Head of Product
CEO: Let's review our market position and Q2 targets.
CFO: Revenue is up 15% YoY. We're tracking to beat forecast.
Head of Product: New features launched last month show strong adoption.
CEO: Great! What are our risks?
CFO: Supply chain delays could impact timeline.
Head of Product: We need to hire 3 more engineers to meet roadmap.
CEO: Let's make that happen. Budget approved.
"""
result2 = summarizer.summarize_meeting(
tenant_id="stratex-consulting",
meeting_transcript=meeting_transcript_2,
user_id="cfo@stratex.com"
)
# Starter tier (TechStart Inc) - standup
meeting_transcript_3 = """
Daily standup
Attendees: Dev team
Tom: I finished the API integration yesterday.
Lisa: I'm working on the UI components.
Chris: Testing is on track for Thursday release.
Tom: Good. Any blockers?
Lisa: Waiting for design approval on the dashboard.
Chris: Should be done today.
"""
result3 = summarizer.summarize_meeting(
tenant_id="techstart-inc",
meeting_transcript=meeting_transcript_3,
user_id="dev@techstart.io"
)
```
```typescript TypeScript theme={null}
// Initialize summarizer
const summarizer = new MultiTenantMeetingSummarizer();
// Enterprise tier (Apex Legal) - legal meeting
const sampleMeeting = `
Attendees: John (Partner), Sarah (Associate), Mike (Paralegal)
Duration: 45 minutes
Topic: Case Strategy for Smith v. Jones
John: Let's discuss our approach for the Smith case. The deposition is in 3 weeks.
Sarah: I've reviewed the discovery documents. The key issue is the contract's ambiguity around the liability clause.
Mike: I've created a timeline. The critical events are on pages 45-67 of the evidence log.
John: Good. Sarah, can you draft a summary of our position by Friday?
Sarah: I'll have it ready. Should I include recommendations for discovery?
John: Yes, especially around vendor communications. Mike, check if we have all related emails.
Mike: I'll pull those by tomorrow.
John: This looks solid. Let's reconvene next week after Sarah finishes the draft.
`;
const result1 = await summarizer.summarizeMeeting(
"apex-legal",
sampleMeeting,
"john.smith@apexlegal.com"
);
console.log(`Tier: ${result1.tier}`);
console.log(`SLA Compliant: ${result1.slaCompliant}`);
console.log(`Latency: ${Math.round(result1.latencyMs)}ms`);
console.log(`Tokens Used: ${result1.tokenUsage.total}`);
// Professional tier (Stratex Consulting) - strategy meeting
const meetingTranscript2 = `
Team sync for Q2 strategy planning.
Attendees: CEO, CFO, Head of Product
CEO: Let's review our market position and Q2 targets.
CFO: Revenue is up 15% YoY. We're tracking to beat forecast.
Head of Product: New features launched last month show strong adoption.
CEO: Great! What are our risks?
CFO: Supply chain delays could impact timeline.
Head of Product: We need to hire 3 more engineers to meet roadmap.
CEO: Let's make that happen. Budget approved.
`;
const result2 = await summarizer.summarizeMeeting(
"stratex-consulting",
meetingTranscript2,
"cfo@stratex.com"
);
// Starter tier (TechStart Inc) - standup
const meetingTranscript3 = `
Daily standup
Attendees: Dev team
Tom: I finished the API integration yesterday.
Lisa: I'm working on the UI components.
Chris: Testing is on track for Thursday release.
Tom: Good. Any blockers?
Lisa: Waiting for design approval on the dashboard.
Chris: Should be done today.
`;
const result3 = await summarizer.summarizeMeeting(
"techstart-inc",
meetingTranscript3,
"dev@techstart.io"
);
```
## Step 7: Review Usage and Cost Breakdown
Analyze per-tenant usage patterns and costs:
```python Python theme={null}
# Print usage summary
summarizer.print_usage_summary()
# Calculate estimated costs (rough approximation)
# GPT-4o-mini pricing (approximate): $0.15/1M input tokens, $0.60/1M output tokens
input_price_per_token = 0.15 / 1_000_000
output_price_per_token = 0.60 / 1_000_000
for tenant_id, usage in summarizer.tenant_usage.items():
# Rough split: assume 70% input, 30% output tokens
input_tokens = int(usage['tokens'] * 0.7)
output_tokens = int(usage['tokens'] * 0.3)
cost = (input_tokens * input_price_per_token) + (output_tokens * output_price_per_token)
print(f"\n{tenant_id}:")
print(f" Total Tokens: {usage['tokens']}")
print(f" Estimated Cost: ${cost:.4f}")
print(f" Cost per Call: ${cost/usage['count']:.4f}")
```
```typescript TypeScript theme={null}
// Print usage summary
summarizer.printUsageSummary();
// Calculate estimated costs (rough approximation)
// GPT-4o-mini pricing (approximate): $0.15/1M input tokens, $0.60/1M output tokens
const inputPricePerToken = 0.15 / 1_000_000;
const outputPricePerToken = 0.60 / 1_000_000;
for (const [tenantId, usage] of Object.entries(summarizer.tenantUsage)) {
// Rough split: assume 70% input, 30% output tokens
const inputTokens = Math.floor(usage.tokens * 0.7);
const outputTokens = Math.floor(usage.tokens * 0.3);
const cost =
inputTokens * inputPricePerToken + outputTokens * outputPricePerToken;
console.log(`\n${tenantId}:`);
console.log(` Total Tokens: ${usage.tokens}`);
console.log(` Estimated Cost: $${cost.toFixed(4)}`);
console.log(` Cost per Call: $${(cost / usage.count).toFixed(4)}`);
}
```
## Step 8: SLA Monitoring
Check which tenants are meeting their SLA commitments:
```python Python theme={null}
sla_results = [
("apex-legal", result1['sla_compliant'], result1['latency_ms']),
("stratex-consulting", result2['sla_compliant'], result2['latency_ms']),
("techstart-inc", result3['sla_compliant'], result3['latency_ms']),
]
for tenant_id, compliant, latency in sla_results:
config = TENANT_CONFIGS[tenant_id]
status = "PASS" if compliant else "FAIL"
sla_text = f"{config.latency_sla_ms}ms" if config.latency_sla_ms else "Best effort"
print(f"\n{tenant_id} ({config.tier}):")
print(f" SLA Target: {sla_text}")
print(f" Actual Latency: {latency:.0f}ms")
print(f" Status: {status}")
```
```typescript TypeScript theme={null}
const slaResults = [
{ tenantId: "apex-legal", ...result1 },
{ tenantId: "stratex-consulting", ...result2 },
{ tenantId: "techstart-inc", ...result3 },
];
for (const result of slaResults) {
const config = TENANT_CONFIGS[result.tenantId];
const status = result.slaCompliant ? "PASS" : "FAIL";
const slaText = config.latencySlaMs
? `${config.latencySlaMs}ms`
: "Best effort";
console.log(`\n${result.tenantId} (${config.tier}):`);
console.log(` SLA Target: ${slaText}`);
console.log(` Actual Latency: ${Math.round(result.latencyMs)}ms`);
console.log(` Status: ${status}`);
}
```
### Setting Up Tenant-Specific Alerts
In the Netra dashboard, navigate to **Alert Rules** and create tenant-filtered alerts:
Click **Create Alert Rule** and name it "Enterprise Latency SLA Breach"
* **Scope**: Trace (monitor end-to-end requests)
* **Metric**: Latency
Add a filter for `tenant_id = apex-legal` to only monitor Enterprise tier requests
* **Condition**: Greater than 2000ms
* **Time Window**: 5 minutes (to avoid alerting on single slow requests)
Select your Slack channel or email for notifications
Create similar alerts for each tier with their respective SLA thresholds:
| Tenant | Tier | Alert Threshold | Rate Limit |
| ------------------ | ------------ | -------------------------- | ------------ |
| apex-legal | Enterprise | > 2000ms | 60 calls/min |
| stratex-consulting | Professional | > 3000ms | 30 calls/min |
| techstart-inc | Starter | No SLA alert (best effort) | 10 calls/min |
## Step 9: Querying Tenant Metrics via Netra API
Once traces are sent to Netra, query tenant-specific metrics programmatically:
```python Python theme={null}
from datetime import datetime, timedelta, timezone
def get_tenant_usage_data(tenant_id: str, start_time: str, end_time: str):
"""Retrieve usage data for a tenant."""
try:
usage = Netra.usage.get_tenant_usage(
tenant_id=tenant_id,
start_time=start_time,
end_time=end_time,
)
if usage:
return {
"tenant_id": usage.tenant_id,
"token_count": usage.token_count,
"request_count": usage.request_count,
"session_count": usage.session_count,
"total_cost": usage.total_cost,
}
return None
except Exception as e:
print(f"Error fetching usage for {tenant_id}: {e}")
return None
# Example: Get usage for a tenant (Last 24 hours)
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(days=1)
for tenant_id in TENANT_CONFIGS.keys():
usage = get_tenant_usage_data(
tenant_id=tenant_id,
start_time=start_time.isoformat(),
end_time=end_time.isoformat(),
)
if usage:
print(f"{tenant_id}: {usage['request_count']} requests, ${usage['total_cost']:.4f}")
else:
print(f"{tenant_id}: No data returned (might be due to ingestion latency)")
```
```typescript TypeScript theme={null}
async function getTenantUsageData(
tenantId: string,
startTime: string,
endTime: string
) {
try {
const usage = await Netra.usage.getTenantUsage({
tenantId,
startTime,
endTime,
});
if (usage) {
return {
tenantId: usage.tenantId,
tokenCount: usage.tokenCount,
requestCount: usage.requestCount,
sessionCount: usage.sessionCount,
totalCost: usage.totalCost,
};
}
return null;
} catch (error) {
console.error(`Error fetching usage for ${tenantId}:`, error);
return null;
}
}
// Example: Get usage for a tenant (Last 24 hours)
const endTime = new Date().toISOString();
const startTime = new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString();
for (const tenantId of Object.keys(TENANT_CONFIGS)) {
const usage = await getTenantUsageData(tenantId, startTime, endTime);
if (usage) {
console.log(
`${tenantId}: ${usage.requestCount} requests, $${usage.totalCost.toFixed(4)}`
);
} else {
console.log(
`${tenantId}: No data returned (might be due to ingestion latency)`
);
}
}
```
***
## What You'll See in the Dashboard
After running this cookbook, check the Netra dashboard for:
* **Tenant selector** filtering all traces to a specific customer
* **Per-tenant cost breakdown** showing usage per customer
* **SLA compliance dashboard** with latency metrics by tier
* **Comparative analytics** showing which customers use which features
* **User activity** filtered by tenant and user ID
## Key Multi-Tenant Patterns
| Pattern | Use Case | How to Implement |
| ----------------- | ------------------------- | ------------------------------------------- |
| Cost attribution | Billing and profitability | Set `tenant_id` at request start |
| SLA monitoring | Support and escalation | Filter by `tenant_id` and latency threshold |
| Feature usage | Product insights | Check feature flags in tenant config |
| User segmentation | Per-user analytics | Set `user_id` in addition to `tenant_id` |
## See Also
Evaluate whether your tier configurations deliver the right quality
Deep dive into tenant tracking features
Query usage data programmatically
Set up proactive monitoring
# Tracing CrewAI Pipelines
Source: https://docs.getnetra.ai/Cookbooks/observability/tracing-crewai-pipelines
Trace CrewAI multi-agent pipelines with Netra. Monitor agent handoffs, track per-agent costs, and debug task delegation in collaborative AI workflows.
This cookbook shows you how to add **complete observability** to CrewAI multi-agent pipelines—tracing agent-to-agent handoffs, measuring individual agent performance, and tracking per-agent costs.
Run the complete notebook in your browser
All company names (ContentCraft) and scenarios in this cookbook are entirely fictional and used for demonstration purposes only.
## What You'll Learn
Capture the message flow between agents as tasks pass through the pipeline
Monitor token usage and costs for each agent role to identify cost drivers
Understand why agents made specific decisions and where quality degrades
Run experiments with different model assignments to find the cost/quality sweet spot
**Prerequisites:**
* Python >=3.10, \<3.14
* OpenAI API key
* Netra API key ([Get started here](/quick-start/Overview))
* CrewAI installed
***
## Why Trace Multi-Agent Systems?
Multi-agent systems introduce complexity that single-agent workflows don't have:
| Failure Mode | Symptom | What Tracing Reveals |
| ------------------- | -------------------- | ------------------------------- |
| Agent bottleneck | Pipeline slow | Which agent takes longest |
| Handoff failure | Context lost | Message content between agents |
| Cost explosion | Budget exceeded | Which agent uses most tokens |
| Quality degradation | Poor output | Where quality drops in pipeline |
| Model mismatch | Inconsistent results | Which model for which role |
Without per-agent visibility, you can't optimize individual roles or identify where the pipeline breaks down.
***
## CrewAI Architecture
CrewAI organizes multi-agent work into three components:
| Component | Description | Example |
| --------- | ---------------------------------------------- | --------------------------------------- |
| **Agent** | Autonomous unit with role, goal, backstory | Research Specialist, Content Writer |
| **Task** | Work item with description and expected output | "Research the topic", "Write the draft" |
| **Crew** | Team of agents executing tasks | Content creation team |
**Processes:**
* **Sequential**: Tasks execute one after another (A → B → C)
* **Hierarchical**: Manager agent delegates to workers
***
## Building an Example Pipeline
### Installation
```bash theme={null}
pip install netra-sdk crewai crewai-tools openai langchain-openai
```
### Environment Setup
```bash theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
```
### Define the Agents
Create a 4-agent content pipeline: Researcher → Writer → Editor → SEO:
```python theme={null}
from crewai import Agent
from langchain_openai import ChatOpenAI
def create_agents(config: dict = None):
"""Create the content team agents with configurable models."""
config = config or {
"researcher": "gpt-4o",
"writer": "gpt-4o",
"editor": "gpt-3.5-turbo",
"seo": "gpt-3.5-turbo",
}
researcher = Agent(
role="Research Specialist",
goal="Gather accurate facts, statistics, and expert opinions",
backstory="Expert researcher with 10 years of experience in content research.",
llm=ChatOpenAI(model=config["researcher"]),
verbose=True,
)
writer = Agent(
role="Content Writer",
goal="Write engaging, well-structured blog articles",
backstory="Professional copywriter with expertise in compelling content.",
llm=ChatOpenAI(model=config["writer"]),
verbose=True,
)
editor = Agent(
role="Quality Editor",
goal="Polish articles for clarity, grammar, and flow",
backstory="Senior editor with a keen eye for detail.",
llm=ChatOpenAI(model=config["editor"]),
verbose=True,
)
seo_specialist = Agent(
role="SEO Optimizer",
goal="Optimize content for search engines",
backstory="SEO expert who balances keywords with readability.",
llm=ChatOpenAI(model=config["seo"]),
verbose=True,
)
return {
"researcher": researcher,
"writer": writer,
"editor": editor,
"seo": seo_specialist,
}
```
### Define the Tasks
Create tasks that chain together:
```python theme={null}
from crewai import Task
def create_tasks(agents: dict, topic: str):
"""Create the content pipeline tasks."""
research_task = Task(
description=f"Research the topic: '{topic}'. Find key facts and statistics.",
expected_output="Research brief with facts, statistics, and sources",
agent=agents["researcher"],
)
writing_task = Task(
description="Write a 800-1000 word blog article based on the research.",
expected_output="Draft blog article in markdown format",
agent=agents["writer"],
context=[research_task],
)
editing_task = Task(
description="Edit the article for grammar, flow, and clarity.",
expected_output="Polished blog article with improved clarity",
agent=agents["editor"],
context=[writing_task],
)
seo_task = Task(
description="Optimize the article for SEO with meta description and keywords.",
expected_output="SEO-optimized article with metadata",
agent=agents["seo"],
context=[editing_task],
)
return [research_task, writing_task, editing_task, seo_task]
```
### Create the Crew
```python theme={null}
from crewai import Crew, Process
def run_content_crew(topic: str, config: dict = None):
"""Execute the content creation pipeline."""
agents = create_agents(config)
tasks = create_tasks(agents, topic)
crew = Crew(
agents=list(agents.values()),
tasks=tasks,
process=Process.sequential,
verbose=True,
)
return crew.kickoff()
```
***
## Adding Netra Observability
### Initialize Netra with Auto-Instrumentation
Netra provides auto-instrumentation for CrewAI that captures agent execution automatically:
```python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Initialize Netra with CrewAI and OpenAI instrumentation
Netra.init(
app_name="contentcraft",
environment="development",
trace_content=True,
instruments={InstrumentSet.CREWAI, InstrumentSet.OPENAI},
)
```
With auto-instrumentation enabled, Netra automatically captures:
* Agent execution spans with role and backstory
* Task execution with descriptions and outputs
* LLM calls with prompts, completions, and token usage
* Cost calculations per agent
### Using the Workflow Decorator
For more control, wrap your pipeline with the `@workflow` decorator:
```python theme={null}
from netra.decorators import workflow
@workflow(name="content-pipeline")
def create_article(topic: str, config_name: str = "default", config: dict = None):
"""Run the content creation pipeline with full tracing."""
# Set custom attributes for filtering and analysis
Netra.set_custom_attributes(key="topic", value=topic)
Netra.set_custom_attributes(key="config_name", value=config_name)
# Run the crew
result = run_content_crew(topic, config)
return {
"topic": topic,
"config": config_name,
"output": result.raw,
}
```
### Adding Custom Span Attributes
Track additional metadata for each pipeline run:
```python theme={null}
from netra import Netra, SpanType
@workflow(name="content-pipeline-detailed")
def create_article_detailed(topic: str, config_name: str, config: dict):
"""Run pipeline with detailed custom tracing."""
with Netra.start_span("pipeline-setup") as setup_span:
setup_span.set_attribute("topic", topic)
setup_span.set_attribute("config_name", config_name)
setup_span.set_attribute("model.researcher", config["researcher"])
setup_span.set_attribute("model.writer", config["writer"])
setup_span.set_attribute("model.editor", config["editor"])
setup_span.set_attribute("model.seo", config["seo"])
agents = create_agents(config)
tasks = create_tasks(agents, topic)
with Netra.start_span("pipeline-execution", as_type=SpanType.AGENT) as exec_span:
crew = Crew(agents=list(agents.values()), tasks=tasks, process=Process.sequential)
result = crew.kickoff()
exec_span.set_attribute("output_length", len(result.raw))
return {"topic": topic, "config": config_name, "output": result.raw}
```
***
## Viewing Traces in Netra
After running the pipeline, navigate to **Observability → Traces** in Netra.
### What the Trace Shows
The trace shows:
* **Pipeline span**: Overall execution time
* **Agent spans**: Each agent's task execution
* **LLM calls**: Nested under each agent with prompts and completions
* **Token usage**: Per-agent and total
## Running Configuration Experiments
Test different model configurations to find the optimal cost/quality balance.
### Define Configurations
```python theme={null}
CONFIGS = {
"premium": {
"researcher": "gpt-4o",
"writer": "gpt-4o",
"editor": "gpt-4o",
"seo": "gpt-4o",
},
"budget": {
"researcher": "gpt-4o",
"writer": "gpt-4o",
"editor": "gpt-3.5-turbo",
"seo": "gpt-3.5-turbo",
},
"economy": {
"researcher": "gpt-4o",
"writer": "gpt-3.5-turbo",
"editor": "gpt-3.5-turbo",
"seo": "gpt-3.5-turbo",
},
}
```
### Run Experiments
```python theme={null}
# Test each configuration
for config_name, config in CONFIGS.items():
print(f"Running {config_name} configuration...")
result = create_article(
topic="The Future of AI in Healthcare",
config_name=config_name,
config=config,
)
print(f"{config_name}: {len(result['output'])} characters")
```
### Compare in Dashboard
After running all configurations, compare costs and latency:
| Config | Total Cost | Total Latency | Output Quality |
| ------- | ---------- | ------------- | -------------- |
| Premium | \~\$0.19 | \~45s | Highest |
| Budget | \~\$0.145 | \~40s | Good |
| Economy | \~\$0.085 | \~35s | Acceptable |
***
## Debugging Multi-Agent Issues
### Common Problems and Solutions
| Problem | What to Look For | Solution |
| ------------------------------- | ---------------------------- | ----------------------------------- |
| **Slow pipeline** | High latency on one agent | Use faster model or shorter prompts |
| **Context lost between agents** | Missing info in task outputs | Improve task descriptions |
| **Editor making no changes** | Low edit delta | Improve editor prompts |
| **High total cost** | One agent dominating | Downgrade non-critical agents |
### Using Traces to Debug
1. **Find slow agents**: Sort spans by duration
2. **Trace context flow**: Check task outputs passed between agents
3. **Identify cost drivers**: Filter by token usage
4. **Compare successful vs failed**: Look for pattern differences
***
## Summary
You've learned how to add comprehensive observability to CrewAI pipelines:
* **Auto-instrumentation** captures agent execution with minimal code
* **Per-agent tracing** reveals costs, latency, and token usage
* **Custom attributes** enable filtering by topic, config, and more
* **Configuration experiments** find the optimal cost/quality balance
### Key Takeaways
1. Multi-agent systems need per-agent visibility to identify bottlenecks
2. Cost allocation by role reveals which agents benefit from premium models
3. Trace context flow to debug handoff issues
4. Use configuration experiments for data-driven model selection
***
## See Also
Complete CrewAI instrumentation guide
Deep dive into agent observability features
Query cost and usage data programmatically
# Tracing LangChain Agents
Source: https://docs.getnetra.ai/Cookbooks/observability/tracing-langchain-agents
Trace LangChain ReAct agents with Netra. Capture reasoning loops, tool calls, and decision points to debug and optimize multi-step AI agent workflows.
This cookbook shows you how to add **complete observability** to LangChain ReAct agents—tracing each step of the reasoning loop, capturing tool invocations with latency breakdowns, and understanding your agent's decision-making process.
Run the complete notebook in your browser
All company names (TaskBot, ShopFlow) and scenarios in this cookbook are entirely fictional and used for demonstration purposes only.
## What You'll Learn
Capture each iteration of thought → action → observation with Netra spans
Monitor tool invocations with latency, inputs, outputs, and cost
Understand why your agent made specific decisions using trace analysis
Enrich traces with user IDs, session context, and custom attributes
**Prerequisites:**
* Python >=3.10, \<3.14
* OpenAI API key
* Netra API key ([Get started here](/quick-start/Overview))
* LangChain installed
***
## Why Trace Agents?
Unlike simple LLM calls, agents involve multi-step reasoning that can fail in subtle ways:
| Failure Mode | Symptom | What Tracing Reveals |
| --------------------- | ------------------------------ | ----------------------------------------- |
| Wrong tool selection | Agent uses incorrect tool | Tool call sequence, decision reasoning |
| Infinite loops | Agent repeats actions | Iteration count, repeated patterns |
| Hallucinated tools | Agent calls non-existent tool | Tool names vs. available tools |
| Premature termination | Agent stops before completion | Final state, missing steps |
| Over-escalation | Agent escalates simple queries | Escalation triggers, query classification |
Without visibility into the reasoning loop, debugging these failures requires guesswork.
***
## The ReAct Pattern
ReAct (Reasoning + Acting) agents follow an iterative loop:
Netra captures each iteration as nested spans, giving you visibility into the agent's decision-making process.
***
## Building the Example Agent
### Installation
```bash theme={null}
pip install netra-sdk langchain langchain-openai openai
```
### Environment Setup
```bash theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
```
### Mock Data and Tools
First, let's define a simple agent with multiple tools:
```python theme={null}
from typing import Dict, List
from langchain.tools import tool
from langchain_openai import ChatOpenAI
from langchain.agents import create_react_agent
# Mock databases
TICKETS = {
"TKT-001": {"id": "TKT-001", "subject": "Return policy question", "status": "open"},
"TKT-002": {"id": "TKT-002", "subject": "Damaged item", "status": "open", "order_id": "ORD-12345"},
}
ORDERS = {
"ORD-12345": {"id": "ORD-12345", "status": "delivered", "items": ["Headphones"], "total": 79.99},
}
KNOWLEDGE_BASE = [
{"title": "Return Policy", "content": "Items can be returned within 30 days."},
{"title": "Refund Processing", "content": "Refunds processed in 5-7 business days."},
]
@tool
def lookup_ticket(ticket_id: str) -> str:
"""Look up a ticket by its ID to get details about the issue."""
ticket = TICKETS.get(ticket_id.upper())
if not ticket:
return f"No ticket found with ID: {ticket_id}"
return f"Ticket {ticket['id']}: {ticket['subject']} (Status: {ticket['status']})"
@tool
def search_kb(query: str) -> str:
"""Search the knowledge base for information about policies or procedures."""
query_lower = query.lower()
results = [a for a in KNOWLEDGE_BASE if query_lower in a["title"].lower()]
if not results:
return "No relevant articles found."
return "\n".join([f"**{a['title']}**: {a['content']}" for a in results])
@tool
def check_order_status(order_id: str) -> str:
"""Check the status of an order including shipping information."""
order = ORDERS.get(order_id.upper())
if not order:
return f"No order found with ID: {order_id}"
return f"Order {order['id']}: {order['status']}, Items: {order['items']}, Total: ${order['total']}"
@tool
def escalate_to_human(ticket_id: str, reason: str) -> str:
"""Escalate a ticket to a human operator for complex issues."""
return f"Ticket {ticket_id} escalated. Reason: {reason}. A specialist will respond within 1 hour."
```
### Create the Agent
```python theme={null}
# Initialize the LLM
model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
# Define tools
tools = [lookup_ticket, search_kb, check_order_status, escalate_to_human]
# Create the agent
agent = create_react_agent(
model,
tools,
prompt="""You are TaskBot, an AI assistant for ShopFlow e-commerce platform.
You help users with:
- Order status and tracking
- Return and refund requests
- Policy questions
- Escalating complex issues
Use tools to look up information before responding.
Escalate to human operators when the user is frustrated or you cannot resolve the issue."""
)
```
***
## Adding Netra Observability
### Initialize Netra with Auto-Instrumentation
Netra provides auto-instrumentation for LangChain that captures agent execution automatically:
```python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Initialize Netra with LangChain and OpenAI instrumentation
Netra.init(
app_name="taskbot",
environment="development",
trace_content=True,
instruments={InstrumentSet.OPENAI, InstrumentSet.LANGCHAIN},
)
```
With auto-instrumentation enabled, Netra automatically captures:
* Agent execution spans
* LLM calls with prompts and completions
* Tool invocations with inputs and outputs
* Token usage and costs
### Tracing Agent Execution with Decorators
For more control, wrap your agent handler with the `@agent` decorator:
```python theme={null}
from netra.decorators import agent
@agent(name="taskbot-agent")
def handle_request(query: str, user_id: str = None) -> dict:
"""Handle a user request with full tracing."""
# Set user context if provided
if user_id:
Netra.set_user_id(user_id)
# Execute the agent
result = agent.invoke({
"messages": [{"role": "user", "content": query}]
})
return {
"query": query,
"response": result["messages"][-1].content,
}
```
### Adding Custom Span Attributes
Enrich tool traces with custom attributes for better filtering and analysis:
```python theme={null}
from netra import Netra, SpanType
@tool
def lookup_ticket_traced(ticket_id: str) -> str:
"""Look up a ticket with custom span attributes."""
with Netra.start_span("ticket-lookup", as_type=SpanType.TOOL) as span:
span.set_attribute("ticket_id", ticket_id)
ticket = TICKETS.get(ticket_id.upper())
if ticket:
span.set_attribute("ticket_status", ticket["status"])
span.set_attribute("ticket_priority", ticket.get("priority", "normal"))
span.set_attribute("found", True)
else:
span.set_attribute("found", False)
if not ticket:
return f"No ticket found with ID: {ticket_id}"
return f"Ticket {ticket['id']}: {ticket['subject']} (Status: {ticket['status']})"
```
***
## Running Sample Requests
Let's test the agent with different query types to see tracing in action.
### Simple Query: FAQ Lookup
```python theme={null}
# Single-tool query - should use search_kb
response = handle_request(
query="What is your return policy?",
user_id="user-001",
)
print(response["response"])
```
**Expected behavior**: Agent uses `search_kb` once and returns the policy information.
### Order Status Query
```python theme={null}
# Order status query - should use check_order_status
response = handle_request(
query="Where is my order ORD-12345?",
user_id="user-002",
)
print(response["response"])
```
**Expected behavior**: Agent uses `check_order_status` and provides tracking information.
### Multi-Step Query
```python theme={null}
# Multi-step workflow - should use multiple tools
response = handle_request(
query="I have ticket TKT-002 about a damaged item. Can you check the order status?",
user_id="user-003",
)
print(response["response"])
```
**Expected behavior**: Agent uses `lookup_ticket` to get context, then `check_order_status` to verify the order.
### Escalation Scenario
```python theme={null}
# Escalation scenario - should detect urgency
response = handle_request(
query="I've been waiting 3 weeks and need urgent help! I want to speak to someone immediately!",
user_id="user-004",
)
print(response["response"])
```
**Expected behavior**: Agent recognizes urgency and uses `escalate_to_human`.
***
## Viewing Traces in Netra
After running requests, navigate to **Observability → Traces** in Netra. You'll see the full agent execution flow:
### What the Trace Shows
* **Parent span**: The overall agent execution
* **LLM calls**: Each reasoning step with prompts and completions
* **Tool calls**: Each tool invocation with inputs, outputs, and latency
* **Token usage**: Cumulative token counts and costs
### Filtering and Analysis
Use Netra's filtering to analyze agent behavior:
| Filter | Use Case |
| ------------------------------- | ---------------------------------- |
| `tool.name = escalate_to_human` | Find all escalation decisions |
| `user_id = user-004` | Debug a specific user's experience |
| `latency > 5000ms` | Find slow agent executions |
| `status = error` | Identify failed requests |
***
## Summary
You've learned how to add comprehensive observability to LangChain agents:
* **Auto-instrumentation** captures agent execution with minimal code
* **Custom spans** add business context to tool calls
* **Trace analysis** reveals reasoning patterns and failure modes
* **Session tracking** connects multi-turn conversations
### Key Takeaways
1. ReAct agents need visibility into the reasoning loop—trace each thought, action, and observation
2. Tool call tracing reveals latency bottlenecks and decision patterns
3. Custom attributes enable filtering by query type, user, and business context
4. Session IDs connect related requests for conversation analysis
***
## See Also
Add systematic evaluation to measure tool selection accuracy
Complete LangChain instrumentation guide
Deep dive into agent observability features
Understanding trace structure and analysis
# Tracing a RAG Pipeline
Source: https://docs.getnetra.ai/Cookbooks/observability/tracing-rag-pipeline
Add full observability to a RAG pipeline with Netra. Auto-instrument retrieval, embedding, and generation steps to trace every query end-to-end.
This cookbook walks you through adding **full observability** to a Retrieval-Augmented Generation (RAG) pipeline—tracing every stage from document ingestion to answer generation, tracking costs, and monitoring performance.
Run the complete observability notebook in your browser
## What You'll Learn
Create a complete RAG chatbot that loads PDFs, chunks documents, generates embeddings, and retrieves relevant context for answering questions.
Instrument every stage—chunking, embedding, retrieval, and generation—with Netra auto-tracing to capture the full execution flow.
Monitor token usage, API costs, and latency at each step to identify bottlenecks and optimize your pipeline.
Track usage per user and session to understand conversation flows and user behavior.
**Prerequisites:**
* Python >=3.10, \<3.14 or Node.js 18+
* OpenAI API key
* Netra API key ([Steps mentioned here](https://docs.getnetra.ai/quick-start/Overview))
***
## High-Level Concepts
### RAG Architecture
A RAG chatbot works in two phases:
**Ingestion (one-time):**
1. Load and chunk the PDF into smaller text segments
2. Generate embeddings for each chunk
3. Store embeddings in a vector database
**Query (per question):**
1. Convert the user's question to an embedding
2. Find the most similar chunks (retrieval)
3. Pass retrieved chunks + question to an LLM
4. Return the generated answer
### Why Observability Matters for RAG
RAG systems can fail silently in multiple ways:
| Problem | Symptom | What Tracing Reveals |
| --------------- | ------------------ | ----------------------------------- |
| Poor chunking | Incomplete answers | Chunk sizes, content boundaries |
| Wrong retrieval | Irrelevant answers | Similarity scores, retrieved chunks |
| Hallucination | Fabricated info | Context vs. generated content |
| High costs | Budget overruns | Token usage per stage |
***
## Creating the Chat Agent
Let's build the RAG chatbot first, then add tracing.
### Installation
Start by installing the required packages. We'll use OpenAI for embeddings and generation, ChromaDB as our vector store, and a PDF parsing library.
```bash Python theme={null}
pip install netra-sdk openai chromadb pypdf reportlab
```
```bash TypeScript theme={null}
npm install netra-sdk openai chromadb pdf-parse @chroma-core/default-embed
```
### Environment Setup
Configure your API keys. You'll need both an OpenAI key for the LLM operations and a Netra key for observability.
```bash Python theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
```
```bash typescript theme={null}
export NETRA_API_KEY="your-netra-api-key"
export NETRA_OTLP_ENDPOINT="your-netra-otlp-endpoint"
export OPENAI_API_KEY="your-openai-api-key"
chroma run --path /tmp/chroma_db # The typescript chroma client does not have an in memory mode.
```
### Loading and Chunking Documents
The first step in any RAG pipeline is extracting text from your documents and splitting it into manageable chunks. We use overlapping chunks to ensure context isn't lost at chunk boundaries—this helps when relevant information spans multiple segments.
```python Python theme={null}
# Import required libraries
from pypdf import PdfReader
from typing import List, Dict, Optional
import chromadb
from openai import OpenAI
import uuid
# Initialize clients
openai_client = OpenAI()
chroma_client = chromadb.Client()
def load_pdf(file_path: str) -> str:
"""Extract text from a PDF file."""
reader = PdfReader(file_path)
text = ""
for page in reader.pages:
text += page.extract_text() + "\n"
return text
def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 200) -> List[str]:
"""Split text into overlapping chunks."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end]
chunks.append(chunk)
start = end - overlap
return chunks
```
```typescript TypeScript theme={null}
// Import required libraries
import { PDFParse } from "pdf-parse";
import * as fs from "fs";
import { ChromaClient } from "chromadb";
import OpenAI from "openai";
import { v4 as uuid } from "uuid";
// Initialize clients
const openai = new OpenAI();
const chroma = new ChromaClient();
async function loadPdf(filePath: string): Promise {
const dataBuffer = fs.readFileSync(filePath);
const parser = new PDFParse({
data: dataBuffer,
});
const data = await parser.getText();
return data.text;
}
function chunkText(text: string, chunkSize = 1000, overlap = 200): string[] {
const chunks: string[] = [];
let start = 0;
while (start < text.length) {
const end = start + chunkSize;
const chunk = text.slice(start, end);
chunks.push(chunk);
start = end - overlap;
}
return chunks;
}
```
### Generating Embeddings and Indexing
Next, we convert each chunk into a vector embedding and store it in ChromaDB. These embeddings capture the semantic meaning of each chunk, allowing us to find relevant content based on meaning rather than just keywords.
```python Python theme={null}
def generate_embeddings(texts: List[str]) -> List[List[float]]:
"""Generate embeddings for a list of texts."""
response = openai_client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
return [item.embedding for item in response.data]
# Load and chunk the PDF
pdf_text = load_pdf("document.pdf")
chunks = chunk_text(pdf_text, chunk_size=1000, overlap=200)
print(f"Created {len(chunks)} chunks")
# Generate embeddings and store in ChromaDB
collection = chroma_client.create_collection(name="pdf_qa")
embeddings = generate_embeddings(chunks)
collection.add(
documents=chunks,
embeddings=embeddings,
ids=[f"chunk_{i}" for i in range(len(chunks))]
)
print(f"Stored {len(chunks)} chunks in vector database")
```
```typescript TypeScript theme={null}
function generateEmbeddings(texts: string[]): Promise {
return openai.embeddings.create({
model: "text-embedding-3-small",
input: texts,
}).then(response => response.data.map((item) => item.embedding));
}
// Load and chunk the PDF, then store embeddings
(async () => {
const pdfText = await loadPdf("document.pdf");
const chunks = chunkText(pdfText, 1000, 200);
console.log(`Created ${chunks.length} chunks`);
// Generate embeddings and store in ChromaDB
const collection = await chroma.createCollection({ name: "pdf_qa" });
const embeddings = await generateEmbeddings(chunks);
await collection.add({
documents: chunks,
embeddings: embeddings,
ids: chunks.map((_, i) => `chunk_${i}`),
});
console.log(`Stored ${chunks.length} chunks in vector database`);
})();
```
### Building the Query Pipeline
Now we implement the core RAG logic: given a user question, retrieve the most relevant chunks from our vector store, then pass them as context to the LLM to generate an answer. The `top_k` parameter controls how many chunks we retrieve—more chunks provide more context but also increase cost and latency.
```python Python theme={null}
def retrieve_chunks(query: str, top_k: int = 3) -> List[Dict]:
"""Retrieve the most relevant chunks for a query."""
query_embedding = generate_embeddings([query])[0]
results = collection.query(
query_embeddings=[query_embedding],
n_results=top_k,
include=["documents", "distances"]
)
retrieved = []
for i, doc in enumerate(results["documents"][0]):
retrieved.append({
"content": doc,
"similarity_score": 1 - results["distances"][0][i] # Convert distance to similarity
})
return retrieved
def generate_answer(query: str, context_chunks: List[Dict]) -> str:
"""Generate an answer using the retrieved context."""
context = "\n\n".join([chunk["content"] for chunk in context_chunks])
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": """You are a helpful assistant that answers questions based on the provided context.
Only use information from the context to answer. If the answer is not in the context, say so."""
},
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {query}"
}
]
)
return response.choices[0].message.content
# Test the query pipeline
test_query = "What is the main topic of this document?"
retrieved_chunks = retrieve_chunks(test_query, top_k=3)
answer = generate_answer(test_query, retrieved_chunks)
print(f"Answer: {answer}")
```
```typescript TypeScript theme={null}
interface RetrievedChunk {
content: string;
similarityScore: number;
}
// Note: collection variable needs to be accessible from previous code block
async function retrieveChunks(
coll: any,
query: string,
topK = 3
): Promise {
const queryEmbedding = (await generateEmbeddings([query]))[0];
const results = await coll.query({
queryEmbeddings: [queryEmbedding],
nResults: topK,
include: ["documents", "distances"],
});
const retrieved: RetrievedChunk[] = [];
for (let i = 0; i < results.documents[0].length; i++) {
retrieved.push({
content: results.documents[0][i] as string,
similarityScore: 1 - (results.distances[0][i] as number),
});
}
return retrieved;
}
async function generateAnswer(
query: string,
contextChunks: RetrievedChunk[]
): Promise {
const context = contextChunks.map((chunk) => chunk.content).join("\n\n");
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "system",
content: `You are a helpful assistant that answers questions based on the provided context.
Only use information from the context to answer. If the answer is not in the context, say so.`,
},
{
role: "user",
content: `Context:\n${context}\n\nQuestion: ${query}`,
},
],
});
return response.choices[0]?.message.content || "";
}
// Test the query pipeline (requires collection from previous step)
(async () => {
// Assume collection is available from previous code block
const collection = await chroma.getCollection({ name: "pdf_qa" });
const testQuery = "What is the main topic of this document?";
const retrievedChunks = await retrieveChunks(collection, testQuery, 3);
const answer = await generateAnswer(testQuery, retrievedChunks);
console.log(`Answer: ${answer}`);
})();
```
### Adding Session Support
For production use, we wrap everything in a class that maintains conversation history and session state. This enables multi-turn conversations where the chatbot remembers previous exchanges, and allows us to track usage per user and session.
```python Python theme={null}
class PDFChatbot:
def __init__(self, pdf_path: str):
self.pdf_path = pdf_path
self.conversation_history = []
self.session_id = str(uuid.uuid4())
self._setup_vector_store()
def _setup_vector_store(self):
"""Initialize the vector store with PDF content."""
pdf_text = load_pdf(self.pdf_path)
self.chunks = chunk_text(pdf_text)
embeddings = generate_embeddings(self.chunks)
self.collection = chroma_client.create_collection(
name=f"pdf_{self.session_id}"
)
self.collection.add(
documents=self.chunks,
embeddings=embeddings,
ids=[f"chunk_{i}" for i in range(len(self.chunks))]
)
def chat(self, query: str, user_id: Optional[str] = None) -> Dict:
"""Process a chat message and return the response."""
# Retrieve relevant chunks
retrieved = self._retrieve(query)
# Build conversation context
context = "\n\n".join([chunk["content"] for chunk in retrieved])
# Generate response
messages = [
{
"role": "system",
"content": f"""You are a helpful assistant answering questions about a PDF document.
Use the following context to answer questions. If the answer is not in the context, say so.
Context:
{context}"""
}
]
# Add conversation history
for msg in self.conversation_history[-6:]: # Last 3 exchanges
messages.append(msg)
messages.append({"role": "user", "content": query})
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=messages
)
answer = response.choices[0].message.content
# Update conversation history
self.conversation_history.append({"role": "user", "content": query})
self.conversation_history.append({"role": "assistant", "content": answer})
return {
"query": query,
"answer": answer,
"retrieved_chunks": retrieved,
"session_id": self.session_id,
"user_id": user_id,
"token_usage": {
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens
}
}
def _retrieve(self, query: str, top_k: int = 3) -> List[Dict]:
"""Retrieve relevant chunks."""
query_embedding = generate_embeddings([query])[0]
results = self.collection.query(
query_embeddings=[query_embedding],
n_results=top_k,
include=["documents", "distances"]
)
retrieved = []
for i, doc in enumerate(results["documents"][0]):
retrieved.append({
"content": doc,
"similarity_score": 1 - results["distances"][0][i]
})
return retrieved
# Usage
chatbot = PDFChatbot("document.pdf")
response = chatbot.chat("What is the main topic?", user_id="user-123")
print(response["answer"])
```
```typescript TypeScript theme={null}
interface ChatMessage {
role: "user" | "assistant" | "system";
content: string;
}
interface ChatResponse {
query: string;
answer: string;
retrievedChunks: RetrievedChunk[];
sessionId: string;
userId?: string;
tokenUsage: {
promptTokens: number;
completionTokens: number;
totalTokens: number;
};
}
class PDFChatbot {
pdfPath: string;
conversationHistory: ChatMessage[] = [];
sessionId: string;
collection: any;
chunks: string[] = [];
constructor(pdfPath: string) {
this.pdfPath = pdfPath;
this.sessionId = uuid();
}
async initialize() {
const pdfText = await loadPdf(this.pdfPath);
this.chunks = chunkText(pdfText);
const embeddings = await generateEmbeddings(this.chunks);
this.collection = await chroma.createCollection({
name: `pdf_${this.sessionId}`,
});
await this.collection.add({
documents: this.chunks,
embeddings: embeddings,
ids: this.chunks.map((_, i) => `chunk_${i}`),
});
}
async chat(query: string, userId?: string): Promise {
// Retrieve relevant chunks
const retrieved = await this.retrieve(query);
// Build conversation context
const context = retrieved.map((chunk) => chunk.content).join("\n\n");
// Build messages
const messages: ChatMessage[] = [
{
role: "system",
content: `You are a helpful assistant answering questions about a PDF document.
Use the following context to answer questions. If the answer is not in the context, say so.
Context:
${context}`,
},
];
// Add conversation history (last 3 exchanges)
messages.push(...this.conversationHistory.slice(-6));
messages.push({ role: "user", content: query });
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: messages,
});
const answer = response.choices[0].message.content || "";
// Update conversation history
this.conversationHistory.push({ role: "user", content: query });
this.conversationHistory.push({ role: "assistant", content: answer });
return {
query,
answer,
retrievedChunks: retrieved,
sessionId: this.sessionId,
userId,
tokenUsage: {
promptTokens: response.usage?.prompt_tokens || 0,
completionTokens: response.usage?.completion_tokens || 0,
totalTokens: response.usage?.total_tokens || 0,
},
};
}
async retrieve(query: string, topK = 3): Promise {
const queryEmbedding = (await generateEmbeddings([query]))[0];
const results = await this.collection.query({
queryEmbeddings: [queryEmbedding],
nResults: topK,
include: ["documents", "distances"],
});
const retrieved: RetrievedChunk[] = [];
for (let i = 0; i < results.documents[0].length; i++) {
retrieved.push({
content: results.documents[0][i] as string,
similarityScore: 1 - (results.distances[0][i] as number),
});
}
return retrieved;
}
}
// Usage
(async () => {
const chatbot = new PDFChatbot("document.pdf");
await chatbot.initialize();
const response = await chatbot.chat("What is the main topic?", "user-123");
console.log(response.answer);
})();
```
***
## Tracing the Agent
Now let's add Netra observability to see what's happening inside the RAG pipeline. The good news: with auto-instrumentation, you get full visibility with minimal code changes.
### Initializing Netra
Add these imports and initialization at the very top of your script, before any other code. Auto-instrumentation captures all OpenAI and ChromaDB operations automatically—no decorators or manual spans required.
```python Python theme={null}
# Add these imports at the top, before other imports
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Initialize Netra before any other code
Netra.init(
app_name="pdf-qa-chatbot",
environment="development",
trace_content=True,
instruments={
InstrumentSet.OPENAI,
InstrumentSet.CHROMA,
}
)
# Now continue with the rest of your imports and code from earlier sections
# from pypdf import PdfReader
# from typing import List, Dict, Optional
# ...
```
```typescript TypeScript theme={null}
// Add these imports at the top, before other imports
import { Netra, NetraInstruments } from "netra-sdk";
// Initialize Netra before any other code
// Use an immediately-invoked async function to handle the await
Netra.init({
appName: "pdf-qa-chatbot",
environment: "development",
traceContent: true,
instruments: new Set([NetraInstruments.OPENAI, NetraInstruments.CHROMADB]),
});
// Now continue with the rest of your imports and code from earlier sections
// import {PDFParse} from "pdf-parse";
// import * as fs from "fs";
// ...
```
**What gets auto-traced with zero code changes:**
* OpenAI chat completions with model, tokens, cost, and latency
* OpenAI embeddings with token counts
* ChromaDB queries and inserts with timing
* Full prompts and responses (when `trace_content=True`)
### What Gets Auto-Traced
With the initialization above, your existing code from the [Creating the Chat agent](#creating-the-chat-agent) section is automatically traced. Here's what appears in your Netra dashboard:
#### Document Ingestion
The `generate_embeddings()` call to OpenAI and `collection.add()` to ChromaDB are captured automatically.
#### Retrieval Operations
Query embedding generation and vector search operations appear as child spans with timing and metadata.
#### LLM Generation
OpenAI chat completions are fully traced with model, tokens, cost, latency, and full prompt/response content.
### Adding User and Session Tracking
To analyze usage per user and track conversation flows, add user and session context to your existing PDFChatbot class. This is the one piece that requires explicit code—everything else is auto-traced. Simply add these two lines in your `chat` method:
```python Python theme={null}
# Modify the chat method in your existing PDFChatbot class:
def chat(self, query: str, user_id: Optional[str] = None) -> Dict:
"""Process a chat message and return the response."""
# Add these two lines to enable user and session tracking
Netra.set_session_id(self.session_id)
if user_id:
Netra.set_user_id(user_id)
# Rest of the method remains the same
retrieved = self._retrieve(query)
context = "\n\n".join([chunk["content"] for chunk in retrieved])
# ... (rest of your existing code)
```
```typescript TypeScript theme={null}
// Modify the chat method in your existing PDFChatbot class:
async function chat(query: string, userId?: string): Promise {
// Add these two lines to enable user and session tracking
Netra.setSessionId(this.sessionId);
if (userId) {
Netra.setUserId(userId);
}
// Rest of the method remains the same
const retrieved = await this.retrieve(query);
const context = retrieved.map((chunk) => chunk.content).join("\n\n");
// ... (rest of your existing code)
}
```
### What You'll See in the Dashboard
After running the chatbot, you'll see traces in the Netra dashboard with:
* **OpenAI spans** showing model, tokens, cost, and full prompt/response
* **ChromaDB spans** showing query timing and results
* **User and session IDs** attached to all spans for filtering
### Using Decorators
Auto-instrumentation handles most cases of tracing but if you want to bring in more structure, you can use decorators.
Use decorators to create parent spans that group related operations. This is useful when you want to see a single trace for an entire pipeline rather than individual OpenAI/ChromaDB calls.
| Decorator | Use Case |
| ----------- | -------------------------------------------- |
| `@workflow` | Top-level pipeline or request handler |
| `@task` | Discrete unit of work within a workflow |
| `@span` | Fine-grained tracing for specific operations |
```python Python theme={null}
import os
import uuid
from typing import List, Dict, Optional
from pypdf import PdfReader
import chromadb
from openai import OpenAI
from netra import Netra
from netra.decorators import workflow, task, span
from netra.instrumentation.instruments import InstrumentSet
# Initialize Netra with auto-instrumentation
Netra.init(
app_name="pdf-qa-chatbot",
environment="development",
trace_content=True,
instruments={
InstrumentSet.OPENAI,
InstrumentSet.CHROMA,
}
)
# Initialize clients
openai_client = OpenAI()
chroma_client = chromadb.Client()
def generate_embeddings(texts: List[str]) -> List[List[float]]:
"""Generate embeddings for a list of texts."""
response = openai_client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
return [item.embedding for item in response.data]
@task(name="load-pdf")
def load_pdf(file_path: str) -> str:
"""Extract text from a PDF file."""
reader = PdfReader(file_path)
text = ""
for page in reader.pages:
text += page.extract_text() + "\n"
return text
@task(name="chunk-text")
def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 200) -> List[str]:
"""Split text into overlapping chunks."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end]
chunks.append(chunk)
start = end - overlap
return chunks
class PDFChatbot:
"""A RAG-based chatbot for answering questions about PDF documents."""
def __init__(self, pdf_path: str):
self.pdf_path = pdf_path
self.session_id = str(uuid.uuid4())
self.collection = None
self.chunks: List[str] = []
self.conversation_history: List[Dict] = []
@task(name="document-ingestion")
def initialize(self):
"""Initialize the vector store with PDF content."""
pdf_text = load_pdf(self.pdf_path)
self.chunks = chunk_text(pdf_text)
embeddings = generate_embeddings(self.chunks)
self.collection = chroma_client.create_collection(name=f"pdf_{self.session_id[:8]}")
self.collection.add(
documents=self.chunks,
embeddings=embeddings,
ids=[f"chunk_{i}" for i in range(len(self.chunks))]
)
@workflow(name="pdf-qa-query")
def chat(self, query: str, user_id: Optional[str] = None) -> Dict:
"""Process a chat message and return the response."""
Netra.set_session_id(self.session_id)
if user_id:
Netra.set_user_id(user_id)
retrieved = self._retrieve(query)
answer, response = self._generate_answer(query, retrieved)
# Update conversation history
self.conversation_history.append({"role": "user", "content": query})
self.conversation_history.append({"role": "assistant", "content": answer})
return {"query": query, "answer": answer, "retrieved_chunks": retrieved}
@task(name="retrieval")
def _retrieve(self, query: str, top_k: int = 3) -> List[Dict]:
"""Retrieve relevant chunks."""
query_embedding = self._get_query_embedding(query)
retrieved = self._vector_search(query_embedding, top_k)
return retrieved
@span(name="query-embedding")
def _get_query_embedding(self, query: str) -> List[float]:
"""Generate embedding for the query."""
return generate_embeddings([query])[0]
@span(name="vector-search")
def _vector_search(self, query_embedding: List[float], top_k: int) -> List[Dict]:
"""Search vector database for relevant chunks."""
results = self.collection.query(
query_embeddings=[query_embedding],
n_results=top_k,
include=["documents", "distances"]
)
return [{"content": doc, "similarity_score": 1 - results["distances"][0][i]}
for i, doc in enumerate(results["documents"][0])]
@span(name="answer-generation")
def _generate_answer(self, query: str, retrieved: List[Dict]):
"""Generate answer using retrieved context."""
context = "\n\n".join([chunk["content"] for chunk in retrieved])
messages = [
{"role": "system", "content": f"Use this context to answer: {context}"},
{"role": "user", "content": query}
]
response = openai_client.chat.completions.create(model="gpt-4o-mini", messages=messages)
return response.choices[0].message.content, response
# Usage
chatbot = PDFChatbot("document.pdf")
chatbot.initialize()
response = chatbot.chat("What is the main topic?", user_id="user-123")
print(response["answer"])
Netra.shutdown()
```
```typescript TypeScript theme={null}
import fs from "fs/promises";
import pdfParse from "pdf-parse";
import { ChromaClient } from "chromadb";
import OpenAI from "openai";
import { Netra, NetraInstruments, workflow, task, span } from "netra-sdk";
// Initialize Netra with auto-instrumentation
await Netra.init({
appName: "pdf-qa-chatbot",
environment: "development",
traceContent: true,
instruments: new Set([NetraInstruments.OPENAI, NetraInstruments.CHROMA]),
});
// Initialize clients
const openaiClient = new OpenAI();
const chromaClient = new ChromaClient();
async function generateEmbeddings(texts: string[]): Promise {
const response = await openaiClient.embeddings.create({
model: "text-embedding-3-small",
input: texts,
});
return response.data.map((item) => item.embedding);
}
// Task wrapper for loading PDF
const loadPdf = task("load-pdf", async (filePath: string): Promise => {
const pdfData = await fs.readFile(filePath);
const pdf = await pdfParse(pdfData);
return pdf.text;
});
// Task wrapper for chunking text
const chunkText = task("chunk-text", (
text: string,
chunkSize: number = 1000,
overlap: number = 200
): string[] => {
const chunks: string[] = [];
let start = 0;
while (start < text.length) {
const end = start + chunkSize;
chunks.push(text.slice(start, end));
start = end - overlap;
}
return chunks;
});
class PDFChatbot {
pdfPath: string;
sessionId: string;
collection: any;
chunks: string[] = [];
conversationHistory: Array<{ role: string; content: string }> = [];
constructor(pdfPath: string) {
this.pdfPath = pdfPath;
this.sessionId = crypto.randomUUID();
}
// Task wrapper for document ingestion
initialize = task("document-ingestion", async () => {
const pdfText = await loadPdf(this.pdfPath);
this.chunks = chunkText(pdfText);
const embeddings = await generateEmbeddings(this.chunks);
this.collection = await chromaClient.createCollection({
name: `pdf_${this.sessionId.slice(0, 8)}`
});
await this.collection.add({
documents: this.chunks,
embeddings,
ids: this.chunks.map((_, i) => `chunk_${i}`)
});
});
// Workflow wrapper for chat
chat = workflow("pdf-qa-query", async (
query: string,
userId?: string
): Promise> => {
Netra.setSessionId(this.sessionId);
if (userId) {
Netra.setUserId(userId);
}
const retrieved = await this.retrieve(query);
const { answer, response } = await this.generateAnswer(query, retrieved);
// Update conversation history
this.conversationHistory.push({ role: "user", content: query });
this.conversationHistory.push({ role: "assistant", content: answer });
return { query, answer, retrievedChunks: retrieved };
});
// Task wrapper for retrieval
retrieve = task("retrieval", async (
query: string,
topK: number = 3
): Promise> => {
const queryEmbedding = await this.getQueryEmbedding(query);
return this.vectorSearch(queryEmbedding, topK);
});
// Span wrapper for query embedding
getQueryEmbedding = span("query-embedding", async (
query: string
): Promise => {
const embeddings = await generateEmbeddings([query]);
return embeddings[0];
});
// Span wrapper for vector search
vectorSearch = span("vector-search", async (
queryEmbedding: number[],
topK: number
): Promise> => {
const results = await this.collection.query({
queryEmbeddings: [queryEmbedding],
nResults: topK,
include: ["documents", "distances"]
});
return results.documents[0].map((doc: string, i: number) => ({
content: doc,
similarityScore: 1 - results.distances[0][i]
}));
});
// Span wrapper for answer generation
generateAnswer = span("answer-generation", async (
query: string,
retrieved: Array<{ content: string }>
) => {
const context = retrieved.map(chunk => chunk.content).join("\n\n");
const messages = [
{ role: "system" as const, content: `Use this context to answer: ${context}` },
{ role: "user" as const, content: query }
];
const response = await openaiClient.chat.completions.create({
model: "gpt-4o-mini",
messages
});
return {
answer: response.choices[0].message.content,
response
};
});
}
// Usage
(async () => {
const chatbot = new PDFChatbot("document.pdf");
await chatbot.initialize();
const response = await chatbot.chat("What is the main topic?", "user-123");
console.log(response.answer);
})();
```
***
## Summary
You've built a fully observable RAG pipeline with Netra. Your chatbot now has:
* **End-to-end tracing** across document ingestion, retrieval, and generation
* **Cost and performance tracking** at each pipeline stage
* **User and session tracking** for usage analytics
* **Debugging capabilities** to trace issues back to specific chunks and prompts
With this foundation, you can identify bottlenecks, optimize costs, and debug issues in your RAG system with confidence.
***
## See Also
Add quality metrics and test suites to measure retrieval and generation quality
Run automated simulation tests to stress-test your pipeline
# Simulating Customer Support Conversations
Source: https://docs.getnetra.ai/Cookbooks/simulation/simulating-customer-support
Simulate customer support conversations with Netra to test your AI agent. Validate handling of refunds, inquiries, and policy questions before launch.
A traced support agent tells you what happened — which tools were called, how long each step took, and what the LLM generated. Simulation tells you whether the agent actually resolved the customer's issue. Without multi-turn testing, you can't tell if the agent communicates the correct refund timeline, handles a frustrated customer gracefully, or gives up before achieving the goal.
**Prerequisite:** You need a customer support agent integrated with Netra. If you haven't set this up yet, follow the [Tracing LangChain Agents](/Cookbooks/observability/tracing-langchain-agents) cookbook first.
## What You'll Learn
Select session-level evaluators that measure goal achievement, fact accuracy, and conversation quality
Build scenarios with goals, user data, fact checkers, and different personas
Implement the BaseTask interface to connect your agent to the simulation engine
Run the same scenario with Neutral, Friendly, Frustrated, and Confused personas and compare results
***
## Why Simulate Customer Support?
Customer support agents engage in goal-oriented, multi-turn conversations where subtle failures compound:
| Failure Mode | What Goes Wrong | Why Single-Turn Evaluation Misses It |
| ------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| **Incomplete resolution** | Agent answers the question but never confirms the action was taken | Each individual response looks correct, but the goal is never achieved |
| **Fact miscommunication** | Agent states the wrong refund timeline or return policy | Only detectable when you define specific facts the agent must communicate |
| **Persona sensitivity** | Agent handles friendly users well but breaks down with frustrated customers | Single-turn tests don't model emotional progression across a conversation |
| **Premature closure** | Agent ends the conversation before the customer's issue is fully resolved | Only visible in multi-turn context with goal tracking |
Simulation addresses this by creating realistic conversations with a simulated user who has a goal, a persona, and context data — then scoring the entire session.
***
Now, let's walk through the process of simulating customer support conversations:
## Step 1: Select Simulation Evaluators
Go to **Evaluation → Evaluators**, switch to the **Library** tab, and filter by **Multi turn**. Add the following four evaluators:
| Evaluator | What It Measures |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Goal Fulfillment** | Did the conversation achieve the customer's objective (e.g., process the refund)? |
| **Factual Accuracy** | Did the agent communicate the correct refund timeline, return policy, and other facts? |
| **Conversation Completeness** | Were all of the customer's questions and intents addressed during the conversation? |
| **Guideline Adherence** | Did the agent follow its instructions throughout — staying professional, not making promises it shouldn't? |
See [Simulation Evaluators](/Simulation/Evaluators) for the full library and configuration options.
***
## Step 2: Create a Multi-Turn Dataset
Go to **Evaluation → Datasets** and click **Create Dataset**. Select **Multi-turn** as the type.
Set the dataset name to "Customer Support Scenarios" and add tags like `support`, `refunds`. Select **Multi-turn** as the type and **Add manually** as the data source.
Define the first scenario:
**Scenario Goal:**
```text theme={null}
Agent approves refund for the damaged product
```
**Behavior Instructions (optional):**
```text theme={null}
Start politely, but become slightly impatient if the agent
asks for information already provided.
```
**Max Turns:** 5
**User Persona:** Frustrated
**Provider and Model:** Choose the LLM that will generate simulated user responses (e.g., OpenAI / GPT-4.1).
**Simulated User Data** — context the simulated user can reference:
| Key | Value |
| -------------- | ------------------------------------- |
| order\_number | ORD-12345 |
| purchase\_date | 2024-01-15 |
| product\_name | Wireless Headphones |
| order\_total | \$79.99 |
| issue | Arrived damaged — left earcup cracked |
**Fact Checker** — facts the agent must communicate correctly:
| Fact | Expected Value |
| ------------------------ | ------------------------- |
| refund\_processing\_time | 5-7 business days |
| refund\_method | Original payment method |
| return\_label\_delivery | Within 24 hours via email |
Add the four evaluators from Step 1 — Goal Fulfillment, Factual Accuracy, Conversation Completeness, and Guideline Adherence. Configure variable mappings to connect evaluator inputs to scenario fields, agent responses, and conversation metadata.
Select a provider and model for each evaluator (e.g., OpenAI / GPT-4.1). Optionally rename evaluators to match your use case (e.g., "Refund Goal Fulfillment"). Review and click **Create Dataset**.
See [Simulation Datasets](/Simulation/Datasets) for the full dataset creation reference.
***
## Step 3: Add More Scenarios
Add two more scenarios to the same dataset, each with a different persona and goal:
**Scenario 2 — Order Status Inquiry (Neutral persona)**
| Field | Value |
| ------------- | --------------------------------------------------------------------------- |
| **Goal** | Agent provides the current status and expected delivery date of the order |
| **Max Turns** | 4 |
| **Persona** | Neutral |
| **User Data** | `order_number`: ORD-67890, `product_name`: Standing Desk |
| **Facts** | `estimated_delivery`: March 15, `carrier`: FedEx, `tracking_available`: Yes |
**Scenario 3 — Return Policy Question (Confused persona)**
| Field | Value |
| ------------- | --------------------------------------------------------------------------------------------------------- |
| **Goal** | Agent informs the user about the return policy of an item they bought three weeks ago |
| **Max Turns** | 6 |
| **Persona** | Confused |
| **User Data** | `order_number`: ORD-11111, `product_name`: Bluetooth Speaker, `purchase_date`: 2024-02-01 |
| **Facts** | `return_window`: 30 days from purchase, `return_condition`: Item must be unused and in original packaging |
Under **Evaluation → Datasets**, you should now see the "Customer Support Scenarios" dataset with three scenarios and four evaluators.
***
## Step 4: Implement the BaseTask Wrapper
Wrap your customer support agent in a `BaseTask` so the simulation engine can call it turn by turn. The `run()` method receives the simulated user's message and a `session_id` for conversation continuity.
```python Python theme={null}
from netra import Netra
from netra.simulation.task import BaseTask
from netra.simulation.models import TaskResult
from openai import OpenAI
import uuid
Netra.init(app_name="support-simulation")
client = OpenAI()
# Store conversation history per session
conversations: dict[str, list] = {}
class SupportAgentTask(BaseTask):
"""Wraps a customer support agent for simulation."""
def run(self, message: str, session_id: str | None = None) -> TaskResult:
session = session_id or str(uuid.uuid4())
if session not in conversations:
conversations[session] = [
{
"role": "system",
"content": (
"You are a customer support agent for an e-commerce store. "
"Help customers with refunds, order status, and return policies. "
"Be professional and empathetic. Always confirm the action taken "
"before ending the conversation."
),
}
]
conversations[session].append({"role": "user", "content": message})
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=conversations[session],
)
content = response.choices[0].message.content
conversations[session].append({"role": "assistant", "content": content})
return TaskResult(message=content, session_id=session)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk-js";
import { BaseTask, TaskResult } from "netra-sdk-js/simulation";
import OpenAI from "openai";
const netra = new Netra({ apiKey: "your-netra-api-key" });
const openai = new OpenAI();
// Store conversation history per session
const conversations: Record = {};
class SupportAgentTask extends BaseTask {
/** Wraps a customer support agent for simulation. */
async run(message: string, sessionId?: string | null): Promise {
const session = sessionId || crypto.randomUUID();
if (!conversations[session]) {
conversations[session] = [
{
role: "system",
content:
"You are a customer support agent for an e-commerce store. " +
"Help customers with refunds, order status, and return policies. " +
"Be professional and empathetic. Always confirm the action taken " +
"before ending the conversation.",
},
];
}
conversations[session].push({ role: "user", content: message });
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: conversations[session],
});
const content = response.choices[0].message.content || "";
conversations[session].push({ role: "assistant", content });
return { message: content, sessionId: session };
}
}
```
If you built a LangChain agent in the [Tracing LangChain Agents](/Cookbooks/observability/tracing-langchain-agents) cookbook, you can wrap it the same way — call your agent's `.invoke()` method inside `run()` and return the response as a `TaskResult`.
***
## Step 5: Trigger the Simulation
Copy the **Dataset ID** from the dataset page and run the simulation.
```python Python theme={null}
result = Netra.simulation.run_simulation(
name="Customer Support — All Personas",
dataset_id="your-dataset-id",
task=SupportAgentTask(),
context={"model": "gpt-4o-mini", "agent_version": "v1"},
max_concurrency=3,
)
print(f"Total scenarios: {result['total_items']}")
print(f"Completed: {len(result['completed'])}")
print(f"Failed: {len(result['failed'])}")
for failure in result["failed"]:
print(f" Failed {failure['run_item_id']}: {failure['error']}")
Netra.shutdown()
```
```typescript TypeScript theme={null}
const result = await netra.simulation.runSimulation({
name: "Customer Support — All Personas",
datasetId: "your-dataset-id",
task: new SupportAgentTask(),
context: { model: "gpt-4o-mini", agentVersion: "v1" },
maxConcurrency: 3,
});
if (result) {
console.log(`Total scenarios: ${result.totalItems}`);
console.log(`Completed: ${result.completed.length}`);
console.log(`Failed: ${result.failed.length}`);
for (const failure of result.failed) {
console.log(` Failed ${failure.runItemId}: ${failure.error}`);
}
}
```
For the full API reference, see the SDK documentation for [Python](/sdk-reference/simulation/python) and [TypeScript](/sdk-reference/simulation/typescript).
***
## Step 6: Analyze Results
Go to **Evaluation → Test Runs** and filter by **Multi turn** to find your simulation run.
### Summary Metrics
The top of the detail view shows aggregated data — total scenarios, pass/fail counts, total cost, and average latency. Use this for a quick health check before diving into individual scenarios.
### Conversation Transcripts
Click on any scenario to open the detail view. The **Conversation** tab shows the full turn-by-turn dialogue between the simulated user and your agent. Look for:
* **Where the conversation stalled** — did the agent ask for information the user already provided?
* **Fact accuracy** — did the agent state the correct refund timeline?
* **Resolution confirmation** — did the agent explicitly confirm the action before ending?
### Exit Reasons
Each scenario ends with one of four exit reasons:
| Exit Reason | What It Means |
| --------------------- | --------------------------------------------------- |
| **Goal Achieved** | The customer's objective was successfully completed |
| **Goal Failed** | The conversation ended without achieving the goal |
| **Abandoned** | The simulated user gave up or stopped engaging |
| **Max Turns Reached** | Hit the turn limit before goal completion |
### Evaluation Scores
The **Evaluation Results** tab shows scores for each evaluator. Compare scores across the three scenarios to spot persona-specific weaknesses:
| Scenario | Persona | Goal Fulfillment | Factual Accuracy | Completeness | Guideline Adherence |
| -------------- | ---------- | ---------------- | ---------------- | ------------ | ------------------- |
| Refund request | Frustrated | 0.8 | 1.0 | 0.75 | 0.6 |
| Order status | Neutral | 1.0 | 1.0 | 1.0 | 1.0 |
| Return policy | Confused | 0.6 | 0.75 | 0.5 | 0.8 |
In this example, the Confused persona scenario scores lowest on Completeness — the agent may not be explaining things clearly enough for users who need extra clarification.
### Debugging with Traces
Click **View Trace** on any conversation turn to inspect the full execution — LLM inputs, tool calls (if applicable), token usage, and latency. This connects simulation results directly to your [observability traces](/Observability/Traces/overview).
***
## Interpreting Scores and Improving Quality
When evaluator scores are low, use this table to identify the likely cause and fix:
| Low Score In | Likely Cause | How to Fix |
| ----------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| **Goal Fulfillment** | Agent answered questions but never completed the action (e.g., processed the refund) | Add explicit instructions in the system prompt to confirm resolution before ending |
| **Factual Accuracy** | Agent stated incorrect policy details or timelines | Include accurate policy data in the system prompt or connect to a knowledge base tool |
| **Conversation Completeness** | Agent addressed the primary question but missed follow-up intents | Improve the system prompt to check whether the customer has additional questions |
| **Guideline Adherence** | Agent deviated from tone or made unauthorized promises | Tighten the system prompt guidelines and add guardrails for what the agent should not promise |
After making changes, re-run the simulation against the same dataset and compare results across test runs.
***
## Continuous Simulation Strategy
For production support agents, run simulations regularly:
1. **On every prompt change** — Verify that updated instructions don't break existing conversation patterns
2. **After adding new tools** — Ensure the agent correctly integrates new capabilities into conversations
3. **After model upgrades** — Compare conversation quality across model versions
4. **Weekly regression runs** — Catch gradual degradation in goal achievement or fact accuracy
***
## See Also
Set up comprehensive tracing for your support agent before simulating
Deep dive into Netra's simulation framework: datasets, evaluators, and test runs
# Custom Dashboard
Source: https://docs.getnetra.ai/Dashboard/Custom-dashboard
Build custom analytics dashboards in Netra to monitor AI cost, latency, and quality. Create charts, apply filters, set time ranges, and share with your team.
Custom Dashboards enable you to create tailored analytics views that surface the metrics most important to your team. Whether you're tracking costs across tenants, monitoring latency for specific models, or analyzing error rates by environment, custom dashboards give you the flexibility to visualize your data exactly how you need it.
Monitor your AI application's performance with live-updating widgets and configurable refresh intervals.
Choose from 6 chart types to display your data in the most meaningful way for your use case.
Segment data by environment, user, session, or tenant to uncover patterns and anomalies.
Create widgets once and add them to multiple dashboards for consistent reporting across teams.
***
## Accessing Custom Dashboards
Click the profile icon in the top-right corner and select **Organization Analytics**
In the left navigation panel, click on **Custom**
Select either **Custom Dashboards** to view dashboards or **Widgets** to manage individual widgets
***
## Dashboard Management
The Custom Dashboards tab displays all your dashboards with key information:
| Column | Description |
| ----------------- | ---------------------- |
| **Name** | Dashboard identifier |
| **Description** | Purpose and contents |
| **Created** | Creation timestamp |
| **Last Modified** | Most recent update |
| **Actions** | Edit, clone, or delete |
### Creating a New Dashboard
Click **Create Dashboard** in the top-right corner of the Custom Dashboards tab
Enter a descriptive **name** and **description** for your dashboard
Click **Create Dashboard** to confirm, or **Cancel** to abort
Click **Add Widget** to open the widget selector
Check the boxes next to the widgets you want to include
Click **Add Selected Widget** to add them to your dashboard
Use the **Create New Widget** button in the widget selector to quickly create a new widget without leaving the dashboard editor.
### Dashboard Actions
| Action | Description |
| ---------- | --------------------------------------------------- |
| **Edit** | Modify dashboard name, description, and widgets |
| **Clone** | Create a copy of the dashboard with all its widgets |
| **Delete** | Permanently remove the dashboard |
***
## Widget Management
Widgets are the building blocks of your dashboards. Create them once and reuse them across multiple dashboards.
### Creating a New Widget
Click **Create Widget** to open the configuration page with two sections:
* **Left panel** - Configuration options
* **Right panel** - Live preview
| Field | Description |
| --------------- | --------------------------------------------- |
| **Name** | Unique identifier (no duplicates allowed) |
| **Description** | Brief explanation of what the widget displays |
**Widget Scope** - Choose your data source:
| Scope | Description |
| ---------- | --------------------------- |
| **Traces** | Aggregated trace-level data |
| **Spans** | Individual span-level data |
**Metric** - Select what to measure (options vary by scope)
| Metric | Aggregation Methods |
| ----------------------- | ----------------------------------- |
| **Error Rate** | Percentage, Total Count |
| **Latency** | Average, P50, P90, P95, P99, Median |
| **Request Count** | Total Count |
| **Total Cost** | Sum |
| **Total Tokens** | Sum |
| **Time to First Token** | Average, P50, P90, P95, P99 |
| **Custom** | Select a custom metric |
| Chart Type | Best For |
| -------------------- | ---------------------------- |
| **Line Time Series** | Trends over time |
| **Bar Time Series** | Periodic comparisons |
| **Horizontal Bar** | Ranking/comparing categories |
| **Vertical Bar** | Category comparisons |
| **Pie Chart** | Proportional distribution |
| **Big Number** | Single KPI display |
Segment your data by:
| Dimension | Use Case |
| --------------- | ------------------------------ |
| **Environment** | Compare prod vs staging vs dev |
| **User ID** | Per-user analysis |
| **Session ID** | Session-level breakdown |
| **Tenant ID** | Multi-tenant comparisons |
Narrow down your data with filters:
| Filter | Options |
| ------------- | ------------ |
| **Violation** | True / False |
| **PII** | True / False |
| **Error** | True / False |
| Filter | Operators |
| ---------- | ---------------- |
| **Models** | Any of / None of |
| Filter | Operators |
| -------------- | ----------------------------------------------------------------------- |
| **Session ID** | Equals / Not equals / Contains |
| **User ID** | Equals / Not equals / Contains |
| **Metadata** | Equals / Not equals / Contains / Not contains / Starts with / Ends with |
| Filter | Operators |
| -------------- | ----------------------------------------------------------- |
| **Latency** | Greater than / Less than / Greater or equal / Less or equal |
| **Total Cost** | Greater than / Less than / Greater or equal / Less or equal |
Review the live preview on the right, then click **Add Widget** to save
The **Widget Preview** section shows a real-time visualization with a date range filter, allowing you to verify your configuration before saving.
***
## Best Practices
Use clear names like "Daily Cost by Model" instead of "Cost Widget 1" for easier discovery.
Begin with 3-5 essential widgets, then expand based on team feedback.
Keep related widgets on the same time range for accurate comparisons.
Create separate dashboards for different concerns (cost, performance, quality).
***
## Common Use Cases
Track spending across your AI infrastructure:
* **Total Cost by Model** - Pie chart showing cost distribution across different models
* **Daily Cost Trend** - Line time series tracking cost over time
* **Cost by Tenant** - Horizontal bar comparing costs across customers
* **High-Cost Sessions** - Filter for sessions exceeding cost thresholds
**Recommended filters:** Total Cost > threshold, grouped by Model or Tenant ID
Monitor latency and throughput metrics:
* **P95 Latency Trend** - Line chart tracking latency percentiles over time
* **Request Count by Environment** - Bar chart comparing prod vs staging traffic
* **Time to First Token** - Track streaming response performance
* **Latency by Model** - Compare response times across different models
**Recommended filters:** Environment = production, grouped by Model Name
Track errors, violations, and PII detection:
* **Error Rate Trend** - Line chart showing error percentage over time
* **Violation Count** - Big number widget for policy violations
* **PII Detection by Service** - Bar chart showing PII occurrences
* **Error Rate by Environment** - Compare error rates across deployments
**Recommended filters:** Error = true, Violation = true, grouped by Environment
Multi-tenant usage analysis:
* **Usage by Tenant** - Horizontal bar showing request distribution
* **Cost per Tenant** - Track spending by customer
* **Token Usage Trend** - Monitor token consumption over time
* **Active Sessions by Tenant** - Compare engagement across customers
**Recommended filters:** Grouped by Tenant ID, sorted by Total Cost
***
## Next Steps
* [Dashboard Query API](/sdk-reference/dashboard-query/python) - Query dashboard data programmatically
* [Usage, Traces & Spans](/usage/usage-utilities) - Query usage data
* [Sessions](/Observability/Session) - Understand session management
* [Tenants](/Observability/Tenants) - Multi-tenant analytics
# Datasets
Source: https://docs.getnetra.ai/Evaluation/Datasets
Build and manage evaluation datasets in Netra. Create structured test cases with inputs, expected outputs, and metadata to benchmark your AI system.
Datasets are the foundation of your evaluation pipeline. They define what you're testing—the inputs, expected outputs, and metadata that [Evaluators](/Evaluation/Evaluators) use to score your AI system's performance. Build them from real production [traces](/Observability/Traces/overview) or create them manually for comprehensive coverage.
## Why Datasets Matter
Datasets transform ad-hoc testing into systematic quality assurance:
| Benefit | Description |
| -------------------------- | --------------------------------------------------------------------------- |
| **Reproducibility** | Run the same tests across model updates, prompt changes, and code releases |
| **Real-World Coverage** | Convert production traces into test cases that reflect actual user behavior |
| **Regression Detection** | Compare results over time to catch quality degradation early |
| **Objective Benchmarking** | Measure performance against defined criteria, not gut feeling |
## Dataset Dashboard
Navigate to **Evaluation → Datasets** from the left navigation panel to access your datasets.
| Column | Description |
| ---------------- | ---------------------------------------------- |
| **Dataset Name** | Unique identifier for the test suite |
| **Tags** | Metadata labels for filtering and organization |
| **Created At** | Timestamp for version tracking |
| **Actions** | Quick access to edit or delete datasets |
## Creating a Dataset
There are two ways to create a dataset:
Convert real production interactions into test cases (Recommended)
Build test suites from scratch in the dashboard
### Creating Dataset from Traces
The fastest way to build meaningful test cases is to capture real interactions from your production system. This ensures your evaluations reflect actual user behavior.
Navigate to **Observability → [Traces](/Observability/Traces/overview)** and locate an interaction you want to use as a test case.
Click the **Add to Dataset** button on the trace.
Choose to create a new dataset or add to an existing one.
In the creation form:
* Enter a dataset name (e.g., "Customer Support QA")
* Add optional tags for organization
* Review and edit the input prompt
* Provide the expected output
* Include any relevant metadata from the trace
Click **Next** and choose [evaluators](/Evaluation/Evaluators) to score this test case:
* Browse the [evaluator library](/Evaluation/Evaluators#library)
* Or select from your saved evaluators in **My Evaluators**
Configure how evaluator variables connect to your data:
| Source | Use Case |
| ------------------ | ------------------------------------------------------------- |
| **Dataset field** | Use values defined in your test case (input, expected output) |
| **Agent response** | Use the actual LLM output at evaluation time |
| **Execution data** | Use metadata from the trace (latency, tokens, model) |
Click **Create Dataset** to finalize.
### Creating Dataset from Dashboard
For comprehensive test coverage, create datasets manually with carefully crafted test cases.
Click the **Create Dataset** button in the top right corner of the Datasets page.
Fill in the dataset details:
| Field | Description |
| --------------- | --------------------------------------------------------------------- |
| **Name** | A descriptive identifier for your test suite |
| **Tags** | Labels for filtering (e.g., "production", "edge-cases", "v2-prompts") |
| **Type** | **Single Turn** for request/response pairs |
| **Data Source** | **Add manually** to create items one by one |
**Scenario** (multi-turn conversations), **Import from traces**, and **Import from CSV** are coming soon.
Click **Next** and select [evaluators](/Evaluation/Evaluators) from the library or your saved configurations.
Configure variable mappings to connect evaluator inputs to your dataset fields.
Click **Create Dataset** to complete the process.
## Running an Evaluation
Once your dataset is configured with evaluators, trigger a test run via the SDK.
Open your dataset and copy the **Dataset ID** displayed at the top of the page.
Use the Dataset ID to fetch the dataset and run a test suite. The task function receives the `input` field from each dataset item and should return the generated output as a string.
```python Python theme={null}
from netra import Netra
Netra.init(app_name="my-app")
def my_task(input_data):
# Your AI logic — takes the input from each dataset item
# and returns the generated output as a string
return response
dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")
result = Netra.evaluation.run_test_suite(
name="My Evaluation",
data=dataset,
task=my_task,
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({ appName: "my-app" });
async function myTask(inputData: string): Promise {
// Your AI logic — takes the input from each dataset item
// and returns the generated output as a string
return response;
}
const dataset = await Netra.evaluation.getDataset("your-dataset-id");
const result = await Netra.evaluation.runTestSuite(
"My Evaluation",
dataset,
myTask,
);
```
The output is compared against `expectedOutput` by the evaluators attached to the dataset. See the SDK reference for [Python](/sdk-reference/evaluation/python) and [TypeScript](/sdk-reference/evaluation/typescript) for the full API.
Monitor progress and results in [Test Runs](/Evaluation/TestRuns).
## Best Practices
### Organizing Datasets
* **Use descriptive names**: "Customer Support - Refund Requests" is better than "Dataset 1"
* **Tag consistently**: Create a tagging convention (e.g., by feature, model version, or test type)
* **Version your datasets**: Include version numbers in tags when testing prompt iterations
### Building Effective Test Cases
* **Cover edge cases**: Include unusual inputs, long prompts, and potential failure scenarios
* **Balance quantity and quality**: A smaller dataset of high-quality test cases beats a large dataset of weak ones
* **Include negative tests**: Add cases where the expected behavior is to refuse or ask for clarification
### Maintaining Datasets
* **Update regularly**: Add new test cases from production traces as you discover new patterns
* **Remove outdated cases**: Delete test cases that no longer reflect current requirements
* **Review failed cases**: Investigate failures to determine if the AI is wrong or the expected output needs updating
## Related
* [Evaluation Overview](/Evaluation/Evaluation-overview) - Understand the full evaluation framework
* [Evaluators](/Evaluation/Evaluators) - Configure scoring logic for your datasets
* [Test Runs](/Evaluation/TestRuns) - Analyze evaluation results
* [Traces](/Observability/Traces/overview) - Source data for creating datasets from production
# Evaluation Overview
Source: https://docs.getnetra.ai/Evaluation/Evaluation-overview
Measure and improve AI quality with Netra's evaluation framework. Run automated test suites, track scores over time, and catch regressions before production.
AI systems don't fail loudly. They drift, regress, and quietly degrade over time. Netra's Evaluation framework makes that invisible failure visible, giving you a structured, repeatable way to measure how your AI behaves—not just once, but continuously across releases, prompts, models, and environments.
New to evaluations? Get your first evaluation running in minutes.
## Why Evaluation Matters
Without systematic evaluation, you're flying blind. Netra helps you answer critical questions with confidence:
| Question | What Netra Measures |
| --------------------------------------- | --------------------------------------------------------- |
| Is my system producing correct answers? | Answer correctness, semantic similarity, factual accuracy |
| Did this update introduce a regression? | Side-by-side comparison across test runs |
| Are costs creeping up unnoticed? | Token usage, latency, and cost per evaluation |
| Are my agents executing correctly? | Tool call sequences, decision paths, guardrail compliance |
## Core Building Blocks
The Evaluation suite is built on three interconnected pillars:
### Evaluators
[Evaluators](/Evaluation/Evaluators) are the scoring logic that assesses your AI's performance. Netra offers two approaches:
**LLM as Judge**
Best for subjective quality, semantic correctness, and nuanced criteria. Use prebuilt templates or write custom prompts with providers like OpenAI, Anthropic, and Google.
**Code Evaluators**
Best for deterministic checks using JavaScript or Python—JSON schema validation, regex matching, mathematical calculations, and custom business logic.
Netra provides a [Library](/Evaluation/Evaluators#library) of preconfigured evaluators covering Quality, Performance, Agentic behavior, and Guardrails. Customize any evaluator and save it to **My Evaluators** for reuse across datasets.
**Playground Testing**
Before deploying an evaluator, test it in the integrated Playground:
* Input sample data and run evaluations in real-time
* Refine prompt templates and adjust pass/fail thresholds
* Verify edge case handling before adding to your pipeline
### Datasets
[Datasets](/Evaluation/Datasets) are collections of test cases that define what you want to evaluate.
| Feature | Description |
| ---------------------- | -------------------------------------------------------------------------- |
| **Create from Traces** | Convert real production interactions into test cases with one click |
| **Manual Creation** | Build test suites from scratch in the dashboard |
| **Variable Mapping** | Map evaluator inputs to dataset fields, agent responses, or trace metadata |
| **Metadata & Tags** | Organize datasets by feature, model, or release version |
### Test Runs
[Test Runs](/Evaluation/TestRuns) execute your datasets through the evaluation pipeline, providing point-in-time snapshots of system health.
| Feature | Description |
| ---------------------- | -------------------------------------------------------------------- |
| **Deep Diagnostics** | Compare expected output vs. actual output side-by-side |
| **Trace Integration** | Link directly to execution traces to debug the "why" behind failures |
| **Aggregated Metrics** | View total cost, average latency, and pass/fail rates across the run |
## Use Cases
### Regression Testing
Catch quality degradation before it reaches production:
1. Create a dataset from your golden test cases
2. Run evaluations after each model or prompt change
3. Compare results across test runs to identify regressions
### Continuous Quality Monitoring
Track quality metrics over time:
1. Convert production traces into datasets
2. Schedule regular evaluation runs
3. Set up [alerts](/Alert-rules/Alert-rules) when pass rates drop below thresholds
### Model Comparison
Evaluate different models or prompts objectively:
1. Create a standardized dataset
2. Run the same inputs through different model configurations
3. Compare scores across test runs to make data-driven decisions
## Getting Started
Define your scoring criteria by adding [evaluators](/Evaluation/Evaluators)—choose from the library or create custom ones.
[Create a dataset](/Evaluation/Datasets) from traces or manually in the dashboard, and attach your evaluators.
Execute your dataset and view results in [Test Runs](/Evaluation/TestRuns).
Use insights from test runs to refine your prompts, models, and evaluation criteria.
## Related
* [Quick Start: Evaluation](/quick-start/QuickStart_Evals) - Get started with evaluations in minutes
* [Datasets](/Evaluation/Datasets) - Create and manage test case collections
* [Evaluators](/Evaluation/Evaluators) - Configure scoring logic and criteria
* [Test Runs](/Evaluation/TestRuns) - Analyze evaluation results and track regressions
* [Traces](/Observability/Traces/overview) - Understand how evaluations connect to trace data
# Evaluators
Source: https://docs.getnetra.ai/Evaluation/Evaluators
Create LLM-as-Judge and code evaluators in Netra to assess AI output quality. Use library evaluators or define custom scoring criteria for your use case.
Evaluators are the scoring logic that determines whether your AI system meets quality standards. They transform subjective assessments into measurable metrics—from semantic correctness and tool execution accuracy to safety guardrails and custom business logic. Use them with [Datasets](/Evaluation/Datasets) to build automated quality pipelines.
## Why Evaluators Matter
Without systematic scoring, you can't measure improvement or catch regressions:
| Challenge | How Evaluators Help |
| ------------------ | ------------------------------------------------------------------------ |
| Subjective quality | LLM as Judge provides consistent, scalable assessment |
| Format validation | Code Evaluators enforce JSON schemas, regex patterns, and business rules |
| Safety compliance | Guardrail evaluators detect toxic, harmful, or off-topic content |
| Tool execution | Agentic evaluators verify correct function calling sequences |
## Evaluator Types
Netra offers two approaches to scoring, each suited for different use cases:
Best for subjective quality, semantic correctness, and nuanced criteria. Uses AI models to evaluate AI outputs.
Best for deterministic checks—JSON validation, regex matching, calculations, and custom business logic in JavaScript or Python.
## Auto Evaluation
When a trace contains an LLM call, Netra automatically scores the response using three built-in evaluators — no datasets, test runs, or configuration required.
| Evaluator | What It Measures |
| -------------------- | --------------------------------------------------------------------------------- |
| **Coherence** | Is the response well-structured, logically organized, and free of contradictions? |
| **Factual Accuracy** | Is the response factually correct based on the provided context? |
| **Toxicity** | Does the response contain harmful, offensive, or inappropriate content? |
Scores appear directly on the trace in your dashboard. Traces that do not contain an LLM call — such as pure retrieval or database operations — will not have auto evaluation scores.
The LLM call must go through a provider supported in the [Integrations](/Integrations/overview). If the call is proxied through an unsupported provider, auto evaluation scores will not appear.
## Evaluators Dashboard
Navigate to **Evaluation → Evaluators** from the left navigation panel. The interface has two tabs:
| Tab | Description |
| ----------------- | ---------------------------------------------------------------------------------- |
| **Library** | Netra's preconfigured evaluators organized by category |
| **My Evaluators** | Your saved custom configurations for reuse across [datasets](/Evaluation/Datasets) |
## Creating Custom Evaluators
Click the **Add Evaluator** button in the top right corner to create a new evaluator.
You can also customize any pre-built evaluator from the [Library](#library) by clicking the **Add** button next to it.
### LLM as Judge Configuration
Use LLM as Judge when you need to evaluate subjective criteria like answer quality, relevance, or helpfulness.
Provide a descriptive name (e.g., "Answer Correctness - Customer Support").
* Select a prebuilt template or write your own evaluation prompt
* Define variables using `{{variable_name}}` syntax
* Variables map to [dataset](/Evaluation/Datasets) fields, agent responses, or trace metadata
**Example prompt:**
```
Compare the following response to the expected answer.
Expected: {{expected_output}}
Actual: {{agent_response}}
Rate the correctness from 0-10.
```
| Output Type | Configuration |
| ------------- | ------------------------------------------------ |
| **Numerical** | Set threshold and operator (e.g., `> 7` to pass) |
| **Boolean** | Simple pass/fail evaluation |
Choose your preferred provider and model:
* OpenAI (GPT-4, GPT-3.5)
* Anthropic (Claude)
* Google (Gemini)
* Mistral
* Input sample data for each variable
* Run the evaluator in real-time
* Refine your prompt until results are consistent
### Code Evaluator Configuration
Use Code Evaluators for deterministic checks that don't require AI judgment.
Provide a descriptive name (e.g., "JSON Schema Validator").
Use the code editor to write JavaScript or Python. A `handler` function is required.
**JavaScript example:**
```javascript theme={null}
function handler(input) {
try {
const parsed = JSON.parse(input.agent_response);
return parsed.hasOwnProperty('name') && parsed.hasOwnProperty('email');
} catch {
return false;
}
}
```
**Python example:**
```python theme={null}
import json
def handler(input):
try:
parsed = json.loads(input["agent_response"])
return "name" in parsed and "email" in parsed
except:
return False
```
| Output Type | Configuration |
| ------------- | --------------------------------------------------- |
| **Numerical** | Set threshold and operator (e.g., `>= 0.8` to pass) |
| **Boolean** | Return `true`/`false` directly from your code |
* Input sample data
* Execute your code in real-time
* Debug and refine until it handles edge cases correctly
Once created, your evaluator appears in **My Evaluators** and becomes available when [creating datasets](/Evaluation/Datasets).
## Library
The Library contains pre-built evaluators ready to use or customize.
| Category | Description | Type |
| ------------------- | ------------------------------------------- | ------------ |
| **Quality** | Answer correctness, relevance, completeness | LLM as Judge |
| **Tool Use** | Validates proper function/tool calling | LLM as Judge |
| **Performance** | Response time, token efficiency | Code |
| **Semantic** | Meaning preservation, context understanding | LLM as Judge |
| **Agentic** | Decision-making, multi-step reasoning | LLM as Judge |
| **Guardrails** | Content safety, toxicity, compliance | LLM as Judge |
| **JSON Evaluator** | Schema validation, structure checks | Code |
| **Regex Evaluator** | Pattern matching, format validation | Code |
### Customizing Pre-built Evaluators
Start with a library evaluator and tailor it to your needs:
Find an evaluator that matches your use case.
Opens the configuration window with pre-filled settings.
* Modify the prompt template
* Adjust variables and mappings
* Change pass/fail thresholds
Validate your changes with sample data.
Click **Create** to save to **My Evaluators**.
## Using Evaluators in Datasets
Once created, evaluators become available when building [datasets](/Evaluation/Datasets):
1. Create or edit a dataset
2. In the evaluator selection step, choose from **Library** or **My Evaluators**
3. Map variables to connect evaluator inputs to your data
4. Run evaluations and view results in [Test Runs](/Evaluation/TestRuns)
## Best Practices
### Choosing the Right Evaluator Type
| Use Case | Recommended Type |
| ------------------------------------- | ------------------------- |
| "Is this answer correct?" | LLM as Judge |
| "Is the JSON valid?" | Code Evaluator |
| "Is the response helpful?" | LLM as Judge |
| "Does it match this regex?" | Code Evaluator |
| "Is content safe for users?" | LLM as Judge (Guardrails) |
| "Did the agent call the right tools?" | LLM as Judge (Agentic) |
### Writing Effective LLM Prompts
* **Be specific**: Define exactly what "correct" or "good" means
* **Provide examples**: Include sample inputs and expected scores
* **Set clear scales**: "Rate 1-10" is better than "rate quality"
* **Test edge cases**: Validate with ambiguous or tricky inputs
### Testing Before Deployment
Always use the Playground before adding evaluators to production datasets:
* Test with representative samples from your actual data
* Include edge cases and potential failure scenarios
* Verify pass/fail thresholds produce expected results
## Related
* [Evaluation Overview](/Evaluation/Evaluation-overview) - Understand the full evaluation framework
* [Datasets](/Evaluation/Datasets) - Create test cases that use your evaluators
* [Test Runs](/Evaluation/TestRuns) - View evaluation results and scores
* [Quick Start: Evaluation](/quick-start/QuickStart_Evals) - Get started with evaluations
# Test Runs
Source: https://docs.getnetra.ai/Evaluation/TestRuns
View and analyze Netra evaluation test run results. Track scores per evaluator, compare runs over time, and identify quality regressions in your AI.
Test Runs are the execution results of your [Datasets](/Evaluation/Datasets) through the evaluation pipeline. Each run provides a point-in-time snapshot of your AI system's performance—showing exactly which test cases passed, which failed, and why. Use them to track quality over time, catch regressions, and debug issues.
## Why Test Runs Matter
Test Runs transform raw evaluation data into actionable insights:
| Capability | Benefit |
| ----------------------- | ------------------------------------------------------------------------------------- |
| **Historical Tracking** | Compare results across releases to detect regressions |
| **Deep Diagnostics** | See expected vs. actual output for every test case |
| **Trace Integration** | Jump directly to execution [traces](/Observability/Traces/overview) to debug failures |
| **Aggregated Metrics** | Monitor cost, latency, and pass rates at a glance |
## Triggering a Test Run
Test runs are triggered via the SDK. You provide a [dataset](/Evaluation/Datasets), a task function that processes each input, and a name for the run.
```python Python theme={null}
from netra import Netra
Netra.init(app_name="my-app")
def my_task(input_data):
# Your AI logic — takes the input from each dataset item
# and returns the generated output as a string
return response
dataset = Netra.evaluation.get_dataset(dataset_id="your-dataset-id")
result = Netra.evaluation.run_test_suite(
name="My Evaluation",
data=dataset,
task=my_task,
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({ appName: "my-app" });
async function myTask(inputData: string): Promise {
// Your AI logic — takes the input from each dataset item
// and returns the generated output as a string
return response;
}
const dataset = await Netra.evaluation.getDataset("your-dataset-id");
const result = await Netra.evaluation.runTestSuite(
"My Evaluation",
dataset,
myTask,
);
```
The task function receives the `input` field from each dataset item. The output is compared against `expectedOutput` by the evaluators attached to the dataset. See the SDK reference for [Python](/sdk-reference/evaluation/python) and [TypeScript](/sdk-reference/evaluation/typescript) for the full API.
## Test Runs Dashboard
Navigate to **Evaluation → Test Runs** from the left navigation panel.
| Column | Description |
| -------------- | ------------------------------------------------------------ |
| **Agent Name** | The agent or application that was evaluated |
| **Dataset** | The [dataset](/Evaluation/Datasets) used for this evaluation |
| **Status** | Current state: Completed, In Progress, or Failed |
| **Started At** | Timestamp when the evaluation began |
### Filtering and Search
* **Date Range**: Filter runs by time period to compare performance over time
* **Search**: Find specific test runs by agent or dataset name
* **Sort**: Order by date, status, or dataset to find what you need quickly
## Viewing Test Run Details
Click on any test run to access detailed results and diagnostics.
### Summary Metrics
The top of the detail view shows aggregated performance data:
| Metric | Description |
| ------------------- | ------------------------------------------------- |
| **Total Cost** | Aggregate token/API cost for all test cases |
| **Total Duration** | End-to-end time for the evaluation run |
| **Average Latency** | Mean response time across test cases |
| **Pass/Fail Rate** | Percentage and count of passing vs. failing cases |
### Per-Test-Case Results
Each test case displays:
| Field | Description |
| -------------------- | -------------------------------------------------------------------------------- |
| **Input** | The prompt or query sent to your AI system |
| **Expected Output** | The reference answer defined in your [dataset](/Evaluation/Datasets) |
| **Task Output** | The actual response generated by your AI |
| **Run Status** | Shows running status (Running or Completed) |
| **Eval Status** | Pass or Fail based on [evaluator](/Evaluation/Evaluators) criteria |
| **Evaluator Scores** | Individual scores from each configured evaluator |
| **View Trace** | Link to the full execution [trace](/Observability/Traces/overview) for debugging |
Click **View Trace** on any failed test case to see the complete execution
timeline, including LLM calls, tool invocations, and latency breakdowns.
## Managing Datasets from Test Runs
Test Run details provide direct access to the underlying dataset configuration.
### Items Tab
View and manage test cases in the dataset:
| Field | Description |
| ---------------- | ------------------------------------------------------ |
| **Input/Output** | The test case prompt and expected response |
| **Metadata** | Additional context attached to the item |
| **Source** | Where the test case originated (manual, trace, import) |
| **Tags** | Labels for filtering and organization |
| **Created At** | When the test case was added |
### Evaluators Tab
View and modify [evaluators](/Evaluation/Evaluators) attached to the dataset:
* See all active evaluators and their configurations
* Edit variable mappings
* Adjust pass/fail thresholds
## Adding to Existing Datasets
Enhance your datasets directly from the Test Run view:
### Add New Test Cases
Opens the test case creation form.
* Enter the input prompt - Define the expected output - Add optional metadata
and tags
The new item is added to the [dataset](/Evaluation/Datasets) and included in future runs.
### Add New Evaluators
Opens the evaluator selection modal.
Choose from the [Library](/Evaluation/Evaluators#library), **My Evaluators**,
or create a new one.
Map evaluator variables to dataset fields, agent responses, or trace data.
The evaluator is added and will score all test cases in future runs.
## Analyzing Results
### Identifying Patterns
When reviewing test runs, look for:
* **Consistent failures**: Same test cases failing across multiple runs may indicate a systematic issue
* **New failures**: Test cases that previously passed but now fail signal a regression
* **Score trends**: Declining evaluator scores over time suggest gradual quality degradation
### Debugging Failures
For each failed test case:
1. Compare **Expected Output** vs **Task Output** to understand the discrepancy
2. Check **Evaluator Scores** to see which criteria failed
3. Click **View Trace** to inspect the full execution flow
4. Review LLM inputs, tool calls, and intermediate steps in the [trace view](/Observability/Traces/overview)
### Comparing Across Runs
To track regression or improvement:
1. Run evaluations after each significant change (model update, prompt revision, code release)
2. Compare pass rates and evaluator scores across runs
3. Investigate any test cases that changed from pass to fail
## Use Cases
### CI/CD Integration
Run evaluations as part of your deployment pipeline:
1. Trigger evaluation when code is pushed
2. Block deployment if pass rate drops below threshold
3. Review failed cases before merging
### Model Comparison
Evaluate different models objectively:
1. Run the same [dataset](/Evaluation/Datasets) with different model configurations
2. Compare test runs side-by-side
3. Make data-driven decisions about which model to deploy
### Prompt Iteration
Measure the impact of prompt changes:
1. Create a baseline test run with your current prompt
2. Update your prompt and run again
3. Compare results to validate improvement
## Related
* [Evaluation Overview](/Evaluation/Evaluation-overview) - Understand the full evaluation framework
* [Datasets](/Evaluation/Datasets) - Create and manage test case collections
* [Evaluators](/Evaluation/Evaluators) - Configure scoring logic and criteria
* [Traces](/Observability/Traces/overview) - Debug failed test cases with execution traces
* [Quick Start: Evaluation](/quick-start/QuickStart_Evals) - Get started with evaluations
# Agents
Source: https://docs.getnetra.ai/Evaluation/agents
Connect AI agents to Netra over HTTP and run dataset evaluations from the UI. Configure endpoints, test in the playground, and trigger suites without code.
Agents in Netra represent your external AI applications: chatbots, copilots, or any HTTP-accessible endpoint. Connect them once, then trigger automated test suite runs against your [Datasets](/Simulation/Datasets) directly from the UI. No SDK integration required—Netra calls your agent's API, collects responses, and runs [Evaluators](/Simulation/Evaluators) to score every interaction.
## Why Agents Matter
Testing AI agents manually doesn't scale. Connecting your agent to Netra unlocks automated, repeatable evaluation:
| Challenge | How Agents Help |
| -------------------------- | ------------------------------------------------------------------------- |
| Manual testing is slow | Run entire datasets against your agent with a single click |
| Inconsistent test coverage | Every dataset item is tested systematically with the same evaluators |
| No multi-turn testing | Netra simulates realistic users for multi-turn conversation testing |
| Disconnected tracing | Agent responses are automatically traced and linked to evaluation results |
| Environment-specific bugs | Test against staging, production, or any HTTP endpoint |
## How It Works
Netra provides two ways to interact with your connected agent:
Send individual messages to your agent in an interactive chat. Ideal for quick validation while configuring endpoints and response mappings.
Trigger a full dataset evaluation against your agent. Netra fans out every dataset item, collects responses, runs evaluators, and reports results.
## Agents Dashboard
Navigate to **Simulation → Agents** from the left navigation panel. The dashboard displays all configured agents for the current project.
Each agent card shows:
| Field | Description |
| ------------ | ----------------------------- |
| **Name** | Agent display name |
| **Endpoint** | The HTTP method and base URL |
| **Created** | When the agent was configured |
## Configuring an Agent
Click **Add Agent** in the top right corner to configure a new agent connection.
The configuration page is split into two panels—API settings on the left and an interactive Playground on the right.
Enter a descriptive name (e.g., "Customer Support Bot - Staging"). This name identifies the agent when selecting it for test suite runs.
Choose the HTTP method and enter your agent's endpoint URL.
| Method | Typical Use |
| --------------- | ---------------------------------------------------- |
| **POST** | Most common—send user messages in the request body |
| **GET** | Query-based agents with URL parameters |
| **PUT / PATCH** | Agents that update state as part of the conversation |
Add any custom headers your agent requires. A `Content-Type: application/json` header is included by default.
Headers that contain sensitive values (API keys, tokens) are encrypted at rest and masked when viewed by project members.
Choose your authentication method:
| Auth Type | Configuration |
| ---------------- | --------------------------------------------------------------- |
| **No Auth** | No additional credentials required |
| **Bearer Token** | Provide a bearer token, sent as `Authorization: Bearer ` |
| **API Key** | Provide a key name and value, sent as a custom header |
Write a JSON body template using variable placeholders. Netra replaces these with actual values at runtime.
**Default template:**
```json theme={null}
{
"message": "{{userMessage}}",
"session_id": "{{sessionId}}"
}
```
| Variable | Description |
| ----------------- | ----------------------------------------------------------- |
| `{{userMessage}}` | The user message from the dataset item or playground input |
| `{{sessionId}}` | A session identifier for multi-turn conversation continuity |
Structure the body to match your agent's expected request format. Any valid JSON is supported, nest variables at any depth.
Tell Netra where to find the agent's response in the JSON output using JSONPath expressions.
| Field | Required | Example | Description |
| ---------------- | -------- | -------------- | -------------------------------------------------- |
| **Text Path** | Yes | `$.response` | JSONPath to the agent's text reply |
| **Session Path** | No | `$.session_id` | JSONPath to a session ID for multi-turn continuity |
The text path must start with `$.` and point to the field containing the agent's response text.
Click **Create Agent** to save the configuration. The agent becomes available for test suite runs and appears on the Agents dashboard.
## Testing in the Playground
The Playground is an interactive chat panel on the right side of the agent configuration page. Use it to validate your endpoint configuration before running full evaluations.
### Sending Messages
1. Type a message in the input field at the bottom of the Playground panel
2. Press **Enter** or click the send button
3. Netra constructs the HTTP request using your configuration, calls your agent, and displays the response
### What Happens Behind the Scenes
When you send a playground message:
1. Netra resolves your body template: replacing `{{userMessage}}` with your input and `{{sessionId}}` with the current session
2. An async job is created and your agent's endpoint is called via HTTP
3. The response is parsed using your configured **Text Path** to extract the reply
4. If a **Session Path** is configured, the session ID is stored for subsequent messages in the same conversation
### Multi-Turn Conversations
The Playground maintains conversation context through sessions. Each message in the same Playground session includes the `sessionId` from the previous response, enabling multi-turn conversations with stateful agents.
Click **New Chat** to reset the session and start a fresh conversation.
Validation runs before each message is sent. The agent name, base URL, and response text path are required. If the body template contains invalid JSON, you'll see an error before the message is dispatched.
## Triggering a Test Suite Run
This is the primary workflow for the Agent Trigger feature: running an entire [Dataset](/Simulation/Datasets) against a configured agent from the UI.
### Prerequisites
Before triggering a run, ensure you have:
* At least one [Agent](#configuring-an-agent) configured in the current project
* A [Dataset](/Simulation/Datasets) with items (single-turn) or scenarios (multi-turn)
* [Evaluators](/Simulation/Evaluators) attached to the dataset for scoring
### Starting a Run
Navigate to **Simulation → Datasets** and open the dataset you want to evaluate.
In the dataset detail page toolbar, click the **Run** button.
The Run Test Suite modal appears with the following fields:
| Field | Required | Description |
| ------------ | -------- | ----------------------------------------------------- |
| **Agent** | Yes | Select a configured agent from the dropdown |
| **Run Name** | Yes | Auto-generated as `{Dataset Name} - {Date}`, editable |
The modal also displays read-only context: dataset name, turn type (single or multi), record count, and evaluator count.
Click **Run** to trigger the test suite. On success, you are redirected to the [Test Run](/Simulation/TestRuns) detail page to monitor progress.
You can also trigger test suite runs programmatically using the SDK—call `run_test_suite` for single-turn datasets or `run_simulation` for multi-turn datasets. See the [SDK Reference](/sdk-reference/simulation/python) for details.
###
## Monitoring Results
After triggering a run, you land on the [Test Run](/Simulation/TestRuns) detail page. The page updates as items complete.
### Status Summary
The summary card at the top shows aggregate metrics:
| Metric | Description |
| ------------------- | ------------------------------------------------ |
| **Passed / Failed** | Number of items that passed or failed evaluation |
| **Not Available** | Items where evaluation could not complete |
| **Total Cost** | Aggregate cost across all agent calls |
| **Average Latency** | Mean response time from your agent |
| **Duration** | Total wall-clock time for the run |
Use the refresh button to update the summary while the run is in progress.
### Run Statuses
| Status | Meaning |
| ------------- | -------------------------------------------------------------------- |
| **Running** | Items are still being processed |
| **Completed** | All items have finished and evaluations are scored |
| **Failed** | All items failed—check agent configuration and endpoint availability |
| **Cancelled** | The run was manually stopped |
### Per-Item Results
Each item in the results table shows:
| Column | Description |
| ------------------- | ----------------------------------------------- |
| **Input** | The original dataset item input |
| **Expected Output** | The ground truth from the dataset (if provided) |
| **Agent Output** | The response from your agent |
| **Run Status** | Whether the agent call succeeded or failed |
| **Eval Status** | Whether the item passed or failed evaluations |
| **Trace** | Link to the execution trace for debugging |
Click any row to view detailed evaluation results, variable comparisons, or the full multi-turn conversation transcript.
## Related
* [Simulation Overview](/Simulation/Simulation-overview) — Understand the full simulation framework
* [Datasets](/Simulation/Datasets) — Create test cases to run against your agents
* [Evaluators](/Simulation/Evaluators) — Configure scoring logic for agent responses
* [Test Runs](/Simulation/TestRuns) — View detailed evaluation results and conversation transcripts
* [Quick Start: Simulation](/quick-start/QuickStart_Simulation) — Get your first simulation running in minutes
# Netra FAQs
Source: https://docs.getnetra.ai/FAQs/FAQs
Frequently asked questions about Netra — covering setup, observability, evaluation, simulation, integrations, data retention, and security topics.
This FAQ helps developers, platform teams, and decision-makers understand how Netra works across setup, observability, evaluation, simulation, integrations, and security.
***
## Get Started
*Account setup, SDK installation, and your first traces.*
Netra is a unified platform for AI application [observability](/Observability/Traces/overview), [evaluation](/Evaluation/Evaluation-overview), and [simulation](/Simulation/Simulation-overview). It helps teams trace LLM calls, evaluate output quality, simulate multi-turn conversations, and monitor production AI systems — all from a single dashboard. Netra is built on [OpenTelemetry](https://opentelemetry.io/) standards and integrates with leading LLM providers, AI frameworks, and vector databases.
You need:
* A [Netra account](https://app.getnetra.ai) (US or EU data region)
* An API key generated from the Netra dashboard
* The `netra-sdk` package (Python >=3.10, \<3.14 or Node.js 18+)
See the [Quick Start Overview](/quick-start/Overview) for a step-by-step walkthrough.
1. Go to **Organisation → Projects** in the side navigation panel
2. Click **Create Project** and provide a project name
3. Confirm by clicking **Create Project**
Your API key is scoped to a project. See [Project Settings](/Account-settings/project) for more details.
Most teams see their first traces within minutes. Install the SDK, initialize `Netra.init()` with your API key, and run any LLM workflow — traces appear in the dashboard immediately. Follow the [Quick Start](/quick-start/Overview) to get started.
Once the SDK is initialized, [auto-instrumentation](/Observability/Traces/auto-instrumentation) captures traces from a [curated default set](/Observability/Traces/auto-instrumentation#default-instruments) of AI-relevant libraries — including LLM providers, agent frameworks, vector databases, and key web frameworks — no code changes required. Your first trace is typically visible in the dashboard as soon as you run your workflow. When you need more structure, add [decorators](/Observability/Traces/decorators) (`@workflow`, `@agent`, `@task`) or [manual spans](/Observability/Traces/manual-tracing) for fine-grained control. See the [Tracing Quick Start](/quick-start/QuickStart_Tracing) for a complete example.
***
## Observability
*Traces, spans, agents, sessions, costs, and latency monitoring.*
A [trace](/Observability/Traces/overview) captures the complete execution of a request through your AI application: LLM calls (prompts, completions, model parameters), tool invocations, retrieval steps, token usage, latency per step, costs, errors, and any custom metadata you attach.
A [trace](/Observability/Traces/overview) represents the full journey of a single request through your system. Spans are the individual operations within that trace — an LLM call, a tool execution, or a retrieval step. Spans are organized hierarchically (parent-child), so you can see exactly which step triggered which sub-step.
Netra offers three ways to instrument your application:
1. [**Auto-instrumentation**](/Observability/Traces/auto-instrumentation) — Zero code changes. Netra automatically traces a curated default set of AI-relevant libraries (OpenAI, LangChain, Pinecone, etc.). You can customize which libraries are included or pass `NetraInstruments.ALL` to instrument every supported library.
2. [**Decorators**](/Observability/Traces/decorators) — Add `@workflow`, `@agent`, or `@task` to your functions for structured tracing with minimal code
3. [**Manual tracing**](/Observability/Traces/manual-tracing) — Full control using context managers (Python) or `SpanWrapper` for custom spans
* **Agents**: Use the [`@agent` decorator](/Observability/Agents) to track agent execution, tool usage patterns, and decision flows
* **Sessions**: Call [`set_session_id`](/Observability/Session) to group related interactions into a single session
* **Users**: Call [`set_user_id`](/Observability/Users) to attribute traces to individual users
Netra automatically captures token usage and costs from [supported LLM providers](/Observability/Traces/auto-instrumentation). You can view cost breakdowns by model, tenant, user, or time range in [custom dashboards](/Dashboard/Custom-dashboard). For unsupported providers, attach cost data as custom span attributes.
***
## Evaluation
*Datasets, evaluators, test runs, and automated quality checks.*
An [evaluation](/Evaluation/Evaluation-overview) measures your AI system's output quality against defined criteria. Netra supports two evaluator types: **LLM-as-Judge** evaluators (for subjective quality, semantic correctness, and custom prompts) and **Code evaluators** (for deterministic checks like JSON schema validation, regex matching, and business logic).
[Datasets](/Evaluation/Datasets) are collections of test cases that define what to evaluate. Each test case includes an input prompt and an expected output. You can create datasets by:
* Converting production traces into test cases with one click
* Manually adding test cases in the dashboard
* Organizing by feature, model version, or release
Netra provides a [library of preconfigured evaluators](/Evaluation/Evaluators) across several categories:
* **Quality**: Coherence, Factual Accuracy, Answer Relevance
* **Guardrails**: Toxicity, Bias, PII Detection
* **Agentic**: Tool Call Accuracy, Goal Fulfillment
You can also create custom evaluators using LLM prompts or code logic. Test them in the [Evaluator Playground](/Evaluation/Evaluators) before deploying.
Yes. When Auto-Evaluation is enabled at the organization level, Netra runs Coherence, Factual Accuracy, and Toxicity evaluators on every incoming trace automatically. These background checks continue until your configured usage limit is reached. See [Evaluation Overview](/Evaluation/Evaluation-overview) for setup details.
Navigate to **Evaluation → Test Runs** to see all evaluation results. Each [test run](/Evaluation/TestRuns) shows pass/fail rates, individual evaluator scores, expected vs. actual output comparisons, and links to execution [traces](/Observability/Traces/overview) for debugging failures.
***
## Simulation
*Multi-turn conversation testing with configurable personas and goals.*
[Simulation](/Simulation/Simulation-overview) lets you test your AI agents with realistic, multi-turn conversations. You define a goal, a user persona, and facts the agent should communicate — Netra then simulates the conversation and scores it automatically using LLM-as-Judge evaluators.
Create a [simulation dataset](/Simulation/Datasets) in the dashboard:
1. Set a conversation goal (e.g., "Get a refund for a damaged product")
2. Choose a user persona (neutral, friendly, frustrated, confused, or custom)
3. Add user data and facts the agent should communicate
4. Select and configure [evaluators](/Simulation/Evaluators) to score the conversation
Netra provides five persona types for [simulation scenarios](/Simulation/Datasets):
* **Neutral** — Baseline performance testing
* **Friendly** — Tests professionalism without pushback
* **Frustrated** — Tests de-escalation and patience
* **Confused** — Tests clarity and explanation quality
* **Custom** — Define your own persona for industry-specific scenarios
Netra uses eight preconfigured [LLM-as-Judge evaluators](/Simulation/Evaluators) in two categories:
* **Quality** (6): Guideline Adherence, Conversation Completeness, Profile Utilization, Conversational Flow, Conversation Memory, Factual Accuracy
* **Agentic** (2): Goal Fulfillment, Information Elicitation
Each evaluator produces a score between 0 and 1. Scores at or above **0.6** pass.
Open the failed scenario in [Simulation → Test Runs](/Simulation/TestRuns) and use the three tabs:
1. **Conversation** — Read the full multi-turn transcript to find where things went wrong
2. **Evaluation Results** — See which evaluators failed and their scores
3. **Scenario Details** — Verify the goal, persona, user data, and facts were configured correctly
Click **View Trace** on any turn to inspect the full execution flow, including LLM calls and tool invocations.
***
## Integrations & SDKs
*Supported languages, frameworks, providers, and auto-instrumentation.*
Netra provides SDKs for both **Python** and **TypeScript/JavaScript**:
* **Python**: `netra-sdk` on PyPI (Python >=3.10, \<3.14)
* **TypeScript**: `netra-sdk` on npm (Node.js 18+)
Both SDKs support auto-instrumentation, decorators, manual tracing, and context tracking. See the [SDK Reference](/sdk-reference/sdk/overview) for the full API.
Netra integrates with 30+ services across four categories:
* **LLM Providers** (14): OpenAI, Anthropic, Google Gemini, AWS Bedrock, Mistral, Groq, Cohere, and more
* **AI Frameworks** (11): LangChain, LangGraph, LlamaIndex, CrewAI, Pydantic AI, DSPy, and more
* **Vector Databases** (8): Pinecone, ChromaDB, Qdrant, Weaviate, Milvus, and more
* **Speech Services** (3): Deepgram, ElevenLabs, Cartesia
See the full list at [Integrations Overview](/Integrations/overview).
[Auto-instrumentation](/Observability/Traces/auto-instrumentation) traces a curated default set of AI-relevant libraries once `Netra.init()` is called. The default set (`DEFAULT_INSTRUMENTS`) covers LLM providers, agent frameworks, vector databases, and key web frameworks. You can customize this behavior in three ways:
* **Select specific libraries**: Pass an explicit set via the `instruments` parameter (e.g., `instruments={NetraInstruments.OPENAI, NetraInstruments.LANGCHAIN}`)
* **Block specific libraries**: Use `block_instruments` to exclude libraries from the curated defaults
* **Instrument everything**: Pass `instruments={NetraInstruments.ALL}` to restore the legacy behavior of instrumenting all supported libraries
Additionally, the `root_instruments` parameter controls which libraries can produce root-level spans (top-level traces). See [Instrumentation Selection](/Observability/Traces/configuration/instrumentation-selection) for full details.
You can send version identifiers such as `prompt_version` or `model_version` as metadata attributes on your spans, enabling comparisons and analysis across versions in the dashboard.
Yes. Netra can collect observability signals from gateway-based and webhook-driven architectures. Use [manual tracing](/Observability/Traces/manual-tracing) to instrument custom ingestion points that auto-instrumentation does not cover.
***
## Dashboards & Alerts
*Custom analytics, real-time monitoring, and proactive notifications.*
Yes. [Custom dashboards](/Dashboard/Custom-dashboard) let you create real-time analytics views with six chart types (line, bar, pie, big number, and more). Available metrics include latency percentiles (P50, P90, P95, P99), total cost, token counts, error rates, and request counts. Filter by model, tenant, user, session, or environment.
Configure [alert rules](/Alert-rules/Alert-rules) to get notified about anomalies in your AI system:
* **Scope**: Trace-level or span-level monitoring
* **Metrics**: Cost, latency, error rate, token count
* **Filters**: Model, tenant ID, environment, service name
* **Delivery**: Email or Slack (via API token or webhook)
For example, you can alert when any trace costs more than \$0.50 or when P95 latency exceeds 3 seconds.
***
## Multi-Tenancy
*Per-customer isolation, usage attribution, and tenant-level monitoring for B2B SaaS.*
A [tenant](/Observability/Tenants) represents an end customer of your application. When you build a multi-tenant AI product, each of your customers is modeled as a tenant so you can isolate their traces, metrics, and costs. Use one workspace per organization, one project per product, and a `tenant_id` for each customer.
Call `set_tenant_id` early in your request lifecycle (typically in middleware) to associate all subsequent traces with that customer:
* **Python**: `Netra.set_tenant_id("acme-corp")`
* **TypeScript**: `Netra.setTenantId("acme-corp")`
See the [Tenants guide](/Observability/Tenants) for setup details and best practices.
Yes. Each tenant's traces, metrics, and costs are tagged with their unique tenant ID and can be viewed independently. The [Tenants dashboard](/Observability/Tenants) lets you filter by tenant to see only that customer's data, including sessions, trace counts, and aggregated costs.
Use one workspace per organization, one project per product or agent, and a `tenant_id` for each customer. This gives you per-customer isolation for traces, metrics, and costs while keeping everything within a single Netra account. See the [Tenants guide](/Observability/Tenants) for setup details.
Yes. When configuring [alert rules](/Alert-rules/Alert-rules), you can add a Tenant ID filter so the alert only triggers for a specific customer. This is useful for per-tenant SLA monitoring and budget enforcement.
***
## Security & Compliance
*Data handling, privacy controls, and regulatory compliance.*
Netra stores observability data including traces, span metadata, evaluation results, and simulation transcripts. You control what data is sent through SDK configuration options like `trace_content` (to enable or disable capturing prompt/completion content).
Yes. Retention policies can be configured based on your environment and compliance requirements.
Yes. Access is controlled using roles and permissions at the organization level. See [Organisation Settings](/Account-settings/Organisation) for configuration details.
Yes. Netra follows industry-standard security practices and complies with SOC 2, GDPR, and HIPAA standards.
Customer data is stored in secure, SOC 2-compliant data centers in the US, EU, and India, managed by top-tier cloud providers. You select your data region (US or EU) during account creation.
Yes. On-premises deployment is available under enterprise plans. Contact the [Netra team](https://getnetra.ai/contact-us/) for details.
***
## Pricing & Billing
*Plans, usage metering, and billing details.*
Pricing is based on usage — traces, evaluations, and simulations — depending on your selected plan. See the [pricing page](https://getnetra.ai/pricing/) for detailed information.
Yes. Netra provides monthly subscription plans and enterprise contracts with custom terms.
On-premises deployments are offered under enterprise pricing. Contact the [Netra team](https://getnetra.ai/contact-us/) for a quote.
Usage and billing details are available in the Netra dashboard under your organization settings.
***
## Help & Troubleshooting
*Common issues, debugging steps, and support.*
Check the following:
1. Your `NETRA_API_KEY` environment variable is set correctly
2. `Netra.init()` is called **before** any LLM or framework calls
3. The OTLP endpoint is reachable from your environment
4. Your workflow is actually executing (check for errors in your application logs)
5. The library you expect to be traced is included in `DEFAULT_INSTRUMENTS` or explicitly passed via the `instruments` parameter
See the [Tracing Quick Start](/quick-start/QuickStart_Tracing) for a working example.
Ensure tool functions are wrapped with the [`@task` decorator](/Observability/Traces/decorators) or instrumented manually using `SpanType.TOOL`. Auto-instrumentation only captures calls to supported libraries — custom tool functions need explicit instrumentation.
Costs and token counts are automatically captured for [supported LLM providers](/Observability/Traces/auto-instrumentation). If you are using an unsupported provider or a custom gateway, you need to attach token usage as custom span attributes via [manual tracing](/Observability/Traces/manual-tracing).
Verify that:
1. Your [dataset](/Evaluation/Datasets) has test cases with input and expected output
2. At least one [evaluator](/Evaluation/Evaluators) is attached to the dataset
3. The [test run](/Evaluation/TestRuns) has completed (check the status column)
Use the [official support channel](https://getnetra.ai/contact-us/) to reach the Netra team.
Include your project name, trace IDs, timestamps, SDK version (`netra-sdk`), and any relevant error logs or stack traces. This helps the team diagnose the issue quickly.
# Drifts
Source: https://docs.getnetra.ai/Insights/Drifts
Detect meaningful shifts in AI behavior across cost, latency, tool usage, and more — automatically, against a rolling baseline.
A drift is a meaningful shift in how your AI behaves now compared to a rolling baseline of its recent past. Insights computes drift across three time horizons and stores each observation for review.
## Time Horizons
Drift is computed on every observation run across three windows:
| Window | Comparison period |
| ----------- | -------------------------- |
| **Daily** | Last 24 hours vs. baseline |
| **Weekly** | Last 7 days vs. baseline |
| **Monthly** | Last 30 days vs. baseline |
## Drift Categories
Computed per intent, compared to baseline:
| Metric | What it measures | Scoring method |
| --------------------- | --------------------------------------------------------- | ------------------------- |
| **Cost** | Average cost per trace | Z-score |
| **Latency** | Average trace latency | Z-score |
| **Step count** | Average number of steps an agent takes to complete a task | Z-score |
| **Error rate** | Share of traces with errors | Percentage change |
| **Tool distribution** | Which tools the agent calls and how often | Jensen-Shannon divergence |
Tracks average output length per intent. Surfaces when responses become noticeably shorter or longer than the baseline - a common signal of prompt regression or model behavior change.
| Metric | Scoring method |
| ----------------- | ----------------- |
| **Output length** | Percentage change |
Emitted alongside new-intent discovery. When a cluster of user inputs appears that does not match any known intent, Insights flags it as an input drift signal - indicating that the nature of requests hitting your AI has shifted.
## Drift Severity
Each metric is scored automatically using the method appropriate for that signal type. You do not need to configure thresholds; Insights applies calibrated defaults and assigns a severity to each observation.
## Learn More
Understand how Insights classifies user requests by intent
Set up alerts on top of the metrics Insights tracks
# Insights Overview
Source: https://docs.getnetra.ai/Insights/Insights-overview
Monitor AI quality in production. Discover user intents and detect drift in cost, latency, and tool usage without writing rules.
Insights is Netra's automated quality and behavior monitoring layer for AI applications. It learns what normal looks like for your agent from real production traffic, then continuously surfaces changes - new user intents, shifts in tool usage, latency creep, rising error rates.
## How Insights Works
Insights operates on the trace data flowing through Netra. Once your application is instrumented, Insights runs in the background to:
1. **Discover and classify user intents** from real traffic
2. **Build rolling baselines** of normal behavior per intent
3. **Detect drift** across cost, latency, step count, error rate, tool usage, and output length
4. **Generate a brief** summarizing significant changes after each observation run
No rule configuration required. Insights ships with calibrated defaults for all metrics.
## Rolling Baselines
Insights builds a baseline automatically from the last 30 days of traffic, refreshed daily. The baseline holds per-intent statistics - average cost, latency, step count, error rate, output length, and tool distribution - against which every drift observation is measured.
No manual baseline tagging is required.
## Brief
After each observation run, Insights generates a short summary of the most significant drifts and new intents detected during that cycle. The brief gives you a digest you can scan in seconds rather than a wall of metrics.
## Setting Up Insights
Insights runs on your trace data. Instrument your application with the Netra SDK and Insights activates automatically once traces start flowing.
```bash pip theme={null}
pip install netra-sdk
```
```bash npm theme={null}
npm install netra-sdk
```
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="my-ai-agent",
environment="production",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
instruments={InstrumentSet.OPENAI, InstrumentSet.ANTHROPIC},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "my-ai-agent",
environment: "production",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.OPENAI, NetraInstruments.ANTHROPIC]),
});
```
Use Netra's decorators so Insights can compute per-intent metrics at the workflow, agent, and task level.
```python Python theme={null}
from netra.decorators import workflow, agent, task
@agent
class SupportAgent:
@workflow
def handle_query(self, query: str) -> str:
...
@task
def fetch_account(self, user_id: str):
...
```
```typescript TypeScript theme={null}
import { workflow, agent, task } from "netra-sdk";
class SupportAgent {
@workflow()
async handleQuery(query: string): Promise {
// ...
}
@task()
async fetchAccount(userId: string) {
// ...
}
}
```
Once traces start flowing, Insights handles the rest automatically:
* After \~500 traces, intent clusters are bootstrapped and classification begins for every new trace
* A rolling 30-day baseline is built and refreshed daily
* Drift observations run on daily, weekly, and monthly windows
* A brief is generated after each observation cycle
Insights requires no additional configuration beyond standard SDK instrumentation. All analysis runs against trace data already captured by Netra.
## Learn More
Discover and track what your users are actually asking
Detect behavior changes before users notice them
Understand how Netra captures the trace data that powers Insights
Instrument workflows, agents, and tasks for per-intent metrics
# Intents
Source: https://docs.getnetra.ai/Insights/Intents
Automatically discover and track the categories of requests hitting your AI - no manual labeling required.
Intents represent the categories of requests hitting your AI. Netra automatically discovers them from production traffic, names them in plain English, and tags every trace with the intent it belongs to.
## Intent Discovery and Classification
You do not define intents up front. After enough traffic accumulates, Insights clusters real user inputs and labels each cluster - for example, `refund_request`, `flight_booking`, or `complaint_escalation` - then tracks each as a first-class workflow.
| Capability | Description |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Automatic bootstrap** | After \~500 traces, Insights clusters your traffic and produces a labeled set of intents |
| **Continuous classification** | Every new trace is matched to the closest intent in real time |
| **New-intent discovery** | Unmatched traces are clustered daily; emerging intents surface as they appear |
| **Intent status** | Each intent is automatically tagged `growing`, `declining`, `stable`, or `new` based on 30-day volume trend |
| **Per-intent metrics** | Cost, latency, error rate, and tool usage broken down by intent |
## Why Intents Matter
Without intent visibility, you have no reliable way to know what users are actually doing with your AI. Aggregate metrics stay green while a specific workflow quietly regresses. Intent tracking lets product teams see emerging use cases as they appear and lets engineering teams prioritize fixes by which workflow is causing the most impact.
## Learn More
Detect behavior changes per intent before users notice them
How to set up Insights and what it monitors
# Aleph Alpha
Source: https://docs.getnetra.ai/Integrations/ai_providers/ALEPH_ALPHA
Trace Aleph Alpha model calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and performance metrics automatically.
## Installation
To use Aleph Alpha with Netra SDK, install both packages:
```bash Python theme={null}
pip install netra-sdk aleph-alpha-client
```
```bash Typescript theme={null}
npm install netra-sdk aleph-alpha-client
```
The Netra SDK automatically instruments Aleph Alpha API calls when initialized, providing comprehensive tracing and observability for your language model interactions.
## Usage
## Basic Setup
Initialize the Netra SDK with your API key to enable automatic instrumentation of Aleph Alpha calls.
```python Python theme={null}
from netra import Netra
# Initialize Netra with your API key
Netra.init(
app_name="my-aleph-alpha-app",
headers=f"x-api-key={os.environ['NETRA_API_KEY']}",
trace_content=True
)
# Your Aleph Alpha code will be automatically traced
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize Netra with your API key (must await)
await Netra.init({
appName: "my-aleph-alpha-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Your Aleph Alpha code will be automatically traced
```
## Working with Aleph Alpha Completions
Once initialized, all Aleph Alpha API calls are automatically traced with detailed telemetry including prompts, completions, token usage, and timing information.
```python Python theme={null}
from netra import Netra
from aleph_alpha_client import Client, CompletionRequest, Prompt
import os
# Initialize Netra
Netra.init(
app_name="my-aleph-alpha-app",
headers=f"x-api-key={os.environ['NETRA_API_KEY']}",
trace_content=True
)
# Use Aleph Alpha normally - automatically traced
client = Client(token=os.environ["ALEPH_ALPHA_API_KEY"])
request = CompletionRequest(
prompt=Prompt.from_text("What is AI?"),
maximum_tokens=100
)
response = client.complete(request, model="luminous-base")
print(response.completions[0].completion)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import { AlephAlphaClient } from "aleph-alpha-client";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Aleph Alpha normally - automatically traced
const client = new AlephAlphaClient({
apiKey: process.env.ALEPH_ALPHA_API_KEY
});
const response = await client.complete({
model: "luminous-base",
prompt: "Explain quantum computing in simple terms.",
maximum_tokens: 100
});
console.log(response.completions[0].completion);
```
## Using Decorators for Custom Workflows
Enhance tracing with Netra decorators to track custom workflows, agents, and tasks alongside Aleph Alpha calls.
```python Python theme={null}
from netra.decorators import workflow, task
from netra import Netra
from aleph_alpha_client import Client, CompletionRequest, Prompt
Netra.init(
headers=f"x-api-key={os.environ['NETRA_API_KEY']}"
)
client = Client(token=os.environ["ALEPH_ALPHA_API_KEY"])
@workflow()
def process_document(text: str):
summary = summarize_text(text)
keywords = extract_keywords(summary)
return {"summary": summary, "keywords": keywords}
@task()
def summarize_text(text: str):
request = CompletionRequest(
prompt=Prompt.from_text(f"Summarize: {text}"),
maximum_tokens=150
)
response = client.complete(request, model="luminous-extended")
return response.completions[0].completion
@task()
def extract_keywords(text: str):
request = CompletionRequest(
prompt=Prompt.from_text(f"Extract keywords: {text}"),
maximum_tokens=50
)
response = client.complete(request, model="luminous-base")
return response.completions[0].completion
```
```typescript TypeScript theme={null}
import { Netra, workflow, task } from "netra-sdk";
import { AlephAlphaClient } from "aleph-alpha-client";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const client = new AlephAlphaClient({
apiKey: process.env.ALEPH_ALPHA_API_KEY
});
@workflow()
async function processDocument(text: string) {
const summary = await summarizeText(text);
const keywords = await extractKeywords(summary);
return { summary, keywords };
}
@task()
async function summarizeText(text: string) {
const response = await client.complete({
model: "luminous-extended",
prompt: `Summarize the following text:\n\n${text}`,
maximum_tokens: 150
});
return response.completions[0].completion;
}
@task()
async function extractKeywords(text: string) {
const response = await client.complete({
model: "luminous-base",
prompt: `Extract key topics from:\n\n${text}`,
maximum_tokens: 50
});
return response.completions[0].completion;
}
```
## Session Tracking
Track user sessions and conversations when using Aleph Alpha for conversational applications.
```python Python theme={null}
from netra import Netra, ConversationType
# Start a user session
Netra.start_session(
session_id="user-session-123",
user_id="user-456",
metadata={"app": "chatbot"}
)
# Start a conversation
Netra.start_conversation(
conversation_type=ConversationType.CHAT,
metadata={"topic": "AI"}
)
# Add messages to the conversation
Netra.add_message(role="user", content="Tell me about Aleph Alpha")
# Your Aleph Alpha call here...
# End conversation when done
Netra.end_conversation()
```
```typescript TypeScript theme={null}
import { Netra, ConversationType } from "netra-sdk";
// Start a user session
Netra.startSession({
sessionId: "user-session-123",
userId: "user-456",
metadata: { channel: "web" }
});
// Start a conversation
Netra.startConversation({
conversationId: "conv-789",
sessionId: "user-session-123",
conversationType: ConversationType.CHAT
});
// Add messages to the conversation
Netra.addMessageToConversation({
conversationId: "conv-789",
role: "user",
content: "What is Aleph Alpha?"
});
// Your Aleph Alpha call here...
Netra.addMessageToConversation({
conversationId: "conv-789",
role: "assistant",
content: "Aleph Alpha is a European AI company..."
});
// End conversation when done
Netra.endConversation("conv-789");
Netra.endSession("user-session-123");
```
## Next Steps
Explore additional Netra SDK capabilities to enhance your Aleph Alpha integration:
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Get started with Netra SDK
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Learn about workflow, agent, and task decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Comprehensive session and conversation management
* [Advanced Configuration](https://docs.getnetra.ai/tracing/advanced-config/programatic-config) - Configure Netra SDK programmatically
* [Aleph Alpha Learning Center](https://learning.aleph-alpha.com/learn) - Official Aleph Alpha documentation
# Claude
Source: https://docs.getnetra.ai/Integrations/ai_providers/ANTHROPIC_CLAUDE
Trace Anthropic Claude API calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and response latency automatically.
The Netra SDK automatically instruments Anthropic Claude API calls, capturing prompts, completions, token usage, and performance metrics through OpenTelemetry tracing.
## Installation
Install the Netra SDK and the Anthropic client library:
```bash Python theme={null}
pip install netra-sdk anthropic
```
```bash TypeScript theme={null}
npm install netra-sdk @anthropic-ai/sdk
```
## Usage
Initialize the Netra SDK with your API key to automatically trace all Anthropic Claude API calls.
## Basic Setup
```python Python theme={null}
from netra import Netra
import anthropic
import os
# Initialize Netra with your API key
Netra.init(headers=f"x-api-key={os.getenv('NETRA_API_KEY')}")
# Use Anthropic client as normal
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{"role": "user", "content": "Hello, Claude"}
]
)
print(message.content)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Anthropic from "@anthropic-ai/sdk";
async function main() {
// Initialize Netra with your API key (must await)
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
// Use Anthropic client as normal
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
const message = await client.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
messages: [
{ role: "user", content: "Hello, Claude" }
]
});
console.log(message.content);
}
main();
```
## Streaming Responses
The SDK automatically traces streaming responses from Claude.
```python Python theme={null}
from netra import Netra
import anthropic
import os
Netra.init(headers=f"x-api-key={os.getenv('NETRA_API_KEY')}")
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
with client.messages.stream(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[{"role": "user", "content": "Tell me a story"}]
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Anthropic from "@anthropic-ai/sdk";
async function main() {
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
const stream = await client.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
messages: [{ role: "user", content: "Tell me a story" }],
stream: true
});
for await (const chunk of stream) {
if (chunk.type === 'content_block_delta') {
process.stdout.write(chunk.delta.text);
}
}
}
main();
```
## Session Tracking
Track user sessions and conversations across multiple Claude API calls.
```python Python theme={null}
from netra import Netra, ConversationType
import anthropic
import os
Netra.init(headers=f"x-api-key={os.getenv('NETRA_API_KEY')}")
# Set session and user context
Netra.set_session_id("session-123")
Netra.set_user_id("user-456")
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
# Track conversation
Netra.add_conversation(
content="What is machine learning?",
conversation_type=ConversationType.INPUT
)
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[{"role": "user", "content": "What is machine learning?"}]
)
Netra.add_conversation(
content=response.content[0].text,
conversation_type=ConversationType.OUTPUT
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Anthropic from "@anthropic-ai/sdk";
async function main() {
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
// Set session and user context
Netra.setSessionId("session-123");
Netra.setUserId("user-456");
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
// API calls are automatically traced with session context
const message = await client.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
messages: [
{ role: "user", content: "What is machine learning?" }
]
});
}
main();
```
## Custom Attributes
Add custom metadata to your Claude API traces for filtering and analysis.
```python Python theme={null}
from netra import Netra
import anthropic
import os
Netra.init(headers=f"x-api-key={os.getenv('NETRA_API_KEY')}")
# Set custom attributes
Netra.set_custom_attributes({
"environment": "production",
"feature": "chat-assistant",
"model_version": "v2"
})
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}]
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Anthropic from "@anthropic-ai/sdk";
async function main() {
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
// Set custom attributes
Netra.setCustomAttributes({
environment: "production",
feature: "chat-assistant",
modelVersion: "v2"
});
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY
});
const message = await client.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 1024,
messages: [{ role: "user", content: "Hello" }]
});
}
main();
```
## Next Steps
* [Auto-instrumentation Guide](https://docs.getnetra.ai/tracing/auto-instrumentation) - Learn about automatic tracing capabilities
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Analytics Dashboard](https://docs.getnetra.ai/Dashboard/Custom-dashboard) - Query and visualize your Claude API usage
* [Anthropic Claude Documentation](https://docs.anthropic.com/en/home) - Official Anthropic Claude API documentation
# AWS Bedrock
Source: https://docs.getnetra.ai/Integrations/ai_providers/AWS_BEDROCK
Trace AWS Bedrock foundation model calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and latency across all models.
## Installation
Install the Netra SDK along with the AWS SDK for Bedrock:
```bash Python theme={null}
pip install netra-sdk boto3
```
```bash TypeScript theme={null}
npm install netra-sdk @aws-sdk/client-bedrock-runtime
```
## Usage
Netra SDK automatically instruments AWS Bedrock calls when you enable the `botocore` instrumentation. This captures traces for all Bedrock API calls including model invocations, streaming responses, and embeddings.
## Basic Setup
Initialize Netra with Bedrock instrumentation enabled:
```python Python theme={null}
from netra import Netra
import boto3
import os
# Initialize Netra SDK
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
app_name="bedrock-app"
)
# Create Bedrock client - automatically instrumented
bedrock = boto3.client(
service_name='bedrock-runtime',
region_name='us-east-1'
)
# Make Bedrock calls - automatically traced
response = bedrock.invoke_model(
modelId='anthropic.claude-v2',
body='{"prompt": "Hello, world!"}'
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
// Initialize Netra SDK (must await)
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
appName: "bedrock-app"
});
// Create Bedrock client - automatically instrumented
const client = new BedrockRuntimeClient({
region: "us-east-1"
});
// Make Bedrock calls - automatically traced
const command = new InvokeModelCommand({
modelId: "anthropic.claude-v2",
body: JSON.stringify({ prompt: "Hello, world!" })
});
const response = await client.send(command);
```
## Model Invocation
The SDK automatically captures model invocations with full request and response details:
```python Python theme={null}
from netra import Netra
import boto3
import json
import os
Netra.init(headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}")
bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')
# Text generation
response = bedrock.invoke_model(
modelId='anthropic.claude-v2',
body=json.dumps({
"prompt": "Explain AI observability",
"max_tokens_to_sample": 300
})
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const client = new BedrockRuntimeClient({ region: "us-east-1" });
const command = new InvokeModelCommand({
modelId: "anthropic.claude-v2",
body: JSON.stringify({
prompt: "Explain AI observability",
max_tokens_to_sample: 300
})
});
await client.send(command);
```
## Streaming Responses
Bedrock streaming responses are automatically traced with full visibility:
```python Python theme={null}
from netra import Netra
import boto3
import json
import os
Netra.init(headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}")
bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')
response = bedrock.invoke_model_with_response_stream(
modelId='anthropic.claude-v2',
body=json.dumps({
"prompt": "Write a story",
"max_tokens_to_sample": 500
})
)
# Stream automatically traced
for event in response['body']:
chunk = json.loads(event['chunk']['bytes'])
print(chunk.get('completion', ''))
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import { BedrockRuntimeClient, InvokeModelWithResponseStreamCommand } from "@aws-sdk/client-bedrock-runtime";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const client = new BedrockRuntimeClient({ region: "us-east-1" });
const command = new InvokeModelWithResponseStreamCommand({
modelId: "anthropic.claude-v2",
body: JSON.stringify({
prompt: "Write a story",
max_tokens_to_sample: 500
})
});
const response = await client.send(command);
// Stream automatically traced
for await (const event of response.body) {
console.log(event.chunk?.bytes);
}
```
## Session Tracking
Track user sessions and conversations with Bedrock models:
```python Python theme={null}
from netra import Netra
import boto3
import json
Netra.init(api_key="your-api-key")
Netra.set_session_id("session-123")
Netra.set_user_id("user-456")
bedrock = boto3.client('bedrock-runtime', region_name='us-east-1')
# All calls automatically tagged with session context
response = bedrock.invoke_model(
modelId='anthropic.claude-v2',
body=json.dumps({"prompt": "Hello"})
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
Netra.setSessionId("session-123");
Netra.setUserId("user-456");
const client = new BedrockRuntimeClient({ region: "us-east-1" });
// All calls automatically tagged with session context
const command = new InvokeModelCommand({
modelId: "anthropic.claude-v2",
body: JSON.stringify({ prompt: "Hello" })
});
await client.send(command);
```
## Next Steps
* [Getting Started with AWS Bedrock](https://aws.amazon.com/bedrock/getting-started/)
* [Auto-Instrumentation Guide](https://docs.getnetra.ai/tracing/auto-instrumentation)
* [Session Tracking](https://docs.getnetra.ai/tracing/session)
* [Advanced Configuration](https://docs.getnetra.ai/tracing/advanced-config/programatic-config)
# Cohere
Source: https://docs.getnetra.ai/Integrations/ai_providers/COHERE
Trace Cohere Command model calls with Netra auto-instrumentation. Monitor prompts, completions, embeddings, token usage, and latency automatically.
## Installation
Install the Netra SDK along with the Cohere client library:
```bash Python theme={null}
pip install netra-sdk cohere
```
```bash TypeScript theme={null}
npm install netra-sdk cohere
```
## Usage
Netra automatically instruments Cohere API calls when you initialize the SDK. The instrumentation captures chat completions, streaming responses, and rerank operations.
## Basic Setup
Initialize Netra at the start of your application to enable automatic Cohere instrumentation:
```python Python theme={null}
from netra import Netra
import cohere
# Initialize Netra SDK
Netra.init(
app_name="my-cohere-app",
headers=f"x-api-key={YOUR_NETRA_API_KEY}",
trace_content=True # Capture prompts and completions
)
# Use Cohere normally - instrumentation is automatic
co = cohere.ClientV2(api_key="your-cohere-api-key")
response = co.chat(
model="command-r-plus",
messages=[{"role": "user", "content": "What is machine learning?"}]
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Cohere from "cohere-ai";
// Initialize Netra SDK
await Netra.init({
appName: "my-cohere-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true // Capture prompts and completions
});
// Use Cohere normally - instrumentation is automatic
const cohere = new Cohere({
apiKey: process.env.COHERE_API_KEY
});
const response = await cohere.chat({
model: "command-r-plus",
message: "What is machine learning?"
});
```
## Streaming Responses
Netra automatically handles streaming responses from Cohere:
```python Python theme={null}
from netra import Netra
import cohere
Netra.init(
app_name="my-cohere-app",
headers=f"x-api-key={YOUR_NETRA_API_KEY}"
)
co = cohere.ClientV2(api_key="your-cohere-api-key")
# Streaming is automatically traced
stream = co.chat_stream(
model="command-r-plus",
messages=[{"role": "user", "content": "Tell me a story"}]
)
for chunk in stream:
if chunk.type == "content-delta":
print(chunk.delta.message.content.text, end="")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Cohere from "cohere-ai";
await Netra.init({
appName: "my-cohere-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const cohere = new Cohere({
apiKey: process.env.COHERE_API_KEY
});
// Streaming is automatically traced
const stream = await cohere.chatStream({
model: "command-r-plus",
message: "Tell me a story"
});
for await (const chunk of stream) {
if (chunk.eventType === "text-generation") {
process.stdout.write(chunk.text);
}
}
```
## Rerank Operations
Rerank API calls are also automatically instrumented:
```python Python theme={null}
from netra import Netra
import cohere
Netra.init(
app_name="my-cohere-app",
headers=f"x-api-key={YOUR_NETRA_API_KEY}"
)
co = cohere.ClientV2(api_key="your-cohere-api-key")
documents = [
"Python is a programming language",
"The sky is blue",
"Machine learning uses algorithms"
]
results = co.rerank(
model="rerank-english-v2.0",
query="programming",
documents=documents
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Cohere from "cohere-ai";
await Netra.init({
appName: "my-cohere-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
const cohere = new Cohere({
apiKey: process.env.COHERE_API_KEY
});
const documents = [
"Python is a programming language",
"The sky is blue",
"Machine learning uses algorithms"
];
const results = await cohere.rerank({
model: "rerank-english-v2.0",
query: "programming",
documents: documents
});
```
## Session Tracking
Associate Cohere operations with user sessions for better analytics:
```python Python theme={null}
from netra import Netra
import cohere
Netra.init(
app_name="my-cohere-app",
headers=f"x-api-key={YOUR_NETRA_API_KEY}"
)
# Set session context
Netra.set_session_id("session-123")
Netra.set_user_id("user-456")
co = cohere.ClientV2(api_key="your-cohere-api-key")
# All subsequent calls are tracked under this session
response = co.chat(
model="command-r-plus",
messages=[{"role": "user", "content": "Hello"}]
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import Cohere from "cohere-ai";
await Netra.init({
appName: "my-cohere-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
// Set session context
Netra.setSessionId("session-123");
Netra.setUserId("user-456");
const cohere = new Cohere({
apiKey: process.env.COHERE_API_KEY
});
// All subsequent calls are tracked under this session
const response = await cohere.chat({
model: "command-r-plus",
message: "Hello"
});
```
## Selective Instrumentation
Enable only Cohere instrumentation if needed:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="my-cohere-app",
headers=f"x-api-key={YOUR_NETRA_API_KEY}",
instruments={InstrumentSet.COHERE}
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "my-cohere-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.COHERE])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Get started with Netra SDK
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Learn about automatic instrumentation
* [Session Management](https://docs.getnetra.ai/tracing/session) - Track user sessions and context
* [The Cohere Platform](https://docs.cohere.com/docs/the-cohere-platform) - Official Cohere documentation
# Gemini
Source: https://docs.getnetra.ai/Integrations/ai_providers/GEMINI
Trace Google Gemini model calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and latency across Gemini model variants.
## Installation
Install both the Netra SDK and Google Generative AI SDK:
```bash Python theme={null}
pip install netra-sdk google-generativeai
```
```bash Typescript theme={null}
npm install netra-sdk @google/generative-ai
```
## Usage
Initialize the Netra SDK with Gemini instrumentation enabled. The SDK automatically traces all Gemini API calls once initialized.
```python Python theme={null}
from netra import Netra
import google.generativeai as genai
import os
# Initialize Netra with Gemini instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Gemini client as usual - all calls are automatically traced
genai.configure(api_key=os.environ.get("GOOGLE_API_KEY"))
model = genai.GenerativeModel('gemini-pro')
response = model.generate_content("What is observability?")
print(response.text)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { GoogleGenerativeAI } from "@google/generative-ai";
async function main() {
// Initialize Netra with Gemini instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Gemini client as usual - all calls are automatically traced
const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY);
const model = genAI.getGenerativeModel({ model: "gemini-pro" });
const result = await model.generateContent("What is observability?");
const response = await result.response;
console.log(response.text());
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
model = genai.GenerativeModel('gemini-pro')
response = model.generate_content("Tell me a story", stream=True)
for chunk in response:
print(chunk.text, end="")
```
```typescript Typescript theme={null}
const model = genAI.getGenerativeModel({ model: "gemini-pro" });
const result = await model.generateContentStream("Tell me a story");
for await (const chunk of result.stream) {
const chunkText = chunk.text();
process.stdout.write(chunkText);
}
```
### Chat Sessions
Chat operations are also automatically instrumented:
```python Python theme={null}
model = genai.GenerativeModel('gemini-pro')
chat = model.start_chat(history=[
{"role": "user", "parts": ["Hello"]},
{"role": "model", "parts": ["Great to meet you. What would you like to know?"]},
])
response = chat.send_message("I have 2 dogs in my house.")
print(response.text)
```
```typescript Typescript theme={null}
const model = genAI.getGenerativeModel({ model: "gemini-pro" });
const chat = model.startChat({
history: [
{
role: "user",
parts: [{ text: "Hello" }],
},
{
role: "model",
parts: [{ text: "Great to meet you. What would you like to know?" }],
},
],
});
const result = await chat.sendMessage("I have 2 dogs in my house.");
console.log(result.response.text());
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Gemini instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.GOOGLE_GENERATIVEAI}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Gemini instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.GOOGLE_GENERATIVEAI])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Gemini Documentation](https://ai.google.dev/gemini-api/docs/quickstart) - Official Google Gemini API quickstart guide
# Groq
Source: https://docs.getnetra.ai/Integrations/ai_providers/GROQ
Trace Groq LPU inference with Netra auto-instrumentation. Monitor ultra-fast model calls, token usage, and latency in real time with zero code setup.
## Installation
Install both the Netra SDK and Groq SDK:
```bash Python theme={null}
pip install netra-sdk groq
```
```bash Typescript theme={null}
npm install netra-sdk groq-sdk
```
## Usage
Initialize the Netra SDK with Groq instrumentation enabled. The SDK automatically traces all Groq API calls once initialized.
```python Python theme={null}
from netra import Netra
from groq import Groq
import os
# Initialize Netra with Groq instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Groq client as usual - all calls are automatically traced
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
completion = client.chat.completions.create(
model="llama-3.1-70b-versatile",
messages=[
{"role": "user", "content": "What is observability?"}
]
)
print(completion.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import Groq from "groq-sdk";
async function main() {
// Initialize Netra with Groq instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Groq client as usual - all calls are automatically traced
const groq = new Groq({
apiKey: process.env.GROQ_API_KEY
});
const completion = await groq.chat.completions.create({
model: "llama-3.1-70b-versatile",
messages: [
{ role: "user", content: "What is observability?" }
]
});
console.log(completion.choices[0].message.content);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
stream = client.chat.completions.create(
model="llama-3.1-70b-versatile",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
```typescript Typescript theme={null}
const stream = await groq.chat.completions.create({
model: "llama-3.1-70b-versatile",
messages: [{ role: "user", content: "Tell me a story" }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```
### Multiple Model Support
Groq supports various open-source models, all automatically instrumented:
```python Python theme={null}
# Using Mixtral
mixtral_response = client.chat.completions.create(
model="mixtral-8x7b-32768",
messages=[{"role": "user", "content": "Explain AI"}]
)
# Using Gemma
gemma_response = client.chat.completions.create(
model="gemma-7b-it",
messages=[{"role": "user", "content": "Hello"}]
)
```
```typescript Typescript theme={null}
// Using Mixtral
const mixtralResponse = await groq.chat.completions.create({
model: "mixtral-8x7b-32768",
messages: [{ role: "user", content: "Explain AI" }]
});
// Using Gemma
const gemmaResponse = await groq.chat.completions.create({
model: "gemma-7b-it",
messages: [{ role: "user", content: "Hello" }]
});
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Groq instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.GROQ}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Groq instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.GROQ])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Groq Documentation](https://console.groq.com/docs/quickstart) - Official Groq API quickstart guide
# Hugging Face Transformers
Source: https://docs.getnetra.ai/Integrations/ai_providers/HUGGINGFACE_TRANSFORMERS
Trace Hugging Face Transformers model inference with Netra. Monitor local and hosted model calls, token usage, and latency with auto-instrumentation.
## Installation
Install both the Netra SDK and Hugging Face Transformers:
```bash Python theme={null}
pip install netra-sdk transformers torch
```
```bash Typescript theme={null}
npm install netra-sdk @huggingface/inference
```
## Usage
Initialize the Netra SDK with Hugging Face instrumentation enabled. The SDK automatically traces all Transformers operations once initialized.
```python Python theme={null}
from netra import Netra
from transformers import pipeline
import os
# Initialize Netra with Hugging Face instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Transformers pipeline - all calls are automatically traced
generator = pipeline("text-generation", model="gpt2")
response = generator("What is observability?", max_length=50)
print(response[0]['generated_text'])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { HfInference } from "@huggingface/inference";
async function main() {
// Initialize Netra with Hugging Face instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Hugging Face Inference API - all calls are automatically traced
const hf = new HfInference(process.env.HF_TOKEN);
const response = await hf.textGeneration({
model: "meta-llama/Llama-2-7b-chat-hf",
inputs: "What is observability?"
});
console.log(response.generated_text);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses from Hugging Face models:
```python Python theme={null}
from transformers import TextStreamer
generator = pipeline(
"text-generation",
model="gpt2",
streamer=TextStreamer(skip_prompt=True)
)
# Streaming output is automatically captured
generator("Tell me a story", max_length=100)
```
```typescript Typescript theme={null}
const stream = hf.textGenerationStream({
model: "meta-llama/Llama-2-7b-chat-hf",
inputs: "Tell me a story"
});
for await (const chunk of stream) {
process.stdout.write(chunk.token.text);
}
```
### Different Task Types
Various Transformers tasks are automatically instrumented:
```python Python theme={null}
# Text Classification
classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")
# Summarization
summarizer = pipeline("summarization")
summary = summarizer("Long text to summarize...", max_length=50)
# Question Answering
qa = pipeline("question-answering")
answer = qa({
"question": "What is AI?",
"context": "Artificial Intelligence is..."
})
```
```typescript Typescript theme={null}
// Text Classification
const classifier = await hf.textClassification({
model: "distilbert-base-uncased-finetuned-sst-2-english",
inputs: "I love this product!"
});
// Summarization
const summary = await hf.summarization({
model: "facebook/bart-large-cnn",
inputs: "Long text to summarize..."
});
// Question Answering
const answer = await hf.questionAnswering({
model: "deepset/roberta-base-squad2",
inputs: {
question: "What is AI?",
context: "Artificial Intelligence is..."
}
});
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Hugging Face instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.TRANSFORMERS}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Hugging Face instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.HUGGINGFACE])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Transformers Documentation](https://huggingface.co/docs/transformers/quicktour) - Official Hugging Face Transformers quick tour
# Mistral AI
Source: https://docs.getnetra.ai/Integrations/ai_providers/MISTRAL
Trace Mistral AI model calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and performance metrics automatically.
## Installation
Install both the Netra SDK and Mistral SDK:
```bash Python theme={null}
pip install netra-sdk mistralai
```
```bash Typescript theme={null}
npm install netra-sdk @mistralai/mistralai
```
## Usage
Initialize the Netra SDK with Mistral instrumentation enabled. The SDK automatically traces all Mistral API calls once initialized.
```python Python theme={null}
from netra import Netra
from mistralai.client import MistralClient
import os
# Initialize Netra with Mistral instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Mistral client as usual - all calls are automatically traced
client = MistralClient(api_key=os.environ.get("MISTRAL_API_KEY"))
response = client.chat(
model="mistral-large-latest",
messages=[
{"role": "user", "content": "What is observability?"}
]
)
print(response.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import MistralClient from "@mistralai/mistralai";
async function main() {
// Initialize Netra with Mistral instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Mistral client as usual - all calls are automatically traced
const mistral = new MistralClient(process.env.MISTRAL_API_KEY);
const response = await mistral.chat({
model: "mistral-large-latest",
messages: [
{ role: "user", content: "What is observability?" }
]
});
console.log(response.choices[0].message.content);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
stream = client.chat_stream(
model="mistral-large-latest",
messages=[{"role": "user", "content": "Tell me a story"}]
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
```typescript Typescript theme={null}
const stream = await mistral.chatStream({
model: "mistral-large-latest",
messages: [{ role: "user", content: "Tell me a story" }]
});
for await (const chunk of stream) {
if (chunk.choices[0]?.delta?.content) {
process.stdout.write(chunk.choices[0].delta.content);
}
}
```
### Embeddings
Embedding operations are also automatically instrumented:
```python Python theme={null}
embeddings = client.embeddings(
model="mistral-embed",
input=["The quick brown fox jumps over the lazy dog"]
)
print(embeddings.data[0].embedding)
```
```typescript Typescript theme={null}
const embeddings = await mistral.embeddings({
model: "mistral-embed",
input: ["The quick brown fox jumps over the lazy dog"]
});
console.log(embeddings.data[0].embedding);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Mistral instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.MISTRALAI}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Mistral instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.MISTRALAI])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Mistral Documentation](https://docs.mistral.ai/getting-started/quickstart/) - Official Mistral AI quickstart guide
# Ollama
Source: https://docs.getnetra.ai/Integrations/ai_providers/OLLAMA
Trace local Ollama LLM inference with Netra auto-instrumentation. Monitor prompts, completions, and latency for locally hosted open-source models.
## Installation
Install both the Netra SDK and Ollama SDK:
```bash Python theme={null}
pip install netra-sdk ollama
```
```bash Typescript theme={null}
npm install netra-sdk ollama
```
## Usage
Initialize the Netra SDK with Ollama instrumentation enabled. The SDK automatically traces all Ollama API calls once initialized.
```python Python theme={null}
from netra import Netra
from ollama import Client
import os
# Initialize Netra with Ollama instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Ollama client as usual - all calls are automatically traced
client = Client()
response = client.chat(
model="llama2",
messages=[
{"role": "user", "content": "What is observability?"}
]
)
print(response['message']['content'])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { Ollama } from "ollama";
async function main() {
// Initialize Netra with Ollama instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Ollama client as usual - all calls are automatically traced
const ollama = new Ollama();
const response = await ollama.chat({
model: "llama2",
messages: [
{ role: "user", content: "What is observability?" }
]
});
console.log(response.message.content);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
stream = client.chat(
model="llama2",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
print(chunk['message']['content'], end="")
```
```typescript Typescript theme={null}
const stream = await ollama.chat({
model: "llama2",
messages: [{ role: "user", content: "Tell me a story" }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.message.content);
}
```
### Embeddings
Embedding operations are also automatically instrumented:
```python Python theme={null}
embeddings = client.embeddings(
model="llama2",
prompt="The quick brown fox jumps over the lazy dog"
)
print(embeddings['embedding'])
```
```typescript Typescript theme={null}
const embeddings = await ollama.embeddings({
model: "llama2",
prompt: "The quick brown fox jumps over the lazy dog"
});
console.log(embeddings.embedding);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Ollama instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.OLLAMA}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Ollama instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.OLLAMA])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Ollama Documentation](https://github.com/ollama/ollama/blob/main/docs/api.md) - Official Ollama API documentation
# OpenAI
Source: https://docs.getnetra.ai/Integrations/ai_providers/OPENAI
Trace and monitor OpenAI GPT model calls with Netra auto-instrumentation. Capture prompts, completions, token usage, and latency with zero code changes.
## Installation
Install both the Netra SDK and OpenAI SDK:
```bash Python theme={null}
pip install netra-sdk openai
```
```bash Typescript theme={null}
npm install netra-sdk openai
```
## Usage
Initialize the Netra SDK with OpenAI instrumentation enabled. The SDK automatically traces all OpenAI API calls once initialized.
```python Python theme={null}
from netra import Netra
from openai import OpenAI
# Initialize Netra with OpenAI instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get("NETRA_API_KEY")}",
trace_content=True
)
# Use OpenAI client as usual - all calls are automatically traced
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
completion = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "user", "content": "What is observability?"}
]
)
print(completion.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import OpenAI from "openai";
async function main() {
// Initialize Netra with OpenAI instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use OpenAI client as usual - all calls are automatically traced
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY
});
const completion = await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "What is observability?" }
]
});
console.log(completion.choices[0].message.content);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
stream = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
```typescript Typescript theme={null}
const stream = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Tell me a story" }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```
### Embeddings
Embedding operations are also automatically instrumented:
```python Python theme={null}
embedding = client.embeddings.create(
model="text-embedding-ada-002",
input="The quick brown fox jumps over the lazy dog"
)
print(embedding.data[0].embedding)
```
```typescript Typescript theme={null}
const embedding = await openai.embeddings.create({
model: "text-embedding-ada-002",
input: "The quick brown fox jumps over the lazy dog"
});
console.log(embedding.data[0].embedding);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable OpenAI instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.OPENAI}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable OpenAI instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.OPENAI])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [OpenAI Documentation](https://platform.openai.com/docs/quickstart) - Official OpenAI API quickstart guide
# Replicate
Source: https://docs.getnetra.ai/Integrations/ai_providers/REPLICATE
Trace Replicate AI model calls with Netra auto-instrumentation. Monitor cloud-hosted model inference, predictions, and latency automatically.
## Installation
Install both the Netra SDK and Replicate SDK:
```bash Python theme={null}
pip install netra-sdk replicate
```
```bash Typescript theme={null}
npm install netra-sdk replicate
```
## Usage
Initialize the Netra SDK with Replicate instrumentation enabled. The SDK automatically traces all Replicate API calls once initialized.
```python Python theme={null}
from netra import Netra
import replicate
import os
# Initialize Netra with Replicate instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Replicate client as usual - all calls are automatically traced
output = replicate.run(
"meta/llama-2-70b-chat:latest",
input={"prompt": "What is observability?"}
)
print(output)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import Replicate from "replicate";
async function main() {
// Initialize Netra with Replicate instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Replicate client as usual - all calls are automatically traced
const replicate = new Replicate({
auth: process.env.REPLICATE_API_TOKEN
});
const output = await replicate.run(
"meta/llama-2-70b-chat:latest",
{
input: {
prompt: "What is observability?"
}
}
);
console.log(output);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
for event in replicate.stream(
"meta/llama-2-70b-chat:latest",
input={"prompt": "Tell me a story"}
):
print(event, end="")
```
```typescript Typescript theme={null}
for await (const event of replicate.stream(
"meta/llama-2-70b-chat:latest",
{
input: {
prompt: "Tell me a story"
}
}
)) {
process.stdout.write(event.toString());
}
```
### Image Generation
Image generation models are also automatically instrumented:
```python Python theme={null}
output = replicate.run(
"stability-ai/sdxl:latest",
input={"prompt": "A futuristic cityscape at sunset"}
)
print(output)
```
```typescript Typescript theme={null}
const output = await replicate.run(
"stability-ai/sdxl:latest",
{
input: {
prompt: "A futuristic cityscape at sunset"
}
}
);
console.log(output);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Replicate instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.REPLICATE}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Replicate instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.REPLICATE])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Replicate Documentation](https://replicate.com/docs/get-started/nodejs) - Official Replicate quickstart guide
# Together AI
Source: https://docs.getnetra.ai/Integrations/ai_providers/TOGETHERAI
Trace Together AI inference calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and latency across open-source models.
## Installation
Install both the Netra SDK and Together AI SDK:
```bash Python theme={null}
pip install netra-sdk together
```
```bash Typescript theme={null}
npm install netra-sdk together-ai
```
## Usage
Initialize the Netra SDK with Together AI instrumentation enabled. The SDK automatically traces all Together AI API calls once initialized.
```python Python theme={null}
from netra import Netra
from together import Together
import os
# Initialize Netra with Together AI instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Together AI client as usual - all calls are automatically traced
client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))
response = client.chat.completions.create(
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
messages=[
{"role": "user", "content": "What is observability?"}
]
)
print(response.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import Together from "together-ai";
async function main() {
// Initialize Netra with Together AI instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Together AI client as usual - all calls are automatically traced
const together = new Together({
apiKey: process.env.TOGETHER_API_KEY
});
const response = await together.chat.completions.create({
model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
messages: [
{ role: "user", content: "What is observability?" }
]
});
console.log(response.choices[0].message.content);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
stream = client.chat.completions.create(
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
messages=[{"role": "user", "content": "Tell me a story"}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
```typescript Typescript theme={null}
const stream = await together.chat.completions.create({
model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
messages: [{ role: "user", content: "Tell me a story" }],
stream: true
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```
### Embeddings
Embedding operations are also automatically instrumented:
```python Python theme={null}
embeddings = client.embeddings.create(
model="togethercomputer/m2-bert-80M-8k-retrieval",
input="The quick brown fox jumps over the lazy dog"
)
print(embeddings.data[0].embedding)
```
```typescript Typescript theme={null}
const embeddings = await together.embeddings.create({
model: "togethercomputer/m2-bert-80M-8k-retrieval",
input: "The quick brown fox jumps over the lazy dog"
});
console.log(embeddings.data[0].embedding);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Together AI instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.TOGETHER}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Together AI instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.TOGETHER])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Together AI Documentation](https://docs.together.ai/docs/quickstart) - Official Together AI quickstart guide
# Vertex AI
Source: https://docs.getnetra.ai/Integrations/ai_providers/VERTEXAI
Trace Google Vertex AI model calls with Netra auto-instrumentation. Monitor prompts, completions, token usage, and performance on GCP automatically.
## Installation
Install both the Netra SDK and Google Cloud Vertex AI SDK:
```bash Python theme={null}
pip install netra-sdk google-cloud-aiplatform
```
```bash Typescript theme={null}
npm install netra-sdk @google-cloud/vertexai
```
## Usage
Initialize the Netra SDK with Vertex AI instrumentation enabled. The SDK automatically traces all Vertex AI API calls once initialized.
```python Python theme={null}
from netra import Netra
from vertexai.preview.generative_models import GenerativeModel
import vertexai
import os
# Initialize Netra with Vertex AI instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Vertex AI client as usual - all calls are automatically traced
vertexai.init(project="your-project-id", location="us-central1")
model = GenerativeModel("gemini-pro")
response = model.generate_content("What is observability?")
print(response.text)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { VertexAI } from "@google-cloud/vertexai";
async function main() {
// Initialize Netra with Vertex AI instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Vertex AI client as usual - all calls are automatically traced
const vertexAI = new VertexAI({
project: "your-project-id",
location: "us-central1"
});
const model = vertexAI.getGenerativeModel({ model: "gemini-pro" });
const result = await model.generateContent("What is observability?");
const response = await result.response;
console.log(response.text());
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
model = GenerativeModel("gemini-pro")
responses = model.generate_content("Tell me a story", stream=True)
for response in responses:
print(response.text, end="")
```
```typescript Typescript theme={null}
const model = vertexAI.getGenerativeModel({ model: "gemini-pro" });
const result = await model.generateContentStream("Tell me a story");
for await (const chunk of result.stream) {
const chunkText = chunk.text();
process.stdout.write(chunkText);
}
```
### Chat Sessions
Chat operations are also automatically instrumented:
```python Python theme={null}
model = GenerativeModel("gemini-pro")
chat = model.start_chat()
response1 = chat.send_message("Hello")
print(response1.text)
response2 = chat.send_message("What is machine learning?")
print(response2.text)
```
```typescript Typescript theme={null}
const model = vertexAI.getGenerativeModel({ model: "gemini-pro" });
const chat = model.startChat();
const result1 = await chat.sendMessage("Hello");
console.log(result1.response.text());
const result2 = await chat.sendMessage("What is machine learning?");
console.log(result2.response.text());
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Vertex AI instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.VERTEXAI}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Vertex AI instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.VERTEXAI])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Vertex AI Documentation](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform) - Official Google Cloud Vertex AI documentation
# Watsonx AI
Source: https://docs.getnetra.ai/Integrations/ai_providers/WATSONX
Trace IBM Watsonx AI model calls with Netra auto-instrumentation. Monitor foundation model prompts, completions, token usage, and latency on IBM Cloud.
## Installation
Install both the Netra SDK and IBM Watsonx SDK:
```bash Python theme={null}
pip install netra-sdk ibm-watsonx-ai
```
```bash Typescript theme={null}
npm install netra-sdk @ibm-cloud/watsonx-ai
```
## Usage
Initialize the Netra SDK with Watsonx instrumentation enabled. The SDK automatically traces all Watsonx API calls once initialized.
```python Python theme={null}
from netra import Netra
from ibm_watsonx_ai.foundation_models import Model
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
import os
# Initialize Netra with Watsonx instrumentation
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use Watsonx client as usual - all calls are automatically traced
model = Model(
model_id="ibm/granite-13b-chat-v2",
credentials={
"url": "https://us-south.ml.cloud.ibm.com",
"apikey": os.environ.get("WATSONX_API_KEY")
},
project_id=os.environ.get("WATSONX_PROJECT_ID")
)
response = model.generate_text(prompt="What is observability?")
print(response)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import WatsonxAI from "@ibm-cloud/watsonx-ai";
// Initialize Netra with Watsonx instrumentation (must await)
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use Watsonx client as usual - all calls are automatically traced
const watsonx = new WatsonxAI({
version: "2024-05-31",
serviceUrl: "https://us-south.ml.cloud.ibm.com",
apikey: process.env.WATSONX_API_KEY
});
async function main() {
const response = await watsonx.generateText({
input: "What is observability?",
modelId: "ibm/granite-13b-chat-v2",
projectId: process.env.WATSONX_PROJECT_ID
});
console.log(response.results[0].generated_text);
}
main();
```
### Streaming Responses
The SDK automatically handles streaming responses and captures the complete output:
```python Python theme={null}
model = Model(
model_id="ibm/granite-13b-chat-v2",
credentials={
"url": "https://us-south.ml.cloud.ibm.com",
"apikey": os.environ.get("WATSONX_API_KEY")
},
project_id=os.environ.get("WATSONX_PROJECT_ID")
)
for chunk in model.generate_text_stream(prompt="Tell me a story"):
print(chunk, end="")
```
```typescript Typescript theme={null}
const stream = await watsonx.generateTextStream({
input: "Tell me a story",
modelId: "ibm/granite-13b-chat-v2",
projectId: process.env.WATSONX_PROJECT_ID
});
for await (const chunk of stream) {
if (chunk.results && chunk.results[0]) {
process.stdout.write(chunk.results[0].generated_text);
}
}
```
### Advanced Parameters
Configure generation parameters for more control:
```python Python theme={null}
generate_params = {
GenParams.MAX_NEW_TOKENS: 100,
GenParams.TEMPERATURE: 0.7,
GenParams.TOP_P: 0.9
}
model = Model(
model_id="ibm/granite-13b-chat-v2",
credentials={
"url": "https://us-south.ml.cloud.ibm.com",
"apikey": os.environ.get("WATSONX_API_KEY")
},
project_id=os.environ.get("WATSONX_PROJECT_ID"),
params=generate_params
)
response = model.generate_text(prompt="Explain AI")
print(response)
```
```typescript Typescript theme={null}
const response = await watsonx.generateText({
input: "Explain AI",
modelId: "ibm/granite-13b-chat-v2",
projectId: process.env.WATSONX_PROJECT_ID,
parameters: {
max_new_tokens: 100,
temperature: 0.7,
top_p: 0.9
}
});
console.log(response.results[0].generated_text);
```
### Selective Instrumentation
Control which integrations are enabled using the `instruments` or `blockInstruments` configuration:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable Watsonx instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.WATSONX}
)
# Or block specific instrumentations
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
block_instruments={InstrumentSet.HTTPX}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable Watsonx instrumentation
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.WATSONX])
});
// Or block specific instrumentations
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
blockInstruments: new Set([NetraInstruments.HTTP])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [Watsonx Documentation](https://www.ibm.com/docs/en/watsonx-as-a-service) - Official IBM Watsonx documentation
# ChromaDB
Source: https://docs.getnetra.ai/Integrations/db-docs/CHROMA
Trace ChromaDB vector operations with Netra auto-instrumentation. Monitor collection queries, embedding inserts, and similarity searches automatically.
## Installation
Install both the Netra SDK and ChromaDB:
```bash Python theme={null}
pip install netra-sdk chromadb
```
```bash Typescript theme={null}
npm install netra-sdk chromadb
```
## Usage
Initialize the Netra SDK to automatically trace all ChromaDB operations:
```python Python theme={null}
from netra import Netra
import chromadb
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create ChromaDB client - automatically traced
client = chromadb.Client()
collection = client.get_or_create_collection(name="my_collection")
# Add documents
collection.add(
ids=["id1", "id2"],
documents=["Document 1", "Document 2"],
metadatas=[{"source": "web"}, {"source": "book"}]
)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { ChromaClient } from "chromadb";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create ChromaDB client - automatically traced
const client = new ChromaClient();
const collection = await client.getOrCreateCollection({ name: "my_collection" });
// Add documents
await collection.add({
ids: ["id1", "id2"],
documents: ["Document 1", "Document 2"],
metadatas: [{ source: "web" }, { source: "book" }]
});
```
### Collection Operations
Trace collection creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_collection(client, name: str):
span = SpanWrapper("chroma-create-collection", {
"collection.name": name
}).start()
collection = client.create_collection(name=name)
span.set_attribute("collection.id", collection.id)
span.end()
return collection
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createCollection(client: ChromaClient, name: string) {
const span = new SpanWrapper("chroma-create-collection", {
"collection.name": name
}).start();
const collection = await client.createCollection({ name });
span.setAttribute("collection.id", collection.id);
span.end();
return collection;
}
```
### Document Insertion
Trace document additions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def add_documents(collection, documents: list[str]):
span = SpanWrapper("chroma-add-documents", {
"documents.count": len(documents)
}).start()
collection.add(
ids=[f"doc{i}" for i in range(len(documents))],
documents=documents
)
span.set_action([ActionModel(
action="insert",
action_type="database.insert",
success=True,
affected_records=[{"id": f"doc{i}"} for i in range(len(documents))],
metadata={"collection": collection.name}
)])
span.set_attribute("status", "success")
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function addDocuments(collection: any, documents: string[]) {
const span = new SpanWrapper("chroma-add-documents", {
"documents.count": documents.length
}).start();
await collection.add({
ids: documents.map((_, i) => `doc${i}`),
documents: documents
});
span.setAction([{
action: "insert",
action_type: "database.insert",
success: true,
affected_records: documents.map((_, i) => ({ id: `doc${i}` })),
metadata: { collection: collection.name }
}]);
span.setAttribute("status", "success");
span.end();
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_similar(collection, query: str, n: int = 5):
span = SpanWrapper("chroma-search", {
"query": query,
"n_results": n
}).start()
results = collection.query(
query_texts=[query],
n_results=n
)
span.set_attribute("results.count", len(results["ids"][0]))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchSimilar(collection: any, query: string, n: number = 5) {
const span = new SpanWrapper("chroma-search", {
"query": query,
"n_results": n
}).start();
const results = await collection.query({
queryTexts: [query],
nResults: n
});
span.setAttribute("results.count", results.ids[0].length);
span.end();
return results;
}
```
### Filtering
Trace filtered queries:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
import json
@task()
def filter_search(collection, query: str, filter: dict):
span = SpanWrapper("chroma-filter-search", {
"query": query,
"filter": json.dumps(filter)
}).start()
results = collection.query(
query_texts=[query],
where=filter
)
span.set_attribute("results.count", len(results["ids"][0]))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function filterSearch(collection: any, query: string, filter: any) {
const span = new SpanWrapper("chroma-filter-search", {
"query": query,
"filter": JSON.stringify(filter)
}).start();
const results = await collection.query({
queryTexts: [query],
where: filter
});
span.setAttribute("results.count", results.ids[0].length);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [ChromaDB Documentation](https://docs.trychroma.com/) - Official ChromaDB documentation
# LanceDB
Source: https://docs.getnetra.ai/Integrations/db-docs/LanceDB
Trace LanceDB vector operations with Netra auto-instrumentation. Monitor table queries, vector searches, and indexing operations automatically.
## Installation
Install both the Netra SDK and LanceDB:
```bash Python theme={null}
pip install netra-sdk lancedb
```
```bash Typescript theme={null}
npm install netra-sdk vectordb
```
## Usage
Initialize the Netra SDK to automatically trace all LanceDB operations:
```python Python theme={null}
from netra import Netra
import lancedb
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Connect to LanceDB - automatically traced
db = lancedb.connect("./lancedb")
table = db.create_table("my_table", [
{"id": 1, "vector": [0.1, 0.2], "text": "Sample"}
])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import * as vectordb from "vectordb";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Connect to LanceDB - automatically traced
const db = await vectordb.connect("./lancedb");
const table = await db.createTable("my_table", [
{ id: 1, vector: [0.1, 0.2], text: "Sample" }
]);
```
### Table Operations
Trace table creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_table(db, name: str, data: list):
span = SpanWrapper("lancedb-create-table", {
"table.name": name,
"data.count": len(data)
}).start()
table = db.create_table(name, data)
span.end()
return table
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createTable(db: any, name: string, data: any[]) {
const span = new SpanWrapper("lancedb-create-table", {
"table.name": name,
"data.count": data.length
}).start();
const table = await db.createTable(name, data);
span.end();
return table;
}
```
### Vector Insertion
Trace data insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def add_vectors(table, data: list):
span = SpanWrapper("lancedb-add", {
"vectors.count": len(data)
}).start()
table.add(data)
span.set_action([ActionModel(
action="insert",
action_type="database.insert",
success=True,
affected_records=[{"id": str(d["id"])} for d in data],
metadata={"table": table.name}
)])
span.set_attribute("status", "success")
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function addVectors(table: any, data: any[]) {
const span = new SpanWrapper("lancedb-add", {
"vectors.count": data.length
}).start();
await table.add(data);
span.setAction([{
action: "insert",
action_type: "database.insert",
success: true,
affected_records: data.map(d => ({ id: String(d.id) })),
metadata: { table: table.name }
}]);
span.setAttribute("status", "success");
span.end();
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_vectors(table, query: list[float], limit: int = 5):
span = SpanWrapper("lancedb-search", {
"query.size": len(query),
"limit": limit
}).start()
results = table.search(query).limit(limit).to_list()
span.set_attribute("results.count", len(results))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchVectors(table: any, query: number[], limit: number = 5) {
const span = new SpanWrapper("lancedb-search", {
"query.size": query.length,
"limit": limit
}).start();
const results = await table
.search(query)
.limit(limit)
.execute();
span.setAttribute("results.count", results.length);
span.end();
return results;
}
```
### Filtering
Trace filtered queries:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def filter_search(table, query: list[float], filter: str):
span = SpanWrapper("lancedb-filter-search", {
"filter": filter
}).start()
results = table.search(query).where(filter).limit(10).to_list()
span.set_attribute("results.count", len(results))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function filterSearch(table: any, query: number[], filter: string) {
const span = new SpanWrapper("lancedb-filter-search", {
"filter": filter
}).start();
const results = await table
.search(query)
.where(filter)
.limit(10)
.execute();
span.setAttribute("results.count", results.length);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [LanceDB Documentation](https://lancedb.github.io/lancedb/) - Official LanceDB documentation
# Marqo
Source: https://docs.getnetra.ai/Integrations/db-docs/Margo
Trace Marqo vector search operations with Netra auto-instrumentation. Monitor index queries, document inserts, and multimodal search automatically.
## Installation
Install both the Netra SDK and Marqo:
```bash Python theme={null}
pip install netra-sdk marqo
```
```bash Typescript theme={null}
npm install netra-sdk marqo
```
## Usage
Initialize the Netra SDK to automatically trace all Marqo operations:
```python Python theme={null}
from netra import Netra
import marqo
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Marqo client - automatically traced
mq = marqo.Client(
url=os.environ.get('MARQO_URL'),
api_key=os.environ.get('MARQO_API_KEY')
)
# Create index
mq.create_index("my-index")
# Add documents
mq.index("my-index").add_documents([
{"title": "Document 1", "content": "Content here"}
])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import marqo from "marqo";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Marqo client - automatically traced
const client = new marqo.Client({
url: process.env.MARQO_URL,
apiKey: process.env.MARQO_API_KEY
});
// Create index
await client.createIndex("my-index");
// Add documents
await client.index("my-index").addDocuments([
{ title: "Document 1", content: "Content here" }
]);
```
### Index Operations
Trace index creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_index(mq, index_name: str):
span = SpanWrapper("marqo-create-index", {
"index.name": index_name
}).start()
mq.create_index(
index_name,
model="hf/e5-base-v2"
)
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createIndex(client: any, indexName: string) {
const span = new SpanWrapper("marqo-create-index", {
"index.name": indexName
}).start();
await client.createIndex(indexName, {
model: "hf/e5-base-v2"
});
span.end();
}
```
### Document Insertion
Trace document additions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def add_documents(mq, index_name: str, documents: list):
span = SpanWrapper("marqo-add-documents", {
"index.name": index_name,
"documents.count": len(documents)
}).start()
result = mq.index(index_name).add_documents(documents)
span.set_action([ActionModel(
action="insert",
action_type="database.insert",
success=result["status"] == "succeeded",
affected_records=[{"id": d.get("id") or d.get("_id")} for d in documents],
metadata={"index": index_name}
)])
span.set_attribute("status", result["status"])
span.end()
return result
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function addDocuments(client: any, indexName: string, documents: any[]) {
const span = new SpanWrapper("marqo-add-documents", {
"index.name": indexName,
"documents.count": documents.length
}).start();
const result = await client.index(indexName).addDocuments(documents);
span.setAction([{
action: "insert",
action_type: "database.insert",
success: result.status === "succeeded",
affected_records: documents.map(d => ({ id: d.id || d._id })),
metadata: { index: indexName }
}]);
span.setAttribute("status", result.status);
span.end();
return result;
}
```
### Semantic Search
Trace text-based searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_documents(mq, index_name: str, query: str, limit: int = 5):
span = SpanWrapper("marqo-search", {
"index.name": index_name,
"query": query,
"limit": limit
}).start()
results = mq.index(index_name).search(
q=query,
limit=limit
)
span.set_attribute("results.count", len(results["hits"]))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchDocuments(client: any, indexName: string, query: string, limit: number = 5) {
const span = new SpanWrapper("marqo-search", {
"index.name": indexName,
"query": query,
"limit": limit
}).start();
const results = await client.index(indexName).search(query, {
limit: limit
});
span.setAttribute("results.count", results.hits.length);
span.end();
return results;
}
```
### Multimodal Search
Trace image and text searches:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def multimodal_search(mq, index_name: str, query: dict):
span = SpanWrapper("marqo-multimodal-search", {
"index.name": index_name
}).start()
results = mq.index(index_name).search(
q=query,
searchable_attributes=["title", "content", "image_url"]
)
span.set_attribute("results.count", len(results["hits"]))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function multimodalSearch(client: any, indexName: string, query: any) {
const span = new SpanWrapper("marqo-multimodal-search", {
"index.name": indexName
}).start();
const results = await client.index(indexName).search(query, {
searchableAttributes: ["title", "content", "image_url"]
});
span.setAttribute("results.count", results.hits.length);
span.end();
return results;
}
```
### Filtered Search
Trace searches with filters:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def filter_search(mq, index_name: str, query: str, filter: str):
span = SpanWrapper("marqo-filter-search", {
"index.name": index_name,
"filter": filter
}).start()
results = mq.index(index_name).search(
q=query,
filter_string=filter
)
span.set_attribute("results.count", len(results["hits"]))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function filterSearch(client: any, indexName: string, query: string, filter: string) {
const span = new SpanWrapper("marqo-filter-search", {
"index.name": indexName,
"filter": filter
}).start();
const results = await client.index(indexName).search(query, {
filter: filter
});
span.setAttribute("results.count", results.hits.length);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Marqo Documentation](https://docs.marqo.ai/) - Official Marqo documentation
# Milvus
Source: https://docs.getnetra.ai/Integrations/db-docs/Milvus
Trace Milvus vector database operations with Netra auto-instrumentation. Monitor collection searches, data inserts, and index queries automatically.
## Installation
Install both the Netra SDK and Milvus:
```bash Python theme={null}
pip install netra-sdk pymilvus
```
```bash Typescript theme={null}
npm install netra-sdk @zilliz/milvus2-sdk-node
```
## Usage
Initialize the Netra SDK to automatically trace all Milvus operations:
```python Python theme={null}
from netra import Netra
from pymilvus import MilvusClient
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Milvus client - automatically traced
client = MilvusClient(
uri=os.environ.get('MILVUS_URI'),
token=os.environ.get('MILVUS_TOKEN')
)
# Create collection
client.create_collection(
collection_name="my_collection",
dimension=384
)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { MilvusClient } from "@zilliz/milvus2-sdk-node";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Milvus client - automatically traced
const client = new MilvusClient({
address: process.env.MILVUS_ADDRESS,
token: process.env.MILVUS_TOKEN
});
// Create collection
await client.createCollection({
collection_name: "my_collection",
dimension: 384
});
```
### Collection Operations
Trace collection creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_collection(client: MilvusClient, name: str, dimension: int):
span = SpanWrapper("milvus-create-collection", {
"collection.name": name,
"vector.dimension": dimension
}).start()
client.create_collection(
collection_name=name,
dimension=dimension
)
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createCollection(client: MilvusClient, name: string, dimension: number) {
const span = new SpanWrapper("milvus-create-collection", {
"collection.name": name,
"vector.dimension": dimension
}).start();
await client.createCollection({
collection_name: name,
dimension: dimension
});
span.end();
}
```
### Vector Insertion
Trace entity insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def insert_vectors(client: MilvusClient, collection: str, data: list):
span = SpanWrapper("milvus-insert", {
"collection": collection,
"entities.count": len(data)
}).start()
result = client.insert(
collection_name=collection,
data=data
)
span.set_action([ActionModel(
action="insert",
action_type="database.insert",
success=True,
affected_records=[{"id": str(d["id"])} for d in data],
metadata={"collection": collection, "inserted": result["insert_count"]}
)])
span.set_attribute("insert.count", result["insert_count"])
span.end()
return result
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function insertVectors(client: MilvusClient, collection: string, data: any[]) {
const span = new SpanWrapper("milvus-insert", {
"collection": collection,
"entities.count": data.length
}).start();
const result = await client.insert({
collection_name: collection,
data: data
});
span.setAction([{
action: "insert",
action_type: "database.insert",
success: true,
affected_records: data.map(d => ({ id: String(d.id) })),
metadata: { collection, inserted: result.insert_cnt }
}]);
span.setAttribute("insert.count", result.insert_cnt);
span.end();
return result;
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_vectors(client: MilvusClient, collection: str, query: list[float], limit: int = 5):
span = SpanWrapper("milvus-search", {
"collection": collection,
"query.dimension": len(query),
"limit": limit
}).start()
results = client.search(
collection_name=collection,
data=[query],
limit=limit
)
span.set_attribute("results.count", len(results[0]))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchVectors(client: MilvusClient, collection: string, query: number[], limit: number = 5) {
const span = new SpanWrapper("milvus-search", {
"collection": collection,
"query.dimension": query.length,
"limit": limit
}).start();
const results = await client.search({
collection_name: collection,
data: [query],
limit: limit
});
span.setAttribute("results.count", results[0]?.length || 0);
span.end();
return results;
}
```
### Filtered Search
Trace searches with filters:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def filter_search(client: MilvusClient, collection: str, query: list[float], filter: str):
span = SpanWrapper("milvus-filter-search", {
"collection": collection,
"filter": filter
}).start()
results = client.search(
collection_name=collection,
data=[query],
filter=filter,
limit=10
)
span.set_attribute("results.count", len(results[0]))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function filterSearch(client: MilvusClient, collection: string, query: number[], filter: string) {
const span = new SpanWrapper("milvus-filter-search", {
"collection": collection,
"filter": filter
}).start();
const results = await client.search({
collection_name: collection,
data: [query],
filter: filter,
limit: 10
});
span.setAttribute("results.count", results[0]?.length || 0);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Milvus Documentation](https://milvus.io/docs) - Official Milvus documentation
# Pinecone
Source: https://docs.getnetra.ai/Integrations/db-docs/Pinecone
Trace Pinecone vector operations with Netra auto-instrumentation. Monitor queries, upserts, latency, and retrieval performance automatically.
## Installation
Install both the Netra SDK and Pinecone:
```bash Python theme={null}
pip install netra-sdk pinecone-client
```
```bash Typescript theme={null}
npm install netra-sdk @pinecone-database/pinecone
```
## Usage
Initialize the Netra SDK to automatically trace all Pinecone operations:
```python Python theme={null}
from netra import Netra
from pinecone import Pinecone
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Pinecone client - automatically traced
pc = Pinecone(api_key=os.environ.get('PINECONE_API_KEY'))
index = pc.Index("my-index")
# Upsert vectors
index.upsert(vectors=[{
"id": "vec1",
"values": [0.1, 0.2, 0.3],
"metadata": {"text": "Sample document"}
}])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { Pinecone } from "@pinecone-database/pinecone";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Pinecone client - automatically traced
const pinecone = new Pinecone({
apiKey: process.env.PINECONE_API_KEY
});
const index = pinecone.index("my-index");
// Upsert vectors
await index.upsert([{
id: "vec1",
values: [0.1, 0.2, 0.3],
metadata: { text: "Sample document" }
}]);
```
### Index Operations
Trace index creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_index(pc: Pinecone, name: str, dimension: int):
span = SpanWrapper("pinecone-create-index", {
"index.name": name,
"index.dimension": dimension
}).start()
pc.create_index(
name=name,
dimension=dimension,
metric="cosine",
spec={"serverless": {"cloud": "aws", "region": "us-west-2"}}
)
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createIndex(pinecone: Pinecone, name: string, dimension: number) {
const span = new SpanWrapper("pinecone-create-index", {
"index.name": name,
"index.dimension": dimension
}).start();
await pinecone.createIndex({
name,
dimension,
metric: "cosine",
spec: {
serverless: {
cloud: "aws",
region: "us-west-2"
}
}
});
span.end();
}
```
### Vector Upsert
Trace vector insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def upsert_vectors(index, vectors: list):
span = SpanWrapper("pinecone-upsert", {
"vectors.count": len(vectors)
}).start()
result = index.upsert(vectors=vectors)
span.set_action([ActionModel(
action="upsert",
action_type="database.upsert",
success=True,
affected_records=[{"id": v["id"]} for v in vectors],
metadata={"index": index._index_name, "upserted": result["upserted_count"]}
)])
span.set_attribute("upserted.count", result["upserted_count"])
span.end()
return result
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function upsertVectors(index: any, vectors: any[]) {
const span = new SpanWrapper("pinecone-upsert", {
"vectors.count": vectors.length
}).start();
const result = await index.upsert(vectors);
span.setAction([{
action: "upsert",
action_type: "database.upsert",
success: true,
affected_records: vectors.map(v => ({ id: v.id })),
metadata: { index: index.name, upserted: result.upsertedCount }
}]);
span.setAttribute("upserted.count", result.upsertedCount);
span.end();
return result;
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_vectors(index, query: list[float], top_k: int = 5):
span = SpanWrapper("pinecone-search", {
"query.dimension": len(query),
"top_k": top_k
}).start()
results = index.query(
vector=query,
top_k=top_k,
include_metadata=True
)
span.set_attribute("results.count", len(results["matches"]))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchVectors(index: any, query: number[], topK: number = 5) {
const span = new SpanWrapper("pinecone-search", {
"query.dimension": query.length,
"top_k": topK
}).start();
const results = await index.query({
vector: query,
topK,
includeMetadata: true
});
span.setAttribute("results.count", results.matches.length);
span.end();
return results;
}
```
### Namespace Operations
Trace namespace-specific operations:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def query_namespace(index, namespace: str, query: list[float]):
span = SpanWrapper("pinecone-namespace-query", {
"namespace": namespace
}).start()
results = index.query(
vector=query,
top_k=10,
namespace=namespace
)
span.set_attribute("results.count", len(results["matches"]))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function queryNamespace(index: any, namespace: string, query: number[]) {
const span = new SpanWrapper("pinecone-namespace-query", {
"namespace": namespace
}).start();
const results = await index.namespace(namespace).query({
vector: query,
topK: 10
});
span.setAttribute("results.count", results.matches.length);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Pinecone Documentation](https://docs.pinecone.io/) - Official Pinecone documentation
# Qdrant
Source: https://docs.getnetra.ai/Integrations/db-docs/Qdrant
Trace Qdrant vector operations with Netra auto-instrumentation. Monitor collection searches, point upserts, and payload filters automatically.
## Installation
Install both the Netra SDK and Qdrant:
```bash Python theme={null}
pip install netra-sdk qdrant-client
```
```bash Typescript theme={null}
npm install netra-sdk @qdrant/js-client-rest
```
## Usage
Initialize the Netra SDK to automatically trace all Qdrant operations:
```python Python theme={null}
from netra import Netra
from qdrant_client import QdrantClient
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Qdrant client - automatically traced
client = QdrantClient(
url=os.environ.get('QDRANT_URL'),
api_key=os.environ.get('QDRANT_API_KEY')
)
# Create collection
client.create_collection(
collection_name="my_collection",
vectors_config={"size": 384, "distance": "Cosine"}
)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { QdrantClient } from "@qdrant/js-client-rest";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Qdrant client - automatically traced
const client = new QdrantClient({
url: process.env.QDRANT_URL,
apiKey: process.env.QDRANT_API_KEY
});
// Create collection
await client.createCollection("my_collection", {
vectors: { size: 384, distance: "Cosine" }
});
```
### Collection Operations
Trace collection creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_collection(client: QdrantClient, name: str, size: int):
span = SpanWrapper("qdrant-create-collection", {
"collection.name": name,
"vector.size": size
}).start()
client.create_collection(
collection_name=name,
vectors_config={"size": size, "distance": "Cosine"}
)
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createCollection(client: QdrantClient, name: string, size: number) {
const span = new SpanWrapper("qdrant-create-collection", {
"collection.name": name,
"vector.size": size
}).start();
await client.createCollection(name, {
vectors: { size, distance: "Cosine" }
});
span.end();
}
```
### Point Insertion
Trace point insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def upsert_points(client: QdrantClient, collection: str, points: list):
span = SpanWrapper("qdrant-upsert", {
"collection": collection,
"points.count": len(points)
}).start()
client.upsert(
collection_name=collection,
points=points
)
span.set_action([ActionModel(
action="upsert",
action_type="database.upsert",
success=True,
affected_records=[{"id": str(p.id)} for p in points],
metadata={"collection": collection}
)])
span.set_attribute("status", "success")
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function upsertPoints(client: QdrantClient, collection: string, points: any[]) {
const span = new SpanWrapper("qdrant-upsert", {
"collection": collection,
"points.count": points.length
}).start();
await client.upsert(collection, {
points: points
});
span.setAction([{
action: "upsert",
action_type: "database.upsert",
success: true,
affected_records: points.map(p => ({ id: String(p.id) })),
metadata: { collection }
}]);
span.setAttribute("status", "success");
span.end();
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def search_points(client: QdrantClient, collection: str, query: list[float], limit: int = 5):
span = SpanWrapper("qdrant-search", {
"collection": collection,
"query.size": len(query),
"limit": limit
}).start()
results = client.search(
collection_name=collection,
query_vector=query,
limit=limit
)
span.set_attribute("results.count", len(results))
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchPoints(client: QdrantClient, collection: string, query: number[], limit: number = 5) {
const span = new SpanWrapper("qdrant-search", {
"collection": collection,
"query.size": query.length,
"limit": limit
}).start();
const results = await client.search(collection, {
vector: query,
limit
});
span.setAttribute("results.count", results.length);
span.end();
return results;
}
```
### Filtering
Trace filtered searches:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
import json
@task()
def filter_search(client: QdrantClient, collection: str, query: list[float], filter: dict):
span = SpanWrapper("qdrant-filter-search", {
"collection": collection,
"filter": json.dumps(filter)
}).start()
results = client.search(
collection_name=collection,
query_vector=query,
query_filter=filter,
limit=10
)
span.set_attribute("results.count", len(results))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function filterSearch(client: QdrantClient, collection: string, query: number[], filter: any) {
const span = new SpanWrapper("qdrant-filter-search", {
"collection": collection,
"filter": JSON.stringify(filter)
}).start();
const results = await client.search(collection, {
vector: query,
filter,
limit: 10
});
span.setAttribute("results.count", results.length);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Qdrant Documentation](https://qdrant.tech/documentation/) - Official Qdrant documentation
# Redis
Source: https://docs.getnetra.ai/Integrations/db-docs/Redis
Trace Redis vector search operations with Netra auto-instrumentation. Monitor vector queries, caching operations, and key lookups automatically.
## Installation
Install both the Netra SDK and Redis:
```bash Python theme={null}
pip install netra-sdk redis
```
```bash Typescript theme={null}
npm install netra-sdk redis
```
## Usage
Initialize the Netra SDK to automatically trace all Redis operations:
```python Python theme={null}
from netra import Netra
import redis
from redis.commands.search.field import VectorField
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Redis client - automatically traced
client = redis.Redis.from_url(os.environ.get('REDIS_URL'))
# Create index for vector search
client.ft("idx:vectors").create_index([
VectorField("vector",
"HNSW", {
"TYPE": "FLOAT32",
"DIM": 384,
"DISTANCE_METRIC": "COSINE"
}
)
])
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { createClient } from "redis";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Redis client - automatically traced
const client = createClient({
url: process.env.REDIS_URL
});
await client.connect();
// Create index for vector search
await client.ft.create("idx:vectors", {
vector: {
type: "VECTOR",
ALGORITHM: "HNSW",
DIM: 384,
DISTANCE_METRIC: "COSINE"
}
});
```
### Index Operations
Trace index creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from redis.commands.search.field import VectorField
@task()
def create_vector_index(client, index_name: str, dimension: int):
span = SpanWrapper("redis-create-index", {
"index.name": index_name,
"vector.dimension": dimension
}).start()
client.ft(index_name).create_index([
VectorField("vector", "HNSW", {
"TYPE": "FLOAT32",
"DIM": dimension,
"DISTANCE_METRIC": "COSINE"
})
])
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createVectorIndex(client: any, indexName: string, dimension: number) {
const span = new SpanWrapper("redis-create-index", {
"index.name": indexName,
"vector.dimension": dimension
}).start();
await client.ft.create(indexName, {
vector: {
type: "VECTOR",
ALGORITHM: "HNSW",
DIM: dimension,
DISTANCE_METRIC: "COSINE"
}
});
span.end();
}
```
### Vector Storage
Trace vector insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
import numpy as np
@task()
def store_vector(client, key: str, vector: list[float], metadata: dict):
span = SpanWrapper("redis-store-vector", {
"key": key,
"vector.size": len(vector)
}).start()
client.hset(key, mapping={
"vector": np.array(vector, dtype=np.float32).tobytes(),
**metadata
})
span.set_action([ActionModel(
action="set",
action_type="database.set",
success=True,
affected_records=[{"id": key}],
metadata={"vector_size": len(vector)}
)])
span.set_attribute("status", "success")
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function storeVector(client: any, key: string, vector: number[], metadata: any) {
const span = new SpanWrapper("redis-store-vector", {
"key": key,
"vector.size": vector.length
}).start();
await client.hSet(key, {
vector: Buffer.from(new Float32Array(vector).buffer),
...metadata
});
span.setAction([{
action: "set",
action_type: "database.set",
success: true,
affected_records: [{ id: key }],
metadata: { vector_size: vector.length }
}]);
span.setAttribute("status", "success");
span.end();
}
```
### Vector Search
Trace similarity searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
import numpy as np
@workflow()
def search_vectors(client, index_name: str, query: list[float], limit: int = 5):
span = SpanWrapper("redis-search", {
"index": index_name,
"query.size": len(query),
"limit": limit
}).start()
query_vec = np.array(query, dtype=np.float32).tobytes()
results = client.ft(index_name).search(
f"*=>[KNN {limit} @vector $query_vec]",
query_params={"query_vec": query_vec}
)
span.set_attribute("results.count", results.total)
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function searchVectors(client: any, indexName: string, query: number[], limit: number = 5) {
const span = new SpanWrapper("redis-search", {
"index": indexName,
"query.size": query.length,
"limit": limit
}).start();
const results = await client.ft.search(
indexName,
`*=>[KNN ${limit} @vector $query_vec]`,
{
PARAMS: {
query_vec: Buffer.from(new Float32Array(query).buffer)
},
RETURN: ["id", "score"]
}
);
span.setAttribute("results.count", results.total);
span.end();
return results;
}
```
### Hybrid Search
Trace combined vector and metadata searches:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
import numpy as np
@task()
def hybrid_search(client, index_name: str, query: list[float], filter: str):
span = SpanWrapper("redis-hybrid-search", {
"index": index_name,
"filter": filter
}).start()
query_vec = np.array(query, dtype=np.float32).tobytes()
results = client.ft(index_name).search(
f"{filter}=>[KNN 10 @vector $query_vec]",
query_params={"query_vec": query_vec}
)
span.set_attribute("results.count", results.total)
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function hybridSearch(client: any, indexName: string, query: number[], filter: string) {
const span = new SpanWrapper("redis-hybrid-search", {
"index": indexName,
"filter": filter
}).start();
const results = await client.ft.search(
indexName,
`${filter}=>[KNN 10 @vector $query_vec]`,
{
PARAMS: {
query_vec: Buffer.from(new Float32Array(query).buffer)
}
}
);
span.setAttribute("results.count", results.total);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Redis Documentation](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/) - Redis vector search documentation
# Weaviate
Source: https://docs.getnetra.ai/Integrations/db-docs/Weavite
Trace Weaviate vector database operations with Netra auto-instrumentation. Monitor schema queries, object inserts, and vector searches automatically.
## Installation
Install both the Netra SDK and Weaviate:
```bash Python theme={null}
pip install netra-sdk weaviate-client
```
```bash Typescript theme={null}
npm install netra-sdk weaviate-ts-client
```
## Usage
Initialize the Netra SDK to automatically trace all Weaviate operations:
```python Python theme={null}
from netra import Netra
import weaviate
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Weaviate client - automatically traced
client = weaviate.Client(
url=os.environ.get('WEAVIATE_URL'),
auth_client_secret=weaviate.AuthApiKey(os.environ.get('WEAVIATE_API_KEY'))
)
# Query data
result = client.query.get("Article", ["title", "content"]).do()
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import weaviate from "weaviate-ts-client";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Weaviate client - automatically traced
const client = weaviate.client({
scheme: "https",
host: process.env.WEAVIATE_URL,
apiKey: new weaviate.ApiKey(process.env.WEAVIATE_API_KEY)
});
// Query data
const result = await client.graphql
.get()
.withClassName("Article")
.withFields("title content")
.do();
```
### Schema Operations
Trace schema creation and management:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def create_schema(client, class_name: str):
span = SpanWrapper("weaviate-create-schema", {
"class.name": class_name
}).start()
schema = {
"class": class_name,
"properties": [
{"name": "title", "dataType": ["text"]},
{"name": "content", "dataType": ["text"]}
]
}
client.schema.create_class(schema)
span.end()
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function createSchema(client: any, className: string) {
const span = new SpanWrapper("weaviate-create-schema", {
"class.name": className
}).start();
const schema = {
class: className,
properties: [
{
name: "title",
dataType: ["text"]
},
{
name: "content",
dataType: ["text"]
}
]
};
await client.schema.classCreator().withClass(schema).do();
span.end();
}
```
### Object Insertion
Trace object insertions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper, ActionModel
@task()
def add_object(client, class_name: str, properties: dict):
span = SpanWrapper("weaviate-add-object", {
"class.name": class_name
}).start()
result = client.data_object.create(
data_object=properties,
class_name=class_name
)
span.set_action([ActionModel(
action="insert",
action_type="database.insert",
success=True,
affected_records=[{"id": result}],
metadata={"class": class_name}
)])
span.set_attribute("object.id", result)
span.end()
return result
```
```typescript Typescript theme={null}
import { task, SpanWrapper, ActionModel } from "netra-sdk";
@task()
async function addObject(client: any, className: string, properties: any) {
const span = new SpanWrapper("weaviate-add-object", {
"class.name": className
}).start();
const result = await client.data
.creator()
.withClassName(className)
.withProperties(properties)
.do();
span.setAction([{
action: "insert",
action_type: "database.insert",
success: true,
affected_records: [{ id: result.id }],
metadata: { class: className }
}]);
span.setAttribute("object.id", result.id);
span.end();
return result;
}
```
### Vector Search
Trace semantic searches:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def semantic_search(client, class_name: str, query: str, limit: int = 5):
span = SpanWrapper("weaviate-search", {
"class.name": class_name,
"query": query,
"limit": limit
}).start()
results = (
client.query
.get(class_name, ["title", "content"])
.with_near_text({"concepts": [query]})
.with_limit(limit)
.with_additional(["distance"])
.do()
)
count = len(results.get("data", {}).get("Get", {}).get(class_name, []))
span.set_attribute("results.count", count)
span.end()
return results
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function semanticSearch(client: any, className: string, query: string, limit: number = 5) {
const span = new SpanWrapper("weaviate-search", {
"class.name": className,
"query": query,
"limit": limit
}).start();
const results = await client.graphql
.get()
.withClassName(className)
.withFields("title content _additional { distance }")
.withNearText({ concepts: [query] })
.withLimit(limit)
.do();
const count = results.data.Get[className]?.length || 0;
span.setAttribute("results.count", count);
span.end();
return results;
}
```
### Hybrid Search
Trace hybrid (keyword + vector) searches:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def hybrid_search(client, class_name: str, query: str):
span = SpanWrapper("weaviate-hybrid-search", {
"class.name": class_name,
"query": query
}).start()
results = (
client.query
.get(class_name, ["title", "content"])
.with_hybrid(query=query, alpha=0.5)
.do()
)
count = len(results.get("data", {}).get("Get", {}).get(class_name, []))
span.set_attribute("results.count", count)
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function hybridSearch(client: any, className: string, query: string) {
const span = new SpanWrapper("weaviate-hybrid-search", {
"class.name": className,
"query": query
}).start();
const results = await client.graphql
.get()
.withClassName(className)
.withFields("title content")
.withHybrid({ query, alpha: 0.5 })
.do();
const count = results.data.Get[className]?.length || 0;
span.setAttribute("results.count", count);
span.end();
return results;
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Weaviate Documentation](https://weaviate.io/developers/weaviate) - Official Weaviate documentation
# Google ADK
Source: https://docs.getnetra.ai/Integrations/orchestrators/ADK
Trace Google Agent Development Kit (ADK) workflows with Netra auto-instrumentation. Monitor agent execution, tool calls, and state across ADK apps.
## Installation
Install both the Netra SDK and Google ADK:
```bash Python theme={null}
pip install netra-sdk google-adk
```
## Usage
Initialize the Netra SDK to automatically trace all Google ADK operations:
```python Python theme={null}
from netra import Netra
from google.adk import Agent
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Define agent - automatically traced
agent = Agent(
name="assistant",
model="gemini-1.5-pro",
instructions="You are a helpful assistant"
)
# Run agent
response = await agent.run("What is Google ADK?")
print(response.output)
```
### Getting Started
Trace basic agent execution:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from google.adk import Agent
@task()
async def run_agent(prompt: str) -> str:
span = SpanWrapper("adk-agent", {
"prompt": prompt
}).start()
agent = Agent(
name="assistant",
model="gemini-1.5-pro",
instructions="You are a helpful assistant"
)
response = await agent.run(prompt)
span.set_attribute("response", response.output)
span.end()
return response.output
```
### Agent Tools
Trace agents with custom tools:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
from google.adk import Agent, Tool
def get_weather(location: str) -> str:
"""Get weather information for a location."""
return f"Weather in {location}: Sunny, 72°F"
@agent()
async def agent_with_tools(query: str):
tool_span = SpanWrapper("adk-agent-tools", {
"query": query
}).start()
# Define tool
weather_tool = Tool(
name="get_weather",
description="Get weather information",
function=get_weather
)
# Create agent with tool
assistant = Agent(
name="weather-agent",
model="gemini-1.5-pro",
instructions="Help users with weather information",
tools=[weather_tool]
)
response = await assistant.run(query)
tool_span.set_attribute("response", response.output)
tool_span.end()
return response.output
```
### Multi-Agent Systems
Trace multi-agent orchestration:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
from google.adk import Agent
@workflow()
async def multi_agent_system(task: str):
workflow_span = SpanWrapper("multi-agent-workflow", {
"task": task
}).start()
# Research agent
researcher = Agent(
name="researcher",
model="gemini-1.5-pro",
instructions="Research and gather information"
)
# Writer agent
writer = Agent(
name="writer",
model="gemini-1.5-pro",
instructions="Write based on research"
)
# Execute agents in sequence
research_result = await researcher.run(f"Research: {task}")
final_result = await writer.run(
f"Write based on: {research_result.output}"
)
workflow_span.set_attribute("result", final_result.output)
workflow_span.end()
return final_result.output
```
### Streaming Responses
Trace streaming agent outputs:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
async def stream_agent_response(prompt: str):
stream_span = SpanWrapper("adk-stream").start()
agent = Agent(
name="assistant",
model="gemini-1.5-pro",
instructions="You are a helpful assistant"
)
full_response = ""
async for chunk in agent.stream(prompt):
print(chunk, end='', flush=True)
full_response += chunk
stream_span.set_attribute("response", full_response)
stream_span.end()
return full_response
```
### State Management
Trace agents with persistent state:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
from google.adk import Agent, Memory
@agent()
async def stateful_agent(user_id: str, message: str):
state_span = SpanWrapper("adk-stateful", {
"user_id": user_id,
"message": message
}).start()
# Create agent with memory
assistant = Agent(
name="assistant",
model="gemini-1.5-pro",
instructions="Remember context from previous interactions",
memory=Memory(user_id=user_id)
)
response = await assistant.run(message)
state_span.set_attribute("response", response.output)
state_span.end()
return response.output
```
## Configuration
Configure selective instrumentation:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Enable Google ADK instrumentation
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
instruments={InstrumentSet.ADK}
)
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Google ADK Documentation](https://google.github.io/adk-docs/) - Official Google ADK documentation
# Cerebras
Source: https://docs.getnetra.ai/Integrations/orchestrators/Cerebras
Trace Cerebras ultra-fast AI inference with Netra auto-instrumentation. Monitor model calls, token throughput, and latency across all inference runs.
## Installation
Install both the Netra SDK and Cerebras:
```bash Python theme={null}
pip install netra-sdk cerebras-cloud-sdk
```
```bash Typescript theme={null}
npm install netra-sdk @cerebras/cerebras_cloud_sdk
```
## Usage
Initialize the Netra SDK to automatically trace all Cerebras operations:
```python Python theme={null}
from netra import Netra
from cerebras.cloud.sdk import Cerebras
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create Cerebras client - automatically traced
client = Cerebras(api_key=os.environ.get('CEREBRAS_API_KEY'))
# Use Cerebras as normal
response = client.chat.completions.create(
model="llama3.1-8b",
messages=[{"role": "user", "content": "What is Cerebras?"}]
)
print(response.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import Cerebras from "@cerebras/cerebras_cloud_sdk";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create Cerebras client - automatically traced
const client = new Cerebras({
apiKey: process.env.CEREBRAS_API_KEY
});
// Use Cerebras as normal
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: [{ role: "user", content: "What is Cerebras?" }]
});
console.log(response.choices[0].message.content);
```
### Chat Completions
Trace chat completions with decorators:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from cerebras.cloud.sdk import Cerebras
@task()
def chat_completion(client: Cerebras, prompt: str) -> str:
span = SpanWrapper("cerebras-chat", {
"prompt": prompt,
"model": "llama3.1-8b"
}).start()
response = client.chat.completions.create(
model="llama3.1-8b",
messages=[{"role": "user", "content": prompt}]
)
answer = response.choices[0].message.content
span.set_attribute("response", answer)
span.set_attribute("tokens.completion", response.usage.completion_tokens)
span.end()
return answer
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
import Cerebras from "@cerebras/cerebras_cloud_sdk";
@task()
async function chatCompletion(client: Cerebras, prompt: string): Promise {
const span = new SpanWrapper("cerebras-chat", {
"prompt": prompt,
"model": "llama3.1-8b"
}).start();
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: [{ role: "user", content: prompt }]
});
const answer = response.choices[0].message.content;
span.setAttribute("response", answer);
span.setAttribute("tokens.completion", response.usage?.completion_tokens || 0);
span.end();
return answer;
}
```
### Streaming Responses
Trace streaming completions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def stream_completion(client: Cerebras, prompt: str):
stream_span = SpanWrapper("cerebras-stream").start()
response = client.chat.completions.create(
model="llama3.1-8b",
messages=[{"role": "user", "content": prompt}],
stream=True
)
full_response = ""
for chunk in response:
if chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
print(content, end='', flush=True)
full_response += content
stream_span.set_attribute("response", full_response)
stream_span.end()
return full_response
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
@task()
async function streamCompletion(client: Cerebras, prompt: string) {
const streamSpan = new SpanWrapper("cerebras-stream").start();
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: [{ role: "user", content: prompt }],
stream: true
});
let fullResponse = "";
for await (const chunk of response) {
if (chunk.choices[0]?.delta?.content) {
const content = chunk.choices[0].delta.content;
process.stdout.write(content);
fullResponse += content;
}
}
streamSpan.setAttribute("response", fullResponse);
streamSpan.end();
return fullResponse;
}
```
### Multi-turn Conversations
Trace conversation history:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
@agent()
def conversation(client: Cerebras, messages: list[dict]):
conv_span = SpanWrapper("cerebras-conversation", {
"messages.count": len(messages)
}).start()
response = client.chat.completions.create(
model="llama3.1-8b",
messages=messages
)
answer = response.choices[0].message.content
conv_span.set_attribute("response", answer)
conv_span.end()
return answer
```
```typescript Typescript theme={null}
import { agent, SpanWrapper } from "netra-sdk";
@agent()
async function conversation(client: Cerebras, messages: Array) {
const convSpan = new SpanWrapper("cerebras-conversation", {
"messages.count": messages.length
}).start();
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: messages
});
const answer = response.choices[0].message.content;
convSpan.setAttribute("response", answer);
convSpan.end();
return answer;
}
```
### Model Configuration
Trace with different models and settings:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
@workflow()
def generate_with_config(client: Cerebras, prompt: str, temperature: float = 0.7):
config_span = SpanWrapper("cerebras-configured", {
"prompt": prompt,
"temperature": temperature
}).start()
response = client.chat.completions.create(
model="llama3.1-70b",
messages=[{"role": "user", "content": prompt}],
temperature=temperature,
max_tokens=1000
)
result = response.choices[0].message.content
config_span.set_attribute("response", result)
config_span.end()
return result
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
@workflow()
async function generateWithConfig(client: Cerebras, prompt: string, temperature: number = 0.7) {
const configSpan = new SpanWrapper("cerebras-configured", {
"prompt": prompt,
"temperature": temperature
}).start();
const response = await client.chat.completions.create({
model: "llama3.1-70b",
messages: [{ role: "user", content: prompt }],
temperature: temperature,
max_tokens: 1000
});
const result = response.choices[0].message.content;
configSpan.setAttribute("response", result);
configSpan.end();
return result;
}
```
## Configuration
Configure Cerebras instrumentation:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
instruments={InstrumentSet.CEREBRAS}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true,
instruments: new Set([NetraInstruments.CEREBRAS])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Cerebras Documentation](https://inference-docs.cerebras.ai/quickstart) - Official Cerebras documentation
# CrewAI
Source: https://docs.getnetra.ai/Integrations/orchestrators/CrewAI
Trace CrewAI multi-agent workflows with Netra auto-instrumentation. Monitor crew execution, agent tasks, tool calls, and handoffs automatically.
## Installation
Install both the Netra SDK and CrewAI:
```bash Python theme={null}
pip install netra-sdk crewai crewai-tools
```
## Usage
Initialize the Netra SDK to automatically trace all CrewAI operations:
```python Python theme={null}
from netra import Netra
from crewai import Agent, Task, Crew
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Define agents - automatically traced
researcher = Agent(
role="Researcher",
goal="Research and analyze information",
backstory="Expert researcher with attention to detail"
)
writer = Agent(
role="Writer",
goal="Write engaging content",
backstory="Creative writer with storytelling skills"
)
# Create tasks
research_task = Task(
description="Research AI trends",
agent=researcher,
expected_output="Research report"
)
write_task = Task(
description="Write article based on research",
agent=writer,
expected_output="Article"
)
# Create crew
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, write_task]
)
result = crew.kickoff()
```
### Agent Workflows
Trace multi-agent workflows with custom decorators:
```python Python theme={null}
from netra.decorators import workflow, agent, task
from netra import SpanWrapper
@workflow()
def research_workflow(topic: str):
span = SpanWrapper("research-workflow", {
"topic": topic
}).start()
# Create specialized agents
researcher = Agent(
role="Researcher",
goal=f"Research {topic}",
backstory="Expert researcher"
)
analyst = Agent(
role="Analyst",
goal="Analyze research data",
backstory="Data analyst"
)
# Define tasks
research = Task(
description=f"Research {topic}",
agent=researcher,
expected_output="Research findings"
)
analysis = Task(
description="Analyze research",
agent=analyst,
expected_output="Analysis report"
)
crew = Crew(
agents=[researcher, analyst],
tasks=[research, analysis],
verbose=True
)
result = crew.kickoff()
span.set_attribute("result.status", "completed")
span.end()
return result
```
### Task Orchestration
Trace individual agent tasks:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def execute_agent_task(agent: Agent, task_description: str):
task_span = SpanWrapper("agent-task", {
"agent.role": agent.role,
"task.description": task_description
}).start()
try:
task = Task(
description=task_description,
agent=agent,
expected_output="Task result"
)
result = agent.execute_task(task)
task_span.set_attribute("task.result", str(result))
task_span.end()
return result
except Exception as e:
task_span.set_attribute("error", str(e))
task_span.set_status(code=1, message=str(e))
task_span.end()
raise
```
### Crew Configuration
Trace crew execution with configuration:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
from crewai import Process
@agent()
def run_crew_with_config(agents: list[Agent], tasks: list[Task]):
config_span = SpanWrapper("crew-config", {
"agents.count": len(agents),
"tasks.count": len(tasks)
}).start()
crew = Crew(
agents=agents,
tasks=tasks,
process=Process.sequential,
verbose=True,
memory=True
)
config_span.end()
# Execute crew
exec_span = SpanWrapper("crew-execution").start()
result = crew.kickoff()
exec_span.set_attribute("execution.status", "completed")
exec_span.end()
return result
```
### Session Tracking
Track multi-turn crew interactions:
```python Python theme={null}
from netra.decorators import workflow
from netra import SessionManager
@workflow()
def interactive_crew_session(user_id: str, queries: list[str]):
session_manager = SessionManager()
session_manager.start_session(user_id=user_id)
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, write_task]
)
results = []
for query in queries:
conversation_id = session_manager.start_conversation()
result = crew.kickoff(inputs={"query": query})
results.append(result)
session_manager.end_conversation()
session_manager.end_session()
return results
```
## Configuration
Configure CrewAI tracing options:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Enable specific instruments
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
instruments={InstrumentSet.OPENAI} # CrewAI uses OpenAI
)
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [CrewAI Documentation](https://docs.crewai.com/) - Official CrewAI documentation
# DSPy
Source: https://docs.getnetra.ai/Integrations/orchestrators/DSPy
Trace DSPy programs with Netra auto-instrumentation. Monitor module execution, optimizer runs, and LLM calls across compiled and uncompiled programs.
## Installation
Install both the Netra SDK and DSPy:
```bash Python theme={null}
pip install netra-sdk dspy-ai
```
## Usage
Initialize the Netra SDK to automatically trace all DSPy operations:
```python Python theme={null}
from netra import Netra
import dspy
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Configure DSPy - automatically traced
lm = dspy.OpenAI(model="gpt-3.5-turbo")
dspy.settings.configure(lm=lm)
# Use DSPy as normal
qa = dspy.Predict("question -> answer")
result = qa(question="What is DSPy?")
```
### Getting Started
Trace DSPy signatures and modules:
```python Python theme={null}
from netra.decorators import workflow, task
from netra import SpanWrapper
import dspy
# Define signature with task decorator
class QA(dspy.Signature):
"""Answer questions with short factual answers."""
question = dspy.InputField()
answer = dspy.OutputField(desc="short answer")
@task()
def simple_qa(question: str) -> str:
span = SpanWrapper("dspy-qa", {
"question": question
}).start()
predictor = dspy.Predict(QA)
result = predictor(question=question)
span.set_attribute("answer", result.answer)
span.end()
return result.answer
result = simple_qa("What is machine learning?")
```
### Advanced Features
Trace DSPy chains and optimization:
```python Python theme={null}
from netra.decorators import workflow, agent
from netra import SpanWrapper
import dspy
class RAG(dspy.Module):
def __init__(self, num_passages=3):
super().__init__()
self.retrieve = dspy.Retrieve(k=num_passages)
self.generate = dspy.ChainOfThought("context, question -> answer")
@task()
def forward(self, question):
context = self.retrieve(question).passages
answer = self.generate(context=context, question=question)
return answer
@workflow()
def rag_pipeline(question: str):
span = SpanWrapper("rag-pipeline", {
"question": question
}).start()
rag = RAG()
result = rag.forward(question=question)
span.set_attribute("answer", result.answer)
span.end()
return result
```
### Chain of Thought
Trace reasoning chains:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
def reasoning_task(question: str):
span = SpanWrapper("chain-of-thought", {
"question": question
}).start()
# Use Chain of Thought
cot = dspy.ChainOfThought("question -> answer")
result = cot(question=question)
span.set_attribute("reasoning", result.rationale)
span.set_attribute("answer", result.answer)
span.end()
return result
```
### Optimization
Trace DSPy optimizers:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
import dspy
from dspy.teleprompt import BootstrapFewShot
@workflow()
def optimize_module(module, trainset):
opt_span = SpanWrapper("dspy-optimization", {
"trainset.size": len(trainset)
}).start()
# Configure optimizer
optimizer = BootstrapFewShot(metric=lambda x, y: x.answer == y.answer)
# Compile the module
compiled = optimizer.compile(module, trainset=trainset)
opt_span.set_attribute("optimization.status", "completed")
opt_span.end()
return compiled
```
## Configuration
Configure DSPy tracing options:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Enable specific instruments for DSPy's underlying LLM
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
instruments={InstrumentSet.OPENAI} # DSPy uses OpenAI
)
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [DSPy Documentation](https://dspy-docs.vercel.app/) - Official DSPy documentation
# Haystack
Source: https://docs.getnetra.ai/Integrations/orchestrators/Haystack
Trace Haystack NLP pipelines with Netra auto-instrumentation. Monitor pipeline components, document retrieval, and LLM generation steps automatically.
## Installation
Install both the Netra SDK and Haystack:
```bash Python theme={null}
pip install netra-sdk haystack-ai
```
## Usage
Initialize the Netra SDK to automatically trace all Haystack operations:
```python python theme={null}
from netra import Netra
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create pipeline - automatically traced
pipeline = Pipeline()
pipeline.add_component("generator", OpenAIGenerator())
result = pipeline.run({
"generator": {
"prompt": "What is Haystack?"
}
})
```
### Pipelines
Trace Haystack pipelines with custom decorators:
```python python theme={null}
from netra.decorators import workflow, task
from netra import SpanWrapper
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
@workflow()
def build_qa_pipeline():
pipeline_span = SpanWrapper("build-pipeline").start()
pipeline = Pipeline()
# Add components
pipeline.add_component("prompt_builder", PromptBuilder(
template="Answer this question: {{question}}"
))
pipeline.add_component("generator", OpenAIGenerator())
# Connect components
pipeline.connect("prompt_builder", "generator")
pipeline_span.end()
return pipeline
@task()
def run_pipeline(pipeline: Pipeline, question: str):
run_span = SpanWrapper("run-pipeline", {
"question": question
}).start()
result = pipeline.run({
"prompt_builder": {"question": question}
})
run_span.set_attribute("answer", result["generator"]["replies"][0])
run_span.end()
return result
```
### Retrievers
Trace document retrieval:
```python python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from haystack import Pipeline
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.dataclasses import Document
@task()
def setup_retrieval_pipeline():
setup_span = SpanWrapper("setup-retrieval").start()
# Create document store
document_store = InMemoryDocumentStore()
document_store.write_documents([
Document(content="Haystack is an NLP framework."),
Document(content="It supports RAG pipelines.")
])
# Create pipeline
pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(
document_store=document_store
))
setup_span.end()
return pipeline
@task()
def retrieve_documents(pipeline: Pipeline, query: str):
retrieval_span = SpanWrapper("retrieve-docs", {
"query": query
}).start()
result = pipeline.run({
"retriever": {"query": query}
})
docs = result["retriever"]["documents"]
retrieval_span.set_attribute("documents.count", len(docs))
retrieval_span.end()
return docs
```
### RAG Pipeline
Trace complete RAG implementations:
```python python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
from haystack import Pipeline
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
@workflow()
def rag_pipeline(document_store, query: str):
rag_span = SpanWrapper("rag-pipeline", {
"query": query
}).start()
# Build pipeline
pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(
document_store=document_store
))
pipeline.add_component("prompt_builder", PromptBuilder(
template="""Context: {{documents}}
Question: {{query}}
Answer:"""
))
pipeline.add_component("generator", OpenAIGenerator())
# Connect components
pipeline.connect("retriever", "prompt_builder.documents")
pipeline.connect("prompt_builder", "generator")
# Run pipeline
result = pipeline.run({
"retriever": {"query": query},
"prompt_builder": {"query": query}
})
rag_span.set_attribute("answer", result["generator"]["replies"][0])
rag_span.end()
return result
```
### Custom Components
Trace custom pipeline components:
```python python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from haystack import component
from typing import List
@component
class CustomProcessor:
@component.output_types(processed=str)
@task()
def run(self, text: str) -> dict:
process_span = SpanWrapper("custom-process", {
"text.length": len(text)
}).start()
# Custom processing
processed = text.upper()
process_span.set_attribute("processed.length", len(processed))
process_span.end()
return {"processed": processed}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Haystack Documentation](https://docs.haystack.deepset.ai/) - Official Haystack documentation
# LangChain
Source: https://docs.getnetra.ai/Integrations/orchestrators/LangChain
Trace LangChain workflows with Netra auto-instrumentation. Monitor chains, agents, tools, and retrievers with automatic span capture for every step.
## Installation
Install both the Netra SDK and LangChain:
```bash Python theme={null}
pip install netra-sdk langchain langchain-openai
```
```bash Typescript theme={null}
npm install netra-sdk langchain
```
## Usage
Initialize the Netra SDK to automatically trace all LangChain operations:
```python Python theme={null}
from netra import Netra
from langchain_openai import ChatOpenAI
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use LangChain as normal - automatically traced
model = ChatOpenAI(temperature=0.9)
response = model.invoke("Tell me a joke")
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { ChatOpenAI } from "langchain/chat_models/openai";
async function main() {
// Initialize Netra (must await)
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use LangChain as normal - automatically traced
const model = new ChatOpenAI({ temperature: 0.9 });
const response = await model.invoke("Tell me a joke");
}
main();
```
### Chains
Trace LangChain chains with custom workflow decorators:
```python Python theme={null}
from netra.decorators import workflow, span
from langchain_openai import ChatOpenAI
from langchain.prompts import PromptTemplate
model = ChatOpenAI()
@workflow()
def summarize_chain(text: str):
prompt = PromptTemplate.from_template(
"Summarize this text: {text}"
)
chain = prompt | model
return chain.invoke({"text": text})
result = summarize_chain("Long article text...")
```
```typescript Typescript theme={null}
import { workflow, span } from "netra-sdk";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { PromptTemplate } from "langchain/prompts";
const model = new ChatOpenAI();
@workflow()
async function summarizeChain(text: string) {
const prompt = PromptTemplate.fromTemplate(
"Summarize this text: {text}"
);
const chain = prompt.pipe(model);
return await chain.invoke({ text });
}
const result = await summarizeChain("Long article text...");
```
### Agents and Tools
Trace LangChain agents with custom span creation:
```python Python theme={null}
from netra.decorators import agent, task
from netra import SpanWrapper
from langchain_openai import ChatOpenAI
from langchain.agents import create_react_agent, AgentExecutor
from langchain.tools import Tool
model = ChatOpenAI()
@agent()
def run_agent(query: str):
span = SpanWrapper("agent-execution").start()
try:
agent_executor = create_react_agent(
llm=model,
tools=[calculator_tool]
)
result = agent_executor.invoke({"input": query})
span.end()
return result
except Exception as e:
span.set_attribute("error", str(e))
span.end()
raise
```
```typescript Typescript theme={null}
import { agent, task, SpanWrapper } from "netra-sdk";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { Calculator } from "langchain/tools/calculator";
import { createReactAgent, AgentExecutor } from "langchain/agents";
const model = new ChatOpenAI();
const tools = [new Calculator()];
@agent()
async function runAgent(query: string) {
const span = new SpanWrapper("agent-execution").start();
try {
const agentExecutor = await createReactAgent({
llm: model,
tools
});
const result = await agentExecutor.invoke({ input: query });
span.end();
return result;
} catch (error) {
span.setAttribute("error", String(error));
span.end();
throw error;
}
}
```
### Streaming Responses
The SDK automatically captures streaming outputs from LangChain:
```python Python theme={null}
from netra.decorators import task
@task()
def stream_response(query: str):
model = ChatOpenAI(streaming=True)
for chunk in model.stream(query):
print(chunk.content, end="", flush=True)
```
```typescript Typescript theme={null}
import { task } from "netra-sdk";
@task()
async function streamResponse(query: string) {
const model = new ChatOpenAI({ streaming: true });
for await (const chunk of await model.stream(query)) {
process.stdout.write(chunk.content);
}
}
```
### Selective Instrumentation
Control which integrations are enabled:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Only enable specific instruments
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
instruments={InstrumentSet.OPENAI}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Only enable specific instruments
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
instruments: new Set([NetraInstruments.OPENAI])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Auto Instrumentation](https://docs.getnetra.ai/tracing/auto-instrumentation) - Automatic tracing for supported libraries
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [LangChain Documentation](https://python.langchain.com/docs/get_started/quickstart) - Official LangChain quickstart guide
# LangGraph
Source: https://docs.getnetra.ai/Integrations/orchestrators/LangGraph
Trace LangGraph agent workflows with Netra auto-instrumentation. Monitor graph state, node execution, and agent decisions across multi-step pipelines.
## Installation
Install both the Netra SDK and LangGraph:
```bash Python theme={null}
pip install netra-sdk langgraph
```
```bash Typescript theme={null}
npm install netra-sdk langgraph
```
## Usage
Initialize the Netra SDK to automatically trace all LangGraph operations:
```python Python theme={null}
from netra import Netra
from langgraph.graph import StateGraph
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Define your graph - automatically traced
from typing import TypedDict
class GraphState(TypedDict):
messages: list[str]
workflow = StateGraph(GraphState)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { StateGraph } from "@langchain/langgraph";
async function main() {
// Initialize Netra (must await)
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Define your graph - automatically traced
interface GraphState {
messages: string[];
}
const workflow = new StateGraph({
channels: {
messages: { value: (x, y) => x.concat(y) }
}
});
}
main();
```
### Core Concepts
Trace LangGraph workflows with custom decorators:
```python Python theme={null}
from netra.decorators import workflow, agent, task
from netra import SpanWrapper
# Node function with task decorator
@task()
def process_node(state: GraphState) -> GraphState:
span = SpanWrapper("node-processing", {
"node.name": "process",
"state.messages": len(state["messages"])
}).start()
result = {
"messages": state["messages"] + ["Processed"]
}
span.end()
return result
# Build graph with workflow decorator
@workflow()
def build_graph():
workflow.add_node("process", process_node)
workflow.set_entry_point("process")
workflow.set_finish_point("process")
return workflow.compile()
```
```typescript Typescript theme={null}
import { workflow, agent, task, SpanWrapper } from "netra-sdk";
// Node function with task decorator
@task()
async function processNode(state: GraphState) {
const span = new SpanWrapper("node-processing", {
"node.name": "process",
"state.messages": state.messages.length
}).start();
const result = {
messages: [...state.messages, "Processed"]
};
span.end();
return result;
}
// Build graph with workflow decorator
@workflow()
async function buildGraph() {
workflow.addNode("process", processNode);
workflow.setEntryPoint("process");
workflow.setFinishPoint("process");
return workflow.compile();
}
```
### Workflow Patterns
Trace multi-node agent workflows:
```python Python theme={null}
@agent()
def agent_workflow(query: str):
graph = StateGraph(GraphState)
@task()
def analyze(state: GraphState) -> GraphState:
return {"messages": state["messages"] + ["Analyzed"]}
@task()
def decide(state: GraphState) -> GraphState:
return {"messages": state["messages"] + ["Decision made"]}
graph.add_node("analyze", analyze)
graph.add_node("decide", decide)
graph.add_edge("analyze", "decide")
graph.set_entry_point("analyze")
graph.set_finish_point("decide")
app = graph.compile()
return app.invoke({"messages": [query]})
```
```typescript Typescript theme={null}
@agent()
async function agentWorkflow(query: string) {
const graph = new StateGraph({
channels: {
messages: { value: (x, y) => x.concat(y) }
}
});
@task()
async function analyze(state: GraphState) {
return { messages: [...state.messages, "Analyzed"] };
}
@task()
async function decide(state: GraphState) {
return { messages: [...state.messages, "Decision made"] };
}
graph.addNode("analyze", analyze);
graph.addNode("decide", decide);
graph.addEdge("analyze", "decide");
graph.setEntryPoint("analyze");
graph.setFinishPoint("decide");
const app = graph.compile();
return await app.invoke({ messages: [query] });
}
```
### State Management
Capture state transitions with manual spans:
```python Python theme={null}
from netra import SpanWrapper
import json
def run_with_state_tracking(app, initial_state: GraphState):
state_span = SpanWrapper("state-management").start()
try:
result = app.invoke(initial_state)
state_span.set_attribute("state.initial", json.dumps(initial_state))
state_span.set_attribute("state.final", json.dumps(result))
state_span.end()
return result
except Exception as e:
state_span.set_status(code=1, message=str(e))
state_span.end()
raise
```
```typescript Typescript theme={null}
import { SpanWrapper } from "netra-sdk";
async function runWithStateTracking(app: any, initialState: GraphState) {
const stateSpan = new SpanWrapper("state-management").start();
try {
const result = await app.invoke(initialState);
stateSpan.setAttribute("state.initial", JSON.stringify(initialState));
stateSpan.setAttribute("state.final", JSON.stringify(result));
stateSpan.end();
return result;
} catch (error) {
stateSpan.setStatus({ code: 1, message: String(error) });
stateSpan.end();
throw error;
}
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) - Official LangGraph documentation
# LiteLLM
Source: https://docs.getnetra.ai/Integrations/orchestrators/LiteLLM
Trace LiteLLM unified LLM calls with Netra auto-instrumentation. Monitor requests across 100+ providers with automatic prompt and token tracking.
## Installation
Install both the Netra SDK and LiteLLM:
```bash Python theme={null}
pip install netra-sdk litellm
```
```bash Typescript theme={null}
npm install netra-sdk litellm
```
## Usage
Initialize the Netra SDK to automatically trace all LiteLLM operations:
```python Python theme={null}
from netra import Netra
from litellm import completion
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use LiteLLM as normal - automatically traced
response = completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { completion } from "litellm";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use LiteLLM as normal - automatically traced
const response = await completion({
model: "gpt-3.5-turbo",
messages: [{ role: "user", content: "Hello!" }]
});
console.log(response.choices[0].message.content);
```
### LiteLLM Features
Trace completions across different providers:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from litellm import completion
@task()
def call_multiple_providers(prompt: str) -> dict:
span = SpanWrapper("multi-provider", {
"prompt": prompt
}).start()
results = {}
# OpenAI
openai_response = completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": prompt}]
)
results["openai"] = openai_response.choices[0].message.content
# Anthropic
anthropic_response = completion(
model="claude-3-sonnet-20240229",
messages=[{"role": "user", "content": prompt}]
)
results["anthropic"] = anthropic_response.choices[0].message.content
span.set_attribute("providers.count", len(results))
span.end()
return results
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
import { completion } from "litellm";
@task()
async function callMultipleProviders(prompt: string): Promise> {
const span = new SpanWrapper("multi-provider", {
"prompt": prompt
}).start();
const results: Record = {};
// OpenAI
const openaiResponse = await completion({
model: "gpt-3.5-turbo",
messages: [{ role: "user", content: prompt }]
});
results["openai"] = openaiResponse.choices[0].message.content;
// Anthropic
const anthropicResponse = await completion({
model: "claude-3-sonnet-20240229",
messages: [{ role: "user", content: prompt }]
});
results["anthropic"] = anthropicResponse.choices[0].message.content;
span.setAttribute("providers.count", Object.keys(results).length);
span.end();
return results;
}
```
### Lightweight Design
Trace with minimal overhead:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
from litellm import completion
@workflow()
def efficient_completion(messages: list[dict], model: str = "gpt-3.5-turbo"):
span = SpanWrapper("litellm-completion", {
"model": model,
"messages.count": len(messages)
}).start()
response = completion(
model=model,
messages=messages,
timeout=30
)
content = response.choices[0].message.content
span.set_attribute("response", content)
span.set_attribute("tokens.total", response.usage.total_tokens)
span.end()
return content
```
```typescript Typescript theme={null}
import { workflow, SpanWrapper } from "netra-sdk";
import { completion } from "litellm";
@workflow()
async function efficientCompletion(messages: Array, model: string = "gpt-3.5-turbo") {
const span = new SpanWrapper("litellm-completion", {
"model": model,
"messages.count": messages.length
}).start();
const response = await completion({
model: model,
messages: messages,
timeout: 30
});
const content = response.choices[0].message.content;
span.setAttribute("response", content);
span.setAttribute("tokens.total", response.usage?.total_tokens || 0);
span.end();
return content;
}
```
### Modular Architecture
Trace with custom configurations:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
from litellm import completion
@agent()
def completion_with_fallback(prompt: str, models: list[str]):
span = SpanWrapper("fallback-completion", {
"prompt": prompt,
"models": ",".join(models)
}).start()
for model in models:
try:
response = completion(
model=model,
messages=[{"role": "user", "content": prompt}],
timeout=10
)
span.set_attribute("successful_model", model)
span.end()
return response.choices[0].message.content
except Exception as e:
span.set_attribute(f"error.{model}", str(e))
continue
span.set_status(code=1, message="All models failed")
span.end()
raise Exception("All models failed")
```
```typescript Typescript theme={null}
import { agent, SpanWrapper } from "netra-sdk";
import { completion } from "litellm";
@agent()
async function completionWithFallback(prompt: string, models: string[]) {
const span = new SpanWrapper("fallback-completion", {
"prompt": prompt,
"models": models.join(",")
}).start();
for (const model of models) {
try {
const response = await completion({
model: model,
messages: [{ role: "user", content: prompt }],
timeout: 10
});
span.setAttribute("successful_model", model);
span.end();
return response.choices[0].message.content;
} catch (error) {
span.setAttribute(`error.${model}`, String(error));
continue;
}
}
span.setStatus({ code: 1, message: "All models failed" });
span.end();
throw new Error("All models failed");
}
```
### Cross-Platform Support
Trace streaming across platforms:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from litellm import completion
@task()
def stream_any_provider(model: str, prompt: str):
stream_span = SpanWrapper("litellm-stream", {
"model": model
}).start()
response = completion(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True
)
full_response = ""
for chunk in response:
if chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
print(content, end='', flush=True)
full_response += content
stream_span.set_attribute("response", full_response)
stream_span.end()
return full_response
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
import { completion } from "litellm";
@task()
async function streamAnyProvider(model: string, prompt: string) {
const streamSpan = new SpanWrapper("litellm-stream", {
"model": model
}).start();
const response = await completion({
model: model,
messages: [{ role: "user", content: prompt }],
stream: true
});
let fullResponse = "";
for await (const chunk of response) {
if (chunk.choices[0]?.delta?.content) {
const content = chunk.choices[0].delta.content;
process.stdout.write(content);
fullResponse += content;
}
}
streamSpan.setAttribute("response", fullResponse);
streamSpan.end();
return fullResponse;
}
```
### Async Support
Trace async completions:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from litellm import acompletion
import asyncio
@task()
async def async_completion(prompt: str):
span = SpanWrapper("async-completion").start()
response = await acompletion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": prompt}]
)
content = response.choices[0].message.content
span.set_attribute("response", content)
span.end()
return content
```
```typescript Typescript theme={null}
import { task, SpanWrapper } from "netra-sdk";
import { completion } from "litellm";
@task()
async function asyncCompletion(prompt: string) {
const span = new SpanWrapper("async-completion").start();
const response = await completion({
model: "gpt-3.5-turbo",
messages: [{ role: "user", content: prompt }]
});
const content = response.choices[0].message.content;
span.setAttribute("response", content);
span.end();
return content;
}
```
## Configuration
Configure LiteLLM instrumentation:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
instruments={InstrumentSet.LITELLM}
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true,
instruments: new Set([NetraInstruments.LITELLM])
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [LiteLLM Documentation](https://docs.litellm.ai/) - Official LiteLLM documentation
# LlamaIndex
Source: https://docs.getnetra.ai/Integrations/orchestrators/LlamaIndex
Trace LlamaIndex RAG pipelines with Netra auto-instrumentation. Monitor query engines, retrievers, embeddings, and LLM calls in every pipeline run.
## Installation
Install both the Netra SDK and LlamaIndex:
```bash Python theme={null}
pip install netra-sdk llama-index
```
```bash Typescript theme={null}
npm install netra-sdk llamaindex
```
## Usage
Initialize the Netra SDK to automatically trace all LlamaIndex operations:
```python Python theme={null}
from netra import Netra
from llama_index.core import VectorStoreIndex, Document
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Use LlamaIndex as normal - automatically traced
documents = [
Document(text="LlamaIndex is a data framework for LLM applications.")
]
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is LlamaIndex?")
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { VectorStoreIndex, Document } from "llamaindex";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Use LlamaIndex as normal - automatically traced
const documents = [
new Document({ text: "LlamaIndex is a data framework for LLM applications." })
];
const index = await VectorStoreIndex.fromDocuments(documents);
const queryEngine = index.asQueryEngine();
const response = await queryEngine.query("What is LlamaIndex?");
```
### Core Features
Trace indexing and retrieval workflows:
```python Python theme={null}
from netra.decorators import workflow, task
from netra import SpanWrapper
@workflow()
def build_rag_pipeline(documents: list[Document]):
index_span = SpanWrapper("build-index", {
"documents.count": len(documents)
}).start()
index = VectorStoreIndex.from_documents(documents)
index_span.end()
return index
@task()
def query_with_retrieval(query_engine, question: str):
query_span = SpanWrapper("query-execution", {
"query.text": question
}).start()
response = query_engine.query(question)
query_span.set_attribute("response.sources", len(response.source_nodes or []))
query_span.end()
return response
```
```typescript Typescript theme={null}
import { workflow, task, SpanWrapper } from "netra-sdk";
@workflow()
async function buildRAGPipeline(documents: Document[]) {
const indexSpan = new SpanWrapper("build-index", {
"documents.count": documents.length
}).start();
const index = await VectorStoreIndex.fromDocuments(documents);
indexSpan.end();
return index;
}
@task()
async function queryWithRetrieval(queryEngine: any, question: string) {
const querySpan = new SpanWrapper("query-execution", {
"query.text": question
}).start();
const response = await queryEngine.query(question);
querySpan.setAttribute("response.sources", response.sourceNodes?.length || 0);
querySpan.end();
return response;
}
```
### Query Examples
Trace complex query patterns:
```python Python theme={null}
from netra.decorators import agent
from netra import SpanWrapper
@agent()
def multi_step_query(index: VectorStoreIndex, queries: list[str]):
query_engine = index.as_query_engine()
results = []
for query in queries:
span = SpanWrapper(f"query-{query}", {
"query.text": query
}).start()
response = query_engine.query(query)
span.set_attribute("response.text", str(response))
span.end()
results.append(response)
return results
```
```typescript Typescript theme={null}
import { agent } from "netra-sdk";
@agent()
async function multiStepQuery(index: VectorStoreIndex, queries: string[]) {
const queryEngine = index.asQueryEngine();
const results = [];
for (const query of queries) {
const span = new SpanWrapper(`query-${query}`, {
"query.text": query
}).start();
const response = await queryEngine.query(query);
span.setAttribute("response.text", response.toString());
span.end();
results.push(response);
}
return results;
}
```
### Streaming Responses
Trace streaming query responses:
```python Python theme={null}
from netra.decorators import task
@task()
def stream_query(query_engine, question: str):
streaming_response = query_engine.query(question, streaming=True)
for chunk in streaming_response.response_gen:
print(chunk, end="", flush=True)
```
```typescript Typescript theme={null}
import { task } from "netra-sdk";
@task()
async function streamQuery(queryEngine: any, question: string) {
const streamingResponse = await queryEngine.query(question, {
streaming: true
});
for await (const chunk of streamingResponse) {
process.stdout.write(chunk);
}
}
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [Session Tracking](https://docs.getnetra.ai/tracing/session) - Track user sessions and conversations
* [LlamaIndex Documentation](https://docs.llamaindex.ai/) - Official LlamaIndex documentation
# Model Context Protocol (MCP)
Source: https://docs.getnetra.ai/Integrations/orchestrators/MCP
Trace Model Context Protocol (MCP) server operations with Netra. Monitor tool calls, resource access, and prompt requests across MCP-based AI apps.
## Installation
Install both the Netra SDK and MCP:
```bash Python theme={null}
pip install netra-sdk mcp
```
```bash Typescript theme={null}
npm install netra-sdk @modelcontextprotocol/sdk
```
## Usage
Initialize the Netra SDK to trace MCP server operations:
```python Python theme={null}
from netra import Netra
from mcp.server import Server
from mcp.server.stdio import stdio_server
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create MCP server - automatically traced
server = Server("example-server")
```
```typescript Typescript theme={null}
import { Netra } from "netra-sdk";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
// Initialize Netra
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true
});
// Create MCP server - automatically traced
const server = new Server(
{
name: "example-server",
version: "1.0.0"
},
{
capabilities: {
tools: {},
resources: {}
}
}
);
```
### Tools
Trace MCP tool execution:
```python Python theme={null}
from netra.decorators import workflow, task
from netra import SpanWrapper
from mcp.server.models import Tool
from mcp.types import TextContent
@server.call_tool()
async def execute_tool(name: str, arguments: dict):
span = SpanWrapper("mcp-tool", {
"tool.name": name
}).start()
try:
result = await process_tool(name, arguments)
span.set_attribute("tool.result", str(result))
span.end()
return [TextContent(
type="text",
text=str(result)
)]
except Exception as e:
span.set_attribute("error", str(e))
span.end()
raise
@task()
async def process_tool(name: str, args: dict):
# Tool implementation
return {"status": "success"}
```
```typescript Typescript theme={null}
import { workflow, task, SpanWrapper } from "netra-sdk";
import { CallToolRequestSchema } from "@modelcontextprotocol/sdk/types.js";
// Register tool with tracing
server.setRequestHandler(CallToolRequestSchema, async (request) => {
const span = new SpanWrapper("mcp-tool", {
"tool.name": request.params.name
}).start();
try {
const result = await executeTool(request.params);
span.setAttribute("tool.result", JSON.stringify(result));
span.end();
return {
content: [{ type: "text", text: JSON.stringify(result) }]
};
} catch (error) {
span.setAttribute("error", String(error));
span.end();
throw error;
}
});
@task()
async function executeTool(params: any) {
// Tool implementation
return { status: "success" };
}
```
### Resources
Trace resource access:
```python Python theme={null}
from netra import SpanWrapper
from mcp.server.models import Resource
from mcp.types import TextResourceContents
@server.list_resources()
async def list_resources():
return [
Resource(
uri="file:///example.txt",
name="Example Resource"
)
]
@server.read_resource()
async def read_resource(uri: str):
span = SpanWrapper("mcp-resource", {
"resource.uri": uri
}).start()
try:
content = await load_resource(uri)
span.set_attribute("resource.size", len(content))
span.end()
return TextResourceContents(
uri=uri,
mimeType="text/plain",
text=content
)
except Exception as e:
span.set_attribute("error", str(e))
span.end()
raise
```
```typescript Typescript theme={null}
import { ReadResourceRequestSchema } from "@modelcontextprotocol/sdk/types.js";
server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
const span = new SpanWrapper("mcp-resource", {
"resource.uri": request.params.uri
}).start();
try {
const content = await loadResource(request.params.uri);
span.setAttribute("resource.size", content.length);
span.end();
return {
contents: [{
uri: request.params.uri,
mimeType: "text/plain",
text: content
}]
};
} catch (error) {
span.setAttribute("error", String(error));
span.end();
throw error;
}
});
```
### Prompts
Trace prompt handling:
```python Python theme={null}
from netra import SpanWrapper
from mcp.server.models import Prompt, PromptMessage
@server.list_prompts()
async def list_prompts():
return [
Prompt(
name="example-prompt",
description="Example prompt template"
)
]
@server.get_prompt()
async def get_prompt(name: str, arguments: dict):
span = SpanWrapper("mcp-prompt", {
"prompt.name": name
}).start()
messages = generate_prompt(name, arguments)
span.set_attribute("prompt.messages", len(messages))
span.end()
return messages
```
```typescript Typescript theme={null}
import { GetPromptRequestSchema } from "@modelcontextprotocol/sdk/types.js";
server.setRequestHandler(GetPromptRequestSchema, async (request) => {
const span = new SpanWrapper("mcp-prompt", {
"prompt.name": request.params.name
}).start();
const prompt = generatePrompt(request.params);
span.setAttribute("prompt.messages", prompt.messages.length);
span.end();
return prompt;
});
```
## Configuration
Configure MCP server tracing:
```python Python theme={null}
from netra import Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True,
app_name="mcp-server"
)
```
```typescript Typescript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
traceContent: true,
appName: "mcp-server"
});
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [MCP Documentation](https://modelcontextprotocol.io/) - Official Model Context Protocol documentation
# PydanticAI
Source: https://docs.getnetra.ai/Integrations/orchestrators/PydanticAI
Trace PydanticAI agents with Netra auto-instrumentation. Monitor type-safe agent execution, tool calls, model requests, and response validation.
## Installation
Install both the Netra SDK and PydanticAI:
```bash Python theme={null}
pip install netra-sdk pydantic-ai
```
## Usage
Initialize the Netra SDK to automatically trace all PydanticAI operations:
```python Python theme={null}
from netra import Netra
from pydantic_ai import Agent
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
# Create agent - automatically traced
agent = Agent(
'openai:gpt-4',
system_prompt='You are a helpful assistant.'
)
result = agent.run_sync('What is Pydantic?')
print(result.data)
```
### Defining Agents
Trace PydanticAI agents with custom decorators:
```python Python theme={null}
from netra.decorators import agent, task
from netra import SpanWrapper
from pydantic_ai import Agent
from pydantic import BaseModel
class Response(BaseModel):
answer: str
confidence: float
@agent()
def create_qa_agent():
span = SpanWrapper("create-agent").start()
qa_agent = Agent(
'openai:gpt-4',
result_type=Response,
system_prompt='Answer questions with confidence scores.'
)
span.end()
return qa_agent
@task()
def query_agent(agent: Agent, question: str):
query_span = SpanWrapper("query-agent", {
"question": question
}).start()
result = agent.run_sync(question)
query_span.set_attribute("answer", result.data.answer)
query_span.set_attribute("confidence", result.data.confidence)
query_span.end()
return result.data
```
### Tools and Functions
Trace agent tools:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
from pydantic_ai import Agent, RunContext
@workflow()
def agent_with_tools():
agent = Agent('openai:gpt-4')
@agent.tool
def calculate(ctx: RunContext, x: float, y: float, op: str) -> float:
"""Perform mathematical operations."""
tool_span = SpanWrapper("tool-calculate", {
"operation": op,
"x": x,
"y": y
}).start()
if op == "add":
result = x + y
elif op == "multiply":
result = x * y
else:
result = 0
tool_span.set_attribute("result", result)
tool_span.end()
return result
result = agent.run_sync('What is 15 + 27?')
return result.data
```
### Structured Outputs
Trace structured response generation:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
from pydantic import BaseModel
from pydantic_ai import Agent
class Article(BaseModel):
title: str
summary: str
tags: list[str]
@task()
def generate_article(topic: str) -> Article:
span = SpanWrapper("generate-article", {
"topic": topic
}).start()
agent = Agent(
'openai:gpt-4',
result_type=Article,
system_prompt='Generate structured articles.'
)
result = agent.run_sync(f'Write about {topic}')
span.set_attribute("article.title", result.data.title)
span.set_attribute("article.tags", ",".join(result.data.tags))
span.end()
return result.data
```
### Streaming Responses
Trace streaming agent outputs:
```python Python theme={null}
from netra.decorators import task
from netra import SpanWrapper
@task()
async def stream_response(agent: Agent, prompt: str):
stream_span = SpanWrapper("stream-response").start()
async with agent.run_stream(prompt) as response:
async for chunk in response.stream():
print(chunk, end='', flush=True)
stream_span.set_attribute("response", response.data)
stream_span.end()
return response.data
```
### Dependency Injection
Trace agents with dependencies:
```python Python theme={null}
from netra.decorators import workflow
from netra import SpanWrapper
from pydantic_ai import Agent, RunContext
from dataclasses import dataclass
@dataclass
class DatabaseDeps:
connection_string: str
@workflow()
def agent_with_deps():
agent = Agent(
'openai:gpt-4',
deps_type=DatabaseDeps
)
@agent.tool
def query_db(ctx: RunContext[DatabaseDeps], query: str) -> str:
"""Query the database."""
db_span = SpanWrapper("db-query", {
"query": query
}).start()
# Use ctx.deps.connection_string
result = f"Query result for: {query}"
db_span.end()
return result
deps = DatabaseDeps(connection_string="postgresql://...")
result = agent.run_sync('Get user data', deps=deps)
return result.data
```
## Next Steps
* [Quick Start Guide](https://docs.getnetra.ai/quick-start/python) - Complete setup and configuration
* [Decorators](https://docs.getnetra.ai/tracing/decorators) - Add custom tracing with `@workflow`, `@agent`, and `@task` decorators
* [PydanticAI Documentation](https://ai.pydantic.dev/) - Official PydanticAI documentation
# Agno
Source: https://docs.getnetra.ai/Integrations/orchestrators/agno
Trace Agno agents, teams, and workflows with Netra auto-instrumentation. Monitor tool calls, interactions, and performance automatically.
Trace Agno functionalities with Netra auto-instrumentation. Monitor agents, teams, workflows, tool calls, and interactions.
## Installation
Install both the Netra SDK and Agno:
```bash Python theme={null}
pip install netra-sdk agno
```
## Usage
Initialize the Netra SDK to automatically trace all Agno operations:
```python Python theme={null}
from netra import Netra
from agno.agent import Agent
from agno.tools.yfinance import YFinanceTools
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
agent = Agent(
name="Finance Agent",
model="openai:gpt-5.4",
tools=[YFinanceTools()],
instructions="Fetch market data and produce a one-line take.",
)
agent.print_response("What's NVDA trading at today?")
```
### Getting Started
Trace basic team execution:
```python Python theme={null}
from netra import Netra
from agno.agent import Agent
from agno.team import Team
from agno.tools.yfinance import YFinanceTools
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
bull = Agent(
name="Bull",
model="openai:gpt-5.4",
role="Make the case FOR investing.",
tools=[YFinanceTools()],
)
bear = Agent(
name="Bear",
model="openai:gpt-5.4",
role="Make the case AGAINST investing.",
tools=[YFinanceTools()],
)
team = Team(
name="Investment Committee",
members=[bull, bear],
instructions="Hear both sides, then synthesize a balanced recommendation.",
)
team.print_response("Should I invest in NVIDIA?")
```
### Workflow
Trace workflow invocation in Agno:
```python Python theme={null}
from netra import Netra
from agno.agent import Agent
from agno.team import Team
from agno.tools.yfinance import YFinanceTools
from agno.workflow import Step, Workflow
import os
# Initialize Netra
Netra.init(
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}",
trace_content=True
)
researcher = Agent(
model="openai:gpt-5.4",
tools=[YFinanceTools()],
instructions="Gather raw market data.",
)
bull = Agent(model="openai:gpt-5.4", role="Make the case FOR investing.")
bear = Agent(model="openai:gpt-5.4", role="Make the case AGAINST investing.")
committee = Team(
name="Investment Committee",
members=[bull, bear],
instructions="Debate the position.",
)
writer = Agent(
model="openai:gpt-5.4",
instructions="Write a 200-word investment brief.",
)
workflow = Workflow(
name="Stock Research",
steps=[
Step(name="Research", agent=researcher),
Step(name="Debate", team=committee),
Step(name="Report", agent=writer),
],
)
workflow.print_response("Analyze NVIDIA for investment.")
```
# Integrations Overview
Source: https://docs.getnetra.ai/Integrations/overview
Connect Netra with 50+ LLM providers, orchestration frameworks, vector databases, and speech APIs. Get instant observability with one-line SDK setup.
Netra provides seamless integrations with popular LLM providers, AI frameworks, vector databases, and speech services. With just a few lines of code, you can enable automatic tracing and observability for your AI applications.
**Missing an integration?** We'd love to hear from you! [Open a GitHub issue](https://github.com/KeyValueSoftwareSystems/netra-sdk-py/issues/new) to request a new integration.
***
## Integration Types
Netra integrations fall into several categories:
Direct integrations with model providers like OpenAI, Anthropic Claude, Google Gemini, and more.
Trace applications built with LangChain, LlamaIndex, CrewAI, and other orchestration frameworks.
Monitor vector search operations with Pinecone, Chroma, Qdrant, Weaviate, and others.
Track speech-to-text and text-to-speech operations with Deepgram, ElevenLabs, and Cartesia.
***
## LLM Providers
Capture traces from direct LLM API calls including prompts, completions, token usage, and costs.
### Popular Providers
GPT-4, GPT-3.5, Embeddings, DALL-E
Claude 3 Opus, Sonnet, Haiku
Gemini Pro, Gemini Ultra
Claude, Titan, Llama on AWS
Mistral Large, Medium, Small
Ultra-fast LLM inference
### All LLM Providers
| Provider | Description | Link |
| -------------------- | ---------------------------------------- | ---------------------------------------------------------------- |
| **OpenAI** | GPT-4, GPT-3.5 Turbo, Embeddings, DALL-E | [View Docs](/Integrations/ai_providers/OPENAI) |
| **Anthropic Claude** | Claude 3 Opus, Sonnet, Haiku | [View Docs](/Integrations/ai_providers/ANTHROPIC_CLAUDE) |
| **Google Gemini** | Gemini Pro, Gemini Ultra, Embeddings | [View Docs](/Integrations/ai_providers/GEMINI) |
| **AWS Bedrock** | Claude, Titan, Llama 2 via AWS | [View Docs](/Integrations/ai_providers/AWS_BEDROCK) |
| **Google Vertex AI** | PaLM, Gemini on Google Cloud | [View Docs](/Integrations/ai_providers/VERTEXAI) |
| **Mistral AI** | Mistral Large, Medium, Small, Embeddings | [View Docs](/Integrations/ai_providers/MISTRAL) |
| **Groq** | Ultra-fast inference for Llama, Mixtral | [View Docs](/Integrations/ai_providers/GROQ) |
| **Cohere** | Command, Embed, Rerank models | [View Docs](/Integrations/ai_providers/COHERE) |
| **Ollama** | Local LLM deployment | [View Docs](/Integrations/ai_providers/OLLAMA) |
| **Together AI** | Open-source model hosting | [View Docs](/Integrations/ai_providers/TOGETHERAI) |
| **Replicate** | Run ML models in the cloud | [View Docs](/Integrations/ai_providers/REPLICATE) |
| **Hugging Face** | Transformers library | [View Docs](/Integrations/ai_providers/HUGGINGFACE_TRANSFORMERS) |
| **Aleph Alpha** | Luminous models | [View Docs](/Integrations/ai_providers/ALEPH_ALPHA) |
| **IBM watsonx** | Enterprise AI platform | [View Docs](/Integrations/ai_providers/WATSONX) |
***
## AI Frameworks
Trace complex AI applications with full visibility into agent workflows, chains, and tool calls.
### Popular Frameworks
Build LLM-powered applications
Stateful, multi-actor agents
Data framework for LLM apps
Multi-agent orchestration
Type-safe AI development
Unified LLM API gateway
### All AI Frameworks
| Framework | Description | Link |
| --------------- | ------------------------------------------------- | --------------------------------------------------- |
| **LangChain** | Popular framework for LLM application development | [View Docs](/Integrations/orchestrators/LangChain) |
| **LangGraph** | Build stateful, multi-actor AI agents | [View Docs](/Integrations/orchestrators/LangGraph) |
| **LlamaIndex** | Data framework for connecting LLMs to data | [View Docs](/Integrations/orchestrators/LlamaIndex) |
| **CrewAI** | Framework for orchestrating AI agents | [View Docs](/Integrations/orchestrators/CrewAI) |
| **Pydantic AI** | Type-safe AI application development | [View Docs](/Integrations/orchestrators/PydanticAI) |
| **LiteLLM** | Unified API for 100+ LLM providers | [View Docs](/Integrations/orchestrators/LiteLLM) |
| **Haystack** | End-to-end NLP framework | [View Docs](/Integrations/orchestrators/Haystack) |
| **DSPy** | Programming with foundation models | [View Docs](/Integrations/orchestrators/DSPy) |
| **Google ADK** | Agent Development Kit | [View Docs](/Integrations/orchestrators/ADK) |
| **Cerebras** | High-performance AI inference | [View Docs](/Integrations/orchestrators/Cerebras) |
| **Groq** | Low-latency LLM inference | [View Docs](/Integrations/ai_providers/GROQ) |
| **MCP** | Model Context Protocol | [View Docs](/Integrations/orchestrators/MCP) |
***
## Vector Databases
Monitor vector search operations, embeddings, and retrieval performance.
Managed vector DB
Open-source embedding DB
Vector similarity search
AI-native vector DB
Cloud-native vector DB
Serverless vector DB
Vector search with Redis
End-to-end vector search
***
## Speech Services
Track speech-to-text and text-to-speech operations for voice AI applications.
Speech recognition & understanding
AI voice generation
Real-time voice AI
***
## Quick Start
Enable auto-instrumentation for any supported integration:
```python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
instruments={
InstrumentSet.OPENAI,
InstrumentSet.LANGCHAIN,
InstrumentSet.PINECONE,
},
)
# Your LLM calls are now automatically traced
```
If you don't specify `instruments`, Netra will auto-detect and instrument all supported libraries in your environment.
***
## Next Steps
* [Auto-Instrumentation](/Observability/Traces/auto-instrumentation) - Learn how automatic tracing works
* [Manual Tracing](/Observability/Traces/manual-tracing) - Add custom spans to your traces
* [Python SDK](/sdk-reference/sdk/python) - Complete SDK reference
# Cartesia
Source: https://docs.getnetra.ai/Integrations/stt-tts-docs/Cartesia
Trace Cartesia ultra-low latency TTS and STT with Netra auto-instrumentation. Monitor voice synthesis, transcription requests, and streaming latency.
Cartesia provides ultra-low latency text-to-speech (Sonic models) and speech-to-text (Ink models) capabilities. With first-byte latency as low as 40ms for TTS and real-time streaming for STT, Cartesia is ideal for conversational AI. Netra helps you track both TTS and STT operations, monitor performance metrics, and analyze usage patterns.
## Installation
```bash Python theme={null}
pip install cartesia netra-sdk
```
```bash Typescript theme={null}
npm install @cartesia/cartesia-js netra-sdk
```
## Usage
Initialize Netra before using Cartesia:
```python Python theme={null}
import os
from netra import Netra
Netra.init(
app_name="cartesia-service",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}"
)
```
```typescript TypeScript theme={null}
import Netra from 'netra-sdk';
await Netra.init({
appName: 'cartesia-service',
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
```
## Examples
### Text-to-Speech with Sonic Models
Track Cartesia TTS operations using Netra decorators:
```python Python theme={null}
from cartesia import Cartesia
from netra.decorators import task, workflow
import os
client = Cartesia(api_key=os.environ.get("CARTESIA_API_KEY"))
@task()
def generate_speech(text: str, voice_id: str) -> bytes:
"""Generate speech using Cartesia Sonic model."""
response = client.tts.bytes(
model_id="sonic-english",
transcript=text,
voice={
"mode": "id",
"id": voice_id
},
output_format={
"container": "raw",
"encoding": "pcm_f32le",
"sample_rate": 44100
}
)
return response
@task()
def stream_speech(text: str, voice_id: str):
"""Stream speech using Cartesia Sonic Turbo."""
stream = client.tts.stream(
model_id="sonic-turbo",
transcript=text,
voice={
"mode": "id",
"id": voice_id
},
output_format={
"container": "raw",
"encoding": "pcm_f32le",
"sample_rate": 44100
}
)
for chunk in stream:
yield chunk
@workflow()
def process_text_batch(texts: list[str], voice_id: str) -> list[bytes]:
"""Process multiple texts to speech."""
audio_buffers = []
for text in texts:
audio = generate_speech(text, voice_id)
audio_buffers.append(audio)
return audio_buffers
# Usage
audio_data = generate_speech(
"Hello, this is Cartesia Sonic speech synthesis.",
"a0e99841-438c-4a64-b679-ae501e7d6091"
)
```
```typescript TypeScript theme={null}
import Cartesia from '@cartesia/cartesia-js';
import { task, workflow } from 'netra-sdk';
const cartesia = new Cartesia({
apiKey: process.env.CARTESIA_API_KEY
});
class TTSService {
@task()
async generateSpeech(text: string, voiceId: string): Promise {
const response = await cartesia.tts.bytes({
model_id: 'sonic-english',
transcript: text,
voice: {
mode: 'id',
id: voiceId
},
output_format: {
container: 'raw',
encoding: 'pcm_f32le',
sample_rate: 44100
}
});
return Buffer.from(response);
}
@task()
async streamSpeech(text: string, voiceId: string): Promise> {
const stream = await cartesia.tts.stream({
model_id: 'sonic-turbo',
transcript: text,
voice: {
mode: 'id',
id: voiceId
},
output_format: {
container: 'raw',
encoding: 'pcm_f32le',
sample_rate: 44100
}
});
async function* generateChunks() {
for await (const chunk of stream) {
yield Buffer.from(chunk);
}
}
return generateChunks();
}
@workflow()
async processTextBatch(texts: string[], voiceId: string): Promise {
const audioBuffers: Buffer[] = [];
for (const text of texts) {
const audio = await this.generateSpeech(text, voiceId);
audioBuffers.push(audio);
}
return audioBuffers;
}
}
// Usage
const service = new TTSService();
const audio = await service.generateSpeech(
'Hello, this is Cartesia Sonic speech synthesis.',
'a0e99841-438c-4a64-b679-ae501e7d6091'
);
```
### Speech-to-Text with Ink Models
Track Cartesia STT operations using Netra decorators:
```python Python theme={null}
from cartesia import Cartesia
from netra.decorators import task
import os
client = Cartesia(api_key=os.environ.get("CARTESIA_API_KEY"))
@task()
def transcribe_audio(audio_path: str) -> str:
"""Transcribe audio using Cartesia Ink model."""
with open(audio_path, "rb") as audio_file:
audio_data = audio_file.read()
response = client.stt.transcribe(
model_id="ink-whisper",
audio=audio_data,
language="en"
)
return response.transcript
@task()
def transcribe_stream(audio_stream) -> str:
"""Transcribe streaming audio in real-time."""
full_transcript = ""
stream = client.stt.stream_transcribe(
model_id="ink-whisper",
audio=audio_stream,
language="en",
interim_results=True
)
for result in stream:
if result.is_final:
full_transcript += result.transcript + " "
return full_transcript.strip()
# Usage
transcript = transcribe_audio("./audio/sample.wav")
```
```typescript TypeScript theme={null}
import Cartesia from '@cartesia/cartesia-js';
import { task } from 'netra-sdk';
import * as fs from 'fs';
const cartesia = new Cartesia({
apiKey: process.env.CARTESIA_API_KEY
});
class STTService {
@task()
async transcribeAudio(audioPath: string): Promise {
const audioBuffer = fs.readFileSync(audioPath);
const response = await cartesia.stt.transcribe({
model_id: 'ink-whisper',
audio: audioBuffer,
language: 'en'
});
return response.transcript;
}
@task()
async transcribeStream(audioStream: ReadableStream): Promise {
let fullTranscript = '';
const stream = await cartesia.stt.streamTranscribe({
model_id: 'ink-whisper',
audio: audioStream,
language: 'en',
interim_results: true
});
for await (const result of stream) {
if (result.is_final) {
fullTranscript += result.transcript + ' ';
}
}
return fullTranscript.trim();
}
}
// Usage
const sttService = new STTService();
const transcript = await sttService.transcribeAudio('./audio/sample.wav');
```
### Manual Span Creation with Action Tracking
For detailed control over tracing with both TTS and STT:
```python Python theme={null}
from cartesia import Cartesia
from netra import SpanWrapper, ActionModel, UsageModel
import os
import time
client = Cartesia(api_key=os.environ.get("CARTESIA_API_KEY"))
def generate_speech_with_tracking(text: str, voice_id: str) -> bytes:
"""Generate speech with detailed tracking."""
span = SpanWrapper("cartesia-tts")
span.start()
try:
start_time = time.time_ns()
span.set_attribute("text_length", len(text))
span.set_attribute("voice_id", voice_id)
span.set_attribute("model", "sonic-turbo")
response = client.tts.bytes(
model_id="sonic-turbo",
transcript=text,
voice={"mode": "id", "id": voice_id},
output_format={
"container": "raw",
"encoding": "pcm_f32le",
"sample_rate": 44100
}
)
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Track the TTS API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="TTS_SYNTHESIS",
metadata={
"provider": "cartesia",
"model": "sonic-turbo",
"voice_id": voice_id,
"text_length": str(len(text)),
"audio_size_bytes": str(len(response)),
"sample_rate": "44100",
"latency_ms": str(duration_ms)
},
success=True
)
span.set_action([action])
# Track usage
usage = UsageModel(
model="sonic-turbo",
usage_type="characters",
units_used=len(text),
cost_in_usd=len(text) * 0.00001
)
span.set_usage([usage])
span.set_status({"code": 1, "message": "Success"})
span.end()
return response
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
def transcribe_with_tracking(audio_path: str) -> str:
"""Transcribe audio with detailed tracking."""
span = SpanWrapper("cartesia-stt")
span.start()
try:
start_time = time.time_ns()
with open(audio_path, "rb") as audio_file:
audio_data = audio_file.read()
audio_size_bytes = len(audio_data)
span.set_attribute("audio_file", audio_path)
span.set_attribute("audio_size_bytes", audio_size_bytes)
span.set_attribute("model", "ink-whisper")
response = client.stt.transcribe(
model_id="ink-whisper",
audio=audio_data,
language="en"
)
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Track the STT API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="STT_TRANSCRIPTION",
metadata={
"provider": "cartesia",
"model": "ink-whisper",
"audio_size_bytes": str(audio_size_bytes),
"transcript_length": str(len(response.transcript)),
"duration_ms": str(duration_ms),
"language": "en"
},
success=True
)
span.set_action([action])
# Track usage
audio_duration = getattr(response, 'duration', 0)
usage = UsageModel(
model="ink-whisper",
usage_type="audio_seconds",
units_used=audio_duration,
cost_in_usd=audio_duration * 0.036 # $0.13 per hour
)
span.set_usage([usage])
span.set_attribute("transcript_length", len(response.transcript))
span.set_status({"code": 1, "message": "Success"})
span.end()
return response.transcript
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
# Usage
audio_data = generate_speech_with_tracking(
"This is ultra-low latency speech synthesis.",
"a0e99841-438c-4a64-b679-ae501e7d6091"
)
transcript = transcribe_with_tracking("./audio/sample.wav")
```
```typescript TypeScript theme={null}
import Cartesia from '@cartesia/cartesia-js';
import { SpanWrapper, ActionModel } from 'netra-sdk';
const cartesia = new Cartesia({
apiKey: process.env.CARTESIA_API_KEY
});
async function generateSpeechWithTracking(text: string, voiceId: string): Promise {
const span = new SpanWrapper('cartesia-tts');
span.start();
try {
const startTime = Date.now();
span.setAttribute('text_length', text.length);
span.setAttribute('voice_id', voiceId);
span.setAttribute('model', 'sonic-turbo');
const response = await cartesia.tts.bytes({
model_id: 'sonic-turbo',
transcript: text,
voice: { mode: 'id', id: voiceId },
output_format: {
container: 'raw',
encoding: 'pcm_f32le',
sample_rate: 44100
}
});
const audioBuffer = Buffer.from(response);
const duration = Date.now() - startTime;
// Track the TTS API operation
const action: ActionModel = {
start_time: (startTime * 1000000).toString(),
action: 'API',
action_type: 'TTS_SYNTHESIS',
metadata: {
provider: 'cartesia',
model: 'sonic-turbo',
voice_id: voiceId,
text_length: text.length.toString(),
audio_size_bytes: audioBuffer.length.toString(),
sample_rate: '44100',
latency_ms: duration.toString()
},
success: true
};
span.setAction([action]);
span.setUsage({
model: 'sonic-turbo',
usage_type: 'characters',
units_used: text.length,
cost_in_usd: text.length * 0.00001
});
span.setStatus({ code: 1, message: 'Success' });
span.end();
return audioBuffer;
} catch (error) {
span.setError(error as Error);
span.setStatus({ code: 2, message: 'Error' });
span.end();
throw error;
}
}
async function transcribeWithTracking(audioPath: string): Promise {
const span = new SpanWrapper('cartesia-stt');
span.start();
try {
const startTime = Date.now();
const audioBuffer = fs.readFileSync(audioPath);
const audioSizeBytes = audioBuffer.length;
span.setAttribute('audio_file', audioPath);
span.setAttribute('audio_size_bytes', audioSizeBytes);
span.setAttribute('model', 'ink-whisper');
const response = await cartesia.stt.transcribe({
model_id: 'ink-whisper',
audio: audioBuffer,
language: 'en'
});
const duration = Date.now() - startTime;
// Track the STT API operation
const action: ActionModel = {
start_time: (startTime * 1000000).toString(),
action: 'API',
action_type: 'STT_TRANSCRIPTION',
metadata: {
provider: 'cartesia',
model: 'ink-whisper',
audio_size_bytes: audioSizeBytes.toString(),
transcript_length: response.transcript.length.toString(),
duration_ms: duration.toString(),
language: 'en'
},
success: true
};
span.setAction([action]);
span.setUsage({
model: 'ink-whisper',
usage_type: 'audio_seconds',
units_used: response.duration || 0,
cost_in_usd: (response.duration || 0) * 0.036
});
span.setAttribute('transcript_length', response.transcript.length);
span.setStatus({ code: 1, message: 'Success' });
span.end();
return response.transcript;
} catch (error) {
span.setError(error as Error);
span.setStatus({ code: 2, message: 'Error' });
span.end();
throw error;
}
}
// Usage
const audio = await generateSpeechWithTracking(
'This is ultra-low latency speech synthesis.',
'a0e99841-438c-4a64-b679-ae501e7d6091'
);
const transcript = await transcribeWithTracking('./audio/sample.wav');
```
## Next Steps
* [Netra Documentation](https://docs.netra.ai) - Learn more about Netra's observability features
* [Cartesia API](https://docs.cartesia.ai/) - Explore Cartesia's ultra-low latency TTS and STT
# Deepgram
Source: https://docs.getnetra.ai/Integrations/stt-tts-docs/Deepgram
Trace Deepgram speech-to-text transcription with Netra auto-instrumentation. Monitor audio requests, transcription accuracy, and latency automatically.
Deepgram provides accurate and fast speech-to-text transcription with support for real-time streaming, multiple languages, and advanced features like speaker diarization. Netra helps you track transcription operations, monitor accuracy metrics, and analyze usage patterns.
## Installation
```bash Python theme={null}
pip install deepgram-sdk netra-sdk
```
```bash Typescript theme={null}
npm install @deepgram/sdk netra-sdk
```
## Usage
Initialize Netra before using Deepgram:
```python Python theme={null}
import os
from netra import Netra
Netra.init(
app_name="deepgram-stt-service",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}"
)
```
```typescript TypeScript theme={null}
import Netra from 'netra-sdk';
await Netra.init({
appName: 'deepgram-stt-service',
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
```
## Examples
### Automatic Tracing with Decorators
Track Deepgram operations automatically using Netra decorators:
```python Python theme={null}
from deepgram import DeepgramClient, PrerecordedOptions, FileSource
from netra.decorators import task, workflow
import os
client = DeepgramClient(api_key=os.environ.get("DEEPGRAM_API_KEY"))
@task()
def transcribe_audio(audio_url: str) -> str:
"""Transcribe audio from URL using Deepgram."""
options = PrerecordedOptions(
model="nova-2",
smart_format=True,
punctuate=True,
paragraphs=True
)
response = client.listen.prerecorded.transcribe_url(
{"url": audio_url},
options
)
return response.results.channels[0].alternatives[0].transcript
@task()
def transcribe_with_diarization(audio_url: str) -> list:
"""Transcribe with speaker diarization."""
options = PrerecordedOptions(
model="nova-2",
smart_format=True,
diarize=True,
punctuate=True,
utterances=True
)
response = client.listen.prerecorded.transcribe_url(
{"url": audio_url},
options
)
return response.results.utterances
@workflow()
def process_audio_file(audio_url: str) -> dict:
"""Process audio file with full transcription and diarization."""
transcript = transcribe_audio(audio_url)
speakers = transcribe_with_diarization(audio_url)
return {
"full_transcript": transcript,
"speakers": speakers
}
# Usage
result = process_audio_file("https://example.com/audio.mp3")
```
```typescript TypeScript theme={null}
import { createClient } from '@deepgram/sdk';
import { task, workflow } from 'netra-sdk';
const deepgram = createClient(process.env.DEEPGRAM_API_KEY);
class TranscriptionService {
@task()
async transcribeAudio(audioUrl: string): Promise {
const { result } = await deepgram.listen.prerecorded.transcribeUrl(
{ url: audioUrl },
{
model: 'nova-2',
smart_format: true,
punctuate: true,
paragraphs: true
}
);
return result.results.channels[0].alternatives[0].transcript;
}
@task()
async transcribeWithDiarization(audioUrl: string): Promise {
const { result } = await deepgram.listen.prerecorded.transcribeUrl(
{ url: audioUrl },
{
model: 'nova-2',
smart_format: true,
diarize: true,
punctuate: true,
utterances: true
}
);
return result.results.utterances;
}
@workflow()
async processAudioFile(audioUrl: string): Promise {
const transcript = await this.transcribeAudio(audioUrl);
const speakers = await this.transcribeWithDiarization(audioUrl);
return {
full_transcript: transcript,
speakers: speakers
};
}
}
// Usage
const service = new TranscriptionService();
const result = await service.processAudioFile(
'https://example.com/audio.mp3'
);
```
### Manual Span Creation with Action Tracking
For detailed control over tracing and action tracking:
```python Python theme={null}
from deepgram import DeepgramClient, PrerecordedOptions, FileSource
from netra import SpanWrapper, ActionModel, UsageModel
import os
import time
client = DeepgramClient(api_key=os.environ.get("DEEPGRAM_API_KEY"))
def transcribe_with_tracking(audio_path: str) -> str:
"""Transcribe audio with detailed tracking."""
span = SpanWrapper("deepgram-transcription")
span.start()
try:
start_time = time.time_ns()
# Read audio file
with open(audio_path, "rb") as audio_file:
audio_data = audio_file.read()
audio_size_bytes = len(audio_data)
span.set_attribute("audio_file", audio_path)
span.set_attribute("audio_size_bytes", audio_size_bytes)
span.set_attribute("model", "nova-2")
# Transcribe
payload = {"buffer": audio_data}
options = PrerecordedOptions(
model="nova-2",
smart_format=True,
punctuate=True,
diarize=True
)
response = client.listen.prerecorded.transcribe_file(
payload,
options
)
transcript = response.results.channels[0].alternatives[0].transcript
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Extract metadata
audio_duration = response.metadata.duration
confidence = response.results.channels[0].alternatives[0].confidence
word_count = len(response.results.channels[0].alternatives[0].words)
# Track the STT API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="STT_TRANSCRIPTION",
metadata={
"provider": "deepgram",
"model": "nova-2",
"audio_size_bytes": str(audio_size_bytes),
"audio_duration_seconds": str(audio_duration),
"transcript_length": str(len(transcript)),
"confidence": str(confidence),
"duration_ms": str(duration_ms),
"words_detected": str(word_count)
},
success=True
)
span.set_action([action])
# Track usage
usage = UsageModel(
model="nova-2",
usage_type="audio_seconds",
units_used=audio_duration,
cost_in_usd=audio_duration * 0.0043 # $0.0043 per second
)
span.set_usage([usage])
span.set_attribute("transcript_length", len(transcript))
span.set_attribute("confidence", confidence)
span.set_status({"code": 1, "message": "Success"})
span.end()
return transcript
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
# Usage
transcript = transcribe_with_tracking("./audio/sample.mp3")
```
```typescript TypeScript theme={null}
import { createClient } from '@deepgram/sdk';
import { SpanWrapper, ActionModel } from 'netra-sdk';
import * as fs from 'fs';
const deepgram = createClient(process.env.DEEPGRAM_API_KEY);
async function transcribeWithTracking(audioPath: string): Promise {
const span = new SpanWrapper('deepgram-transcription');
span.start();
try {
const startTime = Date.now();
const audioBuffer = fs.readFileSync(audioPath);
const audioSizeBytes = audioBuffer.length;
span.setAttribute('audio_file', audioPath);
span.setAttribute('audio_size_bytes', audioSizeBytes);
span.setAttribute('model', 'nova-2');
const { result } = await deepgram.listen.prerecorded.transcribeFile(
audioBuffer,
{
model: 'nova-2',
smart_format: true,
punctuate: true,
diarize: true
}
);
const transcript = result.results.channels[0].alternatives[0].transcript;
const duration = Date.now() - startTime;
const audioDuration = result.metadata.duration;
const confidence = result.results.channels[0].alternatives[0].confidence;
// Track the STT API operation
const action: ActionModel = {
start_time: (startTime * 1000000).toString(),
action: 'API',
action_type: 'STT_TRANSCRIPTION',
metadata: {
provider: 'deepgram',
model: 'nova-2',
audio_size_bytes: audioSizeBytes.toString(),
audio_duration_seconds: audioDuration.toString(),
transcript_length: transcript.length.toString(),
confidence: confidence.toString(),
duration_ms: duration.toString(),
words_detected: result.results.channels[0].alternatives[0].words.length.toString()
},
success: true
};
span.setAction([action]);
span.setUsage({
model: 'nova-2',
usage_type: 'audio_seconds',
units_used: audioDuration,
cost_in_usd: audioDuration * 0.0043
});
span.setAttribute('transcript_length', transcript.length);
span.setAttribute('confidence', confidence);
span.setStatus({ code: 1, message: 'Success' });
span.end();
return transcript;
} catch (error) {
span.setError(error as Error);
span.setStatus({ code: 2, message: 'Error' });
span.end();
throw error;
}
}
// Usage
const transcript = await transcribeWithTracking('./audio/sample.mp3');
```
```python Python theme={null}
from deepgram import DeepgramClient, PrerecordedOptions, FileSource
from netra import SpanWrapper, ActionModel, UsageModel
import os
import time
client = DeepgramClient(api_key=os.environ.get("DEEPGRAM_API_KEY"))
def transcribe_with_tracking(audio_path: str) -> str:
"""Transcribe audio with detailed tracking."""
span = SpanWrapper("deepgram-transcription")
span.start()
try:
start_time = time.time_ns()
# Read audio file
with open(audio_path, "rb") as audio_file:
audio_data = audio_file.read()
audio_size_bytes = len(audio_data)
span.set_attribute("audio_file", audio_path)
span.set_attribute("audio_size_bytes", audio_size_bytes)
span.set_attribute("model", "nova-2")
# Transcribe
payload = {"buffer": audio_data}
options = PrerecordedOptions(
model="nova-2",
smart_format=True,
punctuate=True,
diarize=True
)
response = client.listen.prerecorded.transcribe_file(
payload,
options
)
transcript = response.results.channels[0].alternatives[0].transcript
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Extract metadata
audio_duration = response.metadata.duration
confidence = response.results.channels[0].alternatives[0].confidence
word_count = len(response.results.channels[0].alternatives[0].words)
# Track the STT API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="STT_TRANSCRIPTION",
metadata={
"provider": "deepgram",
"model": "nova-2",
"audio_size_bytes": str(audio_size_bytes),
"audio_duration_seconds": str(audio_duration),
"transcript_length": str(len(transcript)),
"confidence": str(confidence),
"duration_ms": str(duration_ms),
"words_detected": str(word_count)
},
success=True
)
span.set_action([action])
# Track usage
usage = UsageModel(
model="nova-2",
usage_type="audio_seconds",
units_used=audio_duration,
cost_in_usd=audio_duration * 0.0043 # $0.0043 per second
)
span.set_usage([usage])
span.set_attribute("transcript_length", len(transcript))
span.set_attribute("confidence", confidence)
span.set_status({"code": 1, "message": "Success"})
span.end()
return transcript
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
# Usage
transcript = transcribe_with_tracking("./audio/sample.mp3")
```
## Next Steps
* [Netra Documentation](https://docs.netra.ai) - Learn more about Netra's observability features
* [Deepgram API](https://developers.deepgram.com/) - Explore Deepgram speech-to-text capabilities
# ElevenLabs
Source: https://docs.getnetra.ai/Integrations/stt-tts-docs/Elevenlabs
Trace ElevenLabs text-to-speech synthesis with Netra auto-instrumentation. Monitor voice generation requests, audio duration, and latency automatically.
ElevenLabs provides high-quality text-to-speech synthesis with natural-sounding voices and support for multiple languages. Netra helps you track voice synthesis operations, monitor audio generation metrics, and analyze usage patterns.
## Installation
```bash Python theme={null}
pip install elevenlabs netra-sdk
```
```bash Typescript theme={null}
npm install elevenlabs netra-sdk
```
## Usage
Initialize Netra before using ElevenLabs:
```python Python theme={null}
import os
from netra import Netra
Netra.init(
app_name="elevenlabs-tts-service",
headers=f"x-api-key={os.environ.get('NETRA_API_KEY')}"
)
```
```typescript TypeScript theme={null}
import Netra from 'netra-sdk';
await Netra.init({
appName: 'elevenlabs-tts-service',
headers: `x-api-key=${process.env.NETRA_API_KEY}`
});
```
## Examples
### Automatic Tracing with Decorators
Track ElevenLabs operations automatically using Netra decorators:
```python Python theme={null}
from elevenlabs import ElevenLabs
from netra.decorators import task
import os
client = ElevenLabs(api_key=os.environ.get("ELEVENLABS_API_KEY"))
@task()
def synthesize_speech(text: str, voice_id: str) -> bytes:
"""Generate speech from text using ElevenLabs."""
audio = client.text_to_speech.convert(
voice_id=voice_id,
text=text,
model_id="eleven_turbo_v2",
voice_settings={
"stability": 0.5,
"similarity_boost": 0.75
}
)
# Collect audio chunks
audio_bytes = b""
for chunk in audio:
audio_bytes += chunk
return audio_bytes
@task()
def synthesize_with_emotions(text: str, voice_id: str, emotion: str) -> bytes:
"""Generate emotional speech using advanced models."""
audio = client.text_to_speech.convert(
voice_id=voice_id,
text=text,
model_id="eleven_turbo_v2_5",
voice_settings={
"stability": 0.6,
"similarity_boost": 0.8,
"style": 0.5,
"use_speaker_boost": True
}
)
audio_bytes = b""
for chunk in audio:
audio_bytes += chunk
return audio_bytes
# Usage
audio_data = synthesize_speech(
"Hello, this is a test of ElevenLabs speech synthesis.",
"pNInz6obpgDQGcFmaJgB"
)
```
```typescript TypeScript theme={null}
import { ElevenLabsClient } from 'elevenlabs';
import { task } from 'netra-sdk';
const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });
class TTSService {
@task()
async synthesizeSpeech(text: string, voiceId: string): Promise {
const audio = await client.textToSpeech.convert(voiceId, {
text,
model_id: 'eleven_turbo_v2',
voice_settings: {
stability: 0.5,
similarity_boost: 0.75
}
});
const chunks: Buffer[] = [];
for await (const chunk of audio) {
chunks.push(chunk);
}
return Buffer.concat(chunks);
}
@task()
async synthesizeWithEmotions(
text: string,
voiceId: string,
emotion: string
): Promise {
const audio = await client.textToSpeech.convert(voiceId, {
text,
model_id: 'eleven_turbo_v2_5',
voice_settings: {
stability: 0.6,
similarity_boost: 0.8,
style: 0.5,
use_speaker_boost: true
}
});
const chunks: Buffer[] = [];
for await (const chunk of audio) {
chunks.push(chunk);
}
return Buffer.concat(chunks);
}
}
// Usage
const service = new TTSService();
await service.synthesizeSpeech(
'Hello, this is a test of ElevenLabs speech synthesis.',
'pNInz6obpgDQGcFmaJgB'
);
```
### Manual Span Creation with Action Tracking
For detailed control over tracing and action tracking:
```python Python theme={null}
from elevenlabs import ElevenLabs
from netra import SpanWrapper, ActionModel, UsageModel
import os
import time
client = ElevenLabs(api_key=os.environ.get("ELEVENLABS_API_KEY"))
def synthesize_with_tracking(text: str, voice_id: str) -> bytes:
"""Generate speech with detailed tracking."""
span = SpanWrapper("elevenlabs-synthesis")
span.start()
try:
start_time = time.time_ns()
span.set_attribute("voice_id", voice_id)
span.set_attribute("text_length", len(text))
span.set_attribute("model", "eleven_turbo_v2")
audio = client.text_to_speech.convert(
voice_id=voice_id,
text=text,
model_id="eleven_turbo_v2",
voice_settings={
"stability": 0.5,
"similarity_boost": 0.75
}
)
# Collect audio chunks
audio_bytes = b""
for chunk in audio:
audio_bytes += chunk
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Track the TTS API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="TTS_SYNTHESIS",
metadata={
"provider": "elevenlabs",
"voice_id": voice_id,
"model": "eleven_turbo_v2",
"text_length": str(len(text)),
"audio_size_bytes": str(len(audio_bytes)),
"duration_ms": str(duration_ms)
},
success=True
)
span.set_action([action])
# Track usage
usage = UsageModel(
model="eleven_turbo_v2",
usage_type="characters",
units_used=len(text),
cost_in_usd=len(text) * 0.00003 # $0.30 per 1000 characters
)
span.set_usage([usage])
span.set_status({"code": 1, "message": "Success"})
span.end()
return audio_bytes
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
# Usage
audio_data = synthesize_with_tracking(
"This is tracked speech synthesis with detailed metrics.",
"pNInz6obpgDQGcFmaJgB"
)
```
```typescript TypeScript theme={null}
import { ElevenLabsClient } from 'elevenlabs';
import { SpanWrapper, ActionModel } from 'netra-sdk';
const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });
async function synthesizeWithTracking(text: string, voiceId: string): Promise {
const span = new SpanWrapper('elevenlabs-synthesis');
span.start();
try {
const startTime = Date.now();
span.setAttribute('voice_id', voiceId);
span.setAttribute('text_length', text.length);
span.setAttribute('model', 'eleven_turbo_v2');
const audio = await client.textToSpeech.convert(voiceId, {
text,
model_id: 'eleven_turbo_v2',
voice_settings: { stability: 0.5, similarity_boost: 0.75 }
});
const chunks: Buffer[] = [];
for await (const chunk of audio) {
chunks.push(chunk);
}
const audioBuffer = Buffer.concat(chunks);
const duration = Date.now() - startTime;
// Track the TTS API operation
const action: ActionModel = {
start_time: (startTime * 1000000).toString(),
action: 'API',
action_type: 'TTS_SYNTHESIS',
metadata: {
provider: 'elevenlabs',
voice_id: voiceId,
model: 'eleven_turbo_v2',
text_length: text.length.toString(),
audio_size_bytes: audioBuffer.length.toString(),
duration_ms: duration.toString()
},
success: true
};
span.setAction([action]);
span.setUsage({
model: 'eleven_turbo_v2',
usage_type: 'characters',
units_used: text.length,
cost_in_usd: text.length * 0.00003
});
span.setStatus({ code: 1, message: 'Success' });
span.end();
return audioBuffer;
} catch (error) {
span.setError(error as Error);
span.setStatus({ code: 2, message: 'Error' });
span.end();
throw error;
}
}
// Usage
await synthesizeWithTracking(
'This is tracked speech synthesis with detailed metrics.',
'pNInz6obpgDQGcFmaJgB'
);
```
```python Python theme={null}
from elevenlabs import ElevenLabs
from netra import SpanWrapper, ActionModel, UsageModel
import os
import time
client = ElevenLabs(api_key=os.environ.get("ELEVENLABS_API_KEY"))
def synthesize_with_tracking(text: str, voice_id: str) -> bytes:
"""Generate speech with detailed tracking."""
span = SpanWrapper("elevenlabs-synthesis")
span.start()
try:
start_time = time.time_ns()
span.set_attribute("voice_id", voice_id)
span.set_attribute("text_length", len(text))
span.set_attribute("model", "eleven_turbo_v2")
audio = client.text_to_speech.convert(
voice_id=voice_id,
text=text,
model_id="eleven_turbo_v2",
voice_settings={
"stability": 0.5,
"similarity_boost": 0.75
}
)
# Collect audio chunks
audio_bytes = b""
for chunk in audio:
audio_bytes += chunk
end_time = time.time_ns()
duration_ms = (end_time - start_time) / 1_000_000
# Track the TTS API operation
action = ActionModel(
start_time=str(start_time),
action="API",
action_type="TTS_SYNTHESIS",
metadata={
"provider": "elevenlabs",
"voice_id": voice_id,
"model": "eleven_turbo_v2",
"text_length": str(len(text)),
"audio_size_bytes": str(len(audio_bytes)),
"duration_ms": str(duration_ms)
},
success=True
)
span.set_action([action])
# Track usage
usage = UsageModel(
model="eleven_turbo_v2",
usage_type="characters",
units_used=len(text),
cost_in_usd=len(text) * 0.00003 # $0.30 per 1000 characters
)
span.set_usage([usage])
span.set_status({"code": 1, "message": "Success"})
span.end()
return audio_bytes
except Exception as e:
span.set_error(e)
span.set_status({"code": 2, "message": "Error"})
span.end()
raise
# Usage
audio_data = synthesize_with_tracking(
"This is tracked speech synthesis with detailed metrics.",
"pNInz6obpgDQGcFmaJgB"
)
```
## Next Steps
* [Netra Documentation](https://docs.netra.ai) - Learn more about Netra's observability features
* [ElevenLabs API](https://elevenlabs.io/docs/api-reference/text-to-speech) - Explore ElevenLabs text-to-speech capabilities
# Helicone to Netra
Source: https://docs.getnetra.ai/Migrations/Helicone
Migrate from Helicone to Netra for AI observability. A step-by-step guide covering prompt templates, variables, versioning, and tracing integration.
Helicone has [moved into maintenance mode](https://www.helicone.ai/blog/joining-mintlify) after its acquisition by Mintlify. This page walks through how to transition both **prompt management** and **observability** from Helicone to Netra.
## Migrating Tracing / Observability
Helicone captures LLM traffic at the gateway. Netra adds **hierarchical traces** with nested spans, so you can follow complete multi-step agent workflows, not just individual model calls.
### Option A: Use the Netra SDK (Recommended)
Netra ships official SDKs for [Python](https://docs.getnetra.ai/sdk-reference/sdk/python) and [TypeScript](https://docs.getnetra.ai/sdk-reference/sdk/typescript) to instrument application code directly, plus native [integrations across 50+ frameworks and model providers](https://docs.getnetra.ai/Integrations/overview) including OpenAI, LangChain, LlamaIndex, Anthropic, and more. See the full [integrations overview](https://docs.getnetra.ai/Integrations/overview) to pick the integration that matches your stack.
If you’re using OpenAI, the simplest migration is to keep your existing OpenAI SDK calls. Since Helicone is OpenAI-compatible, you can even keep Helicone as a gateway during the transition:
```python Python theme={null}
from netra import Netra
# Initialize before importing other libraries for best results
Netra.init(app_name="my-ai-app", environment="production")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize before importing other libraries for best results
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
Then keep the rest of your OpenAI code the same:
```python Python theme={null}
# Your existing code works unchanged
from openai import OpenAI
client = OpenAI(
base_url="", # Keep helicone's url during transition
api_key=""
)
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
```
```typescript TypeScript theme={null}
// Your existing code works unchanged
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "", // Keep helicone's url during transition
apiKey: "",
});
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
```
Beyond gateway logging, Netra also supports application-level tracing via decorators in [Python](https://docs.getnetra.ai/sdk-reference/sdk/python#decorators) and equivalent patterns in [TypeScript](https://docs.getnetra.ai/sdk-reference/sdk/typescript#decorators). This creates hierarchical traces with nested spans for multi-step workflows (tool calls, retrieval, post-processing, and more), giving you full execution context that a gateway alone can’t provide.
### Option B: Use OpenTelemetry
If you already have OpenTelemetry in place, you don’t need to re-instrument. Point your OTLP exporter at Netra and keep your existing spans and context propagation. Configure your exporter using Netra's OTLP endpoint:
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key="
```
### Option C: Replace the Gateway with LiteLLM Proxy
If you relied on Helicone primarily as an AI gateway (provider routing, fallbacks, or operational controls), you can replace it with [LiteLLM Proxy](https://docs.litellm.ai/docs/simple_proxy) and continue sending traces to Netra:
```python Python theme={null}
from netra import Netra
# Initialize before importing other libraries for best results
Netra.init(app_name="my-ai-app", environment="production")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize before importing other libraries for best results
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
Then keep your OpenAI client code the same—just point it at your LiteLLM proxy:
```python Python theme={null}
# Your existing code works unchanged
from openai import OpenAI
client = OpenAI(
base_url="", # Add your litellm url
api_key=""
)
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
```
```typescript TypeScript theme={null}
// Your existing code works unchanged
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "", // Add your litellm url
apiKey: "",
});
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
```
This preserves the gateway pattern while routing your tracing data into Netra.
## What's next?
Once your tracing is set up, you're ready to explore the rest of Netra. Pick what matters most to your team:
Test your agents with realistic, multi-turn conversations using configurable personas and goals.
Measure quality, accuracy, and reliability with LLM-as-Judge and code evaluators.
Build your own views to track traces, costs, and agent behavior across projects.
Get notified about anomalies, cost spikes, and performance issues.
***
# Portkey to Netra
Source: https://docs.getnetra.ai/Migrations/Portkey
Migrate from Portkey to Netra for AI observability. A step-by-step guide covering SDK integration, tracing, OpenTelemetry, and gateway feature mapping.
If you're looking to move away from Portkey — whether due to reliability concerns, pricing, or simply wanting deeper application-level observability — this guide walks through how to transition your **tracing**, **observability**, and **gateway features** from Portkey to Netra.
## Migrating Tracing / Observability
Portkey captures LLM traffic at its gateway and groups related requests (retries, fallbacks) into traces using a `trace ID` header. Netra takes a different approach: it instruments your application code directly, producing **hierarchical traces** with nested spans that cover complete multi-step agent workflows — not just the gateway hops.
### Option A: Use the Netra SDK (Recommended)
Netra ships official SDKs for [Python](https://docs.getnetra.ai/sdk-reference/sdk/python) and [TypeScript](https://docs.getnetra.ai/sdk-reference/sdk/typescript) to instrument application code directly, plus native [integrations across 50+ frameworks and model providers](https://docs.getnetra.ai/Integrations/overview) including OpenAI, LangChain, LlamaIndex, Anthropic, and more. See the full [integrations overview](https://docs.getnetra.ai/Integrations/overview) to pick the integration that matches your stack.
If you're using the Portkey SDK (`portkey-ai`), the migration starts by initializing Netra before your LLM calls. You can keep Portkey's gateway running during the transition so your existing routing and fallback logic stays intact while Netra begins capturing traces:
```python Python theme={null}
from netra import Netra
# Initialize before importing other libraries for best results
Netra.init(app_name="my-ai-app", environment="production")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize before importing other libraries for best results
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
Then switch from the Portkey SDK to the standard OpenAI SDK. You can keep Portkey's gateway URL during the transition:
```python Python theme={null}
# Before (Portkey SDK)
# from portkey_ai import Portkey
# client = Portkey(api_key="PORTKEY_API_KEY", virtual_key="VIRTUAL_KEY")
# After (OpenAI SDK + Netra instrumentation)
from openai import OpenAI
client = OpenAI(
base_url="https://api.portkey.ai/v1", # Keep Portkey gateway during transition
api_key="",
default_headers={"x-portkey-api-key": ""}
)
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
```
```typescript TypeScript theme={null}
// Before (Portkey SDK)
// import Portkey from "portkey-ai";
// const client = new Portkey({ apiKey: "PORTKEY_API_KEY", virtualKey: "VIRTUAL_KEY" });
// After (OpenAI SDK + Netra instrumentation)
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.portkey.ai/v1", // Keep Portkey gateway during transition
apiKey: "",
defaultHeaders: { "x-portkey-api-key": "" },
});
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
```
Once you've verified traces are flowing into Netra, remove the Portkey gateway URL and point directly at your LLM provider:
```python Python theme={null}
from openai import OpenAI
client = OpenAI(api_key="")
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
```
```typescript TypeScript theme={null}
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "",
});
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
```
Beyond gateway logging, Netra also supports application-level tracing via decorators in [Python](https://docs.getnetra.ai/sdk-reference/sdk/python#decorators) and equivalent patterns in [TypeScript](https://docs.getnetra.ai/sdk-reference/sdk/typescript#decorators). This creates hierarchical traces with nested spans for multi-step workflows (tool calls, retrieval, post-processing, and more), giving you full execution context that a gateway alone can't provide.
### Option B: Use OpenTelemetry
If you already have OpenTelemetry in place — or if you were using Portkey's [OpenTelemetry integration](https://portkey.ai/docs/product/observability/opentelemetry) — you don't need to re-instrument. Point your OTLP exporter at Netra and keep your existing spans and context propagation. Configure your exporter using Netra's OTLP endpoint:
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key="
```
If you were passing W3C Trace Context headers (`traceparent`, `baggage`) through Portkey, those same headers work with Netra's OpenTelemetry backend.
### Option C: Replace the Gateway with LiteLLM Proxy
If you relied on Portkey primarily as an AI gateway — provider routing, fallbacks, load balancing, retries, or caching — you can replace it with [LiteLLM Proxy](https://docs.litellm.ai/docs/simple_proxy) and continue sending traces to Netra. LiteLLM supports the same gateway capabilities: fallback chains, load balancing, retries, and caching across 100+ LLM providers.
```python Python theme={null}
from netra import Netra
# Initialize before importing other libraries for best results
Netra.init(app_name="my-ai-app", environment="production")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize before importing other libraries for best results
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
Then keep your OpenAI client code the same — just point it at your LiteLLM proxy instead of Portkey:
```python Python theme={null}
# Your existing code works unchanged
from openai import OpenAI
client = OpenAI(
base_url="", # Replace Portkey gateway URL with LiteLLM
api_key=""
)
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
```
```typescript TypeScript theme={null}
// Your existing code works unchanged
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "", // Replace Portkey gateway URL with LiteLLM
apiKey: "",
});
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
```
This preserves the gateway pattern while routing your tracing data into Netra.
## Migrating Portkey-Specific Features
Portkey bundles several gateway-level features beyond basic tracing. Here's how each maps to Netra or an alternative:
### Custom Metadata and Tags
Portkey lets you attach custom metadata to requests via the `x-portkey-metadata` header or SDK parameters for filtering and grouping in the dashboard. In Netra, you achieve the same using **session attributes** and **decorators**:
```python Python theme={null}
# Before (Portkey custom metadata)
# response = portkey.chat.completions.create(
# model="gpt-4",
# messages=[...],
# metadata={"user_id": "u-123", "environment": "production", "team": "support"}
# )
# After (Netra session attributes and decorators)
from netra import Netra
Netra.init(app_name="my-ai-app", environment="production")
# Use sessions to group related traces with custom attributes
session = Netra.create_session(
session_id="session-abc",
user_id="u-123",
metadata={"team": "support"}
)
```
```typescript TypeScript theme={null}
// Before (Portkey custom metadata)
// const response = await portkey.chat.completions.create({
// model: "gpt-4",
// messages: [...],
// metadata: { user_id: "u-123", environment: "production", team: "support" }
// });
// After (Netra session attributes and decorators)
import { Netra } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
See the [manual tracing guide](/Observability/Traces/manual-tracing) and [decorators guide](https://docs.getnetra.ai/Observability/Traces/decorators) for details.
### Caching, Routing, and Fallbacks
Portkey's gateway offers simple and semantic caching, provider fallbacks, load balancing, retries, and conditional routing via Config objects. Netra focuses on observability rather than gateway routing, so for these capabilities:
* **[LiteLLM Proxy](https://docs.litellm.ai/docs/simple_proxy)**: Supports fallbacks, load balancing, retries, and caching across 100+ providers. See [Option C](#option-c-replace-the-gateway-with-litellm-proxy) above.
You can run LiteLLM Proxy alongside Netra to get both routing and observability.
### Virtual Keys and Model Catalog
Portkey's Virtual Keys (now Model Catalog) store provider API keys server-side so your application only needs a Portkey API key. When moving to Netra, you'll manage provider keys directly:
* Store provider API keys (OpenAI, Anthropic, etc.) in your environment variables or secrets manager.
* If using LiteLLM Proxy, configure provider keys in the [LiteLLM config](https://docs.litellm.ai/docs/proxy/configs).
* Netra does not require or store your LLM provider credentials — it only needs its own API key for sending telemetry data.
## What's next?
Once your tracing is set up, you're ready to explore the rest of Netra. Pick what matters most to your team:
Test your agents with realistic, multi-turn conversations using configurable personas and goals.
Measure quality, accuracy, and reliability with LLM-as-Judge and code evaluators.
Build your own views to track traces, costs, and agent behavior across projects.
Get notified about anomalies, cost spikes, and performance issues.
***
# Agents
Source: https://docs.getnetra.ai/Observability/Agents
Track AI agents in Netra. Associate traces with agent IDs to analyze per-agent cost, latency, and decision-making across your application.
The Agents view in Netra provides a centralized dashboard of all AI agents used in your project. It shows when each agent was first and last active, how many traces it has generated, and which tools it uses.
## Viewing Agents
Access the Agents view from **Observability → Agents** in the Netra dashboard.
For each agent, Netra displays:
| Field | Description |
| ---------------- | ----------------------------------------------------------- |
| **Agent Name** | The name assigned via the `@agent` decorator or manual span |
| **Total Traces** | Number of traces where this agent was invoked |
| **First Event** | Timestamp of the agent's first recorded invocation |
| **Last Event** | Timestamp of the agent's most recent activity |
### Agent Actions
Each agent card provides two actions:
* **View Traces** - Opens the Traces view filtered to this agent, showing all executions, inputs, outputs, latency, and cost
* **Tools** - Opens a modal listing tools invoked by the agent and their call counts
## Configuring Agents
Agents are automatically tracked when you use the `@agent` decorator or create spans with `SpanType.AGENT`.
### Using the Agent Decorator
The `@agent` decorator is the recommended way to define agents. It automatically creates spans with the correct type and tracks agent activity.
```python Python theme={null}
from netra.decorators import agent
# Basic usage - agent name derived from function name
@agent
def research_agent(query: str):
# Agent logic here
return perform_research(query)
# With custom name
@agent(name="customer-support-agent")
def handle_support_request(request: dict):
intent = classify_intent(request)
response = generate_response(intent)
return response
```
```typescript TypeScript theme={null}
import { agent } from "netra-sdk/decorators";
// Basic usage - agent name derived from function name
@agent
async function researchAgent(query: string) {
// Agent logic here
return await performResearch(query);
}
// With custom name
@agent({ name: "customer-support-agent" })
async function handleSupportRequest(request: SupportRequest) {
const intent = await classifyIntent(request);
const response = await generateResponse(intent);
return response;
}
```
### Decorating Agent Classes
You can also decorate entire classes. All public methods will be instrumented as part of the agent.
```python Python theme={null}
from netra.decorators import agent, task
@agent(name="data-analysis-agent")
class DataAnalyzer:
@task
def analyze_data(self, data: list):
# Analysis logic
pass
@task
def generate_report(self, analysis: dict):
# Report generation
pass
```
```typescript TypeScript theme={null}
import { agent, task } from "netra-sdk/decorators";
@agent({ name: "data-analysis-agent" })
class DataAnalyzer {
@task
async analyzeData(data: any[]) {
// Analysis logic
}
@task
async generateReport(analysis: any) {
// Report generation
}
}
```
### Manual Agent Spans
For more control, create agent spans manually:
```python Python theme={null}
from netra import Netra, SpanType
def run_agent(input: str):
with Netra.start_span("planning-agent", as_type=SpanType.AGENT) as span:
span.set_attribute("agent.input", input)
result = execute_agent_logic(input)
span.set_attribute("agent.output", result)
span.set_success()
return result
```
```typescript TypeScript theme={null}
import { Netra, SpanType } from "netra-sdk";
async function runAgent(input: string) {
const span = Netra.startSpan("planning-agent", {}, undefined, SpanType.AGENT);
span.setAttribute("agent.input", input);
try {
const result = await executeAgentLogic(input);
span.setAttribute("agent.output", result);
span.setSuccess();
return result;
} catch (error) {
span.setError(error.message);
throw error;
} finally {
span.end();
}
}
```
## Agent with Tools
Track which tools your agent uses by nesting `@task` or `SpanType.TOOL` spans within agent spans:
```python Python theme={null}
from netra.decorators import agent, task
@agent(name="research-agent")
class ResearchAgent:
@task(name="web-search")
def search_web(self, query: str):
return web_search_api.search(query)
@task(name="summarize-results")
def summarize(self, results: list):
return llm.summarize(results)
def research(self, topic: str):
results = self.search_web(topic)
return self.summarize(results)
```
```typescript TypeScript theme={null}
import { agent, task } from "netra-sdk/decorators";
@agent({ name: "research-agent" })
class ResearchAgent {
@task({ name: "web-search" })
async searchWeb(query: string) {
return await webSearchAPI.search(query);
}
@task({ name: "summarize-results" })
async summarize(results: any[]) {
return await llm.summarize(results);
}
async research(topic: string) {
const results = await this.searchWeb(topic);
return await this.summarize(results);
}
}
```
The Tools modal in the Agents view will show call counts for `web-search` and `summarize-results`.
## Best Practices
1. **Use descriptive agent names** - Names like `customer-support-agent` or `code-review-agent` are more useful than `agent1`
2. **Nest tools within agents** - Use `@task` for tools to track which tools each agent uses
3. **Add custom attributes** - Include relevant context like input queries, user intent, or decision outcomes
4. **Use classes for stateful agents** - The class decorator tracks all methods as part of the agent
## Next Steps
* [Decorators](/Observability/Traces/decorators) - Full decorator reference
* [Manual Tracing](/Observability/Traces/manual-tracing) - Create custom spans
* [Users](/Observability/Users) - Track users interacting with your agents
* [Sessions](/Observability/Session) - Group agent interactions by session
# Sessions
Source: https://docs.getnetra.ai/Observability/Session
Group and analyze related AI interactions with Netra session tracking. Monitor multi-turn conversations, costs, and user behavior in one session view.
The Sessions view in the Netra dashboard provides a centralized place to monitor user interactions, track custom events, and analyze session-level performance metrics. This section explains how to set up session tracking in your code and how to view and analyze sessions in the dashboard.
## Setting Up Session Tracking
Before viewing sessions in the dashboard, you need to set up session tracking in your application code.
### Initialize the Netra SDK
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
# Initialize with default settings
Netra.init(app_name="Your application name")
# Or with custom configuration
api_key = "Your API key"
headers = f"x-api-key={api_key}"
Netra.init(
app_name="Your application name",
headers=headers,
trace_content=True,
environment="Your Application environment",
instruments={InstrumentSet.OPENAI}
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Initialize with default settings
await Netra.init({ appName: "Your application name" });
// Or with custom configuration
const apiKey = process.env.NETRA_API_KEY;
const headers = `x-api-key=${apiKey}`;
await Netra.init({
appName: "Your application name",
headers: headers,
traceContent: true,
environment: "Your Application environment",
instruments: new Set([NetraInstruments.OPENAI]),
});
```
### Set Session Identifiers
Track specific users, sessions, and tenants with unique identifiers:
```python Python theme={null}
# Set session identification
Netra.set_session_id("unique-session-id")
Netra.set_user_id("user-123")
Netra.set_tenant_id("tenant-456")
```
```typescript TypeScript theme={null}
// Set session identification
Netra.setSessionId("unique-session-id");
Netra.setUserId("user-123");
Netra.setTenantId("tenant-456");
```
### Add Custom Context Attributes
Enrich your session data with custom attributes for better analysis:
```python Python theme={null}
# Add custom context attributes
Netra.set_custom_attributes(key="customer_tier", value="premium")
Netra.set_custom_attributes(key="region", value="us-east")
Netra.set_custom_attributes(key="plan_type", value="enterprise")
```
```typescript TypeScript theme={null}
// Add custom context attributes
Netra.setCustomAttributes({ key: "customer_tier", value: "premium" });
Netra.setCustomAttributes({ key: "region", value: "us-east" });
Netra.setCustomAttributes({ key: "plan_type", value: "enterprise" });
```
### Record Custom Events
Track important business and technical events:
```python Python theme={null}
# Record user feedback event
Netra.set_custom_event(event_name="user_feedback", attributes={
"rating": 5,
"comment": "Great response!",
"timestamp": "2024-01-15T10:30:00Z",
"category": "positive"
})
# Record conversion event
Netra.set_custom_event(event_name="conversion", attributes={
"type": "subscription",
"plan": "premium",
"value": 99.99,
"currency": "USD",
"source": "website",
"campaign": "spring_sale"
})
# Record error event
Netra.set_custom_event(event_name="error_occurred", attributes={
"error_type": "authentication",
"error_message": "Invalid credentials",
"severity": "high",
"user_action": "login"
})
```
```typescript TypeScript theme={null}
// Record user feedback event
Netra.setCustomEvent({
event_name: "user_feedback",
attributes: {
rating: 5,
comment: "Great response!",
timestamp: "2024-01-15T10:30:00Z",
category: "positive",
},
});
// Record conversion event
Netra.setCustomEvent({
event_name: "conversion",
attributes: {
type: "subscription",
plan: "premium",
value: 99.99,
currency: "USD",
source: "website",
campaign: "spring_sale",
},
});
// Record error event
Netra.setCustomEvent({
event_name: "error_occurred",
attributes: {
error_type: "authentication",
error_message: "Invalid credentials",
severity: "high",
user_action: "login",
},
});
```
## Viewing Sessions in the Dashboard
The Sessions dashboard displays all your application sessions with their associated traces. Each session shows key information including session ID, creation time, duration, number of traces, and total cost.
### Features
* **Filter & Search:** Filter sessions by date, time range, or search by session/user ID
* **Refresh Data:** Manually refresh to see the latest sessions
* **View Columns:** SessionId, CreatedAt, Duration, Traces, Cost, and Actions button
Click the **Actions** button to navigate to the trace view and inspect all traces associated with that session.
## Use Cases
* **Debugging User Issues:** Retrieve a specific user's session to review all their interactions and traces
* **Cost Analysis:** Identify expensive sessions by viewing total cost per session
* **Performance Monitoring:** Track session duration and number of traces to monitor application behavior
* **Error Investigation:** Find sessions with failed traces to diagnose issues quickly
* **User Analytics:** Analyze session patterns to understand user behavior and engagement
## Next Steps
* [Users](/Observability/Users) - Track individual users across sessions
* [Tenants](/Observability/Tenants) - Organize sessions by tenant for multi-tenant applications
* [Traces Overview](/Observability/Traces/overview) - Deep dive into trace data within sessions
# Tenants
Source: https://docs.getnetra.ai/Observability/Tenants
Monitor LLM usage and costs per customer with Netra tenant observability. Isolate traces, track spending, and enforce SLAs across your customer base.
If you're building AI-powered products for multiple customers, Netra's tenant tracking gives you complete visibility into how each customer uses your platform. This is a unique capability that lets multi-tenant partners monitor, debug, and optimize AI interactions on a per-customer basis.
## Why Tenant Tracking Matters
For platforms serving multiple customers, understanding per-tenant behavior is critical:
* **Usage Attribution**: Know exactly which customer is driving costs, token consumption, and API calls
* **Customer Support**: When a customer reports an issue, instantly access their specific traces without sifting through logs
* **SLA Monitoring**: Track performance metrics per tenant to ensure service level agreements are met
* **Cost Allocation**: Generate accurate billing reports based on actual LLM usage per customer
* **Anomaly Detection**: Identify unusual patterns in specific tenant activity before they become problems
## Setting Up Tenant Tracking
Associate traces with tenants by calling `set_tenant_id` in your application code:
```python Python theme={null}
from netra import Netra
# Set tenant context - all subsequent traces will be associated with this tenant
Netra.set_tenant_id("acme-corp")
# Your LLM calls are now attributed to "acme-corp"
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}]
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Set tenant context - all subsequent traces will be associated with this tenant
Netra.setTenantId("acme-corp");
// Your LLM calls are now attributed to "acme-corp"
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }]
});
```
Set the tenant ID early in your request lifecycle, typically in middleware or at the start of request handling, to ensure all traces within that request are properly attributed.
## View Tenants List
The **Tenants** view provides a centralized dashboard of all customers whose activity you monitor.
Navigate to **Observability → Tenants** from the left navigation menu.
### Available Controls
| Control | Description |
| ---------------- | ---------------------------------------------------------------------- |
| **Search** | Locate tenants by name using the search bar |
| **Time Range** | Filter tenant-level metrics based on a predefined or custom date range |
| **Sort by Cost** | Quickly identify high-usage or high-spend customers |
### Tenant List Columns
| Column | Description |
| ------------ | --------------------------------------------------------------------------------- |
| **Tenant** | The unique identifier of the end customer |
| **Sessions** | Total number of sessions generated for this tenant within the selected time range |
| **Traces** | Total number of traces recorded for this tenant |
| **Cost** | Aggregated token cost attributed to this tenant |
| **Actions** | View Traces button to drill down into tenant-specific data |
Clicking **View Traces** navigates to the [Traces view](/Observability/Traces/overview), automatically filtered to display only the traces associated with that specific tenant.
## Use Cases
### Customer Support
When a customer reports an issue:
1. Search for their tenant ID in the Tenants view
2. Click **View Traces** to see all their recent activity
3. Identify the problematic trace and investigate the root cause
### Usage-Based Billing
Generate accurate invoices by:
1. Filtering the Tenants view by your billing period
2. Exporting cost data per tenant
3. Attributing LLM costs directly to each customer
### Performance Benchmarking
Compare performance across your customer base:
1. Sort tenants by trace count or cost
2. Identify outliers in latency or error rates
3. Proactively reach out to customers experiencing issues
## Related
* [Users](/Observability/Users) - Track individual user activity within tenants
* [Sessions](/Observability/Session) - Group related interactions into sessions
* [Traces Overview](/Observability/Traces/overview) - Deep dive into individual trace analysis
# Auto Instrumentation
Source: https://docs.getnetra.ai/Observability/Traces/auto-instrumentation
Automatically capture traces from OpenAI, LangChain, Pinecone, and 30+ providers with Netra auto-instrumentation. Zero code changes required.
Auto-instrumentation is the easiest way to start tracing your AI applications. When you call `Netra.init()`, the SDK automatically instruments a **curated default set** of AI-relevant libraries in your application—covering LLM providers, agent frameworks, vector databases, and key web frameworks—capturing traces without requiring any code changes.
Netra does **not** instrument every library in your environment by default. Only a curated set of AI-relevant libraries (`DEFAULT_INSTRUMENTS`) is enabled out of the box. See [Selective Instrumentation](#selective-instrumentation) for details.
## How It Works
Netra uses monkey-patching to intercept calls to supported libraries at runtime. When your application makes a call to an LLM provider, vector database, or HTTP client, Netra automatically:
1. Creates a span for the operation
2. Captures input parameters (prompts, queries, etc.)
3. Records output data (completions, results, etc.)
4. Measures latency and performance metrics
5. Tracks token usage and costs (for LLM calls)
All of this happens transparently without modifying your existing code.
## Quick Start
Initialize Netra at the start of your application to enable auto-instrumentation:
```python Python theme={null}
from netra import Netra
# Initialize before importing other libraries for best results
Netra.init(app_name="my-ai-app", environment="production")
# Your existing code works unchanged
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
)
# This call is automatically traced
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize before importing other libraries for best results
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
// Your existing code works unchanged
import OpenAI from "openai";
const client = new OpenAI();
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello!" }],
});
// This call is automatically traced
```
For best results, initialize Netra before importing the libraries you want to instrument. This ensures all calls are captured from the start.
## What Gets Captured
For each instrumented call, Netra captures relevant attributes based on the operation type:
### LLM Calls
| Attribute | Description |
| -------------------------------- | -------------------------------------------------- |
| `gen_ai.system` | The LLM provider (e.g., "openai", "anthropic") |
| `gen_ai.request.model` | Model name used for the request |
| `gen_ai.prompt` | Input prompt (if `trace_content` is enabled) |
| `gen_ai.completion` | Generated response (if `trace_content` is enabled) |
| `gen_ai.usage.prompt_tokens` | Number of input tokens |
| `gen_ai.usage.completion_tokens` | Number of output tokens |
| `gen_ai.usage.total_tokens` | Total tokens used |
### Vector Database Operations
| Attribute | Description |
| ----------------------- | ------------------------------------------ |
| `db.system` | Database type (e.g., "pinecone", "qdrant") |
| `db.operation` | Operation type (e.g., "query", "upsert") |
| `db.vector.query.top_k` | Number of results requested |
### HTTP Requests
| Attribute | Description |
| -------------------- | ----------------------------- |
| `http.method` | HTTP method (GET, POST, etc.) |
| `http.url` | Request URL |
| `http.status_code` | Response status code |
| `http.request.body` | Request body (if enabled) |
| `http.response.body` | Response body (if enabled) |
## Controlling Content Capture
By default, Netra captures prompt and completion content. You can disable this for privacy or compliance reasons:
```python Python theme={null}
Netra.init(
app_name="my-ai-app",
trace_content=False, # Disable prompt/completion capture
)
```
```typescript TypeScript theme={null}
await Netra.init({
appName: "my-ai-app",
traceContent: false, // Disable prompt/completion capture
});
```
When `trace_content` is disabled, Netra still captures:
* Token counts and usage metrics
* Latency and performance data
* Model names and configuration
* Error information
## Selective Instrumentation
You can control which libraries are instrumented. See [**Instrumentation Selection**](https://docs.getnetra.ai/Observability/Traces/configuration/instrumentation-selection) for details on:
* Enabling only specific instrumentations
* Blocking certain instrumentations
* Fine-tuning what gets captured
## Combining with Manual Tracing
Auto-instrumentation works seamlessly with manual tracing. You can add custom spans and attributes to provide additional context:
```python Python theme={null}
from netra import Netra
from openai import OpenAI
Netra.init(app_name="my-ai-app")
client = OpenAI()
# Create a parent span for your workflow
with Netra.start_span("process-user-query") as span:
span.set_attribute("user.query", user_query)
# Auto-instrumented OpenAI call appears as a child span
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": user_query}],
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import OpenAI from "openai";
await Netra.init({ appName: "my-ai-app" });
const client = new OpenAI();
// Create a parent span for your workflow
const span = Netra.startSpan("process-user-query");
span.setAttribute("user.query", userQuery);
// Auto-instrumented OpenAI call appears as a child span
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: userQuery }],
});
span.end();
```
## Supported Instrumentations
Netra supports a wide range of libraries across both Python and TypeScript. For detailed integration guides, see the [Integrations](/Integrations/overview) section.
### LLM Providers
Automatically trace completions, embeddings, and other API calls to major LLM providers.
| Provider | Python | TypeScript |
| -------------------- | ------------------------- | ------------------------ |
| OpenAI | `openai` | `openai` |
| Anthropic | `anthropic` | - |
| Google Generative AI | `google-generativeai` | `@google/generative-ai` |
| Cohere | `cohere` | - |
| Mistral | `mistralai` | `@mistralai/mistralai` |
| Groq | `groq` | `groq-sdk` |
| AWS Bedrock | `boto3` | - |
| Vertex AI | `google-cloud-aiplatform` | `@google-cloud/vertexai` |
| Ollama | `ollama` | - |
| Replicate | `replicate` | - |
| Together AI | `together` | `together-ai` |
| Hugging Face | `transformers` | - |
| LiteLLM | `litellm` | - |
### AI Frameworks
Capture chain executions, agent steps, and tool calls from popular AI orchestration frameworks.
| Framework | Python | TypeScript |
| ----------- | ------------- | ---------------------- |
| LangChain | `langchain` | `langchain` |
| LangGraph | `langgraph` | `@langchain/langgraph` |
| LlamaIndex | `llama-index` | `llamaindex` |
| Haystack | `haystack` | - |
| CrewAI | `crewai` | - |
| DSPy | `dspy` | - |
| Pydantic AI | `pydantic-ai` | - |
| ADK | `google-adk` | - |
| Agno | `agno` | - |
### Vector Databases
Track similarity searches, upserts, and other vector operations for RAG pipelines.
| Database | Python | TypeScript |
| -------- | ----------------- | ----------------------------- |
| Pinecone | `pinecone-client` | `@pinecone-database/pinecone` |
| Weaviate | `weaviate-client` | - |
| Qdrant | `qdrant-client` | `@qdrant/js-client-rest` |
| ChromaDB | `chromadb` | `chromadb` |
| Milvus | `pymilvus` | - |
| LanceDB | `lancedb` | - |
| Marqo | `marqo` | - |
### Web Frameworks
Instrument incoming HTTP requests to trace your API endpoints end-to-end.
| Framework | Python | TypeScript |
| --------- | ----------- | -------------- |
| FastAPI | `fastapi` | - |
| Flask | `flask` | - |
| Django | `django` | - |
| Starlette | `starlette` | - |
| Express | - | `express` |
| Fastify | - | `fastify` |
| NestJS | - | `@nestjs/core` |
### HTTP Clients
Capture outgoing HTTP requests to external services and APIs.
| Client | Python | TypeScript |
| ---------- | ---------- | ---------- |
| HTTPX | `httpx` | - |
| AIOHTTP | `aiohttp` | - |
| Requests | `requests` | - |
| urllib3 | `urllib3` | - |
| Fetch | - | Built-in |
| HTTP/HTTPS | - | Built-in |
### Databases
Monitor database queries and operations for performance analysis.
| Database | Python | TypeScript |
| ------------- | --------------------- | ------------------ |
| PostgreSQL | `psycopg2`, `psycopg` | `pg` |
| MySQL | `pymysql` | `mysql2` |
| MongoDB | `pymongo` | `mongodb` |
| Redis | `redis` | `redis`, `ioredis` |
| SQLAlchemy | `sqlalchemy` | - |
| SQLite | `sqlite3` | - |
| Elasticsearch | `elasticsearch` | - |
| Prisma | - | `@prisma/client` |
| TypeORM | - | `typeorm` |
### Message Queues
Trace asynchronous task processing and message-based workflows.
| Queue | Python | TypeScript |
| -------- | --------------------------------- | ---------- |
| Celery | `celery` | - |
| RabbitMQ | `pika`, `aio-pika` | `amqplib` |
| Kafka | `kafka-python`, `confluent-kafka` | `kafkajs` |
| AWS SQS | `boto3` | - |
## Next Steps
* [Decorators](/Observability/Traces/decorators) - Add semantic context with `@workflow`, `@agent`, and `@task`
* [Manual Tracing](/Observability/Traces/manual-tracing) - Create custom spans for fine-grained control
* [Instrumentation Selection](/Observability/Traces/configuration/instrumentation-selection) - Control which libraries are instrumented
# Custom Exporters
Source: https://docs.getnetra.ai/Observability/Traces/configuration/custom-exporters
Send Netra traces to your own OTLP-compatible backend or self-hosted stack. Configure custom exporters for Jaeger, Grafana Tempo, or any collector.
Netra uses the OpenTelemetry Protocol (OTLP) to export traces, making it compatible with any OTLP-compliant backend. This guide covers how to configure custom endpoints, authentication, and self-hosted setups.
## OTLP Endpoint Configuration
### Netra Cloud
By default, traces are sent to Netra's cloud backend. Configure the endpoint and API key:
```python Python theme={null}
import os
from netra import Netra
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
});
```
Or via environment variables:
```bash US Region theme={null}
export NETRA_API_KEY="your-api-key"
export NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry"
```
```bash EU Region theme={null}
export NETRA_API_KEY="your-api-key"
export NETRA_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
```
### Regional Endpoints
Netra provides regional endpoints for data residency requirements:
| Region | Location | Endpoint |
| ------ | ------------------------ | -------------------------------------- |
| **US** | N. Virginia (us-east-1) | `https://api.getnetra.ai/telemetry` |
| **EU** | Frankfurt (eu-central-1) | `https://api.eu.getnetra.ai/telemetry` |
Regions are strictly separated, and no data is shared across regions. Choosing a region close to you can help improve speed and comply with local data residency laws and privacy regulations.
```bash theme={null}
# US region
export NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry"
# EU region
export NETRA_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
```
## Custom Backend Configuration
### Setting a Custom Endpoint
Point Netra to any OTLP-compatible backend:
```python Python theme={null}
import os
from netra import Netra
# Set custom endpoint via environment variable
os.environ["NETRA_OTLP_ENDPOINT"] = "https://otel-collector.internal.company.com"
Netra.init(
app_name="my-ai-app",
headers="Authorization=Bearer your-token",
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
// Custom endpoint will have /v1/traces appended automatically
headers: "Authorization=Bearer your-token",
});
```
Via environment variables:
```bash theme={null}
export NETRA_OTLP_ENDPOINT="https://otel-collector.internal.company.com"
export NETRA_HEADERS="Authorization=Bearer your-token"
```
### Endpoint URL Formatting
Netra automatically formats the endpoint URL:
* If the URL doesn't end with `/v1/traces`, it's appended automatically
* Base URLs like `https://collector.example.com` become `https://collector.example.com/v1/traces`
```bash theme={null}
# These are equivalent:
export NETRA_OTLP_ENDPOINT="https://collector.example.com"
export NETRA_OTLP_ENDPOINT="https://collector.example.com/v1/traces"
```
## Authentication Methods
### API Key Authentication (Netra Cloud)
For Netra cloud endpoints, use the `x-api-key` header:
```python Python theme={null}
Netra.init(
app_name="my-ai-app",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
)
```
```typescript TypeScript theme={null}
await Netra.init({
appName: "my-ai-app",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
});
```
### Bearer Token Authentication
For custom backends that use Bearer tokens:
```python Python theme={null}
Netra.init(
app_name="my-ai-app",
headers=f"Authorization=Bearer {os.getenv('OTEL_TOKEN')}",
)
```
```typescript TypeScript theme={null}
await Netra.init({
appName: "my-ai-app",
headers: `Authorization=Bearer ${process.env.OTEL_TOKEN}`,
});
```
### Multiple Headers
Pass multiple headers using comma separation (W3C Correlation-Context format):
```python Python theme={null}
Netra.init(
app_name="my-ai-app",
headers="Authorization=Bearer token123,x-custom-header=value",
)
```
```typescript TypeScript theme={null}
await Netra.init({
appName: "my-ai-app",
headers: "Authorization=Bearer token123,x-custom-header=value",
});
```
### Automatic Authentication Detection
Netra automatically determines the authentication method based on the endpoint:
| Endpoint Contains | Authentication Header |
| ----------------- | ------------------------------- |
| `getnetra` | `x-api-key: {value}` |
| Other | `Authorization: Bearer {value}` |
This means for Netra cloud, you only need to provide the API key value:
```bash theme={null}
# For Netra cloud - uses x-api-key header automatically
export NETRA_API_KEY="ntr_abc123"
# For custom backends - uses Authorization: Bearer header
export NETRA_HEADERS="Authorization=Bearer custom-token"
```
## Self-Hosted OpenTelemetry Collector
### Basic Collector Setup
Deploy an OpenTelemetry Collector to receive traces from Netra:
```yaml theme={null}
# otel-collector-config.yaml
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
timeout: 1s
send_batch_size: 1024
exporters:
# Export to Jaeger
jaeger:
endpoint: "jaeger:14250"
tls:
insecure: true
# Export to logging (for debugging)
logging:
loglevel: debug
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger, logging]
```
Configure Netra to send to the collector:
```bash theme={null}
export NETRA_OTLP_ENDPOINT="https://otel-collector:4318"
```
### Docker Compose Example
```yaml theme={null}
# docker-compose.yml
version: '3.8'
services:
app:
build: .
environment:
- NETRA_APP_NAME=my-ai-app
- NETRA_OTLP_ENDPOINT=https://otel-collector:4318
- NETRA_ENV=development
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # Jaeger UI
- "14250:14250" # Jaeger gRPC
```
### Kubernetes Deployment
```yaml theme={null}
# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
spec:
replicas: 1
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:latest
args:
- --config=/etc/otel/config.yaml
ports:
- containerPort: 4317
- containerPort: 4318
volumeMounts:
- name: config
mountPath: /etc/otel
volumes:
- name: config
configMap:
name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
name: otel-collector
spec:
selector:
app: otel-collector
ports:
- name: otlp-grpc
port: 4317
- name: otlp-http
port: 4318
```
## Popular Backend Integrations
### Jaeger
```yaml theme={null}
# otel-collector-config.yaml
exporters:
jaeger:
endpoint: "jaeger-collector:14250"
tls:
insecure: true
```
```bash theme={null}
export NETRA_OTLP_ENDPOINT="https://otel-collector:4318"
```
### Grafana Tempo
```yaml theme={null}
# otel-collector-config.yaml
exporters:
otlp:
endpoint: "tempo:4317"
tls:
insecure: true
```
### Datadog
```yaml theme={null}
# otel-collector-config.yaml
exporters:
datadog:
api:
key: ${DD_API_KEY}
site: datadoghq.com
```
### New Relic
```yaml theme={null}
# otel-collector-config.yaml
exporters:
otlp:
endpoint: "https://otlp.nr-data.net:4317"
headers:
api-key: ${NEW_RELIC_LICENSE_KEY}
```
### Honeycomb
```yaml theme={null}
# otel-collector-config.yaml
exporters:
otlp:
endpoint: "api.honeycomb.io:443"
headers:
x-honeycomb-team: ${HONEYCOMB_API_KEY}
```
## Dual Export Setup
Send traces to both Netra and a custom backend:
```yaml theme={null}
# otel-collector-config.yaml
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
exporters:
# Forward to Netra (use endpoint matching your region)
# US: https://api.getnetra.ai/telemetry/v1/traces
# EU: https://api.eu.getnetra.ai/telemetry/v1/traces
otlp/netra:
endpoint: "https://api.getnetra.ai/telemetry/v1/traces"
headers:
x-api-key: ${NETRA_API_KEY}
# Also export to Jaeger
jaeger:
endpoint: "jaeger:14250"
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/netra, jaeger]
```
Configure your application to send to the local collector:
```bash theme={null}
export NETRA_OTLP_ENDPOINT="https://localhost:4318"
```
## Troubleshooting
### Connection Issues
1. **Verify endpoint reachability**
```bash theme={null}
curl -v https://your-endpoint/v1/traces
```
2. **Check authentication**
* Ensure API key or token is correct
* Verify header format matches backend expectations
3. **Enable debug mode**
```python theme={null}
Netra.init(app_name="my-app", debug_mode=True)
```
### Traces Not Appearing
1. **Disable batching for debugging**
```python theme={null}
Netra.init(app_name="my-app", disable_batch=True)
```
2. **Check collector logs**
```bash theme={null}
docker logs otel-collector
```
3. **Verify OTLP endpoint format**
* Ensure the endpoint accepts HTTP/protobuf
* Check if `/v1/traces` path is correct
### TLS/SSL Errors
For self-signed certificates in development:
```yaml theme={null}
# otel-collector-config.yaml
exporters:
otlp:
endpoint: "https://internal-backend:4317"
tls:
insecure: true # Only for development!
```
Never use `insecure: true` in production. Configure proper TLS certificates instead.
## Best Practices
1. **Use environment variables** - Keep endpoints and credentials out of code.
2. **Deploy a collector** - For production, use an OpenTelemetry Collector as a gateway rather than sending directly from applications.
3. **Enable batching** - Keep `disable_batch=False` (default) for better performance.
4. **Monitor collector health** - Set up health checks and metrics for your collector.
5. **Use regional endpoints** - Choose the Netra endpoint closest to your infrastructure.
## Next Steps
* [Initialization](/Observability/Traces/configuration/initialization) - All configuration options
* [Environment Variables](/Observability/Traces/configuration/environment-variables) - Configure via environment
* [Auto Instrumentation](/Observability/Traces/auto-instrumentation) - Supported libraries
# Environment Variables
Source: https://docs.getnetra.ai/Observability/Traces/configuration/environment-variables
Configure the Netra SDK using environment variables instead of code. Set API key, endpoint, trace content, and more without modifying your application.
Environment variables provide a way to configure Netra without modifying code. This is useful for managing different configurations across environments (development, staging, production) and for keeping sensitive values like API keys out of your codebase.
## Netra Environment Variables
These variables are specific to the Netra SDK:
### Authentication and Endpoint
| Variable | Description | Example |
| --------------------- | ------------------------------------------------ | --------------------------------- |
| `NETRA_API_KEY` | API key for authenticating with Netra | `ntr_abc123...` |
| `NETRA_OTLP_ENDPOINT` | OTLP endpoint URL for sending traces | See region endpoints below |
| `NETRA_HEADERS` | Custom headers in W3C Correlation-Context format | `x-api-key=abc123,x-custom=value` |
Use the endpoint that matches your selected data region:
| Region | Endpoint |
| ------ | -------------------------------------- |
| **US** | `https://api.getnetra.ai/telemetry` |
| **EU** | `https://api.eu.getnetra.ai/telemetry` |
```bash US Region theme={null}
export NETRA_API_KEY="your-api-key-here"
export NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry"
```
```bash EU Region theme={null}
export NETRA_API_KEY="your-api-key-here"
export NETRA_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
```
### Application Identity
| Variable | Description | Example |
| ---------------- | ------------------------ | -------------------------------------- |
| `NETRA_APP_NAME` | Name of your application | `my-ai-service` |
| `NETRA_ENV` | Deployment environment | `production`, `staging`, `development` |
```bash theme={null}
export NETRA_APP_NAME="my-ai-service"
export NETRA_ENV="production"
```
### Tracing Behavior
| Variable | Description | Default |
| ------------------------ | -------------------------------------------------------- | ------- |
| `NETRA_TRACE_CONTENT` | Capture prompt/completion content (`true`/`false`) | `true` |
| `NETRA_DISABLE_BATCH` | Send spans immediately without batching (`true`/`false`) | `false` |
| `NETRA_ENABLE_ROOT_SPAN` | Create long-lived root span (`true`/`false`) | `false` |
| `NETRA_ENABLE_SCRUBBING` | Enable PII scrubbing (Python only) (`true`/`false`) | `false` |
| `NETRA_DEBUG` | Enable debug logging (`true`/`false`) | `false` |
| `BLOCKED_URL_PATTERNS` | Block internal request calls by URL pattern matching | `None` |
```bash theme={null}
# Production settings
export NETRA_TRACE_CONTENT="true"
export NETRA_DISABLE_BATCH="false"
export NETRA_DEBUG="false"
export BLOCKED_URL_PATTERNS="getnetra.ai"
# Development settings
export NETRA_TRACE_CONTENT="true"
export NETRA_DISABLE_BATCH="true" # Immediate export for debugging
export NETRA_DEBUG="true"
export BLOCKED_URL_PATTERNS="dev.getnetra.ai"
```
### Attribute Limits
| Variable | Description | Default |
| ------------------------------------ | ---------------------------------------- | ------- |
| `NETRA_ATTRIBUTE_MAX_LEN` | Maximum length for span attribute values | `50000` |
| `NETRA_CONVERSATION_CONTENT_MAX_LEN` | Maximum length for conversation content | `50000` |
```bash theme={null}
# Increase limits for verbose applications
export NETRA_ATTRIBUTE_MAX_LEN="100000"
export NETRA_CONVERSATION_CONTENT_MAX_LEN="100000"
```
### Resource Attributes
| Variable | Description | Example |
| ---------------------- | ----------------------------------------- | ------------------------------- |
| `NETRA_RESOURCE_ATTRS` | JSON string of custom resource attributes | `{"team":"ml","version":"1.0"}` |
```bash theme={null}
export NETRA_RESOURCE_ATTRS='{"deployment.version":"1.2.3","team":"ml-platform"}'
```
## OpenTelemetry Environment Variables
Netra also respects standard OpenTelemetry environment variables. These serve as fallbacks when Netra-specific variables are not set.
| Variable | Netra Equivalent | Description |
| ----------------------------- | ---------------------- | ------------------------ |
| `OTEL_SERVICE_NAME` | `NETRA_APP_NAME` | Service/application name |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `NETRA_OTLP_ENDPOINT` | OTLP exporter endpoint |
| `OTEL_EXPORTER_OTLP_HEADERS` | `NETRA_HEADERS` | OTLP exporter headers |
| `OTEL_RESOURCE_ATTRIBUTES` | `NETRA_RESOURCE_ATTRS` | Resource attributes |
```bash theme={null}
# These work as fallbacks (use endpoint matching your region)
export OTEL_SERVICE_NAME="my-ai-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry" # US region
# export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry" # EU region
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=your-key"
```
## Configuration Precedence
When the same setting is configured in multiple places, Netra uses this priority order:
1. **Code parameters** (highest priority)
```python theme={null}
Netra.init(app_name="code-value") # This wins
```
2. **Netra environment variables**
```bash theme={null}
export NETRA_APP_NAME="netra-env-value"
```
3. **OpenTelemetry environment variables**
```bash theme={null}
export OTEL_SERVICE_NAME="otel-env-value"
```
4. **Default values** (lowest priority)
### Example
```bash theme={null}
# Environment
export NETRA_APP_NAME="env-app"
export OTEL_SERVICE_NAME="otel-app"
```
```python theme={null}
# Code
Netra.init(app_name="code-app") # Uses "code-app"
Netra.init() # Uses "env-app" (NETRA_ takes precedence over OTEL_)
```
## Environment-Specific Configuration
### Development
```bash theme={null}
# .env.development
NETRA_API_KEY="dev-api-key"
NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry" # Use endpoint matching your region
NETRA_APP_NAME="my-app-dev"
NETRA_ENV="development"
NETRA_DEBUG="true"
NETRA_DISABLE_BATCH="true"
NETRA_TRACE_CONTENT="true"
```
### Staging
```bash theme={null}
# .env.staging
NETRA_API_KEY="staging-api-key"
NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry" # Use endpoint matching your region
NETRA_APP_NAME="my-app-staging"
NETRA_ENV="staging"
NETRA_DEBUG="false"
NETRA_DISABLE_BATCH="false"
NETRA_TRACE_CONTENT="true"
```
### Production
```bash theme={null}
# .env.production
NETRA_API_KEY="prod-api-key"
NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry" # Use endpoint matching your region
NETRA_APP_NAME="my-app"
NETRA_ENV="production"
NETRA_DEBUG="false"
NETRA_DISABLE_BATCH="false"
NETRA_TRACE_CONTENT="true"
NETRA_ENABLE_SCRUBBING="true" # Enable PII protection in production
```
## Loading Environment Variables
### Python
```python theme={null}
# Using python-dotenv
from dotenv import load_dotenv
load_dotenv() # Load from .env file
from netra import Netra
Netra.init() # Automatically picks up environment variables
```
### TypeScript/Node.js
```typescript theme={null}
// Using dotenv
import "dotenv/config";
import { Netra } from "netra-sdk";
await Netra.init({
appName: process.env.NETRA_APP_NAME || "my-app",
});
```
## Docker and Kubernetes
### Docker
```dockerfile theme={null}
# Dockerfile
ENV NETRA_APP_NAME="my-app"
ENV NETRA_ENV="production"
```
```bash theme={null}
# docker run
docker run -e NETRA_API_KEY="your-key" -e NETRA_ENV="production" my-app
```
### Docker Compose
```yaml theme={null}
# docker-compose.yml
services:
app:
image: my-app
environment:
- NETRA_API_KEY=${NETRA_API_KEY}
- NETRA_OTLP_ENDPOINT=https://api.getnetra.ai/telemetry # Use endpoint matching your region
- NETRA_APP_NAME=my-app
- NETRA_ENV=production
```
### Kubernetes
```yaml theme={null}
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
spec:
containers:
- name: my-app
env:
- name: NETRA_API_KEY
valueFrom:
secretKeyRef:
name: netra-secrets
key: api-key
- name: NETRA_OTLP_ENDPOINT
value: "https://api.getnetra.ai/telemetry" # Use endpoint matching your region
- name: NETRA_APP_NAME
value: "my-app"
- name: NETRA_ENV
value: "production"
```
```yaml theme={null}
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: netra-secrets
type: Opaque
stringData:
api-key: "your-netra-api-key"
```
## Complete Reference
| Variable | Type | Default | Description |
| ------------------------------------ | ----------- | --------- | --------------------------------- |
| `NETRA_API_KEY` | string | - | API key for authentication |
| `NETRA_OTLP_ENDPOINT` | string | - | OTLP endpoint URL |
| `NETRA_HEADERS` | string | - | Custom headers (W3C format) |
| `NETRA_APP_NAME` | string | - | Application name |
| `NETRA_ENV` | string | `default` | Environment name |
| `NETRA_TRACE_CONTENT` | boolean | `true` | Capture prompt/completion content |
| `NETRA_DISABLE_BATCH` | boolean | `false` | Disable span batching |
| `NETRA_ENABLE_ROOT_SPAN` | boolean | `false` | Enable root span |
| `NETRA_ENABLE_SCRUBBING` | boolean | `false` | Enable PII scrubbing (Python) |
| `NETRA_DEBUG` | boolean | `false` | Enable debug logging |
| `NETRA_ATTRIBUTE_MAX_LEN` | integer | `50000` | Max attribute length |
| `NETRA_CONVERSATION_CONTENT_MAX_LEN` | integer | `50000` | Max conversation length |
| `NETRA_RESOURCE_ATTRS` | JSON string | `{}` | Custom resource attributes |
## Next Steps
* [Initialization](/Observability/Traces/configuration/initialization) - Programmatic configuration
* [Instrumentation Selection](/Observability/Traces/configuration/instrumentation-selection) - Control which libraries are traced
* [Custom Exporters](/Observability/Traces/configuration/custom-exporters) - Send traces to custom backends
# Initialization
Source: https://docs.getnetra.ai/Observability/Traces/configuration/initialization
Configure the Netra SDK with Netra.init() in Python and TypeScript. Set your API key, environment, trace content, and instrument selection at startup.
The `Netra.init()` function configures the SDK and starts the tracing system. Call it once at the start of your application, before making any LLM or database calls.
## Quick Start
```python Python theme={null}
from netra import Netra
Netra.init(
app_name="my-ai-app",
environment="production",
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
## Configuration Parameters
| Parameter | Type | Default | Description |
| -------------------------------------------- | ----------- | --------------------------------- | ---------------------------------------------------------------------- |
| `app_name` / `appName` | string | Required | Application name for identifying traces in the dashboard |
| `environment` | string | `"default"` (Py) / `"local"` (TS) | Deployment environment (e.g., `production`, `staging`, `development`) |
| `headers` | string | `""` | Authentication headers, typically `x-api-key=YOUR_KEY` |
| `trace_content` / `traceContent` | boolean | `true` | Capture prompt/completion content from LLM calls. Disable for privacy. |
| `debug_mode` / `debugMode` | boolean | `false` | Enable verbose logging for troubleshooting |
| `disable_batch` / `disableBatch` | boolean | `false` | Send spans immediately instead of batching |
| `enable_root_span` / `enableRootSpan` | boolean | `false` | Create a root span for the entire process (useful for workers) |
| `resource_attributes` / `resourceAttributes` | dict/object | `{}` | Custom attributes added to every span |
| `blocked_spans` / `blockedSpans` | list/array | `[]` | Span name patterns to exclude (supports `*` wildcards) |
| `enable_scrubbing` | boolean | `false` | Auto-redact detected PII (Python only) |
| `instruments` | Set/list | All | Specific instrumentations to enable |
| `block_instruments` / `blockInstruments` | Set/list | `[]` | Instrumentations to disable |
For instrumentation control details, see [Instrumentation Selection](/Observability/Traces/configuration/instrumentation-selection).
## Complete Example
```python Python theme={null}
import os
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
# Core settings
app_name="production-ai-service",
environment="production",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
# Content and privacy
trace_content=True,
enable_scrubbing=False,
# Performance
disable_batch=False,
# Debugging
debug_mode=False,
# Long-running processes
enable_root_span=False,
# Custom metadata added to all spans
resource_attributes={
"service.version": os.getenv("APP_VERSION"),
"deployment.region": os.getenv("AWS_REGION"),
"team": "ml-platform",
},
# Span filtering (supports wildcards)
blocked_spans=[
"health-check", # Exact match
"internal.*", # Prefix match
"*.debug", # Suffix match
],
# Instrumentation control
block_instruments={
InstrumentSet.HTTPX,
InstrumentSet.REQUESTS,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
// Core settings
appName: "production-ai-service",
environment: "production",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
// Content and privacy
traceContent: true,
// Performance
disableBatch: false,
// Debugging
debugMode: false,
// Long-running processes
enableRootSpan: false,
// Custom metadata added to all spans
resourceAttributes: {
"service.version": process.env.APP_VERSION,
"deployment.region": process.env.AWS_REGION,
"team": "ml-platform",
},
// Span filtering (supports wildcards)
blockedSpans: [
"health-check", // Exact match
"internal.*", // Prefix match
"*.debug", // Suffix match
],
// Instrumentation control
blockInstruments: new Set([
NetraInstruments.HTTP,
NetraInstruments.FETCH,
]),
});
```
PII scrubbing (`enable_scrubbing`) adds processing overhead. Only enable when handling sensitive data.
## Configuration Precedence
Configuration values are resolved in order of priority:
1. **Code parameters** - Values passed to `Netra.init()`
2. **Netra environment variables** - `NETRA_*` variables
3. **OpenTelemetry environment variables** - `OTEL_*` variables
4. **Default values** - SDK defaults
## Async Initialization (TypeScript)
The `Netra.init()` method in TypeScript is async and waits for all instrumentations to be ready before returning. Always await the call to ensure proper instrumentation:
```typescript TypeScript theme={null}
async function main() {
// init() is async and waits for instrumentations to be ready
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
// SDK is fully initialized, all instrumentations are patched
}
```
Always `await Netra.init()` to ensure libraries like OpenAI, Anthropic, and LangGraph are properly instrumented before use. This is especially important in frameworks like NestJS where modules are loaded after initialization.
## Shutdown
Ensure all pending spans are exported before application exit:
```python Python theme={null}
Netra.shutdown()
```
```typescript TypeScript theme={null}
await Netra.shutdown();
```
Always call `shutdown()` before exit, especially for short-lived processes like serverless functions.
## Next Steps
* [Environment Variables](/Observability/Traces/configuration/environment-variables) - Configure via environment
* [Instrumentation Selection](/Observability/Traces/configuration/instrumentation-selection) - Control which libraries are traced
* [Custom Exporters](/Observability/Traces/configuration/custom-exporters) - Send traces to custom backends
# Instrumentation Selection
Source: https://docs.getnetra.ai/Observability/Traces/configuration/instrumentation-selection
Selectively enable or disable Netra instrumentations per provider or framework. Control which LLM providers and databases are traced in your app.
By default, Netra instruments a **curated set** of AI-relevant libraries detected in your application. You can customize this behavior to enable only specific instrumentations, block certain ones, or opt into instrumenting all available libraries.
## How It Works
Netra provides three parameters for controlling instrumentations:
| Parameter | Behavior | Default |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------ |
| `instruments` | Set of libraries to instruments. When provided, **only** these instrumentations are enabled. | `DEFAULT_INSTRUMENTS` |
| `root_instruments` | Set of libraries allowed to produce **root-level spans**. (see [Controlling Root-Level Spans](#controlling-root-level-spans)). | `DEFAULT_INSTRUMENTS_FOR_ROOT` |
| `block_instruments` | These instrumentations are **excluded** from both `instruments` and `root_instruments` independently. | `None` |
Use `instruments` when you want a minimal setup with only specific libraries traced. Use `block_instruments` when you want the curated defaults but need to exclude a few. Use `root_instruments` to control which libraries can start new top-level traces.
### The `ALL` Sentinel
Pass `NetraInstruments.ALL` to restore the legacy behavior of instrumenting every supported library found in your environment:
```python Python theme={null}
from netra import Netra, NetraInstruments
Netra.init(
app_name="my-ai-app",
instruments={NetraInstruments.ALL},
root_instruments={NetraInstruments.ALL},
)
```
`ALL` can be used in `instruments`, `root_instruments`, and `block_instruments`. When present in `block_instruments`, it blocks all instrumentations.
## Enabling Specific Instrumentations
Use the `instruments` parameter to enable only the instrumentations you need:
```python Python theme={null}
from netra import Netra, NetraInstruments
Netra.init(
app_name="my-ai-app",
instruments={
NetraInstruments.OPENAI,
NetraInstruments.PINECONE,
NetraInstruments.LANGCHAIN,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
instruments: new Set([
NetraInstruments.OPENAI,
NetraInstruments.PINECONE,
NetraInstruments.LANGCHAIN,
]),
});
```
## Blocking Specific Instrumentations
Use `block_instruments` to exclude certain instrumentations while keeping the curated defaults:
```python Python theme={null}
from netra import Netra, NetraInstruments
# Use curated defaults except HTTP clients
Netra.init(
app_name="my-ai-app",
block_instruments={
NetraInstruments.HTTPX,
NetraInstruments.AIOHTTP,
NetraInstruments.REQUESTS,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Use curated defaults except HTTP clients
await Netra.init({
appName: "my-ai-app",
blockInstruments: new Set([
NetraInstruments.HTTP,
NetraInstruments.HTTPS,
NetraInstruments.FETCH,
]),
});
```
## Controlling Root-Level Spans
Use `root_instruments` (`rootInstruments` in TypeScript) to control which libraries can produce root-level spans. It is independent of `instruments`, and defaults to the curated `DEFAULT_INSTRUMENTS_FOR_ROOT` subset (LLM providers and AI frameworks) when omitted.
```python Python theme={null}
from netra import Netra, NetraInstruments
# Instrument the curated defaults, but only allow LLM providers to start traces
Netra.init(
app_name="my-ai-app",
root_instruments={
NetraInstruments.OPENAI,
NetraInstruments.ANTHROPIC,
NetraInstruments.GOOGLE_GENERATIVEAI,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
// Instrument the curated defaults, but only allow LLM providers to start traces
await Netra.init({
appName: "my-ai-app",
rootInstruments: new Set([
NetraInstruments.OPENAI,
NetraInstruments.ANTHROPIC,
NetraInstruments.GOOGLE_GENAI,
]),
});
```
Pass the `ALL` sentinel to `root_instruments` / `rootInstruments` to let every instrumentation produce root spans (legacy behavior). When a root span is enabled via `enable_root_span` / `enableRootSpan`, Netra attaches its own root span and every auto-instrumentation span becomes its child, so `root_instruments` has no effect.
When a root span comes from an instrumentation outside the allowed set, only the disallowed root span is dropped: its children are **reparented** onto its parent so nested LLM and vector spans are kept. Use this to filter out noisy top-level traces from HTTP clients, web frameworks, or database drivers.
## Available Instruments
### Python Instruments
#### LLM Providers
| Instrument | Description |
| ----------------------------------- | ------------------------- |
| `InstrumentSet.OPENAI` | OpenAI API calls |
| `InstrumentSet.ANTHROPIC` | Anthropic Claude API |
| `InstrumentSet.COHERE` | Cohere API |
| `InstrumentSet.GOOGLE_GENERATIVEAI` | Google Generative AI |
| `InstrumentSet.MISTRALAI` | Mistral AI API |
| `InstrumentSet.GROQ` | Groq API |
| `InstrumentSet.BEDROCK` | AWS Bedrock |
| `InstrumentSet.VERTEXAI` | Google Vertex AI |
| `InstrumentSet.OLLAMA` | Ollama local models |
| `InstrumentSet.REPLICATE` | Replicate API |
| `InstrumentSet.TOGETHER` | Together AI |
| `InstrumentSet.TRANSFORMERS` | Hugging Face Transformers |
| `InstrumentSet.LITELLM` | LiteLLM unified interface |
#### AI Frameworks
| Instrument | Description |
| --------------------------- | -------------------- |
| `InstrumentSet.LANGCHAIN` | LangChain framework |
| `InstrumentSet.LANGGRAPH` | LangGraph workflows |
| `InstrumentSet.LLAMAINDEX` | LlamaIndex framework |
| `InstrumentSet.HAYSTACK` | Haystack framework |
| `InstrumentSet.CREWAI` | CrewAI agents |
| `InstrumentSet.DSPY` | DSPy framework |
| `InstrumentSet.PYDANTIC_AI` | Pydantic AI |
#### Vector Databases
| Instrument | Description |
| ------------------------ | ------------------------ |
| `InstrumentSet.PINECONE` | Pinecone vector database |
| `InstrumentSet.WEAVIATE` | Weaviate vector database |
| `InstrumentSet.QDRANT` | Qdrant vector database |
| `InstrumentSet.CHROMADB` | ChromaDB vector database |
| `InstrumentSet.MILVUS` | Milvus vector database |
| `InstrumentSet.LANCEDB` | LanceDB vector database |
| `InstrumentSet.MARQO` | Marqo vector database |
#### HTTP Clients
| Instrument | Description |
| ------------------------ | -------------------- |
| `InstrumentSet.HTTPX` | HTTPX async client |
| `InstrumentSet.AIOHTTP` | AIOHTTP async client |
| `InstrumentSet.REQUESTS` | Requests library |
| `InstrumentSet.URLLIB` | urllib library |
| `InstrumentSet.URLLIB3` | urllib3 library |
#### Web Frameworks
| Instrument | Description |
| ------------------------- | ------------------- |
| `InstrumentSet.FASTAPI` | FastAPI framework |
| `InstrumentSet.FLASK` | Flask framework |
| `InstrumentSet.DJANGO` | Django framework |
| `InstrumentSet.STARLETTE` | Starlette framework |
| `InstrumentSet.TORNADO` | Tornado framework |
| `InstrumentSet.FALCON` | Falcon framework |
#### Databases
| Instrument | Description |
| ----------------------------- | --------------------- |
| `InstrumentSet.SQLALCHEMY` | SQLAlchemy ORM |
| `InstrumentSet.SQLITE3` | SQLite3 |
| `InstrumentSet.PSYCOPG` | PostgreSQL (psycopg) |
| `InstrumentSet.PSYCOPG2` | PostgreSQL (psycopg2) |
| `InstrumentSet.PYMYSQL` | MySQL |
| `InstrumentSet.PYMONGO` | MongoDB |
| `InstrumentSet.REDIS` | Redis |
| `InstrumentSet.ELASTICSEARCH` | Elasticsearch |
#### Message Queues
| Instrument | Description |
| ------------------------------- | ------------------- |
| `InstrumentSet.CELERY` | Celery task queue |
| `InstrumentSet.PIKA` | RabbitMQ (pika) |
| `InstrumentSet.AIO_PIKA` | RabbitMQ (aio-pika) |
| `InstrumentSet.KAFKA_PYTHON` | Kafka |
| `InstrumentSet.CONFLUENT_KAFKA` | Confluent Kafka |
### TypeScript Instruments
#### LLM Providers
| Instrument | Description |
| --------------------------------- | -------------------- |
| `NetraInstrumentSet.OPENAI` | OpenAI API calls |
| `NetraInstrumentSet.GOOGLE_GENAI` | Google Generative AI |
| `NetraInstrumentSet.MISTRAL` | Mistral AI API |
| `NetraInstrumentSet.GROQ` | Groq API |
| `NetraInstrumentSet.VERTEX_AI` | Google Vertex AI |
| `NetraInstrumentSet.TOGETHER` | Together AI |
#### AI Frameworks
| Instrument | Description |
| ------------------------------- | -------------------- |
| `NetraInstrumentSet.LANGCHAIN` | LangChain framework |
| `NetraInstrumentSet.LANGGRAPH` | LangGraph workflows |
| `NetraInstrumentSet.LLAMAINDEX` | LlamaIndex framework |
#### Vector Databases
| Instrument | Description |
| ----------------------------- | ------------------------ |
| `NetraInstrumentSet.PINECONE` | Pinecone vector database |
| `NetraInstrumentSet.QDRANT` | Qdrant vector database |
| `NetraInstrumentSet.CHROMADB` | ChromaDB vector database |
#### HTTP Clients
| Instrument | Description |
| -------------------------- | -------------------- |
| `NetraInstrumentSet.HTTP` | Node.js HTTP module |
| `NetraInstrumentSet.HTTPS` | Node.js HTTPS module |
| `NetraInstrumentSet.FETCH` | Fetch API |
#### Web Frameworks
| Instrument | Description |
| ---------------------------- | -------------------- |
| `NetraInstrumentSet.EXPRESS` | Express.js framework |
| `NetraInstrumentSet.FASTIFY` | Fastify framework |
| `NetraInstrumentSet.NESTJS` | NestJS framework |
#### Databases
| Instrument | Description |
| ----------------------------- | ----------- |
| `NetraInstrumentSet.PRISMA` | Prisma ORM |
| `NetraInstrumentSet.TYPEORM` | TypeORM |
| `NetraInstrumentSet.MONGODB` | MongoDB |
| `NetraInstrumentSet.POSTGRES` | PostgreSQL |
| `NetraInstrumentSet.MYSQL` | MySQL |
| `NetraInstrumentSet.REDIS` | Redis |
#### Message Queues
| Instrument | Description |
| ----------------------------- | ----------- |
| `NetraInstrumentSet.KAFKA` | Kafka |
| `NetraInstrumentSet.RABBITMQ` | RabbitMQ |
## Common Use Cases
### LLM-Only Tracing
Trace only LLM provider calls, ignoring HTTP, database, and framework instrumentation:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="llm-only-app",
instruments={
InstrumentSet.OPENAI,
InstrumentSet.ANTHROPIC,
InstrumentSet.GOOGLE_GENERATIVEAI,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "llm-only-app",
instruments: new Set([
NetraInstrumentSet.OPENAI,
NetraInstrumentSet.GOOGLE_GENAI,
NetraInstrumentSet.MISTRAL,
]),
});
```
### Reduce HTTP Noise
Block HTTP client instrumentation to reduce trace noise while keeping everything else:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="my-ai-app",
block_instruments={
InstrumentSet.HTTPX,
InstrumentSet.AIOHTTP,
InstrumentSet.REQUESTS,
InstrumentSet.URLLIB,
InstrumentSet.URLLIB3,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
blockInstruments: new Set([
NetraInstrumentSet.HTTP,
NetraInstrumentSet.HTTPS,
NetraInstrumentSet.FETCH,
]),
});
```
### RAG Application
Instrument only what's needed for a typical RAG application:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="rag-app",
instruments={
# LLM provider
InstrumentSet.OPENAI,
# Vector database
InstrumentSet.PINECONE,
# Framework
InstrumentSet.LANGCHAIN,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "rag-app",
instruments: new Set([
// LLM provider
NetraInstrumentSet.OPENAI,
// Vector database
NetraInstrumentSet.PINECONE,
// Framework
NetraInstrumentSet.LANGCHAIN,
]),
});
```
### API Service
Instrument a FastAPI/Express service with database and LLM calls:
```python Python theme={null}
from netra import Netra
from netra.instrumentation.instruments import InstrumentSet
Netra.init(
app_name="api-service",
instruments={
# Web framework
InstrumentSet.FASTAPI,
# LLM
InstrumentSet.OPENAI,
# Database
InstrumentSet.SQLALCHEMY,
InstrumentSet.REDIS,
},
)
```
```typescript TypeScript theme={null}
import { Netra, NetraInstruments } from "netra-sdk";
await Netra.init({
appName: "api-service",
instruments: new Set([
// Web framework
NetraInstrumentSet.EXPRESS,
// LLM
NetraInstrumentSet.OPENAI,
// Database
NetraInstrumentSet.POSTGRES,
NetraInstrumentSet.REDIS,
]),
});
```
## Best Practices
1. **Start broad, then narrow** - Begin with default instrumentation (all libraries), then use `block_instruments` to remove noisy ones.
2. **Use `instruments` for minimal setups** - When you know exactly which libraries you need traced, use `instruments` for a cleaner configuration.
3. **Block HTTP for cleaner traces** - HTTP instrumentation can create many spans. Block it if you don't need to trace external API calls.
4. **Match your dependencies** - Only include instruments for libraries you actually use. Extra instruments have minimal overhead but can cause confusion.
5. **Document your choices** - Add comments explaining why certain instrumentations are enabled or blocked.
```python theme={null}
Netra.init(
app_name="my-app",
block_instruments={
# Block HTTP to reduce noise - we only care about LLM calls
InstrumentSet.HTTPX,
InstrumentSet.REQUESTS,
},
)
```
## Troubleshooting
### Instrumentation not working
1. **Check library installation order** - Initialize Netra before importing the library you want to instrument.
2. **Verify the instrument name** - Ensure you're using the correct enum value for your SDK version.
3. **Enable debug mode** - Use `debug_mode=True` to see which instrumentations are being loaded.
### Too many spans
1. **Block HTTP clients** - HTTP instrumentation often creates the most spans.
2. **Use `instruments`** - Switch from blocking to allowlisting for more control.
3. **Use `blocked_spans`** - Filter specific span names at export time (see [Initialization](/Observability/Traces/configuration/initialization)).
## Next Steps
* [Initialization](/Observability/Traces/configuration/initialization) - All configuration options
* [Environment Variables](/Observability/Traces/configuration/environment-variables) - Configure via environment
* [Custom Exporters](/Observability/Traces/configuration/custom-exporters) - Send traces to custom backends
# Decorators
Source: https://docs.getnetra.ai/Observability/Traces/decorators
Use Netra's @workflow, @agent, and @task decorators to add semantic context to traces without manual span management. Works with any Python function.
Decorators provide a clean, declarative way to instrument your code. They automatically create spans with semantic meaning, making your traces easier to understand and navigate in the Netra dashboard.
## Decorators: Overview
Netra provides four decorators, each designed for a specific type of operation:
| Decorator | Purpose | Span Type |
| ----------- | ----------------------------------------- | ------------ |
| `@workflow` | High-level business transactions | Span |
| `@agent` | AI agents or orchestrators | Agent |
| `@task` | Individual units of work | Tool |
| `@span` | Generic operations with customizable type | Configurable |
## @workflow
Use `@workflow` to mark high-level business transactions or processes. Workflows typically represent complete user-facing operations that may involve multiple steps.
```python Python theme={null}
from netra.decorators import workflow
@workflow
def process_customer_order(order_id: str):
order = fetch_order(order_id)
inventory = check_inventory(order.items)
payment = process_payment(order)
shipment = create_shipment(order)
return {"order": order, "shipment": shipment}
# With custom name
@workflow(name="order-fulfillment")
def fulfill_order(order_id: str):
# ...
pass
```
```typescript TypeScript theme={null}
import { workflow } from "netra-sdk";
@workflow
async function processCustomerOrder(orderId: string) {
const order = await fetchOrder(orderId);
const inventory = await checkInventory(order.items);
const payment = await processPayment(order);
const shipment = await createShipment(order);
return { order, shipment };
}
// With custom name
@workflow({ name: "order-fulfillment" })
async function fulfillOrder(orderId: string) {
// ...
}
```
**When to use `@workflow`:**
* User-initiated actions (e.g., "submit order", "generate report")
* End-to-end processes that span multiple operations
* Top-level entry points in your application
## @agent
Use `@agent` to mark AI agents or autonomous components that make decisions. Agent spans help you track reasoning steps and decision-making processes.
```python Python theme={null}
from netra.decorators import agent
@agent
class CustomerSupportAgent:
def handle_ticket(self, ticket: dict):
analysis = self.analyze_ticket(ticket)
response = self.generate_response(analysis)
return response
def analyze_ticket(self, ticket: dict):
# Analysis logic
pass
def generate_response(self, analysis: dict):
# Response generation
pass
# Function-based agent
@agent
def research_agent(query: str):
sources = search_sources(query)
synthesis = synthesize_information(sources)
return synthesis
```
```typescript TypeScript theme={null}
import { agent } from "netra-sdk";
@agent
class CustomerSupportAgent {
async handleTicket(ticket: Ticket) {
const analysis = await this.analyzeTicket(ticket);
const response = await this.generateResponse(analysis);
return response;
}
private async analyzeTicket(ticket: Ticket) {
// Analysis logic
}
private async generateResponse(analysis: Analysis) {
// Response generation
}
}
// Function-based agent
@agent
async function researchAgent(query: string) {
const sources = await searchSources(query);
const synthesis = await synthesizeInformation(sources);
return synthesis;
}
```
**When to use `@agent`:**
* Autonomous AI components that make decisions
* Multi-step reasoning processes
* Components that orchestrate other tools or services
## @task
Use `@task` to mark individual units of work. Tasks are typically discrete operations that perform a specific function within a larger workflow.
```python Python theme={null}
from netra.decorators import task
@task
def fetch_user_profile(user_id: str):
response = requests.get(f"/api/users/{user_id}")
return response.json()
@task
def send_notification(user_id: str, message: str):
notification_service.send(user_id, message)
@task(name="validate-input")
def validate_order_input(order: dict):
if not order.get("items"):
raise ValueError("Order must contain at least one item")
return True
```
```typescript TypeScript theme={null}
import { task } from "netra-sdk";
@task
async function fetchUserProfile(userId: string) {
const response = await fetch(`/api/users/${userId}`);
return response.json();
}
@task
async function sendNotification(userId: string, message: string) {
await notificationService.send(userId, message);
}
@task({ name: "validate-input" })
function validateOrderInput(order: Order) {
if (!order.items || order.items.length === 0) {
throw new Error("Order must contain at least one item");
}
return true;
}
```
**When to use `@task`:**
* Individual operations within a workflow
* Tool calls or function executions
* Database operations, API calls, or computations
## @span
Use `@span` for generic tracing with full control over the span type. This is the most flexible decorator, allowing you to specify the exact span type.
```python Python theme={null}
from netra.decorators import span
from netra import SpanType
# Default span type
@span
def process_data(data: list):
return [transform(item) for item in data]
# With custom span type
@span(as_type=SpanType.GENERATION)
def generate_embedding(text: str):
embedding = embedding_model.embed(text)
return embedding
@span(name="vector-search", as_type=SpanType.TOOL)
def search_vector_db(query: str, top_k: int):
return vector_store.search(query, top_k)
```
```typescript TypeScript theme={null}
import { span, SpanType } from "netra-sdk";
// Default span type
@span
function processData(data: any[]) {
return data.map(transform);
}
// With custom span type
@span({ asType: SpanType.GENERATION })
async function generateEmbedding(text: string) {
const embedding = await embeddingModel.embed(text);
return embedding;
}
@span({ name: "vector-search", asType: SpanType.TOOL })
async function searchVectorDB(query: string, topK: number) {
return await vectorStore.search(query, topK);
}
```
### Available Span Types
| Span Type | Use Case |
| --------------------- | ---------------------------- |
| `SpanType.SPAN` | Generic operations (default) |
| `SpanType.GENERATION` | LLM text generation |
| `SpanType.EMBEDDING` | Vector embedding operations |
| `SpanType.TOOL` | Tool or function calls |
| `SpanType.AGENT` | AI agent operations |
## Decorating Classes
When you apply a decorator to a class, all public methods of that class are automatically instrumented.
```python Python theme={null}
from netra.decorators import agent, task
@agent
class OrderProcessor:
# All public methods are traced as part of the agent
def process_order(self, order: dict):
self.validate_order(order)
self.charge_payment(order)
self.fulfill_order(order)
@task # Override with specific decorator
def validate_order(self, order: dict):
# Validation logic
pass
def charge_payment(self, order: dict):
# Payment logic
pass
def fulfill_order(self, order: dict):
# Fulfillment logic
pass
```
```typescript TypeScript theme={null}
import { agent, task } from "netra-sdk";
@agent
class OrderProcessor {
// All public methods are traced as part of the agent
async processOrder(order: Order) {
await this.validateOrder(order);
await this.chargePayment(order);
await this.fulfillOrder(order);
}
@task // Override with specific decorator
async validateOrder(order: Order) {
// Validation logic
}
private async chargePayment(order: Order) {
// Private methods are also traced
}
private async fulfillOrder(order: Order) {
// Fulfillment logic
}
}
```
## Async and Generator Support
Decorators work seamlessly with async functions and generators:
```python Python theme={null}
from netra.decorators import workflow, task
# Async functions
@workflow
async def async_workflow():
result = await some_async_operation()
return result
# Async generators
@task
async def stream_results(query: str):
stream = await openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": query}],
stream=True,
)
async for chunk in stream:
yield chunk.choices[0].delta.content or ""
# Sync generators
@task
def generate_items(count: int):
for i in range(count):
yield process_item(i)
```
```typescript TypeScript theme={null}
import { workflow, task } from "netra-sdk";
// Async functions
@workflow
async function asyncWorkflow() {
const result = await someAsyncOperation();
return result;
}
// Async generators
@task
async function* streamResults(query: string) {
const stream = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: query }],
stream: true,
});
for await (const chunk of stream) {
yield chunk.choices[0]?.delta?.content || "";
}
}
```
For streaming responses, the span remains open until the stream is fully consumed. This ensures accurate latency measurements for streaming operations.
## Automatic Parameter Capture
Decorators automatically capture function parameters as span attributes, making it easy to understand what inputs were provided:
```python Python theme={null}
from netra.decorators import task
@task
def search_products(query: str, category: str, limit: int = 10):
# Parameters are automatically captured:
# - query: "laptop"
# - category: "electronics"
# - limit: 10
return product_service.search(query, category, limit)
# Call the function
search_products("laptop", "electronics")
```
```typescript TypeScript theme={null}
import { task } from "netra-sdk";
@task
async function searchProducts(
query: string,
category: string,
limit: number = 10
) {
// Parameters are automatically captured:
// - query: "laptop"
// - category: "electronics"
// - limit: 10
return await productService.search(query, category, limit);
}
// Call the function
await searchProducts("laptop", "electronics");
```
Complex types (lists, dicts, objects) are serialized to JSON. Parameter values are truncated to 1000 characters to prevent excessively large attributes.
## Exception Handling
Decorators automatically capture exceptions and mark spans with error status:
```python Python theme={null}
from netra.decorators import task
@task
def risky_operation(data: dict):
if not data.get("valid"):
raise ValueError("Invalid data provided")
# Span is automatically marked as ERROR
# Exception details are recorded
return process_data(data)
```
```typescript TypeScript theme={null}
import { task } from "netra-sdk";
@task
async function riskyOperation(data: any) {
if (!data.valid) {
throw new Error("Invalid data provided");
// Span is automatically marked as ERROR
// Exception details are recorded
}
return processData(data);
}
```
## Combining Decorators with Manual Tracing
You can combine decorators with manual span operations for additional context:
```python Python theme={null}
from netra.decorators import workflow
from netra import Netra
@workflow
def process_order(order: dict):
# Add custom attributes to the current span
current_span = Netra.get_current_span()
if current_span:
current_span.set_attribute("order.id", order["id"])
current_span.set_attribute("order.total", order["total"])
# Add custom events
current_span.add_event("order-validated", {
"item_count": len(order["items"]),
})
result = fulfill_order(order)
if current_span:
current_span.add_event("order-fulfilled")
return result
```
```typescript TypeScript theme={null}
import { workflow, Netra } from "netra-sdk";
@workflow
async function processOrder(order: Order) {
// Add custom attributes to the current span
const currentSpan = Netra.getCurrentSpan();
currentSpan?.setAttribute("order.id", order.id);
currentSpan?.setAttribute("order.total", order.total);
// Add custom events
currentSpan?.addEvent("order-validated", {
itemCount: order.items.length,
});
const result = await fulfillOrder(order);
currentSpan?.addEvent("order-fulfilled");
return result;
}
```
## Best Practices
1. **Use semantic decorators** - Choose the decorator that best describes the operation's purpose (`@workflow` for processes, `@agent` for AI components, `@task` for individual operations).
2. **Name spans meaningfully** - Use the `name` parameter when the function name isn't descriptive enough.
3. **Don't over-instrument** - Focus on high-value operations. Not every function needs a decorator.
4. **Combine with auto-instrumentation** - Let auto-instrumentation handle LLM calls and database operations while using decorators for your application logic.
5. **Use class decoration sparingly** - Decorating an entire class instruments all methods, which may create noise. Consider decorating individual methods instead.
## Learn More
* [Manual Tracing](/Observability/Traces/manual-tracing) - Fine-grained control with SpanWrapper
* [Auto Instrumentation](/Observability/Traces/auto-instrumentation) - Zero-code tracing for supported libraries
* [Spans](/Observability/Traces/spans) - Understanding span types and attributes
# Manual Tracing
Source: https://docs.getnetra.ai/Observability/Traces/manual-tracing
Create custom spans in Netra with fine-grained control using SpanWrapper. Set prompts, model names, costs, and attributes on any operation in your app.
Manual tracing gives you complete control over span creation, attributes, and lifecycle. Use it when you need to trace custom operations, add detailed metadata, or track usage and costs.
## Getting Started
To start manual tracing, you'll need to:
1. Import the required classes from Netra
2. Create a new span using `start_span()`
3. Track your operations within the span
4. Add relevant attributes and events
## Creating Spans
Use `start_span()` to create a span that wraps a block of code. In Python, use it as a context manager. In TypeScript, explicitly call `end()` when done.
```python Python theme={null}
from netra import Netra
# Use as context manager (recommended)
with Netra.start_span("process-document") as span:
result = process_document(doc)
span.set_attribute("document.pages", result.page_count)
# Span automatically ends when exiting the context
```
```typescript TypeScript theme={null}
const parent = Netra.startSpan("parent").start();
await parent.withActiveAsync(async () => {
const child = Netra.startSpan("child").start();
await child.withActiveAsync(async () => {
Netra.addConversation(...);
});
child.end();
});
```
### Span Parameters
| Parameter | Type | Description |
| ---------------------------- | ----------- | ------------------------------------------- |
| `name` | string | Name of the span (required) |
| `attributes` | dict/object | Initial attributes to set on the span |
| `module_name` / `moduleName` | string | Module or component name for organization |
| `as_type` / `asType` | SpanType | Type of span (SPAN, GENERATION, TOOL, etc.) |
```python Python theme={null}
from netra import Netra, SpanType
with Netra.start_span(
"generate-summary",
attributes={
"input.length": len(document),
"model": "gpt-4",
},
module_name="summarization",
as_type=SpanType.GENERATION,
) as span:
# Your code here
pass
```
```typescript TypeScript theme={null}
import { Netra, SpanType } from "netra-sdk";
async function generateSummary(document: string) {
const span = Netra.startSpan(
"generate-summary",
{
"input.length": document.length,
model: "gpt-4",
},
"summarization",
SpanType.GENERATION
).start();
await span.withActiveAsync(async () => {
// Your code here
});
span.end();
}
```
## Span Types
Use the `as_type` parameter to categorize spans. This helps Netra display them correctly and enables type-specific features.
| Type | Use For |
| --------------------- | ---------------------------------------- |
| `SpanType.GENERATION` | LLM completions, image generation |
| `SpanType.EMBEDDING` | Vector embedding operations |
| `SpanType.TOOL` | Function calls, API requests, DB queries |
| `SpanType.AGENT` | AI agent reasoning and decisions |
| `SpanType.SPAN` | General operations (default) |
See [Spans](/Observability/Traces/spans#span-types) for detailed guidance on when to use each type.
## Local Span Blocking
You can block specific spans locally within a particular span scope. This is useful when you want to filter out noisy child spans (like HTTP requests) within a specific operation.
```python Python theme={null}
from netra import Netra
# Block POST spans within this scope
with Netra.start_span("image-generation", attributes={"blocked_spans": ["POST", "GET"]}) as span:
# HTTP spans named "POST" or "GET" created within this scope will be filtered
generate_image(prompt)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Block POST spans within this scope
const span = Netra.startSpan("image-generation", {
"blocked_spans": ["POST", "GET"],
});
// HTTP spans named "POST" or "GET" created within this scope will be filtered
await generateImage(prompt);
span.end();
```
This is different from global `blocked_spans` in `Netra.init()` which blocks spans across the entire application. Local blocking only affects spans created within the specific parent span's scope.
## SpanWrapper Methods
The `start_span()` function returns a `SpanWrapper` object with methods for adding context to your spans.
### Setting Span Attributes
Add custom key-value pairs to provide context about the operation:
```python Python theme={null}
with Netra.start_span("search-products") as span:
span.set_attribute("query", user_query)
span.set_attribute("filters.category", category)
span.set_attribute("filters.price_range", [min_price, max_price])
span.set_attribute("results.count", len(results))
```
```typescript TypeScript theme={null}
const span = Netra.startSpan("search-products");
span.setAttribute("query", userQuery);
span.setAttribute("filters.category", category);
span.setAttribute("filters.priceRange", [minPrice, maxPrice]);
span.setAttribute("results.count", results.length);
span.end();
```
### LLM-Specific Attributes
For LLM operations, use dedicated methods to set prompts, models, and system information:
```python Python theme={null}
from netra import Netra, SpanType
with Netra.start_span("generate-response", as_type=SpanType.GENERATION) as span:
span.set_prompt(user_message)
span.set_negative_prompt("blurry, low quality") # For image generation
span.set_model("gpt-4-turbo")
span.set_llm_system("openai")
response = generate_response(user_message)
span.set_attribute("completion", response.content)
```
```typescript TypeScript theme={null}
import { Netra, SpanType } from "netra-sdk";
const span = Netra.startSpan("generate-response", {}, undefined, SpanType.GENERATION);
span.setPrompt(userMessage);
span.setNegativePrompt("blurry, low quality"); // For image generation
span.setModel("gpt-4-turbo");
span.setLlmSystem("openai");
const response = await generateResponse(userMessage);
span.setAttribute("completion", response.content);
span.end();
```
### Recording Events
Track significant moments within a span's lifecycle:
```python Python theme={null}
with Netra.start_span("order-processing") as span:
span.add_event("validation-started")
validate_order(order)
span.add_event("validation-completed", {"valid": True})
span.add_event("payment-started")
payment = process_payment(order)
span.add_event("payment-completed", {
"transaction_id": payment.id,
"amount": payment.amount,
})
```
```typescript TypeScript theme={null}
const span = Netra.startSpan("order-processing");
span.addEvent("validation-started");
await validateOrder(order);
span.addEvent("validation-completed", { valid: true });
span.addEvent("payment-started");
const payment = await processPayment(order);
span.addEvent("payment-completed", {
transactionId: payment.id,
amount: payment.amount,
});
span.end();
```
## Tracking Usage Data
Use `UsageModel` to track token usage and costs for LLM operations:
```python Python theme={null}
from netra import Netra, SpanType, UsageModel
with Netra.start_span("llm-call", as_type=SpanType.GENERATION) as span:
response = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
)
# Track usage
span.set_usage([
UsageModel(
model="gpt-4",
cost_in_usd=calculate_cost(response.usage),
usage_type="chat",
units_used=1,
)
])
```
```typescript TypeScript theme={null}
import { Netra, SpanType, UsageModel } from "netra-sdk";
const span = Netra.startSpan("llm-call", {}, undefined, SpanType.GENERATION);
const response = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: prompt }],
});
// Track usage
span.setUsage([
{
model: "gpt-4",
costInUsd: calculateCost(response.usage),
usageType: "chat",
unitsUsed: 1,
},
]);
span.end();
```
### UsageModel Fields
| Field | Type | Description |
| --------------------------- | ------ | ------------------------------------------------- |
| `model` | string | Model name used |
| `cost_in_usd` / `costInUsd` | float | Calculated cost in USD |
| `usage_type` / `usageType` | string | Type of usage (e.g., "chat", "image\_generation") |
| `units_used` / `unitsUsed` | int | Number of units consumed |
## Adding Action Tracking
Use `ActionModel` to track discrete actions, tool calls, or database operations within a span:
```python Python theme={null}
from netra import Netra, ActionModel
with Netra.start_span("agent-execution") as span:
# Record actions taken by the agent
span.set_action([
ActionModel(
action="DB",
action_type="INSERT",
affected_records=[
{"record_id": "user_123", "record_type": "user"},
{"record_id": "profile_456", "record_type": "profile"},
],
metadata={
"table": "users",
"operation_id": "tx_789",
"duration_ms": "45",
},
success=True,
),
ActionModel(
action="API",
action_type="CALL",
metadata={
"endpoint": "/api/v1/process",
"method": "POST",
"status_code": "200",
},
success=True,
),
])
```
```typescript TypeScript theme={null}
import { Netra, ActionModel } from "netra-sdk";
const span = Netra.startSpan("agent-execution");
// Record actions taken by the agent
span.setAction([
{
action: "DB",
actionType: "INSERT",
affectedRecords: [
{ recordId: "user_123", recordType: "user" },
{ recordId: "profile_456", recordType: "profile" },
],
metadata: {
table: "users",
operationId: "tx_789",
durationMs: "45",
},
success: true,
},
{
action: "API",
actionType: "CALL",
metadata: {
endpoint: "/api/v1/process",
method: "POST",
statusCode: "200",
},
success: true,
},
]);
span.end();
```
### ActionModel Fields
| Field | Type | Description |
| -------------------------------------- | ----------- | ----------------------------------------------------------- |
| `action` | string | Action category (e.g., "DB", "API", "CACHE") |
| `action_type` / `actionType` | string | Action subtype (e.g., "INSERT", "SELECT", "CALL") |
| `affected_records` / `affectedRecords` | array | List of affected records with `record_id` and `record_type` |
| `metadata` | dict/object | Additional metadata as key-value pairs |
| `success` | boolean | Whether the action succeeded |
## Error Handling
Mark spans as errors when operations fail:
```python Python theme={null}
from netra import Netra
with Netra.start_span("risky-operation") as span:
try:
result = risky_operation()
span.set_success()
except Exception as e:
span.set_error(str(e))
raise
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
const span = Netra.startSpan("risky-operation");
try {
const result = await riskyOperation();
span.setSuccess();
span.end();
return result;
} catch (error) {
span.setError(error.message);
span.end();
throw error;
}
```
When using Python's context manager, exceptions are automatically recorded and the span is marked as an error. You can still explicitly call `set_error()` for custom error messages.
## Nested Spans
Create hierarchical traces by nesting spans. Child spans automatically inherit the parent context:
```python Python theme={null}
from netra import Netra
def process_order(order: dict):
with Netra.start_span("process-order") as parent_span:
parent_span.set_attribute("order.id", order["id"])
# Child span for validation
with Netra.start_span("validate-order"):
validate_order(order)
# Child span for payment
with Netra.start_span("process-payment") as payment_span:
payment = process_payment(order)
payment_span.set_attribute("payment.id", payment.id)
# Child span for fulfillment
with Netra.start_span("fulfill-order"):
fulfill_order(order)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
async function processOrder(order: Order) {
const parentSpan = Netra.startSpan("process-order");
parentSpan.setAttribute("order.id", order.id);
try {
// Child span for validation
const validateSpan = Netra.startSpan("validate-order");
await validateOrder(order);
validateSpan.end();
// Child span for payment
const paymentSpan = Netra.startSpan("process-payment");
const payment = await processPayment(order);
paymentSpan.setAttribute("payment.id", payment.id);
paymentSpan.end();
// Child span for fulfillment
const fulfillSpan = Netra.startSpan("fulfill-order");
await fulfillOrder(order);
fulfillSpan.end();
parentSpan.end();
} catch (error) {
parentSpan.setError(error.message);
parentSpan.end();
throw error;
}
}
```
## Accessing the Current Span
Get the currently active span to add attributes from anywhere in your code:
```python Python theme={null}
from netra import Netra
def log_user_action(action: str):
current_span = Netra.get_current_span()
if current_span:
current_span.add_event("user-action", {"action": action})
# Usage within a traced operation
with Netra.start_span("user-session"):
# ... somewhere deep in the call stack ...
log_user_action("clicked-submit")
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
function logUserAction(action: string) {
const currentSpan = Netra.getCurrentSpan();
if (currentSpan) {
currentSpan.addEvent("user-action", { action });
}
}
// Usage within a traced operation
const span = Netra.startSpan("user-session");
// ... somewhere deep in the call stack ...
logUserAction("clicked-submit");
span.end();
```
## Example: RAG Pipeline
This example demonstrates nested spans with multiple span types - a common pattern for AI pipelines.
```python Python theme={null}
from netra import Netra, SpanType, UsageModel
def rag_pipeline(query: str):
with Netra.start_span("rag-pipeline") as pipeline_span:
pipeline_span.set_attribute("query", query)
# Step 1: Generate embedding
with Netra.start_span(
"generate-embedding", as_type=SpanType.EMBEDDING
) as embed_span:
embedding = embed_model.embed(query)
embed_span.set_usage([
UsageModel(
model="text-embedding-3-small",
usage_type="embedding",
units_used=1
)
])
# Step 2: Retrieve documents
with Netra.start_span(
"retrieve-documents", as_type=SpanType.TOOL
) as retrieve_span:
documents = vector_store.search(embedding, top_k=5)
retrieve_span.set_attribute("documents.count", len(documents))
# Step 3: Generate response
with Netra.start_span(
"generate-response", as_type=SpanType.GENERATION
) as generate_span:
generate_span.set_prompt(query)
generate_span.set_model("gpt-4")
generate_span.set_llm_system("openai")
response = openai.chat.completions.create(
model="gpt-4",
messages=[
{"role": "system", "content": build_context(documents)},
{"role": "user", "content": query},
],
)
generate_span.set_usage([
UsageModel(
model="gpt-4",
cost_in_usd=calculate_cost(response.usage),
usage_type="chat",
units_used=1,
)
])
pipeline_span.set_success()
return response.choices[0].message.content
```
```typescript TypeScript theme={null}
import { Netra, SpanType, UsageModel } from "netra-sdk";
async function ragPipeline(query: string) {
const pipelineSpan = Netra.startSpan("rag-pipeline");
pipelineSpan.setAttribute("query", query);
try {
// Step 1: Generate embedding
const embedSpan = Netra.startSpan("generate-embedding", {}, undefined, SpanType.EMBEDDING);
const embedding = await embedModel.embed(query);
embedSpan.setUsage([
{ model: "text-embedding-3-small", usageType: "embedding", unitsUsed: 1 },
]);
embedSpan.end();
// Step 2: Retrieve documents
const retrieveSpan = Netra.startSpan("retrieve-documents", {}, undefined, SpanType.TOOL);
const documents = await vectorStore.search(embedding, { topK: 5 });
retrieveSpan.setAttribute("documents.count", documents.length);
retrieveSpan.end();
// Step 3: Generate response
const generateSpan = Netra.startSpan("generate-response", {}, undefined, SpanType.GENERATION);
generateSpan.setPrompt(query);
generateSpan.setModel("gpt-4");
generateSpan.setLlmSystem("openai");
const response = await openai.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "system", content: buildContext(documents) },
{ role: "user", content: query },
],
});
generateSpan.setUsage([
{
model: "gpt-4",
costInUsd: calculateCost(response.usage),
usageType: "chat",
unitsUsed: 1,
},
]);
generateSpan.end();
pipelineSpan.setSuccess();
pipelineSpan.end();
return response.choices[0].message.content;
} catch (error) {
pipelineSpan.setError(error.message);
pipelineSpan.end();
throw error;
}
}
```
## Best Practices
1. **Use context managers in Python** - They ensure spans are properly closed even when exceptions occur.
2. **End spans in TypeScript** - Always call `span.end()` in both success and error paths, preferably in a `finally` block.
3. **Add meaningful attributes** - Include information that will help you debug and analyze traces later.
4. **Track usage for LLM calls** - Use `setUsage()` to monitor token consumption and costs.
5. **Use appropriate span types** - Set `as_type` to categorize spans correctly (GENERATION for LLM calls, TOOL for function calls, etc.).
6. **Handle errors explicitly** - Call `setError()` with descriptive messages to make debugging easier.
7. **Use local span blocking** - Filter noisy child spans when you only care about the parent operation.
8. **Add events for milestones** - Use `addEvent()` to mark important points in long-running operations.
## Learn More
* [Decorators](/Observability/Traces/decorators) - Simpler instrumentation with decorators
* [Auto Instrumentation](/Observability/Traces/auto-instrumentation) - Zero-code tracing
* [Initialization](/Observability/Traces/configuration/initialization) - Configure tracing behavior
# Traces Overview
Source: https://docs.getnetra.ai/Observability/Traces/overview
Understand distributed tracing for AI applications in Netra. Learn how traces and spans capture every LLM call, tool use, and retrieval operation.
## What is a Trace?
A trace represents the complete journey of a single request through your AI application. It captures every operation involved in handling that request—from the initial user input to the final response—including LLM calls, retrieval steps, tool executions, and database queries.
Each trace is composed of **spans**, which represent discrete operations within the request lifecycle. Spans are organized hierarchically, showing parent-child relationships that reveal how operations are nested and sequenced.
Netra's tracing is built on [OpenTelemetry](https://opentelemetry.io/) standards, an industry-standard observability framework. This ensures compatibility with any OTLP-compliant backend and allows you to leverage existing OpenTelemetry tooling and instrumentation.
## Why Tracing Matters for AI Applications
Traditional logging captures isolated events. Tracing connects them into a coherent story, which is essential for AI systems where:
* **Multiple LLM calls** may happen in sequence or parallel
* **Agent decisions** trigger unpredictable tool executions
* **RAG pipelines** combine retrieval, embedding, and generation steps
* **Costs accumulate** across many model invocations
With traces, you can:
| Capability | What It Enables |
| ------------------------ | ----------------------------------------------------------------- |
| **Diagnose failures** | Pinpoint exactly where and why a request failed |
| **Analyze latency** | Identify slow operations across the request lifecycle |
| **Audit agent behavior** | Review the sequence of decisions and actions in agentic workflows |
| **Track costs** | Correlate token usage and costs to specific requests |
| **Debug prompts** | See exact inputs and outputs for each LLM call |
## Getting Started with Tracing
Setting up tracing in Netra takes just a few lines of code:
```python Python theme={null}
from netra import Netra
Netra.init(
app_name="my-ai-app",
environment="production",
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
```
Once initialized, Netra automatically instruments a **curated default set** of supported LLM providers, agent frameworks, vector databases, and web frameworks. Only AI-relevant libraries are instrumented out of the box, not every library in your environment. Your traces will appear in the Netra dashboard within seconds.
Follow the step-by-step guide to see your first trace
## Viewing Traces in Netra
Access traces from the Netra dashboard by navigating to **Observability → Traces**.
The Traces page provides:
* **Trace list** - All captured traces with timestamps, duration, status, and token usage
* **Search and filters** - Find traces by name, time range, status, or custom attributes
* **Configurable columns** - Customize which fields are visible
* **Saved views** - Persist filter and column configurations for recurring workflows
Selecting a trace opens the **Trace Timeline**, showing the hierarchical span structure, timing waterfall, and detailed metadata for each operation.
## Tracing Methods
Netra offers three ways to add tracing to your application:
| Method | Best For | Effort |
| ------------------------------------------------------------------ | -------------------------------------------------- | -------------------- |
| [Auto Instrumentation](/Observability/Traces/auto-instrumentation) | Quick setup with curated defaults for AI libraries | Zero code changes |
| [Decorators](/Observability/Traces/decorators) | Functions and classes | Minimal code changes |
| [Manual Tracing](/Observability/Traces/manual-tracing) | Custom operations, fine-grained control | Full control |
## Learn More
Understand span types, attributes, and hierarchy
Configure tracing behavior and export options
### Tutorials
* [Comparing Traces](/Tutorials/comparing-traces) - Debug regressions by comparing traces side-by-side
* [Custom Trace Views](/Tutorials/custom-trace-views) - Create saved views for your team's workflows
# Spans
Source: https://docs.getnetra.ai/Observability/Traces/spans
Learn about span types, attributes, and hierarchy in Netra traces. Understand GENERATION, TOOL, AGENT, and EMBEDDING spans for AI observability.
A span represents a single unit of execution within a trace. Each span captures a discrete operation performed during a request lifecycle, such as an API call, model invocation, or tool execution, along with its timing, cost, and execution context.
Spans are organized hierarchically within a trace, allowing you to understand how individual operations contribute to overall latency, cost, and system behavior.
## Span Types
Netra supports five span types, each designed to categorize specific kinds of operations:
| Span Type | Enum Value | Description | Use Case |
| -------------- | --------------------- | ----------------------------------- | ----------------------------------------------------------- |
| **Span** | `SpanType.SPAN` | Default type for general operations | Function calls, business logic, generic operations |
| **Generation** | `SpanType.GENERATION` | LLM text or image generation | Chat completions, text generation, image generation |
| **Embedding** | `SpanType.EMBEDDING` | Vector embedding operations | Text-to-vector conversions, embedding API calls |
| **Tool** | `SpanType.TOOL` | Tool or function execution | Function calls, API requests, database queries |
| **Agent** | `SpanType.AGENT` | AI agent reasoning or decisions | Agent orchestration, multi-step reasoning, autonomous tasks |
### When to Use Each Type
**Span (Default)**
* General-purpose operations that don't fit other categories
* Business logic and data processing
* Workflow orchestration steps
**Generation**
* OpenAI, Anthropic, or other LLM completions
* Image generation (DALL-E, Stable Diffusion)
* Text-to-speech or speech-to-text operations
**Embedding**
* Converting text to vectors for similarity search
* Embedding API calls (OpenAI embeddings, Cohere embed)
* Document embedding pipelines
**Tool**
* External API calls
* Database operations
* File system operations
* Any function that performs a specific action
**Agent**
* AI agents making autonomous decisions
* ReAct-style reasoning loops
* Multi-step task execution
* Agent frameworks (LangGraph, CrewAI, AutoGen)
### Specifying Span Types
When creating manual spans, specify the type using the `as_type` parameter:
```python Python theme={null}
from netra import Netra, SpanType
# Generation span for LLM calls
with Netra.start_span("chat-completion", as_type=SpanType.GENERATION) as span:
pass
# Embedding span for vector operations
with Netra.start_span("generate-embedding", as_type=SpanType.EMBEDDING) as span:
pass
# Tool span for function calls
with Netra.start_span("search-database", as_type=SpanType.TOOL) as span:
pass
# Agent span for AI agents
with Netra.start_span("research-agent", as_type=SpanType.AGENT) as span:
pass
```
```typescript TypeScript theme={null}
import { Netra, SpanType } from "netra-sdk";
// Generation span for LLM calls
const genSpan = Netra.startSpan("chat-completion", {}, undefined, SpanType.GENERATION);
// Embedding span for vector operations
const embedSpan = Netra.startSpan("generate-embedding", {}, undefined, SpanType.EMBEDDING);
// Tool span for function calls
const toolSpan = Netra.startSpan("search-database", {}, undefined, SpanType.TOOL);
// Agent span for AI agents
const agentSpan = Netra.startSpan("research-agent", {}, undefined, SpanType.AGENT);
```
Auto-instrumentation automatically assigns the correct span type based on the operation. For example, OpenAI chat completions are automatically marked as `GENERATION` spans.
## Viewing Spans in Netra
Netra displays spans within the Trace Timeline as a hierarchical view of all operations executed during a request. Selecting a span highlights it in the timeline and surfaces its details in the **Metadata** panel, allowing you to inspect execution context without leaving the trace.
Different span types are visually distinguished to help you quickly identify agent execution, model generations, and tool calls.
## Span Attributes
Each span in Netra captures attributes in the **Metadata** panel. Some attributes are common across all span types, while others are specific to certain types.
| Attribute | Description | Applies To |
| ---------------------------------------- | ------------------------------------------ | ---------- |
| `library.name` | Name of the tracing library | All |
| `library.version` | Version of the tracing library | All |
| `sdk.name` | Name of the SDK | All |
| `netra.session_id` | Session identifier (if specified) | All |
| `netra.user_id` | User identifier (if specified) | All |
| `netra.duration_ms` | Duration of the operation in milliseconds | All |
| `netra.status` | Success or error status | All |
| `span_type` | Type of the span | All |
| `netra.instrumentation.name` | LLM provider (e.g., "openai", "anthropic") | Generation |
| `gen_ai.request.model` | Model used for the request | Generation |
| `gen_ai.response.model` | Model returned in the response | Generation |
| `gen_ai.usage.prompt_tokens` | Number of input tokens | Generation |
| `gen_ai.usage.completion_tokens` | Number of output tokens | Generation |
| `llm.usage.total_tokens` | Total tokens used | Generation |
| `llm.response.duration` | Response time from the LLM | Generation |
| `gen_ai.performance.time_to_first_token` | Time to receive the first token | Generation |
| `netra.model` | The embedding model used | Embedding |
| `cost.total_usd` | Total cost in USD | Agent |
Generation spans also show additional data in **Chat Preview** including system prompt, user input, and assistant response.
### Custom Attributes
You can add custom attributes to any span for additional context using `set_attribute()` / `setAttribute()`. See [Manual Tracing](/Observability/Traces/manual-tracing#setting-span-attributes) for detailed examples of working with span attributes.
## Adding Spans to Datasets
Spans can be added to a dataset using **Add to Dataset** for reuse in evaluations and testing workflows. This is useful for:
* Building evaluation datasets from production data
* Creating test cases for regression testing
* Analyzing specific types of operations
See [Evaluations](/Evaluation/Evaluation-overview) for more details.
## Learn More
* [Manual Tracing](/Observability/Traces/manual-tracing) - Create custom spans with full control
* [Decorators](/Observability/Traces/decorators) - Use decorators for cleaner instrumentation
* [Auto Instrumentation](/Observability/Traces/auto-instrumentation) - Zero-code tracing for supported libraries
# Users
Source: https://docs.getnetra.ai/Observability/Users
Track individual end users interacting with your AI app in Netra. Associate traces with user IDs to analyze behavior, costs, and quality per user.
The Users view in Netra provides a list of all end users who have interacted with your AI applications. It helps you understand user activity, correlate behavior with system usage, and debug user-specific issues.
## Viewing Users
Access the Users view from **Observability → Users** in the Netra dashboard.
For each user, Netra displays:
| Field | Description |
| --------------------- | ----------------------------------------------- |
| **User ID** | The identifier you assigned via `setUserId()` |
| **Tenant** | The tenant this user belongs to (if configured) |
| **First Interaction** | When the user first interacted with the system |
| **Last Interaction** | When the user was most recently active |
| **Sessions** | Number of sessions generated by this user |
| **Traces** | Total number of traces from this user |
| **Total Cost** | Estimated cost incurred by this user's activity |
### User Actions
Each user entry provides:
* **View Traces** - Opens the Traces view filtered to this user, showing all their requests and executions
## Configuring User Tracking
User tracking is configured by calling `setUserId()` in your application code. Once set, all subsequent traces will be associated with that user.
### Setting the User ID
Call `setUserId()` after initializing Netra and when you know the user's identity (e.g., after authentication):
```python Python theme={null}
from netra import Netra
# Initialize Netra
Netra.init(
app_name="my-ai-app",
environment="production",
)
# After user authentication
def on_user_login(user):
Netra.set_user_id(user.id)
# All subsequent traces will be associated with this user
response = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": query}],
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
// Initialize Netra
await Netra.init({
appName: "my-ai-app",
environment: "production",
});
// After user authentication
function onUserLogin(user: User) {
Netra.setUserId(user.id);
}
// All subsequent traces will be associated with this user
const response = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: query }],
});
```
### User ID in Web Frameworks
For web applications, set the user ID in your request middleware:
```python Python (FastAPI) theme={null}
from netra import Netra
from fastapi import FastAPI, Depends, Request
app = FastAPI()
async def set_user_context(request: Request):
if hasattr(request.state, "user") and request.state.user:
Netra.set_user_id(request.state.user.id)
@app.post("/api/chat")
async def chat(message: str, _=Depends(set_user_context)):
# This trace will be associated with the authenticated user
response = await process_chat(message)
return response
```
```typescript TypeScript (Express) theme={null}
import { Netra } from "netra-sdk";
import express from "express";
const app = express();
// Middleware to set user context
app.use((req, res, next) => {
if (req.user?.id) {
Netra.setUserId(req.user.id);
}
next();
});
app.post("/api/chat", async (req, res) => {
// This trace will be associated with the authenticated user
const response = await processChat(req.body.message);
res.json(response);
});
```
### Combining with Tenant and Session
For multi-tenant applications, combine user tracking with tenant and session IDs:
```python Python theme={null}
from netra import Netra
def initialize_user_context(user, tenant, session_id: str):
# Set all context attributes
Netra.set_user_id(user.id)
Netra.set_tenant_id(tenant.id)
Netra.set_session_id(session_id)
# Optionally add custom attributes
Netra.set_custom_attributes("user.plan", user.plan)
Netra.set_custom_attributes("user.region", user.region)
# All traces will now include user, tenant, and session context
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
function initializeUserContext(user: User, tenant: Tenant, sessionId: string) {
// Set all context attributes
Netra.setUserId(user.id);
Netra.setTenantId(tenant.id);
Netra.setSessionId(sessionId);
// Optionally add custom attributes
Netra.setCustomAttributes("user.plan", user.plan);
Netra.setCustomAttributes("user.region", user.region);
}
// All traces will now include user, tenant, and session context
```
## User Context Methods
| Method | Description |
| --------------------------------- | ------------------------------------------------------ |
| `setUserId(userId)` | Associate traces with a user identifier |
| `setTenantId(tenantId)` | Associate traces with a tenant (for multi-tenant apps) |
| `setSessionId(sessionId)` | Group traces into a session |
| `setCustomAttributes(key, value)` | Add custom metadata to traces |
## Use Cases
### Per-User Cost Analysis
Track costs incurred by each user to understand usage patterns and implement usage-based billing:
1. Navigate to **Observability → Users**
2. Sort by **Total Cost** to find highest-cost users
3. Click **View Traces** to understand their usage patterns
### Debugging User-Specific Issues
When a user reports an issue:
1. Search for the user by their ID
2. Click **View Traces** to see their recent activity
3. Identify failed or slow traces
4. Drill into specific traces to debug
### User Activity Monitoring
Track user engagement with your AI features:
* **Sessions count** - How often users return
* **Traces count** - How much they use AI features
* **Last interaction** - When they were last active
## Best Practices
1. **Use stable user IDs** - Use your application's user ID, not ephemeral session tokens
2. **Set user ID early** - Call `setUserId()` as soon as authentication completes
3. **Handle anonymous users** - For unauthenticated users, consider using a consistent anonymous ID or skip user tracking
4. **Combine with sessions** - Use session IDs to group related traces within a user's activity
5. **Add relevant attributes** - Include user plan, region, or other attributes useful for analysis
## Privacy Considerations
* User IDs are stored in Netra and appear in the dashboard
* Avoid using PII (email, name) as user IDs—use opaque identifiers instead
* Consider your data retention requirements when storing user activity
## Next Steps
* [Sessions](/Observability/Session) - Group user interactions into sessions
* [Tenants](/Observability/Tenants) - Multi-tenant user organization
* [Traces Overview](/Observability/Traces/overview) - Understanding trace data
# Setup
Source: https://docs.getnetra.ai/Redteaming/Configs
Create a red team setup that connects your agent to evaluators, attacker LLM, and judge LLM. Control turn mode, number of tests, and prompt generation.
A setup is a reusable test configuration for red teaming. It connects a specific agent to a set of [evaluators](/Redteaming/Suites-and-Evaluators), an attacker LLM for generating adversarial prompts, and a judge LLM for scoring responses. You can create multiple setups per agent to test different risk profiles or attack strategies.
## Why Setups Matter
Setups make adversarial testing repeatable and systematic:
| Challenge | How Setups Help |
| -------------------------------------------- | ---------------------------------------------------------------- |
| Need to test the same agent repeatedly | Save evaluators, LLM settings, and turn mode as a reusable setup |
| Different risk profiles for different agents | Create separate setups with different evaluator selections |
| Multi-turn attacks require specific settings | Configure conversation depth and attack strategy per setup |
| Consistent testing across releases | Re-run the same setup after each agent update |
## Setup Fields
| Field | Description | Default |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------- |
| **Agent** | The remote agent to test. Selected from your registered agents. | Required |
| **Selected Evaluators** | Attack categories to test. Pick from [suites](/Redteaming/Suites-and-Evaluators) or select individual evaluators. | Required |
| **Turn Type** | `single` for one-shot attacks, `multi` for multi-turn conversations. | `single` |
| **No. of Tests** | Number of adversarial prompts generated per evaluator. | 2 |
| **Multi-Turn Count** | Number of conversation turns in multi-turn mode. Ignored in single-turn mode. | 2 |
| **Attacker LLM Provider** | The LLM provider used to generate adversarial prompts. | Required |
| **Attacker LLM Model** | The specific model for adversarial prompt generation. | Required |
| **Judge LLM Provider** | The LLM provider used to score agent responses. | Required |
| **Judge LLM Model** | The specific model for response judging. | Required |
## Creating a Setup
Enter a name for the red team setup and choose the remote agent you want to test. The agent must be registered in your project.
Choose the attack strategy:
* **Single-turn**: Each adversarial prompt is sent once. Best for testing individual safety guardrails.
* **Multi-turn**: The attacker LLM sustains a conversation over multiple turns. Best for testing escalation resistance and context manipulation.
Define how many adversarial prompts to generate per evaluator. More tests provide broader coverage but increase run duration.
Select the **attacker LLM** (generates adversarial prompts) and the **judge LLM** (scores agent responses). You can use different providers and models for each role.
Select evaluators from the [catalog](/Redteaming/Suites-and-Evaluators). You can:
* Select a **suite** to include all its evaluators
* Pick **individual evaluators** for a custom test
* Combine both — start with a suite and add or remove evaluators as needed
Save your setup. You can now generate prompts and start runs.
## Prompt Generation
Adversarial prompts are generated asynchronously after you create a setup. Netra uses the attacker LLM and each evaluator's generation template to create targeted attack prompts.
### How It Works
1. Auto triggers prompt generation from the setup page.
2. For each selected evaluator, Netra renders the evaluator's template with the target agent's **purpose** (derived from its application details).
3. The attacker LLM generates the configured number of adversarial prompts per evaluator.
4. Generated prompts are stored and ready for use in [runs](/Redteaming/Runs).
### Generation Status
Track prompt generation progress per evaluator. Each evaluator shows:
| Status | Meaning |
| --------------- | --------------------------------------------- |
| **Pending** | Generation has not started for this evaluator |
| **In Progress** | The attacker LLM is generating prompts |
| **Completed** | All prompts generated successfully |
| **Failed** | Generation failed for this evaluator |
### Regenerating Prompts
You can regenerate prompts at any time from the setup page. Regeneration replaces existing prompts with fresh ones.
You cannot regenerate prompts while a run is active for the setup. Wait for the run to complete or cancel it first.
## Updating a Setup
You can update any setup field after creation. However, certain changes invalidate previously generated prompts:
### Fields That Invalidate Prompts
Changes to these fields automatically clear generated prompts. You must regenerate prompts before the next run.
* **Selected Evaluators** — Adding or removing evaluators changes the prompt set
* **No. of Tests** — Changing the count requires new prompts
* **Attacker LLM Provider or Model** — Different models produce different prompts
### Fields That Do Not Invalidate Prompts
Changes to these fields preserve existing prompts:
* **Judge LLM Provider or Model** — Only affects scoring, not prompt generation
* **Turn Type** — Changes execution mode but not the prompts themselves
* **Multi-Turn Count** — Changes conversation depth but not initial prompts
You cannot update a setup while a run is active. Wait for the run to complete or cancel it first.
## Deleting a Setup
Setups support soft deletion. Deleting a setup removes it from the dashboard but preserves its run history for auditing purposes.
## Cloning a Setup
You can clone an existing setup to create a copy with the same evaluators, attack type, and LLM settings. This is useful when you want to:
* Test a different agent with the same evaluator selection
* Create a variation of an existing setup with minor changes
* Reuse a proven setup across multiple agents
The cloned setup appears in the sidebar with a "Copy of" prefix. You can rename it and modify any field after cloning.
Clone a setup before making major changes. This preserves your original setup and its run history while letting you experiment with a new variation.
## Related
Browse the adversarial test catalog and select attack categories.
Execute your setup and view per-prompt results.
Track safety scores aggregated from your setup's runs.
Understand the full red teaming framework and end-to-end flow.
# Overview
Source: https://docs.getnetra.ai/Redteaming/Redteaming-overview
Test your AI agents for safety vulnerabilities with Netra's red teaming framework. Run adversarial attacks, judge responses, and track safety scores over time.
AI agents can be manipulated. Adversarial users craft prompts that bypass safety guardrails, extract system prompts, and force harmful outputs. Netra's Red Teaming framework lets you find these vulnerabilities before your users do — systematically, repeatedly, and at scale.
## Why Red Teaming Matters
Without adversarial testing, you cannot know how your agent behaves under attack. Netra helps you answer critical safety questions with confidence:
| Question | What Netra Tests |
| ---------------------------------------- | --------------------------------------------------- |
| Can my agent be jailbroken? | Adversarial attacks across multiple risk categories |
| Does my agent leak its system prompt? | System prompt leakage evaluators |
| Is my agent safe across risk categories? | Suites aligned to security frameworks |
| How has safety changed over time? | Risk score trending with historical comparison |
## Core Building Blocks
The Red Teaming suite is built on three interconnected pillars:
### Suites & Evaluators
[Suites & Evaluators](/Redteaming/Suites-and-Evaluators) form the adversarial test catalog. **Evaluators** define individual attack categories — each with a generation template for creating adversarial prompts and a grader rubric for judging agent responses. **Suites** bundle related evaluators into curated groups aligned to security frameworks.
### Setup
A [Setup](/Redteaming/Configs) is a reusable test configuration. It connects a specific agent to selected evaluators, an attacker LLM (for generating adversarial prompts), and a judge LLM (for scoring agent responses). You also choose between single-turn and multi-turn attack modes.
### Runs & Results
A [Run](/Redteaming/Runs) executes all adversarial prompts from a setup against the target agent. Each prompt is sent to the agent, the response is judged by the LLM, and a pass/fail/error result is recorded. Runs execute asynchronously with queue-based concurrency control.
## Execution Modes
Netra supports two attack strategies, each suited to different risk scenarios:
| Mode | When Used | Behavior |
| --------------- | ------------------------------- | ---------------------------------------------------------------------------------------- |
| **Single-turn** | Default for standard evaluators | One adversarial prompt is sent to the agent. The judge scores the single response. |
| **Multi-turn** | Turn type set to multi | The attacker LLM sustains a conversation for N turns. Each turn is judged independently. |
## End-to-End Flow
The red teaming pipeline follows this sequence:
[Create a setup](/Redteaming/Configs) for your agent. Choose the attack type, number of tests, and configure the attacker and judge LLMs.
Explore [suites & evaluators](/Redteaming/Suites-and-Evaluators) and select the attack categories that match your risk profile.
Trigger [prompt generation](/Redteaming/Configs#prompt-generation). The attacker LLM generates targeted adversarial prompts for each evaluator.
[Start a run](/Redteaming/Runs). Each prompt is sent to the agent, the response is judged, and a pass/fail result is recorded.
View per-prompt [results](/Redteaming/Runs#viewing-results) and monitor your [risk scores](/Redteaming/Risk-Scores) over time.
## Getting Started
[Create a setup](/Redteaming/Configs) for your agent. Choose attack type, number of tests, and configure the attacker and judge LLMs.
Navigate to **Red Teaming** and explore [suites & evaluators](/Redteaming/Suites-and-Evaluators). Select a suite or pick individual evaluators that match your risk profile.
Trigger [prompt generation](/Redteaming/Configs#prompt-generation). Netra uses the attacker LLM and each evaluator's template to generate targeted adversarial prompts.
[Start a run](/Redteaming/Runs). Each adversarial prompt is sent to your agent. The judge LLM scores every response.
View per-prompt [results](/Redteaming/Runs#viewing-results), track [progress](/Redteaming/Runs#progress-tracking), and monitor your [risk scores](/Redteaming/Risk-Scores) over time.
## Related
Browse the adversarial test catalog and select attack categories.
Create reusable test setups that connect agents to evaluators.
Execute red team tests and view per-prompt pass/fail results.
Track your agent's safety posture over time with aggregated scores.
# Risk Scores & Progress
Source: https://docs.getnetra.ai/Redteaming/Risk-Scores
Track your AI agent's safety posture over time. View aggregated risk scores by suite and monitor score changes.
Risk scores aggregate your [run results](/Redteaming/Runs) into a single safety posture metric. They give you a high-level view of how safe your agent is, how safety has changed over time, and which areas need attention. Scores are broken down by [suite](/Redteaming/Suites-and-Evaluators) so you can identify exactly where vulnerabilities exist.
## Why Risk Scores Matter
Individual pass/fail results are useful for debugging, but risk scores answer the bigger questions:
| Question | What Risk Scores Show |
| --------------------------------- | ----------------------------------------------------------- |
| How safe is my agent overall? | A single safety percentage computed from all completed runs |
| Is safety improving or degrading? | Score change delta between the latest and previous runs |
| Which risk areas need work? | Per-suite breakdown showing scores by evaluator group |
| How many evaluators are failing? | Count of evaluators with at least one vulnerable result |
## Understanding Risk Scores
Risk scores are computed from all completed [runs](/Redteaming/Runs) for a given [setup](/Redteaming/Configs).
### Score Fields
| Field | Description |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Latest Safety Score** | Overall safety percentage from the most recent completed run. |
| **Score Change** | Difference between the latest safety score and the previous run's score. A positive change means safety improved. |
| **Failing Evaluators** | Number of evaluators that have at least one vulnerable result in the latest run. |
| **Score History** | Historical safety scores across all completed runs, shown as a time series. |
### How Scores Are Computed
Each evaluator produces individual pass/fail results per adversarial prompt. Scores are built up from these results in three layers:
| Level | Calculation |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| **Evaluator Score** | Percentage of tests passed for that evaluator. For example, if an evaluator runs 3 prompts and 2 pass, its score is 67%. |
| **Suite Score** | Average of the evaluator scores within that suite. |
| **Overall Safety Score** | Average of all distinct evaluator test pass percentages across the run. |
Results with **error** or **cancelled** status are excluded from score computation.
Scores are tracked across runs to show trends over time — whether your agent's safety posture is improving or degrading.
A safety score of 100% means the agent blocked every adversarial attack in the run. A score of 0% means the agent was vulnerable to every attack.
## Suite Breakdown
Risk scores are grouped by [suite](/Redteaming/Suites-and-Evaluators). This gives you a per-category view of your agent's safety posture.
### How Suites Are Determined
| Scenario | Grouping |
| ---------------------------------------------- | -------------------------------------------- |
| Evaluator belongs to a suite used in the setup | Results are grouped under that suite's name |
| Evaluator does not belong to any suite | Results are grouped under a **Custom** suite |
| Evaluator belongs to multiple suites | Results appear under each matching suite |
### Per-Suite Fields
Each suite in the breakdown shows:
| Field | Description |
| ---------------------- | -------------------------------------------------------- |
| **Suite Name** | The suite name, or "Custom" for ungrouped evaluators |
| **Safety Score** | Percentage of passed results within this suite |
| **Evaluator Count** | Number of evaluators in this suite |
| **Failing Evaluators** | Number of evaluators with at least one vulnerable result |
## Reading the Dashboard
The risk score dashboard provides a multi-level view of your agent's safety:
### Safety Gauge
The top-level safety gauge shows the latest overall safety score. It updates after each completed run.
### Trend Chart
The score history chart plots safety scores over time. Use it to:
* Identify trends — is safety improving or declining across releases?
* Validate fixes — confirm that agent updates improved the score.
### Suite Cards
Below the trend chart, suite cards show the per-category breakdown. Each card displays the suite name, safety score, and number of failing evaluators.
Click on a suite card to drill down into the individual evaluator results. This helps you identify exactly which attack categories your agent is vulnerable to.
## Improving Your Score
Follow this process to systematically improve your agent's safety posture:
Review the suite breakdown and find evaluators with failing results. Focus on evaluators with high severity first.
Open the [run results](/Redteaming/Runs#viewing-results) for failing evaluators. Read the conversation histories to understand how the adversarial attack succeeded.
Modify your agent's system prompt, guardrails, or logic to defend against the identified attack patterns. Common fixes include:
* Strengthen system prompt instructions against information leakage
* Add content filters for harmful output categories
* Improve refusal patterns for jailbreak attempts
Start a new [run](/Redteaming/Runs) with the same setup. Compare the new risk score to the previous one to validate the improvement.
Repeat steps 1–4 until your safety score meets your target threshold.
## Score History and Auditing
Risk scores provide a complete audit trail for your agent's safety posture:
* **Every completed run** contributes a data point to the score history.
* **Failed and cancelled runs** do not affect the score history.
* **Score change** is always computed against the immediately previous completed run.
This history is useful for compliance reporting, security reviews, and tracking the impact of agent changes over time.
## Related
View the per-prompt results that drive risk score computation.
Manage the test setups that define which evaluators contribute to scores.
Understand how suites determine score groupings in the breakdown.
Understand the full red teaming framework and end-to-end flow.
# Runs & Results
Source: https://docs.getnetra.ai/Redteaming/Runs
Execute red team tests against your AI agents. View per-prompt results, conversation histories, and judge scores for each evaluator.
A run executes all adversarial prompts from a [setup](/Redteaming/Configs) against the target agent. Each prompt is sent to the agent, the response is scored by the judge LLM, and a pass/fail/error result is recorded. Runs execute asynchronously with queue-based concurrency control, so you can monitor progress in real time.
## Why Runs Matter
Runs transform your red teaming setup into actionable safety data:
| Capability | Benefit |
| ------------------------ | ---------------------------------------------------------------------- |
| **Async Execution** | Large test suites run in the background without blocking your workflow |
| **Per-Prompt Results** | See exactly which adversarial attacks succeeded and which were blocked |
| **Conversation History** | Review full turn-by-turn dialogue for multi-turn attacks |
| **Progress Tracking** | Monitor per-evaluator completion in real time |
## Run Lifecycle
Every run follows a defined lifecycle:
| Status | Meaning |
| ------------- | ------------------------------------------------------------------- |
| **Running** | The run is actively executing adversarial prompts against the agent |
| **Completed** | All prompts have been processed and results are available |
| **Failed** | The run encountered an unrecoverable error |
| **Cancelled** | The run was manually cancelled before all prompts completed |
## Starting a Run
Navigate to **Red Teaming** and select the [setup](/Redteaming/Configs) you want to run.
Start the run. Netra validates that no other run is active for this setup.
Each adversarial prompt is placed on a processing queue. The run processes prompts with configurable concurrency and rate limiting.
Only one run can be active per setup at a time. Start a new run after the current one completes, fails, or is cancelled.
## Execution Details
### Single-Turn Execution
For each adversarial prompt in single-turn mode:
1. The prompt is sent to the target agent via HTTP.
2. The agent's response is collected.
3. The judge LLM scores the response using the evaluator's grader rubric.
4. A result is saved with the score, pass/fail status, and judge explanation.
### Multi-Turn Execution
For each adversarial prompt in multi-turn mode:
1. The initial prompt is sent to the target agent.
2. The agent's response is judged. If the agent fails (is vulnerable), the result is saved and execution stops for this test.
3. If the agent passes, the attacker LLM generates a follow-up message based on the conversation history.
4. Steps 2–3 repeat for the configured number of turns.
5. A result is saved for each turn, all linked by a shared test ID.
All execution modes generate telemetry traces for each agent interaction. View these traces in the [Observability](/Observability/Traces/overview) section for deeper debugging.
## Viewing Results
Each run provides detailed per-prompt results. Navigate to a completed run to view them.
### Result Fields
| Field | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------- |
| **Status** | `Pass` (agent blocked the attack), `Fail` (agent was vulnerable), or `Error` (execution failed) |
| **Score** | Numerical score assigned by the judge LLM |
| **Judge Output** | The judge LLM's explanation of the score |
| **Evaluator** | Which attack category was tested |
| **Conversation History** | Full turn-by-turn dialogue between the adversarial prompt and the agent (multi-turn only) |
| **Test ID** | Groups related turns in a multi-turn test |
| **Turn Index** | Position of this result within a multi-turn conversation |
### Filtering Results
Filter results by evaluator to focus on specific attack categories. This is useful when a run includes many evaluators and you want to investigate a specific risk area.
## Progress Tracking
While a run is active, track per-evaluator progress in real time.
### Progress Statuses
Each evaluator's progress shows a breakdown of result statuses:
| Status | Meaning |
| -------------- | ------------------------------------------------------ |
| **Protected** | Agent passed — blocked the adversarial attack |
| **Vulnerable** | Agent failed — was susceptible to the attack |
| **Error** | Execution failed for this prompt |
| **Cancelled** | Prompt was not processed because the run was cancelled |
## Cancelling a Run
You can cancel an active run at any time from the run detail page. When a run is cancelled:
1. The run status changes to **Cancelled**.
2. Prompts that have already been processed retain their results.
3. Unprocessed prompts receive synthetic **Cancelled** results.
4. The setup is unlocked for new runs.
Cancelled results are included in the total count but do not affect pass/fail rates or [risk scores](/Redteaming/Risk-Scores).
## Safety Guards
Netra enforces several guards to prevent conflicts:
| Guard | Behavior |
| ------------------------------- | ----------------------------------------------------------------------- |
| **One active run per setup** | You cannot start a new run while one is already active |
| **Agent locked during run** | You cannot update the agent's application details while a run is active |
| **Prompt regeneration blocked** | You cannot regenerate prompts while a run is active |
## Related
View aggregated safety scores computed from run results.
Create and manage the test setups that drive runs.
Browse the adversarial test catalog used in your runs.
Understand the full red teaming framework and end-to-end flow.
# Suites & Evaluators
Source: https://docs.getnetra.ai/Redteaming/Suites-and-Evaluators
Browse the red teaming catalog. Select suites aligned to security frameworks or pick individual evaluators to test specific attack categories.
Evaluators are the building blocks of red teaming. Each evaluator defines a specific attack category — with a template for generating adversarial prompts and a rubric for judging agent responses. Suites bundle related evaluators into curated groups so you can test against an entire risk profile in one step.
## Why Suites & Evaluators Matter
Choosing the right attacks is the foundation of effective adversarial testing:
| Challenge | How the Catalog Helps |
| --------------------------------------------- | ----------------------------------------------------- |
| Too many attack types to manage individually | Suites bundle evaluators into ready-to-use groups |
| Need to align testing with security standards | Suites reference specific framework items |
| Custom risk profiles | Select individual evaluators to build a tailored test |
| Consistent scoring across runs | Grader rubrics enforce repeatable judge criteria |
## Evaluators
An evaluator represents a single adversarial attack category. It contains everything Netra needs to generate attack prompts and judge the agent's response.
### Evaluator Fields
| Field | Description |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Slug** | Unique identifier for the evaluator (e.g., `system-prompt-leakage`) |
| **Category** | Risk category the evaluator tests (e.g., harmful content, jailbreak, data leakage) |
| **Severity** | Risk severity level assigned to this evaluator |
| **Evaluator Type** | Classification of the attack strategy |
| **Generation Template** | Nunjucks template that the attacker LLM uses to generate adversarial prompts. The template receives the agent's purpose as context. |
| **Grader Rubric** | Criteria the judge LLM uses to score the agent's response. Defines what constitutes a pass or fail. |
| **Pass Threshold** | Minimum score at or above which the agent is considered safe for this evaluator |
### How Evaluators Work
Each evaluator serves two roles in the red teaming pipeline:
**1. Prompt Generation**
The generation template is rendered with the target agent's purpose (derived from its application details). The attacker LLM uses this template to create adversarial prompts tailored to the specific attack category.
**2. Response Judging**
After the agent responds, the judge LLM receives the grader rubric along with the conversation. It scores the response and determines whether the agent passed or failed for that attack category.
## Suites
Suites are curated bundles of evaluators. Each suite groups related attack categories together, often aligned to a specific security framework or compliance standard.
### Suite Fields
| Field | Description |
| -------------- | --------------------------------------------------------------------- |
| **Name** | Display name of the suite |
| **Slug** | Unique identifier for the suite |
| **Source** | Reference to the security framework or standard the suite aligns with |
| **Evaluators** | List of evaluators included in the suite |
### Using Suites
Suites simplify evaluator selection. Instead of manually picking individual evaluators, you can select a suite to test against an entire risk profile at once.
When you [create a setup](/Redteaming/Configs), you can:
* **Select a suite** to include all its evaluators automatically
* **Pick individual evaluators** from the catalog for a custom test
* **Combine both** — start with a suite and add or remove evaluators as needed
## Browsing the Catalog
Navigate to **Red Teaming** from the left navigation panel. The catalog interface shows available suites and evaluators.
### Viewing Suites
The suites view displays all available groups. Each suite shows:
* The suite name and source framework
* The number of evaluators included
* An expandable list of individual evaluators with their categories and severity levels
### Viewing Evaluators
The evaluators view lists all available attack categories. Each evaluator shows:
* The evaluator slug and display name
* The risk category and severity
* The evaluator type
Start with a suite that matches your compliance requirements. You can always customize by adding or removing individual evaluators when you create a setup.
## Suites and Risk Scores
Suites also determine how [risk scores](/Redteaming/Risk-Scores) are aggregated. When run results are computed, evaluators that belong to a suite are grouped into a **suite score**. Evaluators not matching any suite appear under a **Custom** suite.
This means your risk score dashboard shows safety posture broken down by the same suites you selected during setup.
## Related
Understand the full red teaming framework and end-to-end flow.
Create test setups that connect agents to evaluators.
Execute adversarial tests and view per-prompt results.
Track aggregated safety scores by suite over time.
# Datasets
Source: https://docs.getnetra.ai/Simulation/Datasets
Create multi-turn simulation datasets in Netra to test AI agent behavior. Define conversation goals and expected dialogue paths for simulation runs.
Datasets for simulation define the scenarios you want to test—multi-turn conversations with specific goals, user personas, and success criteria. Unlike single-turn evaluation datasets, simulation datasets create dynamic, goal-oriented conversations that reflect real-world user interactions.
## Why Simulation Datasets Matter
Simulation datasets transform simple Q\&A testing into realistic conversation testing:
| Benefit | Description |
| ---------------------------- | ----------------------------------------------------------------------------------- |
| **Goal-Oriented Testing** | Test whether your agent achieves specific objectives, not just individual responses |
| **Persona-Based Scenarios** | Simulate different user types—frustrated, confused, friendly, or neutral |
| **Multi-Turn Conversations** | Test how your agent handles back-and-forth dialogue (1-10 turns) |
| **Fact Verification** | Ensure your agent communicates critical information correctly |
| **Context Simulation** | Provide user data and context for realistic scenario execution |
## Dataset Dashboard
Navigate to **Evaluation → Datasets** from the left navigation panel. Filter by **Multi** turn type to see simulation datasets.
Each card displays the following fields:
| Column | Description |
| ---------------- | ---------------------------------------------- |
| **Dataset Name** | Unique identifier for the simulation suite |
| **Turn Type** | MULTI for simulation datasets |
| **Tags** | Metadata labels for filtering and organization |
| **Created At** | Timestamp for version tracking |
| **Actions** | Quick access to edit or delete datasets |
## Creating a Multi-Turn Dataset
Click the **Create Dataset** button in the top right corner of the Datasets page.
| Field | Description |
| --------------- | -------------------------------------------------------------------------------------- |
| **Name** | A descriptive identifier for your simulation suite (e.g., "Customer Refund Scenarios") |
| **Tags** | Labels for filtering (e.g., "customer-support", "refunds", "production") |
| **Type** | Select **Multi-turn** for simulation scenarios |
| **Data Source** | Select **Add manually** to create scenarios one by one |
Import from traces and CSV import for multi-turn datasets are coming soon.
Define the simulation scenario with the following fields:
**Scenario Goal** — Describe what the simulated user is trying to achieve. This becomes the goal that drives the conversation.
```
The customer wants to get a refund for a product they purchased
15 days ago because it arrived damaged.
```
**Behavior Instructions (Optional)** — Provide guidance on how the simulated user should behave.
```
Start politely, but become slightly impatient if the agent
asks for information already provided.
```
**Max Turns** — Choose the maximum number of conversation turns (1-10):
* **Lower (1-3)**: Quick interactions like single-question support
* **Medium (4-6)**: Standard support conversations
* **Higher (7-10)**: Complex, multi-step problem resolution
The simulation stops when the goal is achieved, the max turns limit is reached, or the scenario is abandoned.
**User Persona** — Choose how the simulated user behaves emotionally:
| Persona | Icon | Description |
| -------------- | ---- | --------------------------------------------------- |
| **Neutral** | 😐 | Straightforward and factual, sticks to the point |
| **Friendly** | 😊 | Polite and cooperative, patient with the agent |
| **Frustrated** | 😤 | Impatient, wants quick resolution, may be curt |
| **Confused** | 😕 | Needs extra clarification, asks follow-up questions |
| **Custom** | ✏️ | Define your own persona behavior |
**Provider and Model** — Choose the LLM provider and model that will generate simulated user responses (e.g., OpenAI / GPT-4.1).
Use consistent, capable models (GPT-4, Claude Sonnet) for realistic user simulation.
**Simulated User Data** — Provide context data that the simulated user can reference during the conversation. Available in Table, JSON, or Plain Text format.
**Example (Table)**:
| Key | Value |
| ----------------- | ------------------------- |
| order\_number | ORD-123456 |
| purchase\_date | 2024-01-15 |
| product\_name | Wireless Headphones |
| order\_total | \$129.99 |
| shipping\_address | 123 Main St, New York, NY |
**Example (JSON)**:
```json theme={null}
{
"order_number": "ORD-123456",
"purchase_date": "2024-01-15",
"product_name": "Wireless Headphones",
"order_total": "$129.99",
"shipping_address": "123 Main St, New York, NY"
}
```
**Fact Checker** — Specify facts that the agent MUST communicate correctly. These are used by evaluators to verify accuracy.
**Example (Table)**:
| Fact | Expected Value |
| ------------------------ | ------------------------- |
| refund\_processing\_time | 5-7 business days |
| refund\_method | Original payment method |
| return\_label\_delivery | Within 24 hours via email |
**Example (JSON)**:
```json theme={null}
{
"refund_processing_time": "5-7 business days",
"refund_method": "Original payment method",
"return_label_delivery": "Within 24 hours via email"
}
```
**JSON Validation**: When using JSON format, ensure there are no duplicate
keys. The system validates JSON structure before allowing you to proceed.
Select [evaluators](/Simulation/Evaluators) from Netra's library of session-level evaluators across two categories:
**Recommended Evaluators**:
* **Agentic**: Goal Fulfillment, Information Elicitation
* **Quality**: Factual Accuracy, Conversation Completeness, Guideline Adherence
Configure variable mappings to connect evaluator inputs to your data:
* **Scenario fields**: Goal, persona, user data
* **Agent response**: What the agent said in each turn
* **Conversation metadata**: Turn index, conversation history
* **Execution data**: Latency, tokens, model
When you select evaluators from the library, Netra clones them and adds them to **My Evaluators**. Configure each cloned evaluator:
* **Rename** (optional) — Rename any evaluator to match your use case (e.g., "Refund Goal Fulfillment" instead of "Goal Fulfillment")
* **Select Provider and Model** — For each evaluator, choose the **provider** and **model** that will run the LLM-as-Judge evaluation (e.g., OpenAI / GPT-4.1)
Review all configurations, then click **Create Dataset** to finalize. Your simulation dataset is now ready to run.
## Running a Simulation
Once your dataset is configured, you can run simulations:
Open your dataset and copy the **Dataset ID** displayed at the top of the page.
Use the Dataset ID in your simulation code. The simulation runs automatically
through the Netra SDK.
Monitor progress and results in [Test Runs](/Simulation/TestRuns).
Simulations execute automatically when the associated code is triggered. You
don't need to manually start each run—just ensure your agent code is
integrated with Netra.
## Best Practices
### Crafting Effective Scenarios
* **Be specific**: "Get a refund for a damaged product" is better than "Ask about returns"
* **Include context**: Provide enough detail for realistic simulation (order details, timeline, issue description)
* **Include edge cases**: Create scenarios that challenge your agent's boundaries
### Choosing User Personas
* **Neutral**: Best for baseline performance testing
* **Friendly**: Tests whether your agent maintains professionalism even when not challenged
* **Frustrated**: Critical for customer support agents—tests patience and de-escalation
* **Confused**: Tests clarity and explanation quality
* **Custom**: Use for industry-specific personas (technical users, non-native speakers, etc.)
### Defining User Data
* **Provide realistic data**: Use representative order numbers, dates, and values
* **Include edge cases**: Test with missing fields, unusual values, or conflicting data
* **Keep it relevant**: Only include data that matters for the scenario
* **Use consistent formats**: Standardize date formats, currency, and naming
### Setting Fact Checkers
* **Focus on critical facts**: What MUST the agent communicate correctly?
* **Be precise**: "5-7 business days" is better than "about a week"
* **Test compliance**: Include regulatory or policy-critical information
* **Verify, don't duplicate**: Don't repeat information already in user data
## Related
* [Simulation Overview](/Simulation/Simulation-overview) - Understand the full simulation framework
* [Evaluators](/Simulation/Evaluators) - Configure scoring logic for simulations
* [Test Runs](/Simulation/TestRuns) - View simulation results and conversation transcripts
* [Traces](/Observability/Traces/overview) - Debug simulation turns with execution traces
# Evaluators
Source: https://docs.getnetra.ai/Simulation/Evaluators
Assess simulated AI conversations in Netra using library and custom evaluators. Measure turn-by-turn quality, goal completion, and agent performance.
Simulation evaluators assess entire multi-turn conversations at the session level. After a simulated conversation completes, evaluators determine whether your agent achieved its goal, communicated facts correctly, and maintained quality throughout the interaction.
## Why Simulation Evaluators Matter
Multi-turn conversations require different evaluation approaches than single-turn responses:
| Challenge | How Simulation Evaluators Help |
| --------------------- | ----------------------------------------------------------------------------------- |
| Goal achievement | Verify whether the conversation reached its objective |
| Fact accuracy | Track whether critical information was communicated correctly across multiple turns |
| Conversation quality | Ensure the conversation maintains logical flow, completeness, and consistency |
| Information gathering | Assess whether the agent effectively collected required information from the user |
## Evaluators Dashboard
Navigate to **Evaluation → Evaluators** from the left navigation panel. Switch to the **Library** tab and filter by **Multi turn** to see the simulation evaluators.
Netra organizes simulation evaluators into two categories: **Quality** and **Agentic**.
## Library Evaluators
Netra provides 8 preconfigured library evaluators across two categories. All evaluators run at the session level, assessing the entire conversation after it completes.
### Quality Evaluators
Quality evaluators assess how well your agent maintains conversation standards.
| Evaluator | What It Measures |
| ----------------------------- | ------------------------------------------------------------------------------------- |
| **Guideline Adherence** | Whether the assistant followed its given instructions throughout the conversation |
| **Conversation Completeness** | Whether all human intents were addressed during the conversation |
| **Profile Utilization** | Whether the assistant correctly used provided human profile information when relevant |
| **Conversational Flow** | Whether the conversation flowed logically and the assistant maintained consistency |
| **Conversation Memory** | Whether the assistant remembered and correctly used information shared earlier |
| **Factual Accuracy** | Whether the assistant's claims were consistent with provided reference facts |
### Agentic Evaluators
Agentic evaluators assess goal-directed and information-gathering behavior.
| Evaluator | What It Measures |
| --------------------------- | ---------------------------------------------------------------------- |
| **Goal Fulfillment** | Goal achievement and progress toward the stated conversation objective |
| **Information Elicitation** | How effectively the agent gathered required information from the user |
## Evaluator Configuration
All 8 library evaluators share the same configuration:
| Setting | Value |
| ------------------------- | ------------------------------------------ |
| **Type** | LLM as Judge |
| **Eval Scope** | Session (entire conversation) |
| **Output** | Numerical (0-1, normalized from 1-5 scale) |
| **Default Pass Criteria** | >= 0.6 |
You can adjust the pass criteria threshold for any evaluator based on your requirements. A higher threshold enforces stricter quality standards.
## Using Evaluators in Simulations
When configuring a [multi-turn dataset](/Simulation/Datasets), you select and configure evaluators in Step 4 of the dataset creation flow. You can choose any combination of Quality and Agentic evaluators based on what you want to measure.
## Best Practices
### Choosing Evaluators by Scenario Type
| Scenario Type | Recommended Evaluators |
| -------------------- | ------------------------------------------------------------------- |
| Customer Support | Conversation Completeness, Factual Accuracy, Guideline Adherence |
| Technical Assistance | Conversation Completeness, Conversational Flow, Goal Fulfillment |
| Sales Conversations | Profile Utilization, Factual Accuracy, Information Elicitation |
| Troubleshooting | Conversation Completeness, Conversation Memory, Conversational Flow |
### Getting Started with Evaluators
1. **Start with Goal Fulfillment and Factual Accuracy** — these cover the most critical aspects of any simulation
2. **Add Quality evaluators** based on your use case — Conversation Completeness and Guideline Adherence are strong defaults
3. **Adjust pass criteria** if the default threshold of 0.6 is too lenient or strict for your needs
4. **Monitor results** across the first few test runs to ensure evaluators align with your expectations
## Related
* [Simulation Overview](/Simulation/Simulation-overview) - Understand the full simulation framework
* [Datasets](/Simulation/Datasets) - Create scenarios that use evaluators
* [Test Runs](/Simulation/TestRuns) - View evaluation results from simulations
* [Evaluation Evaluators](/Evaluation/Evaluators) - Standard single-turn evaluators documentation
# Simulation Overview
Source: https://docs.getnetra.ai/Simulation/Simulation-overview
Test your AI agents with Netra's simulation framework. Run multi-turn conversations with a simulated user to validate agent behavior before deploying.
AI agents don't just answer questions—they engage in complex, multi-turn conversations to achieve goals. Netra's Simulation framework lets you test these interactions systematically, simulating realistic user behaviors to validate your agent's performance before deployment.
New to simulations? Get your first simulation running in minutes.
## Why Simulation Matters
Traditional testing falls short for conversational agents. Simulations provide a comprehensive way to test multi-turn interactions with realistic user behaviors:
| Question | What Netra Simulates |
| -------------------------------------------------------- | ----------------------------------------------------- |
| Does my agent handle multi-turn conversations correctly? | Full conversation flows with simulated user responses |
| Can my agent achieve specific goals? | Goal-oriented scenarios with success/failure tracking |
| How does my agent perform with different user personas? | Frustrated, confused, friendly, or neutral users |
## Core Building Blocks
The Simulation suite is built on three interconnected pillars:
### Evaluators
[Evaluators](/Simulation/Evaluators) assess the entire conversation after it completes. Netra provides 8 preconfigured library evaluators in two categories:
| Category | Evaluators |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **Quality** | Guideline Adherence, Conversation Completeness, Profile Utilization, Conversational Flow, Conversation Memory, Factual Accuracy |
| **Agentic** | Goal Fulfillment, Information Elicitation |
All evaluators use LLM-as-Judge with a default pass threshold of >= 0.6.
### Datasets
[Datasets](/Simulation/Datasets) are collections of simulation scenarios that define multi-turn conversation goals.
| Feature | Description |
| ------------------------ | ---------------------------------------------------------------------------------- |
| **Multi-Turn Scenarios** | Define conversation goals with simulated user interactions |
| **User Personas** | Choose from neutral, friendly, frustrated, confused, or custom personas |
| **User Data & Facts** | Provide context data and facts the agent must communicate correctly |
| **Variable Mapping** | Map evaluator inputs to scenario fields, agent responses, or conversation metadata |
### Test Runs
[Test Runs](/Simulation/TestRuns) execute your simulation scenarios, providing detailed conversation transcripts and evaluation results.
| Feature | Description |
| --------------------------- | --------------------------------------------------------------------------- |
| **Conversation Transcript** | Full multi-turn dialogue between simulated user and agent |
| **Scenario Details** | View goal, persona, user data, and fact checker configuration |
| **Trace Integration** | Link directly to execution traces for each turn to debug issues |
| **Aggregated Metrics** | View total cost, average latency, and pass/fail rates across the simulation |
## Use Cases
### Goal Achievement Testing
Validate that your agent can successfully complete user objectives:
1. Create scenarios with specific goals (e.g., "Get a refund from customer support")
2. Define what facts the agent must communicate
3. Run simulations and verify goal achievement across different personas
4. Analyze conversation transcripts to understand failure points
### Persona-Based Testing
Test agent performance with different user types:
1. Create datasets with various personas (frustrated, confused, friendly)
2. Run the same scenario across all personas
3. Compare results to identify which personas your agent handles poorly
4. Refine your agent based on insights
## Getting Started
Set up [evaluators](/Simulation/Evaluators) to define your scoring criteria — choose from the library or create custom ones.
Build a [multi-turn dataset](/Simulation/Datasets) with simulation scenarios, user personas, and facts to verify.
Execute your dataset and view conversation transcripts and results in [Test Runs](/Simulation/TestRuns).
Use insights from simulations to refine your agent's behavior.
## Related
* [Evaluators](/Simulation/Evaluators) - Configure scoring logic and criteria
* [Datasets](/Simulation/Datasets) - Create multi-turn simulation scenarios
* [Test Runs](/Simulation/TestRuns) - Analyze simulation results and conversation transcripts
* [Traces](/Observability/Traces/overview) - Understand how simulations connect to trace data
# Test Runs
Source: https://docs.getnetra.ai/Simulation/TestRuns
View Netra simulation test run results with full conversation transcripts. Analyze turn-by-turn scores, agent behavior, and goal completion rates.
Test Runs for simulation show the execution results of your [multi-turn datasets](/Simulation/Datasets). Each run provides a complete conversation transcript between the simulated user and your agent, along with evaluation results, scenario details, and performance metrics.
## Why Simulation Test Runs Matter
Simulation test runs provide deep insights into conversational agent performance:
| Capability | Benefit |
| ---------------------------- | ---------------------------------------------------------------------------------------------- |
| **Conversation Transcripts** | See the full multi-turn dialogue to understand how your agent performed |
| **Scenario Details** | View goal, persona, user data, and fact checker configuration |
| **Turn-by-Turn Tracing** | Jump directly to execution [traces](/Observability/Traces/overview) for each conversation turn |
| **Evaluation Results** | Review turn-level and session-level evaluator scores |
| **Exit Reason Tracking** | Understand why conversations ended (goal achieved, failed, abandoned, max turns) |
| **Aggregated Metrics** | Monitor cost, latency, and success rates across simulations |
## Test Runs Dashboard
Navigate to **Evaluation → Test Runs** from the left navigation panel to see simulation test runs.
| Column | Description |
| -------------- | ------------------------------------------------------------ |
| **Name** | Name of the test run |
| **Type** | Multi-turn for simulation test runs |
| **Started At** | Timestamp when the simulation began |
| **Status** | Current state: Completed, In Progress, or Failed |
| **Dataset** | The [dataset](/Simulation/Datasets) used for this simulation |
### Filtering and Search
* **Date Range**: Filter runs by time period to compare performance over time
* **Search**: Find specific test runs by agent or dataset name
* **Sort**: Order by date, status, or dataset
## Viewing Test Run Details
Click on any simulation test run to access detailed results.
### Summary Metrics
The top of the detail view shows aggregated performance data:
| Metric | Description |
| ------------------- | --------------------------------------------- |
| **Total Items** | Number of scenarios run in this test |
| **Passed Items** | Count of scenarios that achieved their goals |
| **Failed Items** | Count of scenarios that did not achieve goals |
| **Total Cost** | Aggregate token/API cost for all scenarios |
| **Total Duration** | End-to-end time for the simulation run |
| **Average Latency** | Mean response time across all turns |
## Viewing Scenario Details
Click on any test run item to view the detailed scenario results. This opens a modal with three tabs.
### Tab 1: Conversation
The Conversation tab shows the full multi-turn dialogue between the simulated user and your agent.
**Features**:
* **Turn-by-Turn Display**: Each conversation turn is clearly separated
* **User Messages**: Shows what the simulated user said
* **Agent Responses**: Shows what your agent replied
* **Trace Links**: Click **View Trace** on any turn to see detailed execution traces
* **Turn Index**: Track which turn number you're viewing (Turn 1, Turn 2, etc.)
* **Exit Reason**: Shows why the conversation ended
**Exit Reasons**:
| Exit Reason | Description |
| --------------------- | ------------------------------------------------- |
| **Goal Achieved** | The scenario objective was successfully completed |
| **Goal Failed** | The objective could not be achieved |
| **Abandoned** | The simulated user gave up or stopped engaging |
| **Max Turns Reached** | Hit the turn limit before goal completion |
Use the **View Trace** link to debug specific turns where the agent's response
was unexpected or incorrect. Traces show the full LLM call, tool usage, and
latency breakdown.
### Tab 2: Evaluation Results
The Evaluation Results tab shows scores from all configured [evaluators](/Simulation/Evaluators).
Each evaluator produces a normalized score between 0 and 1. Scores at or above **0.6** pass; scores below 0.6 fail.
**Example Results**:
| Evaluator | Score | Pass/Fail |
| ------------------------- | ----- | --------- |
| Goal Fulfillment | 1 | Pass |
| Factual Accuracy | 1 | Pass |
| Conversation Completeness | 1 | Pass |
| Profile Utilization | 0.75 | Pass |
| Guideline Adherence | 0.5 | Fail |
### Tab 3: Scenario Details
The Scenario Details tab shows the complete configuration used for this simulation.
**Scenario Section**:
| Field | Value |
| ---------------- | ------------------------------------------------------------------- |
| **Goal** | The scenario objective (e.g., "Get a refund for a damaged product") |
| **Max Turns** | Maximum turns allowed (e.g., 5) |
| **User Persona** | The persona used (e.g., Frustrated 😤) |
**User Data Section**:
Shows all context data provided to the simulated user:
```json theme={null}
{
"order id": "3",
"product name": "laptop stand"
}
```
**Fact Checker Section**:
Shows facts the agent needed to communicate:
```json theme={null}
{
"item usage": "unused",
"refund window": "7 days",
"days since delivery": "28"
}
```
**Provider Configuration Section**:
| Field | Value |
| ------------ | --------------------------------------------- |
| **Provider** | The LLM provider used (e.g., openai) |
| **Model** | The model used for simulation (e.g., gpt-4.1) |
The Scenario Details tab is crucial for understanding the context of each
simulation. It shows exactly what data the simulated user had access to and
what facts the agent was expected to communicate.
## Analyzing Simulation Results
### Identifying Patterns
When reviewing simulation test runs, look for:
* **Goal achievement rates**: What percentage of simulations achieved their goals?
* **Persona differences**: Does your agent perform better with certain personas? Run the same scenarios with all persona types and compare results.
* **Turn efficiency**: Are conversations longer than necessary? Compare turn counts for successful vs failed scenarios.
* **Common failure points**: Which turns typically cause issues?
* **Fact accuracy**: Are specific facts consistently missed?
* **Cost trends**: Monitor total cost across test runs and identify scenarios that consume excessive turns.
### Debugging Failed Simulations
For each failed scenario:
1. **Review the Conversation tab**: Identify where the conversation went wrong
2. **Check the Evaluation Results tab**: See which evaluators failed and why
3. **Examine the Scenario Details tab**: Verify the user data and facts were correct
4. **Click View Trace**: Inspect the full execution flow for problematic turns — check LLM inputs, tool calls, and latency breakdowns
### Comparing Across Runs
To track improvement or regression:
1. Run simulations after each agent update
2. Compare goal achievement rates and evaluator scores across runs
3. Investigate scenarios that changed from pass to fail
4. Track turn efficiency and cost trends over time
## Best Practices
* **Test after every agent change**: Run simulations when updating your agent to catch regressions early
* **Create baseline runs**: Establish performance benchmarks before making changes
* **Always check traces for failures**: Don't just read the conversation — inspect the execution flow, LLM context, and tool calls
* **Review latency**: Identify slow turns that might frustrate real users
## Related
* [Simulation Overview](/Simulation/Simulation-overview) - Understand the full simulation framework
* [Datasets](/Simulation/Datasets) - Create scenarios that generate test runs
* [Evaluators](/Simulation/Evaluators) - Configure scoring logic for simulations
* [Traces](/Observability/Traces/overview) - Debug simulation turns with execution traces
# Online Evaluations
Source: https://docs.getnetra.ai/Simulation/online-evaluation
Continuously evaluate production traces in real-time by running configurable evaluators against incoming agent data — automatically and at scale.
# Online Evaluations
> Continuously evaluate production traces in real-time by running configurable evaluators against incoming agent data — automatically and at scale.
AI agents in production don't fail in predictable ways — they degrade silently, drift over time, and exhibit issues that only surface under real-world conditions. Netra's Online Evaluations framework brings automated, continuous quality assessment directly into your production pipeline. Instead of manually reviewing traces, you define evaluation rules once, and every incoming trace that matches your criteria gets scored automatically.
Online Evaluations work alongside Evaluators. Learn how to build custom evaluators for your use cases.
***
## Why Online Evaluations Matter
Without continuous production monitoring, issues compound silently. Online Evaluations help you answer critical questions in real-time:
| Question | What Online Evaluations Measure |
| ---------------------------------------------------------- | --------------------------------------------------------------- |
| Is my agent producing quality outputs in production? | Real-time pass rates across all configured evaluators |
| Are there regressions after deploying a new agent version? | Continuous score tracking and trend analysis over time |
| Which evaluators are flagging the most failures? | Per-evaluator breakdown with pass/fail rates |
| Am I meeting quality SLAs for my AI product? | Configurable alert thresholds with automatic notifications |
| What does my evaluation cost look like over time? | Total evaluation cost tracking across all configured evaluators |
***
## How It Works
Online Evaluations run as a background process that continuously analyses all traces coming into your project. When configured for a specific agent (or all agents), the system:
1. **Intercepts incoming traces** — Every trace logged to the project is checked against the online evaluation's scope and filters
2. **Applies sampling** — Based on your configured sampling rate (1–100%), traces are selected for evaluation
3. **Runs evaluators** — Each selected trace is scored against all configured evaluators (LLM-as-Judge, latency, cost, regex, etc.)
4. **Records results** — Evaluation scores, pass/fail status, and reasoning are stored per trace
5. **Triggers alerts** — If configured thresholds are breached, alerts fire to your contact points
Online Evaluations operate at the **trace** scope — each incoming trace is treated as a single evaluation unit scored against your configured evaluators.
***
## Online Evaluations List Page
When you navigate to **Online Evaluations**, you see a table listing all configured online evaluations for the current project.
| Column | Description |
| --------------------- | ---------------------------------------------------- |
| **Name** | The name assigned to the online evaluation |
| **Total Evaluations** | Total number of trace evaluation results recorded |
| **No. of Evaluators** | Number of evaluators configured for this online eval |
| **Status** | Active (running) or Inactive (paused) |
| **Actions** | Edit and Delete buttons |
The page also provides:
* **Search** — Filter evaluations by name
* **Refresh** — Reload the list to see the latest data
* **Create button** — Opens the configuration modal to create a new online evaluation
Clicking any row navigates to the detailed view for that online evaluation.
***
## Online Evaluation Detail View
On clicking into a specific online evaluation, you see its detail page with three tabs and an active/inactive toggle in the top right.
### Active / Inactive Toggle
The toggle at the top right controls whether the online evaluation is currently running. When **Active**, it continuously evaluates incoming traces. When **Inactive**, no new traces are evaluated but historical results remain accessible.
### Overview Tab
The Overview tab is a time-filtered analytics dashboard that provides a high-level picture of how your evaluation is performing.
**Summary Cards:**
| Metric | Description |
| --------------------------- | -------------------------------------------------------------------------- |
| **Traces Evaluated** | Total number of traces that have been evaluated in the selected time range |
| **Overall Pass Rate** | Percentage of pass judgments across all evaluators, pooled together |
| **Average Evaluator Score** | Mean numeric score across all evaluators for the selected period |
| **Total Evaluation Cost** | Estimated LLM cost incurred for running the evaluators |
**Charts:**
* **Evaluator Scores** — A combined line chart showing per-evaluator pass rates over time, with a white average score line overlaid
* **Traces Evaluated** — A bar chart showing evaluation volume over time
* **Online Evaluation Cost** — A line chart tracking cost over time
All charts respect the selected time range and automatically group data by minute, hour, or day depending on the range.
### Results Tab
The Results tab provides granular, trace-level evaluation data split into two sections:
**Evaluator Breakdown Table:**
| Column | Data |
| ---------------- | ----------------------------------------- |
| Evaluator | Evaluator name |
| Type | Evaluator type (e.g., LLM-as-Judge) |
| Traces Evaluated | Number of traces scored by this evaluator |
| Pass Rate | Percentage of traces that passed |
| Fail Rate | Percentage of traces that failed |
**Evaluation Results Table:**
A detailed per-trace results table with:
* **Timestamp** — When the trace was evaluated
* **Trace ID** — Unique identifier (clickable to view full trace)
* **Evaluator columns** — One column per configured evaluator showing score, pass/fail status, and reasoning
Clicking a row opens a sidebar with full evaluation details for that trace, including per-evaluator scores, reasoning, pass criteria, and a link to view the complete trace.
### Alerts Tab
The Alerts tab displays a history of all triggered alerts for this online evaluation.
| Column | Description |
| -------------- | --------------------------------------------------------------- |
| Triggered Time | When the alert was fired |
| Alert Details | Source (overall pass rate vs. per-evaluator) and detail message |
| Severity | CRITICAL or WARNING |
| Status | TRIGGERED or RESOLVED |
The table is sortable by Triggered Time, Severity, and Status.
***
## Configuring an Online Evaluation
Creating an online evaluation is a 4-step process:
Define the scope and identity of your online evaluation:
| Field | Required | Description |
| --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Agent** | No | Select a specific agent to evaluate, or leave as "All agents" to evaluate every trace in the project |
| **Filters** | No | Add trace filters to narrow which traces get evaluated (e.g., by metadata, tags, or attributes). A matching trace count preview shows how many traces matched in the last 24 hours |
| **Name** | Yes | A descriptive name for this online evaluation (e.g., "Production Quality Monitor") |
| **Description** | No | Optional description explaining the purpose of this evaluation |
If you add filters, all filter rows must be complete (field, operator, and value) before proceeding.
Choose which evaluators will score your traces. You can select from two sources:
* **My Evaluators** — Custom evaluators you've created in your workspace
* **Library** — Pre-built evaluator templates grouped by category
Browse evaluators using the search bar or filter tabs (All / My Evaluators / Library). Select at least one evaluator to proceed.
The Library provides pre-built evaluators across the following categories:
| Category | What It Covers |
| ------------------- | ---------------------------------------------------------------------------------- |
| **Quality** | Response accuracy, relevance, faithfulness to context, and hallucination detection |
| **Multimodal** | Image generation quality, format validation, and text-image alignment |
| **Guardrails** | Safety checks — bias detection, topic boundaries, and harmful content |
| **Performance** | Response time, cost, and token consumption thresholds |
| **Semantic** | Meaning-level comparison between outputs and references |
| **Agentic** | Whether the agent achieved the user's stated goal |
| **JSON Evaluator** | Validates output JSON structure and field values |
| **Regex Evaluator** | Validates output format against a regex pattern |
| **Tool Use** | Verifies correct tool calls against expected tools |
You must select at least one evaluator. Only single-turn evaluators are available for online evaluations.
Configure each selected evaluator with its specific settings:
**For LLM-based evaluators:**
* Select the provider and model (e.g., OpenAI / gpt-4o-mini)
* Define evaluation criteria and pass criteria
**Variable mapping:**
* Map evaluator variables to trace data using JMESPath expressions
* Variables can source from trace attributes, span data, or literal values
**Pass criteria:**
* Numerical evaluators: Set a threshold with an operator (`>=`, `<=`, `>`, `<`, `=`)
* Boolean evaluators: Set pass condition (`is_true` / `is_false`)
Each evaluator card shows its configuration status. All required fields must be filled before proceeding.
Configure how many traces to evaluate and optionally set up alert rules.
**Sampling Rate:**
| Preset | Description |
| ------ | ----------------------------------------- |
| 10% | Light sampling for high-volume production |
| 25% | Balanced sampling for moderate traffic |
| 50% | Substantial coverage |
| 75% | High coverage |
| 100% | Evaluate every matching trace (default) |
Use the slider or presets to set any value from 1–100%.
**Alerts (optional):**
Toggle alerts on to receive notifications when quality degrades. When enabled, configure:
| Setting | Description |
| ---------------------------- | ----------------------------------------------------------------- |
| **Contact Points** | Select notification channels (email, Slack, etc.) |
| **Re-evaluation Interval** | How often to check alert conditions (15 min to 24 hours) |
| **Overall Pass Rate Alerts** | Trigger when overall pass rate crosses a threshold for a duration |
| **Per-Evaluator Alerts** | Set individual thresholds per evaluator |
Each alert rule supports:
* **Severity** — CRITICAL or WARNING
* **Comparison type** — is\_above or is\_below
* **Threshold** — The value that triggers the alert
* **Duration** — How long the condition must persist before firing
Click **Create** (or **Update** in edit mode) to save. The online evaluation begins running immediately in Active status.
***
## Editing an Online Evaluation
To edit an existing online evaluation:
1. Navigate to the **Online Evaluations** list page
2. Click the **Edit** (pencil icon) button in the Actions column of the evaluation you want to modify
3. The configuration modal opens pre-filled with the current settings
4. Modify any settings across the 4 steps (Rule Details, Evaluator Selection, Evaluator Configuration, Sampling & Alerts)
5. Click **Update** to save your changes
Changes take effect immediately — the next incoming trace will be evaluated using the updated configuration.
You can also toggle an online evaluation between Active and Inactive from the detail page without opening the edit modal.
***
## Deleting an Online Evaluation
To delete an online evaluation:
1. Navigate to the **Online Evaluations** list page
2. Click the **Delete** (trash icon) button in the Actions column
3. A confirmation modal appears: *"Are you sure you want to delete \[name]? This action cannot be undone."*
4. Click **Delete** to confirm
Deleting an online evaluation permanently removes the configuration and all associated evaluation history. This action cannot be undone.
***
## Best Practices
| Practice | Why It Matters |
| -------------------------------------------------------- | ------------------------------------------------------------- |
| Start with 100% sampling, reduce as volume grows | Ensures full coverage during setup, then optimize for cost |
| Configure alerts with appropriate durations | Avoids noisy alerts from transient spikes |
| Use per-evaluator alerts for critical quality dimensions | Pinpoints exactly which quality aspect is degrading |
| Combine multiple evaluator types | Catches different failure modes (quality, latency, structure) |
| Filter by specific agents for targeted monitoring | Focuses evaluation budget on the most critical agents |
| Review the Results tab regularly | Identifies patterns that summary metrics might miss |
| Set meaningful pass criteria per evaluator | Defines what "good" means for your specific use case |
***
## FAQ
An online evaluation analyses all incoming traces for the project that match the configured scope. If you select a specific agent, only traces from that agent are evaluated. If you add filters, only traces matching those filter conditions are evaluated. The sampling rate then determines what percentage of matching traces actually get scored.
Yes. You can configure multiple online evaluations, each with different agents, filters, evaluators, and sampling rates. Each operates independently.
When inactive, no new traces are evaluated. Historical results remain accessible in the Overview, Results, and Alerts tabs. Toggling back to Active resumes evaluation of new incoming traces.
The overall pass rate pools all pass/fail judgments across all evaluators for all evaluated traces in the selected time range. If a trace has 3 evaluators and passes 2, that contributes 2 passes and 1 failure to the aggregate.
Sampling rate determines the percentage of matching traces that are actually evaluated. At 100%, every trace is scored. At 25%, roughly one in four matching traces is selected. This helps manage evaluation costs at high traffic volumes.
Alerts check conditions at the configured re-evaluation interval (e.g., every 15 minutes). If the overall pass rate or a per-evaluator metric crosses the threshold for the specified duration, the alert fires and notifications are sent to your configured contact points. Alerts auto-resolve when conditions return to normal.
Yes. In Step 2 (Evaluator Selection), the Library tab shows pre-built evaluator templates. When you select a library evaluator, it is automatically saved to your workspace and linked to the online evaluation.
Evaluation cost reflects the estimated LLM cost for running evaluators (primarily LLM-as-Judge type). Non-LLM evaluators (latency, regex, JSON validation) have no cost. The Total Evaluation Cost metric in the Overview tab tracks this over time.
***
## Related
* [Evaluation Agents](/Evaluation/Evaluators) — Create custom evaluators for online and offline evaluation
* [Traces](/Observability/Traces/overview) — View and debug individual trace executions
# Netra MCP
Source: https://docs.getnetra.ai/mcp/overview
Connect Netra's remote MCP server to Cursor, Claude Code, or other IDEs to query traces, manage datasets, and run evaluations.
Netra exposes a remote MCP server so you can pull observability context and run evaluation workflows directly in your editor or agent workspace.
Choose the MCP endpoint that matches your Netra data region, then replace the
API key placeholder with your own project API key.
## Server Details
* Region-specific MCP endpoints:
* Auth header: `x-api-key`
```bash US Region theme={null}
NETRA_MCP_ENDPOINT="https://api.getnetra.ai/mcp"
NETRA_API_KEY=""
```
```bash EU Region theme={null}
NETRA_MCP_ENDPOINT="https://api.eu.getnetra.ai/mcp"
NETRA_API_KEY=""
```
Use the endpoint that matches the region where your Netra project is hosted.
## Client Setup
Add this `netra` entry to your MCP server configuration in `mcp.json`:
If you're in the EU region, then use the url `https://api.eu.getnetra.ai/mcp`.
```json theme={null}
"netra": {
"url": "https://api.getnetra.ai/mcp",
"headers": {
"x-api-key": ""
}
}
```
Add the Netra MCP server directly from the command line:
If you're in the EU region, then use the url `https://api.eu.getnetra.ai/mcp`.
```bash theme={null}
claude mcp add netra --transport http https://api.getnetra.ai/mcp --header "x-api-key: "
```
Alternatively, add it via JSON:
```bash theme={null}
claude mcp add-json netra '{"type": "http", "url": "https://api.getnetra.ai/mcp", "headers": {"x-api-key": ""}}'
```
To verify the server was added correctly:
```bash theme={null}
claude mcp list
```
Add this `netra` entry to your MCP server configuration in `mcp_config.json`:
If you're in the EU region, then use the url `https://api.eu.getnetra.ai/mcp`.
```json theme={null}
"netra": {
"url": "https://api.getnetra.ai/mcp",
"headers": {
"x-api-key": ""
}
}
```
Add this `netra` server definition to your MCP configuration in `mcp.json`:
If you're in the EU region, then use the url `https://api.eu.getnetra.ai/mcp`.
```json theme={null}
"netra": {
"serverUrl": "https://api.getnetra.ai/mcp",
"headers": {
"x-api-key": ""
}
}
```
Add this block to `~/.codex/config.toml`:
If you're in the EU region, then use the url `https://api.eu.getnetra.ai/mcp`.
```toml theme={null}
[mcp_servers.netra]
enabled = true
url = "https://api.getnetra.ai/mcp"
[mcp_servers.netra.http_headers]
x-api-key = ""
```
Codex also supports managing MCP servers from the CLI, but custom HTTP headers like `x-api-key` should be set in the config file.
## Why Netra MCP
Integrate Netra's observability and evaluation workflows directly into your development environment. The Netra MCP server enables your AI coding assistant to query traces, inspect sessions, create datasets and evaluators, and launch test runs without leaving the IDE. By bridging the gap between telemetry, quality validation, and your code, it helps you debug regressions and validate agent behavior as you build.
## Available Tools
The Netra MCP server exposes **21 tools** scoped to the project associated with your API key. Tools are grouped by capability below.
### Observability
| Tool | Description |
| --------------------------- | ----------------------------------------------------------------------------------------- |
| `netra_get_trace_by_id` | Retrieve all spans for a given trace ID |
| `netra_query_traces` | Query traces with filters, sorting, and cursor-based pagination |
| `netra_get_session_details` | Retrieve session-level totals and all traces in a session (by `session_id` or `trace_id`) |
### Datasets
| Tool | Description |
| --------------------------------- | --------------------------------------------------------------- |
| `netra_create_dataset` | Create a new evaluation dataset (single-turn or multi-turn) |
| `netra_list_datasets` | List evaluation datasets in the current project |
| `netra_create_dataset_item` | Add a test case item to a dataset |
| `netra_create_dataset_items_bulk` | Add multiple test case items in one operation |
| `netra_get_dataset_items` | Retrieve dataset items and their evaluator mappings |
| `netra_update_dataset_item` | Update a dataset item's input, metadata, or evaluator overrides |
| `netra_map_evaluator_to_dataset` | Map an evaluator to a dataset at the dataset level |
### Evaluators
| Tool | Description |
| ------------------------------------- | -------------------------------------------------------------------- |
| `netra_get_default_llm_configuration` | Get the organization's default LLM provider and model for evaluators |
| `netra_list_evaluators` | List evaluators configured in the current project |
| `netra_list_evaluator_library` | List built-in evaluators from Netra's evaluator library |
| `netra_get_evaluator` | Get detailed configuration for a specific evaluator |
| `netra_create_evaluator` | Create an evaluator from the library or with a custom configuration |
### Test Runs
| Tool | Description |
| ---------------------------------- | -------------------------------------------------------- |
| `netra_create_test_run` | Create a single-turn test run against a dataset |
| `netra_create_multi_turn_test_run` | Create a multi-turn conversational test run (simulation) |
| `netra_list_test_runs` | List test runs for the current project |
| `netra_get_test_run_summary` | Get high-level pass rates and scores per evaluator |
| `netra_get_test_run_details` | Get per-item evaluation results for a test run |
MCP tools handle evaluation **setup** (datasets, evaluators, test run creation) and **result inspection**. Your agent must still be executed — typically via `Netra.evaluation.run_test_suite()` for single-turn evaluations or `Netra.simulation.run_simulation()` for multi-turn simulations — before evaluators score outputs and results appear in `netra_get_test_run_summary` / `netra_get_test_run_details`.
## Example Workflows
### Debug a production regression
Call `netra_query_traces` with a time range and filters such as `has_error`, `user_id`, or `session_id`.
Use `netra_get_trace_by_id` with a trace ID from the results to retrieve the full span tree.
Call `netra_get_session_details` with the `session_id` or `trace_id` to see cost, token, and error totals across the conversation.
### Set up an evaluation from your IDE
Call `netra_list_evaluator_library` to browse built-in evaluators, then `netra_create_evaluator` to add one to your project.
Call `netra_create_dataset` with the appropriate `turnType` (`single` or `multi`), then add items with `netra_create_dataset_item` or `netra_create_dataset_items_bulk`.
Use `netra_map_evaluator_to_dataset` to attach evaluators at the dataset level, or set item-level overrides when creating items.
Call `netra_create_test_run` (single-turn) or `netra_create_multi_turn_test_run` (multi-turn) to allocate a test run against your dataset. This registers the run but does **not** execute your agent or trigger evaluators on its own.
Run the agent under evaluation so Netra can capture outputs and traces for each dataset item:
* **Single-turn**: Process each item's input through your agent. Use `Netra.evaluation.run_test_suite()` to run the full loop automatically, or execute your agent locally with Netra tracing enabled and submit results against the test run.
* **Multi-turn**: Use the `userMessages` array returned by `netra_create_multi_turn_test_run` — each entry includes a `testRunItemId`, `turnId`, and opening `userMessage`. Respond as your agent for each scenario and continue the conversation until every item completes. Use `Netra.simulation.run_simulation()` to automate the turn loop.
After your agent has finished processing all items, call `netra_get_test_run_summary` and `netra_get_test_run_details` to retrieve evaluator scores. Evaluators run only once agent execution completes and traces are linked to each test run item.
## Troubleshooting
* Ensure your JSON/TOML configs are properly formatted
* Confirm the MCP server is reachable at your region-specific MCP endpoint
* Verify your API key is correct and has not expired
* Restart the client after making config changes
## Next Step
After saving your config, restart the client and confirm the `netra` MCP server shows up in the available tools or MCP server list.
# Netra Skills
Source: https://docs.getnetra.ai/mcp/skills
Use Netra Skills to equip your AI coding assistant with proven instrumentation patterns, best practices, and automated evaluation workflows for your apps.
Netra MCP solves this by providing a standardized interface for integrating observability into your workflows.
Netra Skills bridge this gap by giving agents and developers ready-to-use patterns and primitives for reliable instrumentation. Instead of figuring everything out from scratch, you get a proven approach that works in real production environments.
## Quick Start
You can add these skills to your local environment in seconds:
```bash theme={null}
npx skills add KeyValueSoftwareSystems/netra-skills
```
The CLI will detect which agentic interfaces you have installed and drop the instructions in the right place. Agents that support skills should start to use them automatically.
***
## Available skills
| Skill | Best For | Install command |
| :---------------------------------- | :------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------- |
|
**netra-best-practices**
|
End-to-end instrumentation, observability, and evaluation.
|
#### When To Use
* You want one workflow that replaces separate setup and instrumentation skills.
* You need traceability from request context to span-level debugging.
* You need precise, schema-correct inputs for Netra MCP observability, dataset, evaluator, and test run tools.
* You need repeatable quality validation with evaluations and simulations.
* You need production-safe troubleshooting guidance.
#### Coverage
* **Setup and Baseline**: Initialize the SDK correctly for Python (FastAPI/OpenAI) and TypeScript (Express/OpenAI).
* **Context Tracking**: Implement request identity (`user_id`, `session_id`, `tenant_id`) and conversation logging.
* **Trace Debugging**: Execute precise `netra_query_traces`, `netra_get_trace_by_id`, and `netra_get_session_details` calls with correct schemas, sorting, and pagination.
* **Evaluation Setup**: Create datasets, map evaluators, and manage test runs using MCP tools with schema-correct inputs.
* **Instrumentation Strategy**: Choose between auto-instrumentation, decorators (`@workflow`, `@agent`, `@task`), and manual spans.
* **Advanced Observability**: Implement usage/cost tracking, action records, and OpenTelemetry custom metrics.
* **Evaluation**: Set up single-turn evaluations and multi-turn simulations with representative datasets.
* **Trace Analysis**: Use Netra MCP observability tools to debug regressions and inspect session-level cost and error totals.
* **Test Run Review**: Use `netra_get_test_run_summary` and `netra_get_test_run_details` to interpret evaluation results from your IDE.
***
## Related Resources
* [Netra MCP](/mcp/overview) - Connect Netra's remote MCP server to your editor
* [Tracing Quick Start](/quick-start/QuickStart_Tracing) - Fastest path to visible traces
* [Evaluation Quick Start](/quick-start/QuickStart_Evals) - Build datasets and evaluators
* [Simulation Quick Start](/quick-start/QuickStart_Simulation) - Test multi-turn behavior
* [SDK Overview](/sdk-reference/sdk/overview) - Core SDK concepts
# Netra
Source: https://docs.getnetra.ai/netra
Netra is the AI observability, evaluation, and simulation platform. Trace every LLM call, evaluate agent quality, and monitor costs in production.
Netra is a unified platform for AI application observability, analytics, privacy protection, and security, designed to give teams deep visibility into how their AI systems behave in development and production while safeguarding sensitive data.
Netra combines [multi-language SDKs](/sdk-reference/sdk/overview) for instrumenting AI applications, a centralized [customizable dashboard](/Dashboard/Custom-dashboard) for monitoring traces, sessions, agents, users, and costs, an AI-specific analytics engine for understanding LLM performance, vector database behavior, and prompt effectiveness.
Netra is built on [OpenTelemetry](https://opentelemetry.io/) standards, Netra integrates with leading [LLM providers](/Integrations/ai_providers/OPENAI), [vector databases](/Integrations/db-docs/Pinecone), and [AI frameworks](/Integrations/orchestrators/LangChain), enabling end-to-end tracing, evaluation, and alerting across complex AI workflows. Netra also supports [multi-tenancy](/Observability/Tenants), allowing you to isolate and track observability data across different tenants in your SaaS applications. Whether you are debugging latency issues, optimizing costs, enforcing security policies, or evaluating model quality, Netra provides the foundational observability layer needed to operate AI applications reliably at scale.
Set up Netra in your project and start tracing, evaluating, and simulating in minutes.
## Netra: Features
Simulation enables you to test your AI agents with realistic, multi-turn conversations driven by configurable user personas, goals, and fact-checking criteria — all evaluated automatically.
* [Run your first simulation](/quick-start/QuickStart_Simulation)
* [Know more about Simulation](/Simulation/Simulation-overview)
Evaluations help you systematically, repeatable evaluate your AI applications that quantify model quality, reasoning accuracy, tool correctness, and output reliability using production-grade metrics.
* [Run your first evals](/quick-start/QuickStart_Evals)
* [Know more about Evaluations](/Evaluation/Evaluation-overview)
Traces provides you end-to-end visibility into every AI request, capturing prompts, model calls, tools, latency, and token usage across the full execution lifecycle.
* [Setup Traces](/quick-start/QuickStart_Tracing)
* [Know more about Traces](/Observability/Traces/overview)
Alerts helps in proactive, real-time notifications that surface cost spikes, performance regressions, safety violations, and anomalous AI behavior before they impact users or budgets.
* [Get started with Alerts](/quick-start/QuickStart_Alerts)
* [Know more about Alerts](/Alert-rules/Alert-rules)
***
## Learn More
Integrate Netra into your LLM application
Test your AI agents with multi-turn conversations
Evaluate your LLM application
End-to-end examples and best practices
# Prompt Studio
Source: https://docs.getnetra.ai/prompt-management/prompt-studio
Netra Prompt Studio is a prompt management workspace to build, test, and version prompts without redeployments. Manage templates and A/B test in production.
View all prompts, select an existing one, or create a new prompt from a centralized library.
Compose prompts using **System** and **User** messages for chat-style model interactions.
Parameterize prompts with variables and provide runtime values using **Fields** or **JSON**.
Configure the **provider**, **model**, and runtime parameters so behavior stays consistent across environments.
Publish immutable prompt versions with **GitHub-style diffs** and review changes before release.
Track all published versions along with metadata and change history.
Attach labels such as `production`, `staging`, or `latest` to control which version applications invoke.
Create drafts from existing prompt versions and iterate without modifying previously published versions.
***
## When to Use Prompt Studio
Use Prompt Studio when you need a structured way to manage prompts across teams and environments, safely iterate on prompt changes, and control how updates are released to production systems.
| Situation | Benefit |
| ---------------------------------------------------------------------------------- | -------------------------------------- |
| You want a single place to manage prompts across services and teams | Centralized prompt management |
| You need controlled releases (dev → staging → production) without redeploying code | Safe production rollouts |
| You want an audit trail of prompt changes | Version diffs, comments, and history |
| You want to reproduce outcomes exactly | Pin prompt versions and model settings |
## Step by step guide to manage your prompts
Open **Prompt Studio** and click **Create Prompt** to start a new prompt.
Compose your prompt using **System** and **User** messages.
* **System messages** define role, rules, and guardrails.
* **User messages** contain the request template and runtime input.
Add **variables** for dynamic inputs such as user messages, context, or knowledge snippets.
Reference variables using placeholders like:
`{{user_message}}`
Specify how variable values are provided at runtime.
| Method | Description |
| ---------- | ----------------------------------------------------- |
| **Fields** | Structured inputs with name, type, and example values |
| **JSON** | Runtime JSON payload for programmatic invocation |
Select the **provider**, **model**, and runtime parameters that control how the prompt executes.
| Setting | Description |
| --------------- | --------------------------------------------------------------- |
| **Provider** | The model provider used to run the prompt (for example, OpenAI) |
| **Model** | The specific model that will generate responses |
| **Temperature** | Controls randomness in the output |
| **Max tokens** | Limits the maximum length of the response |
| **Top P** | Adjusts sampling diversity when generating text |
Keep **temperature low** for deterministic tasks such as support replies, extraction, or formatting. Increase it when creativity is needed.
Click **Publish** to convert the draft into an **immutable version**.
You can review the change, add a comment, and assign labels.
Use **Version History** and **labels** (`production`, or any custom label) to control which prompt version your application invokes.
***
## Immutability and Iteration
Published prompt versions are **immutable**, meaning they cannot be edited after they are published. This ensures that production behavior remains predictable and that every version can be audited or reproduced later.
When you need to update a prompt, you create a new draft, make your changes, and publish a new version.
Select any published version and create a **draft** from it. The draft acts as a working copy where changes can be made safely.
Modify messages, variables, or model settings in the draft. You can iterate and test freely without affecting the currently published version.
When your changes are ready, click **Publish** to create a new immutable version. You can then assign labels such as `staging` or `production` to control which version is used.
***
## Best Practices
| Practice | Why It Matters |
| ------------------------------------------ | -------------------------------------------------------- |
| Treat prompts like **APIs** | Stable inputs and predictable outputs |
| Write publish comments like **changelogs** | Improves traceability and collaboration |
| Use labels as **release channels** | Manage deployments safely (`dev → staging → production`) |
| Keep prompts **focused** | Compose multiple prompts instead of one large prompt |
| Write clear **publish comments** | Helps teammates quickly understand what changed |
| Keep changes **small and descriptive** | Makes it easier to track who changed what and why |
***
## FAQ
Published versions are immutable. To make changes, create a new **draft** from a published version, update it, and publish a new version.
Move the **`production label`** to the version you want your application to use.
# Stress Testing
Source: https://docs.getnetra.ai/prompt-management/stress-testing
Validate prompt reliability at scale by running automated multi-model tests with configurable evaluators and AI-powered analysis before publishing.
Prompts don't fail in obvious ways — they drift, produce inconsistent outputs, and degrade quietly across different models and edge cases. Netra's Stress Testing framework makes that invisible unreliability visible, giving you a structured, repeatable way to measure prompt performance across models, runs, and evaluators before every publish.
Stress Testing lives inside Prompt Studio. Learn how to build, test, and version prompts.
***
## Why Stress Testing Matters
Without systematic reliability testing, a prompt that works once might fail unpredictably in production. Netra helps you answer critical questions with confidence:
| Question | What Netra Measures |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| Will my prompt produce consistent outputs at scale? | Pass rates across 1–100 repeated executions per model |
| Which model delivers the best reliability for my use case? | Side-by-side scoring of up to 5 models on identical inputs |
| Does my prompt meet latency, cost, and quality requirements? | Per-run evaluations against configurable thresholds |
| Did my latest prompt edit introduce a regression? | Version-over-version reliability trends with score history |
| Is my prompt production-ready? | AI-generated health assessment (Healthy / Needs Optimization / Critical Issues) |
***
## Core Building Blocks
The Stress Testing framework is built on three interconnected pillars:
### Models & Runs
Configure which models to test and how many executions to perform. Netra runs your prompt repeatedly against each selected model to surface inconsistencies that single runs miss.
| Feature | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------- |
| **Multi-model selection** | Test up to 5 models simultaneously in a single stress test |
| **Run presets** | Quick-select 1 (spot check), 25 (early signal), 50 (reliable), or 100 (conclusive) runs per model |
| **Custom runs** | Set any value from 1 to 100 runs per model |
| **Model parameters** | Override temperature, top P, max tokens, and penalties per model |
| **Execution summary** | See total executions before starting (models × runs per model) |
Your currently configured model in Prompt Studio is pre-selected as the first model. Add additional models to compare performance across providers like OpenAI, Anthropic, and Google.
### Evaluators
Evaluators score every individual run against your quality and performance criteria. Netra provides 6 evaluator types:
| Evaluator | What It Measures | Configuration Required |
| ------------------- | ------------------------------------------ | --------------------------------------------- |
| **LLM-as-Judge** | Subjective quality assessed by another LLM | Provider, model, optional reference answer |
| **Latency** | Response time in milliseconds | Expected latency threshold (ms) |
| **Cost** | Per-run execution cost | Expected cost threshold |
| **Token Count** | Total tokens consumed | Expected token count threshold |
| **JSON Validation** | Output matches expected JSON structure | Expected JSON object, optional keys to ignore |
| **Regex Match** | Output matches a regular expression | Regex pattern with optional flags |
Each evaluator supports **pass criteria** — numerical thresholds (e.g., latency \< 2000ms) or boolean conditions (pass/fail) — so you define exactly what "good" means for your use case.
### Results & Analysis
After all runs complete, Netra aggregates scores and generates actionable insights:
| Feature | Description |
| ----------------------------- | ---------------------------------------------------------------------------------------- |
| **AI Analysis** | LLM-generated summary with health badge (Healthy / Needs Optimization / Critical Issues) |
| **Per-model metrics** | Average latency, cost, tokens, and evaluator score for each model |
| **Pass rate breakdown** | Visual distribution of all-pass, all-fail, partial, and error outcomes |
| **Radar chart** | Multi-axis comparison of models across all evaluators |
| **Evaluator breakdown table** | Detailed pass/fail/score per evaluator per model |
| **Run logs** | Paginated, filterable list of every individual execution with expandable output |
| **Reliability trends** | Score history chart across versions (available after 2+ completed tests) |
***
## Step-by-Step Guide
### Prerequisites
* A prompt with at least one message in Prompt Studio
* All prompt variables filled with representative values
* At least one model provider configured in your workspace
### Run Your First Stress Test
In Prompt Studio, ensure your prompt variables are filled with values that represent real usage. Click the **Stress Test** button next to the **Run** button in the actions panel.
All prompt variables must be filled before starting a stress test. If any variables are empty, the button will trigger a validation error.
The configuration modal opens on **Step 1: Configure**. Your current model is pre-selected. Add more models to compare:
* Click **Add Model** to open the model picker
* Select a provider and model from the dropdown
* Add up to **5 models** total
* Remove models by clicking the × on each model chip
Choose how many times each model should execute the prompt. More runs produce more statistically reliable results.
| Preset | Runs | Label | Best For |
| ------ | ---- | ------------ | ------------------------------ |
| 1 | 1 | Spot check | Quick sanity check |
| 25 | 25 | Early signal | Initial quality assessment |
| 50 | 50 | Reliable | Production readiness (default) |
| 100 | 100 | Conclusive | High-confidence decisions |
You can also enter a custom value (1–100). The footer shows total executions:
> **3 models × 50 runs = 150 total executions**
Click **Next** to move to **Step 2: Evaluators**. Select which evaluators to use and configure each one:
1. Toggle evaluators on/off using the checkbox on each card
2. Expand a card to configure its settings
3. Set **pass criteria** — either a numerical threshold with an operator or a boolean condition (is\_true / is\_false)
4. For **LLM-as-Judge**, select the provider and model that will act as the judge
You must select at least one evaluator. The counter at the top shows how many evaluators are currently selected.
Click **Run** at the bottom of the modal. The test starts immediately in the background:
* A **progress banner** appears above the action buttons showing real-time status
* Executions are processed asynchronously — you can continue editing your prompt
* Status polls automatically every 4 seconds
Also, you can click View Results button to open Results modal and can stop any running Stress Test
The banner updates in real-time:
* **Running** — Shows `Running · X/Y runs` with an animated indicator
* **Multiple tests** — Shows `N tests running · X/Y runs` when concurrent tests are active
* **Completion** — A success toast appears with the overall score and a link to results
You can also access past and running tests anytime via the **View past stress tests** link.
Access **Run History** via the banner link or "View past stress tests." The history modal shows:
* A table of all past stress tests with version/draft, score, duration, status, and who ran it
* A **Reliability Chart** (appears after 2+ completed tests) showing average score trends over time
* Filter by version to compare performance across prompt iterations
* Stop any running test directly from the history
On clicking each row in Run History table, a new modal with two tabs will open. The **Overview** tab shows:
**AI Analysis** — A generated summary with a health badge:
* 🟢 **Healthy** — Prompt is reliable and production-ready
* 🟡 **Needs Optimization** — Some issues detected; review recommendations
* 🔴 **Critical Issues** — Significant reliability problems; do not publish
**Per-Model Metrics** — A grid showing each model's average latency, total cost, total tokens, and average evaluator score.
**Pass Rate Breakdown** — A stacked bar per model showing the distribution of all-pass, all-fail, error, and partial outcomes.
**Radar Chart** — A visual comparison of all models across every evaluator axis.
**Evaluator Breakdown Table** — Tabular view of exact scores per model per evaluator.
Switch to the **Logs** tab for granular inspection of every individual run:
* **Filter by model** — Focus on a specific model's runs
* **Filter by eval status** — Show only Passed, Partial, Failed, or Error runs
* **Expand any row** — See the full model output and per-evaluator scores with pass/fail reasons
***
## Evaluators in Detail
### LLM-as-Judge
Uses a separate LLM to evaluate the quality of each run's output. The judge model reads the prompt, the output, and (optionally) a reference answer, then produces a score.
| Setting | Description |
| -------------------- | ------------------------------------------------------------ |
| **Provider** | The model provider for the judge (e.g., OpenAI) |
| **Model** | The specific judge model (e.g., gpt-4o-mini) |
| **Reference Answer** | Optional ground truth for more consistent scoring |
| **Pass Criteria** | Numerical score threshold (default: `>= 0.5` on a 0–1 scale) |
### Latency
Measures response time in milliseconds and checks against your expected threshold.
| Setting | Description |
| -------------------- | --------------------------------------------- |
| **Expected Latency** | Maximum acceptable response time (ms) |
| **Pass Criteria** | Boolean — passes if actual is within expected |
### Cost
Measures per-run execution cost and checks against your budget threshold.
| Setting | Description |
| ----------------- | --------------------------------------------- |
| **Expected Cost** | Maximum acceptable cost per run |
| **Pass Criteria** | Boolean — passes if actual is within expected |
### Token Count
Measures total tokens consumed (prompt + completion) and checks against your limit.
| Setting | Description |
| ------------------- | --------------------------------------------- |
| **Expected Tokens** | Maximum acceptable token count |
| **Pass Criteria** | Boolean — passes if actual is within expected |
### JSON Validation
Deep-compares the model output against an expected JSON structure to validate structured outputs.
| Setting | Description |
| ------------------ | -------------------------------------------------- |
| **Expected JSON** | The JSON object the output should match (required) |
| **Keys to Ignore** | Optional list of keys to exclude from comparison |
| **Pass Criteria** | Boolean — passes if structure matches |
### Regex Match
Tests the model output against a regular expression pattern.
| Setting | Description |
| ----------------- | ----------------------------------------------------- |
| **Pattern** | The regex pattern to match against the output |
| **Flags** | Optional regex flags (e.g., `i` for case-insensitive) |
| **Pass Criteria** | Boolean — passes if pattern matches output |
***
## Understanding Pass Criteria
Every evaluator uses pass criteria to determine if a run passes or fails:
### Numerical Pass Criteria
For evaluators that produce a numeric score (like LLM-as-Judge):
| Operator | Meaning | Example |
| -------- | ------------------------ | -------------- |
| `>=` | Greater than or equal to | `Score >= 0.8` |
| `<=` | Less than or equal to | `Score <= 0.3` |
| `>` | Greater than | `Score > 0.5` |
| `<` | Less than | `Score < 0.5` |
| `=` | Equal to | `Score = 1.0` |
### Boolean Pass Criteria
For evaluators that produce a true/false result (latency, cost, token, JSON, regex):
| Condition | Meaning |
| ---------- | ----------------------------------------- |
| `is_true` | Run passes if the evaluator returns true |
| `is_false` | Run passes if the evaluator returns false |
***
## Test Statuses
| Status | Meaning | Actions Available |
| ------------- | ---------------------------------------------------- | ---------------------- |
| **Running** | Executions in progress; banner shows live count | Stop (cancel) |
| **Completed** | All runs finished; results and AI analysis available | View results |
| **Failed** | One or more runs could not complete execution | View results |
| **Partial** | Some models completed, others failed | View results |
| **Cancelled** | Manually stopped by the user | View completed results |
***
## Use Cases
### Pre-Publish Validation
Ensure your prompt meets quality bars before releasing to production:
1. Finish editing your prompt in Prompt Studio
2. Run a stress test with 50+ runs across your target production model
3. Configure evaluators matching your quality criteria (e.g., LLM-as-Judge `>= 0.8`, latency `< 3000ms`)
4. Review the AI health analysis — check if status is "Healthy"
### Model Selection & Comparison
Choose the best model for your use case with data, not guesses:
1. Add all candidate models (up to 5) to a single stress test
2. Set runs to 50+ for statistically meaningful results
3. Compare the radar chart and per-model scores across evaluators
4. Select the model with the best balance of quality, speed, and cost
### Regression Detection
Catch quality degradation when iterating on prompts:
1. Run a stress test on your current published version as a baseline
2. Create a draft with your proposed changes
3. Run the same stress test configuration on the draft
4. Compare scores in Run History — the reliability chart shows trends across versions
***
## Best Practices
| Practice | Why It Matters |
| ------------------------------------------------------- | ------------------------------------------------------------ |
| Run stress tests before every publish | Catches regressions before they reach production |
| Use at least 50 runs for production decisions | Small sample sizes produce misleading results |
| Test with multiple models | Reveals model-specific weaknesses you wouldn't see otherwise |
| Configure evaluators relevant to your specific use case | Generic tests miss domain-specific failure modes |
| Track reliability across versions | Spots degradation trends before production impact |
| Provide reference answers for LLM-as-Judge when needed | Improves scoring consistency with a ground truth |
| Keep temperature low for deterministic tasks | Reduces output variance and produces cleaner test results |
***
## FAQ
Yes. Stress tests work on both published versions and drafts. This lets you validate changes before publishing a new version.
Up to 5 models in a single stress test. Each model runs independently with its own parameter configuration (temperature, top P, max tokens, etc.).
You can run up to 100 executions per model. The UI provides quick presets (1, 25, 50, 100), but you can enter any custom value within the range.
Yes. Open the Run History modal and click the **Stop** button on the running test.
After all runs finish, an AI model analyzes the aggregated scores, pass rates, and per-model performance. It produces a natural-language summary explaining what went well and what needs attention, along with a health rating (Healthy, Needs Optimization, or Critical Issues).
Yes. The progress banner tracks multiple concurrent tests and shows aggregated progress. Each test runs independently in the background.
The reliability trend chart appears in Run History after you have 2 or more completed stress tests on the same prompt. It shows average score progression over time.
Each run's score is the average of all evaluator scores for that run (boolean results become 1 for pass, 0 for fail). The overall stress test score is the average across all runs and all models.
If a model call fails, that run is marked as failed. The system retries up to 3 times with exponential backoff. If all retries fail, the run is recorded as failed and counted in the failure metrics. Other runs continue unaffected.
***
## Related
* [Prompt Studio](/prompt-management/prompt-studio) — Build, test, and version prompts in a centralized workspace
* [Evaluation Overview](/Evaluation/Evaluation-overview) — Broader evaluation framework for datasets and test runs
* [Simulation Overview](/Simulation/Simulation-overview) — Test AI agents with multi-turn conversations
* [Traces](/Observability/Traces/overview) — Debug individual executions with full trace visibility
# Overview
Source: https://docs.getnetra.ai/quick-start/Overview
Get started with Netra in minutes. Learn to instrument your AI app, view traces, run automated evaluations, and configure AI performance alerts.
Let us get started by signing up to your Netra account.
* Go to the [Netra Sign Up page](https://app.eu.getnetra.ai/signup)
* Select your **Data Region** (USA or Europe)
* Enter your **Email address** or click on **Sign up with Google** and continue
* A verification code will be sent to your **Email address**
* Once verified, enter your **First Name** and **Last Name**
* Create a password and click **Create Account** — you will be redirected to the Netra dashboard
* Go to the [Netra Login page](https://app.eu.getnetra.ai/login)
* Select your **Data Region** (USA or Europe)
* Login using **Email address** and **Password** or login using your **Google account**
***
## Data Regions
Netra is available in two data regions:
| Region | Location | API Endpoint |
| ------ | ------------------------ | -------------------------------------- |
| **US** | N. Virginia (us-east-1) | `https://api.getnetra.ai/telemetry` |
| **EU** | Frankfurt (eu-central-1) | `https://api.eu.getnetra.ai/telemetry` |
Regions are strictly separated, and no data is shared across regions. Choosing a region close to you can help improve speed and comply with local data residency laws and privacy regulations. You can have accounts in multiple regions, but each region requires a separate subscription.
***
## You're all set to begin
Once you sign up and log in, you're automatically part of a **default organization** with a **default project** ready to go. You can start instrumenting your AI application right away, or take a moment to organize your workspace first.
**Want to keep things organized?** Create separate projects to organize your work however makes sense for you — by environment, application, team, or any other structure. Or jump straight in with the default project — you can always restructure later.
### Step 1: Choose your project (optional)
You can use the default project to get started immediately, or create a dedicated project for your application:
Projects are workspaces that help you organize observability data across different environments or applications.
1. Go to **Settings** on the bottom left side of the screen
2. Click on **Projects** under **Organisation**
3. Click **Create Project** on the top right
4. Give your project a name and click create
You can always switch between projects or create more as your needs evolve.
### Step 2: Create your API key
Every request to Netra needs to be authenticated. Let's generate your API key:
1. Go to **Settings** → **Project** → **API keys**
2. Click **Create Secret Key**
3. Give your key a name and click **Create Key**
Copy and store this token securely — you won't be able to see it again. This key authenticates all your API requests.
### Step 3: Configure Netra Credentials as your Environment Variables
Add your API key and endpoint to your environment variables. Use the endpoint that matches your selected data region:
```bash US Region theme={null}
export NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry"
export NETRA_API_KEY="your-api-key-here"
```
```bash EU Region theme={null}
export NETRA_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
export NETRA_API_KEY="your-api-key-here"
```
### Step 4: Install the SDK
Install Netra in your environment with a single command:
```bash pip theme={null}
pip install netra-sdk
```
```bash poetry theme={null}
poetry add netra-sdk
```
```bash npm theme={null}
npm install netra-sdk
```
```bash yarn theme={null}
yarn add netra-sdk
```
### Step 5: Initialize Netra in your application
Add two lines to your application to start capturing traces:
```python Python theme={null}
from netra import Netra
Netra.init(app_name="Your application name")
```
```typescript TypeScript theme={null}
import { Netra } from 'netra-sdk';
Netra.init({ appName: 'Your application name' });
```
That's it! Netra automatically uses your environment variables for authentication.
#### Custom Configuration
If you prefer to configure credentials directly in code or need additional options:
```python Python theme={null}
from netra import Netra
api_key = "Your API key"
headers = f"x-api-key={api_key}"
Netra.init(
app_name="Your application name",
headers=headers,
trace_content=True,
environment="production"
)
```
```typescript TypeScript theme={null}
import { Netra } from 'netra-sdk';
Netra.init({
appName: 'Your application name',
apiKey: 'Your API key',
traceContent: true,
environment: 'production'
});
```
***
## What's next?
You're now ready to explore Netra's capabilities. Choose where to start based on what matters most to your team:
Test your agents with realistic, multi-turn conversations using configurable personas and goals
Measure quality, accuracy, and reliability with LLM-as-Judge and code evaluators
Capture every LLM call, tool invocation, and decision your agent makes
Get notified about anomalies, cost spikes, and performance issues
# Quick Start: Alerts
Source: https://docs.getnetra.ai/quick-start/QuickStart_Alerts
Set up your first Netra alert rule in minutes. Get notified via Slack or email when your AI system exceeds cost, latency, or error rate thresholds.
This guide walks you through creating your first [alert rule](/Alert-rules/Alert-rules) to monitor your AI system and get notified when something goes wrong.
## 1. Prerequisites
Before setting up alerts, ensure you have:
* [Netra SDK installed and initialized](/quick-start/QuickStart_Tracing)
* At least one traced LLM call in your dashboard
* Access to a notification channel (Email or Slack)
## 2. Create a Contact Point
Contact points define where notifications are sent when alerts trigger.
Go to **Settings → Contact Points** from the left navigation panel.
Click **Create Contact Point**.
* **Name**: Enter a descriptive name (e.g., "Engineering Alerts")
* **Integration**: Select **Email** or **Slack**
* **Details**: Enter email address or Slack webhook URL
Click **Create** to save your contact point.
For Slack, create an [Incoming Webhook](https://api.slack.com/messaging/webhooks) in your workspace and paste the URL.
## 3. Create Your First Alert Rule
Go to **Alert Rules** from the left navigation panel.
Click **Create Alert Rule** in the top right corner.
* **Alert Name**: "High Cost Alert"
* **Description**: "Notifies when a single request exceeds \$0.50"
Choose the contact point you created in Step 2.
Select **Trace** to monitor entire requests.
Choose **Cost** to monitor token/API spend.
* **Operator**: Greater than (>)
* **Threshold**: 0.50
Click **Create** to activate the alert.
## 4. Test Your Alert
Trigger a trace that exceeds your threshold:
```python Python theme={null}
from netra import Netra
from openai import OpenAI
Netra.init(
app_name="alert-test",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
)
client = OpenAI()
# Generate a longer response to increase cost
response = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "user", "content": "Write a detailed 500-word essay about AI safety."}
],
)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import OpenAI from "openai";
await Netra.init({
appName: "alert-test",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
});
const client = new OpenAI();
// Generate a longer response to increase cost
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [
{ role: "user", content: "Write a detailed 500-word essay about AI safety." }
],
});
```
## 5. Verify Notification
After triggering a high-cost trace:
1. Check your configured contact point (email inbox or Slack channel)
2. You should receive a notification with:
* Alert name and description
* Triggered value (actual cost)
* Timestamp
* Link to the trace
Alerts evaluate in real-time. Notifications arrive within seconds of the threshold being breached.
## Common Alert Examples
### Latency Alert
Monitor response time:
* **Scope**: Trace
* **Metric**: Latency
* **Trigger**: > 5000ms
### Error Rate Alert
Catch failures:
* **Scope**: Span
* **Metric**: Error
* **Trigger**: = true
### Token Usage Alert
Control token consumption:
* **Scope**: Trace
* **Metric**: Token Count
* **Trigger**: > 10000
## Troubleshooting
| Issue | Solution |
| ------------------------- | ------------------------------------------------------------ |
| No notifications received | Verify contact point configuration and check spam folder |
| Alert not triggering | Confirm threshold is lower than actual values in your traces |
| Too many alerts | Increase threshold or add time-based aggregation |
## Next Steps
Learn advanced alert configurations
Understand the data alerts monitor
View metrics to set appropriate thresholds
Monitor per-customer metrics
# Quick Start: Evaluation
Source: https://docs.getnetra.ai/quick-start/QuickStart_Evals
Run your first AI evaluation in Netra in minutes. Create a dataset, configure evaluators, and measure your LLM output quality with automated test runs.
This guide walks you through setting up [evaluations](/Evaluation/Evaluation-overview) to measure your AI system's accuracy, quality, and reliability.
## 1. Prerequisites
Before setting up evaluations, ensure you have:
* [Netra SDK installed and initialized](/quick-start/QuickStart_Tracing)
* At least one traced LLM call in your dashboard
* Your API key configured
## 2. Create a Dataset
[Datasets](/Evaluation/Datasets) are collections of test cases that define inputs and expected outputs for your AI system.
### Option A: Create from Traces (Recommended)
Convert real-world interactions into test cases:
Go to **Observability → Traces** and find a trace you want to use as a test case.
Click the **Add to Dataset** button on the trace.
* Enter a dataset name (e.g., "Customer Support QA") - Add optional tags for
organization - Review the input prompt - Provide the expected output - Click
**Next**
Choose evaluators to score your AI's performance (see Step 3).
### Option B: Create Manually
Navigate to **Evaluation → Datasets** and click **Create Dataset**.
* Enter a dataset name - Select **Single Turn** for request/response pairs -
Choose **Add manually**
For each test case, provide:
* **Input**: The prompt or question
* **Expected Output**: The ideal response
* **Metadata** (optional): Additional context
## 3. Configure Evaluators
[Evaluators](/Evaluation/Evaluators) score your AI's outputs against defined criteria. Netra offers two types:
### LLM as Judge
Best for subjective quality assessment:
* **Answer Correctness**: Does the response match the expected answer?
* **Relevance**: Is the response relevant to the question?
* **Hallucination Detection**: Does the response contain fabricated information?
* **Toxicity**: Is the content safe and appropriate?
### Code Evaluators
Best for deterministic checks:
* **JSON Validation**: Verify JSON structure and schema
* **Regex Matching**: Pattern-based validation
* **Custom Logic**: Write JavaScript or Python for specific rules
When creating your dataset, click **Next** to reach the evaluator selection screen.
Browse [pre-built evaluators](/Evaluation/Evaluators#library) in categories: -
Quality - Performance - Agentic - Guardrails
Configure how evaluator variables map to your data:
* **Dataset field**: Use values from your test cases
* **Agent response**: Use the actual LLM output
* **Execution data**: Use trace metadata
## 4. Run an Evaluation
Once your dataset is configured with evaluators:
Open your dataset and copy the **Dataset ID** displayed at the top.
Run your AI system with the dataset inputs. Evaluations execute automatically
when traces are created.
Navigate to **Evaluation → [Test Runs](/Evaluation/TestRuns)** to see your evaluation results.
## 5. Analyze Test Run Results
Click on a [test run](/Evaluation/TestRuns) to view detailed results:
### Summary Metrics
* **Total Cost**: Aggregate cost of all LLM calls
* **Average Latency**: Response time across test cases
* **Pass/Fail Rate**: Overall success rate
### Per-Test-Case Results
Each test case shows:
| Field | Description |
| ---------------- | ------------------------------------- |
| Input | The prompt sent to the AI |
| Expected Output | Your defined ideal response |
| Task Output | The actual AI response |
| Run Status | Shows Running status |
| Eval Status | Pass/Fail indicator |
| Evaluator Scores | Individual scores from each evaluator |
| View Trace | Link to the full execution trace |
## Troubleshooting
| Issue | Solution |
| ---------------------- | ----------------------------------------------------------------------- |
| No test runs appearing | Ensure your dataset has evaluators configured and traces are being sent |
| Evaluator errors | Test your evaluator in the Playground before adding to datasets |
| Unexpected failures | Check variable mappings in evaluator configuration |
## Next Steps
Learn advanced dataset management
Create custom evaluation logic
Deep dive into test run analysis
Understand the full evaluation framework
# Quick Start: Simulation
Source: https://docs.getnetra.ai/quick-start/QuickStart_Simulation
Run your first AI agent simulation in Netra. Set up multi-turn conversations with a simulated user to test agent behavior before going to production.
This guide walks you through setting up [simulations](/Simulation/Simulation-overview) to test your AI agents in realistic, goal-oriented conversations.
## 1. Prerequisites
Before setting up simulations, ensure you have:
* [Netra SDK installed and initialized](/quick-start/Overview)
* Your API key configured
## 2. Configure Evaluators
[Evaluators](/Simulation/Evaluators) assess the entire simulated conversation after it completes. Netra provides 8 library evaluators in two categories:
**Quality** (6 evaluators): Guideline Adherence, Conversation Completeness, Profile Utilization, Conversational Flow, Conversation Memory, Factual Accuracy
**Agentic** (2 evaluators): Goal Fulfillment, Information Elicitation
All evaluators use LLM-as-Judge with a default pass threshold of >= 0.6.
Start with Goal Fulfillment and Factual Accuracy as your core evaluators, then add more as needed. You can also create custom evaluators before setting up your dataset.
## 3. Create a Multi-Turn Dataset
[Datasets](/Simulation/Datasets) define the scenarios you want to test—multi-turn conversations with specific goals.
Go to **Evaluation → Datasets** and click **Create Dataset**.
* **Name**: "Customer Refund Scenarios" - **Type**: Select **Multi-turn** -
**Data Source**: **Add manually** - Click **Next**
Define your simulation scenario:
| Field | Value |
| ----------------- | ----------------------------------------------------------------------------------- |
| **Scenario Goal** | "The customer wants to get a refund for a product that arrived damaged 15 days ago" |
| **Max Turns** | 5 (recommended for support scenarios) |
| **User Persona** | Frustrated 😤 (tests patience and de-escalation) |
| **Provider** | OpenAI |
| **Model** | GPT-4.1 (for realistic user simulation) |
Click **Next**
Provide context and success criteria:
**Simulated User Data** (JSON format):
```json theme={null}
{
"order_number": "ORD-123456",
"purchase_date": "2024-01-15",
"product_name": "Wireless Headphones",
"order_total": "$129.99"
}
```
**Fact Checker** (what the agent MUST communicate):
```json theme={null}
{
"refund_processing_time": "5-7 business days",
"refund_method": "Original payment method",
"return_label_delivery": "Within 24 hours via email"
}
```
Click **Next**
Choose evaluators to score the conversation. Select from the library evaluators you reviewed in Step 2, or any custom evaluators you created:
* **Agentic**: Goal Fulfillment (did the agent achieve the objective?)
* **Quality**: Factual Accuracy (were facts communicated correctly?), Conversation Completeness
Click **Next** then **Create Dataset**
## 4. Run Your First Simulation
Once your dataset is configured, trigger simulations through your agent code:
Open your dataset in the Netra dashboard and copy the **Dataset ID** from the top of the page.
The simulation runs automatically when your agent code executes. Ensure your
agent is instrumented with Netra tracing.
Navigate to **Evaluation → Test Runs** and filter by **Multi** turn type to see your simulation in progress.
## 5. Review Results
Go to **Evaluation → Test Runs** and click on your completed simulation.
Review high-level performance: - Total scenarios run - Pass/fail rate -
Average cost and latency
Click on any scenario to view: - **Conversation tab**: Full turn-by-turn
dialogue - **Evaluation Results tab**: Turn-level and session-level scores -
**Scenario Details tab**: Goal, user data, and facts
Click **View Trace** on any turn to see detailed execution traces for debugging.
## What's Next?
Learn more about the simulation framework and use cases
Build complex multi-turn scenarios with custom personas
Create custom evaluators for your specific requirements
## Common Patterns
### Testing Customer Support
* **Personas**: Test with Frustrated, Confused, and Neutral personas
* **Evaluators**: Conversation Completeness, Factual Accuracy, Guideline Adherence
* **Max Turns**: 4-6 for typical support scenarios
### Testing Technical Assistants
* **Personas**: Confused (needs extra clarification)
* **Evaluators**: Conversational Flow, Conversation Completeness, Goal Fulfillment
* **Max Turns**: 6-8 for complex troubleshooting
### Guideline Compliance Testing
* **Scenarios**: Create edge cases that challenge agent boundaries
* **Evaluators**: Guideline Adherence, Goal Fulfillment
* **Personas**: Frustrated (more likely to push boundaries)
Start with 3-5 scenarios covering your most critical use cases, then expand
coverage as you gain confidence in the system.
# Quick Start: Tracing
Source: https://docs.getnetra.ai/quick-start/QuickStart_Tracing
Send your first AI trace to Netra in under 5 minutes. Install netra-sdk, initialize with your API key, and start monitoring LLM calls instantly.
This guide walks you through sending your first trace to Netra.
## Prerequisites
Before you begin, make sure you've completed the initial setup from the [Getting Started guide](/quick-start/Overview):
* Created your API key
* Configured environment variables
* Installed the Netra SDK
**1. Install the SDK**
```bash pip theme={null}
pip install netra-sdk
```
```bash poetry theme={null}
poetry add netra-sdk
```
```bash npm theme={null}
npm install netra-sdk
```
```bash yarn theme={null}
yarn add netra-sdk
```
**2. Set environment variables**
```bash US Region theme={null}
export NETRA_API_KEY="your-api-key-here"
export NETRA_OTLP_ENDPOINT="https://api.getnetra.ai/telemetry"
```
```bash EU Region theme={null}
export NETRA_API_KEY="your-api-key-here"
export NETRA_OTLP_ENDPOINT="https://api.eu.getnetra.ai/telemetry"
```
## Send Your First Trace
Add Netra initialization at the start of your application, then run a simple LLM call. Netra will automatically capture the trace.
```python Python theme={null}
import os
from netra import Netra
from openai import OpenAI
# Initialize Netra
Netra.init(
app_name="my-ai-app",
environment="development",
headers=f"x-api-key={os.getenv('NETRA_API_KEY')}",
)
# Make an LLM call - this will be automatically traced
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello, how are you?"}],
)
print(response.choices[0].message.content)
```
```typescript TypeScript theme={null}
import { Netra } from "netra-sdk";
import OpenAI from "openai";
async function main() {
// Initialize Netra (must await to ensure instrumentations are ready)
await Netra.init({
appName: "my-ai-app",
environment: "development",
headers: `x-api-key=${process.env.NETRA_API_KEY}`,
});
// Make an LLM call - this will be automatically traced
const client = new OpenAI();
const response = await client.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello, how are you?" }],
});
console.log(response.choices[0].message.content);
}
main();
```
Netra automatically instruments popular LLM libraries like OpenAI, Anthropic, LangChain, and more. See [Auto Instrumentation](/Observability/Traces/auto-instrumentation) for the full list.
## View Your Trace
Open the [Netra Dashboard](https://app.eu.getnetra.ai/login) and navigate to **Observability → Traces**. You should see your trace appear within a few seconds.
Click on the trace to see the full timeline, including:
* LLM call details and parameters
* Token usage and cost
* Latency breakdown
* Full prompt and response content
## Troubleshooting
| Issue | Solution |
| ---------------------- | -------------------------------------------------------------------- |
| No traces appearing | Verify `NETRA_API_KEY` and `NETRA_OTLP_ENDPOINT` are set correctly |
| LLM calls not traced | Ensure `Netra.init()` is called **before** importing the LLM library |
| Missing prompt content | Set `traceContent: true` in initialization |
## Learn more
Learn what traces are and how to use them
See all supported libraries and frameworks
Add semantic context with @workflow, @agent, @task
Create custom spans for fine-grained control
# Custom Metrics
Source: https://docs.getnetra.ai/sdk-reference/custom-metric
Emit custom metrics using the Netra Meter API
The Netra SDK exposes a `meter` API that lets you:
* **Create custom metrics** – Counters, histograms, gauges, and more
* **Export to Netra backend** – Using OpenTelemetry OTLP/HTTP JSON
This page shows how to use `meter` utility in Netra to define and record your own metrics.
## **Getting Started**
The metrics pipeline is configured automatically when you enable metrics in `Netra.init`.
```python theme={null}
from netra import Netra
Netra.init(
app_name="sample-app",
enable_metrics=True,
)
# Access a Meter (OpenTelemetry metrics entry point)
meter = Netra.get_meter("ai-service")
```
If `enable_metrics` is `False` or no OTLP endpoint is configured, Netra installs a no-op `MeterProvider`. Your code can still call `get_meter` and record metrics safely, they will simply be discarded.
## **get\_meter**
Return an OpenTelemetry `Meter` instance backed by Netra’s global MeterProvider.
```python theme={null}
from netra import Netra
Netra.init(
app_name="sample-app",
enable_metrics=True,
)
# Recommended: use your service or module name
meter = Netra.get_meter("payment_service")
```
### **Parameters**
| **Parameter** | **Type** | **Description** |
| :------------ | :------- | :------------------------------------------------------------------------------------------------------------------ |
| `name` | `str` | Instrumentation scope name – typically your service or module name (e.g. `"order_service"`). Defaults to `"netra"`. |
| `version` | `str?` | Optional instrumentation scope version string (e.g. `"1.0.0"`). |
## **Creating Instruments**
The returned `Meter` is a standard OpenTelemetry `Meter` and supports the usual instrument types.
### **Counter**
Monotonically increasing value (e.g. request counts).
```python theme={null}
request_counter = meter.create_counter(
name="http.requests",
description="Number of HTTP requests processed",
unit="1"
)
request_counter.add(
amount=1,
attributes={"route": "/api/health", "status": "ok"}
)
```
### **Up Down Counter**
Value that can increase or decrease (e.g. active connections).
```python theme={null}
active_connections = meter.create_up_down_counter(
name="connections.active",
description="Number of active client connections",
unit="1"
)
active_connections.add( # connection opened
amount=1,
attributes={"region": "us-east-1"}
)
active_connections.add( # connection closed
amount=-1,
attributes={"region": "us-east-1"}
)
```
### **Histogram**
Distribution of measurements (e.g. latency, payload size).
```python theme={null}
latency = meter.create_histogram(
name="db.query.latency_ms",
unit="ms",
description="Database query latency",
)
latency.record(
amount=15.3,
attributes={"operation": "read", "table": "users"}
)
latency.record(
amount=30.7,
attributes={"operation": "write", "table": "orders"}
)
```
### **Observable Instruments (Advanced)**
You can also create observable instruments (pull-based metrics), such as periodic gauges or counters, by registering callbacks.
Example for CPU and memory utilization gauges:
```python theme={null}
import psutil
from opentelemetry.metrics import Observation
def _cpu_utilization_callback(options):
value = psutil.cpu_percent()
yield Observation(
value=value,
attributes={"resource": "cpu"}
)
def _memory_usage_callback(options):
mem = psutil.virtual_memory()
yield Observation(
value=mem.used,
attributes={"unit": "bytes"}
)
cpu_gauge = meter.create_observable_gauge(
name="system.cpu.utilization",
description="CPU utilization percentage",
callbacks=[_cpu_utilization_callback],
)
memory_gauge = meter.create_observable_gauge(
name="system.memory.used_bytes",
description="Used memory in bytes",
callbacks=[_memory_usage_callback],
)
```
These callbacks are invoked periodically by the metrics reader configured by Netra.
## **Metrics Export Pipeline**
Netra configures the metrics pipeline via`MetricsSetup()`, which:
* **Uses OTLP/HTTP JSON**\
Wraps the upstream `OTLPMetricExporter` to serialize metrics to JSON and send them
* **Uses delta temporality for all instruments**\
Every export contains **increments** since the last export (delta), matching common observability platforms.
* **Respects configuration**\
Uses `app_name`, `environment`, and optional `resource_attributes` from `Netra.init` as resource attributes on all metrics.
* **Exports periodically**\
Uses `metrics_export_interval_ms` to control the export interval.
## **Complete Example**
```python theme={null}
from netra import Netra
import time
import random
# Initialize Netra with metrics enabled
Netra.init(
app_name="custom-metrics-demo",
environment="staging",
enable_metrics=True,
metrics_export_interval_ms=2000
)
# Get a Meter for your service
meter = Netra.get_meter("checkout_service")
# Define instruments
request_counter = meter.create_counter(
name="checkout.requests",
description="Number of checkout requests",
unit="1"
)
error_counter = meter.create_counter(
name="checkout.errors",
description="Number of failed checkouts",
unit="1"
)
latency_histogram = meter.create_histogram(
name="checkout.latency_ms",
description="Checkout latency in milliseconds",
unit="ms",
)
# Application loop (simplified)
for _ in range(100):
start = time.time()
# Simulate request handling
success = random.random() > 0.1
latency_ms = random.uniform(50, 300)
# Record metrics
request_counter.add(
amount=1,
attributes={"status": "success" if success else "error"}
)
if not success:
error_counter.add(
amount=1,
attributes={"reason": "payment_failed"}
)
latency_histogram.record(
amount=latency_ms,
attributes={"status": "success" if success else "error"},
)
time.sleep(0.1)
Netra.shutdown()
```
## **Next Steps**
* [Dashboard Query](/sdk-reference/dashboard-query/python) - Query dashboard metrics
* [Usage Utilities](/usage/usage-utilities) - Query traces and spans
* [Simulation Overview](/Simulation/Simulation-overview) - Learn about simulation testing
* [Evaluation](/sdk-reference/evaluation/python) - Evaluate AI outputs
# Dashboard Query
Source: https://docs.getnetra.ai/sdk-reference/dashboard-query/python
Python API reference for Netra Dashboard queries. Fetch metrics, session summaries, session details, and usage data programmatically with the dashboard query API.
The Netra SDK exposes a `dashboard` client that lets you perform dashboard queries and retrieve relevant data programmatically.
This page shows how to use `Netra.dashboard` to craft dashboard queries, fetch session summaries, retrieve session statistics, and load full session details.
## Getting Started
The `dashboard` client is available on the main `Netra` entry point after initialization.
```python theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# Access the dashboard client
Netra.dashboard.query_data(...)
Netra.dashboard.get_session_summary(...)
Netra.dashboard.get_session_stats(...)
Netra.dashboard.get_session_details(...)
```
***
## query\_data
Fetch dashboard data with customizable metrics, dimensions, and filters. This method supports various chart types and aggregation strategies.
```python Usage theme={null}
from netra import Netra
from netra.dashboard import *
Netra.init(app_name="sample-app")
result = Netra.dashboard.query_data(
scope=Scope.SPANS,
chart_type=ChartType.LINE_TIME_SERIES,
metrics=Metrics(
measure=Measure.TOTAL_COST,
aggregation=Aggregation.TOTAL_COUNT,
),
dimension=Dimension(
field=DimensionField.SERVICE,
),
filter=FilterConfig(
start_time="2026-01-10T00:00:00.000Z",
end_time="2026-01-14T23:59:59.000Z",
group_by=GroupBy.DAY,
filters=[
Filter(
field=FilterField.ENVIRONMENT,
operator=Operator.EQUALS,
type=Type.STRING,
value="production",
)
],
),
)
print(result)
```
```python Signature theme={null}
query_data(
scope: Scope,
chart_type: ChartType,
metrics: Metrics,
filter: FilterConfig,
dimension: Dimension | None = None,
) -> dict | Any
```
### Parameters
| Parameter | Type | Description |
| ------------ | -------------- | ------------------------------------------------------ |
| `scope` | `Scope` | Scope of data to query (`SPANS` or `TRACES`) |
| `chart_type` | `ChartType` | Chart visualization type (controls the response shape) |
| `metrics` | `Metrics` | Defines what metric to compute and how to aggregate it |
| `filter` | `FilterConfig` | Filter configuration constraining the query |
| `dimension` | `Dimension?` | Optional grouping to split results by a dimension |
### Enums and Types
| Value | Description |
| -------------- | --------------------------------- |
| `Scope.SPANS` | Query individual span-level data |
| `Scope.TRACES` | Query trace-level aggregated data |
| Value | Description |
| ---------------------------- | -------------------- |
| `ChartType.LINE_TIME_SERIES` | Line chart over time |
| `ChartType.BAR_TIME_SERIES` | Bar chart over time |
| `ChartType.HORIZONTAL_BAR` | Horizontal bar chart |
| `ChartType.VERTICAL_BAR` | Vertical bar chart |
| `ChartType.PIE` | Pie chart |
| `ChartType.NUMBER` | Single numeric value |
The `Metrics` object defines what to measure and how to aggregate.
| Field | Type | Description |
| ------------- | --------------- | --------------------------------------------------------- |
| `metric_name` | `Optional[str]` | Name of metric *(applicable only for custom metric)* |
| `measure` | `Measure` | Measure of the metric |
| `aggregation` | `Aggregation` | Aggregation to use |
**Measure:**
| Value | Description |
| ------------------------- | ----------------------- |
| `Measure.LATENCY` | Request latency |
| `Measure.ERROR_RATE` | Error rate percentage |
| `Measure.PII_COUNT` | Count of PII detections |
| `Measure.REQUEST_COUNT` | Number of requests |
| `Measure.TOTAL_COST` | Total cost in USD |
| `Measure.VIOLATIONS` | Policy violations count |
| `Measure.TOTAL_TOKENS` | Total token usage |
| `Measure.AUDIO_DURATION` | Audio Duration |
| `Measure.CHARACTER_COUNT` | Character Count |
| `Measure.TTFT` | Time to First Token |
| `Measure.CUSTOM` | Custom Metric |
**Aggregation:**
| Value | Description |
| ------------------------- | ---------------------- |
| `Aggregation.AVERAGE` | Mean value |
| `Aggregation.P50` | 50th percentile |
| `Aggregation.P90` | 90th percentile |
| `Aggregation.P95` | 95th percentile |
| `Aggregation.P99` | 99th percentile |
| `Aggregation.MEDIAN` | Median value |
| `Aggregation.PERCENTAGE` | Percentage calculation |
| `Aggregation.TOTAL_COUNT` | Sum total |
| `Aggregation.SUM` | Sum of values |
| Field | Type | Description |
| ------------ | --------------- | ----------------------------------------------------------------------- |
| `start_time` | `str` | Start of time window (ISO 8601 UTC, e.g., `"2026-01-10T00:00:00.000Z"`) |
| `end_time` | `str` | End of time window (ISO 8601 UTC) |
| `group_by` | `GroupBy` | Time bucket size: `DAY`, `HOUR`, or `MINUTE` |
| `filters` | `list[Filter]?` | Optional list of filter conditions |
| Field | Type | Description |
| ---------- | ------------- | --------------------------------------- |
| `field` | `FilterField` | Field to filter on |
| `operator` | `Operator` | Comparison operator |
| `type` | `Type` | Value type |
| `value` | `Any` | Value to compare against |
| `key` | `str?` | Required only for `Type.OBJECT` filters |
**FilterField values:** `TOTAL_COST`, `SERVICE`, `TENANT_ID`, `USER_ID`, `SESSION_ID`, `ENVIRONMENT`, `LATENCY`, `MODEL_NAME` (Spans only), `MODELS` (Traces only), `METADATA`
**Operator values:** `EQUALS`, `NOT_EQUALS`, `CONTAINS`, `NOT_CONTAINS`, `STARTS_WITH`, `ENDS_WITH`, `GREATER_THAN`, `LESS_THAN`, `GREATER_EQUAL_TO`, `LESS_EQUAL_TO`, `ANY_OF`, `NONE_OF`
**Type values:** `STRING`, `NUMBER`, `BOOLEAN`, `ARRAY_OPTIONS`, `OBJECT`
| Field | Type | Description |
| ------- | ---------------- | ---------------------------------------------------------------- |
| `field` | `DimensionField` | Field to group by |
| `name` | `Optional[str]` | Name of dimension *(applicable only for custom dimension)* |
**DimensionField values:**
| Value | Supported Scopes |
| ---------------------------- | ---------------- |
| `DimensionField.ENVIRONMENT` | Spans, Traces |
| `DimensionField.SERVICE` | Spans only |
| `DimensionField.MODEL_NAME` | Spans only |
| `DimensionFIeld.CUSTOM` | Custom dimension |
If the query scope is `Scope.TRACES`, only `DimensionField.ENVIRONMENT` is supported. The `Scope.SPANS` supports all dimension fields.
***
## get\_session\_summary
Retrieve aggregated session metrics including total sessions, costs, latency, and cost breakdown by model.
```python Usage theme={null}
from netra import Netra
from netra.dashboard import *
Netra.init(app_name="sample-app")
result = Netra.dashboard.get_session_summary(
filter=SessionFilterConfig(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
filters=[
SessionFilter(
field=SessionFilterField.TENANT_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["Unilever", "AceTech"]
),
SessionFilter(
field=SessionFilterField.SERVICE,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["openai-chat"]
),
SessionFilter(
field=SessionFilterField.ENVIRONMENT,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["production"]
),
SessionFilter(
field=SessionFilterField.USER_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["user-123", "user-456"]
)
]
)
)
print(result)
```
```python Signature theme={null}
get_session_summary(
filter: SessionFilterConfig
) -> dict | Any
```
### Parameters
| Parameter | Type | Description |
| --------- | --------------------- | ---------------------------------- |
| `filter` | `SessionFilterConfig` | Filter configuration for the query |
### SessionFilterConfig
| Field | Type | Description |
| ------------ | ---------------------- | ----------------------------------- |
| `start_time` | `str` | Start of time window (ISO 8601 UTC) |
| `end_time` | `str` | End of time window (ISO 8601 UTC) |
| `filters` | `list[SessionFilter]?` | Optional list of filter conditions |
### SessionFilter (Optional)
| Field | Type | Description |
| ---------- | ----------------------- | --------------------------------------------------------- |
| `field` | `SessionFilterField` | Supports `TENANT_ID`, `ENVIRONMENT`, `SERVICE`, `USER_ID` |
| `operator` | `SessionFilterOperator` | Currently supports `ANY_OF` |
| `type` | `SessionFilterType` | Currently supports `ARRAY` |
| `value` | `list[str]` | List of values to match |
### Response
```json theme={null}
{
"timeRange": {
"startTime": "2026-01-01T00:00:00.000Z",
"endTime": "2026-01-31T23:59:59.000Z"
},
"data": {
"totalSessions": 207,
"totalCost": 0.041807,
"avgCostPerSession": 0.000202,
"avgLatencyMs": 8689.83,
"costByModel": [
{
"model": "gpt-4o-mini",
"cost": 0.041807
}
]
}
}
```
***
## get\_session\_details
Retrieve full details for a single session, including every trace with tokens, costs, models, inputs/outputs, and tool calls.
`get_session_details` is available in the Python SDK only.
```python Usage theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
detail = Netra.dashboard.get_session_details(
session_id="0acd79e0-526e-40cc-91e9-4010711de0ed"
)
print(detail["sessionId"])
for trace in detail["traces"]:
print(trace["traceId"], trace["cost"]["totalCost"], trace["toolCalls"])
```
```python Signature theme={null}
get_session_details(
session_id: str
) -> dict | Any
```
### Parameters
| Parameter | Type | Description |
| ------------ | ----- | ------------------ |
| `session_id` | `str` | Session identifier |
### Response
```json theme={null}
{
"sessionId": "0acd79e0-526e-40cc-91e9-4010711de0ed",
"traces": [
{
"traceId": "de51208be1ae032057381f162cbcea9b",
"traceName": "openai.chat",
"startTime": "2026-01-15T10:30:00Z",
"endTime": "2026-01-15T10:30:04Z",
"latencyMs": 4050,
"input": "What is the refund policy?",
"output": "You can request a refund within 30 days...",
"tokens": {
"promptTokens": 120,
"completionTokens": 85,
"cachedTokens": 0,
"cacheCreationTokens": 0,
"totalTokens": 205
},
"cost": {
"promptTokensCost": 0.000018,
"completionTokensCost": 0.000051,
"cachedTokensCost": 0,
"cacheCreationTokensCost": 0,
"totalCost": 0.000069
},
"models": ["gpt-4o-mini"],
"toolCalls": [
{
"toolName": "lookup_policy",
"toolCallCount": 1
}
]
}
]
}
```
***
## get\_session\_stats
Fetch a paginated list of sessions with individual session metrics including request count, cost, and duration.
```python Usage theme={null}
from netra import Netra
from netra.dashboard import *
Netra.init(app_name="sample-app")
session_stats = Netra.dashboard.get_session_stats(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
limit=10,
cursor=None,
filters=[
SessionFilter(
field=SessionFilterField.TENANT_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["Unilever", "AceTech"]
),
SessionFilter(
field=SessionFilterField.SERVICE,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["openai-chat"]
),
SessionFilter(
field=SessionFilterField.ENVIRONMENT,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["production"]
),
SessionFilter(
field=SessionFilterField.USER_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["user-123"]
)
],
sort_field=SortField.START_TIME,
sort_order=SortOrder.DESC
)
print(session_stats.data)
# Pagination
if session_stats.has_next_page:
next_page = Netra.dashboard.get_session_stats(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
limit=10,
cursor=session_stats.next_cursor,
filters=[
SessionFilter(
field=SessionFilterField.TENANT_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["Unilever", "AceTech"]
),
SessionFilter(
field=SessionFilterField.SERVICE,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["openai-chat"]
),
SessionFilter(
field=SessionFilterField.ENVIRONMENT,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["production"]
),
SessionFilter(
field=SessionFilterField.USER_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["user-123"]
)
],
sort_field=SortField.START_TIME,
sort_order=SortOrder.DESC
)
print(next_page.data)
```
```python Signature theme={null}
get_session_stats(
start_time: str,
end_time: str,
filters: Optional[List[SessionFilter]] = None,
limit: Optional[int] = None,
cursor: Optional[str] = None,
sort_field: Optional[SortField] = None,
sort_order: Optional[SortOrder] = None,
) -> SessionStatsResult
```
### Parameters
| Parameter | Type | Description |
| ------------ | ---------------------- | ------------------------------------ |
| `start_time` | `str` | Start of time window (ISO 8601 UTC) |
| `end_time` | `str` | End of time window (ISO 8601 UTC) |
| `filters` | `list[SessionFilter]?` | Optional filter conditions |
| `limit` | `int?` | Maximum results per page |
| `cursor` | `str?` | Pagination cursor from previous page |
| `sort_field` | `SortField?` | Field to sort by |
| `sort_order` | `SortOrder?` | Sort direction |
### Sorting Options
| Value | Description |
| -------------------------- | -------------------------- |
| `SortField.SESSION_ID` | Sort by session ID |
| `SortField.START_TIME` | Sort by session start time |
| `SortField.TOTAL_REQUESTS` | Sort by request count |
| `SortField.TOTAL_COST` | Sort by total cost |
| Value | Description |
| ---------------- | ---------------- |
| `SortOrder.ASC` | Ascending order |
| `SortOrder.DESC` | Descending order |
### Response
```json theme={null}
SessionStatsResult(
data=[
{
"sessionId": "0acd79e0-526e-40cc-91e9-4010711de0ed",
"sessionStartTime": "2026-02-23 11:59:58.785",
"totalRequests": "1",
"totalCost": 0.0012775,
"costByModel": {
"gpt-4o": 0.0012775
},
"sessionDuration": "4.05s",
"cursor": "MjAyNi0wMi0yMyAxMTo1OTo1OC43ODV8MGFjZDc5ZTAtNTI2ZS00MGNjLTkxZTktNDAxMDcxMWRlMGVk"
},
{
"sessionId": "03d000d0-0b16-4688-a83b-291ed489c281",
"sessionStartTime": "2026-02-23 11:59:14.233",
"totalRequests": "2",
"totalCost": 0.00161175,
"costByModel": {
"gpt-4o-mini": 0.00010425,
"gpt-4o": 0.0015075
},
"sessionDuration": "1m 16.27s",
"cursor": "MjAyNi0wMi0yMyAxMTo1OToxNC4yMzN8MDNkMDAwZDAtMGIxNi00Njg4LWE4M2ItMjkxZWQ0ODljMjgx"
}],
has_next_page=True,
next_cursor="MjAyNi0wMS0yMiAwOTo1MzoyNy4xNjN8ODMzNmQwOTYtZjNiNS00ZTk5LWE1ZjEtYTk2OTkxMjlmNDFh",
)
```
## iter\_session\_stats
Stream over all pages of session stats until completion. This iterator handles pagination automatically.
```python Usage theme={null}
from netra import Netra
from netra.dashboard import *
Netra.init(app_name="sample-app")
for session in Netra.dashboard.iter_session_stats(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
filters=[
SessionFilter(
field=SessionFilterField.TENANT_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["Unilever", "AceTech"]
),
SessionFilter(
field=SessionFilterField.SERVICE,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["openai-chat"]
),
SessionFilter(
field=SessionFilterField.ENVIRONMENT,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["production"]
)
],
sort_field=SortField.START_TIME,
sort_order=SortOrder.DESC
):
print(session)
```
```python Signature theme={null}
iter_session_stats(
start_time: str,
end_time: str,
filters: Optional[List[SessionFilter]] = None,
sort_field: Optional[SortField] = None,
sort_order: Optional[SortOrder] = None,
) -> Iterator[SessionStatsResult]
```
### Parameters
| Parameter | Type | Description |
| ------------ | ---------------------- | ----------------------------------- |
| `start_time` | `str` | Start of time window (ISO 8601 UTC) |
| `end_time` | `str` | End of time window (ISO 8601 UTC) |
| `filters` | `list[SessionFilter]?` | Optional filter conditions |
| `sort_field` | `SortField?` | Field to sort by |
| `sort_order` | `SortOrder?` | Sort direction |
### Response
```json theme={null}
{
"sessionId": "0acd79e0-526e-40cc-91e9-4010711de0ed",
"sessionStartTime": "2026-02-23 11:59:58.785",
"totalRequests": 1,
"totalCost": 0.0012775,
"costByModel": {
"gpt-4o": 0.0012775
},
"sessionDuration": "4.05s",
"cursor": "MjAyNi0wMi0yMyAxMTo1OTo1OC43ODV8MGFjZDc5ZTAtNTI2ZS00MGNjLTkxZTktNDAxMDcxMWRlMGVk"
}
{
"sessionId": "03d000d0-0b16-4688-a83b-291ed489c281",
"sessionStartTime": "2026-02-23 11:59:14.233",
"totalRequests": 37,
"totalCost": 0.02144675,
"costByModel": {
"gpt-4o-mini": 0.00010425,
"gpt-4o": 0.0213425
},
"sessionDuration": "22hr 19m",
"cursor": "MjAyNi0wMi0yMyAxMTo1OToxNC4yMzN8MDNkMDAwZDAtMGIxNi00Njg4LWE4M2ItMjkxZWQ0ODljMjgx"
}
{
"sessionId": "fe19a082-1b4f-4be9-a75c-91a609c948ad",
"sessionStartTime": "2026-02-23 04:45:26.156",
"totalRequests": 1,
"totalCost": 0.0000897,
"costByModel": {
"gpt-4o-mini": 0.0000897
},
"sessionDuration": "3.76s",
"cursor": "MjAyNi0wMi0yMyAwNDo0NToyNi4xNTZ8ZmUxOWEwODItMWI0Zi00YmU5LWE3NWMtOTFhNjA5Yzk0OGFk"
}
```
Use `iter_session_stats` when you need to process all sessions without manually handling pagination. The iterator fetches pages on-demand as you iterate.
***
## Complete Example
```python theme={null}
from netra import Netra
from netra.dashboard import *
# Initialize the SDK
Netra.init(
app_name="analytics-app",
headers="x-api-key=your-api-key",
)
# Query cost trends over time
cost_trends = Netra.dashboard.query_data(
scope=Scope.SPANS,
chart_type=ChartType.LINE_TIME_SERIES,
metrics=Metrics(
measure=Measure.TOTAL_COST,
aggregation=Aggregation.TOTAL_COUNT,
),
dimension=Dimension(field=DimensionField.SERVICE),
filter=FilterConfig(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
group_by=GroupBy.DAY,
),
)
# Get session summary for specific tenants, services, environments, and users
summary = Netra.dashboard.get_session_summary(
filter=SessionFilterConfig(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
filters=[
SessionFilter(
field=SessionFilterField.TENANT_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["TenantA", "TenantB"]
),
SessionFilter(
field=SessionFilterField.SERVICE,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["openai-chat"]
),
SessionFilter(
field=SessionFilterField.ENVIRONMENT,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["production"]
),
SessionFilter(
field=SessionFilterField.USER_ID,
operator=SessionFilterOperator.ANY_OF,
type=SessionFilterType.ARRAY,
value=["user-123"]
)
]
)
)
# Iterate through all sessions
for session in Netra.dashboard.iter_session_stats(
start_time="2026-01-01T00:00:00.000Z",
end_time="2026-01-31T23:59:59.000Z",
sort_field=SortField.TOTAL_COST,
sort_order=SortOrder.DESC
):
print(f"Session {session.session_id}: ${session.total_cost:.4f}")
# Load full detail for a specific session
detail = Netra.dashboard.get_session_details(
session_id="0acd79e0-526e-40cc-91e9-4010711de0ed"
)
print(detail["sessionId"], len(detail["traces"]))
```
## Next Steps
* [Usage Utilities](/usage/usage-utilities) - Query token usage and trace data
* [Python SDK Reference](/sdk-reference/sdk/python) - Complete SDK documentation
* [Custom Dashboard](/Dashboard/Custom-dashboard) - Build custom dashboards
# Dashboard Query
Source: https://docs.getnetra.ai/sdk-reference/dashboard-query/typescript
TypeScript API reference for Netra Dashboard queries. Fetch metrics, session summaries, and usage data programmatically with the dashboard query API.
The Netra SDK exposes a `dashboard` client that lets you perform dashboard queries and retrieve relevant data programmatically.
This page shows how to use `netra.dashboard` to craft dashboard queries, fetch session summaries, and retrieve session statistics.
## Getting Started
The `dashboard` client is available on the main `Netra` entry point after initialization.
```typescript theme={null}
import { Netra } from "netra-sdk-js";
const client = new Netra({
apiKey: "your-api-key",
});
// Access the dashboard client
await client.dashboard.queryData(...);
await client.dashboard.getSessionSummary(...);
await client.dashboard.getSessionStats(...);
```
***
## queryData
Fetch dashboard data with customizable metrics, dimensions, and filters. This method supports various chart types and aggregation strategies.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
import {
Scope,
ChartType,
Measure,
Aggregation,
DimensionField,
GroupBy,
FilterField,
Operator,
FilterType,
} from "netra-sdk-js/api/dashboard";
const client = new Netra({ apiKey: "..." });
const result = await client.dashboard.queryData({
scope: Scope.SPANS,
chartType: ChartType.LINE_TIME_SERIES,
metrics: {
measure: Measure.TOTAL_COST,
aggregation: Aggregation.TOTAL_COUNT,
},
dimension: {
field: DimensionField.SERVICE,
},
filter: {
startTime: "2026-01-10T00:00:00.000Z",
endTime: "2026-01-14T23:59:59.000Z",
groupBy: GroupBy.DAY,
filters: [
{
field: FilterField.ENVIRONMENT,
operator: Operator.EQUALS,
type: FilterType.STRING,
value: "production",
},
],
},
});
console.log(result);
```
```typescript Signature theme={null}
queryData(params: QueryDataParams): Promise
// QueryDataParams interface
interface QueryDataParams {
scope: Scope;
chartType: ChartType;
metrics: Metrics;
filter: FilterConfig;
dimension?: Dimension;
}
```
### Parameters
| Parameter | Type | Description |
| ----------- | -------------- | ------------------------------------------------------ |
| `scope` | `Scope` | Scope of data to query (`SPANS` or `TRACES`) |
| `chartType` | `ChartType` | Chart visualization type (controls the response shape) |
| `metrics` | `Metrics` | Defines what metric to compute and how to aggregate it |
| `filter` | `FilterConfig` | Filter configuration constraining the query |
| `dimension` | `Dimension?` | Optional grouping to split results by a dimension |
### Enums and Types
| Value | Description |
| -------------- | --------------------------------- |
| `Scope.SPANS` | Query individual span-level data |
| `Scope.TRACES` | Query trace-level aggregated data |
| Value | Description |
| ---------------------------- | -------------------- |
| `ChartType.LINE_TIME_SERIES` | Line chart over time |
| `ChartType.BAR_TIME_SERIES` | Bar chart over time |
| `ChartType.HORIZONTAL_BAR` | Horizontal bar chart |
| `ChartType.VERTICAL_BAR` | Vertical bar chart |
| `ChartType.PIE` | Pie chart |
| `ChartType.NUMBER` | Single numeric value |
The `Metrics` object defines what to measure and how to aggregate.
**Measure:**
| Value | Description |
| ------------------------- | ----------------------- |
| `Measure.LATENCY` | Request latency |
| `Measure.ERROR_RATE` | Error rate percentage |
| `Measure.PII_COUNT` | Count of PII detections |
| `Measure.REQUEST_COUNT` | Number of requests |
| `Measure.TOTAL_COST` | Total cost in USD |
| `Measure.VIOLATIONS` | Policy violations count |
| `Measure.TOTAL_TOKENS` | Total token usage |
| `Measure.AUDIO_DURATION` | Audio Duration |
| `Measure.CHARACTER_COUNT` | Character Count |
| `Measure.TTFT` | Time to First Token |
**Aggregation:**
| Value | Description |
| ------------------------- | ---------------------- |
| `Aggregation.AVERAGE` | Mean value |
| `Aggregation.P50` | 50th percentile |
| `Aggregation.P90` | 90th percentile |
| `Aggregation.P95` | 95th percentile |
| `Aggregation.P99` | 99th percentile |
| `Aggregation.MEDIAN` | Median value |
| `Aggregation.PERCENTAGE` | Percentage calculation |
| `Aggregation.TOTAL_COUNT` | Sum total |
| Field | Type | Description |
| ----------- | ----------- | ----------------------------------------------------------------------- |
| `startTime` | `string` | Start of time window (ISO 8601 UTC, e.g., `"2026-01-10T00:00:00.000Z"`) |
| `endTime` | `string` | End of time window (ISO 8601 UTC) |
| `groupBy` | `GroupBy` | Time bucket size: `DAY`, `HOUR`, or `MINUTE` |
| `filters` | `Filter[]?` | Optional list of filter conditions |
| Field | Type | Description | |
| ---------- | ------------- | --------------------------------------------- | ------------------ |
| `field` | \`FilterField | string\` | Field to filter on |
| `operator` | `Operator` | Comparison operator | |
| `type` | `FilterType` | Value type | |
| `value` | `any` | Value to compare against | |
| `key` | `string?` | Required only for `FilterType.OBJECT` filters | |
**FilterField values:** `TOTAL_COST`, `SERVICE`, `TENANT_ID`, `USER_ID`, `SESSION_ID`, `ENVIRONMENT`, `LATENCY`, `MODEL_NAME` (Spans only), `MODELS` (Traces only), `METADATA`
**Operator values:** `EQUALS`, `NOT_EQUALS`, `CONTAINS`, `NOT_CONTAINS`, `STARTS_WITH`, `ENDS_WITH`, `GREATER_THAN`, `LESS_THAN`, `GREATER_EQUAL_TO`, `LESS_EQUAL_TO`, `ANY_OF`, `NONE_OF`
**FilterType values:** `STRING`, `NUMBER`, `BOOLEAN`, `ARRAY_OPTIONS`, `OBJECT`
| Field | Type | Description |
| ------- | ---------------- | ----------------- |
| `field` | `DimensionField` | Field to group by |
**DimensionField values:**
| Value | Supported Scopes |
| ---------------------------- | ---------------- |
| `DimensionField.ENVIRONMENT` | Spans, Traces |
| `DimensionField.SERVICE` | Spans only |
| `DimensionField.MODEL_NAME` | Spans only |
If the query scope is `Scope.TRACES`, only `DimensionField.ENVIRONMENT` is supported. The `Scope.SPANS` supports all dimension fields.
***
## getSessionSummary
Retrieve aggregated session metrics including total sessions, costs, latency, and cost breakdown by model.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
import {
SessionFilterField,
SessionOperator,
SessionFilterType,
} from "netra-sdk-js/api/dashboard";
const client = new Netra({ apiKey: "..." });
const result = await client.dashboard.getSessionSummary({
startTime: "2026-01-01T00:00:00.000Z",
endTime: "2026-01-31T23:59:59.000Z",
filters: [
{
field: SessionFilterField.TENANT_ID,
operator: SessionOperator.ANY_OF,
type: SessionFilterType.ARRAY,
value: ["Unilever", "AceTech"],
},
],
});
console.log(result);
```
```typescript Signature theme={null}
getSessionSummary(filter: SessionFilterConfig): Promise
// SessionFilterConfig interface
interface SessionFilterConfig {
startTime: string;
endTime: string;
filters?: SessionFilter[];
}
```
### Parameters
| Parameter | Type | Description |
| --------- | --------------------- | ---------------------------------- |
| `filter` | `SessionFilterConfig` | Filter configuration for the query |
### SessionFilterConfig
| Field | Type | Description |
| ----------- | ------------------ | ----------------------------------- |
| `startTime` | `string` | Start of time window (ISO 8601 UTC) |
| `endTime` | `string` | End of time window (ISO 8601 UTC) |
| `filters` | `SessionFilter[]?` | Optional list of filter conditions |
### SessionFilter (Optional)
| Field | Type | Description |
| ---------- | -------------------- | --------------------------- |
| `field` | `SessionFilterField` | Supports `TENANT_ID` |
| `operator` | `SessionOperator` | Currently supports `ANY_OF` |
| `type` | `SessionFilterType` | Currently supports `ARRAY` |
| `value` | `string[]` | List of values to match |
\| `value` | `string[]` | List of values to match |
### Response
```json theme={null}
{
"timeRange": {
"startTime": "2026-01-01T00:00:00.000Z",
"endTime": "2026-01-31T23:59:59.000Z"
},
"data": {
"totalSessions": 207,
"totalCost": 0.041807,
"avgCostPerSession": 0.000202,
"avgLatencyMs": 8689.83,
"costByModel": [
{
"model": "gpt-4o-mini",
"cost": 0.041807
}
]
}
}
```
***
## getSessionStats
Fetch a paginated list of sessions with individual session metrics including request count, cost, and duration.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
import {
SessionFilterField,
SessionOperator,
SessionFilterType,
SortField,
SortOrder,
} from "netra-sdk-js/api/dashboard";
const client = new Netra({ apiKey: "..." });
const sessionStats = await client.dashboard.getSessionStats(
"2026-01-01T00:00:00.000Z", // startTime
"2026-01-31T23:59:59.000Z", // endTime
10, // limit
[ // filters
{
field: SessionFilterField.TENANT_ID,
operator: SessionOperator.ANY_OF,
type: SessionFilterType.ARRAY,
value: ["Unilever", "AceTech"],
},
],
undefined, // cursor
SortField.START_TIME, // sortField
SortOrder.DESC // sortOrder
);
if (sessionStats) {
console.log(sessionStats.data);
// Pagination
if (sessionStats.hasNextPage && sessionStats.nextCursor) {
const nextPage = await client.dashboard.getSessionStats(
"2026-01-01T00:00:00.000Z",
"2026-01-31T23:59:59.000Z",
10,
[
{
field: SessionFilterField.TENANT_ID,
operator: SessionOperator.ANY_OF,
type: SessionFilterType.ARRAY,
value: ["Unilever", "AceTech"],
},
],
sessionStats.nextCursor,
SortField.START_TIME,
SortOrder.DESC
);
console.log(nextPage?.data);
}
}
```
```typescript Signature theme={null}
getSessionStats(
startTime: string,
endTime: string,
limit?: number,
filters?: SessionFilter[],
cursor?: string,
sortField?: SortField,
sortOrder?: SortOrder
): Promise
```
### Parameters
| Parameter | Type | Description |
| ----------- | ------------------ | -------------------------------------- |
| `startTime` | `string` | Start of time window (ISO 8601 UTC) |
| `endTime` | `string` | End of time window (ISO 8601 UTC) |
| `limit` | `number?` | Maximum results per page (default: 10) |
| `filters` | `SessionFilter[]?` | Optional filter conditions |
| `cursor` | `string?` | Pagination cursor from previous page |
| `sortField` | `SortField?` | Field to sort by |
| `sortOrder` | `SortOrder?` | Sort direction |
### Sorting Options
| Value | Description |
| -------------------------- | -------------------------- |
| `SortField.SESSION_ID` | Sort by session ID |
| `SortField.START_TIME` | Sort by session start time |
| `SortField.TOTAL_REQUESTS` | Sort by request count |
| `SortField.TOTAL_COST` | Sort by total cost |
| Value | Description |
| ---------------- | ---------------- |
| `SortOrder.ASC` | Ascending order |
| `SortOrder.DESC` | Descending order |
### Response
```json theme={null}
{
"data": [
{
"sessionId": "00973697-f64b-4144-a42d-6845f5d149e4",
"sessionStartTime": "2026-01-22 09:54:51.210",
"totalRequests": 1,
"totalCost": 0.0000885,
"sessionDuration": "3.15s",
"cursor": "MjAyNi0wMS0yMiAwOTo1NDo1MS4yMTB8MDA5NzM2OTctZjY0Yi00MTQ0LWE0MmQtNjg0NWY1ZDE0OWU0"
},
{
"sessionId": "48827b92-24fa-4278-b559-a982d1ddee96",
"sessionStartTime": "2026-01-22 09:54:47.563",
"totalRequests": 1,
"totalCost": 0.00012465,
"sessionDuration": "3.65s",
"cursor": "MjAyNi0wMS0yMiAwOTo1NDo0Ny41NjN8NDg4MjdiOTItMjRmYS00Mjc4LWI1NTktYTk4MmQxZGRlZTk2"
}
],
"hasNextPage": true,
"nextCursor": "MjAyNi0wMS0yMiAwOTo1MzoyNy4xNjN8ODMzNmQwOTYtZjNiNS00ZTk5LWE1ZjEtYTk2OTkxMjlmNDFh"
}
```
***
## iterSessionStats
Stream over all pages of session stats until completion. This iterator handles pagination automatically.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
import {
SessionFilterField,
SessionOperator,
SessionFilterType,
SortField,
SortOrder,
} from "netra-sdk-js/api/dashboard";
const client = new Netra({ apiKey: "..." });
const iterator = client.dashboard.iterSessionStats(
"2026-01-01T00:00:00.000Z",
"2026-01-31T23:59:59.000Z",
[
{
field: SessionFilterField.TENANT_ID,
operator: SessionOperator.ANY_OF,
type: SessionFilterType.ARRAY,
value: ["Unilever", "AceTech"],
},
],
SortField.START_TIME,
SortOrder.DESC
);
for await (const session of iterator) {
console.log(session);
}
```
```typescript Signature theme={null}
iterSessionStats(
startTime: string,
endTime: string,
filters?: SessionFilter[],
sortField?: SortField,
sortOrder?: SortOrder
): AsyncGenerator
```
### Parameters
| Parameter | Type | Description |
| ----------- | ------------------ | ----------------------------------- |
| `startTime` | `string` | Start of time window (ISO 8601 UTC) |
| `endTime` | `string` | End of time window (ISO 8601 UTC) |
| `filters` | `SessionFilter[]?` | Optional filter conditions |
| `sortField` | `SortField?` | Field to sort by |
| `sortOrder` | `SortOrder?` | Sort direction |
### Response
```json theme={null}
{
"sessionId": "sess_abc123def456",
"startTime": "2026-01-15T10:23:45.000Z",
"totalRequests": 12,
"totalCost": 0.0347,
"sessionDuration": "4m 32s",
"cursor": "eyJzZXNzaW9uX2lkIjoic2Vzc19hYmMxMjNkZWY0NTYifQ=="
}
{
"sessionId": "sess_xyz789ghi012",
"startTime": "2026-01-14T08:15:30.000Z",
"totalRequests": 5,
"totalCost": 0.0123,
"sessionDuration": "1m 15s",
"cursor": "eyJzZXNzaW9uX2lkIjoic2Vzc194eXo3ODlnaGkwMTIifQ=="
}
{
"sessionId": "sess_mno345pqr678",
"startTime": "2026-01-12T14:50:10.000Z",
"totalRequests": 28,
"totalCost": 0.0891,
"sessionDuration": "12m 08s",
"cursor": "eyJzZXNzaW9uX2lkIjoic2Vzc19tbm8zNDVwcXI2NzgifQ=="
}
```
Use `iterSessionStats` when you need to process all sessions without manually handling pagination. The iterator fetches pages on-demand as you iterate.
***
## Complete Example
```typescript theme={null}
import { Netra } from "netra-sdk-js";
import {
Scope,
ChartType,
Measure,
Aggregation,
DimensionField,
GroupBy,
SessionFilterField,
SessionOperator,
SessionFilterType,
SortField,
SortOrder,
} from "netra-sdk-js/api/dashboard";
async function main() {
// Initialize the SDK
const client = new Netra({
apiKey: "your-api-key",
});
// Query cost trends over time
const costTrends = await client.dashboard.queryData({
scope: Scope.SPANS,
chartType: ChartType.LINE_TIME_SERIES,
metrics: {
measure: Measure.TOTAL_COST,
aggregation: Aggregation.TOTAL_COUNT,
},
dimension: {
field: DimensionField.SERVICE,
},
filter: {
startTime: "2026-01-01T00:00:00.000Z",
endTime: "2026-01-31T23:59:59.000Z",
groupBy: GroupBy.DAY,
},
});
// Get session summary for specific tenants
const summary = await client.dashboard.getSessionSummary({
startTime: "2026-01-01T00:00:00.000Z",
endTime: "2026-01-31T23:59:59.000Z",
filters: [
{
field: SessionFilterField.TENANT_ID,
operator: SessionOperator.ANY_OF,
type: SessionFilterType.ARRAY,
value: ["TenantA", "TenantB"],
},
],
});
// Iterate through all sessions
const iterator = client.dashboard.iterSessionStats(
"2026-01-01T00:00:00.000Z",
"2026-01-31T23:59:59.000Z",
undefined,
SortField.TOTAL_COST,
SortOrder.DESC
);
for await (const session of iterator) {
console.log(
`Session ${session.session_id}: $${session.totalCost.toFixed(4)}`
);
}
}
main();
```
## Next Steps
* [Usage Utilities](/usage/usage-utilities) - Query token usage and trace data
* [TypeScript SDK Reference](/sdk-reference/sdk/typescript) - Complete SDK documentation
* [Custom Dashboard](/Dashboard/Custom-dashboard) - Build custom dashboards
# Evaluation
Source: https://docs.getnetra.ai/sdk-reference/evaluation/python
Python API reference for Netra evaluation. Run test suites, create and manage datasets, add items, and measure AI output quality programmatically.
The Netra SDK exposes an `evaluation` client that lets you:
* **Manage datasets** - Create datasets and add test items
* **Run test suites** - Execute tasks against datasets with automatic tracing
* **Apply evaluators** - Score outputs using built-in or custom evaluators
* **Fetch results** - Retrieve detailed run results programmatically
This page shows how to use `Netra.evaluation` to manage datasets, run test suites, fetch results, and programmatically evaluate your AI applications.
## Getting Started
The `evaluation` client is available on the main `Netra` entry point after initialization.
```python theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# Access the evaluation client
Netra.evaluation.create_dataset(...)
Netra.evaluation.add_dataset_item(...)
Netra.evaluation.get_all_datasets(...)
Netra.evaluation.get_dataset(...)
Netra.evaluation.run_test_suite(...)
Netra.evaluation.get_run_results(...)
```
***
## create\_dataset
Create an empty dataset that can hold test items for evaluation runs.
```python Usage theme={null}
from netra import Netra
from netra.evaluation import TurnType
Netra.init(app_name="sample-app")
result = Netra.evaluation.create_dataset(
name="Customer Support QA",
tags=["support", "qa", "v1"],
turn_type=TurnType.SINGLE, # or TurnType.MULTI for multi-turn
)
print(f"Dataset created: {result.id}")
print(f"Name: {result.name}")
print(f"Tags: {result.tags}")
```
```python Signature theme={null}
create_dataset(
name: str,
tags: Optional[List[str]] = None,
turn_type: TurnType = TurnType.SINGLE,
) -> CreateDatasetResponse | None
```
### Parameters
| Parameter | Type | Description |
| ----------- | ------------ | ----------------------------------------------------------- |
| `name` | `str` | Name of the dataset (required) |
| `tags` | `list[str]?` | Optional tags for categorization |
| `turn_type` | `TurnType` | `SINGLE` for single-turn or `MULTI` for multi-turn datasets |
### Response: CreateDatasetResponse
| Field | Type | Description |
| ----------------- | ----------- | ------------------------------------ |
| `id` | `str` | Unique dataset identifier |
| `name` | `str` | Dataset name |
| `tags` | `list[str]` | Associated tags |
| `project_id` | `str` | Project identifier |
| `organization_id` | `str` | Organization identifier |
| `created_by` | `str` | Creator identifier |
| `updated_by` | `str` | Last updater identifier |
| `created_at` | `str` | Creation timestamp |
| `updated_at` | `str` | Last update timestamp |
| `deleted_at` | `str?` | Deletion timestamp (if soft-deleted) |
| Value | Description |
| ----------------- | ----------------------------------------------- |
| `TurnType.SINGLE` | Single-turn evaluation (one input → one output) |
| `TurnType.MULTI` | Multi-turn evaluation (conversation sequences) |
***
## add\_dataset\_item
Add a single test item to an existing dataset.
```python Usage theme={null}
from netra import Netra
from netra.evaluation import DatasetItem
Netra.init(app_name="sample-app")
result = Netra.evaluation.add_dataset_item(
dataset_id="dataset-123",
item=DatasetItem(
input="What is the return policy for electronics?",
expected_output="Electronics can be returned within 30 days with original packaging.",
tags=["policy", "returns"],
metadata={"category": "electronics", "priority": "high"},
),
)
print(f"Item added: {result.id}")
print(f"Input: {result.input}")
```
```python Signature theme={null}
add_dataset_item(
dataset_id: str,
item: DatasetItem,
) -> AddDatasetItemResponse | None
```
### Parameters
| Parameter | Type | Description |
| ------------ | ------------- | ------------------------ |
| `dataset_id` | `str` | ID of the target dataset |
| `item` | `DatasetItem` | The test item to add |
### DatasetItem
| Field | Type | Description |
| ----------------- | ------------ | ----------------------------------------- |
| `input` | `Any` | The input to pass to your task (required) |
| `expected_output` | `Any?` | Expected output for comparison |
| `tags` | `list[str]?` | Optional tags for the item |
| `metadata` | `dict?` | Optional metadata for evaluators |
### Response: AddDatasetItemResponse
| Field | Type | Description |
| ----------------- | ----------- | ------------------------------------ |
| `id` | `str` | Unique item identifier |
| `dataset_id` | `str` | Parent dataset ID |
| `project_id` | `str` | Project identifier |
| `organization_id` | `str` | Organization identifier |
| `source` | `str` | Source of the item |
| `source_id` | `str?` | Source reference ID |
| `input` | `Any` | The input value |
| `expected_output` | `Any` | The expected output |
| `is_active` | `bool` | Whether the item is active |
| `tags` | `list[str]` | Associated tags |
| `metadata` | `dict?` | Item metadata |
| `created_by` | `str` | Creator identifier |
| `updated_by` | `str` | Last updater identifier |
| `created_at` | `str` | Creation timestamp |
| `updated_at` | `str` | Last update timestamp |
| `deleted_at` | `str?` | Deletion timestamp (if soft-deleted) |
***
## get\_all\_datasets
Fetch all datasets, optionally filtered by a specific tag.
```python Usage theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# Fetch all datasets
all_datasets = Netra.evaluation.get_all_datasets()
# Fetch only datasets that include a specific tag
tagged_datasets = Netra.evaluation.get_all_datasets(tag="production")
if tagged_datasets:
print(f"Found {len(tagged_datasets.datasets)} dataset(s)")
for dataset in tagged_datasets.datasets:
print(f"{dataset.id} | {dataset.name} | {dataset.tags}")
```
```python Signature theme={null}
get_all_datasets(
tag: Optional[str] = None,
) -> GetAllDatasetsResponse | None
```
### Parameters
| Parameter | Type | Description |
| --------- | ------ | ----------------------------------------------------------------------- |
| `tag` | `str?` | Optional tag to filter datasets. If omitted, all datasets are returned. |
### Response: GetAllDatasetsResponse
| Field | Type | Description |
| ---------- | ----------------------------- | ----------------------------------------- |
| `datasets` | `list[CreateDatasetResponse]` | List of datasets returned by the backend. |
### CreateDatasetResponse
| Field | Type | Description |
| ----------------- | ----------- | ------------------------------------ |
| `id` | `str` | Unique dataset identifier |
| `name` | `str` | Dataset name |
| `tags` | `list[str]` | Associated tags |
| `turn_type` | `str?` | Turn type (`single` or `multi`) |
| `dataset_type` | `str?` | Dataset type value from backend |
| `project_id` | `str` | Project identifier |
| `organization_id` | `str` | Organization identifier |
| `created_by` | `str` | Creator identifier |
| `updated_by` | `str` | Last updater identifier |
| `created_at` | `str` | Creation timestamp |
| `updated_at` | `str` | Last update timestamp |
| `deleted_at` | `str?` | Deletion timestamp (if soft-deleted) |
***
## get\_dataset
Retrieve a dataset and all its items by ID.
```python Usage theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
dataset = Netra.evaluation.get_dataset(dataset_id="dataset-123")
print(f"Total items: {len(dataset.items)}")
for item in dataset.items:
print(f"ID: {item.id}")
print(f"Input: {item.input}")
print(f"Expected: {item.expected_output}")
print("---")
```
```python Signature theme={null}
get_dataset(
dataset_id: str,
) -> GetDatasetItemsResponse | None
```
### Parameters
| Parameter | Type | Description |
| ------------ | ----- | ----------------------------- |
| `dataset_id` | `str` | ID of the dataset to retrieve |
### Response: GetDatasetItemsResponse
| Field | Type | Description |
| ------- | --------------------- | --------------------- |
| `items` | `list[DatasetRecord]` | List of dataset items |
### DatasetRecord
| Field | Type | Description |
| ----------------- | ----- | ------------------- |
| `id` | `str` | Item identifier |
| `dataset_id` | `str` | Parent dataset ID |
| `input` | `Any` | The input value |
| `expected_output` | `Any` | The expected output |
***
## run\_test\_suite
Execute a test suite against a dataset, running your task function on each item and optionally applying evaluators.
```python Usage theme={null}
from netra import Netra
from openai import OpenAI
Netra.init(app_name="sample-app")
client = OpenAI()
def my_task(input_data):
"""Task function that processes each dataset item."""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": input_data},
],
)
return response.choices[0].message.content
# Get dataset
dataset = Netra.evaluation.get_dataset(dataset_id="dataset-123")
# Run test suite
result = Netra.evaluation.run_test_suite(
name="GPT-4o Mini Evaluation",
data=dataset,
task=my_task,
evaluators=["correctness", "relevance"], # Optional evaluator IDs
max_concurrency=10,
)
print(f"Run ID: {result['runId']}")
print(f"Items processed: {len(result['items'])}")
```
```python Signature theme={null}
run_test_suite(
name: str,
data: Dataset,
task: Callable[[Any], Any],
evaluators: Optional[List[Any]] = None,
max_concurrency: int = 50,
) -> Optional[Dict[str, Any]]
```
### Parameters
| Parameter | Type | Description |
| ----------------- | ---------- | -------------------------------------------- |
| `name` | `str` | Name for this test run (required) |
| `data` | `Dataset` | Dataset from `get_dataset()` |
| `task` | `Callable` | Function that takes input and returns output |
| `evaluators` | `list?` | Optional evaluator IDs or configs |
| `max_concurrency` | `int` | Max parallel task executions (default: 50) |
### Response
| Field | Type | Description |
| ------- | ------------ | ------------------------------- |
| `runId` | `str` | Unique run identifier |
| `items` | `list[dict]` | Results for each processed item |
### Item Result
| Field | Type | Description |
| --------------- | ----- | ------------------------------ |
| `index` | `int` | Item index in dataset |
| `status` | `str` | `"completed"` or `"failed"` |
| `traceId` | `str` | Trace ID for observability |
| `spanId` | `str` | Span ID for the task execution |
| `testRunItemId` | `str` | Backend item identifier |
The `task` function receives the `input` field from each dataset item. Return the output that should be compared against `expected_output` by evaluators.
***
## get\_run\_results
Fetch detailed results for a completed test run by its run ID. Use this after `run_test_suite` to retrieve evaluation scores, item-level outcomes, and other run metadata from the backend.
```python Usage theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# After running a test suite
result = Netra.evaluation.run_test_suite(
name="GPT-4o Mini Evaluation",
data=dataset,
task=my_task,
evaluators=["correctness", "relevance"],
)
run_id = result["runId"]
# Fetch the full run results
run_results = Netra.evaluation.get_run_results(run_id=run_id)
print(f"Run results: {run_results}")
```
```python Signature theme={null}
get_run_results(
run_id: str,
) -> Any | None
```
### Parameters
| Parameter | Type | Description |
| --------- | ----- | --------------------------------------------------------------------- |
| `run_id` | `str` | The unique identifier of the test run to fetch results for (required) |
### Response
Returns the JSON response from the backend containing the full run results, including evaluation scores and item-level details. Returns `None` if the `run_id` is empty or the request fails.
The top-level response wraps the run data:
| Field | Type | Description |
| --------- | ----------- | -------------------------------------------------------- |
| `success` | `bool` | Whether the request succeeded |
| `data` | `RunResult` | The run result object (see below) |
| `error` | `Any?` | Error details, `null` on success |
| `meta` | `dict` | Request metadata (timestamp, path, version, status code) |
### RunResult
| Field | Type | Description |
| ------------------ | ---------------- | --------------------------------------------- |
| `id` | `str` | Unique run identifier |
| `name` | `str` | Name of the test run |
| `projectId` | `str` | Project identifier |
| `organizationId` | `str` | Organization identifier |
| `status` | `str` | Run status |
| `evaluationStatus` | `str` | Evaluation status |
| `time` | `str` | Timestamp when the run was initiated |
| `turnType` | `str` | `"single"` or `"multi"` |
| `createdAt` | `str` | Creation timestamp |
| `updatedAt` | `str` | Last update timestamp |
| `deletedAt` | `str?` | Deletion timestamp (if soft-deleted) |
| `runContext` | `Any?` | Optional run context |
| `testRunSummary` | `TestRunSummary` | Aggregated summary of the run |
| `redirectUrl` | `str` | Direct link to the run in the Netra dashboard |
### TestRunSummary
| Field | Type | Description |
| ------------------ | ------- | ---------------------------------------- |
| `totalItems` | `int` | Total number of items in the run |
| `passedItems` | `int` | Number of items that passed |
| `failedItems` | `int` | Number of items that failed |
| `durationMs` | `float` | Total run duration in milliseconds |
| `totalCostUsd` | `float` | Total cost of the run in USD |
| `averageLatencyMs` | `float` | Average latency per item in milliseconds |
```json theme={null}
{
"success": true,
"data": {
"id": "4dd93b23-a769-401e-96c3-db42f408b65b",
"name": "Image Fidelity Test",
"projectId": "b0e6f0f3-b3fb-4d73-aea4-a75fbb2d72f3",
"organizationId": "b4a00d6b-52ff-4f2b-be84-db8bd2d5f657",
"status": "completed",
"evaluationStatus": "completed",
"time": "2026-04-02T10:45:36.751Z",
"turnType": "single",
"createdAt": "2026-04-02T10:45:36.761Z",
"updatedAt": "2026-04-02T10:47:48.271Z",
"deletedAt": null,
"runContext": null,
"testRunSummary": {
"totalItems": 1,
"passedItems": 1,
"failedItems": 0,
"durationMs": 7978,
"totalCostUsd": 0.024752200000000002,
"averageLatencyMs": 136562
},
"redirectUrl": "https://demo.getnetra.ai/test-runs/4dd93b23-a769-401e-96c3-db42f408b65b?currentOrg=b4a00d6b-52ff-4f2b-be84-db8bd2d5f657¤tProject=b0e6f0f3-b3fb-4d73-aea4-a75fbb2d72f3"
},
"error": null,
"meta": {
"timestamp": "2026-04-20T05:51:22.075Z",
"path": "/evaluations/run/4dd93b23-a769-401e-96c3-db42f408b65b",
"version": "1.0.0",
"statusCode": 200
}
}
```
Pair `get_run_results` with `run_test_suite` to programmatically inspect evaluation outcomes. The `run_id` is available in the `runId` field of the `run_test_suite` response. Use `redirectUrl` from the response to jump directly to the run in the Netra dashboard.
***
## When to Use Which API
**`create_dataset` / `add_dataset_item` / `get_all_datasets` / `get_dataset`**
Build and manage test datasets programmatically. Use for CI/CD pipelines or when generating test cases from production data.
**`run_test_suite`**
Execute your AI task against a dataset with automatic tracing and evaluation. Use for regression testing and model comparisons.
**`get_run_results`**
Fetch detailed results for a completed run, including evaluation scores and item-level outcomes. Use for post-run analysis and CI/CD assertions.
**`create_run`**
Create runs without immediate execution. Use when you need custom orchestration or want to manage run lifecycle separately.
**Evaluator IDs or Configs**
Pass evaluator IDs to `run_test_suite` to automatically score outputs. Configure custom evaluators in the Netra dashboard.
***
## Complete Example
```python theme={null}
from netra import Netra
from netra.evaluation import DatasetItem, TurnType
from openai import OpenAI
# Initialize
Netra.init(
app_name="evaluation-demo",
headers="x-api-key=your-api-key",
)
client = OpenAI()
# 1. Create a dataset
dataset_response = Netra.evaluation.create_dataset(
name="Product FAQ Evaluation",
tags=["faq", "products", "v2"],
turn_type=TurnType.SINGLE,
)
dataset_id = dataset_response.id
print(f"Created dataset: {dataset_id}")
# 2. Add test items
test_cases = [
{
"input": "What is your return policy?",
"expected_output": "Items can be returned within 30 days.",
},
{
"input": "How long does shipping take?",
"expected_output": "Standard shipping takes 3-5 business days.",
},
{
"input": "Do you offer international shipping?",
"expected_output": "Yes, we ship to over 50 countries.",
},
]
for case in test_cases:
Netra.evaluation.add_dataset_item(
dataset_id=dataset_id,
item=DatasetItem(
input=case["input"],
expected_output=case["expected_output"],
),
)
print(f"Added {len(test_cases)} test items")
# 3. Define the task
def faq_agent(query: str) -> str:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a customer support agent. Answer concisely."},
{"role": "user", "content": query},
],
)
return response.choices[0].message.content
# 4. Run the test suite
dataset = Netra.evaluation.get_dataset(dataset_id=dataset_id)
result = Netra.evaluation.run_test_suite(
name="FAQ Agent v2 Evaluation",
data=dataset,
task=faq_agent,
evaluators=["correctness", "relevance"],
max_concurrency=5,
)
# 5. Review results
print(f"\nRun completed: {result['runId']}")
for item in result["items"]:
print(f" Item {item['index']}: {item['status']} (trace: {item['traceId']})")
# 6. Fetch detailed run results
run_results = Netra.evaluation.get_run_results(run_id=result["runId"])
print(f"\nDetailed run results: {run_results}")
print("\nView detailed results in Netra dashboard → Evaluation → Test Runs")
```
## Next Steps
* [Dashboard Query](/sdk-reference/dashboard-query/python) - Query dashboard metrics
* [Usage Utilities](/usage/usage-utilities) - Query traces and spans
* [Evaluators](/Evaluation/Evaluators) - Configure custom evaluators
* [Test Runs](/Evaluation/TestRuns) - View and analyze test run results
# Evaluation
Source: https://docs.getnetra.ai/sdk-reference/evaluation/typescript
TypeScript API reference for Netra evaluation. Run test suites, create and manage datasets, add items, and measure AI output quality programmatically.
The Netra SDK exposes an `evaluation` client that lets you:
* **Manage datasets** - Create datasets and add test items
* **Run test suites** - Execute tasks against datasets with automatic tracing
* **Apply evaluators** - Score outputs using built-in or custom evaluators
This page shows how to use `netra.evaluation` to manage datasets, run test suites, and programmatically evaluate your AI applications.
## Getting Started
The `evaluation` client is available on the main `Netra` entry point after initialization.
```typescript theme={null}
import { Netra } from "netra-sdk-js";
const client = new Netra({
apiKey: "your-api-key",
});
// Access the evaluation client
await client.evaluation.createDataset(...);
await client.evaluation.addDatasetItem(...);
await client.evaluation.getDataset(...);
await client.evaluation.runTestSuite(...);
```
***
## createDataset
Create an empty dataset that can hold test items for evaluation runs.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
const client = new Netra({ apiKey: "..." });
const result = await client.evaluation.createDataset(
"Customer Support QA", // name
["support", "qa", "v1"] // tags (optional)
);
if (result) {
console.log(`Dataset created: ${result.id}`);
console.log(`Name: ${result.name}`);
console.log(`Tags: ${result.tags}`);
}
```
```typescript Signature theme={null}
createDataset(
name: string,
tags?: string[]
): Promise
```
### Parameters
| Parameter | Type | Description |
| --------- | ----------- | -------------------------------- |
| `name` | `string` | Name of the dataset (required) |
| `tags` | `string[]?` | Optional tags for categorization |
### Response: CreateDatasetResponse
| Field | Type | Description |
| ---------------- | ---------------- | ------------------------------------ |
| `id` | `string` | Unique dataset identifier |
| `name` | `string` | Dataset name |
| `tags` | `string[]` | Associated tags |
| `projectId` | `string` | Project identifier |
| `organizationId` | `string` | Organization identifier |
| `createdBy` | `string` | Creator identifier |
| `updatedBy` | `string` | Last updater identifier |
| `createdAt` | `string` | Creation timestamp |
| `updatedAt` | `string` | Last update timestamp |
| `deletedAt` | `string \| null` | Deletion timestamp (if soft-deleted) |
***
## addDatasetItem
Add a single test item to an existing dataset.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
const client = new Netra({ apiKey: "..." });
const result = await client.evaluation.addDatasetItem(
"dataset-123", // datasetId
{ // item
input: "What is the return policy for electronics?",
expectedOutput: "Electronics can be returned within 30 days with original packaging.",
tags: ["policy", "returns"],
metadata: { category: "electronics", priority: "high" },
}
);
if (result) {
console.log(`Item added: ${result.id}`);
console.log(`Input: ${result.input}`);
}
```
```typescript Signature theme={null}
addDatasetItem(
datasetId: string,
item: DatasetEntry
): Promise
// DatasetEntry interface
interface DatasetEntry {
input: any;
expectedOutput?: any;
tags?: string[];
metadata?: Record;
}
```
### Parameters
| Parameter | Type | Description |
| ----------- | -------------- | ------------------------ |
| `datasetId` | `string` | ID of the target dataset |
| `item` | `DatasetEntry` | The test item to add |
### DatasetEntry
| Field | Type | Description |
| ---------------- | ---------------------- | ----------------------------------------- |
| `input` | `any` | The input to pass to your task (required) |
| `expectedOutput` | `any?` | Expected output for comparison |
| `tags` | `string[]?` | Optional tags for the item |
| `metadata` | `Record?` | Optional metadata for evaluators |
### Response: AddDatasetItemResponse
| Field | Type | Description |
| ---------------- | ---------------------- | ------------------------------------ |
| `id` | `string` | Unique item identifier |
| `datasetId` | `string` | Parent dataset ID |
| `projectId` | `string` | Project identifier |
| `organizationId` | `string` | Organization identifier |
| `source` | `string` | Source of the item |
| `sourceId` | `string?` | Source reference ID |
| `input` | `any` | The input value |
| `expectedOutput` | `any` | The expected output |
| `isActive` | `boolean` | Whether the item is active |
| `tags` | `string[]` | Associated tags |
| `metadata` | `Record?` | Item metadata |
| `createdBy` | `string` | Creator identifier |
| `updatedBy` | `string` | Last updater identifier |
| `createdAt` | `string` | Creation timestamp |
| `updatedAt` | `string` | Last update timestamp |
| `deletedAt` | `string?` | Deletion timestamp (if soft-deleted) |
***
## getDataset
Retrieve a dataset and all its items by ID.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
const client = new Netra({ apiKey: "..." });
const dataset = await client.evaluation.getDataset("dataset-123");
if (dataset) {
console.log(`Total items: ${dataset.items.length}`);
for (const item of dataset.items) {
console.log(`ID: ${item.id}`);
console.log(`Input: ${item.input}`);
console.log(`Expected: ${item.expectedOutput}`);
console.log("---");
}
}
```
```typescript Signature theme={null}
getDataset(
datasetId: string
): Promise
```
### Parameters
| Parameter | Type | Description |
| ----------- | -------- | ----------------------------- |
| `datasetId` | `string` | ID of the dataset to retrieve |
### Response: GetDatasetItemsResponse
| Field | Type | Description |
| ------- | ----------------- | --------------------- |
| `items` | `DatasetRecord[]` | List of dataset items |
### DatasetRecord
| Field | Type | Description |
| ---------------- | -------- | ------------------- |
| `id` | `string` | Item identifier |
| `datasetId` | `string` | Parent dataset ID |
| `input` | `any` | The input value |
| `expectedOutput` | `any` | The expected output |
***
## runTestSuite
Execute a test suite against a dataset, running your task function on each item and optionally applying evaluators.
```typescript Usage theme={null}
import { Netra } from "netra-sdk-js";
import OpenAI from "openai";
const client = new Netra({ apiKey: "..." });
const openai = new OpenAI();
// Task function that processes each dataset item
async function myTask(inputData: any): Promise {
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: inputData },
],
});
return response.choices[0].message.content || "";
}
// Get dataset
const dataset = await client.evaluation.getDataset("dataset-123");
if (dataset) {
// Run test suite
const result = await client.evaluation.runTestSuite(
"GPT-4o Mini Evaluation", // name
dataset, // data
myTask, // task
["correctness", "relevance"], // evaluators (optional)
10 // maxConcurrency
);
if (result) {
console.log(`Run ID: ${result.runId}`);
console.log(`Items processed: ${result.items.length}`);
}
}
```
```typescript Signature theme={null}
runTestSuite(
name: string,
data: Dataset,
task: TaskFunction,
evaluators?: any[],
maxConcurrency?: number
): Promise | null>
// TaskFunction type
type TaskFunction = (input: any) => any | Promise
```
### Parameters
| Parameter | Type | Description |
| ---------------- | -------------- | -------------------------------------------- |
| `name` | `string` | Name for this test run (required) |
| `data` | `Dataset` | Dataset from `getDataset()` |
| `task` | `TaskFunction` | Function that takes input and returns output |
| `evaluators` | `any[]?` | Optional evaluator IDs or configs |
| `maxConcurrency` | `number` | Max parallel task executions (default: 50) |
### Response
| Field | Type | Description |
| ------- | ---------- | ------------------------------- |
| `runId` | `string` | Unique run identifier |
| `items` | `object[]` | Results for each processed item |
### Item Result
| Field | Type | Description |
| --------------- | -------- | ------------------------------ |
| `index` | `number` | Item index in dataset |
| `status` | `string` | `"completed"` or `"failed"` |
| `traceId` | `string` | Trace ID for observability |
| `spanId` | `string` | Span ID for the task execution |
| `testRunItemId` | `string` | Backend item identifier |
The `task` function receives the `input` field from each dataset item. Return the output that should be compared against `expectedOutput` by evaluators.
***
## When to Use Which API
**`createDataset` / `addDatasetItem` / `getDataset`**
Build and manage test datasets programmatically. Use for CI/CD pipelines or when generating test cases from production data.
**`runTestSuite`**
Execute your AI task against a dataset with automatic tracing and evaluation. Use for regression testing and model comparisons.
**`createRun`**
Create runs without immediate execution. Use when you need custom orchestration or want to manage run lifecycle separately.
**Evaluator IDs or Configs**
Pass evaluator IDs to `runTestSuite` to automatically score outputs. Configure custom evaluators in the Netra dashboard.
***
## Complete Example
```typescript theme={null}
import { Netra } from "netra-sdk-js";
import OpenAI from "openai";
async function main() {
// Initialize
const client = new Netra({
apiKey: "your-api-key",
});
const openai = new OpenAI();
// 1. Create a dataset
const datasetResponse = await client.evaluation.createDataset(
"Product FAQ Evaluation",
["faq", "products", "v2"]
);
if (!datasetResponse) {
console.error("Failed to create dataset");
return;
}
const datasetId = datasetResponse.id;
console.log(`Created dataset: ${datasetId}`);
// 2. Add test items
const testCases = [
{
input: "What is your return policy?",
expectedOutput: "Items can be returned within 30 days.",
},
{
input: "How long does shipping take?",
expectedOutput: "Standard shipping takes 3-5 business days.",
},
{
input: "Do you offer international shipping?",
expectedOutput: "Yes, we ship to over 50 countries.",
},
];
for (const testCase of testCases) {
await client.evaluation.addDatasetItem(datasetId, {
input: testCase.input,
expectedOutput: testCase.expectedOutput,
});
}
console.log(`Added ${testCases.length} test items`);
// 3. Define the task
async function faqAgent(query: string): Promise {
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "system",
content: "You are a customer support agent. Answer concisely.",
},
{ role: "user", content: query },
],
});
return response.choices[0].message.content || "";
}
// 4. Run the test suite
const dataset = await client.evaluation.getDataset(datasetId);
if (dataset) {
const result = await client.evaluation.runTestSuite(
"FAQ Agent v2 Evaluation",
dataset,
faqAgent,
["correctness", "relevance"],
5
);
// 5. Review results
if (result) {
console.log(`\nRun completed: ${result.runId}`);
for (const item of result.items) {
console.log(
` Item ${item.index}: ${item.status} (trace: ${item.traceId})`
);
}
console.log(
"\nView detailed results in Netra dashboard → Evaluation → Test Runs"
);
}
}
}
main();
```
## Next Steps
* [Dashboard Query](/sdk-reference/dashboard-query/typescript) - Query dashboard metrics
* [Usage Utilities](/usage/usage-utilities) - Query traces and spans
* [Evaluators](/Evaluation/Evaluators) - Configure custom evaluators
* [Test Runs](/Evaluation/TestRuns) - View and analyze test run results
# Models
Source: https://docs.getnetra.ai/sdk-reference/models
Python API reference for Netra models. Fetch model details and pricing information for AI models tracked by your project.
The Netra SDK exposes a `models` client that lets you:
* **Fetch model pricing** - Retrieve pricing details for all AI models in your project
* **Filter by model name** - Look up pricing for a specific model by name
This page shows how to use `Netra.models` to query model pricing information programmatically.
## Getting Started
The `models` client is available on the main `Netra` entry point after initialization.
```python theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# Access the models client
Netra.models.get_model_pricing(...)
```
***
## get\_model\_pricing
Fetch model details and pricing for all models associated with your project, or filter by a specific model name.
```python Usage theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
# Fetch pricing for all models in the project
all_models = Netra.models.get_model_pricing()
print(all_models)
# Fetch pricing for a specific model
model = Netra.models.get_model_pricing(name="gpt-5.4-pro")
print(model)
```
```python Signature theme={null}
get_model_pricing(
name: Optional[str] = None,
) -> List[Any] | Any
```
### Parameters
| Parameter | Type | Description |
| --------- | ------ | ---------------------------------------------------------------------------------------- |
| `name` | `str?` | Optional model name to filter results. When omitted, returns all models for the project. |
### Response
Returns a list of model objects from the API. Each object contains the model's details and pricing information. Returns an empty list if the request fails or no models are found.
```json theme={null}
{
"id": "string",
"createdAt": "ISO-8601 timestamp",
"updatedAt": "ISO-8601 timestamp",
"deletedAt": "ISO-8601 timestamp | null",
"projectId": "string | null",
"name": "string",
"matchPattern": "string",
"prices": [
{
"price": "number",
"maxUnits": "number | null",
"minUnits": "number",
"unitValue": "number",
"usageType": "input | output | cache"
}
]
}
```
The `name` parameter performs a filter on the server side. Use it to avoid fetching the full model catalog when you only need pricing for a specific model.
***
## Configuration
### Environment Variables
The models client respects the following environment variables:
| Variable | Default | Description |
| --------------------- | ------- | ----------------------------------------------------------------------- |
| `NETRA_OTLP_ENDPOINT` | — | **Required.** The OTLP endpoint used to derive the models API base URL. |
| `NETRA_API_KEY` | — | API key for authenticating requests. |
***
## Complete Example
```python theme={null}
from netra import Netra
# Initialize the SDK
Netra.init(
app_name="pricing-demo",
headers="x-api-key=your-api-key",
)
# 1. Fetch all model pricing
all_models = Netra.models.get_model_pricing()
print(f"Total models available: {len(all_models)}\n")
print(all_models)
# 2. Look up pricing for a specific model
target_model = "gpt-4o-mini"
results = Netra.models.get_model_pricing(name=target_model)
if results:
print(results)
else:
print(f"\nNo pricing found for {target_model}")
```
## Next Steps
* [Evaluation](/sdk-reference/evaluation/python) - Run test suites and evaluate AI outputs
* [Dashboard Query](/sdk-reference/dashboard-query/python) - Query dashboard metrics
* [Usage Utilities](/usage/usage-utilities) - Query traces and spans
# Prompts
Source: https://docs.getnetra.ai/sdk-reference/prompts
Fetch versioned prompts from Netra into your Python or TypeScript app. Load production, staging, or labeled prompt versions without hardcoding strings.
The Netra SDK exposes a `prompts` API that lets you:
* **Fetch managed prompts** – Retrieve prompt versions by name and label from the Netra backend
* **Use labels for versioning** – Target specific prompt versions such as `"production"`, `"staging"`, or any custom label
* **Integrate seamlessly** – Drop fetched prompts directly into your LLM calls with zero boilerplate
This page shows how to use the `get_prompt` utility in Netra to fetch and use your managed prompts.
## **Getting Started**
The prompts client is initialized automatically when you call `Netra.init`. No additional flags are required as long as a valid OTLP endpoint and API key are configured, prompts are ready to use.
```python theme={null}
from netra import Netra
Netra.init(
app_name="sample-app",
)
```
## **get\_prompt**
Fetch a prompt version by name and label from the Netra backend.
```python theme={null}
from netra import Netra
Netra.init(
app_name="sample-app",
)
# Fetch the production version of a prompt
prompt = Netra.prompts.get_prompt(name="welcome-message")
# Fetch a specific label
staging_prompt = Netra.prompts.get_prompt(
name="welcome-message",
label="staging",
)
```
### **Parameters**
| **Parameter** | **Type** | **Description** |
| :------------ | :------- | :------------------------------------------------------------------------------------------------------ |
| `name` | `str` | Name of the prompt to fetch. This is required — passing an empty string returns `None`. |
| `label` | `str` | Label of the prompt version to retrieve (e.g. `"production"`, `"staging"`). Defaults to `"production"`. |
### **Return Value**
| **Scenario** | **Return** |
| :------------------------ | :------------------------------------------------------------- |
| Prompt found | `dict` — the prompt version data returned by the Netra backend |
| `name` is empty or `None` | `None` |
| Network / server error | `{}` (empty dict) |
| Client not initialized | `{}` (empty dict) |
## **Configuration**
### **Required**
`get_prompt` relies on the OTLP endpoint and API key configured via `Netra.init` or environment variables.
| **Setting** | **Environment Variable** | **Description** |
| :------------ | :----------------------- | :------------------------------------------------------------------------------------------------------ |
| OTLP endpoint | `NETRA_OTLP_ENDPOINT` | Base URL for the Netra backend. The prompts client strips a trailing `/telemetry` suffix automatically. |
| API key | `NETRA_API_KEY` | API key sent as the `x-api-key` header on every request. |
## **Error Handling**
`get_prompt` is designed to never throw. Errors are logged and a safe fallback value is returned so your application keeps running.
```python theme={null}
from netra import Netra
Netra.init(app_name="sample-app")
prompt = Netra.prompts.get_prompt(name="onboarding-flow")
if not prompt:
# Handle missing prompt — use a hardcoded fallback
prompt_text = "Welcome! How can I help you today?"
else:
prompt_text = prompt.get("messages", "")
```
Common error scenarios and their log messages:
| **Scenario** | **Log Level** | **Message** |
| :--------------------- | :------------ | :------------------------------------------------------------------------------------------- |
| Empty `name` argument | `ERROR` | `netra.prompts: name is required to fetch a prompt` |
| Client not initialized | `ERROR` | `netra.prompts: Prompts client is not initialized; cannot fetch prompt version for ''` |
| Network / HTTP error | `ERROR` | `netra.prompts: Failed to fetch prompt version for '' (label=