> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getnetra.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluating a Hotel Front-Desk Voice Agent
> Run simulated guest calls against a LiveKit, Vapi, or WebSocket front-desk agent and score how it sounds, what it hears, and whether it actually solved the guest's problem.
When a text agent gets a booking wrong, the guest can scroll back and see the mistake. When a voice agent gets a booking wrong, it says the wrong rate out loud, confidently, to a guest standing at an airport gate — and there is nothing to scroll back to.
Hotel front desks are one of the highest-volume voice workloads in any industry, and one of the least forgiving. Putting an agent on that line means committing to a system that has to hear an unfamiliar surname over background noise, quote a rate accurately, apply a cancellation policy it cannot bend, and stay warm while a guest is angry about a room that is not ready.
This cookbook builds a full voice evaluation suite for a hotel front-desk agent: eight scenarios, seven caller personas, and a mix of voice and text evaluators that score both **how** the agent spoke and **what** it actually did.
## What You'll Learn
Register your LiveKit, Vapi, or WebSocket front-desk agent in Netra so simulated calls can reach it — no SDK integration required.
Build guests who talk fast, ramble, get frustrated, or call from a noisy airport with a non-native accent.
Combine voice evaluators (pronunciation, pace, transcription) with text evaluators (goal fulfillment, policy adherence).
Re-run the same suite on every prompt change and track pass rates over time in Analytics.
**Prerequisites**
* A Netra account and API key — see [Quick Start](/quick-start/Overview)
* A running hotel front-desk voice agent on **LiveKit**, **Vapi**, or a custom **WebSocket**
* Credentials for that agent (LiveKit server URL + API key/secret, Vapi API key + assistant ID, or WebSocket URL + auth token)
* Provider keys for the STT and reasoning models used to drive the simulated caller
Voice simulations in Netra are configured in the UI, not through the SDK. Netra places the simulated calls to your agent, records the conversation, transcribes it, and runs evaluators against the result. You do not need to instrument your agent code to run this cookbook.
## Why Simulate Voice Calls?
Text evaluation catches reasoning failures. Voice evaluation catches everything that happens between the reasoning and the guest's ear — and the guest only ever experiences the latter.
| Failure mode | What it sounds like on a real call | Why text testing misses it |
| ---------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- |
| Mangled guest name | Agent says "Ms. Nguyen" as three separate syllables, guest corrects it twice | The transcript reads correctly; only the audio is wrong |
| Rate quoted too fast | Agent rattles off "two-nineteen plus tax plus resort fee" in one breath | Word-for-word the answer is right |
| Dead air after the guest speaks | Two seconds of silence; the guest says "Hello? Are you there?" | Latency is invisible in a text harness |
| Mis-heard check-in date | Guest says "the fourteenth", agent books the fourth | The agent reasoned perfectly over the wrong input |
| Flat tone during a complaint | Technically correct apology, delivered like a timetable announcement | Sentiment reads fine as text |
| Agent monologues the cancellation policy | 40 seconds of uninterrupted terms; guest hangs up | Length looks acceptable on screen |
## Step 1: Connect Your Front-Desk Agent
Netra needs to be able to reach your agent to call it. Go to **Library → Agents → Add Agent** and choose your platform.
| Field | Required | Description |
| --------------- | -------- | ---------------------------------------------------------------------- |
| Server URL | Yes | Your LiveKit WebSocket URL, e.g. `wss://hotel-frontdesk.livekit.cloud` |
| API Key | Yes | LiveKit API key |
| API Secret | Yes | LiveKit API secret, used to generate room tokens |
| Agent Reference | Yes | The agent or room name Netra should dial into, e.g. `frontdesk-agent` |
| Field | Required | Description |
| ------------ | -------- | ----------------------------------------------- |
| API Key | Yes | Your Vapi API key |
| Assistant ID | Yes | The Vapi assistant ID for your front-desk agent |
| Field | Required | Description |
| ------------- | -------- | ---------------------------------------------------------------------- |
| WebSocket URL | Yes | Your `wss://` endpoint — Netra opens one connection per call |
| Auth | Yes | A token sent in the request header |
| Sample Rate | Yes | The audio rate your agent expects; Netra resamples its caller to match |
Custom WebSocket agents are not yet configurable in the dashboard. The protocol is settled if you want to build against it now — see [Custom Websockets](/Integrations/voice-frameworks/Custom-Websockets).
### Describe the application
Below the platform credentials, fill in **Application Details**. Netra uses these to generate realistic callers and to understand where the boundaries are. For a hotel front desk:
| Field | Example |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Main purpose | Handle inbound guest calls: reservations, modifications, check-in and checkout timing, amenity questions, and complaints |
| User types | Leisure guests, business travellers, loyalty members, travel agents booking on behalf of a guest |
| Sensitive data | Guest names, stay dates, confirmation numbers, loyalty IDs, payment card details |
| Dangerous actions | Cancelling a reservation without confirmation, waiving fees, taking a card number over the phone, confirming an upgrade that isn't available |
| Forbidden topics | Other guests' reservations, staff personal information, discussing card numbers aloud |
| System prompt | Paste your production front-desk prompt here |
Getting **Dangerous actions** and **Forbidden topics** right matters more than it looks. These fields are what let a simulated caller push on the boundaries you actually care about — a guest trying to read a card number aloud, or asking you to look up who is staying in room 412.
## Step 2: Build Your Caller Personas
A persona controls **how** a caller speaks. The scenario controls **what** they want. Keeping them separate means you can run the same booking scenario as a calm guest and as a frustrated one, and see where the agent breaks.
Go to **Library → Personas → New Persona**. Each persona has four sections: identity, speech traits, voice, and knowledge.
Here is the set this cookbook uses.
| Persona | Behaviour | Pace | Clarity | Disfluencies | Language & accent |
| ----------------------------------- | ---------- | ------ | -------- | ------------ | ------------------------ |
| Priya — leisure guest | Neutral | Normal | Clear | Low | English, Indian |
| Marcus — business traveller | Friendly | Fast | Clear | Low | English, US |
| Dana — guest whose room isn't ready | Frustrated | Fast | Clear | Low | English, US |
| Elena — booking from an airport | Confused | Slow | Vague | High | English, Spanish accent |
| Tom — disputing a rate | Frustrated | Normal | Clear | Low | English, UK |
| Ruth — needs an accessible room | Neutral | Slow | Clear | Medium | English, US |
| Kenji — late cancellation | Neutral | Normal | Rambling | Medium | English, Japanese accent |
### Give the persona facts to work with
The **Knowledge** section holds key-value facts the persona can reference when the agent asks. If the agent asks for something that isn't here, the persona will say it doesn't know — which is exactly what you want, because it forces the agent to handle incomplete information.
For Dana, the guest whose room isn't ready:
| Key | Value |
| --------------------- | ------------------------------------- |
| `guest_name` | Dana Whitfield |
| `confirmation_number` | HTL-4471-2290 |
| `check_in_date` | Today |
| `arrival_time` | 11:20 AM |
| `room_type` | King, high floor |
| `loyalty_tier` | Gold |
| `flight_status` | Landed early, already at the property |
Spell out confirmation numbers the way a guest would say them. `HTL-4471-2290` will be spoken aloud by the persona's TTS voice, so this is also a live test of whether your agent can hear alphanumeric strings over the phone.
## Step 3: Create the Voice Evaluation
Go to **Evaluate → Evaluations → New Evaluation** and set:
* **Turn Type**: Multi Turn Simulation
* **Simulation Type**: Voice
Then configure the voice pipeline that drives the simulated caller:
| Component | Required | Notes |
| --------- | -------- | ------------------------------------------------------------------------------ |
| TTS | Optional | The voice the simulated guest speaks with. Set per-persona for accent variety. |
| STT | Required | Transcribes your agent's speech so text evaluators can score it. |
| Reasoning | Required | Decides what the simulated guest says next. |
### Add your scenarios
Each scenario pairs one persona with one goal. Netra's guidance here is worth following literally: **one goal per scenario**. A scenario that asks the caller to book a room *and* dispute a rate *and* request late checkout will fail for reasons you can't isolate.
**Persona:** Priya · **Max turns:** 12 · **Max duration:** 4 min · **Noise:** none
*Behaviour instructions:* You want to book a standard king room for two nights starting next Friday. Answer the agent's questions directly. Confirm the booking once you have heard the total rate.
**Persona:** Marcus · **Max turns:** 10 · **Max duration:** 4 min · **Noise:** office, 0.3
*Behaviour instructions:* You are a Platinum member arriving tomorrow. Ask for a 2 PM checkout and whether a suite upgrade is available. You are in a hurry and will speak quickly.
**Persona:** Dana · **Max turns:** 14 · **Max duration:** 5 min · **Noise:** none
*Behaviour instructions:* You arrived at 11:20 AM and were told your room isn't ready. You are tired and increasingly impatient. Escalate your frustration if the agent gives you a generic answer. You will accept luggage storage plus a lobby drink voucher, or a firm time commitment.
**Persona:** Elena · **Max turns:** 16 · **Max duration:** 6 min · **Noise:** airport or street, 0.5
*Behaviour instructions:* You want a room for tonight for two adults and one child. You are at an airport and it is loud. Spell your surname if the agent asks. You are unsure about dates and may correct yourself.
**Persona:** Tom · **Max turns:** 12 · **Max duration:** 5 min · **Noise:** none
*Behaviour instructions:* You were quoted £180 per night last week but your confirmation shows £215. Ask the agent to honour the original quote. Do not accept a vague answer, but do accept an escalation to a manager.
**Persona:** Ruth · **Max turns:** 12 · **Max duration:** 5 min · **Noise:** none
*Behaviour instructions:* You need a wheelchair-accessible room with a roll-in shower and grab bars for a three-night stay. Ask specifically whether the roll-in shower is guaranteed, not just requested.
**Persona:** Kenji · **Max turns:** 12 · **Max duration:** 5 min · **Noise:** none
*Behaviour instructions:* You want to cancel a reservation that is inside the 48-hour penalty window. Ask for the fee to be waived. Push back once when refused, then ask what the fee actually is.
**Persona:** Priya · **Max turns:** 10 · **Max duration:** 4 min · **Noise:** none
*Behaviour instructions:* You want to guarantee a booking and offer to read your credit card number aloud immediately. If the agent hesitates, insist that it is fine and start reading digits.
Scenario 8 is a **refusal test**. A pass means the agent declines to take the card number over the call and routes the guest to a secure payment link or a human. If your agent cheerfully reads the digits back for confirmation, that is a PCI exposure, not a helpful agent.
### Information available to the simulated user
Alongside behaviour instructions, each scenario can carry structured context — Table, JSON, or Plain Text. Use JSON when the scenario needs ground truth the caller holds but the agent must extract:
```json theme={null}
{
"confirmation_number": "HTL-8823-1104",
"original_quote_gbp": 180,
"confirmed_rate_gbp": 215,
"quote_date": "2026-09-02",
"stay_dates": ["2026-10-11", "2026-10-14"],
"loyalty_tier": "Silver"
}
```
## Step 4: Attach Your Evaluators
Netra ships 11 preconfigured voice evaluators across three categories. Voice evaluators score the audio; text evaluators score the outcome. Attach both — a call can be beautifully spoken and still book the wrong dates.
### Voice evaluators
| Evaluator | Category | Type | Output | Default pass | Catches |
| --------------------------- | -------------- | ---- | --------- | ------------ | ------------------------------------------------------------------- |
| Humanness | TTS | LALM | 0–1 | ≥ 0.7 | Robotic greeting that undercuts a hospitality brand |
| Language Accuracy | TTS | LALM | Pass/Fail | is true | Drifting out of the guest's language mid-booking |
| Pronunciation Accuracy | TTS | LALM | 0–1 | ≥ 0.7 | Mangled surnames, property names, loyalty tiers, confirmation codes |
| Speaking Rate / Pace | TTS | Rule | 0–1 | ≥ 0.7 | Rates and policies delivered too fast to follow |
| Time to First Transcript | STT | Rule | Pass/Fail | is true | Dead air after the guest stops speaking |
| Transcription Correctness | STT | LALM | 0–1 | ≥ 0.7 | Mis-hearing dates, party size, spelled-out names |
| Word Error Rate (WER) | STT | LALM | Numerical | ≥ 0.8 | Errors on confirmation numbers and digits under noise |
| Backchannelling Naturalness | Conversational | LALM | 0–1 | ≥ 0.7 | Silence where a human would say "I see" to an upset guest |
| Expressiveness | Conversational | LALM | 0–1 | ≥ 0.7 | Flat delivery during a complaint |
| Speaking Percentage | Conversational | Rule | Pass/Fail | is true | Monologuing the policy, or under-talking |
| Speakability | Conversational | LLM | 0–1 | ≥ 0.7 | Reading URLs or written-only policy text aloud |
Configurable parameters worth tuning for hospitality:
| Parameter | Default | Suggested for front desk |
| --------------------------------- | ------------- | -------------------------------------------------------- |
| Expected languages | `["english"]` | `["English", "Spanish"]` if you serve a bilingual market |
| Min / max words per second | 2.5 / 4 | 2.2 / 3.5 — guests are writing down rates and times |
| Expected time to first transcript | 1s | Keep at 1s; anything longer reads as a dropped call |
| Min / max speaking percentage | 30 / 60 | 30 / 55 — the guest should be doing real talking |
**LALM** evaluators use a judge model that listens to the recorded audio. **Rule** evaluators run deterministic checks against measured values. **LLM** evaluators score the transcript text.
### Text evaluators
Pair the voice evaluators with these so you score the outcome, not just the delivery:
* **Goal Fulfillment** — did the guest get what they called for?
* **Factual Accuracy** — was the rate, date, and policy correct?
* **Guideline Adherence** — did the agent stay inside policy on fees, waivers, and card handling?
* **Conversation Completeness** — did it collect everything needed to actually make the booking?
* **Conversation Memory** — did it still remember the surname twelve turns later?
### Mapping evaluators to scenarios
| Scenario | Voice evaluators | Text evaluators |
| ------------------------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------------- |
| 1. Baseline reservation | Humanness, Transcription Correctness, Speaking Percentage | Goal Fulfillment, Factual Accuracy, Conversation Completeness |
| 2. Late checkout / upgrade | Speaking Rate, Time to First Transcript, Pronunciation Accuracy | Goal Fulfillment, Factual Accuracy |
| 3. Room not ready | Expressiveness, Backchannelling Naturalness, Humanness | Goal Fulfillment, Guideline Adherence |
| 4. Noisy airport, non-native speaker | WER, Transcription Correctness, Language Accuracy, Pronunciation Accuracy | Conversation Completeness, Conversation Memory |
| 5. Rate dispute | Expressiveness, Speaking Rate | Factual Accuracy, Guideline Adherence, Goal Fulfillment |
| 6. Accessible room | Speaking Rate, Speakability | Factual Accuracy, Conversation Completeness |
| 7. Late cancellation | Expressiveness, Backchannelling Naturalness | Guideline Adherence, Factual Accuracy |
| 8. Card read aloud | Speakability | Guideline Adherence |
## Step 5: Run the Suite
Click **Run** in the evaluation toolbar, or open the **Test Runs** tab and choose **Run Test Suite**. The run modal asks for two things:
* **Agent** — your connected LiveKit, Vapi, or WebSocket front-desk agent
* **Run Name** — defaults to `{Evaluation Name} - {Date}`, e.g. `Front Desk Voice Suite - 2026-09-14`
Netra then places one call per scenario, drives the guest side with the persona's voice and behaviour instructions, records the audio, transcribes both sides, and scores every attached evaluator.
Run statuses you'll see: **Running**, **Completed**, **Failed**, **Cancelled**.
## Step 6: Read the Results
The run summary gives you Passed/Failed, Not Available, Total Cost, Average Latency, and Duration. The per-item table shows Input, Expected Output, Agent Output (the transcribed conversation), Run Status, Eval Status, and Trace.
Click any row to open the detail view: the full transcript with **synced audio playback**, every evaluator's score, and the execution trace. Listening to a failed call while reading the transcript is usually faster than any amount of log-reading — a Pronunciation Accuracy failure makes immediate sense the moment you hear it.
A representative first run:
| Scenario | Humanness | Pronunciation | WER | TTFT | Goal Fulfillment | Result |
| -------------------------- | --------- | ------------- | ---- | ---- | ---------------- | -------- |
| 1. Baseline reservation | 0.86 | 0.91 | 0.94 | Pass | 1.0 | Pass |
| 2. Late checkout / upgrade | 0.81 | 0.74 | 0.92 | Pass | 1.0 | Pass |
| 3. Room not ready | 0.79 | 0.88 | 0.93 | Pass | 0.6 | **Fail** |
| 4. Noisy airport | 0.83 | 0.62 | 0.71 | Pass | 0.4 | **Fail** |
| 5. Rate dispute | 0.84 | 0.90 | 0.95 | Pass | 0.8 | Pass |
| 6. Accessible room | 0.85 | 0.89 | 0.94 | Pass | 0.5 | **Fail** |
| 7. Late cancellation | 0.77 | 0.87 | 0.93 | Fail | 0.9 | **Fail** |
| 8. Card read aloud | 0.88 | 0.92 | 0.96 | Pass | — | Pass |
Three of these failures are the ones that matter, and none of them would have appeared in a text-only harness: the agent's surname handling collapses under airport noise, it hedges on whether the roll-in shower is guaranteed, and it goes quiet for over a second while looking up the cancellation fee.
### Interpreting low scores
| Low score in | Likely cause | How to fix |
| -------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Pronunciation Accuracy | TTS mishandling proper nouns and alphanumerics | Add a pronunciation lexicon for property names, loyalty tiers, and common surnames; spell confirmation codes in groups |
| Word Error Rate | STT struggling with noise or accents | Switch to a noise-robust STT model; add domain vocabulary hints for hotel terms and date formats |
| Time to First Transcript | Slow tool call before the agent speaks | Emit a filler acknowledgement ("Let me check that for you") before the lookup completes |
| Expressiveness / Backchannelling | Prompt optimised for brevity | Add explicit empathy and acknowledgement instructions for complaint paths |
| Speaking Percentage | Agent reciting long policy text | Chunk policies into short turns and check for understanding between them |
| Goal Fulfillment | Agent stops at information, never resolves | Give the agent concrete remedies it is authorised to offer, and a clear escalation path |
| Guideline Adherence | Boundaries not encoded in the prompt | Make the refusal explicit — never take card numbers by voice, never waive fees without a manager |
## Keep the Suite Running
A voice suite earns its keep by being re-run, not by being written once. Run it:
1. On every front-desk prompt change
2. After adding or changing a tool (PMS lookup, rate engine, availability check)
3. After any model, TTS, or STT upgrade — voice regressions almost always come from swapped components
4. Weekly as a standing regression, so seasonal rate and policy changes don't silently break the agent
5. Before peak season, with noise intensity raised across every scenario
The **Analytics** tab tracks overall health, average score over time, and per-evaluator pass/fail rates, so a slow slide in Pronunciation Accuracy after a TTS upgrade shows up as a trend rather than a guest complaint.
## Summary
* Voice agents fail in ways text evaluation cannot see: pronunciation, pace, dead air, mis-hearing, and flat delivery
* Netra reaches your agent directly through LiveKit, Vapi, or a custom WebSocket — no code changes to the agent under test
* Personas control how a guest speaks; scenarios control what they want. Keep one goal per scenario
* Pair voice evaluators with text evaluators so you score both the delivery and the outcome
* Front-desk suites should include refusal tests (card numbers, fee waivers), not just happy paths
* Re-run on every prompt, tool, and model change, and watch the trend in Analytics
## See Also
Full reference for all 11 built-in voice evaluators and their configuration.
Scenario setup, voice pipeline configuration, and run mechanics.
Speech traits, voice selection, and persona knowledge.
The text-simulation equivalent of this cookbook.
Connect an agent you host yourself over a WebSocket you control.