AI Testing With Synthetic Customers

October 28, 2026

A testing and telemetry setup where voice assistants are put in conversation with another AI playing the customer — and measured — before they ever meet a real user.

You Don't Test a Voice Assistant by Listening. You Test It by Talking to It.

A few trial conversations tell you almost nothing about a voice assistant's quality; the failures hide in the tail of the conversation, at the moment of interruption, in a misheard product name. For Hardiye we turned this into a testing problem: an AI playing the customer joins the real-time room and talks to the assistant, and the conversation's event timeline is verified automatically against written scenarios.

We don't treat latency as one number. A sentence's mouth-to-ear journey is marked step by step: the moment the user stops speaking, the model's first word, the moment audio starts playing. We see separately how much of the total is spent in the backend and how much in the avatar. A browser driven with a fake microphone replays interruption (barge-in) scenarios on every release.

Measurement drives decisions: the silence threshold is tuned by A/B test, tool outputs are compressed to the size the model handles well, a possibly misheard product name triggers a spoken confirmation, and non-Latin hallucinations in transcripts are counted separately. The same discipline runs through our other AI systems: cost- and latency-targeted scenario tests for the company brain, and leakage measurement on every Motice release.

How It Works

• LLM-simulated customer: speech-to-speech tests in a real-time room
• Event-timeline verification against YAML scenarios
• Mouth-to-ear latency telemetry, with backend and avatar shares split
• Latency and barge-in tests through a fake-microphone browser
• End-to-end per-session traces in Langfuse
• Scenario tests with cost and latency targets

Where We Use It

In Hardiye, Hardline Nutrition's voice assistant; and in the AI layers of APEX and Motice.