We Stopped Trusting Other Models. We Built Our Own.

Every response an AI agent gives on a Replicant conversation gets checked before it’s spoken, and early benchmarks show it outperforming the approach it replaces.

The most important model on the call is the one callers never hear

Every turn of a Replicant AI agent runs through a chain of models. Most of them shape what gets said. One of them checks whether what’s about to be said is actually supported by the conversation. We call it the hallucination check, and it runs on nearly every response.

For most of our history, that check ran on a third-party model. It worked, but it meant paying per call, accepting whatever latency the provider delivered, and having little room to tune a model for one narrow job.

So, we built our own.

Why voice makes this harder

A chat user will wait a few seconds for an answer. A caller on the phone notices a pause immediately. Every check that adds latency lands in the middle of a live conversation, and every error that gets through lands in front of a real customer.

That’s why our model runs constantly, and it has to be right every time.

Trained on synthetic conversations

We trained our first custom model entirely on synthetic data with realistic products and workflows, producing thousands of conversations shaped like real calls. The model learned the job from those conversations.

We tested both checkers and measured the same kinds of replies and how often each one caught a real error and how often it raised a false alarm. The new checker caught 28 of 46 real errors, up from 15% for the previous checker. Most of what it flagged was a real problem, with 93% precision.

The two tests used different samples and labeling methods. The previous test used a much larger set of live production replies labeled by an AI reviewer. The new test used 71 real-call replies labeled by people. Read the results as a clear direction and a large improvement in catching errors, not a precise head-to-head comparison. The tradeoff is more false alarms, with 2 of 25 clean turns flagged, so we’re tracking that closely as the model moves onto live traffic.

Built for speed on the easy cases

Most turns need no correction, so the model is designed to clear those quickly and to spend its precision on the turns that do need a second look. When it flags something, it identifies the specific text in question, and a separate step rewrites it.

The design target for this check is 200 to 500 milliseconds, down from 1 to 3 seconds with a third-party approach.

Governance built in

Training on customer data requires a complete record of where the data came from, how it was de-identified, and who approved its use. So we built a model governance system that tracks that lineage and requires human sign-off on key steps. Teams across the industry face the same problem, and we’re exploring ways to share what we’ve learned.

What’s next

• Retraining on governed customer data. Once the governance system is fully in use, we’ll retrain on de-identified customer conversations, with a clear record of what each model learned from.
• More models, more tasks. Over the next 3 to 6 months, we expect to apply the same approach to other steps in the pipeline, replacing general-purpose calls with models built for each job.
• Published results. Measured accuracy and latency numbers, shared openly.

Replacing general-purpose models is how we expect our AI agents to get faster, more accurate, and more predictable over time. Callers won’t see the models. They’ll hear the difference.