Automatic labelling is only as good as the ground truth people have manually labelled

Understanding what went wrong with your calls requires human hand-labelled data, not just AI

Reviewing calls is addictive. As much as advanced/generative analytics can power thoughtful recommendations about the right changes to make to a call, there is no substitute for listening to them in detail, reviewing call logs, and figuring out what we should have done to improve the call for the next time around.

The problem is this data often lives across disparate systems: people might choose to track them in Google Sheets, Apple Notes, emails, project tracking. Things don’t get tracked, and then issues get swept under the rug. When calls don’t get reviewed, you can’t easily understand the scale of the issue - is it a one-off, or something happening in 1 in 5 calls?

Finding a way to bring these into one central workflow is critical for any AI Agent system, and Replicant is launching its In-App Issue Flagging for customers today.

However, what makes this exciting is that it runs against a trend we’ve seen in the industry: trying to automate this labelling system using AI to extract things happening inside your conversations, obviating the need for any human review of the calls. We're already pioneers in this area: we have an entire product built around Scorecards for human agents, which we apply to our AI Agents as well.

What we observed first-hand is interesting: understanding one individual call requires an immense amount of nuance. It requires context on the design of the AI Agent, and what exactly “common sense” intuition tells us an AI Agent should do. LLMs are not there yet without significant adjustments to the context of the individual AI Agent they are judging.

What we have built is a pipeline that takes the manually labelled data and uses that to improve our upcoming AI Agent analysis, improving the accuracy rate, so that we can achieve trustworthy Recursive Self-Improvement that makes our product better over time.

The end result will be a trustworthy platform that reliably and efficiently gives you confidence in your AI Agent now, and as it improves. It gives you transparency over what you and your broader operations team are finding. And it gets better with time!

‍