A guide to conversational metrics

Conversational interfaces are everywhere. Your website visitors use your live chat to ask questions and inform themselves before they buy, your app users use the support chat to get help and your conversational commerce experiences guide potential customers to purchase the right product, one message at a time.

But how do you measure this new conversational world?
Every time a new channel is added to your channel mix and technical stack, the question comes to metrics: what do you measure, how do you measure it and how do you visualize it?

Of course, you can start with simple channel analytics - which channels are connected and how many conversations and messages go through every day. But this is too high level for the many questions to ask and metrics to answer them: does the app chat support perform better than Whatsapp support, what is used more - by number of conversations but also by number of average user conversations. How do the channels differ in usage - for example what the most active hours per channel are.

All these questions lead to real decisions: how many support agents do you need to hire or retrain for each channel, when do they need to be most active, how does the channel compare to other channels you have running - does it make sense to invest more in conversational channel A or conversational channel B?

And we haven’t even started looking at operating metrics: NLP analytics like most common intents, phrase clustering or analysing conversation paths with flowcharts. Last but not least don’t forget about business metrics like Merchant Response Rates, response times or CSATs. They define how well your conversational experience is actually adopted on the business side.

Conversational Metrics 101

There are many conversational metrics to look at and a whole new world of analytics to track, store, and explore. And everybody with analytics experience knows: with great choice comes great responsibility.

The choice is an important one, thankfully the answer is straight forward:
Conversational interfaces are just that: a new type of interface, a useful and universal one. Perhaps the future of universal interfaces. Perhaps soon all your interfaces will be conversational, at least partially, staffed by Conversational AI and human agents, talking to your users like friends and family.

For this vision to arrive, conversational interfaces need to proof their value first.
So for your conversational tracking you should focus on 2 metrics:
- Standard Conversational Metrics: to keep track of this new channel and improve it
- Value Metrics: to prove the ROI of this channel, focus on the value it brings

An overview of conversational metrics

But first, let's look at the standard conversational metrics. For an overview of Conversational Metrics and their definitions, check this handy list of possible metrics:
https://airy.co/docs/enterprise/analytics/introduction

We divided them into 5 major categories:

Setup & Channel Analytics
Everything you want to know about the conversational channels you have

Conversational Analytics
Metrics for Messages, Conversations and everything in between

NLP Analytics
Intents, Top In & Out Messages and more

Demographics
Understand where your users come from and which languages they speak

Business Analytics
Business Metrics from MRR to CSAT

Out of these, you choose the standard metrics you want to track. In a second we will show you which ones we chose for our standard analytics setup.

Value Metrics

Value metrics explain themselves with their name: what value comes from this channel? They go one step further than the metrics above and try to capture the ROI of the conversational experience.

The trick here is that conversational experiences are often multi-intent: Your website visitors use your live chat to ask pre-purchase questions on a product, and are helped by your sales agents.
A while later they come back after purchasing something to ask customer service questions, answered by an automated FAQ.

So is the livechat a sales channel or a customer service channel? Both in this case. But if it is both then you need to track the value metrics for both: The revenue generated through the answering of sales related questions, and the cost savings through the automated FAQs.

For value metrics thus keep it simple: Track how much revenue is generated if the conversational experience is sales related.
If the experience is customer service related, track how many costs are saved, for example by “fully automated conversations” or “questions deflected”.

Next steps: Storing and visualizing data

Now that you know what you want to track and have chosen your standard metrics & your value metrics, it is time to make working with them a reality.
First you need to make sure that you have a data pipeline up and running, for example a conversational Data Lake. Read our guide on conversational data lakes to learn more about how we solved this particular problem.

Once you have the data and it is readily available you need to find an easy way to visualize it: A conversational dashboard is needed.

Continue your path to conversational analytics mastery by learning how to build your conversational dashboard.