Intelligence Satisfaction Score (ISAT)

 

An industry-first framework to measure the effectiveness of an intelligent virtual assistant

banner
banner

Limitations in the Current Measurement Frameworks

csat-icon
CSAT & NPS

Only 2% of users share feedback about their experience with a virtual assistant

automation-icon
Automation

Useful but does not say anything about queries resolved, false positives, confidence threshold, etc.

 
engagement-rate-icon
Engagement Rate

Standard market metrics such as retention rate, engagement, avg time spent, etc. are not applicable for virtual assistants

sentiment-analysis-icon
Sentiment Analysis

Complex multi-turn conversations can have various sentiments that make it difficult to analyze

 

The ISAT Score Framework

Steps to Arrive at the ISAT for a given virtual assistant or chatbot

1
Categorize
Conversations
categorize-conv
calculate
2
Calculate ISATScore
3
Optimize
Conversation
Flows
bitmap

Classifying User Conversations

Analyzing Positive, Negative and Neutral ISAT Conversations

Positive
positive
  • User comes with a query
  • IVA responds appropriately and solves the query
  • Task completion
Negative
clarifying-user-conversations-negative
  • High user intent. User repeatedly asks the same query
  • Sends > 2 messages
  • IVA unable to comprehend repeats itself, leading to negative sentiments and unhappy users
Neutral
clarifying-user-conversations-neutral
  • Low user intent
  • User interacts with the IVA with no clear goal or objective in mind
  • User drops off randomly

Analyzing Negative ISAT Conversations

Conversations with a negative ISAT score are poor experiences that users undergo when the IVA fails to understand and resolve the user’s query. This model is validated by adding all conversations that had a user repetition (UR), bot repetition (BR), and cuss words (CW).

Score
Clients
  • Insurance
  • Finance
  • Healthcare
  • Govt
  • FMCG
ISAT
Bad Chats
  • 12.2%
  • 1.1%
  • 10.4%
  • 18.0%
  • 3.2%
Machine Analysis
UR BR CW Total Negative Conversations
6.0% 3.0% 2.0% 11.0%
1.0% 0.0% 0.0% 1.0%
4.0% 2.0% 2.0% 8.0%
14.0% 3.0% 3.0% 17.0%
1.0% 1.0% 1.0% 2.0%

Decoding False Positives in ISAT

When the IVA fails to recognize the real intent of the user, it sends a wrong response causing False positives, thereby affecting the user experience negatively.

decoding
decoding

Combusting Bot Breaks in ISAT

An IVA ‘breaks’ when it is unable to recognize the intent of the user. An IVA designer configures a response to be sent to the user in such a scenario to convey the context and scope of the IVA.

combusting-bot-breaks
combusting-bot-breaks
circle line

ISAT Case Study

For one of the world’s largest insurance companies, we used the ISAT as a benchmark to improve the quality of the IVA, thereby improving the overall NPS

month-1
month-6