Intent Detection is a vital part of the Natural Language Understanding (NLU) pipeline of  Task-oriented dialogue systems. Recent advances in NLP have enabled systems that perform quite well on existing intent detection benchmarking datasets like HWU64, CLINC150, BANKING77 as shown in Larson et al., 2019, Casanueva et al., 2020. However, most existing datasets for intent detection are generated using crowdsourcing services. This difference in dataset preparation methodology leads to assumptions about training data which are no longer valid in the real world. In the real world, definition of intent often varies across users, tasks and domains.

Typically, there are 5 levels of Conversational AI and Level 3 is understood as contextual assistants, where the user no longer needs to know how to use the assistant or follow the pre-designed conversational flow.

We recently open-sourced a repository called Multi-Task learning which can help with multiple conversational AI tasks. In this blog, we have explained why such an architecture can change the way we build conversational AI systems and the thought process behind building it.