12/15/2017 18:51:39

The Conversational Intelligence Challenge Finals

On Dec. 8 the Conference on Neural Information Processing Systems (NIPS) hosted the finals of Conversational Intelligence Challenge. The challenge was organized within the collaboration of AI research universities.

Conversational artificial intelligence (AI) is a task of having meaningful dialogue with a user and acting as human as possible. This challenging and complex problem is considered to be one of the core tasks that true AI should pass. Dialogue agents — programmes capable of having a conversation — surround us everywhere: From bots that we chat with on messengers to personal assistants and voice control interfaces. While dialogue systems are becoming more widespread, there is still a great deal of work involved in making conversational intelligence more sophisticated.

To create dialogue systems, we need two things: More data to train on (examples of conversations), and a reliable and fast ways to evaluate the quality of these dialogues. The competition gathered chatbots developed to define which techniques would make a chatbot appear intelligent. The competition also aimed at working out approaches for chatbots evaluation and at human-to-machine dialogues collection for further end-to-end dialog systems training.

During the competition, chatbots had to talk with users, who then assessed these conversations. The competition was held in two stages: At first round, organized by the Neural Networks and Deep Learning Lab this summer, the participants of the hackathon talked to the systems. The final stage took place at the NIPS conference, where the scientific community tested chatbots. Reflecting the wider AI boom, over the past few years the NIPS conference has grown from a small gathering of a few hundred academics to a sprawling event with thousands of attendees, big-name corporate recruiters, and lavish parties. During the Challenge, the bots conducted about 3500 dialogues, about 400 of which were judged by users as meaningful. The dialogues will be further examined to find out what factors involved in maintaining the meaningful conversation and how we could measure its quality. This knowledge will help to improve tools for creating dialogue systems.

Six teams from the University of Wroclaw, Moscow Institute of Physics and Technology, McGill University, KAIST & AIBrain & Crosscert, UMass Lowell’s Text Machine Lab & Trinity College, Hong Kong Polytechnic University, and Fudan University took part in the competition finals.

The first prize was awarded to 2 winning teams: Moscow Institute of Physics and Technology team (Idris Yusupov, Yurii Kuratov) and  University of Wroclaw team (Jan Chorowski, Adrian Lancucki, Szymon Malik, Maciej Pawlikowski, Pawel Rychlikowski, Pawel Zykowski).

Both teams used similar approaches: they developed modules that processed different scenarios (having a small talk, answering factoid questions asked by a user, engaging a user into talk about himself/herself, processing unrecognized input from a user), and a command module that decides which routine should step in. This is now considered a state-of-the-art technique for chatbot development. Even though these chatbots do not contain any novel methods (i.e. methods not published anywhere before), they represent good proofs of concept for existing approaches. Moreover, these solutions will be open-sourced and can now serve as a starting point for further research in dialogue systems.

The Conversational Intelligence Challenge attracted the interest of frontiers in conversational AI - Google AI, Facebook research, Amazon, Microsoft, and will be held annually.

The competition was organized by Moscow Institute of Physics and Technology, Université de Montréal, McGill University, and Carnegie Mellon University in partnership with Facebook Research, Flint Capital, IVADO, Maluuba, and Element AI.


Mikhail Burtsev, Valentin Malykh, Varvara Logacheva, Neural Networks and Deep Learning Lab, MIPT

Iulian Serban, Yoshua Bengio, Montreal Institute for Learning Algorithms, Université de Montréal

Ryan Lowe, Reasoning and Learning Lab, School of Computer Science, McGill University

Alexander Rudnicky, Alan W. Black, Shrimai Prabhumoye, School of Computer Science, Carnegie Mellon University


  • Flint Capital is an international venture capital fund established by IT entrepreneurs with the primary mission of discovering and supporting bold-thinking founders and their teams from Europe, Israel, and the US across a number of industries. Flint capital leads funding rounds at the Seed, Series A, and Series B stages, focusing on companies that have the potential to disrupt their respective industries. Flint Capital has an extensive expertise in artificial intelligence and machine learning. Previously Flint Capital has supported the following AI/ML companies: WalkMe, YouAPPI, CyberX, Epistema, Flo, Findo. Flint Capital has had several significant exits amongst portfolio companies, including Lending Club and Blazemeter, with current investments in dozens of industry-leading companies, including WalkMe, CyberX, Virool, and Findo. Flint Capital has offices in Palo Alto, Berlin, Boston, Moscow and Tel Aviv.
  • Facebook Research
  • Maluuba
  • Element AI

If you have noticed a mistake on this page, select it and press Ctrl + Enter

Related posts

The dispute about the origins of terahertz photoresponse in graphene results in a draw

Electromagnetic wizardry: Wireless power transfer enhanced by backward signal
MIPT delivers world’s first biosensor chips based on copper and graphene oxide
Scientists discover a link between superconductivity and the periodic table