Team Proto earns bronze in Amazon’s Alexa Socialbot competition

By Nicole Capozziello

Under the direction of computer science and engineering professor Rohini Srihari, Team Proto, made up of four PhD students, entered the Alexa Prize Socialbot Grand Challenge 4, sponsored by Amazon.

Their task was to create a socialbot (think Alexa or Siri) that could converse coherently and engagingly for 20 minutes with users on a range of topics, from politics to fashion to technology. Team Proto was one of nine teams from around the world competing not only for a substantial grand prize but to use AI to make a real impact.

After months of building their design, the team’s bot went live in January 2021. Every day, it interacted with over 7,000 people who conversed with the bot at length, and provided valuable feedback after the experience. The process was an iterative one as the team constantly tweaked their bot in response to user interaction.

On July 1, Team Proto received the exciting news that they had qualified as one of five to advance to the finals stage of the competition, which took place from July 27-29. 2021.

In August, the winners were announced. Team Proto, comprised of Sougata Saha and Souvik Das, PhD students in the Department of Computer Science and Engineering, and Elizabeth Soper and Erin Pacquetet, PhD students in the Department of Linguistics, won bronze, which includes a $50,000 prize.

“I am especially proud of our team Proto and the incredible effort they put into this competition.  We were the smallest team and were participating for the first time,” says Srihari. “Given this and the stiff competition, we are more than thrilled with our performance!”

The gold went to Czech Technical University (CTU) of Prague and the silver was awarded to Stanford, both who have participated in the competition with strong showings in the past.

The winning team from CTU had an average user rating of 3.28 (out of 5) and an average conversation duration of 14 minutes and 14 seconds. Team Proto earned an average user rating of 3.16, with an average interaction duration of 14 minutes and 45 seconds. Each participating team also published a research paper on their approach to this year’s competition.

“If we can figure out how to develop socialbots that can conduct purposeful, personalized, empathetic conversations with users, it opens up so many applications, ranging from personal health coaching to combating disinformation,” says Srihari. “There are still many research challenges before this becomes a reality, but we are already using socialbots to help people with amnesia gain confidence in participating in conversations.”

The competition provided the students with an opportunity to learn about the current state of research on AI and socialbots, or chatbots. “People can interact with a socialbot in endless ways. However, identifying each of the possibilities is definitely taxing. A more holistic approach is needed to understand how people think and make decisions during a conversation,” says Souvik Das, whose doctoral thesis will focus on ways of achieving this.

The hands-on competition, in which Team Proto’s bot ultimately had 262,364 recorded conversations between January 1 and August 1, 2021, also challenged the graduate researchers’ perspectives and training.

“It was eye-opening to realize that the way we think about conversations as researchers and linguists is different from the real-world data we obtained every day,” says Erin Pacquetet. “We were forced to re-evaluate how we think about language. In order to build meaningful conversations, we had to come up with clever ways to bridge the gap between our theoretical understanding of dialogue and the real-life expectations of users when conversing with our socialbot.”

For students, a highlight of the competition was working on such an intimate team. The steep learning curve, as well as undertaking such a large project as a small team, made for an experience that was both intense and supportive. “This experience of working on team was different than any I’d had before,” says Das. “The way we operated was like a group of friends working towards building a start-up.”

Pacquetet adds, “Because we are an interdisciplinary team, we have different ways of tackling issues and we learned a lot from each other along the way.” The team was made up of students from the College of Arts and Sciences and the School of Engineering and Applied Sciences.

Team Proto also really enjoyed the experience of getting to interact with and learn from Alexa users across the country. The constant feedback helped to make their socialbot more responsive. “It was exciting to see the conversations that people have with our bot; a lot of times they interacted with it in ways we didn’t expect,” says Liz Soper. “Having so much regular feedback really allowed us to create a better socialbot than we could have if we were the only ones testing our own system.”

Srihari plans to enter the competition with another team, including some of the same students, this year. And this time, the team will come in with a base of experience. “As first-time participants last year, we had to build a social bot from scratch and do a lot of groundwork,” Pacquetet reflects. “Now Proto is a fully-fledged and functioning bot. Running again will allow us to spend more time coming up with creative solutions to conversational AI problems.”

Among other improvements and innovations, Srihari says the team plans to incorporate higher level models of conversational structure that reflect human behavior, as well as more nuanced dialogue generation taking into account factors such as user personality.

This semester, Srihari is offering a seminar in the computer science and engineering department that focuses on the use of socialbots for positive impacts, including combating disinformation, which is the topic of a recent National Science Foundation grant on which Srihari is a co-principal investigator. Students in the course collaborate with an NGO in a developing country looking to use socialbots to train volunteers and collect health-related data from citizens in underserved areas, many who are illiterate. And, of course, Srihari will also be introducing students to the Alexa Socialbot Grand Challenge.

Amazon Science says that since the competition’s launch in 2017, Alexa customers have spent more than 900,000 hours engaging with Alexa Prize socialbots. Alexa customers can continue to engage with the winning teams’ socialbots simply by saying, “Alexa, let’s chat.”

Proto team leader Sougata Saha says, “I love the feeling that at any given point, someone somewhere in the United States is interacting with an AI that we created, and witnessing an experience that we designed.”

Learn more about the journey of the cross-disciplinary team, a collaboration between students from University at Buffalo’s College of Arts and Sciences and the School of Engineering and Applied Sciences.