Harnessing AI to help kids speak

Venu Govindaraju is leading a five-year, $20 million project to address the nationwide shortage of speech-language pathologists and provide services to children ages 3 to 10 who are at increased risk of falling behind in their academic and socio-emotional development —issues exacerbated by the COVID-19 pandemic.

The highly competitive National Science Foundation grant was awarded to the UB team earlier this year to establish a national institute that develops artificial intelligence systems that identify and assist young children with speech and/or language processing challenges.

The award, which will be used to create the National AI Institute for Exceptional Education at UB, also will advance foundational AI technologies, human-centered AI design, and learning science that improve educational outcomes for young children. It is one of the largest federal research grants received by UB.

“There simply aren’t enough speech-language pathologists in the United States and, as a result, children are not receiving life-changing interventions soon enough,” said Govindaraju. “Our multidisciplinary team will create advanced AI systems that address this critical problem, allowing for earlier diagnoses and tailored interventions that close educational gaps and create more inclusive learning environments where children thrive both academically and socially.”

Govindaraju, SUNY Distinguished Professor in the Department of Computer Science and Engineering (CSE) and UB’s vice president for research and economic development, is the principal investigator. Jinjun Xiong, CSE Empire Innovation Professor, is a co-principal investigator and will serve as scientific director; and Srirangaraj Setlur, principal research scientist in CSE, will serve as the managing director. Letitia Thomas, assistant dean for diversity in the School of Engineering and Applied Sciences, will lead the broadening participation, and diversity, equity and inclusion sections of the grant.

The institute also includes over 30 researchers from nine universities including UB; the University of Illinois Urbana-Champaign; Stanford University; the University of Washington; Cornell University; University of Nevada, Reno; University of Texas at El Paso; Penn State University; and University of Oregon. They specialize in AI, natural language processing, social robotics, communicative disorders, diversity and inclusivity, learning science, communication and other fields.

The National AI Institute for Exceptional Education will focus on serving the millions of children nationwide who, under the Individuals with Disabilities Education Act, are legally entitled to speech and language services.

Specifically, it will develop two advanced AI solutions: the AI Screener for early identification of potential speech and/or language impairments and disorders; and the AI Orchestrator, which will act as a virtual teaching assistant by providing students with ability-based interventions.

The AI Screener will listen to and observe children in the classroom, collecting samples of children’s speech, facial expressions, gestures and other data. It will create weekly summaries of these interactions that catalogue each child’s vocabulary, pronunciation, video snippets and more. These summaries will help teachers monitor their students’ speech and language processing abilities and, if needed, suggest a formal evaluation with a speech-language pathologist.

The AI Orchestrator is an app that will help speech-language pathologists, most of whom have caseloads so large that they are forced to provide group-based interventions instead of individualized care for children. The app addresses this by recommending personalized content tailored to students’ needs. It continues to monitor their progress and adjusts lesson plans to ensure that the interventions are working.

Initially, the team intends to deploy prototypes of each system to roughly 80 classrooms, reaching 480 kindergartners.