Students at Rensselaer Polytechnic Institute will feel as if they are learning Chinese on the streets of Beijing this summer when an intelligent, immersive environment developed by Rensselaer and IBM Research is used in an academic credit-bearing course on campus for the first time.

“AI-Assisted Immersive Chinese” is a first-of-its-kind, six-week course in Mandarin Chinese that Rensselaer is offering to students this summer. It serves as the academic debut of a new generation of classrooms that combine artificial intelligence (AI) and other cutting-edge technologies to provide an experiential language-learning experience akin to studying abroad -- with no travel required.

A 360-degree panoramic display system creates the sense that users are in the middle of various computer-generated scenes, such as a restaurant or a street market, that are similar to those they might encounter in China. A combination of the immersive environment’s natural language understanding, narrative generation, and gesture recognition technology capabilities enable students to interact naturally with characters in these scenes. The room is able to engage in dialogue with students and offer immediate feedback on their non-native Chinese speech and pronunciation – and even critique their Tai Chi.

The scenarios used in the class were developed by the Mandarin Project, which is an initiative of the Cognitive and Immersive Systems Laboratory (CISL), an ongoing collaboration between Rensselaer and IBM Research.

It is no coincidence that this course is being offered during the inaugural summer of the new ARCH program at Rensselaer, which reimagines the academic calendar to provide additional opportunities for students while maximizing use of university resources. Beginning in 2019, all rising juniors will remain on campus in the summer and then participate in an away experience during their junior year.

“At Rensselaer, we are transforming education and redefining the global understanding of what a learning environment can be. Pioneering the use of intelligent immersive classrooms is a significant part of that effort,” said Rensselaer President Shirley Ann Jackson. “The Mandarin Project is a demonstration of the interdisciplinary problem-solving and collaboration with industry that are hallmarks of our vision for The New Polytechnic.”

No prior experience in Mandarin Chinese is required for the summer course, which is open to 25 Rensselaer students. Over the course of six weeks, students will spend half their time in the immersive environment, which is generated in a multipurpose studio in the Curtis R. Priem Experimental Media and Performing Arts Center on the Rensselaer campus. Inside the immersive room, they will explore wholly computer generated scenes and enhanced street scenes, interacting with AI-enabled characters who will help them practice conversational skills,  improve their vocabulary, pronunciation, and cultural knowledge.

“The immersive environment is revolutionary,” said course instructor Helen Zhou, a Rensselaer associate professor of communication and media who worked closely with CISL to develop the program. “The language context we designed is based on real life scenes. These interactions give students a sense of how communication will play out in real world scenarios, and the system gives them immediate feedback on how well they did.” 

CISL was launched in 2015, and it immediately began developing its concept for an intelligent immersive environment. After one year, the team completed a prototype computational architecture that seamlessly linked input from sensors, microphones and cameras with multiple AI technologies developed by Rensselaer and IBM Research. CISL has continued to improve and expand the architecture by introducing more interaction capabilities -- such as eye-gaze tracking, face landmark detection, interactive visualization -- and creating new AI technologies that enable multimodal multiparty conversation, goal-oriented dialogue, and reasoning and planning.

“The goal of CISL is to create intelligent, human-scale interactive spaces and experiences that combine AI and immersive technologies to help improve the way people learn and collaborate,” said Hui Su, the director of CISL, IBM Research.

The Mandarin Project originated in 2012 as a Rensselaer-led effort to combine augmented reality and game design with education. When it was incorporated into CISL, the project became a more ambitious effort to explore how the latest technologies can be used to innovate the teaching of foreign languages. In addition to education, CISL envisions these immersive rooms will have applications in many other fields, including business, medicine, and emergency response.