AI enabled foreign language immersion : technology and method to acquire foreign languages with AI in immersive virtual worlds

Divekar, Rahul R.
Thumbnail Image
Other Contributors
Hendler, James A.
Su, Hui
Zhou, Yalun (Helen)
Kephart, Jeffrey O.
Fox, Peter A.
Issue Date
Computer science
Terms of Use
Attribution-NonCommercial-NoDerivs 3.0 United States
This electronic version is a licensed copy owned by Rensselaer Polytechnic Institute, Troy, NY. Copyright of original work retained by author.
Full Citation
Students who learn foreign languages have limited exposure to conversations in the target language as, by definition, a foreign language is the one that is not commonly spoken where they live. However, exposure to the target language is key to acquiring it. This gap is commonly filled by international foreign language immersion programs. International foreign language immersion programs enable a student to move temporarily to a foreign country for several months where opportunities to interact in the target language are ample. However, the overhead of moving to a foreign country makes foreign language education less accessible. This dissertation explores how extensions to state-of-the-art Artificial Intelligence (AI) integrated with Extended Reality (XR) can bring experiences like the ones found in international immersion programs to a student and help them acquire a foreign language; without requiring them to travel extensively.
Specifically, we articulate a pedagogical framework that involves learning a foreign language with AI in XR. We show how AI can be extended from its state-of-the-art and integrated with XR technology to enable the said pedagogical framework. The learning methodology and technology are novel and resulting interactions are unique in that they immerse students without requiring intrusive wearables thereby allowing long-term collaborative experiences in a virtual world where students can communicate multi-modally with several AI Agents and each other. We test the novel interactions to observe users' characteristics in Human-AI conversational interactions and solve key challenges. The resulting novel interactions with AI in XR environments contributed by this dissertation are shown to improve foreign language learners’ proficiency across vocabulary, listening, comprehension and conversation.
Extended Reality (XR) and Artificial Intelligence (AI) are two technologies that can bring these theories to life. Specifically, recent developments in XR make it possible to enable physical immersion by creating a sense of suspended disbelief. Whereas, social immersion can be brought by embodied AI Agents situated in the visual scenes and can hear, see and speak to students. We show that by bringing AI and XR together, we can create an experience that provides authentic conversational opportunities like the ones found in international immersion programs and thus improve students' target language proficiency.
To bring this experience to the students, we rely on two key theories --- Physical immersion and social immersion. Physical immersion lets the students feel as if they are somewhere else while social immersion gives them a context to practice their conversational skills.
August 2020
School of Science
Dept. of Computer Science
Rensselaer Polytechnic Institute, Troy, NY
Rensselaer Theses and Dissertations Online Collection
CC BY-NC-ND. Users may download and share copies with attribution in accordance with a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License. No commercial use or derivatives are permitted without the explicit approval of the author.