AI enabled foreign language immersion : technology and method to acquire foreign languages with AI in immersive virtual worlds

Abstract

Students who learn foreign languages have limited exposure to conversations in the target language as, by definition, a foreign language is the one that is not commonly spoken where they live. However, exposure to the target language is key to acquiring it. This gap is commonly filled by international foreign language immersion programs. International foreign language immersion programs enable a student to move temporarily to a foreign country for several months where opportunities to interact in the target language are ample. However, the overhead of moving to a foreign country makes foreign language education less accessible. This dissertation explores how extensions to state-of-the-art Artificial Intelligence (AI) integrated with Extended Reality (XR) can bring experiences like the ones found in international immersion programs to a student and help them acquire a foreign language; without requiring them to travel extensively.

Specifically, we articulate a pedagogical framework that involves learning a foreign language with AI in XR. We show how AI can be extended from its state-of-the-art and integrated with XR technology to enable the said pedagogical framework. The learning methodology and technology are novel and resulting interactions are unique in that they immerse students without requiring intrusive wearables thereby allowing long-term collaborative experiences in a virtual world where students can communicate multi-modally with several AI Agents and each other. We test the novel interactions to observe users' characteristics in Human-AI conversational interactions and solve key challenges. The resulting novel interactions with AI in XR environments contributed by this dissertation are shown to improve foreign language learners’ proficiency across vocabulary, listening, comprehension and conversation.

Extended Reality (XR) and Artificial Intelligence (AI) are two technologies that can bring these theories to life. Specifically, recent developments in XR make it possible to enable physical immersion by creating a sense of suspended disbelief. Whereas, social immersion can be brought by embodied AI Agents situated in the visual scenes and can hear, see and speak to students. We show that by bringing AI and XR together, we can create an experience that provides authentic conversational opportunities like the ones found in international immersion programs and thus improve students' target language proficiency.

To bring this experience to the students, we rely on two key theories --- Physical immersion and social immersion. Physical immersion lets the students feel as if they are somewhere else while social immersion gives them a context to practice their conversational skills.