Building cognitive and immersive systems: architecture, implementation, and formalization

Abstract

To address these concerns, in this dissertation we introduce aframework for building what we term \textit{cognitive and immersive systems} (CAIS). Within our approach, and to handle the demands above, we emphasize techniques that are well-formalized and that operate at a ``theory of mind'' level. To accomplish this, we look to underpin our system with a formal language that has a high level of expressivity. For this, we turn to the \textit{cognitive event calculus} (CEC), a multi-operator multi-sorted quantified modal logic, and a matching high-expressivity automated reasoner and planner. Stemming from this we can establish a formalized set of principles by which these systems operate; and these systems can operate within scenarios that require not only reasoning about the physical world, but also the cognitive states of our agents within the CAIS. To validate our approach and demonstrate its effectiveness, we provide a real-world open-source implementation of a CAIS, and utilize it on tasks of planning and plan recognition.

As computational power has continued to increase, and high fidelity sensors andlarge-scale displays have become commonplace, so too has the desire for systems that can better fuse artificial intelligence and human-computer interaction. On the AI side of things, there is a demand for artificial intelligent agents to operate at a cognitive level that permits them to reason over an agent's beliefs, knowledge, communications, goals, etc. On the HCI side, these systems must be multi-modal, able to combine speech, gestures, and various interfaces to allow a diverse range of interactions to these systems. Within this fusion, users may develop reasonable and productive expectations regarding the capabilities of these systems, and these capabilities can then be applied to a diverse range of domains and content, in service of said users.