Cross-modal instance grounding for intelligent agent systems

Abstract

Two modules for the OntoAgent cognitive architecture are presented. The first is a visual analyzer, which accomplishes a high-level semantic understanding of the output of computer vision or a simulation thereof. The design of the visual analyzer closely parallels the design of OntoAgent's existing textual analyzer. It infers event instances by comparing snapshots of its environment and using a new knowledge resource known as a "visual lexicon" to determine the meaning of differences it sees. The second module is a reasoner that synthesizes input from the visual analyzer and the already-existing textual analyzer using a variety of knowledge-based heuristics in order to determine coreference between object or event instances from the two modalities.