Web semantics for sensory augmentation

Abstract

Such a system would need to respect the conditions under which the integration of cues will occur, which is a real-time problem as well as a problem of machine perception and semantic reasoning or querying. The applicable thresholds of perceptual synchrony are comparable to, and in many cases exceeded by, the delay of the simplest real-world sensor networks we might like to build, and semantic processing time is also a factor. The results of this dissertation are that the technical problem can in fact be solved, although any just-in-time operations are tightly constrained by the sensory modalities of the mapping, and by geography and network topology. A concrete architecture is provided as a proof of concept, along with a number of applications using cooperative event detection and wearable computers, semantic queries, and dynamic views of Web data for context awareness. A novel semantic query processor is introduced in order to answer complex queries continuously and within the real-time bounds over an unbounded stream of events. An evaluation of the system indicates that applications would scale up to handle infrequent events for very many users, or continuous events for a single user at a frequency high enough to enable recalibration of perceptual synchrony.

This dissertation proposes to apply semantic technology to augment perceived objects and events with new features. Sensory substitution and augmentation systems typically map signals continuously from a sensory input space to a sensory output space with little or no intermediate feature extraction or semantic processing. However, there is evidence that recognition is facilitated by semantics such as categorical relationships, just as it is in everyday perception. It is worthwhile, then, to ask whether a sensory mapping system can directly target perceptual semantics, enabling a preconscious awareness of a feature through semantically mediated correspondences rather than statistical or synesthetic ones. If so, it might even be possible to extend the user's awareness to features not available in any one sensory modality or from any single point of view, but to derive them from some combination of sensors and background knowledge.