Context augmented event and object recognition

Abstract

Secondly, we observe that the contextual information for event recognition generally exists at the feature level, the semantic level, or the prior level. The existing context-based approaches typically employ the contextual information at only one of the three levels. To address this deficiency, we build the hierarchical context model to comprehensively utilize contexts from all the three levels, and simultaneously integrate the contexts for event recognition. We introduce advanced learning and inference methods to learn the parameters of the hierarchical context model, and directly infer the event class through the proposed model. The hierarchical context model can effectively integrate the feature, semantic and prior level contexts to significantly improve event recognition over the state-of-the-art methods.

In summary, by incorporating different types of contextual information through the proposed models, we can effectively mitigate the challenges associated with recognizing real-world events and objects, and significantly improve the recognition performances.

Finally, we utilize the object taxonomy as the prior level context to augment the learning of object classifiers. The object taxonomy represents the hierarchical structural of object categories and super-categories. We utilize two types of taxonomy relationships including the overall relationships and the local relationships. These two types of relationships are captured as the loss function and extra constraints respectively to regularize the object classifier learning. Experiments on the benchmark datasets demonstrate that the classifier learned with the taxonomy can outperform the baseline classifier learned without using taxonomy for object recognition, and state-of-the-art methods that also exploit the object taxonomy.

Thirdly, attribute-based object recognition has become an important topic in computer vision. The existing work, however, treats each object attribute independently and separately, ignoring their semantic relationships. As an important type of context information, semantic relationships among attributes widely exist among different attributes and objects. However, such contextual information is not utilized by the conventional attribute-based object recognition approaches. In this research, we propose to exploit and capture the semantic relationships among attributes and objects for both attribute prediction and object category recognition. Specifically, we introduce a unified probabilistic graphical model to automatically discover and capture both the object-dependent and object-independent attribute relationships. The model leverages the relationships among attributes to improve both attribute prediction and object recognition.

Current computer vision research remains largely target-centered. For many real-world applications, target-centered visual recognition faces great challenges due to large intra-class variation, limited image resolution, and significant variation in background and illumination. Effectively exploiting contexts can alleviate these challenges. In computer vision, context is the extra information that is relevant but is not directly applied to the vision task. In this research, we propose to improve visual recognition by systematically extracting and encoding different types of contextual information.

Firstly, we propose a context model based on the Dynamic Bayesian Networks (DBNs) to incorporate scene, event-object interaction, and event temporal contexts into human event recognition in surveillance videos. We construct the target-centered baseline event DBN models to capture event appearance and dynamics, and then introduce a context model to augment the baseline event models with various contexts. Experiments on benchmark datasets show that event recognition performance can be significantly improved by incorporating the contexts.

Fourthly, unlike traditional approaches for object recognition that treat attributes as a middle level representation and require the estimation of attributes during testing, we further propose to incorporate attributes as hidden context to improve object recognition. To achieve this goal, we develop two different approaches to incorporate attributes, with one approach utilizing attributes as additional features and the other utilizing the relationships between attributes and objects. Both approaches can effectively improve the learning of the object classifiers, and a combination of the two yields the best performance.