Dynamic data modeling, recognition, and synthesis

Abstract

First, the dynamic pattern of interest is often contaminated with irrelevant contents such that the exact location and duration of the patterns of interest are unknown. To address this challenge, we propose a method that incorporates the hidden Markov model (HMM) and the random sample consensus (RANSAC) method for the robust identification and segmentation of relevant signals from time series data. We demonstrate the effectiveness of this approach on electrocorticographic (ECoG) data in brain-computer interface applications, where the identification method helps to improve the classification accuracy of the dynamic model.

Finally, we address the challenge of modeling complex dynamics emerging in dynamic data. The source of complexity is two-fold. First, there exist complex interdependencies among different components that generate the dynamic data. Second, there exist long-term temporal dependencies between data samples observed at different times. We propose a Bayesian neural networks (BNN) model that integrates Bayesian modeling with neural networks (NN) to leverage the benefits of both probabilistic and deterministic approaches. Specifically, we use graph convolution to capture the structural dependencies among different components of the dynamic data, whose temporal dynamics are further modeled by recurrent neural networks (RNN) with long short-term memory (LSTM). The entire model is extended to a probabilistic model to better handle the randomness in the dynamic data. A Bayesian inference framework is formulated to perform a classification task, and an adversarial prior is developed to further improve the generalization of the model. We demonstrate the effectiveness of the proposed framework on skeleton-based human action recognition.

Second, in dynamic data analysis, we are often faced with insufficient amount of labels due to a lack of resources for the annotations. To address this challenge, we propose a method called ordinal support vector regression (OSVR) that exploits the ordinal relationship embedded in time series data. By incorporating ordinal information as additional constraints, OSVR can be learned with partially labeled data. We demonstrate the advantage of our method in the application of facial expression intensity estimation.

Dynamic data modeling is the key to many dynamic data analysis tasks, such as prediction, classification, regression, and synthesis. It provides an automatic tool to characterize and reveal the underlying dynamic process that generates the data. Applications of dynamic data modeling are widespread in various fields of science and engineering. Despite its importance, modeling dynamic data is challenging due to the complexity and randomness of the underlying dynamic process. In this research, we introduce advanced methods to overcome some of the challenges and demonstrate their effectiveness in real-world applications.

Third, we address the challenge of dynamic data analysis under significant intra-class variation. Intra-class variation of dynamic patterns manifests itself in both space and time, occurring in many dynamic data analysis problems. For example, the same human action (e.g., bowling) may vary spatially in terms of body pose as well as temporally in terms of speed, duration, and transition. To address this challenge, we present a probabilistic hierarchical dynamic model (HDM) which handles variation at two levels. At the first level, a hidden semi-Markov model (HSMM) is used to capture the spatio-temporal variation in the data. At the second level, the model parameters are treated as random variables that are governed by prior distributions to further increase the capacity of modeling intra-class variation. We also develop different learning and inference methods to better handle different applications, including recognition and synthesis. We demonstrate the effectiveness of our methods on benchmark human action datasets.