Event extraction with imitation learning and cross-media inference on streaming data

Abstract

Another challenge originates from lack of a mechanism that dynamically assesses wrong and confusing labels, and most conventional approaches treat all instances (and especially errors) equally but usually neglect various extents of difficulty.

In this thesis, we introduce several approaches that improve event extraction. We start from a cross-media integration approach that boosts the performance of the textual task with visual features, and we also demonstrate how visual information improves its down-streaming application -- event coreference. We then dive into imitation learning which dynamically assesses the difficulties among instances in event extraction task. Finally, We propose a framework based on memory network that tackles a scenario where annotation arrives in a stream with different annotation quality.

The last but not the least, availability and quality of training data or annotation also impact the performance. In some practical scenarios, annotated data arrive in small-scale batches -- stream -- so that developers and researchers are able to get familiar with the data and start development of event extraction system as early as possible. However, batches released in different time phases may yield to different qualities; and most traditional learning approaches and strategies do not possess the capability of handling streamed annotation.

One major challenge resides in modality exploitation. Most raw data consist of multiple modalities: texts, images, and videos. However, traditional extraction methods focus on a single modality, and most of them are not able to comprehensively take advantage of information from other modalities that they do not handle.

Extracting structured data from a large and growing amount of unstructured data is a challenging task.