Deep Neural Networks for Detection of Rare Events, Novelties, and Data Augmentation in Multimodal Data Streams

Alina Vasylivna Nesen, Purdue University

Abstract

The abundance of heterogeneous data produced and collected each day via multimodal sources may contain hidden events of interest, but in order to extract them the streams of data need to be analyzed with appropriate algorithms, so these events are presented to the end user at the right moment and at the right time. This dissertation proposes a series of algorithms that shape a comprehensive framework for situational knowledge on demand to address this problem. The framework consists of several modules and approaches, each of them is presented in a separate chapter: I begin with video data analysis in streaming video and video at rest for enhanced object detection of real-life surveillance video. For detecting the rare events of interest, I develop a semantic video analysis algorithm which uses an overlay knowledge graph and a semantical network. I show that the usage of the external knowledge for understanding the semantic analysis outperforms other techniques such as transfer learning.The semantical outliers can be used further for improving the algorithm of detecting new objects in the stream of different modalities. I extend the framework with additional modules for natural language data and apply the extended version of the semantic analysis algorithm to define the events of interest from multimodal streaming data. I present a way of combining several feature extractors which can be extended to multiple heterogeneous streams of data in order to efficiently fuse the data based on its semantical similarity, and then show how the serverless architecture of the framework outperforms conventional cloud software architecture.Besides detecting the rare and semantically incompatible events, the semantic analysis can be used for improving the neural networks performance with the data augmentation. The algorithm presented for augmenting the data with the potentially novel objects to circumvent the data drift problem uses the knowledge graph and generative adversarial networks to present the objects to augment the training datasets for supervised learning. I extend the presented framework with a pipeline for generating synthetic novelties to improve the performance of feature extractors and provide the empirical evaluation of the developed method.

Degree

Ph.D.

Advisors

Bhargava, Purdue University.

Subject Area

Artificial intelligence|Aging|Logic|Medicine|Neurosciences|Transportation|Internet and social media studies

COinS