Modeling Cortical Visual Processing With Recurrent Neural Network

Junxing Shi, Purdue University

Abstract

Human vision is enabled by a cascade of visual processes in the brain. On the other hand, deep neural networks have also enabled computer vision to recognize images to a human-like level. Bearing on this similarity, it has recently attracted interests in using deep neural networks for modeling the human visual cortex with fMRI imaging. Studies have shown convolutional neural networks (CNNs) predicted the cortical activity during subjects viewing natural images. Besides, CNNs revealed the spatial hierarchy of visual processing in the cortex. Yet, neither the brain nor our visual world is static. Therefore, we acquired fMRI data during subjects watching natural movies. We used both a CNN, and a recurrent neural network (RNN) extended upon the CNN, for modeling and understanding the workings of the visual cortex during natural vision. Using neural encoding as the common metric, we found that these models predicted widespread cortical activity. Moreover, the RNN, exploiting temporal structures, was superior to the CNN in a number of aspects. It not only predicted more about the cortical activity and mapped the spatial hierarchy, but also reported the temporal hierarchy in the visual processing for the first time. Our results cast light on the future studies of systems neuroscience using computational models.

Degree

M.S.E.C.E.

Advisors

Liu, Purdue University.

Subject Area

Neurosciences

COinS