Learning Based Image Analysis with Application in Dietary Assessment and Evaluation

Yu Wang, Purdue University

Abstract

Mobile devices will transform the healthcare industry by increasing accessibility to quality care and wellness management. Accurate methods to assess food and nutrient intake are essential. We have developed a dietary assessment system, known as the mobile Food Record (mFR) to automatically estimate food type, nutrients and energy from a food image captured by a mobile device. Color information is of great importance in our mFR system and it serves as a key feature to identify foods. Thus, a preprocessing step including color correction and image deblurring is necessary to ensure that we can utilize the image for the further analysis. We present an image quality enhancement technique combining saliency based image deblurring and color correction using LMS color space. The accurate estimate of nutrients is essentially dependent on the correctly labelled food items and sufficiently well-segmented regions. Since food recognition also largely relies on the interest region detection or segmentation, image segmentation plays a critical role in our mFR system. We propose a generic segmentation method that combines normalized cut and superpixels. Experimental results suggest that the proposed method using multiple simple features is effective for food segmentation. To achieve high classification accuracy in food images is challenging due to large number of food categories, lighting and pose variations, background noise and occlusion. Deep learning with big data has shown its dominance in various object detection tasks. In this thesis, we compare deep features with the handcrafted features in terms of classification performance and we also introduce a weakly supervised segmentation method based on class activation maps using only the label of the input image to deal with sparsity of ground-truth masks or bounding boxes. Furthermore, a 3-stage food localization and identification technique using end-to-end deep networks is proposed. Finally, we integrate contextual information into our mFR system and introduce the personalized learning model to further improve the food recognition accuracy. The result indicates that our contextual models are promising and further investigation is warranted.

Degree

Ph.D.

Advisors

Zhu, Purdue University.

Subject Area

Computer Engineering|Engineering|Electrical engineering

COinS