Incorporating nonverbal features into multimodal models of human-to-human communication

Lei Chen, Purdue University

Abstract

Nonverbal communication (e.g., eye gaze and hand gesture) plays an important role in human conversations, including providing semantic content, expressing emotional status, and regulating conversation turns. However, most analyses of conversations largely focus on spoken content and not important nonverbal cues. In this thesis, we investigated the use of nonverbal cues for enhancing the analysis of conversations. Particularly, we considered three types of conversational structures, speech repairs, sentence units, and floor control shifts, which are important for understanding conversations. To support the research on these three structures, we focused on: (1) collecting multimodal data resources involving nonverbal cues and structural events in human conversations, (2) analyzing the collected data sets to enrich our knowledge about nonverbal cues and the structural events investigated, and (3) building statistical models for detecting structural events using verbal and nonverbal cues. Through collaborations with researchers in psychology and computer vision, we collected the KDI multimodal dialog corpus and the VACE multimodal meeting corpus. Using the KDI data, we analyzed gesture patterns that occur during speech repairs. Using the VACE data, we analyzed gesture patterns for signaling the presence of SUs and gaze and gesture patterns for signaling floor control shifts. Using the VACE data, we then investigated combining gestural features with lexical and prosodic features for detecting SUs and combining gestural and gaze features for detecting floor control shifts. In this thesis, the impact of nonverbal cues was investigated for analysis and detection of structural events. The development of data resources and tools for this type of research is an important contribution. Our data analyses have enriched the knowledge about the relationship between nonverbal communication and structural events. Our statistical modeling research has demonstrated the usefulness of nonverbal cues for conversation analysis. Research in this thesis suggests that nonverbal communication provides useful cues for the analysis and detection of structural events in human conversations. Our results support the view that human conversations are processes involving multimodal cues, and so are more effectively analyzed using information from both verbal and nonverbal channels.

Degree

Ph.D.

Advisors

Harper, Purdue University.

Subject Area

Electrical engineering

COinS