Optimal modality selection for multimodal human-machine systems using RIMAG
Abstract
When robots are expected to work side-by-side with humans, they should be capable of communicating through ways similar to those used in inter-personal communication. To this end, several natural communication channels for human-robot interaction (HRI) have been researched over the years such as gestures, speech, gaze, emotion recognition and brain-based control. It has also been shown that inter-human communication is intrinsically multimodal resulting in several multimodal systems designed for HRI. These interfaces allow humans to communicate with robots through lexicons i.e. a vocabulary of instances of modalities assigned to specific control commands. The process of determining the instances of modalities to be used for certain tasks, and assigning them to specific commands (i.e. building the lexicon) is an open research question. Even though hybrid human-robot communication frameworks and multimodal communication have been studied, a systematic methodology for designing multimodal interfaces does not exist. This gap is addressed by the proposed RIMAG framework to generate multimodal lexicons which maximize multiple performance metrics over a wide range of communication modalities. Due to the extremely numerous possibilities of assigning instances of modalities to control commands (i.e. lexicons), new tools and theoretical methods are required to design and assess the quality of lexicons. In this dissertation, a novel methodology utilizing Random Instance Modeling and Generation (RIMAG) is presented to tackle this problem. RIMAG is designed to generate optimal multimodal lexicons for cooperative human-machine task completion. The task is modeled and lexicons which maximize multiple performance metrics are determined. One of the contributions of this work is its ability to model and simulate both human and machine aspects of interaction. Validation of the generated lexicons was conducted with experiments utilizing the Gestonurse robot for mock abdominal incision and opening. Experimental results indicate that predicted optimal lexicons significantly outperform predicted suboptimal lexicons validating the methodology.
Degree
Ph.D.
Advisors
Wachs, Purdue University.
Subject Area
Industrial engineering|Robotics|Computer science
Off-Campus Purdue Users:
To access this dissertation, please log in to our
proxy server.