Evaluation of Visual Analytics with Application to Social Spambot Labeling

Mosab A Khayat, Purdue University

Abstract

Visual analytics (VA) solutions emerged in the past decade and tackled many problems in a variety of domains. The power of combining the abilities of human and machine creates fertile ground for new solutions to grow. However, the rise of these hybrid solutions complicates the process of evaluation. Unlike automated solutions, VA solutions behavior depends on the user who operates them. This creates a dimension of variability in measured performance. The existence of a human, on the other hand, allows researchers to borrow evaluation methods from domains, such as sociology. The challenge in these methods, however, lies in gathering and analyzing qualitative data to build valid evidence of usefulness. This thesis tackles the challenge of evaluating the usefulness of VA solutions. We survey existing evaluation methods that have been used to assess VA solutions. We then analyze these methods in terms of validity and generalizability of their findings, as well as the feasibility of using them. Subsequently, we propose an evaluation framework which suggests evaluating VA solutions based on judgment analysis theory. The analysis provided by our framework is capable of quantitatively assessing the performance of a solution while providing a reason for the captured performance. We have conducted multiple case studies in social spambot labeling domain to apply our theoretical discussion. We have developed a VA solution that tackles social spambot labeling problem, then use this solution to apply existing evaluation methods and showcase some of their limitations. Furthermore, we have used our solution to show the benefit yielded by our proposed evaluation framework.

Degree

Ph.D.

Advisors

Ghafoor, Purdue University.

Subject Area

Design|Computer science|Web Studies

COinS