A methodology for evaluation of provenance-based user interfaces

Abstract

The Merriam Webster English Dictionary defines provenance as: (i) the origin or source of something; (ii) the history of ownership of a valued object or work of art or literature. In its earliest usage, the provenance of physical objects - such as pieces of artwork - could be used to make assessments of their value. In more recent times, provenance has become an increasingly critical component for assessment of data in digital systems. From the perspective of digital artifacts, the World Wide Web Consortium's (W3C) Provenance Working Group defines provenance as: a record that describes the people, institutions, entities, and activities, involved in producing, influencing, or delivering a piece of data or a thing.

A Comparison of Provenance Querying and Presentation Interfaces: This contribution centers on demonstrating utility of the ProvAnalytics framework using a case-study based evaluation. Three interface configurations were compared in a 36 subject study, aimed at gauging performance across three types of established information retrieval tasks. Findings from this study are intended to demonstrate: (i) that novice users of provenance systems can quickly adapt to working with the developed tasks and data representations, and (ii) that statistically meaningful relationships can be obtained in support of routine usability hypotheses.

An Analysis and Classification for Provenance Querying Tasks: In the related literature, no benchmarks exist for gauging the usability of provenance-based systems for end-users. This contribution builds on prior computational benchmarks - particularly from the Provenance Challenge series - to establish a set of user-centric querying tasks oriented toward the multidimensional analysis paradigm.

The ProvAnalytics Framework: This is a novel framework for exploring provenance record collections expressed as directed graphs. The core focus of ProvAnalytics is to provide a set of approaches for converting collections of provenance graphs into multidimensional datasets for subsequent review by interested users. Additionally, functionality is provided for generating synthetic provenance collections intended to meet the needs of particular interface evaluations.

A major barrier to the widespread adoption of provenance-based technologies is the absence of methodologies to enable provenance exploration and retrieval by users who lack expertise in database querying. This dissertation takes the position that existing tools and techniques from multidimensional data representation and analysis can be applied toward the design of such a methodology. To advance this position, three supporting contributions are made:

Particularly in the past decade, technologies for acquiring, recording, storing and representing provenance data have grown more sophisticated. Additionally, significant interest has been expressed by stakeholders of many digital systems in expanding their usage of provenance. However, in this time, limited work has been done to develop and rigorously evaluate tools for exploration and querying of provenance collections - particularly, for system stakeholders with limited background in database querying languages and tools. The question now remains on not only the best ways to present provenance collections in an intuitive and usable manner, but also how to effectively evaluate interface designs for achieving these goals.