Experimental assessment of higher-level data analysis skills

Abstract

This thesis reports the results of a pilot experiment aimed at learning how to assess and improve students' competence in data analysis. The premise of the research is that, while most engineering statistics courses emphasize technique, students also need to develop judgment about how to apply techniques, and further need to develop "analytic imagination" so that they can see and exploit the potential in datasets. These skills distinguish experts from novices. This pilot study assesses twenty-seven volunteers in two statistics classes on their overall working style and behaviors and attempts to relate those factors to a successful analysis. Ultimately, these experiments should lead to methods and materials that transform classical technique-oriented statistics courses into courses that will better prepare engineers to be effective data analysts. Analysis of the pilot experiment concluded that there are major differences across individual students in terms of their behaviors and the quality of their responses. However, these descriptors were not predictable from which class they were taking or from any of their personal attributes. I introduce new methods describing student problem-solving behavior, volume and variety. There were marginally significant differences of volume and variety of steps in the more advanced course as well as significant differences between younger and older students. Nevertheless, valuable lessons were learned regarding methods for insuring data quality, quantifying student behavior, and assessing the quality of performance. These lessons should be applied to additional experiments describing and contrasting the performance on open-ended problems with larger numbers of students in classes featuring greater differences in statistical background.