Unobtrusive analysis of human behavior in task-based group interactions

Abstract

First, we present a method for estimating the body orientation of seated people by fusing low-resolution range information collected from downward-pointed time-of-flight (ToF) sensors with synchronized speaker identification information from microphone recordings. We propose a Bayesian estimation algorithm for the quantized body orientations in which the likelihood term is based on the observed ToF data and the prior term is based on the occupants' locations and current speakers. We evaluate our algorithm in real meeting scenarios and show that it is possible to accurately estimate seated human orientation even with very low-resolution systems. Next, we present a method for estimating the head pose and visual focus of attention (VFOA) of meeting participants using higher resolution ceiling-mounted ToF sensors. We compute 3D point clouds from the depth maps of the sensors, and apply a rigid transformation that optimally aligns the two point clouds in the least squares sense to build a combined 3D point cloud of the entire scene. The head of each participant is thresholded based on a dynamic thresholding criterion, and an ellipsoid is fit to the 3D head of each seated individual. The head pose is estimated for each participant by tracking the axes of the ellipsoids, together with appropriate filtering. We combine the head pose with contextual information about the active speaker to estimate the VFOA of the meeting participants.

Group meetings can suffer from serious problems that undermine productivity, including bias, "groupthink", fear of speaking, and unfocused discussion. To better understand these issues, propose interventions, and thus improve team performance, we need to study human dynamics in group meetings. Studying group dynamics is an inherently difficult problem, because it requires the identification and measurement of various verbal and non-verbal events that dynamically change within a group interaction at a very fine temporal and spatial granularity. However, this process currently heavily depends on manual coding and video cameras or wearable sensors. Manual coding is tedious, inaccurate, and subjective, while active video cameras can affect the natural behavior of meeting participants. Wearable sensors are invasive and participants may feel uncomfortable or inhibited wearing sensors during a natural conversation. In this thesis, we present unobtrusive sensing and analysis of human behavior in task-based group interactions through a fusion of microphones and ceiling-mounted time-of-flight (ToF) sensors. Using the multimodal sensors, we can track occupants, understand their coarse body, head and arm poses, visual focuses of attention, sitting postures, and derive non-verbal and verbal speech-based metrics. Since the ToF sensors are ceiling-mounted and out of the lines of sight of the participants, we posit that their presence would not disrupt the natural interaction patterns of individuals. We collect a new multimodal dataset of group interactions where participants have to complete a task by reaching a group consensus, and then complete a post-task questionnaire. We use this dataset for the development of our algorithms and analysis of group meetings.

Thirdly, we present a method of estimating the seated body posture and armpose of the participants from the higher resolution ceiling mounted ToF sensors. For body posture estimation, we extract features from the elevation profile of each individual and train a Support Vector Machine (SVM) classifier for classifying leaning forward vs. leaning backward classes. For armpose classification, we use transfer learning on the Mask R-CNN architecture followed by a rule-based algorithm to classify if the arms are together, crossed, on table, or touching the face. We use the recorded audio information to extract non-verbal speech patterns, acoustic features, and discussion content.

Finally, we derive metrics from the rich set of multimodal data and correlate them with user-reported rankings of emergent group leaders and major contributors to produce accurate predictors. We show that our automated techniques of extracting visual and non-verbal speech features can explain 63% of the variance in perceived emergent leadership scores, and can predict emergent group leaders and major contributors with 90% and 95% accuracies respectively. The results are promising and show that it is possible to study group dynamic patterns with just ceiling-mounted, unobtrusive devices and no frontal video cameras.