Enabling human-machine coexistence using depth sensors

Abstract

In this thesis, we investigate the application of sensing technology to scenarios including large scale immersive environments, industrial robot workspaces, and domestic environments. We design frameworks that tackle the individual requirements and challenges of each of these application areas. First, we look at expansive immersive environments, where the addition of multiple 3D sensors enables occupant awareness and gesture based interactions through the application of image filtering techniques and shape approximations. This not only allows the environment to react to the presence of its users, but also lets the users interact with its vast screens through pointing and dragging. We evaluate our system in an experimental immersive space. Second, we look at industrial robot workcells, where through a combination of well-placed 3D sensors, joint configuration information of the robot, highly accurate models of its links, and 3D shape primitives we develop a real-time safety solution. Through simulation and a physical testbed, we demonstrate that this system allows human workers to safely cohabit and collaborate with large, powerful industrial robots. Third, we employ robot-mounted sensors and image-based visual servoing on augmented reality markers to enable the flexible assembly of large structures in an industrial manufacturing scenario. We evaluate our approach in a physical testbed and establish that we can exceed metrics achieved by a fully manual process. Lastly, we integrate room and robot mounted sensors with an assistive robot through the use of markers to enable an activity of daily living for quadriplegic individuals, introducing the possibility of restoring some independence in their lives.

Humans benefit when the spaces they inhabit and the machines they interact with are smarter. Intelligent machines are transitioning from mere tools to partners in many human endeavours, and hybrid human-machine systems are becoming increasingly popular. There is a need to make existing spaces interactive, collaborative, and more accommodating of human presence. From expansive theatre-like spaces to factory floors to kitchens, each space can be augmented with the addition of sensing. Networked, low-cost, consumer-grade 3D sensors have introduced the possibility of adding accurate occupancy sensing to a host of existing and emergent applications. The number and layout of these sensors can be configured according to the requirements of the application, providing a rich representation of the covered volume and its contents. This information can be processed to augment individual spaces in relevant ways. Specifically, we explore three application areas, where the addition of 3D sensors and cameras can help establish a synergy between human beings and their surroundings.