Virtual learning and planning toward autonomous object manipulation

Abstract

Another contribution of the dissertation is our virtual learning from demonstration system, which helps robots learn task knowledge and perform informed motion planning. Successful task motions are demonstrated using human hand tracking and virtual reality technologies. These motions are then fit into Gaussian mixture models (GMMs), which provide biased samples and help calculate costs for an optimizing motion planner. Our experiments show that GMM-based sampling and cost calculation both favor motions similar to the demonstrations, hence capable of providing guidance to motion planning.

Combining our efforts in both directions, we designed a planning system that uses a reachability database together with data and models obtained from virtual demonstrations. The reachability database identifies demonstrated motions reachable from a robot's current base placement. When none of them are reachable, the reachability database can also help the robot find new base placements from where the manipulator can reach as many demonstrated motions as possible. Only these reachable motions are used to train GMMs, so that the models are focused on reachable portions of the manipulator's workspace. By providing biased samples and calculating task-relevant costs, the trained GMMs guide an optimizing motion planner to find effective plans that honor implicit task knowledge conveyed in the demonstrations.

Given an end effector (e.g. robotic gripper) path that accomplishes a task, a robot needs to know where its mobile base should move so that its manipulator can execute the entire path. A reachability database, or a reachability map, characterizes a manipulator's reachable workspace through deterministic discretizations. To find good poses for its base, or "base placements", the robot matches the given end effector path with desirable portions of its manipulator's reachability database. We introduced a new orientation-based structure for reachability database that supports end effector extensions, which allows the robot to construct new reachability databases for grasped tools in fractions of a second, by shifting and rotating elements in an existing reachability database for its gripper. Without such extension capability, recomputing all the elements in the database would take hours or days.

Robotic mobile manipulators can be useful in many scenarios, including warehouses, factories, hospitals, and so forth. Working in these places often requires the robots to have some level of autonomy and intelligence, so that they can automatically adapt to changes without needing constant supervisions from humans. This is non-trivial as many challenges mix into the problem: robust object recognition, compliant control, optimal motion planning, etc. Among all of them, we make contributions in two planning related directions: 1) characterizing a manipulator's workspace structure to take advantage of reachable and dexterous portions of the workspace for executing tasks, and 2) learning and planning with task-relevant knowledge critical to the task's success.