Fachbereich Informatik

Intuitive Gesture-based Robot Teach-Mode

Bachelor- or Master-Thesis at group TAMS


Despite a lot of progress in robot motion planning algorithms, automatically generated trajectories are often far from optimal, and most industrial robots are still programmed to follow human-designed trajectories. This is done either by following hard-coded waypoints or using a teach-mode, where the user can drag the real robot to the desired positions. The first approach results in mathematically precise coordinates, but is highly un-intuitive, while the second approach requires a lot of force and precise way-points are hard to reach due to friction and the high inertia of the robot. With typical current software, the recorded trajectories are fixed and cannot easily be modified.

However, there has been a lot of progress recently in visual object tracking and in particular in markerless tracking of human hands and fingers using deep networks. In the proposed thesis, we will develop a human hand tracking and gesture detection interface to create robot trajectories interactively, by just pointing at the desired waypoints and by moving between waypoints.

While the teaching is in progress, a 3D-visualization of the trajectory will be generated (rviz, Hololens VR) and provide visual feedback to the user. Gestures will be eefined to select waypoints and to move, copy, or delete them, and to adjust and finetune the trajectory. Other gestures will be provided to define collision objects and to draw regions that should be avoided (or preferred) by the robot motions. In addition to the visualization of the tool center point trajectory, we will also enable to visualize the motion of other parts of the robot (elbow, shoulder, knees) and to select and adjust those motions as well.

The precise scope of the software implementation is to be discussed, with a basic functionality achievable in a BSc-thesis, but a really useful function set probably requires a full MSc-thesis. Given the current Corona restrictions, large parts of the software can be designed (and tested) in home office, using a setup consisting of a RGB or RGB-D camera and the underlying hand/finger tracking software. However, actual evaluation of the software will probably require at least some tests and experiments on the real robot in our lab.

Thesis Goals: