Multimodal Human Robot Interaction

Summer School by EURON

(European Robotics Research Network)

Hamburg, Germany

September 7-13, 2003

This planned summer school will give young European researchers the chance to learn about important human-robot interaction modalities from the invited lecturers. It will also allow them to get in touch with each other.

The summer school will introduce new ways to utilize the existing computational approaches to develop new user-centered multimodal interaction principles and will focus on a subset of tasks, which can be generalized to other task domains. Gaze and indicating guestures are very important interaction modalities which are closely coupled with natural language, e.g. "grasp that one" or "go there". We arranged 1 1/2 days to introduce the techniques using weakly calibrated cameras for computing gaze and pointing direction as well as applications in interfaces for mobile robots. Additionally, we will provide our hardware setup and software for the participants to do exercises on what they learned in theory. The second part is about the fundamentals for developing dialog systems for robots, one of the most sophisticated interaction modalities. The invited lecturer will use 1 1/2 days to introduce basic computational linguistics, the role of semantics and inference, both in theory and practise. Touching can be an optimal completion to visualization with applications in robot therapy or manipulation. Principles of touching sensors, development of pet robots, and applications and their combination with other modalities will be discussed in the third part of the summer school (1 day). The fourth part (1 day) is about computer haptics, the art and science of developing software algorithms that synthesize computer-generated forces to be displayed to the user for perception and manipulation of virtual objects through force-reflecting electromechanical systems. At the end, we will discuss the subsequent cross-modal interaction issues when multiple modalities are introduced into a natural human-robot environment. The grand design challenges of multimodal interaction will be summarized at the end of the course.

For most of the lectures, theory, practice and applications will be combined so that the participants can learn the basics of human-robot interactions and then use computers, vision systems, haptic devices, and pet robots to partially apply the learned theory.

Abstract:
Visual recognition is one of the most important perceptual functions for intelligent robots. To achieve natural ways for human-robot interaction, recognition of user's behaviors is especially important. In this course, I will firstly present current technologies of real-time vision for human-robot interaction. The face tracking system and its application to robot interfaces we have developed will also be described. Then participants will have some practices for real-time vision processing. A camera connected to a PC will be used for capturing human images, and simple human interfaces will be implemented based on real-time vision techniques.

Bio:
Yoshio Matsumoto is an associate professor at The Robotics Lab, Nara Institute of Science and Technology. He received the B.E., M.E., and Dr. of Engineering, in 1993, 1995, 1998 respectively from the University of Tokyo. In 1998, he joined the Australian National University as a research fellow, where he developed a real-time vision system for gaze measurement. In 1999 he joined the Graduate School of Information Science, Nara Institute of Science and Technology. His research interests include real-time vision processing technology in real-world which can be applied to robotics and human interfaces.

Material: ppt, pdf

Abstract: This course consists of three parts:

(1) Automatic speech recognition for communication with robots.
Here we will learn how commercial speech recognisers can be modelled for specific dialogue applications, such as communication with robots. We will see how (domain independent) linguistic grammars can be used, how language models can be tuned to specific domains, and how to build meaning representations from the speech recogniser output in a systematic way.

(2) Computational semantics and inference.
Here we will introduce the basics of natural language semantics, refresh aspects of logical inference, and show how to use first-order reasoning tools (such as theorem proving and model building) in human-machine dialogue. This material of this part of the course will be based on the textbook by Blackburn & Bos (see www.comsem.org).

(3) Dialogue Management and System Building
Here we will introduce the information-state approach to dialogue modelling and demonstrate how the various components of a dialogue system (speech recognition, semantic interpretation speech synthesis, the robot interface) communicate with each other using the open agent architecture.

Material: (1) ppt, pdf / (2) pdf / (2) ppt, pdf / (3) ppt, pdf

Abstract:
We have been developing mental commit robots that provide psychological, and social effects to human beings through physical interaction. The appearances of these robots look like real animals such as cat and seal. The seal robot was developed especially for therapy. We have been applied seal robots to assisting therapy of children at a pediatric ward, and to assisting activity of elderly people at a day service center. In order to investigate the effects of seal robots to people, we evaluated the people's moods by face scales (which express person's moods by illustration of person's faces). In the case of elderly people, questionnaires and urinary tests were conducted additionally. Moreover, stress of nursing staff was investigated. We evaluated their mental poverty by "Burnout Scale." As the results, feelings of people were improved by interaction with the robots. Then, urinary tests of elderly people showed that their restoration abilities to the stress were also improved. Moreover, the nursing staff's mental poverty decreased because the elderly people spent their time by themselves with the robots. Consequently, the seal robots were useful at pediatric ward and at elderly institutions.

Material: pdf

Abstract:
Just as computer graphics is concerned with synthesizing and rendering visual images, computer haptics is the art and science of developing software algorithms that synthesize computer generated forces to be displayed to the user for perception and manipulation of virtual objects through force-reflecting electromechanical systems, also known as haptic interfaces. We have already seen various applications of computer haptics in the areas of medicine (surgical simulation, tele-medicine, haptic user interfaces for blind people, rehabilitation of patients with neurological disorders) entertainment (3D painting, character animation, morphing and sculpting), mechanical design (path planning and assembly sequencing), and scientific visualization (geophysical data analysis, molecular manipulation). In this tutorial, I will talk about haptic interfaces, the fundamental concepts of computer haptics, and our related work in the areas of surgical simulation, shared virtual environments, and human perception.

Reference:
Basdogan, C., Srinivasan, M.A., 2001, "Haptic Rendering In Virtual Environments (PDF), "Handbook of Virtual Environments", pp. 117-134 (electronic version is available through http://network.ku.edu.tr/~cbasdogan).

Short Bio:
Dr. Basdogan conducts research in the areas of man-machine interfaces, computer graphics, virtual environments, robotics, medical simulation and visualization, and computational modeling for biological systems. Before joining to Koc University, he was a senior member of technical staff at NASA-Jet Propulsion Laboratory of California Institute of Technology from 1999 to 2002. He moved to JPL from Massachusetts Institute of Technology where he was a research scientist and principal investigator at MIT Research Laboratory of Electronics and a member of the MIT Touch Lab and the Sensory Communication Group from 1

Material: (1) ppt, pdf / (2) ppt, pdf / (3) ppt, pdf

ATTENTION Applicants!
NEW DEADLINE: Please send the application form by August 22 2003 to tetsis@informatik.uni-hamburg.de. A letter of confirmation will be sent to all accepted applicants. Please check the latest information on this website.

ATTENTION: If you had problems last week with sending us mails, please send them again: the problem is solved.

For further adminstrative information, please contact:
Tatjana Tetsis, Secretary Group TAMS
e-mail: see above
fax: +49/40/42883-2397

For further scientific information, please contact:
Jianwei Zhang, Professor and Head of TAMS
e-mail: zhang@informatik.uni-hamburg.de
fax: see above

Schwesternheim Bethanien (Sister's home Bethanien, 41-49, Martinistrasse)
Bus no. 38 heading for Teulesbrueck". Get off at "Tarpenbekstrasse". Go down Tarpenbekstrasse and take the second road on your rigt, which is Martinistrasse. Go to No. 44, the Hospital Bethanien (Distance from bus station: 4 minutes). On the left inside the entrance is the gatekeepers. Say your name, and they will give you a key as well as map showing how to reach house C of Schwesternheim Bethanien (it's a sister's home consisting of 3 houses) opposite the street. The whole journey takes about 22 minutes.
Taxi: costs about 8 Euro.
ATTENTION: inform Tatjana Tetsis if you arrive there after 20:00, as it would be difficult to get your key then.
Here is a link to a map with the Krankenhaus (Hospital) Bethanien. The home is directly opposite.

Schwesternheim Bethanien (Sister's home Bethanien, 41-49, Martinistrasse)
Take one of the following subways: S 31 heading for Altona, or S21 heading for Elbgaustrasse. Get off at "S-Bahn Holstenstrasse". Go down the stairs of the S-Bahn station, take the left exit. At the bus station rightin front of you, take bus no. 20 heading for Ruebenkamp. Get off at "Krankenhaus Bethanien". Cross the street and enter No. 44, the Hospital Bethanien. Rest: see above under "airport .
ATTENTION: inform Tatjana Tetsis if you arrive there after 20:00, as it would be difficult to get your key then.

from the city center
Take the Grindelallee, Hoheluftchaussee, and then Lokstedter Steindamm to Siemers Platz. Turn left into Vogt-Wells-Strasse, take the second street to the right and then turn immediately left into Gazellenkamp. Turn right at the traffic lights and after 200 meters, turn right again into Vogt-Kölln-Stasse.