logo euron-2003
Faculty of Informatics | TAMS | TAMS English | Lectures | Publications
 
 

Multimodal Human Robot Interaction

Summer School by EURON

(European Robotics Research Network)

Coordinated by TAMS
University of Hamburg
Department of Informatics

Hamburg, Germany

September 7-13, 2003



General Introduction

This planned summer school will give young European researchers the chance to learn about important human-robot interaction modalities from the invited lecturers. It will also allow them to get in touch with each other.

The summer school will introduce new ways to utilize the existing computational approaches to develop new user-centered multimodal interaction principles and will focus on a subset of tasks, which can be generalized to other task domains. Gaze and indicating guestures are very important interaction modalities which are closely coupled with natural language, e.g. "grasp that one" or "go there". We arranged 1 1/2 days to introduce the techniques using weakly calibrated cameras for computing gaze and pointing direction as well as applications in interfaces for mobile robots. Additionally, we will provide our hardware setup and software for the participants to do exercises on what they learned in theory. The second part is about the fundamentals for developing dialog systems for robots, one of the most sophisticated interaction modalities. The invited lecturer will use 1 1/2 days to introduce basic computational linguistics, the role of semantics and inference, both in theory and practise. Touching can be an optimal completion to visualization with applications in robot therapy or manipulation. Principles of touching sensors, development of pet robots, and applications and their combination with other modalities will be discussed in the third part of the summer school (1 day). The fourth part (1 day) is about computer haptics, the art and science of developing software algorithms that synthesize computer-generated forces to be displayed to the user for perception and manipulation of virtual objects through force-reflecting electromechanical systems. At the end, we will discuss the subsequent cross-modal interaction issues when multiple modalities are introduced into a natural human-robot environment. The grand design challenges of multimodal interaction will be summarized at the end of the course.

For most of the lectures, theory, practice and applications will be combined so that the participants can learn the basics of human-robot interactions and then use computers, vision systems, haptic devices, and pet robots to partially apply the learned theory.



Contents "Real-time computer vision for human interfaces".
Prof. Yoshio Matsumoto
Robotics Lab, Nara Institute of Science and Technology, Japan

(One of the key developers of vision-based gaze detection software)
http://robotics.aist-nara.ac.jp/~yoshio/index_e.html

Abstract:
Visual recognition is one of the most important perceptual functions for intelligent robots. To achieve natural ways for human-robot interaction, recognition of user's behaviors is especially important. In this course, I will firstly present current technologies of real-time vision for human-robot interaction. The face tracking system and its application to robot interfaces we have developed will also be described. Then participants will have some practices for real-time vision processing. A camera connected to a PC will be used for capturing human images, and simple human interfaces will be implemented based on real-time vision techniques.

Bio:
Yoshio Matsumoto is an associate professor at The Robotics Lab, Nara Institute of Science and Technology. He received the B.E., M.E., and Dr. of Engineering, in 1993, 1995, 1998 respectively from the University of Tokyo. In 1998, he joined the Australian National University as a research fellow, where he developed a real-time vision system for gaze measurement. In 1999 he joined the Graduate School of Information Science, Nara Institute of Science and Technology. His research interests include real-time vision processing technology in real-world which can be applied to robotics and human interfaces.

Material: ppt, pdf

 
  "Building Spoken Dialogue Systems for Embodied Agents".
Dr. Johan Bos,
Division of Informatics, University of Edinburgh, UK.
http://www.iccs.informatics.ed.ac.uk/~jbos/

Abstract: This course consists of three parts:

(1) Automatic speech recognition for communication with robots.
Here we will learn how commercial speech recognisers can be modelled for specific dialogue applications, such as communication with robots. We will see how (domain independent) linguistic grammars can be used, how language models can be tuned to specific domains, and how to build meaning representations from the speech recogniser output in a systematic way.

(2) Computational semantics and inference.
Here we will introduce the basics of natural language semantics, refresh aspects of logical inference, and show how to use first-order reasoning tools (such as theorem proving and model building) in human-machine dialogue. This material of this part of the course will be based on the textbook by Blackburn & Bos (see www.comsem.org).

(3) Dialogue Management and System Building
Here we will introduce the information-state approach to dialogue modelling and demonstrate how the various components of a dialogue system (speech recognition, semantic interpretation speech synthesis, the robot interface) communicate with each other using the open agent architecture.

Material: (1) ppt, pdf / (2) pdf / (2) ppt, pdf / (3) ppt, pdf

 
  "Human-Robot Interaction for Psychological Enrichment".
Dr. Takanori Shibata,
AIST, Japan

(One of the developers of Cat Robot and Seal Robot).
"http://www.aist.go.jp/MEL/soshiki/robot/biorobo/shibata/shibata.html"

Abstract:
We have been developing mental commit robots that provide psychological, and social effects to human beings through physical interaction. The appearances of these robots look like real animals such as cat and seal. The seal robot was developed especially for therapy. We have been applied seal robots to assisting therapy of children at a pediatric ward, and to assisting activity of elderly people at a day service center. In order to investigate the effects of seal robots to people, we evaluated the people's moods by face scales (which express person's moods by illustration of person's faces). In the case of elderly people, questionnaires and urinary tests were conducted additionally. Moreover, stress of nursing staff was investigated. We evaluated their mental poverty by "Burnout Scale." As the results, feelings of people were improved by interaction with the robots. Then, urinary tests of elderly people showed that their restoration abilities to the stress were also improved. Moreover, the nursing staff's mental poverty decreased because the elderly people spent their time by themselves with the robots. Consequently, the seal robots were useful at pediatric ward and at elderly institutions.

Material: pdf

 
  "Computer Haptics and Applications".
Prof. Cagatay Basdogan,
College of Engineering, Koc University, Turkey.

(one of the developers of Phantom haptic devices)
"http://network.ku.edu.tr/~cbasdogan"

Abstract:
Just as computer graphics is concerned with synthesizing and rendering visual images, computer haptics is the art and science of developing software algorithms that synthesize computer generated forces to be displayed to the user for perception and manipulation of virtual objects through force-reflecting electromechanical systems, also known as haptic interfaces. We have already seen various applications of computer haptics in the areas of medicine (surgical simulation, tele-medicine, haptic user interfaces for blind people, rehabilitation of patients with neurological disorders) entertainment (3D painting, character animation, morphing and sculpting), mechanical design (path planning and assembly sequencing), and scientific visualization (geophysical data analysis, molecular manipulation). In this tutorial, I will talk about haptic interfaces, the fundamental concepts of computer haptics, and our related work in the areas of surgical simulation, shared virtual environments, and human perception.

Reference:
Basdogan, C., Srinivasan, M.A., 2001, "Haptic Rendering In Virtual Environments (PDF), "Handbook of Virtual Environments", pp. 117-134 (electronic version is available through http://network.ku.edu.tr/~cbasdogan).

Short Bio:
Dr. Basdogan conducts research in the areas of man-machine interfaces, computer graphics, virtual environments, robotics, medical simulation and visualization, and computational modeling for biological systems. Before joining to Koc University, he was a senior member of technical staff at NASA-Jet Propulsion Laboratory of California Institute of Technology from 1999 to 2002. He moved to JPL from Massachusetts Institute of Technology where he was a research scientist and principal investigator at MIT Research Laboratory of Electronics and a member of the MIT Touch Lab and the Sensory Communication Group from 1

Material: (1) ppt, pdf / (2) ppt, pdf / (3) ppt, pdf

 
Summary presented at the EURON annual meeting 2004 pdf


Program
SundayArrival
Monday
9:30-10:00  Welcome and overview.
Prof. J. Zhang TAMS, Faculty of Informatics, University of Hamburg
10:00-12:00  Real-time computer vision for human interfaces.
Prof. Yoshio Matsumoto, Robotics Lab, Nara Institute of Science and Technology
13:00-18:00  Real-time computer vision for human interfaces (continued)
Tuesday
9:00-12:00  Real-time computer vision for human interfaces (continued)
13:00-18:00  Building Spoken Dialogue Systems for Embodied Agents.
Dr. Johan Bos, Division of Informatics, University of Edinburgh
Wednesday
9:00-12:00  Spoken Dialogue Systems for Embodied Agents (continued)
13:00-18:00  Spoken Dialogue Systems for Embodied Agents (continued)
Thursday
9:00-12:00  Interaction with Pet Robots.
Dr. Takanori Shibata, AIST, Japan.
13:00-15:00  Interaction with Pet Robots (continued)
15:00-18:30  Excursion
Friday
9:00-12:00  Computer Haptics and Applications.
Prof. Cagatay Basdogan, College of Engineering, Koc University
13:00-17:00  Computer Haptics and Applications (continued)
17:00-18:00  Cross-modal interactions and grand challenges - Discussions.
Prof. J. Zhang TAMS, Faculty of Informatics, University of Hamburg
SaturdayDeparture


Admission and Accommodation The number of participants is limited to 40. Each member of EURON can send two students. There is no registration fee. Costs for accommodation and lunch will be carried by EURON. There are single-rooms booked in two student's homes, at around 19 and 35 minutes from the Informatics-Campus. The homes are:
Student's Home Berliner Tor (3, Berliner Tor Street)
Schwesternheim Bethanien (Sister's home Bethanien, 41-49, Martinistrasse)
Details on how to reach them: see below "How to get there".

ATTENTION Applicants!
NEW DEADLINE: Please send the application form by August 22 2003 to tetsis@informatik.uni-hamburg.de. A letter of confirmation will be sent to all accepted applicants. Please check the latest information on this website.

ATTENTION: If you had problems last week with sending us mails, please send them again: the problem is solved.

For further adminstrative information, please contact:
Tatjana Tetsis, Secretary Group TAMS
e-mail: see above
fax: +49/40/42883-2397

For further scientific information, please contact:
Jianwei Zhang, Professor and Head of TAMS
e-mail: zhang@informatik.uni-hamburg.de
fax: see above



How to get there Travel costs cannot be paid. Information on and online-booking of Eurailpasses can e.g. be found at http://www.raileurope.com/us/rail/about_train_travel/perfect_rail_pass.htm.
Timetables Get a personal timetable for reaching your student's home or any other destination in Hamburg by public transport here. Click on "Persönlicher Fahrplan" and then on the tiny British Flag right at the top for the English version, then enter starting point, destination and the required date and time of arrival or departure.
the student's homes
from the airport Student's Home Berliner Tor (3, Berliner Tor Street)
Take bus no. 26 to the subway station "Alsterdorf". Take the blue subway U1 heading for "Ohlstedt", get off at "Lohmuehlenstrasse". Follow the signs to the exit to Berliner Tor street. From there it's a 5 minute walk to Berliner Tor No. 3, where the home is.
Or you can take the bus 172 heading for "Mundsburger Bruecke" and get off at the "U/S Ohlsdorf". From there you take the S-bahn (a subway line) S1 heading for "Blankenese" and get off at "Berliner Tor".
Here you will find a map of the home's (blue circle) surroundings with the subway stations "Berliner Tor" and "Lohmuehlenstrasse".

Schwesternheim Bethanien (Sister's home Bethanien, 41-49, Martinistrasse)
Bus no. 38 heading for Teulesbrueck". Get off at "Tarpenbekstrasse". Go down Tarpenbekstrasse and take the second road on your rigt, which is Martinistrasse. Go to No. 44, the Hospital Bethanien (Distance from bus station: 4 minutes). On the left inside the entrance is the gatekeepers. Say your name, and they will give you a key as well as map showing how to reach house C of Schwesternheim Bethanien (it's a sister's home consisting of 3 houses) opposite the street. The whole journey takes about 22 minutes.
Taxi: costs about 8 Euro.
ATTENTION: inform Tatjana Tetsis if you arrive there after 20:00, as it would be difficult to get your key then.
Here is a link to a map with the Krankenhaus (Hospital) Bethanien. The home is directly opposite.

from the train station
Hauptbahnhof
Student's Home Berliner Tor (3, Berliner Tor Street)
Take one of the following 3 subways to "Berliner Tor" station:
-the red subway U2 heading for "Barmbek" or "Wandsbek Gartenstadt"
-the S-Bahn (also a subway, just called differently) S1 heading for Poppenbüttel
-the yellow U3 heading for "Muemmelmannsberg".
From there it's a 5 minute walk to Berliner Tor No. 3. For a map: see above under "airport"

Schwesternheim Bethanien (Sister's home Bethanien, 41-49, Martinistrasse)
Take one of the following subways: S 31 heading for Altona, or S21 heading for Elbgaustrasse. Get off at "S-Bahn Holstenstrasse". Go down the stairs of the S-Bahn station, take the left exit. At the bus station rightin front of you, take bus no. 20 heading for Ruebenkamp. Get off at "Krankenhaus Bethanien". Cross the street and enter No. 44, the Hospital Bethanien. Rest: see above under "airport .
ATTENTION: inform Tatjana Tetsis if you arrive there after 20:00, as it would be difficult to get your key then.

the campus
  Click here for a map of the campus "Informatikum". The group TAMS is located on the second floor of house F. Lectures will be held on the first floor of house D in room D-220.
from Berliner Tor Walk down Berliner Tor Street towards the subway stop Berliner Tor (8 minutes walk). Take the red subway U2 in the direction of Niendorf Nord (it leaves every 10 minutes, 08:56 is good to be on time Monday morning.) Get off at Hagenbecks Tierpark. Walk till the end of the underpass, climb the stairs, and take one of the buses 181 or 281 heading towards Niendorf. Get off at Informatikum station. Turn right into Vogt-Kölln-Strasse and, after about 100m, right again into the Campus. Walk straight on, take the second path on the left. Turn right into the main entrance of house D. Climb the stairs to the first floor. Opposite on your left a few steps down the corridor is room D-220. Traveling time: about 34 minutes.
from Martinistrasse Walk down Martinistrasse to your right, then turn left into Tarpenbekstrasse. Cross Tarpenbekstrasse and walk on in the same direction till the bus stop for bus 22 heading for S-Bahn Blankenese, or bus 39 heading for Teufelsbrück (but this is an express bus and more expensive). Get off at Hagenbecks Tierpark station. Rest see above. Traveling time: about 22 minutes.
Attention: No. 22 starts only every 20 minutes from Tarpenbekstrasse. To be on time on Monday morning, take the 22 at 08:53. If you take it at 09:13, you will arrive at Informatikum station at 09:30 and cut it very close. Or take the 39 at 09:06.
from the airport Take bus no. 39 from Hamburg Fuhlsbüttel heading to Teufelsbrück. Get off at Hagenbecks Tierpark (zoo) and take one of the buses 181 or 281. Get off at Informatikum station, turn right into Vogt-Kölln-Strasse and, after about 100m, right again into the Campus.
from the train station Coming from Hamburg Hauptbahnhof (main train station) take the red subway U2 heading towards Niendorf to Hagenbecks Tierpark (zoo). Walk till the end of the underpass, climb the stairs, and take one of the buses 181 or 281 heading towards Niendorf. Get off at Informatikum station, see above.
by car from the motorway A7
Take the exit Hamburg-Stellingen and turn left into Kieler Strasse. Turn right at the second crossroads into Wördemanns Weg. Vogt-Kölln-Stasse is the fifth street to the left.

from the city center
Take the Grindelallee, Hoheluftchaussee, and then Lokstedter Steindamm to Siemers Platz. Turn left into Vogt-Wells-Strasse, take the second street to the right and then turn immediately left into Gazellenkamp. Turn right at the traffic lights and after 200 meters, turn right again into Vogt-Kölln-Stasse.



Sightseeing Hamburg has been called the Venice of the North - and is said to have more bridges than Venice and Amsterdam combined. Would you like a taste of its beauty?
360°Panoramic Views, Historic Impressions and a Bird's Eye View
Around the River Alster
Impressions of the harbour on the River Elbe
City Impressions


Impressum
15.03.2004
top
http://tams-www.informatik.uni-hamburg.de/lehre/ss2003/euron_summer_school/euron_summer_school.html