BieColl - Bielefeld eCollections

Schulungstermin - https://katalogplus.ub.uni-bielefeld.d

Pietsch, Christian

Summann, Friedrich

Kataloganreicherung

Palinko, Oskar

Rea, Francesco

Sandini, Giulio

Sciutti, Alessandra

Gaze Tracking for Human Robot Interaction 
Oskar Palinko, Francesco Rea, Giulio Sandini, Alessandra Sciutti 
 Robotics, Brain and Cognitive Sciences Dept., Istituto Italiano di Tecnologia 
1 Introduction 
Humans use a number of communication cues in their daily interaction with other 
humans: primarily speech but also gestures, pointing and gaze [1]. The main pur-
pose of gaze is to provide visual information to the subject, but at the same time a 
person’s gaze implicitly provides information to an outside observer about what 
the subjects are focusing their attention on. There is a number of ways how eye 
gaze is implicitly used during communication: gaze aversion, mutual gaze, gaze 
pointing, join attention, etc.  
Humans are very good at reading other people’s gaze, but robots are less so. This 
ability would be especially important for humanoid robots to be able to mimic 
human abilities. However, most human robot interaction experiments today use 
head pose as a proxy for real eye gaze often times because it’s easier to extract 
than eye gaze [2] [3] [4] [5]. But “head gaze” does not provide all the information 
that eye gaze does [6], thus enabling robots to perform eye tracking could signifi-
cantly improve its abilities and also its acceptance by humans. A proof of concept 
gaze tracker was realized by Matsumoto and Zelinsky [7] and implemented on the 
HRP2 humanoid [8]. More recently Sciutti et al. [9] implemented a mutual gaze 
detection system on the iCub which facilitated a teacher/student scenario. Still, so 
far no extensive use of eye gaze tracking has been done in human-robot interac-
tion. 
Figure 1. Left: example performance of our eye tracking system on the iCub ro-
bot. Right: human robot interaction example where the subject needs to ask for tak-
ing toy building blocks from the robot. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 5
Gaze Tracking for Human Robot Interaction 
2 Our Contribution 
We implemented a monocular feature-based passive gaze tracking algorithm on 
the iCub platform with the goal of facilitating human robot interaction. The first 
step in eye tracking is detecting faces and finding face features. For this purpose 
we used King’s implementation [10] of Khazemi and Sullivan’s approach for find-
ing features like the corners of the eyes and mouth [11]. We also used Baltrusaitis 
implementation of the constrained local models approach for tracking head pose 
[12]. Once these measures were found we proceeded to apply an eye model to the 
detected center of the pupil similarly as in [13]. The model finally provided the es-
timate of the gaze angle of the subject, see Figure 1. We then performed a valida-
tion experiments in which we found the gaze estimates to be quite acceptable for 
our setup: the absolute error in the horizontal plane was 5 degrees on average. The 
accuracy of our system was limited by the cameras used in the iCub setup. We 
employed PointGrey Dragonfly2 cameras in 1024x768 resolution with fixed-focus 
4mm lenses, which produce images of the iris with 20 pixels in diameter when the 
subject is at 60cm. Knowing that the average diameter of the iris [14] is similar in 
size to the average eye radius (12mm) [15], then one pixel difference in the middle 
of the iris corresponds to about 3 degrees difference in gaze. Thus our accuracy is 
greatly influenced by the hardware used. It is foreseeable that the progressive de-
velopment of cheaper and small cameras will allow future robotic platforms to 
have higher resolution sensors, with a consequent improvement of the accuracy of 
our system. In the meanwhile, the current hardware already enables a gaze estima-
tion from the iCub robot that it can exploit to manage human-robot collaboration 
tasks. 
We also conducted a proof of concept human robot interaction experiment in 
which subjects were seated opposite of the robot and experimenter, who held toy 
building blocks in their hands, see Figure 1. The subject’s role was to ask for the 
blocks in specific order, but we did not provide information on how to communi-
cate with the robot. Participants used a combination of speech, pointing and gaze 
to achieve the task, but the robot really only reacted to gaze. More precisely, the 
robot handed over pieces of toy building blocks when it detected a succession of 
mutual gaze and gazing at the requested object. The subjects were not aware of the 
robot’s gaze reading ability, but could still complete the task of building a pillar 
out of these blocks just by using natural eye behavior (paper submitted to Human-
oids 2015). Hence, the robot succeeded in exploiting naturally occurring human 
gaze behavior to control its helping actions in a collaborative manner.  
Future benefits of a built-in gaze tracker in a humanoid robot can be manifold: it 
could improve turn taking, joint attention and in general the processing of all the 
communicative gaze cues typical of human interaction. Furthermore, the robot 
could potentially be used for diagnosing early behavioral problems associated with 
gaze processing as Autism Spectrum Disorders, by monitoring subjects’ gaze in 
real time.  
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 5
Oskar Palinko, Francesco Rea, Giulio Sandini, Alessandra Sciutti 
ACKNOWLEDGMENTS: The research presented here has been supported by the Eu-
ropean CODEFROR project (PIRSES-2013-612555). 
References 
[1] N. George and L. Conty, “Facing the gaze of others.,” Neurophysiol. Clin., vol. 38, no. 3, pp. 
197–207, Jun. 2008. 
[2] M. W. Doniec, G. Sun, and B. Scassellati, “Active Learning of Joint Attention,” in
Humanoid Robots, 2006 6th IEEE-RAS International Conference on, 2006, pp. 34–39. 
[3] H. Kim, H. Jasso, G. Deák, and J. Triesch, “A robotic model of the development of gaze 
following,” in 2008 IEEE 7th International Conference on Development and Learning,
ICDL, 2008, pp. 238–243. 
[4] S. Ivaldi, S. M. Anzalone, W. Rousseau, O. Sigaud, and M. Chetouani, “Robot initiative in a 
team learning task increases the rhythm of interaction but not the perceived engagement,”
Front. Neurorobot., vol. 8, 2014. 
[5] S. Sheikhi and J.-M. Odobez, “Combining dynamic head pose–gaze mapping with the robot
conversational state for attention recognition in human–robot interactions,” Pattern Recognit.
Lett. 
[6] A. Borji, D. Parks, and L. Itti, “Complementary effects of gaze direction and early saliency 
in guiding fixations during free viewing.,” J. Vis., vol. 14, no. 13, p. 3–, Jan. 2014. 
[7] Y. Matsumoto and A. Zelinsky, “An algorithm for real-time stereo vision implementation of 
head pose and gaze direction measurement,” in Automatic Face and Gesture Recognition,
2000. Proceedings. Fourth IEEE International Conference on, 2000, pp. 499–504. 
[8] J. Ido, Y. Matsumoto, T. Ogasawara, and R. Nisimura, “Humanoid with interaction ability 
using vision and speech information,” in IEEE/RSJ International Conference on Intelligent 
Robots and Systems (IROS), 2006, no. November 2008, pp. 1316–1321. 
[9] and G. S. A. Sciutti, L. Schillingmann, O. Palinko, Y. Nagai, “A Gaze-contingent Dictating 
Robot to Study Turn-taking,” in Proceedings of the 10th ACM/IEEE International
Conference on Human-Robot Interaction, 2015. 
[10] D. E. King, “Dlib-ml : A Machine Learning Toolkit,” J. Mach. Learn. Res., vol. 10, pp. 
1755–1758, 2009. 
[11] V. Kazemi and J. Sullivan, “One Millisecond Face Alignment with an Ensemble of 
Regression Trees,” in Computer Vision and Pattern Recognition (CVPR), 2014, 2014. 
[12] T. Baltrusaitis, P. Robinson, and L. P. Morency, “3D Constrained Local Model for rigid and
non-rigid facial tracking,” in Proceedings of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, 2012, pp. 2610–2617. 
[13] T. Ishikawa, S. Baker, I. Matthews, and T. Kanade, “Passive Driver Gaze Tracking with
Active Appearance Models,” in Proc. World Congress on Intelligent Transportation
Systems, 2004, pp. 1–12. 
[14] S. Thainimit, L. A. Alexandre, and V. M. N. de Almeida, “Iris surface deformation and
normalization,” Communications and Information Technologies (ISCIT), 2013 13th
International Symposium on. pp. 501–506, 2013. 
[15] I. Bekerman, P. Gottlieb, and M. Vaiman, “Variations in Eyeball Diameters of the Healthy 
Adults,” J. Ophthalmol., vol. 2014, 2014. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 5


English

Gaze Tracking for Human Robot Interaction

Pfeiffer-Leßmann, Nadine

Renner, Patrick

Pfeiffer, Thies

Analyzing Patterns of Eye Movements  
in Social Interactions 
Nadine Pfeiffer-Leßmann, Patrick Renner, Thies Pfeiffer  
1 Introduction 
Eye gaze plays an important role in human communication. One foundational skill 
in human social interaction is joint attention which is receiving increased interest in 
particular in the area of human-agent or human-robot interaction. We are focusing 
here on patterns of gaze interaction that emerge in the process of establishing joint 
attention. The approach, however, should be applicable to many other aspects of 
social communication in which eye gaze plays an important role. 
Attention has been characterized as an increased awareness [1] and intentionally 
directed perception [2] and is judged to be crucial for goal-directed behavior. Joint 
attention can be defined as simultaneously allocating attention to a target as a con-
sequence of attending to each other's attentional states [3]. In other words: Interloc-
utors have to deliberatively focus on the same target while being mutually aware of 
sharing their focus of attention [2] [4]. 
2 Our Contribution 
2.1. Study 
In a study with 18 pairs of participants (i.e. 36 different participants in total), we 
recorded eye movements while participants were solving a visual search task based 
on a verbal description of the experimenter and a communication task where they 
both had to cooperate on agreeing on a specific target stimulus. 
2.2. Automatic labeling of fixation targets 
Each pair of participants had to solve 32 tasks, which took about 30 minutes, result-
ing in 18 hours of mobile eye movement videos. Annotation time was reduced to 
zero by mapping from fixations to semantic regions automatically based on the Eye-
See3D approach [5]. 
2.3. Analysis of interaction patterns 
Our approach for the analysis is to define several indices to describe basic gazing 
behaviors, such as  
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10
Analyzing Patterns of Eye Movements in Social Interactions 
 Searching | 	 |
| |
→ 1
 Identifying | 	 |
| |
→ 1
 Looking at partner | |
| |
→ 1
 Looking for help | 	 | | |
| |
→ 1
 Helping ≫ 	 	  
 Following 	 	 ≫ 	 	 	  
This provides an abstraction of the concrete gazing behavior, which will then be 
used for the analysis of dyadic patterns. We will present the results of this analysis 
and discuss our approach with existing alternatives. 
3 Discussion 
We present an approach to characterize and analyze eye movement data in a way 
that is compatible with an online analysis. The EyeSee3D approach [5] can be ap-
plied to studies in the domain of social interactions, to significantly reduce the time 
required for the analysis of the eye movement data. In combination, the index-based 
classification and EyeSee3D will allow us to interpret gaze patterns of interacting 
persons in real-time, which is highly relevant for realizing human-agent interac-
tions. 
ACKNOWLEDGMENTS: This research was partly supported by the Cluster of Excellence Cog-
nitive Interaction Technology 'CITEC' (EXC 277) and the SFB 673 Alignment in Communication; 
both are at Bielefeld University and funded by the German Research Foundation (DFG). 
Figure 1: Interaction scenario of our study: Two participants, both eye-tracked, have to solve 
cooperative tasks. Eye movements were recorded to analyze patterns of eye gaze interaction.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10
 Nadine Pfeiffer-Leßmann, Patrick Renner, Thies Pfeiffer 
References 
1. Brinck, I. (2003): The objects of attention. In: Proc. of ESPP2003, Torino.
2. Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H. (2005): Understanding
and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences
28
3. Deak, G.O., Fasel, I., and Movellan, J. (2001): The emergence of shared attention: Using
robots to test developmental theories. In: Proc. of the First Intl. Workshop on Epigenetic
Robotics, Lund University, Cognitive Studies, 85.
4. Hobson, R.P. (2005): What puts the jointness into joint attention? In Eilan, N., Hoerl, C.,
McCormack, T., Roessler, J., eds.: Joint attention: communication and other minds. Ox-
ford University Press.
5. Pfeiffer, T., & Renner, P. (2014). EyeSee3D: a low-cost approach for analysing mobile
3D eye tracking data using augmented reality technology. Proceedings of the Symposium
on Eye Tracking Research and Applications, 195–202.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10


Analyzing Patterns of Eye Movements in Social Interactions

Kübler, Thomas C.

Kasneci, Enkelejda

Automated Comparison of Scanpaths in
Dynamic Scenes
Thomas C. Kübler and Enkelejda Kasneci
1 Introduction
According to the scanpath theory [6], the sequence of eye movements can give in-
sights into the cognitive processes of visual perception. In this light, the comparison
of scanpaths is of particular interest for many applications, e.g., driving [4], advertis-
ing, or activity recognition [1]. Although precise measurement devices and several
approaches for the extraction of the visual scanpath in dynamic scenes exist, e.g. [7],
the temporal and spatial comparison of scanpaths with their natural high variabil-
ity remains challenging and laborious. Despite the variety of existing approaches,
such as ScanMatch [2] or Multimatch [3], most scanpath studies are still limited to
the comparison of time-integrated features, e.g. fixation duration or average saccade
length. Due to the sequential nature of the scan pattern, such metrics are not suffi-
cient to capture essential pattern characteristics of the viewing behavior. Recently,
we proposed SubsMatch [5], a novel method for scanpath comparison in natural
environments. In this work, we will demonstrate the performance of SubsMatch in
comparison to the state-of-the-art when analyzing eye-tracking data derived from a
driving scenario.
Thomas C. Kübler
University of Tübingen, Computer Engineering Department, Sand 13, 72076 Tübingen, Germany
e-mail: thomas.kuebler@uni-tuebingen.de
Enkelejda Kasneci
University of Tübingen, Computer Engineering Department, Sand 13, 72076 Tübingen, Germany
e-mail: enkelejda.kasneci@uni-tuebingen.de
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17
Thomas C. Kübler and Enkelejda Kasneci
2 SubsMatch Algorithm
Instead of addressing the issue whether two scanpaths focus on the same objects
at the same time, SubsMatch addresses the question whether the scanpaths exhibit
the same repetitive exploratory patterns. Figure 1 illustrates the main steps of Subs-
Match as introduced in [5]. First, the scanpath is mapped to a string representation
based on just one dimension of the scanpath, e.g. the horizontal position of fixations.
Letter encoding is chosen to optimally map the scanpath data into equally frequent
letters. This step also compensates for spatial offset and scaling, often caused by
incorrect calibration, different seat positions while driving or different viewing dis-
tances towards the scene. The second step extracts all possible visual subpatterns
from the encoded scanpath. The problem of scanpath comparison is thereby reduced
to the comparison of the frequency of subpatterns.
Fig. 1 Overview of the SubsMatch algorithm.
3 Exemplary Application
SubsMatch was applied to scanpaths derived from simulated driving sessions of 10
subjects with advanced glaucoma and 10 control subjects. Eye movements were
recorded using a Dikablis eye tracker at 25Hz. 50% of the drivers with visual field
defects were able to pass the driving test despite their visual impairment. This result
indicates that the viewing behavior of patients who passed the test differs from the
viewing behavior of patients who failed. Glaucoma subjects who passed (Gp) and
failed (G f ) were compared to each other and to the control subjects (GC) to iden-
tify differences in the viewing patterns by means of SubsMatch, ScanMatch [2] and
MutliMatch [3]. Table 1 shows the statistical evaluation of the pairwise scanpath
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17
Automated Comparison of Scanpaths in Dynamic Scenes
Table 1 Comparison of the pairwise distances within and between scanpath groups (G for glau-
coma patients, p/f for passed or failed the driving test, C for control subjects).
{G f ,Gp} {GC,G f } {GC,Gp}
SubsMatch 0.03 0.03 0.67
ScanMatch 0.05 0.05 0.65
MultiMatch 0.89 0.89 0.89
distances between the groups. SubsMatch revealed that there are no additional com-
pensatory eye movements performed by the Gp group, as often assumed. Instead,
this group was able to maintain a normal exploratory behavior, similar to that in the
GC group. The G f group on the other hand exhibit-ed fewer horizontal, alternating
saccades.
4 Conclusion
The presented algorithm is a significant step towards automated analysis and com-
parison of visual scanpaths derived from eye-tracking data in natural environments.
Beyond further developments of SubsMatch, our future work will focus on the ques-
tion, how the analysis of recurrent patterns can be used to provide added value for
several applications.
References
1. Braunagel C, Kasneci E, Stolzmann W, Rosenstiel W. (2015). Driver-Activity Recognition in
the Context of Conditionally Autonomous Driving. IEEE 18th International Conference on
Intelligent Transportation Systems (ITSC), 2015
2. Cristino F, Matht S, Theeuwes J, Gilchrist ID. (2010). ScanMatch: a novel method for com-
paring fixation sequences. Behavior research methods, 42(3):692-700.
3. Dewhurst R, Nyström M, Jarodzka H, Foulsham T, Johansson R, Holmqvist K. (2012). It
depends on how you look at it: scanpath comparison in multiple dimensions: MultiMatch, a
vector-based approach. Behavior research methods 44(4):1079-100.
4. Kasneci E. (2013). Towards the Automated Recognition of Assistance Need for Drivers with
Impaired Visual Field. PhD thesis,University of Tübingen, Wilhelmstr. 32, 72074 Tübingen.
5. Kübler TC, Kasneci E, Rosenstiel W. (2014). Subsmatch: Scanpath similarity in dynamic
scenes based on subsequence frequencies. In Proceedings of the Symposium on Eye Tracking
Research and Applications, ETRA 2014, 319–322, ACM.
6. Noton D, Stark L. (1971). Scanpaths in saccadic eye movements while viewing and recogniz-
ing patterns. Vision research 11(9):929-42.
7. Tafaj E, Kübler TC, Kasneci G, Rosenstiel W, Bogdan M. (2013). Online classification of
eye tracking data for automated analysis of traffic hazard perception. In Artificial Neural Net-
works and Machine Learning. ICANN 2013. 442–450. Springer Berlin Heidelberg.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17


Automated Comparison of Scanpaths in Dynamic Scenes

Online Visual Attention Monitoring for Mobile 
Assistive Systems 
Patrick Renner, Thies Pfeiffer  
1 Introduction 
Every now and then there are situations in which we are not sure how to proceed 
and thus are seeking for help. For example, choosing the best product out of dozens 
of different brands in a supermarket can be difficult, especially when following a 
specific diet. There are, however, also people who have problems with decision 
making or sequencing actions in everyday life, e.g. because they suffer from de-
mentia. In such situations, it may be welcomed when there is someone around no-
ticing our problem and offering help. In more private situations, e.g. in the bath-
room, help in shape of a human being cannot be expected or even is not welcomed. 
Our research focuses on the design of mobile assistive systems which could as-
sist in everyday life activities. Such a system needs to detect situations of helpless-
ness, identify the interaction context, conclude what would be an appropriate assis-
tance, before finally engaging in interaction with the user.  
2 Our Contribution 
To approach the problem of detecting the cognitive state of helplessness or demand 
for assistance, our idea is to observe the eye movements of the user. This way, the 
system is at the same time attentive but quiet in the background until it detects user 
behaviours which elicit assistance. Only then, the system awakes and enters in direct 
interaction with the user. 
We report about our experiences with such mobile assistance systems which we 
have gathered in two scenarios: shopping and chess playing. For the shopping situ-
ation, the assistance system has been created as a functional prototype in a 3D im-
mersive virtual supermarket scenario (Figure 1). The second system has already 
been tested in real-world scenarios: Here, computer vision-based detection of the 
configuration on a chess board provides the basis for context identification. 
In the virtual supermarket scenario, we currently investigate which basic features 
of eye movements could be used to decide in which phase of a decision process the 
users are currently in [1,2]. In a pilot user study, we tested different ways of pre-
senting augmented information to support the decision process, inter alia an ap-
proach in which additional product information is presented on simulated aug-
mented reality glasses. They appear attached to the product currently focused by the 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 6
Online Visual Attention Monitoring for Mobile Assistive Systems
user. The augmentation thus adapts in real-time to the current target object via anal-
ysis of gaze information. We report about first results of this pilot study. 
For the chess scenario, we show how we map gaze positions to locations on the 
chessboard and present a way to provide feedback on the strength of the fixated 
chess figure based on sonification. We report on a pilot study in which we tested 
the user experience of that system. 
3 Discussion 
Eye movements could provide viable information for mobile assistance systems that 
automatically adapt to the context of use and the cognitive state of the user. We 
present work of several projects and report on first pilot studies on user experience.  
ACKNOWLEDGMENTS: This work is partly funded by the BMBF in the frame of the project 
Adaptive and Mobile Action Assistance in Daily Living Activities (ADAMAAS) and by the Cluster 
of Excellence Cognitive Interaction Technology 'CITEC' (EXC 277) at Bielefeld University, 
which is funded by the German Research Foundation (DFG). 
References 
1. Gidlöf, K., Wallin, A., Dewhurst, R., & Holmqvist, K. (2013). Using eye tracking to trace a
cognitive process: Gaze behaviour during decision making in a natural environment. Journal
of Eye Movement Research, 6(1), 1–14.
2. Pfeiffer, J., Pfeiffer, T., & Meißner, M. (2015). Towards Attentive In-Store Recommender Sys-
tems. In Reshaping Society through Analytics, Collaboration, and Decision Support (S. 161–
173). Springer. Fetched from http://link.springer.com/chapter/10.1007/978-3-319-11575-7_11
Figure 1: Prototyping of the mobile assistance
system in a 3D immersive virtual supermarket. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 6


Online Visual Attention Monitoring for Mobile Assistive Systems

Jahn, Najko

Werkzeuge für die Visualisierung von Bibliotheksdaten

Ferhat, Onur

Llanza, Arcadi

Vilariño, Fernando

Gaze interaction for multi-display systems using
natural light eye-tracker
Onur Ferhat, Arcadi Llanza, Fernando Vilariño
1 Introduction
Natural light-based gaze-tracking constitutes an interesting alternative to commer-
cial eye-trackers, and it requires no additional hardware other than a camera (ubiq-
uitously present on consumer devices). Methods developed to attack this problem
include using raw eye appearance as the feature set [1], using the geometry of iris
boundary ellipse for estimation [2, 3, 4] and using more complicated geometrical
models [5, 6], among others. The problem with these methods is either they are
using unrelated information (skin pixels), or they are summarising the necessary
information too much (to model parameters).
We propose a system implementing a novel feature based approach for webcam
gaze interaction. Our system relies on the automated detection of the iris bound-
ing box, followed by iris segmentation by means of binarisation. We develop this
approach using the OpenGazer eye-tracker as a baseline.
As a validation framework, we propose a two-display context-adaptive system
which modifies in real time the data rendered in a secondary display depending on
the particular object that is observed in the main display. We particularly tune this
approach to Google-glass type devices. The application context of our research can
be straightforwardly tested and extrapolated to different augmented reading expe-
Onur Ferhat
Universitat Autònoma de Barcelona and Computer Vision Center, Edificio O Campus UAB 08193
Bellaterra e-mail: oferhat@cvc.uab.es
Arcadi Llanza
Universitat Autònoma de Barcelona and Computer Vision Center, Edificio O Campus UAB 08193
Bellaterra e-mail: arcadi.llanza@e-campus.uab.cat
Fernando Vilariño
Universitat Autònoma de Barcelona and Computer Vision Center, Edificio O Campus UAB 08193
Bellaterra e-mail: fernando@cvc.uab.es
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 4
Onur Ferhat, Arcadi Llanza, Fernando Vilariño
riences such as maps analysis, artworks detailed observation and music reading,
among others.
2 Our Approach
The system performs an automatic selection of 8 anchor points on the subject’s
face based on facial features such as eye corners, eyebrows, nose and mouth [7, 8].
Over the next frames, optical flow is used to update the points’ positions in the new
images.
The feature extraction stage takes as input the eye images extracted from the
camera image. The iris bounding box is detected with template matching and iris
segmentation is calculated using a binarisation inside this area. The features are ex-
tracted by projecting the segmented pixels into the vertical and horizontal axes and
a feature vector is created by concatenation as seen in Fig. 1. For gaze estimation,
we train a Gaussian Process with the standard squared exponential kernel tuned to
work at a desired distance during the calibration process.
(a) (b)
(c)
Fig. 1 (a) Iris segmentation (b) Vertical features calculated by projecting iris pixels to vertical axis
(c) Horizontal features
The system is complemented with a gaze-interactive interface as shown in Fig. 2.
On the first display, visual selection is natively performed by guiding the gaze to-
wards the desired object (left). Real-time output is then rendered in the secondary
display (top-right) which adapts its information to the item visually selected (in the
context of Google-glass type devices the secondary display plays the role for the
glass display).
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 4
Gaze interaction for multi-display systems using natural light eye-tracker
Fig. 2 System interface: (Left) Main visualization display where stimuli are shown. (Top-Right)
Responsive display: The information is adapted to the object visualized. (Bottom-right) Subject’s
facial reference points and segmented images from the eyes shown for illustrative purposes.
3 Discussion
The proposed method decreases the average gaze estimation error by 34% in hori-
zontal direction and by 12% in the vertical direction when compared to our previous
work on standard databases [8]. The average errors are 2.35◦and 1.82◦ for the base-
line system in both directions; whereas the proposed system achieves error rates
of 1.54◦and 1.61◦. Our experiments show that the errors are not uniform over the
screen area, and horizontal errors can be up to 65% larger near the edges compared
to the center of the screen.
The interaction obtained in this way is smooth and fast. The preliminary results
are promising, and our current work is focused on enriching the feature set by
introducing spatial feature correlation, and using other cues such as exact eye corner
positions. Regarding interaction, the definition of suitable visual action triggers that
handle with the information delivered in the secondary display, not defined in this
work, is an open area of research.
ACKNOWLEDGMENTS: This work was supported by UAB grants and the
Google Faculty Award.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 4
Onur Ferhat, Arcadi Llanza, Fernando Vilariño
References
1. Holland C, Komogortsev OV (2012). Eye tracking on unmodified common tablets: challenges
and solutions. Eye Tracking Research and Applications. 277280.
2. Wood E, Bulling A (2014). EyeTab: Model-based gaze estimation on unmodified tablet com-
puters. Eye Tracking Research and Applications. 207-210.
3. Wang JG, Sung E (2000). Gaze determination via images of irises. British Machine Vision
Conference.
4. Wu H, Chen Q, Wada T (2004). Conic-based algorithm for visual line estimation from one
image. Automatic Face and Gesture Recognition. 260265.
5. Chen J, Ji Q (2000). 3D gaze estimation with a single camera without IR illumination. Inter-
national Conference on Pattern Recognition. 14.
6. Ishikawa T, Baker S, Matthews I, Kanade T (2004). Passive driver gaze tracking with active
appearance models. World Congress on Intelligent Transportation Systems.
7. Santana MC, Déniz-Suárez O, Hernández-Sosa D, Lorenzo J (2011). A comparison of face
and facial feature detectors based on the Viola-Jones general object detection framework.
Mach. Vis. Appl. 22(3), 481494.
8. Ferhat O, Vilarino F, Sanchez FJ (2014). A cheap portable eye-tracker solution for common
setups. Journal of Eye Movement Research, 7(3):2, 1-10.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 4


Gaze interaction for multi-display systems using natural light eye-tracker

Halbrügge, Marc

Online Calibration of Gaze Tracking on the
Computer Screen using Particle Filtering
Marc Halbrügge
1 Introduction
Visual salience is usually not evenly distributed on the computer screen. Most of the
time, there is only a limited number of highly salient points instead. These points
naturally attract fixations. If the points of visual salience are known to a gaze track-
ing system, its calibration could be inspected and also enhanced on their basis. But
how should gaze be mapped to one of the salient points during this process? This
mapping can not be established from a single observation. Particle filtering (Kantas
et al., 2009) allows to maintain several hypotheses about where the user is currently
looking and integrates this with where the user has looked before, finally converging
to a most likely gaze position. We are presenting a particle filter that estimates the
calibration drift of a head-based gaze tracking system.
2 The Problem of Calibration in Eye-Tracking
Eye-tracking, whether applied in the lab or in the field, needs calibration (Poole
and Ball, 2005). While this is usually bearable for normal subjects, standard gaze
calibration procedures often fail for children or some patient groups because they
cannot keep their gaze on a fixation point for the required amount of time. In this
case, the identification of “correct” gazes during calibration has to be done by the
experimenter in a tedious and lengthy process (c.f., Franchak et al., 2011). Pursuit
calibration (Pfeuffer et al., 2013) could be a solution to this, but is still in its infancy.
How can we alleviate this situation? We propose to use particle filtering to esti-
mate the gaze position. Particle Filters have been applied to the machine vision side
of eye-tracking, i.e., iris detection and tracking, before (Hansen and Pece, 2005).
Quality & Usability Lab, Telekom Innovation Laboratories, Technische Universität Berlin,
Ernst-Reuter-Platz 7, 10587 Berlin e-mail: marc.halbruegge@tu-berlin.de
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 16
Marc Halbrügge
Here we extend this idea to the calibration and gaze estimation part of the problem.
As a starting point, we try to estimate a time dependent drift of the estimated gaze
position. Drift may be caused by, e.g., the headband being dragged down by the
weight of the tracker, or subject actions like lifting the eyebrows.
3 Gaze Controlled HTML5 Shooting Game
As empirical basis for the drift estimation, data collected using a simple gaze con-
trolled HTML5 shooting game based on Halbrügge (2015) was chosen. Task of the
game was to “shoot” rectangles of a specific color by looking at them. To make gaze
control easier, a small fixation cross was added to the center of the target objects
(see Figure 1). The game took about three minutes to play with about 250 targets to
shoot. The players’ gaze was tracked using a head-based SR Research EyeLink II
with 250 Hz sampling frequency; no chinrest was used.
The connection from the eye tracker to the game was established using SR Re-
search’s proprietary ethernet link and the web browser bridge described in Halbrügge
(2015). Gaze interaction was provided by triggering virtual clicks on the browser’s
JavaScript layer after a dwell time of 300 ms. Neither eye tracker nor display latency
was compensated as both were negligibly small relative to the dwell time.
Fig. 1 Gaze controlled HTML5 game. The players have to “shoot” colored rectangles by looking
at them. White crosses at the center of each target act as points of maximum salience.
Subsequent analysis showed that significant detoriation of the calibration hap-
pened during the gaming episodes. A fitted linear model showed substantial drift,
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 16
Online Calibration of Gaze Tracking on the Computer Screen using Particle Filtering
especially in the vertical dimension. This is consistent with the rather bulky and
heavy headmount of the gaze tracker used for the game.
4 Particle Filter Results and Conclusions
Post-hoc analysis of the data using the particle filter approach showed that the deto-
riation of the calibration could easily be counteracted. As few as 1000 particles
were enough to attain convergence and considerable error reduction. Each filter step
needed less than 2 ms computation time on a consumer laptop computer, which
would fast enough to run it in sync with the 250 Hz eye tracker used for the game.
As next step, we are planning to integrate the drift estimator into the tracking
system. This should bring us closer to the goal of less intrusive calibration of gaze
tracking systems. In the future, calibration could be performed by playing a simple
game or watching a funny stickman video.
While this research has been guided by the goal of easier eye tracking for spe-
cific subject groups like children, and although it is currently restricted to HTML
applications, it is not necessarily bound to these domains. Future applications could
include calibration-free gaze interaction with large public displays, or continuous
drift correction during mobile eye tracking.
ACKNOWLEDGMENTS: The authors gratefully acknowledges financial sup-
port from the German Research Foundation (DFG) for the project “Automatische
Usability-Evaluierung modellbasierter Interaktionssysteme für Ambient Assisted
Living” (MO 1038/18-1).
References
1. Franchak, J. M., Kretch, K. S., Soska, K. C., and Adolph, K. E. (2011). Head-
mounted eye tracking: A new method to describe infant looking. Child develop-
ment, 82(6):1738–1750.
2. Halbrügge, M. (2015). Automatic online analysis of eye-tracking data for dy-
namic HTML-based user interfaces. In Gramann, K. and Zander, T. O., edi-
tors, Berliner Werkstatt Mensch-Maschine-Systeme, Berlin. Technische Univer-
sität Berlin. accepted.
3. Hansen, D. W. and Pece, A. E. (2005). Eye tracking in the wild. Computer Vision
and Image Understanding, 98(1):155–181.
4. Kantas, N., Doucet, A., Singh, S. S., and Maciejowski, J. M. (2009). An overview
of sequential monte carlo methods for parameter estimation in general state-space
models. In 15th IFAC Symposium on System Identification (SYSID), Saint-Malo,
France.(invited paper), volume 102, page 117.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 16
Marc Halbrügge
5. Pfeuffer, K., Vidal, M., Turner, J., Bulling, A., and Gellersen, H. (2013). Pursuit
calibration: Making gaze calibration less tedious and more flexible. In Proc. of the
26th ACM Symposium on User Interface Software and Technology (UIST 2013),
pages 261–270, New York, NY, USA. ACM.
6. Poole, A. and Ball, L. J. (2005). Eye tracking in human-computer interaction and
usability research: Current status and future. In Ghaoui, C., editor, Encyclopedia
of Human-Computer Interaction, pages 211–219, Hershey, PA. Idea Group.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 16


Online Calibration of Gaze Tracking on the Computer Screen using Particle Filtering

Salb, Simon

Splitt, Markus

Bandow, Nicole

Witte, Kerstin

The influence of spatial occlusion on visual 
search behavior of karate athletes 
Simon Salb, Markus Splitt, Nicole Bandow, Kerstin Witte 
1 Introduction/Related Work 
In martial arts, the ability to anticipate and response appropriately has to be seen 
as a performance determinant. Common methods in anticipation research are tem-
poral and spatial occlusion, as well as eye tracking  (Panchuk & Vickers, 2009; 
Savelsbergh, Williams, van der Kamp & Ward, 2002). For studying anticipatory 
skills in karate the presentation of attack scenes on a life size video screen (Wil-
liams & Elliott, 1999) using spatio-temporal occlusion proved to be a valid meth-
od (Bandow, Emmermacher, Stucke, Masik & Witte, 2014; Zerbe, E., Kirbach, 
M., Bandow, N., Emmermacher, P., Witte, K., 2013; Mori, Ohtani & Imanaka, 
2002). Since there is evidence that spatial occlusion can influence the visual be-
havior of athletes (Hagemann, Schorer, Cañal-Bruland, Lotz & Strauss, 2010) it is 
important to consider this method critically. Prior studies did not record eye 
movements while applying spatial occlusion for studying anticipation in karate 
(Bandow et al., 2014). The aim of the present study is to explore the influence of 
spatial occlusion on the visual search behavior of karate athletes. 
2 Our Contribution 
Seven karate athletes with experience in national and international competition re-
sponded physically to life size videos of a karate reverse punch shown on a back 
projection screen (2 m x 2.5 m). Spatial occlusion was conducted by covering rel-
evant body regions (hip, punching arm, front leg) of the opponent in the video se-
quences with background images. The participants’ visual behavior was recorded 
with a binocular head mounted eye tracker (SensoMotoric Instruments; model: 
SMI Eye Tracking Glasses). Each video sequence was presented three times in 
randomized order to prevent a learning effect. Eye tracking data was analyzed us-
ing the software BeGaze (SensoMotoric Instruments, Teltow, Germany). There-
fore nine areas of interest (AOI) were defined representing nine regions of the op-
ponents’ body relevant for karate kumite. As dependent measures relative fixation 
time per AOI, mean fixation duration per trial and the number fixations per second 
in each trial were chosen (Dicks, Button & Davids, 2010). A reliability analysis 
revealed no significant differences between three repeated measurements. By 
means of one-way ANOVA/ Kruskal-Wallis test significant differences between 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 18
The influence of spatial occlusion on visual search behavior of karate athletes 
the occlusion conditions were tested. The results matched with the findings of 
Williams & Elliott (1999) and showed primary fixation of the head and upper tor-
so under all conditions (e.g. 58% head and 22% upper torso under non occluded 
condition). There was no difference in visual search behavior between occluded 
and non occluded stimuli. Neither mean fixation duration [χ²(3)=4.543, p=.208] 
nor number of fixations per second [F(3,24)=1.747, p=.184] showed significant 
differences between the occluded and non occluded conditions. Analyzing relative 
fixation duration per AOI confirmed those findings. For example, the relative fixa-
tion duration of the head did not significantly differ between conditions 
[F(3,24)=.232, p=.873]. 
3 Discussion 
Due to the small sample size (n=7), variations between the subjects influence the 
results. Therefore it is required to gather more data in order to make general con-
clusions. It will be necessary to conduct further studies occluding regions of pri-
mary fixations (head, upper torso) to determine the influence of spatial occlusion 
on visual search of karate athletes. As long as spatial occlusion is conducted by 
making body regions invisible that are no main fixation locations, there seems to 
be no influence on visual search behavior.  
References 
1. Bandow, N., Emmermacher, P., Stucke, C., Masik, S. & Witte, K. (2014). Comparison of a
video and a virtual based environment using the temporal and spatial occlusion technique for
studying anticipation in karate. International Journal of Computer Science in Sport, 13
(SPLISSUE.1), 44–56.
2. Dicks, M., Button, C. & Davids, K. (2010). Examination of gaze behaviors under in situ and
video simulation task constraints reveals differences in information pickup for perception and
action. Attention, Perception, & Psychophysics, 72 (3), 706–720.
3. Hagemann, N., Schorer, J., Cañal-Bruland, R., Lotz, S. & Strauss, B. (2010). Visual percep-
tion in fencing: do the eye movements of fencers represent their information pickup? Atten-
tion, perception & psychophysics, 72 (8), 2204–2214.
4. Mori, S., Ohtani, Y. & Imanaka, K. (2002). Reaction times and anticipatory skills of karate 
athletes. Human Movement Science, 21 (2), 213-230.
5. Panchuk, D. & Vickers, J.N. (2009). Using spatial occlusion to explore the control strategies
used in rapid interceptive actions: Predictive or prospective control? Journal of sports scienc-
es, 27 (12), 1249–1260
6. Savelsbergh, G. J., Williams, A. M., van der Kamp, J. & Ward, P. (2002). Visual search, antic-
ipation and expertise in soccer goalkeepers. Journal of sports sciences, 20 (3), 279–287.
7. Williams, A.M. & Elliott, D. (1999). Anxiety, expertise, and visual search strategy in karate.
Journal of Sport & Exercise Psychology, 21 (4), 362–375.
8. Zerbe, E., Kirbach, M., Bandow, N., Emmermacher, P., Witte, K. (2013). Comparison of tem-
poral and spatial occluded video karate attacks with regard to predictability. In Fac. of Elec-
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 18
Simon Salb, Markus Splitt, Nicole Bandow, Kerstin Witte
trical Engineering (Hrsg.), International Scientific- Professional Symposium INFOTEH (Vol. 
12, S. 944–950). 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 18


The influence of spatial occlusion on visual search behavior of karate athletes

Kohorst, Petra

Peil, Vitali

PUB als responsive Webapplikation auf Basis von
ElasticSearch und Catmandu
Petra Kohorst, Vitali Peil 8. Mai 2015 Kolloquium Wissensinfrastruktur
Einführung
Open Access Repository Ranking
• 2014 Open Access Repository Ranking
• Repository-Landschaft in Deutschland
• PUB im Ranking
“Repository” als Sammelbegriff
• Institutionelles Repository zur Unterstützung des grünen OA
• Publikationsmanagementsystem
• Hochschulschriftenserver: Primärveröffentlichungen von Hochschulschriften
im Open Access
• Forschungsdatenrepository
• Hochschulbibliographie
Ziele
• Meldeverhalten verbessern
• Erhöhung der OA-Publikationen
• Einfacher Weg zur Publikationsliste
• verbesserte technische Basis für Weiterentwicklungen
1
Das Bisherige System
• Perl + MySQL
• Erstellt in Lund, erweitert in Gent
• Übernommen 2010 von Bielefeld und seit dem von allen dreien erweitert
• 2012 Frontend auf ElasticSearch umgestellt
Warum Neuentwicklung?
• Das System ist inzwischen veraltet, langsam und schwierig zu warten
• Sehr umfangreiche Perl-Sourcen
• Komplizierte Struktur der MySQL-Datenbank
• Überflüssige Features und Altlasten aus Lund und Gent
• Unterschied zwischen Außensicht (Personenseite) und Backend
Responsives Design? - Fehlanzeige
Welche Systeme verwenden wir?
• Sprache: Perl
• Tools: Catmandu, Template Toolkit, Twitter Bootstrap
• Datenbank: ElasticSearch, MySQL als Backup
2
3
Vereinfachung und Vereinheitlichung
• Ziel: Der einfache Weg zur Publikationsliste
• Zitationen, Filter und Suche wie im Frontend nötig
• ALLE Einträge müssen indexiert werden (auch die nicht-öffentlichen)
• Backend muss aussehen wie die öffentliche Personenseite
• Angezeigte Daten sollen an Ort und Stelle editierbar sein
Vereinfachung und Vereinheitlichung
• Preview-Seite auch für nicht-öffentliche Einträge
• Überprüfen aller Angaben im “geschützten” Umfeld möglich
• Wenn alle Daten ok sind, “make public” direkt von der Preview-Seite
Responsives und Verständliches Design
Responsives Design:
• Frontdoor, Personenseite, Webseite allgemein
• Backend, Eingabeformulare
Verständliches Design:
• Eingabeformulare mit Placeholdern, Labels, Beispieltexten und ohne kryp-
tische Bezeichner
• Eingabeformulare aufgeräumt, alles was für die Zitation wichtig ist, steht
oben
4
Importe und Uploads
• Bevorzugt Importe über IDs (DOI, Pubmed, ArXiv, Inspire)
• Besonderer Fokus auf OA Volltexte
• Datei-Upload rückt weit nach vorne und wird vereinfacht durch “Drag &
Drop”
Neue Features - Dropzones und Quick and Easy Uploads
• Neu: Quick and Easy Dateiupload
• Service des PUB-Teams: Vervollständigung anhand des pdfs
• Aber auch: der schnellste Weg zum PUB-Eintrag, ohne viele Klicks
• Auch für PUB Theses genutzt
• Zusätzlich: Neuer Access-Level Request-a-Copy
Ausblick
• Open Source Release auf GitHub unter dem Namen LibreCat
• Catmandu-Tools in Action
• Flagship-application des LibreCat-Projekts
Vielen Dank!
Gibt es Fragen?
5


Not a member yet