BieColl - Bielefeld Electronic Collections

Pietsch, Christian

Summann, Friedrich

Jahn, Najko

Nutzungsstatistiken für Webanwendungen der Universitätsbibliothek

Menke, Peter

Langzeitarchivierung von Forschungsdaten aus dem SFB 673

Gidlöf, Kerstin

Lingonblad, Martin

Wallin, Annika

Effect  of  Familiarity  on  Visual  Attention  and
Choice at the Point of Purchase
Kerstin Gidlöf, Martin Lingonblad, and Annika Wallin
1 Introduction
Being familiar with an environment can have great effects on our eye movements
and visual attention. With experience, we learn to attend to things that are impor-
tant to us and to ignore less relevant information (Droll, Gigone & Hayhoe, 2007;
Haider & Frensch, 1999; Jovancevic-Misic & Hayhoe, 2009; Meisner & Decker,
2010). This is one of the reasons why it is preferable to take studies out of the lab
and into more natural environments. The present study investigated the effect of
familiarity on visual attention and choice of consumers doing their grocery shop-
ping. In a familiar environment where we make decisions repeatedly,  our visual
attention will change over time. For instance, familiarity with the task and task en-
vironment reduces time and effort and the relative influence of bottom up factors
will decrease (Orquin, Bagger & Loose, 2013).
2 Experiment
The eye movements of fifty consumers were recorded in their familiar supermar-
ket (group 1). All participants were instructed to buy a product from three differ -
ent product categories during their regular shopping. These consumers were later
recorded in another, unfamiliar supermarket of the same supermarket chain, again
instructed to buy products from the same categories. A control group of twenty
five consumers, familiar with the second supermarket, was also recorded (group
2), performing the same task, to determine if there was in fact familiarity and not
differences in shelf organization or any other peculiarities of the second supermar-
ket that made an impact on the results.
The SMI ETG glasses were used for this study, recording binocular eye move-
ments at a rate of 30Hz. The collected eye tracking data was analysed manually 
frame-by- frame, using semantic gaze mapping in the SMI BeGazeTM Software 
by four independent coders. The fixations on AOI’s where then divided into 
dwells according to the definition stated in Gidlöf et al. (2013) i.e. all fixations 
within an AOI for a duration of at least 120msec.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 13
Effect of Familiarity on Visual Attention and Choice at the Point of Purchase  
The quality of the decision was calculated by having participants fill out a ques-
tionnaire of how important each attribute of a product category were for them 
when choosing a product. By summing up the values each product got an option 
quality (Gidlöf et al., 2013).
3 Preliminary Results and Discussion
All results are based on two of the three product categories since the third category
is still being analysed. In general participants were more familiar with the yogurt
category compared to the pasta category.  Participants did spend more time in-
specting the products in the unfamiliar supermarket compared to the familiar one,
specifically in the pasta category. A paired t-test on group1 suggests an increase in
total time between group1 in supermarket No.1 and in supermarket No.2. This dif-
ference was also reflected in the quality of their choices with participants making
significantly better choices in pasta category in their familiar supermarket com-
pared to the unfamiliar supermarket. The results showed no general differences 
between group 1 in supermarket no.1 and the control group in supermarket no.2.
The present findings are a first step in examining the effect of familiarity on visual
attention and choice. These effects will be investigated further by for example 
studying the top-down and bottom-up influences on visual attention in these envi-
ronments.
ACKNOWLEDGMENTS: The authors would like to thank Erik Jansson for his contributions
in recording and analysis of the data.
References
1. Droll, J. A.,  Gigone, K.,  & Hayhoe, M. M. (2007).  Learning where to direct gaze during
change detection. Journal of Vision, 7(14), 6.
2.  Haider, H., & Frensch, P. A. (1999). Eye movement during skill acquisition: More evidence
     for the information-reduction hypothesis.
     Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(1), 172.
3.  Jovanevic-Misic, J., & Hayhoe, M. M. (2009) Adaptive Gaze Control in Natural
     Environ ments. Journal of Neuroscience, 29, 6234 – 6238
4.  Meissner, M., & Decker, R. (2010). Eye-tracking information processing in choice-based
     conjoint analysis. International Journal of
Market Research, 52(5).
5. Orquin, J. L., Bagger, M. & Loose, S. M. (2013),”Learning affects top down and bottom up
    modulation of eye movements in decision making”, Judgment and Decision Making, 8(6),    
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 13
Kerstin Gidlöf, Martin Lingonblad, and Annika Wallin
     700-716.
6.  Gidlöf, K., Wallin, A., Møgelvang-Hansen, P., & Holmqvist, K. (2013). Material Distortion
     of Economic Behaviour and Everyday Decision Quality. Journal of Consumer Policy, 36(4), 
     389-402.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 13


English

Effect of Familiarity on Visual Attention and Choice at the Point of Purchase

Werkzeuge für die Visualisierung von Bibliotheksdaten

Pfeiffer-Leßmann, Nadine

Renner, Patrick

Pfeiffer, Thies

Analyzing Patterns of Eye Movements  
in Social Interactions 
Nadine Pfeiffer-Leßmann, Patrick Renner, Thies Pfeiffer  
1 Introduction 
Eye gaze plays an important role in human communication. One foundational skill 
in human social interaction is joint attention which is receiving increased interest in 
particular in the area of human-agent or human-robot interaction. We are focusing 
here on patterns of gaze interaction that emerge in the process of establishing joint 
attention. The approach, however, should be applicable to many other aspects of 
social communication in which eye gaze plays an important role. 
Attention has been characterized as an increased awareness [1] and intentionally 
directed perception [2] and is judged to be crucial for goal-directed behavior. Joint 
attention can be defined as simultaneously allocating attention to a target as a con-
sequence of attending to each other's attentional states [3]. In other words: Interloc-
utors have to deliberatively focus on the same target while being mutually aware of 
sharing their focus of attention [2] [4]. 
2 Our Contribution 
2.1. Study 
In a study with 18 pairs of participants (i.e. 36 different participants in total), we 
recorded eye movements while participants were solving a visual search task based 
on a verbal description of the experimenter and a communication task where they 
both had to cooperate on agreeing on a specific target stimulus. 
2.2. Automatic labeling of fixation targets 
Each pair of participants had to solve 32 tasks, which took about 30 minutes, result-
ing in 18 hours of mobile eye movement videos. Annotation time was reduced to 
zero by mapping from fixations to semantic regions automatically based on the Eye-
See3D approach [5]. 
2.3. Analysis of interaction patterns 
Our approach for the analysis is to define several indices to describe basic gazing 
behaviors, such as  
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10
Analyzing Patterns of Eye Movements in Social Interactions 
 Searching | 	 |
| |
→ 1
 Identifying | 	 |
| |
→ 1
 Looking at partner | |
| |
→ 1
 Looking for help | 	 | | |
| |
→ 1
 Helping ≫ 	 	  
 Following 	 	 ≫ 	 	 	  
This provides an abstraction of the concrete gazing behavior, which will then be 
used for the analysis of dyadic patterns. We will present the results of this analysis 
and discuss our approach with existing alternatives. 
3 Discussion 
We present an approach to characterize and analyze eye movement data in a way 
that is compatible with an online analysis. The EyeSee3D approach [5] can be ap-
plied to studies in the domain of social interactions, to significantly reduce the time 
required for the analysis of the eye movement data. In combination, the index-based 
classification and EyeSee3D will allow us to interpret gaze patterns of interacting 
persons in real-time, which is highly relevant for realizing human-agent interac-
tions. 
ACKNOWLEDGMENTS: This research was partly supported by the Cluster of Excellence Cog-
nitive Interaction Technology 'CITEC' (EXC 277) and the SFB 673 Alignment in Communication; 
both are at Bielefeld University and funded by the German Research Foundation (DFG). 
Figure 1: Interaction scenario of our study: Two participants, both eye-tracked, have to solve 
cooperative tasks. Eye movements were recorded to analyze patterns of eye gaze interaction.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10
 Nadine Pfeiffer-Leßmann, Patrick Renner, Thies Pfeiffer 
References 
1. Brinck, I. (2003): The objects of attention. In: Proc. of ESPP2003, Torino.
2. Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H. (2005): Understanding
and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences
28
3. Deak, G.O., Fasel, I., and Movellan, J. (2001): The emergence of shared attention: Using
robots to test developmental theories. In: Proc. of the First Intl. Workshop on Epigenetic
Robotics, Lund University, Cognitive Studies, 85.
4. Hobson, R.P. (2005): What puts the jointness into joint attention? In Eilan, N., Hoerl, C.,
McCormack, T., Roessler, J., eds.: Joint attention: communication and other minds. Ox-
ford University Press.
5. Pfeiffer, T., & Renner, P. (2014). EyeSee3D: a low-cost approach for analysing mobile
3D eye tracking data using augmented reality technology. Proceedings of the Symposium
on Eye Tracking Research and Applications, 195–202.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 10


Analyzing Patterns of Eye Movements in Social Interactions

Kübler, Thomas C.

Kasneci, Enkelejda

Automated Comparison of Scanpaths in
Dynamic Scenes
Thomas C. Kübler and Enkelejda Kasneci
1 Introduction
According to the scanpath theory [6], the sequence of eye movements can give in-
sights into the cognitive processes of visual perception. In this light, the comparison
of scanpaths is of particular interest for many applications, e.g., driving [4], advertis-
ing, or activity recognition [1]. Although precise measurement devices and several
approaches for the extraction of the visual scanpath in dynamic scenes exist, e.g. [7],
the temporal and spatial comparison of scanpaths with their natural high variabil-
ity remains challenging and laborious. Despite the variety of existing approaches,
such as ScanMatch [2] or Multimatch [3], most scanpath studies are still limited to
the comparison of time-integrated features, e.g. fixation duration or average saccade
length. Due to the sequential nature of the scan pattern, such metrics are not suffi-
cient to capture essential pattern characteristics of the viewing behavior. Recently,
we proposed SubsMatch [5], a novel method for scanpath comparison in natural
environments. In this work, we will demonstrate the performance of SubsMatch in
comparison to the state-of-the-art when analyzing eye-tracking data derived from a
driving scenario.
Thomas C. Kübler
University of Tübingen, Computer Engineering Department, Sand 13, 72076 Tübingen, Germany
e-mail: thomas.kuebler@uni-tuebingen.de
Enkelejda Kasneci
University of Tübingen, Computer Engineering Department, Sand 13, 72076 Tübingen, Germany
e-mail: enkelejda.kasneci@uni-tuebingen.de
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17
Thomas C. Kübler and Enkelejda Kasneci
2 SubsMatch Algorithm
Instead of addressing the issue whether two scanpaths focus on the same objects
at the same time, SubsMatch addresses the question whether the scanpaths exhibit
the same repetitive exploratory patterns. Figure 1 illustrates the main steps of Subs-
Match as introduced in [5]. First, the scanpath is mapped to a string representation
based on just one dimension of the scanpath, e.g. the horizontal position of fixations.
Letter encoding is chosen to optimally map the scanpath data into equally frequent
letters. This step also compensates for spatial offset and scaling, often caused by
incorrect calibration, different seat positions while driving or different viewing dis-
tances towards the scene. The second step extracts all possible visual subpatterns
from the encoded scanpath. The problem of scanpath comparison is thereby reduced
to the comparison of the frequency of subpatterns.
Fig. 1 Overview of the SubsMatch algorithm.
3 Exemplary Application
SubsMatch was applied to scanpaths derived from simulated driving sessions of 10
subjects with advanced glaucoma and 10 control subjects. Eye movements were
recorded using a Dikablis eye tracker at 25Hz. 50% of the drivers with visual field
defects were able to pass the driving test despite their visual impairment. This result
indicates that the viewing behavior of patients who passed the test differs from the
viewing behavior of patients who failed. Glaucoma subjects who passed (Gp) and
failed (G f ) were compared to each other and to the control subjects (GC) to iden-
tify differences in the viewing patterns by means of SubsMatch, ScanMatch [2] and
MutliMatch [3]. Table 1 shows the statistical evaluation of the pairwise scanpath
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17
Automated Comparison of Scanpaths in Dynamic Scenes
Table 1 Comparison of the pairwise distances within and between scanpath groups (G for glau-
coma patients, p/f for passed or failed the driving test, C for control subjects).
{G f ,Gp} {GC,G f } {GC,Gp}
SubsMatch 0.03 0.03 0.67
ScanMatch 0.05 0.05 0.65
MultiMatch 0.89 0.89 0.89
distances between the groups. SubsMatch revealed that there are no additional com-
pensatory eye movements performed by the Gp group, as often assumed. Instead,
this group was able to maintain a normal exploratory behavior, similar to that in the
GC group. The G f group on the other hand exhibit-ed fewer horizontal, alternating
saccades.
4 Conclusion
The presented algorithm is a significant step towards automated analysis and com-
parison of visual scanpaths derived from eye-tracking data in natural environments.
Beyond further developments of SubsMatch, our future work will focus on the ques-
tion, how the analysis of recurrent patterns can be used to provide added value for
several applications.
References
1. Braunagel C, Kasneci E, Stolzmann W, Rosenstiel W. (2015). Driver-Activity Recognition in
the Context of Conditionally Autonomous Driving. IEEE 18th International Conference on
Intelligent Transportation Systems (ITSC), 2015
2. Cristino F, Matht S, Theeuwes J, Gilchrist ID. (2010). ScanMatch: a novel method for com-
paring fixation sequences. Behavior research methods, 42(3):692-700.
3. Dewhurst R, Nyström M, Jarodzka H, Foulsham T, Johansson R, Holmqvist K. (2012). It
depends on how you look at it: scanpath comparison in multiple dimensions: MultiMatch, a
vector-based approach. Behavior research methods 44(4):1079-100.
4. Kasneci E. (2013). Towards the Automated Recognition of Assistance Need for Drivers with
Impaired Visual Field. PhD thesis,University of Tübingen, Wilhelmstr. 32, 72074 Tübingen.
5. Kübler TC, Kasneci E, Rosenstiel W. (2014). Subsmatch: Scanpath similarity in dynamic
scenes based on subsequence frequencies. In Proceedings of the Symposium on Eye Tracking
Research and Applications, ETRA 2014, 319–322, ACM.
6. Noton D, Stark L. (1971). Scanpaths in saccadic eye movements while viewing and recogniz-
ing patterns. Vision research 11(9):929-42.
7. Tafaj E, Kübler TC, Kasneci G, Rosenstiel W, Bogdan M. (2013). Online classification of
eye tracking data for automated analysis of traffic hazard perception. In Artificial Neural Net-
works and Machine Learning. ICANN 2013. 442–450. Springer Berlin Heidelberg.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 17


Automated Comparison of Scanpaths in Dynamic Scenes

Gontar, Patrick

Mulligan, Jeffrey B.

A Metric to Quantify Shared Visual Attention in
Two-Person Teams
Patrick Gontar, Jeffrey B. Mulligan
1 Introduction
Critical tasks in high-risk environments are often performed by teams, the mem-
bers of which must work together efficiently. In some situations, the team mem-
bers may have to work together to solve a particular problem, while in others it
may be better for them to divide the work into separate tasks that can be com-
pleted in parallel. We hypothesize that these two team strategies can be differenti-
ated on the basis of shared visual attention, measured by gaze tracking.
2 Method
Gaze recordings were obtained for two-person flight crews flying a high-fidelity
simulator (Gontar & Hoermann, 2014). Inherent for those teams is that they nor-
mally do not face each other while working in time-critical situations (in contrast
to infant-parents’ shared attention research, such as Redcay,  Kleiner, and Saxe,
2012). Gaze was categorized with respect to 12 areas of interest (AOIs). We used
these data to construct  time series of 12 dimensional vectors,  with each vector
component representing one of the AOIs. At each time step, each vector compo-
nent was set to 0, except for the one corresponding to the currently fixated AOI,
which was set to 1. This time series could then be averaged in time, with the aver-
aging window time (∆t) as a variable parameter. For example, when we average
with a ∆t of one minute, each vector component represents the proportion of time
that the corresponding AOI was fixated within the corresponding one minute in-
terval. We then computed the Pearson product-moment correlation coefficient be-
tween the gaze proportion vectors for each of the two crew members, at each point
in time, resulting in a signal  representing the time-varying correlation between
gaze behaviors. We determined criteria for concluding correlated gaze behavior
using two methods:  first,  a  permutation test  was  applied to  the subjects'  data.
When one crew member's gaze proportion vector is correlated with a random time
sample from the other crew member's data, a distribution of correlation values is
obtained  that  differs  markedly  from the  distribution  obtained  from temporally
aligned samples. In addition to validating that the gaze tracker was functioning
reasonably well, this also allows us to compute probabilities of coordinated behav-
ior for each value of the correlation. As an alternative, we also tabulated distribu-
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 20
A Metric to Quantify Shared Visual Attention in Two-Person Teams
tions of correlation coefficients for synthetic data sets, in which the behavior was
modeled as a first-order Markov process, and compared correlation distributions
for identical processes with those for disparate processes, allowing us to choose
criteria and estimate error rates.
3 Discussion
Our method of gaze correlation is able to measure shared visual attention, and can
distinguish between activities involving different instruments. We plan to analyze
whether pilots’ strategies of sharing visual attention can predict performance. Pos-
sible measurements of performance include expert ratings from instructors, fuel
consumption, total  task time, and failure rate.  While developed for  two-person
crews, our approach can be applied to larger groups, using intra-class correlation
coefficients instead of the Pearson product-moment correlation.
ACKNOWLEDGMENTS: This  work  was  supported  by  the  TUM  Graduate
School and a LuFo IV-2 grant (PG), and NASA's Airspace Operations and Safety
Program (JBM). The authors are grateful to Immanuel Barshi for his help and sup-
port.
References
1. Gontar, P., & Hoermann, H.-J. (2014). Flight Crew Performance and CRM Ratings
Based on Three Different Perceptions. In A. Droog (Ed.), Aviation Psychology: facili-
tating change(s): Proceedings of the 31st EAAP Conference (pp. 310–316). Malta.
2. Redcay, E., Kleiner, M., & Saxe, R. (2012). Look at this: the neural correlates of initi-
ating and responding to bids for joint attention.  Frontiers in human neuroscience, 6,
169.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 20


A Metric to Quantify Shared Visual Attention in Two-Person Teams

Memili, Cem

Capturing and Visualizing Eye Movements  
in 3D Environments 
Thies Pfeiffer, Cem Memili, Patrick Renner 
Figure 1: The presented approach is able to map fixations to 3D models of the environ-
ment and create 3D heatmaps in real-time. The example shows a 3D gaze recording in an 
immersive virtual environment (person to the left inspecting 3D object on the right). Real-
time heatmap generation is shown in the left background (wrong colors due to photography 
of a projection screen). Lower left corner shows a rendering of the created 3D heatmaps. 
1 Introduction 
Visual attention can be a viable source of information to assess human behaviors in 
many different contexts, from human-computer interaction, over sports or social 
interactions, to complex working environments, such as to be found in the context 
of Industry 4.0. In such scenarios in which the user is able to walk around freely, 
mobile eye-tracking systems are used to record eye movements, which are then 
mapped onto an ego-perspective video. The analysis of such recordings then re-
quires large efforts for manually annotating the recorded videos on a frame-by-
frame basis to label the fixations based on their locations to the target objects present 
in the video. There are several problems scientists are faced: 
 Manual annotations are cumbersome and error prone. The time required for
the annotation of the data renders many studies unfeasible.
 The annotated material consists of 2D videos of a 3D environment, which is
an abstraction and the results are difficult to integrate. In particular, it is dif-
ficult to create appropriate visualizations for the collected data.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 3
Capturing and Visualizing Eye Movements in 3D Environments 
2 Our Contribution 
First, we present a method to record eye movements in 3D scenarios and annotate 
fixations with corresponding labels for the objects of interest in real-time [2]. For 
this purpose, we rely on computer-vision methods for the detection of the camera 
position and orientation in the world. Based on a coarse 3D model of the environ-
ment, representing the 3D areas of interest, fixations are mapped to areas of interest. 
As a result, we can identify the position of the fixation in terms of local object co-
ordinates for each relevant object of interest. 
Second, we present a method for real-time creation and visualization of heatmaps 
for 3D objects [1]. Based on a live-streaming of the recorded and analyzed eye 
movements, our solution renders heatmaps on top of the object surfaces. The result-
ing visualizations are more realistic than standard 2D heatmaps, in that we consider 
occlusions, depth of focus and dynamic moving objects. 
Third, we present a new method which allows us to aggregate fixations on a per 
object basis, e.g. similar to regions/areas of interest. This allows us to transfer ex-
isting methods of analysis to 3D environments. 
We present examples from a virtual supermarket, a study on social interactions 
between two humans, examples from real-time gaze mapping on body parts of a 
moving humans and from studying 3D prototypes in a virtual reality environment. 
3 Discussion 
The presented work covers the full workflow from recording to aggregating, ana-
lyzing and visualizing data on visual attention over multiple participants. The tool-
chain is designed in such a way, that the results can be gathered in real-time, during 
the recording of the study. This not only significantly reduces the time required for 
the conduction of the studies. It also increases the quality of the recordings, as prob-
lems can be detected already during the recording session. The annotation is also 
objectified and thus random errors by human annotators are eliminated. 
ACKNOWLEDGMENTS: This research was supported by the Cluster of Excellence Cognitive 
Interaction Technology 'CITEC' (EXC 277) at Bielefeld University, which is funded by the Ger-
man Research Foundation (DFG). 
References 
1. Pfeiffer, T., & Memili, C. (2015). GPU-accelerated Attention Map Generation for Dynamic 3D
Scenes. In T. Höllerer, V. Interrante, A. Lécuyer, & J. E. S. II (Eds.),Proceedings of the IEEE
VR 2015 (pp. 257–258). IEEE.
2. Pfeiffer, T., & Renner, P. (2014). EyeSee3D: a low-cost approach for analysing mobile 3D eye
tracking data using augmented reality technology. Proceedings of the Symposium on Eye
Tracking Research and Applications, 195–202.
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 3


Capturing and Visualizing Eye Movements in 3D Environments

Draschkow, Dejan

Võ, Melissa L.-H.

Collecting memories: the impact of active object 
handling on recall and search times
Dejan Draschkow
1
 & Melissa L.-H. Võ
1 
Johann Wolfgang Goethe-Universität, Frankfurt, Germany
 1 
1 Introduction 
In natural behavior we not only visually inspect our environment, but often active-
ly interact with objects which are part of it. Our perception and cognition is modu-
lated by actual as well as planned actions towards objects (for a review see Witt & 
Riley, 2014). Priorities for selection and representations of a purely visual task do 
not reflect those present in a natural task in which objects are actively manipulated 
(Tatler et al., 2013). There is a strong dependence of memory representations on 
behavioral goals (Droll, Hayhoe, Triesch, & Sullivan, 2005; Droll & Hayhoe, 
2007; Triesch, Ballard, Hayhoe, & Sullivan, 2003), which becomes particularly 
evident in natural behavior (Tatler & Land, 2011). 
2 Our Contribution 
To investigate how physically engaging with objects influences both identity 
memory (free recall) and location memory (subsequent search for these objects), 
we constructed a real-world paradigm in which participants equipped with a mo-
bile eye tracker (SMI Eye Tracking Glasses) first either searched for cued objects 
via verbal response without object interaction (Passive condition) or actively col-
lected the objects they found (Active condition) (Figure 1). Participants conducted 
this task within an actual four-room apartment (kitchen, bedroom, living room, 
and study); with each room containing 10 designated Active, Passive, and Distrac-
tor objects. Additionally, a unique category of objects was designated as relevant 
for each room (e.g. “objects needed for making a sandwich” in the kitchen) and 
participants were instructed to decide if an object was relevant upon finding it. Af-
ter the 80 trials were completed, a surprise recall test was conducted in which par-
ticipants were asked to list every object they remembered from the rooms in order 
to access identity memory performance. After completion, participants had to 
search trial-by-trial for all previously searched (Passive), searched + collected 
(Active), never searched but present (Distractors), and not included (Absent) ob-
jects. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 11
Collecting memories: the impact of active object handling on recall and search times. 
Figure 1: A depiction of the experimental procedure. The top part of the figure il-
lustrates an example trial in the Packing Phase (80 trials): in the Passive as well as Ac-
tive condition participants responded as soon as they found the target object, but par-
ticipants were instructed to collect the object only in the Active condition. In both 
condition the participants had to indicate if the object was relevant. Experimental 
conditions and object searches were randomized in a trial-by-trial fashion. In the sub-
sequent Free Recall Phase participants wrote down all objects they remembered. In 
the final Search Phase a randomized search of all the objects from the Packing Phase 
followed, supplemented with Distractor and Absent trials. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 11
Dejan Draschkow & Melissa L.-H. Võ 
3 Discussion 
Identity memory was substantially modulated by task relevance, with a higher 
memory performance for relevant objects in both the Active and the Passive con-
dition. Distractors were recalled less than targets, but were modulated by 
relevance as well, i.e. there was a higher recall performance for relevant dis-
tractors, even though they were never the target of a search.  
Time to first fixation, as an indicator of location memory, was shorter for task-
relevant objects following physical interaction (Active condition), but did not dif-
fer between relevant and irrelevant objects in the Passive condition. Time to first 
fixation for Distractors was slower than for targets. Task-relevant Distractors were 
fixated faster than irrelevant ones. 
In the current study we demonstrate that active object manipulation interacts with 
task-relevance. Time to first fixation on relevant objects was faster compared to ir-
relevant ones, but only if these objects were previously manipulated. 
ACKNOWLEDGMENTS: This work was supported by Deutsche Forschungsgemeinschaft 
(DFG) Grant VO 1683/2-1 to MLV. We wish to thank Sage Boettcher, Daniela Gresch, Maximi-
lan Scheuplein and Leonie Polzer for valuable help with data collection and analysis. 
References 
Droll, J. A., & Hayhoe, M. M. (2007). Trade-offs between gaze and working memory use. Journal of Experimental 
Psychology. Human Perception and Performance, 33(6), 1352–65. doi:10.1037/0096-1523.33.6.1352 
Droll, J. A., Hayhoe, M. M., Triesch, J., & Sullivan, B. T. (2005). Task demands control acquisition and storage of 
visual information. Journal of Experimental Psychology. Human Perception and Performance, 31(6), 1416–
38. doi:10.1037/0096-1523.31.6.1416
Tatler, B. W., Hirose, Y., Finnegan, S. K., Pievilainen, R., Kirtley, C., & Kennedy, A. (2013). Priorities for selection 
and representation in natural tasks. Philosophical Transactions of the Royal Society of London. Series B, 
Biological Sciences, 368(1628), 20130066. doi:10.1098/rstb.2013.0066 
Tatler, B. W., & Land, M. F. (2011). Vision and the representation of the surroundings in spatial memory. 
Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 366(1564), 596–
610. doi:10.1098/rstb.2010.0188
Triesch, J., Ballard, D. H., Hayhoe, M. M., & Sullivan, B. T. (2003). What you see is what you need. Journal of 
Vision, 3(1), 86–94. doi:10:1167/3.1.9 
Witt, J. K., & Riley, M. A. (2014). Discovering your inner Gibson: Reconciling action-specific and ecological 
approaches to perception–action. Psychonomic Bulletin & Review, 21(6), 1353–1370. doi:10.3758/s13423-
014-0623-4
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 11


Collecting memories: the impact of active object handling on recall and search times

Paletta, Lucas

Schwarz, Stephanie

Bobeth, Jan

Schwarz, Michael

Tscheligi, Manfred

Gaze Analysis in Mobile Pedestrian Navigation: 
Socio-Cultural Aspects and Wayfinding  
Lucas Paletta
1
, Stephanie Schwarz
2
, Jan Bobeth
2
, 
Michael Schwarz
1
, and Manfred Tscheligi
2
 
1
DIGITAL – Institute for Information and Communication Technology, 
JOANNEUM RESEARCH Forschungsgesellschaft mbH, 8010 Graz, Austria 
2
AIT Austrian Institute of Technology GmbH, Vienna, Austria 
1 Introduction 
Using augmented reality is recognized as a suitable alternative to map-based in-
terfaces for mobile pedestrian navigation, as it provides route instructions directly 
into the real visual context of the user [1].  
The analysis of the concrete orientation behavior of using the phone, of paying 
attention to the immediate environment and the navigation landmarks has been of 
minor interest so far. However, researching the impact of socio-cultural aspects on 
wayfinding styles requires detailed information about the user’s focus of attention, 
its preference for information types associated with particular affordances and en-
vironments, and continuous measurements about the user interaction, across ex-
tended periods of time.  
An exploratory analysis of gaze behavior was conducted to identify point-of-
regards (POR) on predefined areas of interest (AOI) within the smartphone dis-
play and towards the environment. To acquire video and eye movement data, SMI 
Eye Tracking Glasses were used, with 30 Hz sampling rate of gaze information 
and 1280 x 960 pixels scene camera (Figure 1). A screencast video of the naviga-
tion app was recorded and synchronized with the eye tracking data. For the post-
processing of the gaze data, the smartphone eye tracking (SMET) system [2] was 
applied, which was demonstrated to be feasible for large scale studies [4]. 
2 Mobile eye tracking study 
Navigation modes. In an outdoor study, gaze movements were recorded to inves-
tigate participants´ gaze during a wayfinding task on a predefined route  on the 
campus at University Hospital Graz, Austria (Figure 1). A mobile navigation tool 
was used providing two alternative presentation modes on the smartphone (Figure 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 14
“Gaze Analysis in Mobile Pedestrian Navigation” 
2a) to indicate the recommended route: (1) a two-dimensional map based view 
(MAP) and (2) an augmented reality-supported view (AR).  
Data capture. The SMET system [2] enables fully automated analysis of atten-
tion in user studies and showed highly accurate POR mappings on smartphone 
displays. Figure 2b shows the automated smartphone localization on a sample vid-
eo frame from the eye tracking scene camera. Synchronization and image analysis 
provide a correlated data stream of smartphone events, geometric transformations 
and heat-mapping for further attention analysis. For the analysis of the gaze pat-
terns we investigated content-dependent AOI assignments (Figure 2Fehler! Ver-
weisquelle konnte nicht gefunden werden.b) including the two interface ele-
ments on the smartphone (MAP, AR) as well as gaze into the physical surrounding 
(SUR).  
Study. In total 20 women participated in the study, including 10 immigrants 
from Turkey and 10 local citizens from Graz, Austria. Among the Turkish partici-
pants the average duration of stay in Graz was 8.55 ±5.431 years. The age of the 
participants was 28.3 ±6.5 (Turkish) and 28.3 ±6.5 years (Austrian). All partici-
pants had experience with smartphones and were familiar with the overall area; 
they all had previously visited the hospital. Turkish participants reported to be less 
experienced with maps.   
The objective of the study was to investigate whether there are principal differ-
ences between the viewing behaviors of the two participant groups, in the context 
of using a mobile interface for navigation, in particular, in the usage of map or 
augmented reality based services for wayfinding.   
3 Results 
Figure 3 shows the average count of PORs with respect to the selected AOIs, 
i.e., AOIs of the augmented reality mode (AR), the map mode (MAP) and the spa-
tial surrounding (SUR).
Analysis revealed that the Austrian participants preferred to look on the MAP 
than on the AR display region whereas Turkish users looked on average more of-
ten on the AR view than on the MAP. Results indicate a tendency in the context of 
the users’ socio-cultural background on the amount of PORs regarding the func-
tional use of the MAP and SUR views, respectively.  
Consequently, the findings drawn from eye tracking data as well as from quali-
tative feedback were capable to reveal that there exist relevant differences in the 
interface preferences of the Turkish users in the study in contrast to behaviors of 
the native Austrian participants. Future work will investigate the attention pro-
cesses in more depth and in the context of the socio-cultural variables, for exam-
ple, whether the social (immigrants vs. permanent citizen) or the cultural back-
1 Note: M ±S with M (=mean) and S (=standard deviation) 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 14
L. Paletta, S. Schwarz, J. Bobeth, M. Schwarz and M. Tscheligi
ground (Turkish vs. Austrian) of the users definitely impacts the wayfinding and 
the tool use in mobile navigation. 
References 
1. Walther-Franks, B (2007) Augmented reality on handhelds for pedestrian navigation. Master
thesis, University of Bremen, Germany.
2. Paletta, L., Neuschmied, H., Schwarz, M., Lodron, H., Pszeida, M., Ladstätter, S., Luley, P.
(2014) (2014), Smartphone Eye Tracking Toolbox: Accurate Gaze Recovery on Mobile Dis-
plays, Proc. Symposium on Eye Tracking Research and Applications, ETRA 2014, Safety 
Harbor, FA, pp. 367-68.
3. Goldberg, J. and Kotval, X. (1999) Computer interface evaluation using eye movements:
methods and constructs, International Journal of Industrial Ergonomics, 24, 631-645.
4. Paletta, L., Neuschmied, H., Schwarz, M., Lodron, H., Pszeida, M., Luley, P., Ladstätter, S.,
Deutsch, S., Bobeth, J., and Tscheligi, M. (2014). Attention in Mobile Interactions: Gaze
Recovery for Large Scale Studies, Proc. Conference on Human Factors in Computing Sys-
tems, CHI’14 extended abstracts, pp. 1717-1722.
(a)     (b) 
Figure 1. Eye tracking glasses used by (a) local and (b) immigrant users. 
(a) (b) 
Figure 2. Navigation app: (a) AR (left) and MAP (right) presentation modes, (b) 
view from eye tracking scene camera with automated gaze recovery (green dot) [2]. 
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 14
“Gaze Analysis in Mobile Pedestrian Navigation” 
Figure 3. Average gaze count on areas-of-interest (AR-augmented reality, MAP-
map, SUR-surrounding) with respect to the mobile display by Austrian (blue) and 
Turkish immigrant (red) participants. The error bars indicate the 95% confidence in-
tervals. Austrian participants clearly preferred to look at the map based information 
than on the augmented reality interface component. Turkish participants focused 
more on the surrounding than Austrian participants and looked on average more on 
intuitive navigation information (AR) than on the map based information display. 
Acknowledgments 
The research leading to these results has received funding from the European 
Community's Seventh Framework Programme (FP7/2007-2013) under grant 
agreement No. 288587 (MASELTOV) as well as from the Austrian Research 
Promotion Agency (FFG) under contract No. 832045 (FACTS).  
Published at SAGA Workshop 2015, proceedings archived under DOI: 10.2390/biecoll-saga2015 0.
Individual article archived under DOI: 10.2390/biecoll-saga2015 14


Gaze Analysis in Mobile Pedestrians Navigation: Socio-Cultural Aspects and Wayfinding

Not a member yet