160 research outputs found
Sonic Interactions in Virtual Environments: the Egocentric Audio Perspective of the Digital Twin
No full textThe relationships between the listener, physical world and virtual environment
(VE) should not only inspire the design of natural multimodal interfaces but
should be discovered to make sense of the mediating action of VR technologies.
This chapter aims to transform an archipelago of studies related to sonic interactions
in virtual environments (SIVE) into a research field equipped with a first theoretical
framework with an inclusive vision of the challenges to come: the egocentric
perspective of the auditory digital twin. In a VE with immersive audio technologies
implemented, the role of VR simulations must be enacted by a participatory
exploration of sense-making in a network of human and non-human agents, called
actors. The guardian of such locus of agency is the auditory digital twin that fosters
intra-actions between humans and technology, dynamically and fluidly redefining
all those configurations that are crucial for an immersive and coherent experience.
The idea of entanglement theory is here mainly declined in an egocentric-spatial
perspective related to emerging knowledge of the listener’s perceptual capabilities.
This is an actively transformative relation with the digital twin potentials to create
movement, transparency, and provocative activities in VEs. The chapter will contain
an original theoretical perspective complemented by several bibliographical references
and links to the other book chapters that have contributed significantly to the
proposal presented here
Mixed Structural Models for 3D Audio in Virtual Environments
No full textIn the world of ICT, strategies for innovation and development are increasingly focusing on applications that require spatial representation and real-time interaction with and within 3D media environments. One of the major challenges that such applications have to address is user-centricity, reflecting e.g. on developing complexity-hiding services so that people can personalize their own delivery of services. In these terms, multimodal interfaces represent a key factor for enabling an inclusive use of the new technology by everyone. In order to achieve this, multimodal realistic models that describe our environment are needed, and in particular models that accurately describe the acoustics of the environment and communication through the auditory modality. Examples of currently active research directions and application areas include 3DTV and future internet, 3D visual-sound scene coding, transmission and reconstruction and teleconferencing systems, to name but a few.
The concurrent presence of multimodal senses and activities make multimodal virtual environments potentially flexible and adaptive, allowing users to switch between modalities as needed during the continuously changing conditions of use situation. Augmentation through additional modalities and sensory substitution techniques are compelling ingredients for presenting information non-visually, when the visual bandwidth is overloaded, when data are visually occluded, or when the visual channel is not available to the user (e.g., for visually impaired people). Multimodal systems for the representation of spatial information will largely benefit from the implementation of audio engines that have extensive knowledge of spatial hearing and virtual acoustics. Models for spatial audio can provide accurate dynamic information about the relation between the sound source and the surrounding environment, including the listener and his/her body which acts as an additional filter. Indeed, this information cannot be substituted by any other modality (i.e., visual or tactile). Nevertheless, today's spatial representation of audio within sonification tends to be simplistic and with poor interaction capabilities, being multimedia systems currently focused on graphics processing mostly, and integrated with simple stereo or multi-channel surround-sound.
On a much different level lie binaural rendering approaches based on headphone reproduction, taking into account that possible disadvantages (e.g. invasiveness, non-flat frequency responses) are counterbalanced by a number of desirable features. Indeed, these systems might control and/or eliminate reverberation and other acoustic effects of the real listening space, reduce background noise, and provide adaptable and portable audio displays, which are all relevant aspects especially in enhanced contexts.
Most of the binaural sound rendering techniques currently exploited in research rely on the use of Head-Related Transfer Functions (HRTFs), i.e. peculiar filters that capture the acoustic effects of the human head and ears. HRTFs allow loyal simulation of the audio signal that arrives at the entrance of the ear canal as a function of the sound source's spatial position. HRTF filters are usually presented under the form of acoustic signals acquired on dummy heads built according to mean anthropometric measurements. Nevertheless, anthropometric features of the human body have a key role in HRTF shaping: several studies have attested how listening to non-individual binaural sounds results in evident localization errors. On the other hand, individual HRTF measurements on a significant number of subjects result both time- and resource-expensive.
Several techniques for synthetic HRTF design have been proposed during the last two decades and the most promising one relies on structural HRTF models. In this revolutionary approach, the most important effects involved in spatial sound perception (acoustic delays and shadowing due to head diffraction, reflections on pinna contours and shoulders, resonances inside the ear cavities) are isolated and modeled separately with a corresponding filtering element. HRTF selection and modeling procedures can be determined by physical interpretation: parameters of each rendering blocks or selection criteria can be estimated from real and simulated data and related to anthropometric geometries.
Effective personal auditory displays represent an innovative breakthrough for a plethora of applications and structural approach can also allow for effective scalability depending on the available computational resources or bandwidth. Scenes with multiple highly realistic audiovisual objects are easily managed exploiting parallelism of increasingly ubiquitous GPUs (Graphics Processing Units). Building individual headphone equalization with perceptually robust inverse filtering techniques represents a fundamental step towards the creation of personal virtual auditory displays (VADs). To this regard, several examples might benefit from these considerations: multi-channel downmix over headphones, personal cinema, spatial audio rendering in mobile devices, computer-game engines and individual binaural audio standards for movie and music production.
This thesis presents a family of approaches that overcome the current limitations of headphone-based 3D audio systems, aiming at building personal auditory displays through structural binaural audio models for an immersive sound reproduction. The resulting models allow for an interesting form of content adaptation and personalization, since they include parameters related to the user's anthropometry in addition to those related to the sound sources and the environment.
The covered research directions converge to a novel framework for synthetic HRTF design and customization that combines the structural modeling paradigm with other HRTF selection techniques (inspired by non-individualized HRTF selection procedures) and represents the main novel contribution of this thesis: the Mixed Structural Modeling (MSM) approach considers the global HRTF as a combination of structural components, which can be chosen to be either synthetic or recorded components. In both cases, customization is based on individual anthropometric data, which are used to either fit the model parameters or to select a measured/simulated component within a set of available responses.
The definition and experimental validation of the MSM approach addresses several pivotal issues towards the acquisition and delivery of binaural sound scenes and designing guidelines for personalized 3D audio virtual environments holding the potential of novel forms of customized communication and interaction with sound and music content.
The thesis also presents a multimodal interactive system which is used to conduct subjective test on multi-sensory integration in virtual environments. Four experimental scenarios are proposed in order to test the capabilities of auditory feedback jointly to tactile or visual modalities. 3D audio feedback related to user’s movements during simple target following tasks is tested as an applicative example of audio-visual rehabilitation system. Perception of direction of footstep sounds interactively generated during walking and provided through headphones highlights how spatial information can clarify the semantic congruence between movement and multimodal feedback. A real time, physically informed audio-tactile interactive system encodes spatial information in the context of virtual map presentation with particular attention to orientation and mobility (O&M) learning processes addressed to visually impaired people. Finally, an experiment analyzes the haptic estimation of size of a virtual 3D object (a stair-step) whereas the exploration is accompanied by a real-time generated auditory feedback whose parameters vary as a function of the height of the interaction point.
The collected data from these experiments suggest that well-designed multimodal feedback, exploiting 3D audio models, can definitely be used to improve performance in virtual reality and learning processes in orientation and complex motor tasks, thanks to the high level of attention, engagement, and presence provided to the user. The research framework, based on the MSM approach, serves as an important evaluation tool with the aim of progressively determining the relevant spatial attributes of sound for each application domain. In this perspective, such studies represent a novelty in the current literature on virtual and augmented reality, especially concerning the use of sonification techniques in several aspects of spatial cognition and internal multisensory representation of the body.
This thesis is organized as follows. An overview of spatial hearing and binaural technology through headphones is given in Chapter 1.
Chapter 2 is devoted to the Mixed Structural Modeling formalism and philosophy. In Chapter 3, topics in structural modeling for each body component are studied, previous research and two new models, i.e. near-field distance dependency and external-ear spectral cue, are presented.
Chapter 4 deals with a complete case study of the mixed structural modeling approach and provides insights about the main innovative aspects of such modus operandi. Chapter 5 gives an overview of number of a number of proposed tools for the analysis and synthesis of HRTFs. System architectural guidelines and constraints are discussed in terms of real-time issues, mobility requirements and customized audio delivery.
In Chapter 6, two case studies investigate the behavioral importance of spatial attribute of
sound and how continuous interaction with virtual environments can benefit from using spatial audio algorithms. Chapter 7 describes a set of experiments aimed at assessing the contribution of binaural audio through headphones in learning processes of spatial cognitive maps and exploration of virtual objects.
Finally, conclusions are drawn and new research horizons for further work are exposed in Chapter 8
Evaluating vertical localization performance of 3D sound rendering models with a perceptual metric
No full textThe head-related transfer functions (HRTFs) describe individual acoustic transformation that sound sources undergo due to human anatomy before arriving at the left and right tympanic membranes. The resulting spectral modifications are the main localization cues for elevation detection in space. In this paper, synthetic HRTF mod- els able to render the vertical spatial dimension in virtual auditory displays, are evaluated via auditory models. Perceptually-motivated metrics describe the output of 4 virtual experiments that numeri- cally simulate real listening experiments for 20 virtual subjects. The current implementation considers a limited set of parameters for a structural model of the pinna acting as a proof-of-concept of such approach. Accordingly, results confirm that the research framework is a flexible tool for systematic evaluation of different instances of structural model
Sonic Interactions in Virtual Environments
No full textThis book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are:
Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies
Sonic interaction: the human-computer interplay through auditory feedback in VE
VR systems: naturally support multimodal integration, impacting different application domains
Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond.
Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments.
This is an open access book
Musiche liquide. XX Colloquio di Informatica Musicale. 20Th Colloquium on Music Informatics. Liquid music
No full textA motion based setup for peri-personal space estimation with virtual auditory displays
No full textThe core idea of the work is to reveal the presence of changes in action preparation as a function of sounds movements (e.g. direction of arrivals and trajectories in space) and sounds semantics (e.g. threatening or pleasant) when they are sent within the Peri-Personal-Space (PPS) of blindfolded listeners. This near-field acoustics is known to activate direct pathways from the motor cortex to the muscular periphery, as a prompt preparation against threats. The proposed system is thought to aid particularly to people with sensory or cognitive impairments
Designing sonic interactions in intelligent reality with egocentric audio technologies
No full textSonic Interactions in Virtual Environments (SIVE) is an emerging field of study that aims to foster the creation of immersive and interactive sonic experiences in Virtual/Augmented Realities (VR/AR). Sound can convey a wealth of information, such as emotion, meaning and narratives, as well as enhance a multimodal experience. However, designing immersive sonic interactions is a complex task that requires a deep understanding of the embodied, situated and enactive aspects of sound in space. This chapter provides an overview of the field of SIVE and discusses the key concepts and challenges involved in designing meaningful sonic interactions. By using the lens of actor-network theory (ANT) and the new technological paradigm of egocentric audio, this chapter will discuss how sonic interactions can be effectively designed in their early stages to improve the sound-driven design in a multimodal perspective. In addition, the chapter proposes a novel theoretical design tool, called the Entanglement eXperience Map (EXMap), which can be used to define different perspectives and dynamic behaviors of actors within immersive sonic experiences. It considers three main concepts introduced by the egocentric audio technologies and their pivotal participation of an auditory digital twin:Â (i) sense of immersion in the VE, (ii)Â coherence of the experience, and (iii) entanglement between the user (real twin) with a cyber-physical system (digital twin). The following case study in the cultural heritage domain guides the reader:Â a user can have a museum visit experience in immersive audio AR, receiving spatialized information about artwork and artifacts based on personal interests, and emotional and psychological states. Personalized sonic interactions aim to uniquely create a meaningful and memorable experience for the user
On the evaluation of head-related transfer functions with probabilistic auditory models of human sound localization
No full textUnderstanding spatial hearing leads to implement efficient and effective auralization rendering algorithms with headphones. Two important aspects contribute to sound localization: (i) acoustic filtering of listener body, and (ii) non-acoustic factors introduced by auditory periphery. Accordingly, head-related transfer functions (HRTFs) describe users acoustics in terms of their spatial filtering. Binaural synthesis through generic HRTFs (commonly a dummy head) is the most simple solution for an auralization framework. In this scenario, a high variability in localization tasks between subjects yields to an unreliable rendering. Listener's acoustic and perceptual characterization require HRTF modeling and auditory models predictions in order to provide an effective auralization on individual basis. Systemic comparisons of HRTF approximations and different user profiles can help to predict listener's performances. We consider a case study on both vertical and horizontal localization with different HRTFs and two probabilistic auditory models. In our analysis, spatial audio rendering with non-individual HRTFs has a special attention for its commercial relevance compared to unpractical and questionable use of individual HRTFs
- …
