1,720,994 research outputs found
Calibration and View Interpolation for Light Field Displays
No full textDisplay technologies play an important role in our daily lives: we use displays for all kinds of
tasks, ranging from information gathering to entertainment. However, most of the commonly
used displays only serve a flat 2D image. As a result, they only stimulate a limited part of
the human’s visual system: several visual cues related to the depth of the scene are left out.
Technologies such as stereo displays and viewer tracking can be used to stimulate some of
these cues, however the result is still very incomplete. In recent years there is an increasing
interest in light field displays: displays that try to recreate the light field of an environment in
order to create a complete viewing experience, including all cues related to depth perception.
However, to accomplish this, light field displays require large amounts of input data.
When one wants to display a virtual scene, this is not much of an issue: the required data
can easily be generated using traditional computer graphics techniques. However, when one
wants to display a real-world scene, other means have to be used to obtain the required data.
One approach could be to capture all information using cameras. However, this would require
a large amount of cameras to capture all information in a single shot. Hence, such a solution
is expensive and often not practical. In this dissertation we describe a view interpolation
approach to generate the input data for light field displays based on a sparse set of input
cameras. The distance between the cameras should be large enough to capture the full scene.
To accommodate the different input requirements for different light field displays, we will
present an intermediate data structure that stores the real-world scene. The data structure
should also allow to easily generate data for different types of displays. This data structure
is based on a light field representation, called Epipolar Plane Images. Epipolar Plane Images
are created by stacking the images from a linear camera array on top of each other and taking
a slice corresponding to a scanline from the resulting cube. The created images contain a set
of lines that implicitly encode the geometry of the scene.
First, we will look at techniques to extract this geometrical information from the input
camera images. We show that utilization of the Epipolar Plane Image properties allows for
an accurate estimation of the geometry of the scene, even for smaller amounts of cameras
and when the distance between the cameras is relatively large. Furthermore we propose
methods to better handle occlusions of the scene as well as color variations in the input data
caused by the material properties of the scene or inconsistencies in camera synchronization. A frequency domain based method is introduced to obtain a per scanline impression of the
scene’s depth distribution. This is achieved by analyzing the lines in the epipolar plane image.
This analysis helps to reduce the search range while extracting the required information, and
hence increase the quality of the obtained data.
After filling the data structure, we generate the data for the light field displays. In this
dissertation we will focus on the generation of data for two kinds of displays: multi-view light
field displays and integral imaging displays. Both types of displays require a different layout
of the input data. The former type requires the input images to be captured at the positions
from which the user will observe the screen. The latter display type requires the images to be
captured at the display plane itself, which is often located inside the scene. We will show that
our approach is able to generate consistent data for both types of displays. Additionally, we
will show that, due to our proposed extraction algorithms, we are able to generate data of a
higher quality than existing view synthesis approaches. With the help of a frequency domain
based filter, we are able to reduce the disturbing effects of view dependent noise.
The advantages that light field displays have over traditional displays are also useful for
augmented reality applications. However, light field displays are often opaque, hence hiding
what is behind them. We will present a custom transparent light field display that can be used
for augmented reality applications. The display solely consists of an off the shelf projector
and a custom holographic optical element that acts as a screen of micromirrors. The system
requires an accurate calibration between the projector and the screen to accurately recreate
the light field. We first propose a calibration approach for a flat display, and show that the
proposed calibration approach is able to correctly align the virtual world with the real world.
Since the material of the screen is flexible it can also be applied on curved surfaces such as
the wind-screen of a car. An adjusted calibration approach is required for a curved screen
due to the extra distortion that the curved surface introduces. We show that the calibration of
a curved transparent light field display is possible if the shape of the surface is known
Projector-camera calibration with non-overlapping fields of view using a planar mirror
No full textProjector-camera systems have numerous applications across diverse domains. Accurate calibration of both intrinsic and extrinsic parameters is crucial for these systems. Intrinsic parameters include focal length, distortion parameters, and the principal point, while extrinsic parameters encompass the position and orientation of the projector and camera. A non-overlapping projector-camera system is required in certain scenarios due to practical limitations, physical arrangements, or specific application requirements. These systems pose a more complex calibration challenge because the devices have no direct correspondences or overlapping fields of view, necessitating intermediate objects or methods. This paper proposes a calibration method for non-overlapping projector-camera systems using a planar mirror. The method involves a straightforward process that requires a calibrated camera and a separate mirror calibration step. In this setup, the projector displays a pattern on a planar calibration board, and the camera has an indirect view of this calibration board through the mirror. Using homography, 3D-2D correspondences of the projector are established, enabling the calibration of the system. This method is empirically evaluated using real-world setups and quantitatively assessed in a synthetic environment. The results demonstrate that the proposed method achieves precise calibration across various setups, proving its effectiveness. This approach is an easy-to-use and accessible calibration process for non-overlapping projector-camera systems, made possible by a mirror.This research was partly funded by the European Union
(HORIZON MAX-R, Mixed Augmented and Extended Reality Media
Pipeline, 101070072), the Flanders Make’s XRTwin SBO project
(R-12528), the Special Research Fund (BOF) of Hasselt University
(R-14360) and the specialized FWO fellowship grant (1SHDZ24N).
Data availability The datasets generated during and/or analyzed during
the current study are available from the corresponding author on
reasonable request
EDM-Research/UE-LASAA: published
No full textUnreal Engine 5.3 plugin for Large-area Spatially Aligned Anchors This project contains a plug-and-play Unreal Engine 5 plugin for Large-area Spatially Aligned Anchors (LASAA). The current version only supports Meta headsets. Other headsets can be integrated by changing the SDK and the code inside LASAA/Source/LASAA/Private/Anchor.cpp and LASAA/Source/LASAA/Public/Anchor.h. Source code for the following scientific publication: Vanherck, J., Zoomers, B., Jorissen, L., Vandebroeck, I., Joris, E., & Michiels, N. (2024). Large-Area Spatially Aligned Anchors. In L. T. De Paolis, P. Arpaia, & M. Sacco (Eds.), International Conference on Extended Reality (pp. 42–60). Cham: Springer Nature Switzerland. Abstract: Extended Reality (XR) technologies, including Virtual Reality (VR) and Augmented Reality (AR), offer immersive experiences merging digital content with the real world. Achieving precise spatial tracking over large areas is a critical challenge in XR development. This paper addresses the drift issue, caused by small errors accumulating over time leading to a discrepancy between the real and virtual worlds. Tackling this issue is crucial for co-located XR experiences where virtual and physical elements interact seamlessly. Building upon the locally accurate spatial anchors, we propose a solution that extends this accuracy to larger areas by exploiting an external, drift-corrected tracking method as a ground truth. During the preparation stage, anchors are placed inside the headset and inside the external tracking method simultaneously, yielding 3D-3D correspondences. Both anchor clouds, and thus tracking methods, are aligned using a suitable cloud registration method during the operational stage. Our method enhances user comfort and mobility by leveraging the headset's built-in tracking capabilities during the operational stage, allowing standalone functionality. Additionally, this method can be used with any XR headset that supports spatial anchors and with any drift-free external tracking method. Empirical evaluation demonstrates the system's effectiveness in aligning virtual content with the real world and expanding the accurate tracking area. In addition, the alignment is evaluated by comparing the camera poses of both tracking methods. This approach may benefit a wide range of industries and applications, including manufacturing and construction, education, and entertainment.MAX-R (Mixed Augmented and eXtended Reality media pipeline). Horizon Europe ProjectXRTwin SBO. Flanders Make (Belgium).Special Research Fund (BOF)FWO fellowship grant. Research Foundation - Flanders. awardNumber:1SHDZ24N
Nonuniform Depth Distribution Selection with Discrete Fourier Transform
No full textinfo:eu-repo/semantics/publishe
Multi-view wide baseline depth estimation robust to sparse input sampling
No full textIn this paper, we propose a depth map estimation algorithm, based on Epipolar Plane Image (EPI) line extraction, that is able to correctly handle partially occluded objects in wide baseline camera setups. Furthermore, we introduce a descriptor matching technique to reduce the negative influence of inaccurate color correction and similarly textured objects on the depth maps. A visual comparison between an existing EPI-line extraction algorithm and our method is provided, showing that our method provides more accurate and consistent depth maps in most cases.SCOPUS: cp.pinfo:eu-repo/semantics/publishe
PRoGS: Progressive Rendering of Gaussian Splats
No full textOver the past year, 3D Gaussian Splatting (3DGS) has received significant
attention for its ability to represent 3D scenes in a perceptually accurate
manner. However, it can require a substantial amount of storage since each
splat's individual data must be stored. While compression techniques offer a
potential solution by reducing the memory footprint, they still necessitate
retrieving the entire scene before any part of it can be rendered. In this
work, we introduce a novel approach for progressively rendering such scenes,
aiming to display visible content that closely approximates the final scene as
early as possible without loading the entire scene into memory. This approach
benefits both on-device rendering applications limited by memory constraints
and streaming applications where minimal bandwidth usage is preferred. To
achieve this, we approximate the contribution of each Gaussian to the final
scene and construct an order of prioritization on their inclusion in the
rendering process. Additionally, we demonstrate that our approach can be
combined with existing compression methods to progressively render (and stream)
3DGS scenes, optimizing bandwidth usage by focusing on the most important
splats within a scene. Overall, our work establishes a foundation for making
remotely hosted 3DGS content more quickly accessible to end-users in
over-the-top consumption scenarios, with our results showing significant
improvements in quality across all metrics compared to existing methods.This research was partly funded by the specialized FWO fellowship grant (1SHDZ24N), the European Union (HORIZONMAX-R,MixedAugmentedandExtended Reality Media Pipeline, 101070072), the Flanders 20 Make’s XRTwin SBO project (R-12528) and the Special Research Fund (BOF) of Hasselt University (R-14360). This work was made possible with support from MAXVR-INFRA, a scalable and flexible infrastructure that facilitates the transition to digital-physical work environments.FWO: R-14292
BOF: R-1443
PRoGS: Progressive Rendering of Gaussian Splats
No full textFigure 1. Progressive Rendering of 3DGS using our approach versus using an existing web-viewer by Antimatter. From left to right, we show 0.2%, 0.5%, 1%, and 10% of the total number of splats, respectively. Our approach results in a faster visualization of a representative version of the scene. At 0.2%, we have loaded in a basic level of the truck while it is completely missing in the alternative approach. Abstract Over the past year, 3D Gaussian Splatting (3DGS) has received significant attention for its ability to represent 3D scenes in a perceptually accurate manner. However, it can require a substantial amount of storage since each splat's individual data must be stored. While compression techniques offer a potential solution by reducing the memory footprint, they still necessitate retrieving the entire scene before any part of it can be rendered. In this work, we introduce a novel approach for progressively rendering such scenes, aiming to display visible content that closely approximates the final scene as early as possible without loading the entire scene into memory. This approach benefits both on-device rendering applications limited by memory constraints and streaming applications where minimal bandwidth usage is preferred. To achieve this, we approximate the contribution of each Gaussian to the final scene and construct an order of prioritization on their inclusion in the rendering process. Additionally, we demonstrate that our approach can be combined with existing compression methods to progressively render (and stream) 3DGS scenes, optimizing bandwidth usage by focusing on the most important splats within a scene. Overall, our work establishes a foundation for making remotely hosted 3DGS content more quickly accessible to end-users in over-the-top consumption scenarios, with our results showing significant improvements in quality across all metrics compared to existing methods
Large-area Spatially Aligned Anchors
No full textExtended Reality (XR) technologies, including Virtual Reality (VR) and Augmented Reality (AR), offer immersive experiences merging digital content with the real world. Achieving precise spatial tracking over large areas is a critical challenge in XR development. This paper addresses the drift issue, caused by small errors accumulating over time leading to a discrepancy between the real and virtual worlds. Tackling this issue is crucial for co-located XR experiences where virtual and physical elements interact seamlessly. Building upon the locally accurate spatial anchors, we propose a solution that extends this accuracy to larger areas by exploiting an external, drift-corrected tracking method as a ground truth. During the preparation stage, anchors are placed inside the headset and inside the external tracking method simultaneously, yielding 3D-3D correspondences. Both anchor clouds, and thus tracking methods, are aligned using a suitable cloud registration method during the operational stage. Our method enhances user comfort and mobility by leveraging the headset's built-in tracking capabilities during the operational stage, allowing standalone functionality. Additionally, this method can be used with any XR headset that supports spatial anchors and with any drift-free external tracking method. Empirical evaluation demonstrates the system's effectiveness in aligning virtual content with the real world and expanding the accurate tracking area. In addition, the alignment is evaluated by comparing the camera poses of both tracking methods. This approach may benefit a wide range of industries and applications, including manufacturing and construction, education, and entertainment
PRoGS: Progressive Rendering of Gaussian Splats
No full textOver the past year, 3D Gaussian Splatting (3DGS) has received significant
attention for its ability to represent 3D scenes in a perceptually accurate
manner. However, it can require a substantial amount of storage since each
splat's individual data must be stored. While compression techniques offer a
potential solution by reducing the memory footprint, they still necessitate
retrieving the entire scene before any part of it can be rendered. In this
work, we introduce a novel approach for progressively rendering such scenes,
aiming to display visible content that closely approximates the final scene as
early as possible without loading the entire scene into memory. This approach
benefits both on-device rendering applications limited by memory constraints
and streaming applications where minimal bandwidth usage is preferred. To
achieve this, we approximate the contribution of each Gaussian to the final
scene and construct an order of prioritization on their inclusion in the
rendering process. Additionally, we demonstrate that our approach can be
combined with existing compression methods to progressively render (and stream)
3DGS scenes, optimizing bandwidth usage by focusing on the most important
splats within a scene. Overall, our work establishes a foundation for making
remotely hosted 3DGS content more quickly accessible to end-users in
over-the-top consumption scenarios, with our results showing significant
improvements in quality across all metrics compared to existing methods.This research was partly funded by the specialized FWO fellowship grant (1SHDZ24N), the European Union (HORIZONMAX-R,MixedAugmentedandExtended Reality Media Pipeline, 101070072), the Flanders 20 Make’s XRTwin SBO project (R-12528) and the Special Research Fund (BOF) of Hasselt University (R-14360). This work was made possible with support from MAXVR-INFRA, a scalable and flexible infrastructure that facilitates the transition to digital-physical work environments.FWO: R-14292
BOF: R-1443
Tracking and co-location of global point clouds for large-area indoor environments
No full textExtended reality (XR) experiences are on the verge of becoming widely adopted in diverse application domains. An essential part of the technology is accurate tracking and localization of the headset to create an immersive experience. A subset of the applications require perfect co-location between the real and the virtual world, where virtual objects are aligned with real-world counterparts. Current headsets support co-location for small areas, but suffer from drift when scaling up to larger ones such as buildings or factories. This paper proposes tools and solutions for this challenge by splitting up the simultaneous localization and mapping (SLAM) into separate mapping and localization stages. In the pre-processing stage, a feature map is built for the entire tracking area. A global optimizer is applied to correct the deformations caused by drift, guided by a sparse set of ground truth markers in the point cloud of a laser scan. Optionally, further refinement is applied by matching features between the ground truth keyframe images and their rendered-out SLAM estimates of the point cloud. In the second, real-time stage, the rectified feature map is used to perform localization and sensor fusion between the global tracking and the headset. The results show that the approach achieves robust co-location between the virtual and the real 3D environment for large and complex tracking environments.This research was funded by the European Union (HORIZON MAX-R, Mixed Augmented and Extended Reality Media Pipeline, 101070072) and the Flanders Make’s XRTwin SBO project (R-12528)
- …
