26 research outputs found
Towards DESRED: A Diverse Explorable Super Resolution Evaluation Dataset
No full textThe task of explorable super resolution (ESR) reconstructs multiple possible high resolution (HR) image variants for a given single low resolution (LR) image. However, due to the novelty and complexity of ESR, there is currently no way to properly evaluate accuracy and diversity across ESR methods, instead relying on single image ground truths for evaluation, which cannot reflect the diverse space of HR images that ESR methods construct. This work aims towards solving this deficiency within the task of ESR by introducing DESRED: a Diverse Explorable Super Resolution Evaluation Dataset. The goal of this dataset is to produce multiple diverse, high-quality, and photorealistic HR images from single LR images, such that each set of HR images correspond exactly to the same LR image when downscaled. In this thesis, we detail the methods attempted towards the creation of this dataset to illustrate the complexity of this problem, from classical image processing techniques to deep learning and generative models. We present our initial classical method incorporating 3D face frontalization and normalization, as well as currently the most promising direction for the creation of this dataset: leveraging the state-of-the-art face image synthesizer StyleGAN with a consistency enforcing module to generate LR-consistent many-to-one HR/LR mappings. Our results show an initial formulation of DESRED with both high perceptual quality and diversity when compared with adjacent SR methods
Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions
No full textProbabilistic diffusion models have achieved state-of-the-art results for
image synthesis, inpainting, and text-to-image tasks. However, they are still
in the early stages of generating complex 3D shapes. This work proposes
Diffusion-SDF, a generative model for shape completion, single-view
reconstruction, and reconstruction of real-scanned point clouds. We use neural
signed distance functions (SDFs) as our 3D representation to parameterize the
geometry of various signals (e.g., point clouds, 2D images) through neural
networks. Neural SDFs are implicit functions and diffusing them amounts to
learning the reversal of their neural network weights, which we solve using a
custom modulation module. Extensive experiments show that our method is capable
of both realistic unconditional generation and conditional generation from
partial inputs. This work expands the domain of diffusion models from learning
2D, explicit representations, to 3D, implicit representations.Comment: revised experiments and added link to code and supplemen
Neural Volume Super-Resolution
No full textNeural volumetric representations have become a widely adopted model for
radiance fields in 3D scenes. These representations are fully implicit or
hybrid function approximators of the instantaneous volumetric radiance in a
scene, which are typically learned from multi-view captures of the scene. We
investigate the new task of neural volume super-resolution - rendering
high-resolution views corresponding to a scene captured at low resolution. To
this end, we propose a neural super-resolution network that operates directly
on the volumetric representation of the scene. This approach allows us to
exploit an advantage of operating in the volumetric domain, namely the ability
to guarantee consistent super-resolution across different viewing directions.
To realize our method, we devise a novel 3D representation that hinges on
multiple 2D feature planes. This allows us to super-resolve the 3D scene
representation by applying 2D convolutional networks on the 2D feature planes.
We validate the proposed method by super-resolving multi-view consistent views
on a diverse set of unseen 3D scenes, confirming qualitative and quantitatively
favorable quality over existing approaches
Neural Point Light Fields
No full textWe introduce Neural Point Light Fields that represent scenes implicitly with
a light field living on a sparse point cloud. Combining differentiable volume
rendering with learned implicit density representations has made it possible to
synthesize photo-realistic images for novel views of small scenes. As neural
volumetric rendering methods require dense sampling of the underlying
functional scene representation, at hundreds of samples along a ray cast
through the volume, they are fundamentally limited to small scenes with the
same objects projected to hundreds of training views. Promoting sparse point
clouds to neural implicit light fields allows us to represent large scenes
effectively with only a single radiance evaluation per ray. These point light
fields are a function of the ray direction, and local point feature
neighborhood, allowing us to interpolate the light field conditioned training
images without dense object coverage and parallax. We assess the proposed
method for novel view synthesis on large driving scenarios, where we synthesize
realistic unseen views that existing implicit approaches fail to represent. We
validate that Neural Point Light Fields make it possible to predict videos
along unseen trajectories previously only feasible to generate by explicitly
modeling the scene.Comment: 9 pages, replacement changed font of equation
From Posterior Sampling to Meaningful Diversity in Image Restoration
No full textImage restoration problems are typically ill-posed in the sense that each
degraded image can be restored in infinitely many valid ways. To accommodate
this, many works generate a diverse set of outputs by attempting to randomly
sample from the posterior distribution of natural images given the degraded
input. Here we argue that this strategy is commonly of limited practical value
because of the heavy tail of the posterior distribution. Consider for example
inpainting a missing region of the sky in an image. Since there is a high
probability that the missing region contains no object but clouds, any set of
samples from the posterior would be entirely dominated by (practically
identical) completions of sky. However, arguably, presenting users with only
one clear sky completion, along with several alternative solutions such as
airships, birds, and balloons, would better outline the set of possibilities.
In this paper, we initiate the study of meaningfully diverse image restoration.
We explore several post-processing approaches that can be combined with any
diverse image restoration method to yield semantically meaningful diversity.
Moreover, we propose a practical approach for allowing diffusion based image
restoration methods to generate meaningfully diverse outputs, while incurring
only negligent computational overhead. We conduct extensive user studies to
analyze the proposed techniques, and find the strategy of reducing similarity
between outputs to be significantly favorable over posterior sampling. Code and
examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.Comment: Accepted for ICLR 2024. Code and examples are available at
https://noa-cohen.github.io/MeaningfulDiversityInI
Atrial fibrillation: a geriatric perspective on the 2020 ESC guidelines
No full textBackground: The Task Force for the diagnosis and management of atrial fibrillation (AF) of the European Society of Cardiology (ESC) published in 2020 the updated Guidelines for the Diagnosis and Management of Atrial Fibrillation with the contribution of the European Heart Rhythm Association (EHRA) of the ESC and the European Association for Cardiothoracic Surgery (EACTS). Methods and results: In this narrative viewpoint, we approach AF from the perspective of aging medicine and try to provide the readers with information usually neglected in clinical routine, mainly due to the fact that while the large majority of AF patients in real life are older, frail and cognitively impaired, these are mostly excluded from clinical trials, and physicians’ attitudes often prevail over standardized algorithms. Conclusions: On the basis of existing evidence, (1) opportunistic AF screening by pulse palpation or ECG rhythm strip is cost-effective, and (2) whereas advanced chronological age by itself is not a contraindication to AF treatment, a Comprehensive Geriatric Assessment (CGA) including frailty, cognitive impairment, falls and bleeding risk may assist in clinical decision making to provide the best individualized treatment. © 2021, The Author(s)
Seeing through obstructions with diffractive cloaking
No full textUnwanted camera obstruction can severely degrade captured images, including both scene occluders near the camera and partial occlusions of the camera cover glass. Such occlusions can cause catastrophic failures for various scene understanding tasks such as semantic segmentation, object detection, and depth estimation. Existing camera arrays capture multiple redundant views of a scene to see around thin occlusions. Such multi-camera systems effectively form a large synthetic aperture, which can suppress nearby occluders with a large defocus blur, but significantly increase the overall form factor of the imaging setup. In this work, we propose a monocular single-shot imaging approach that optically cloaks obstructions by emulating a large array. Instead of relying on different camera views, we learn a diffractive optical element (DOE) that performs depth-dependent optical encoding, scattering nearby occlusions while allowing paraxial wavefronts to be focused. We computationally reconstruct unobstructed images from these superposed measurements with a neural network that is trained jointly with the optical layer of the proposed imaging system. We assess the proposed method in simulation and with an experimental prototype, validating that the proposed computational camera is capable of recovering occluded scene information in the presence of severe camera obstruction. © 2022 Owner/Author.11Nsciescopu
