1,139 research outputs found
Interactive In-Situ Scene Capture on Mobile Devices
No full textThesis (Ph.D.)--University of Washington, 2017-12Architectural visualizations of indoor scenes enable compelling applications in several areas, such as real estate, interior design, cultural heritage preservation, and more recently, immersive virtual reality. Computer-Aided Design (CAD) tools have been invaluable for creating such visualizations. However, creating detailed, attractive visualizations of scenes remains a challenging task, particularly for non-experts. User interfaces for CAD tools tend to be complex and require significant manual effort to operate. These tools are also designed to be used ex-situ, or off-site, making it difficult to record and reproduce details faithfully. In this thesis, I propose novel techniques and systems for interactive in-situ scene capture on mobile devices that let non-expert users quickly and easily capture useful architectural visualizations of indoor scenes. These systems are built upon two key insights; 1) sensors on mobile devices can be leveraged to capture important aspects of the scene such as dimensions, room shape, furniture placement, etc., and 2) an in-situ user can assist in the modeling task by acting as a guide for reconstruction and object recognition algorithms. Based on these insights, the semi-automatic systems that I propose combine the strengths of the user, who is good at high-level semantic reasoning, and the computer, which excels at combinatorics and numerical optimization. I present three systems in this thesis. First, I present a smartphone application designed to visually capture homes, offices and other indoor scenes. The application leverages data from smartphone sensors such as the camera, accelerometer, gyroscope and magnetometer to help reconstruct the indoor scene. The output of the system is two-fold; first, an interactive visual tour of the scene is generated in real time that allows the user to explore each room and transition between connected rooms. Second, by marking distinct room features such as corners and doors, the system generates a 2D floor plan and accompanying 3D model of the scene, under a Manhattan-world assumption. This approach does not require any specialized equipment or training, and is able to produce accurate floor plans. I then describe an interactive system to capture CAD-like 3D models of indoor scenes, on a tablet device. The modeling proceeds in two stages: (1) The user captures the 3D shape and dimensions of the room. (2) The user then uses voice commands and an augmented reality sketching interface to insert objects of interest, such as furniture, artwork, doors and windows. The system recognizes the sketches and add a corresponding 3D model into the scene at the appropriate location. The key contributions of this work are the design of a multi-modal user interface to effectively capture the user's semantic understanding of the scene, a framework for sketch based model retrieval, and the underlying algorithms that process the input to produce useful reconstructions. Finally, I extend the in-situ modeling approach to 3D-aware mobile devices, an emerging class of devices that can more richly sense the 3D nature of our world using depth sensing and self-localization. I propose a novel interactive system to further simplify the process of indoor 3D CAD room modeling on such devices. The proposed system leverages the sensing capabilities of a 3D aware mobile device, recent advances in object recognition, and a novel augmented reality user interface, to author indoor 3D room models in-situ. With a few taps, a user can mark the surface of an object, take a photo, and the system automatically retrieves and places a matching 3D model into the scene, from a large online database -- a modality that proves to be faster, more accurate, and easier than using traditional desktop tools
Reconstruction and Visualization of Architectural Scenes
No full textThesis (Ph.D.)--University of Washington, 2014Can we experience a scene virtually, such as the Colosseum in Rome, without ever having to visit it? Such an experience should replicate the feeling of being physically present, in terms of being able to visualize the scene from different viewpoints as well as quickly assimilating the highlights of the scene. An Internet search can increasingly provide us with a complete photographic record for the scene, but the challenge is in displaying such imagery in a coherent and informative way. In my thesis, I propose an approach for this problem based on 3D reconstruction and path planning. To make this approach feasible, we need to overcome three primary challenges. The first is in scaling the reconstruction algorithms to process millions of 3D points and thousands of images in an efficient manner. We design effective preconditioners to solve the non linear Bundle Adjustment problem efficiently, to obtain significant reductions in execution time. The second challenge involves improving the quality of the 3D reconstructions. Despite decades of research, state-of-the-art stereo algorithms cannot produce quality reconstructions everywhere, due to their dependence on the presence of texture. We complement stereo with monocular cues to overcome this challenge to compute more accurate and complete reconstructions. Finally having computed the reconstructions, a third challenge is in creating compelling visual experiences to aid a user in effectively navigating through the scene. We automatically compute movies, or photo tours, as paths through the reconstruction that are coherent, informative and efficient, for famous sites all over the world
Learning Scene CAD Recomposition
No full textThesis (Ph.D.)--University of Washington, 2020Humans perceive the world in three dimensions and can interpret the hidden part of objects and scenes despite partial observations and occlusions. This level of understanding comes from imagining the hidden surfaces based on the knowledge of object shapes, scene arrangement, and high-level inference based on common scene patterns. Understanding the 3D scene with the object-level composition of its components is essential for meaningful interaction with surrounding objects in the real world physical environments. In this thesis, first, we consider the problem of scene understanding where we show that if we can decompose a scene into its prominent objects, then we can start analyzing the scene. This enables us to infer high-level information about the scene structure, such as scene recognition and scene completion. Second, we introduce a novel perspective of recomposing the CAD model of the scene. We propose transforming raw visual sensory observations in order to re-create the scene with corresponding 3D object-level components. Towards this goal, we take advantage of large databases of object CAD models and leverage learning methods to solve this problem for real-world scenes at scale. Our scene CAD recomposition re-creates the scene by matching, placing, and aligning the objects from a database of thousands of CAD models. We also propose learning-based methods to automatically recompose the scene from a single-view RGB image as well as a sequence of RGB-D images for whole scene recomposition. Finally, we incorporate scene recomposition to solve a robotic object interaction problem. By means of technical analysis and experimental studies on real-world scenes, we validate that our novel object-level scene recomposition perspective provides a useful and yet concise representation that can facilitate accomplishing downstream tasks such as object manipulation in robotics. This thesis takes first steps towards developing a fully automatic system for recomposing scenes, and we hope our work inspires future research both on 3D learning and recomposition applications
Augmenting Visual Memories
No full textThesis (Ph.D.)--University of Washington, 2023We often rely on photographs and other forms of physical and digital media for capturing, preserving, and sharing the visual experiences that we encounter throughout our lives. While many common forms of media (like photographs) are able to effectively capture and reproduce detailed representations of our visual experiences, much of what we remember from these experiences is usually not captured in a photograph. When compared to our (incredibly rich) visual memories, photographs often lack a lot of crucial contextual information that is necessary to our understanding a particular event, e.g., details that help us answer questions like "where did this take place?", "what led to this moment?", and "what happened next?". Unlike human memories, which include information about our unbounded, three-dimensional, and ever-changing surroundings, photographs only capture a fixed field-of-view and a fixed point in time. Although recent technological developments have brought about a number of ways for capturing more detailed, immersive, and realistic depictions of our visual memories (such as 3D cameras and 360-degree videos), the vast majority of our past memories were captured by older technologies, such as the still photograph. As such, these memories remain permanently preserved in outdated forms of media, limiting in the realism, immersion, and fidelity with which they are able to portray our past experiences. In this thesis, I investigate methods for elevating this type of legacy imagery to more modern and immersive visual experiences by recovering or synthesizing the context that was lost during the capture process. First, I provide a survey of modern forms of media, specifically focusing on those which provide more immersion or visual fidelity than the traditional still photograph. I also review methods for augmentation of legacy captures, i.e., methods that turn older, less immersive content into modern forms of media. Finally, I present three novel techniques for enabling automatic augmentation of captured memories: (1) a method for automatically turning captured photos into high-quality video sequences; (2) a method for 3D reconstruction of previously captured smartphone camera videos, enabling different forms of 3D interaction, like virtual object insertion and light manipulation; and (3) a method for large-scale 3D structure from motion targeted towards handheld smartphone footage, which enables 3D reconstruction and visualization of large scenes, like buildings and city blocks that would otherwise be difficult to capture
Generative Keyframing
No full textThesis (Ph.D.)--University of Washington, 2025Keyframing is a fundamental element of animation creation and video editing. It involves defining specific frames, i.e., keyframes, that mark important moments of change and guide how the intermediate frames are filled or interpolated. In early hand-drawn animation, a keyframe is a visual drawing created by animators, with assistants manually drawing the in-between frames. With the advent of digital animation and video editing software, a keyframe became a set of parameters that define the state of the rendered character or object at specific times, with in-between transitions produced by interpolating these parameters. However, such parametric approaches rely heavily on manually designed controls and artist-crafted heuristics, making them difficult to capture complex, nuanced, and realistic motions. Furthermore, they do not naturally generalize to real image and video domains. The rapid progress of visual generative models that are trained on large collections of visual data and capable of learning rich appearance and motion patterns, has made it possible to generate high-fidelity imagery and realistic motion. Building on these advances, this thesis investigates generative keyframing, a data-driven, non-parametric, image-based approach to the keyframing process. To this end, I present a series of works in this thesis that collectively develop and explore this idea. I begin with the basic aspect: using generative models to synthesize transitions directly from images, and even to fully generate in-between motions. I first present a GAN-based technique for smoothing jump cuts in talking head videos, synthesizing seamless transitions between the cuts even in challenging cases involving large head movement. I then introduce a method for generating in-between videos with dynamic motion between more distant key frames by adapting a pretrained large-scale image-to-video diffusion model with minimal fine-tuning effort. Beyond automatically generating transitions between keyframes, I further explore multi-scale keyframing for achieving very deep zoom. Specifically, I introduce a multi-scale joint sampling diffusion approach for generating consistent images (keyframes) across different spatial scales while adhering to their respective input text prompts. This enables deep semantic zoom and a continuous zoom video can be rendered from these images. When working with multiple keyframes, one import question is how they should be ordered in the final video. I address this in the context of dance video generation---specifically, music synchronized and choreography-aware animal dance video---where unordered keyframes representing distinct animal poses are arranged via graph optimization to satisfy a specified choreography pattern of beats that defines the long-range structure of a dance. Finally, I conclude with discussions and directions for future works
Creating a Photorealistic World from Casual Lighting Capture
No full textThesis (Ph.D.)--University of Washington, 2024Light is the most crucial phenomenon for us to perceive and interact with the world. It plays a fundamental role in how we navigate, recognize, and interpret our surroundings. Throughout human history, we have sought to capture and record different lighting effects, from the earliest forms of painting to the invention of photography and the development of video technology. These mediums have allowed us to document and study the world in increasing detail. Many psychological studies have shown that the human visual system excels at deducing depth, shape, and motion from lighting effects such as shading and shadows. This ability underscores the importance of accurately simulating these effects in the creation of a photorealistic world. Creating photorealistic images and virtual environments has been a long-standing goal in the field of computer graphics, driven by applications in virtual reality, film production, and architectural visualization. Achieving high levels of realism requires accurately simulating the interaction of light with various objects and surfaces, as well as generating detailed highlights and realistic shadows. Casual capture --- everyday photos and videos taken with consumer devices --- offer a rich source of data for understanding and replicating real-world lighting effects. In this thesis, I explore techniques for enhancing photorealistic image synthesis by leveraging casual lighting capture. The ultimate goal is to create highly realistic and immersive visual experiences from novel viewpoints, novel scene configurations, and novel illumination. I begin with an overview of the psychophysics of light, focusing on human perception and the use of shading techniques in western art, providing a foundational understanding of how lighting effects influence visual perception. This foundational knowledge is critical for the subsequent development of methods that accurately replicate the nuances of lighting and shading in synthetic imagery. The core contributions of this thesis are as follows: First, I present People as Scene Probes, a method that infers depth, occlusion, lighting, and shadow information from video sequences captured from a single camera viewpoint. This technique enables realistic image composition by accurately modeling scene geometry and shading effects. Second, I introduce Repopulating Street Scenes, a framework that uses learned scene properties from image collections to automatically reconfigure street scenes by populating, depopulating or repopulating them with objects such as pedestrians or vehicles. It enables the realistic removal of existing objects along with their shadows and the insertion of new objects by accurately matching the lighting and casting shadows. This method enhances privacy and generates diverse training data for autonomous driving applications. Next, I introduce SunStage, a lightweight capture setup that replicates the functionality of a light stage using only a smartphone camera and the sun as the light source. With a video of an individual rotating in-place under the sun, SunStage reconstructs a physical model of the subject and the scene lighting, which enables applications such as relighting the subject with realistic reflections and cast shadows. SunStage allows arbitrary lighting and reflectance control in the reconstructed physical space, which can be rendered to produce photo-realistic results. I demonstrate several applications such as editing skin reflectance, relighting, and view synthesis. Finally, I present Infinite Texture, a method for generating arbitrarily large texture images from a text prompt. This technique supports applications in 3D rendering and texture transfer, ensuring consistent shading and depth through the use of a minimal dataset. I demonstrate the effectiveness of this approach in generating high-resolution, high-quality textures that can be seamlessly integrated into various downstream tasks. This work represents significant progress towards the goal of generating high-quality graphics assets from natural language descriptions
- …
