12 research outputs found
다양한 환경에서의 자율 주행을 위한 이미지와 포인트 클라우드 데이터 융합 기술
No full text학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2023. 2. 서승우.Autonomous driving is one of the main topics of robotics research and is becoming part of our lives as they are used in automobiles, indoor robots, military robots, and drones. Perception, which is the starting point of driving intelligence, is a process of generating the knowledge necessary for driving by interpreting data collected from sensors. The data collected by the sensors varies depending on the robot platform, terrain, and light or weather conditions. For autonomous driving, perception must stably generate accurate knowledge by receiving data that changes according to these various conditions. However, many existing studies have focused on benchmark competition to achieve the best performance under limited conditions. We break away from these laboratory-only studies and focus on the real world to develop robust algorithms that can operate in various conditions.
Sensors widely used in autonomous driving include cameras that collect images and LiDARs that collect point clouds. Images contain high-resolution color information and are sensitive to changes in illumination (e.g., sunlight and weather). Point clouds are robust to changes in illumination and provide distance information of 3D space but are of low resolution. The image and point cloud have complementary characteristics. Therefore, the fusion of two data can produce enhanced information that compensates for the shortcomings of each data.
We propose a fusion framework of images and point clouds for autonomous driving in diverse environments. The first step of the fusion framework is a one-to-one match of the two data. This is called image-to-point cloud registration and aims to align a 2D image with a 3D point cloud. The proposed algorithm is designed for robust operation under various terrain conditions. Therefore, it was demonstrated in automobiles on paved roads and UGVs on off-road driving.
The second step is to generate enhanced data using the two matched data. This is called depth completion, which generates a depth image with high-resolution depth information using an image with high-resolution color information and a point cloud with low-resolution depth information. The proposed algorithm is designed to work reliably in various light and weather conditions, from sunrise to sunset, fog and rain, and camera corruption.
The last step is to estimate the traversability from the observed image and point cloud. This is called traversability estimation, and traversability is predicted by learning the robot's driving style. The proposed algorithm is designed to estimate different traversabilities with respect to the driving style of the robot platform, from large ATVs to small UGVs.
In this dissertation, we present an image and point cloud fusion framework that can operate in diverse real-world conditions. We evaluate the proposed algorithm through experiments on various platforms (automobile, large and small UGV), terrain (paved road, open fields, mountain), and light and weather conditions (morning, sunset, rain, fog). The proposed framework serves as a buffer to ensure the stability of further applications by receiving data that fluctuates depending on diverse conditions and generating stable and enhanced knowledge. Therefore, we expect this framework to be a foundation for robust autonomous driving.자율 주행은 로봇 연구의 주요 화두 중 하나로 자동차, 실내로봇, 군용로봇, 드론 등 다양한 플랫폼에 사용되어 우리 삶의 일부가 되고 있다. 그 중 주행 지능의 출발점인 인식은 센서에서 수집한 데이터를 해석해 주행에 필요한 지식을 생성하는 과정이다. 센서가 수집하는 데이터는 로봇 플랫폼, 지형, 빛이나 날씨 조건에 따라 달라진다. 자율 주행을 위한 인식 기술은 이와 같은 다양한 조건에 따라 변동하는 데이터를 입력받아 안정적으로 정확한 지식을 생성할 수 있어야 한다. 그러나 기존의 연구는 대부분 제한된 조건에서 최고의 성능을 달성하기 위한 벤치마크 경쟁에 초점을 맞추었다. 우리는 이러한 실험실 전용 연구에서 벗어나 실제 세계로 뛰어들어 다양한 조건에서 작동할 수 있는 강력한 알고리즘을 개발하고자 한다.
자율 주행에 널리 사용되는 센서로는 이미지를 수집하는 카메라와 포인트 클라우드를 수집하는 라이다(LiDAR) 가 있다. 이미지는 고해상도의 색상 정보를 포함하고 있으며 조도 변화(햇빛, 날씨 등) 에 민감하다. 포인트 클라우드는 조도 변화에 강하고 3차원 공간의 거리 정보를 제공하지만 낮은 해상도를 가진다. 이렇듯, 이미지와 포인트 클라우드는 보완적인 특성을 가지고 있으므로 우리는 두 데이터의 융합을 통해 각 데이터의 단점을 보완하는 보강된 정보를 생성할 수 있다.
우리는 다양한 환경에서의 자율 주행을 위한 이미지와 포인트 클라우드의 융합 프레임워크를 제안한다. 융합 프레임워크의 첫 번째 단계는 두 데이터의 일대일 일치이다. 이를 이미지-투-포인트 클라우드 레지스트레이션 (Image-to-point cloud registration) 이라고 하며 이 단계의 목표는 2D 이미지를 3D 포인트 클라우드와 정렬하는 것이다. 제안하는 알고리즘은 다양한 지형 조건에서 강건하게 동작하도록 설계되었으며, 포장 도로를 주행하는 자동차와 오프로드를 주행하는 UGV에서 그 성능이 검증되었다.
두 번째 단계는 정렬된 이미지와 포인트 클라우드를 바탕으로 하는 보강된 데이터의 생성이다. 이를 뎁스 컴플리션 (Depth completion) 이라고 하며, 이 단계의 목표는 고해상도의 색상 정보를 가진 이미지와 저해상도의 거리 정보를 가진 포인트 클라우드를 이용하여 고해상도의 거리 정보를 가진 뎁스 이미지를 생성하는 것이다. 제안하는 알고리즘은 아침부터 밤, 안개와 비, 카메라 손상 등 다양한 빛과 날씨 조건에서 안정적으로 작동하도록 설계되었다.
마지막 단계는 관찰된 이미지와 포인트 클라우드를 바탕으로 하는 주행 가능성의 추정이다. 이를 주행 가능성 추정 (Traversability estimation) 이라고 하며, 로봇의 주행 스타일을 학습하여 횡단성을 예측한다. 제안하는 알고리즘은 대형 ATV에서 소형 UGV까지 다양한 로봇 플랫폼의 주행 스타일을 반영한 주행 가능성을 추정할 수 있도록 설계되었다.
본 논문에서, 우리는 다양한 실제 세계의 조건에서 동작할 수 있는 이미지와 포인트 클라우드의 융합 프레임워크를 제시한다. 우리는 제안하는 알고리즘을 다양한 플랫폼 조건 (자동차, 대형 및 소형 UGV), 다양한 지형 조건(포장 도로, 들판, 산), 다양한 빛과 날씨 조건(아침, 해질녘, 비, 안개) 에서의 실험을 통해 검증했다. 제안하는 프레임워크는 다양한 조건에 따라 변동하는 데이터를 입력받고 안정적이고 보강된 지식을 생성하여 후속 응용 프로그램의 안정성을 보장하는 완충 역할을 수행할 수 있다. 따라서 우리는 이 프레임워크가 강력한 자율주행을 위한 초석이 되리라 기대한다.1 Introduction 1
1.1 Background and Motivations 1
1.2 Contributions and Outline of the Dissertation 3
1.2.1 EFGHNet: A Versatile Image-to-Point Cloud Registration Network for Extreme Outdoor Environment 3
1.2.2 ABCD: Attentive Bilateral Convolutional Network for Robust Depth Completion 4
1.2.3 Traversability Estimation Based on Footprint Supervision in an Off-road Environment 5
2 EFGHNet: A Versatile Image-to-Point Cloud Registration Network for Extreme Outdoor Environment 6
2.1 Introduction 6
2.2 Related Work 10
2.2.1 Image-based Localization 10
2.2.2 Camera-LiDAR Extrinsic Calibration 11
2.3 Methods 12
2.3.1 E3 Network 12
2.3.2 Horizon Network 14
2.3.3 Forward-axis Network 14
2.3.4 Gather Network 16
2.4 Experiments 18
2.4.1 Implementation Details 18
2.4.2 Test Set Configurations 18
2.4.3 Image-based Localization 18
2.4.4 Camera-LiDAR Extrinsic Calibration 22
2.5 Conclusions 23
3 ABCD: Attentive Bilateral Convolutional Network for Robust Depth Completion 25
3.1 Introduction 25
3.2 Related Work 29
3.2.1 Depth Completion 29
3.2.2 3D Deep Learning 29
3.3 Methods 30
3.3.1 Preliminary: Bilateral Convolutional Layer 30
3.3.2 Attentive Bilateral Convolutional Layer 31
3.3.3 Feature Projection 34
3.3.4 Network Overview 35
3.3.5 Implementation Details 37
3.4 Experiments 37
3.4.1 Experimental Setup 37
3.4.2 Evaluation on the KITTI Dataset 38
3.4.3 Evaluation on the VirtualKITTI2 Dataset 40
3.4.4 Ablation Study 40
3.5 Discussion 42
3.5.1 ABCL-based Point Encoder 42
3.5.2 Weight Map 43
3.6 Conclusions 43
4 Traversability Estimation Based on Footprint Supervision in an Off-road Environment 45
4.1 Introduction 45
4.2 Related Work 49
4.2.1 Traversability Estimation 49
4.2.2 Dynamic Filter Layer 50
4.3 Methods 51
4.3.1 Inter-modality Joint-control Kernel Layer 51
4.3.2 Footprint Supervision 54
4.3.3 Network Architecture 57
4.4 Experiments 58
4.4.1 Dataset 58
4.4.2 Experiments on the Rellis-3D Dataset 59
4.4.3 Experiments on Custom Dataset 61
4.5 Conclusions 61
5 Conclusion 63
Abstract (In Korean) 77박
Follow the Footprints: Self-supervised Traversability Estimation for Off-road Vehicle Navigation based on Geometric and Visual Cues
No full textIn this study, we address the off-road traversability estimation problem,
that predicts areas where a robot can navigate in off-road environments. An
off-road environment is an unstructured environment comprising a combination of
traversable and non-traversable spaces, which presents a challenge for
estimating traversability. This study highlights three primary factors that
affect a robot's traversability in an off-road environment: surface slope,
semantic information, and robot platform. We present two strategies for
estimating traversability, using a guide filter network (GFN) and footprint
supervision module (FSM). The first strategy involves building a novel GFN
using a newly designed guide filter layer. The GFN interprets the surface and
semantic information from the input data and integrates them to extract
features optimized for traversability estimation. The second strategy involves
developing an FSM, which is a self-supervision module that utilizes the path
traversed by the robot in pre-driving, also known as a footprint. This enables
the prediction of traversability that reflects the characteristics of the robot
platform. Based on these two strategies, the proposed method overcomes the
limitations of existing methods, which require laborious human supervision and
lack scalability. Extensive experiments in diverse conditions, including
automobiles and unmanned ground vehicles, herbfields, woodlands, and farmlands,
demonstrate that the proposed method is compatible for various robot platforms
and adaptable to a range of terrains. Code is available at
https://github.com/yurimjeon1892/FtFoot.Comment: Accepted to IEEE International Conference on Robotics and Automation
(ICRA) 202
Dual Targeting of EZH2 Degradation and EGFR/HER2 Inhibition for Enhanced Efficacy against Burkitt’s Lymphoma
No full textEZH2, a histone methyltransferase, contributes significantly to cancer cell survival and proliferation. Although various EZH2 inhibitors have demonstrated promise in treating lymphoma, they have not fully managed to curb lymphoma cell proliferation despite effective reduction of the H3K27me3 mark. We used MS1943, an EZH2 selective degrader, which successfully diminishes EZH2 levels in lymphoma cells. Additionally, lapatinib, a dual inhibitor of the epidermal growth factor receptor (EGFR) and human epidermal growth factor receptor 2 (HER2) tyrosine kinases, targets a receptor protein that regulates cell growth and division. The overexpression of this protein is often observed in lymphoma cells. Our study aims to combine these two therapeutic targets to stimulate apoptosis pathways and potentially suppress Burkitt’s lymphoma cell survival and proliferation in a complementary and synergistic manner. We observed that a combination of MS1943 and lapatinib induced apoptosis in Daudi cells and caused cell cycle arrest at the S and G2/M phases in both Ramos and Daudi cells. This strategy, using a combination of MS1943 and lapatinib, presents a promising therapeutic approach for treating lymphoma and potentially Burkitt’s lymphoma
Overcoming the therapeutic limitations of EZH2 inhibitors in Burkitt’s lymphoma: a comprehensive study on the combined effects of MS1943 and Ibrutinib
No full textEnhancer of zeste homolog 2 (EZH2) and Bruton’s tyrosine kinase (BTK) are both key factors involved in the development and progression of hematological malignancies. Clinical studies have demonstrated the potential of various EZH2 inhibitors, which target the methyltransferase activity of EZH2, for the treatment of lymphomas. However, despite their ability to effectively reduce the H3K27me3 levels, these inhibitors have shown limited efficacy in blocking the proliferation of lymphoma cells. To overcome this challenge, we employed a hydrophobic tagging approach utilizing MS1943, a selective EZH2 degrader. In this study, we investigated the inhibitory effects of two drugs, the FDA-approved EZH2 inhibitor Tazemetostat, currently undergoing clinical trials, and the novel drug MS1943, on Burkitt’s lymphoma. Furthermore, we assessed the potential synergistic effect of combining these drugs with the BTK inhibitor Ibrutinib. In this study, we evaluated the effects of combination therapy with MS1943 and Ibrutinib on the proliferation of three Burkitt’s lymphoma cell lines, namely RPMI1788, Ramos, and Daudi cells. Our results demonstrated that the combination of MS1943 and Ibrutinib significantly suppressed cell proliferation to a greater extent compared to the combination of Tazemetostat and Ibrutinib. Additionally, we investigated the underlying mechanisms of action and found that the combination therapy of MS1943 and Ibrutinib led to the upregulation of miR29B-mediated p53-upregulated modulator of apoptosis PUMA, BAX, cleaved PARP, and cleaved caspase-3 in Burkitt’s lymphoma cells. These findings highlight the potential of this innovative therapeutic strategy as an alternative to traditional EZH2 inhibitors, offering promising prospects for improving treatment outcomes in Burkitt’s lymphoma
Image_1_Overcoming the therapeutic limitations of EZH2 inhibitors in Burkitt’s lymphoma: a comprehensive study on the combined effects of MS1943 and Ibrutinib.jpeg
No full textEnhancer of zeste homolog 2 (EZH2) and Bruton’s tyrosine kinase (BTK) are both key factors involved in the development and progression of hematological malignancies. Clinical studies have demonstrated the potential of various EZH2 inhibitors, which target the methyltransferase activity of EZH2, for the treatment of lymphomas. However, despite their ability to effectively reduce the H3K27me3 levels, these inhibitors have shown limited efficacy in blocking the proliferation of lymphoma cells. To overcome this challenge, we employed a hydrophobic tagging approach utilizing MS1943, a selective EZH2 degrader. In this study, we investigated the inhibitory effects of two drugs, the FDA-approved EZH2 inhibitor Tazemetostat, currently undergoing clinical trials, and the novel drug MS1943, on Burkitt’s lymphoma. Furthermore, we assessed the potential synergistic effect of combining these drugs with the BTK inhibitor Ibrutinib. In this study, we evaluated the effects of combination therapy with MS1943 and Ibrutinib on the proliferation of three Burkitt’s lymphoma cell lines, namely RPMI1788, Ramos, and Daudi cells. Our results demonstrated that the combination of MS1943 and Ibrutinib significantly suppressed cell proliferation to a greater extent compared to the combination of Tazemetostat and Ibrutinib. Additionally, we investigated the underlying mechanisms of action and found that the combination therapy of MS1943 and Ibrutinib led to the upregulation of miR29B-mediated p53-upregulated modulator of apoptosis PUMA, BAX, cleaved PARP, and cleaved caspase-3 in Burkitt’s lymphoma cells. These findings highlight the potential of this innovative therapeutic strategy as an alternative to traditional EZH2 inhibitors, offering promising prospects for improving treatment outcomes in Burkitt’s lymphoma.</p
Image_2.jpeg
No full textEnhancer of zeste homolog 2 (EZH2) and Bruton’s tyrosine kinase (BTK) are both key factors involved in the development and progression of hematological malignancies. Clinical studies have demonstrated the potential of various EZH2 inhibitors, which target the methyltransferase activity of EZH2, for the treatment of lymphomas. However, despite their ability to effectively reduce the H3K27me3 levels, these inhibitors have shown limited efficacy in blocking the proliferation of lymphoma cells. To overcome this challenge, we employed a hydrophobic tagging approach utilizing MS1943, a selective EZH2 degrader. In this study, we investigated the inhibitory effects of two drugs, the FDA-approved EZH2 inhibitor Tazemetostat, currently undergoing clinical trials, and the novel drug MS1943, on Burkitt’s lymphoma. Furthermore, we assessed the potential synergistic effect of combining these drugs with the BTK inhibitor Ibrutinib. In this study, we evaluated the effects of combination therapy with MS1943 and Ibrutinib on the proliferation of three Burkitt’s lymphoma cell lines, namely RPMI1788, Ramos, and Daudi cells. Our results demonstrated that the combination of MS1943 and Ibrutinib significantly suppressed cell proliferation to a greater extent compared to the combination of Tazemetostat and Ibrutinib. Additionally, we investigated the underlying mechanisms of action and found that the combination therapy of MS1943 and Ibrutinib led to the upregulation of miR29B-mediated p53-upregulated modulator of apoptosis PUMA, BAX, cleaved PARP, and cleaved caspase-3 in Burkitt’s lymphoma cells. These findings highlight the potential of this innovative therapeutic strategy as an alternative to traditional EZH2 inhibitors, offering promising prospects for improving treatment outcomes in Burkitt’s lymphoma.</p
Remote Plasma Atomic Layer Deposition of SiNx Using Cyclosilazane and H2/N2 Plasma
No full textSilicon nitride (SiNx) thin films using 1,3-di-isopropylamino-2,4-dimethylcyclosilazane (CSN-2) and N2 plasma were investigated. The growth rate of SiNx thin films was saturated in the range of 200–500 °C, yielding approximately 0.38 Å/cycle, and featuring a wide process window. The physical and chemical properties of the SiNx films were investigated as a function of deposition temperature. As temperature was increased, transmission electron microscopy (TEM) analysis confirmed that a conformal thin film was obtained. Also, we developed a three-step process in which the H2 plasma step was introduced before the N2 plasma step. In order to investigate the effect of H2 plasma, we evaluated the growth rate, step coverage, and wet etch rate according to H2 plasma exposure time (10–30 s). As a result, the side step coverage increased from 82% to 105% and the bottom step coverages increased from 90% to 110% in the narrow pattern. By increasing the H2 plasma to 30 s, the wet etch rate was 32 Å/min, which is much lower than the case of only N2 plasma (43 Å/min)
