29 research outputs found
diffConv: Analyzing Irregular Point Clouds with an Irregular View
No full textStandard spatial convolutions assume input data with a regular neighborhood
structure. Existing methods typically generalize convolution to the irregular
point cloud domain by fixing a regular "view" through e.g. a fixed neighborhood
size, where the convolution kernel size remains the same for each point.
However, since point clouds are not as structured as images, the fixed neighbor
number gives an unfortunate inductive bias. We present a novel graph
convolution named Difference Graph Convolution (diffConv), which does not rely
on a regular view. diffConv operates on spatially-varying and density-dilated
neighborhoods, which are further adapted by a learned masked attention
mechanism. Experiments show that our model is very robust to the noise,
obtaining state-of-the-art performance in 3D shape classification and scene
understanding tasks, along with a faster inference speed.Comment: Accepted by ECCV 202
Effect of HfO2 Particles on Ceramic Coating Fabricated on Ti6Al4V Alloy via Plasma Electrolytic Oxidation
No full textHafnium dioxide (HfO2) has a wide bandgap and high dielectric constant. We prepared ceramic coatings on Ti6Al4V alloys via plasma electrolytic oxidation (PEO) in an electrolyte with HfO2 particles. The influence of the HfO2 particles on the microstructure, phase composition, elemental distribution, and corrosion resistance of the PEO coatings was systematically investigated. The results showed that the addition of HfO2 increased the oxidation voltage (from 462 to 472 V) and promoted the microarc sintering reaction so that the thickness and hardness of the resulting PEO coating increased. Moreover, the quantity of the micropores on the coating surface caused by the discharge decreased after adding the HfO2 particles. The X-ray diffraction patterns confirmed that the HfO2 particles were incorporated into the coating by remelting and sintering the microarc. Furthermore, the corrosion resistance of the PEO coating was remarkably increased after introducing HfO2, which was attributed to the increase in the electrode potential and the densification of the coating structure
DTU-Net: Learning Topological Similarity for Curvilinear Structure Segmentation
No full textCurvilinear structure segmentation is important in medical imaging,
quantifying structures such as vessels, airways, neurons, or organ boundaries
in 2D slices. Segmentation via pixel-wise classification often fails to capture
the small and low-contrast curvilinear structures. Prior topological
information is typically used to address this problem, often at an expensive
computational cost, and sometimes requiring prior knowledge of the expected
topology.
We present DTU-Net, a data-driven approach to topology-preserving curvilinear
structure segmentation. DTU-Net consists of two sequential, lightweight U-Nets,
dedicated to texture and topology, respectively. While the texture net makes a
coarse prediction using image texture information, the topology net learns
topological information from the coarse prediction by employing a triplet loss
trained to recognize false and missed splits in the structure. We conduct
experiments on a challenging multi-class ultrasound scan segmentation dataset
as well as a well-known retinal imaging dataset. Results show that our model
outperforms existing approaches in both pixel-wise segmentation accuracy and
topological continuity, with no need for prior topological knowledge.Comment: 12 pages, 4 figure
S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR
No full textScene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR). However, previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. This pipeline may potentially compromise the flexibility of learning multimodal representations, consequently constraining the overall effectiveness. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed S^2Former-OR, aimed to complementally leverage multi-view 2D scenes and 3D point clouds for SGG in an end-to-end manner. Concretely, our model embraces a View-Sync Transfusion scheme to encourage multi-view visual information interaction. Concurrently, a Geometry-Visual Cohesion operation is designed to integrate the synergic 2D semantic features into 3D point cloud features. Moreover, based on the augmented feature, we propose a novel relation-sensitive transformer decoder that embeds dynamic entity-pair queries and relational trait priors, which enables the direct prediction of entity-pair relations for graph generation without intermediate steps. Extensive experiments have validated the superior SGG performance and lower computational cost of S^2Former-OR on 4D-OR benchmark, compared with current OR-SGG methods, e.g., 3 percentage points Precision increase and 24.2M reduction in model parameters. We further compared our method with generic single-stage SGG methods with broader metrics for a comprehensive evaluation, with consistently better performance achieved.This work has been accepted by TMI202
Incorporating Clinical Guidelines Through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring
No full textThe Prostate Imaging Reporting and Data System (PI-RADS) is pivotal in the diagnosis of clinically significant prostate cancer through MRI imaging. Current deep learning-based PI-RADS scoring methods often lack the incorporation of common PI-RADS clinical guideline (PICG) utilized by radiologists, potentially compromising scoring accuracy. This paper introduces a novel approach that adapts a multi-modal large language model (MLLM) to incorporate PICG into PI-RADS scoring model without additional annotations and network parameters. We present a designed two-stage fine-tuning process aiming at adapting a MLLM originally trained on natural images to the MRI images while effectively integrating the PICG. Specifically, in the first stage, we develop a domain adapter layer tailored for processing 3D MRI inputs and instruct the MLLM to differentiate MRI sequences. In the second stage, we translate PICG for guiding instructions from the model to generate PICG-guided image features. Through such a feature distillation step, we align the scoring network’s features with the PICG-guided image features, which enables the model to effectively incorporate the PICG information. We develop our model on a public dataset and evaluate it on an in-house dataset. Experimental results demonstrate that our approach effectively improves the performance of current scoring networks. Code is available at: https://github.com/med-air/PICG2scorin
Conversion of female germline stem cells from neonatal and prepubertal mice into pluripotent stem cells
No full textTri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms
No full textA comprehensive understanding of surgical scenes allows for monitoring of the
surgical process, reducing the occurrence of accidents and enhancing efficiency
for medical professionals. Semantic modeling within operating rooms, as a scene
graph generation (SGG) task, is challenging since it involves consecutive
recognition of subtle surgical actions over prolonged periods. To address this
challenge, we propose a Tri-modal (i.e., images, point clouds, and language)
confluence with Temporal dynamics framework, termed TriTemp-OR. Diverging from
previous approaches that integrated temporal information via memory graphs, our
method embraces two advantages: 1) we directly exploit bi-modal temporal
information from the video streaming for hierarchical feature interaction, and
2) the prior knowledge from Large Language Models (LLMs) is embedded to
alleviate the class-imbalance problem in the operating theatre. Specifically,
our model performs temporal interactions across 2D frames and 3D point clouds,
including a scale-adaptive multi-view temporal interaction (ViewTemp) and a
geometric-temporal point aggregation (PointTemp). Furthermore, we transfer
knowledge from the biomedical LLM, LLaVA-Med, to deepen the comprehension of
intraoperative relations. The proposed TriTemp-OR enables the aggregation of
tri-modal features through relation-aware unification to predict relations so
as to generate scene graphs. Experimental results on the 4D-OR benchmark
demonstrate the superior performance of our model for long-term OR streaming.Comment: 10 pages, 4 figures, 3 table
Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring
No full textThe Prostate Imaging Reporting and Data System (PI-RADS) is pivotal in the diagnosis of clinically significant prostate cancer through MRI imaging. Current deep learning-based PI-RADS scoring methods often lack the incorporation of common PI-RADS clinical guideline~(PICG) utilized by radiologists, potentially compromising scoring accuracy. This paper introduces a novel approach that adapts a multi-modal large language model (MLLM) to incorporate PICG into PI-RADS scoring model without additional annotations and network parameters. We present a designed two-stage fine-tuning process aiming at adapting a MLLM originally trained on natural images to the MRI images while effectively integrating the PICG. Specifically, in the first stage, we develop a domain adapter layer tailored for processing 3D MRI inputs and instruct the MLLM to differentiate MRI sequences. In the second stage, we translate PICG for guiding instructions from the model to generate PICG-guided image features. Through such a feature distillation step, we align the scoring network\u27s features with the PICG-guided image features, which enables the model to effectively incorporate the PICG information. We develop our model on a public dataset and evaluate it on an in-house dataset. Experimental results demonstrate that our approach effectively improves the performance of current scoring networks. Code is available at: https://github.com/med-air/PICG2scorin
I saw, I conceived, I concluded: Progressive Concepts as Bottlenecks
No full textConcept bottleneck models (CBMs) include a bottleneck of human-interpretable
concepts providing explainability and intervention during inference by
correcting the predicted, intermediate concepts. This makes CBMs attractive for
high-stakes decision-making. In this paper, we take the quality assessment of
fetal ultrasound scans as a real-life use case for CBM decision support in
healthcare. For this case, simple binary concepts are not sufficiently
reliable, as they are mapped directly from images of highly variable quality,
for which variable model calibration might lead to unstable binarized concepts.
Moreover, scalar concepts do not provide the intuitive spatial feedback
requested by users.
To address this, we design a hierarchical CBM imitating the sequential expert
decision-making process of "seeing", "conceiving" and "concluding". Our model
first passes through a layer of visual, segmentation-based concepts, and next a
second layer of property concepts directly associated with the decision-making
task. We note that experts can intervene on both the visual and property
concepts during inference. Additionally, we increase the bottleneck capacity by
considering task-relevant concept interaction.
Our application of ultrasound scan quality assessment is challenging, as it
relies on balancing the (often poor) image quality against an assessment of the
visibility and geometric properties of standardized image content. Our
validation shows that -- in contrast with previous CBM models -- our CBM models
actually outperform equivalent concept-free models in terms of predictive
performance. Moreover, we illustrate how interventions can further improve our
performance over the state-of-the-art
Advances and Challenges in Electrolyte Development for Magnesium-Sulfur Batteries: A Comprehensive Review
No full textMagnesium–sulfur batteries are an emerging technology. With their elevated theoretical energy density, enhanced safety, and cost-efficiency, they have the ability to transform the energy storage market. This review investigates the obstacles and progress made in the field of electrolytes which are especially designed for magnesium–sulfur batteries. The primary focus of the review lies in identifying electrolytes that can facilitate the reversible electroplating and stripping of Mg2+ ions whilst maintaining compatibility with sulfur cathodes and other battery components. The review also addresses the critical issue of managing the shuttle effect on soluble magnesium polysulfide by looking at the innovative engineering methods used at the sulfur cathode’s interface and in the microstructure design, both of which can enhance the reaction kinetics and overall battery efficiency. This review emphasizes the significance of reaction mechanism analysis from the recent studies on magnesium–sulfur batteries. Through analysis of the insights proposed in the latest literature, this review identifies the gaps in the current research and suggests future directions which can enhance the electrochemical performance of Mg-S batteries. Our analysis highlights the importance of innovative electrolyte solutions and provides a deeper understanding of the reaction mechanisms in order to overcome the existing barriers and pave the way for the practical application of Mg-S battery technology
