DR-NTU (Digital Repository of NTU)
Not a member yet
116018 research outputs found
Sort by
Enhanced contrastive learning for semantic sentence embeddings
No full textThe process of semantic sentence embedding is a pivotal component in the field of
natural language processing. This procedure involves the transformation of sentences
from discrete linguistic spaces into a high-dimensional vector space. Such
transformations are designed to retain the semantic integrity of the original sentences,
thereby proving to be instrumental in numerous downstream applications.
The core component of the embedding system is its embedding model. Initial research
on sentence embeddings predominantly adopted the concepts of word2vec
while recent advancements utilize pre-trained language models (PLMs) for sentence
embedidngs. More recently, the application of contrastive learning techniques
to sentence embedding learning has emerged. This approach learns representations
by drawing together positive examples while distancing negative ones.
Consequently, numerous sentence representation methods based on contrastive
learning have been developed and have progressively become the prevalent approach
for sentence embeddings.
Despite their efficacy, several challenges and issues persist that need to be addressed.
A key limitation of contemporary embedding methodologies is their dependence
on supervised data, which is often scarce or non-existent in the context
of in-domain settings. This absence of in-domain data hinders the fine-tuning of
embeddings for specific tasks or domains, thereby diminishing their transferability
in out-of-distribution scenarios. Additionally, contrastive learning methods are often
plagued by a representation degeneration issue, where the embeddings may fail
to encapsulate the complete semantic information of a sentence. This constraint
inhibits the model’s representational ability, potentially leading to subpar performance
in subsequent NLP tasks. Moreover, since semantics is intrinsically linked
to human comprehension of knowledge, it poses a challenge to devise analytical
solutions for enhancing model performance. The crux of the problem lies in formulating
a more efficacious learning framework for sentence embeddings. Lastly,
aligning the learning framework more closely with human semantic understanding
remains a long-term objective.
In this thesis, we aim to tackle the aforementioned issues individually. Firstly,
we examine the transferability properties of sentence embeddings in unsupervised
settings and out-of-domain applications for the first time, discovering that current
popular methods exhibit low transferability. As a result, we introduce a novel
framework, BlendCSE, designed to generate embeddings with enhanced transferability.
Secondly, we enhance recent contrastive learning-based sentence embeddings
by investigating their augmentation strategies and addressing feature collapse
problems. We pinpoint the noise issue in embeddings and the rank bottleneck
issue in data samples, proposing a dimensional contrastive learning approach to
tackle these challenges. Finally, we explore the knowledge distillation strategy to
further improve the baseline performance of sentence embeddings. During our examination,
we observe that the standard knowledge distillation framework yields
only marginal improvement over the distillation teacher, which arises from similarity
logits and results in overfitting issues. To mitigate such variance, we put
forward a Group-P shuffling regulation strategy and a teacher ensemble strategy.
Both approaches significantly decrease the loss gap between training and testing
sets, indicating their robust capability to prevent overfitting.
In conclusion, this dissertation recognizes the issues of transferability, feature
collapse, and overfitting associated with the conventional embedding model. It
suggests multiple strategies to counteract these challenges. Empirical evidence
highlights the effectiveness of these proposed methods in resolving the identified
problems, leading to a substantial enhancement in the baseline performance.Doctor of Philosoph
Bipolar-barrier tunnel heterostructures for high-sensitivity mid-wave infrared photodetection
No full textThe rapid development of modern infrared optoelectronic technology has driven a growing demand for high-sensitivity mid-wave infrared (MWIR) photodetectors capable of reliable room-temperature operation. Achieving optimal specific detectivity, a critical performance metric for MWIR photodetection, remains challenging due to inherent limitations imposed such as high dark current, low optical absorption, or both. To address these challenges, we present an approach based on a bipolar-barrier architecture featuring a black phosphorus (BP)/MoTe2/BP tunnel heterostructure integrated with an Au reflector. This configuration delivers simultaneous electrical and optical enhancement, effectively suppressing dark currents and significantly increasing optical absorption. The bipolar-barrier structure minimizes dark current by blocking thermally excited and bias-induced carrier leakage, while facilitating efficient tunneling of photogenerated carriers via trap-assisted photogating mechanisms. In addition, the Au reflector enhances optical absorption through interference effects. As a result, the heterostructure achieves remarkable performance metrics, including a room-temperature specific detectivity of ∼3.0 × 1010 cm Hz0.5 W−1, a high responsivity of ∼4 A W−1, and an external quantum efficiency of ∼140% within the MWIR range. These results establish the bipolar-barrier tunnel heterostructure as a highly efficient platform, paving the way for the next generation of advanced infrared optoelectronic devices.Agency for Science, Technology and Research (A*STAR)National Research Foundation (NRF)Published versionThis work was paritially supported by the Singapore Agency for Science, Technology and Research (A*STAR) (M22K2c0080, R23I0IR041 and M23M2b0056), and National Research Foundation Singapore (Award No. NRFCRP22-2019-0007, NRF-CRP29-2022-0003, and NRF-MSG-2023-0002)
How well can an LLM chatbot clarify queries by medical students in biomedical TBL?
No full textThis report analyzes an LLM chatbot’s responses in a biomedical team-based learning (TBL) classroom for first-year medical students. The chatbot clarified low-complexity queries but provided incorrect conclusions at higher levels. Mitigating its tone of undue confidence and tuning it to the correct knowledge level are critical for its effective use.Submitted/Accepted versio
Is interfaith dialogue in Southeast Asia losing its relevance? A call for renewal
No full textAs society changes digitally, generationally, and demographically, older interreligious dialogue models struggle to be effective, and a renewal is arguably needed.Published versio
Cardiovascular risk conversations in Singapore's primary care clinics
No full textPrimary health care used to provide acute episodic care but is shifting towards long-term management of patient’s chronic illnesses and health promotion and prevention. This warrants a need for building up better communication and doctor-patient relationships in primary care. This thesis aims to do so through studying conversations between a primary care physician and their patients. A combination of conversation analysis of five video-recorded primary care consultations by a private general practitioner (GP) with five unique patients, and individual interviews with the GP and three patients was used to study the communication between physicians and patients from an interactional perspective together with experiences of the individuals in the conversations. The data demonstrated the use of various interactional devices such as vocalisations, repeats, repairs, numbers, labels, and measurement systems in the collaborative construction of understanding and turn-taking between the physician and patients. The building of trust in the doctor-patient relationship was crucial, with this trust achieved via a history of the GP providing efficient and effective care while allowing the patients feel comfortable raising questions about their health and health care. Time was a main factor reported by the GP in balancing shared decision making while ensuring delivery of care to improve patients’ health, with the challenges intensified when patients display resistance to recommended care. Identifying strategies and support to enable GPs to improve doctor-patient relationships within the time constrains in consultations can be explored in future research.Master's degre
AttenPU: an area efficient attention processor with reconfigurable FP8 precision and dataflow
No full textEfficient numerical representation is crucial for deep learning accelerators, especially for large language models (LLMs). The 8-bit-floating-point (FP8) data representation achieves higher precision and fewer quantization efforts than integer, which has been proven inevitable in attention-based accelerators for LLMs. Therefore, area-efficient design techniques for FP8 play a central role in lowering LLMs chip’s budget. This paper presents AttenPU, which is built upon reconfigurable FP8 units and supports E4M3 for inference and E5M2 for training. Bidirectional dataflow is exploited to enable AttenPU to interact with FP32 coprocessor to reduce latency. The design achieves a low FP8-to-INT8 area ratio of 1.63×, an area efficiency of 193.5 GFLOPS/mm2, with an 87.05% reduction in the latency of RTX3090 GPU.Published versionThis work was funded by NSFC under Grant No.62372442, State Key Lab of Processors, Institute of Computing Technology, CAS under Grant No.CLQ202308, Guangdong Basic and Applied Basic Research Foundation under Grant No.2023A1515012842, Shenzhen Science and Technology Program under Grant No.JCYJ20220818101607015
Reducing T-depth and T-count in quantum multiplication using compressor primitives
No full textOptimization of quantum multiplication is a critical area of study due to its pivotal role in quantum algorithms such as Shor’s factorization. Every time a quantum multiplier is used, it repeatedly executes several key components to perform the multiplication. Most current works have focused on using components such as the basic half and full-adder designs, which have limited efficiency. In this paper, we demonstrate that by using a generalized (m:k) compressor-based Wallace Tree one can significantly improve efficiency; this method achieves reductions of up to 92.8% in T-Depth and 55.6% in T-Count while maintaining a competitive Qubit-Count through brute-force and dynamic programming optimization.National Research Foundation (NRF)Ministry of Education (MOE)Published versionWe gratefully acknowledge the support of QEP grant NRF2021- QEP2-02-P05 and MoE AcRF Tier 1 award RT10/23 for this work
Additive manufacturing in spatial patterning for spinal cord injury treatment.
No full textCombinatorial treatments integrating cells and biomolecules within scaffolds have been investigated to address the multifactorial nature of spinal cord injury (SCI). Current regenerative treatments have been ineffective as they do not consider the spatial positions of various cell types to effectively form functional neural pathways. Emulating the complex heterogeneity of cells in the native spinal cord requires translating the existing biological understanding of spatial patterning in neural development, as well as the influence of biomolecule and mechanical patterning on regional specification and axonal regeneration, to engineer a scaffold for spinal cord regeneration. This review explores the potential of 3D bioprinting to precisely control material, cell and drug patterns in scaffolds, achieving spatial phenotype specification and providing axonal guidance to form appropriate connections. We also discuss the application of extrusion-based and digital light processing bioprinting in integrating mechanical, chemical and biological cues within a scaffold to advance spatially patterned 3D bioprinted scaffold, as well as current challenges and future perspectives in these bioengineering strategies.National Research Foundation (NRF)Ministry of Education (MOE)Submitted/Accepted versionThis research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme (IntraCREATE grant award number: NRF2019-THE002-0001). Partial funding support from the MOE Tier 1 grants (RG92/22, RT21/23 and RG27/24) is also acknowledged. C.K. would like to acknowledge NTU for providing the Nanyang Research Scholarship to carry out the research works
STENCIL: Subject-driven generation with context guidance
No full textThe advent of text-to-image diffusion models marked a breakthrough in generative AI, enabling
users to create diverse images from textual prompts. An application of text-to-image diffusion
models is subject-driven generation, which offers even greater user control over the generation
process by enabling subject customization using a few reference images. However, subject-
driven generation remains a challenge. Existing methods struggle to generate diverse, text-
aligned images while also achieving subject consistency. Moreover, existing methods suffer a
trade-off between quality and efficiency. Achieving high-quality diverse results requires fine-
tuning on large-scale diffusion models, which is time-consuming. Conversely, fine-tuning on
smaller diffusion models improves efficiency but compromises subject consistency and image
quality. To address these limitations, we present Stencil. Stencil leverages Context Guidance,
a novel technique where instead of using a single diffusion model, we use two. Specifically,
we use a large pre-trained diffusion model to contextually guide a smaller fine-tuned model
perform generation. This allows us to combine the superior generalization capabilities of large
models with the efficient fine-tuning of small models. Stencil delivers state-of-the-art perfor-
mance in both text-image alignment and subject consistency and sets a new benchmark on
DreamBench while being ×20 faster than DreamBooth, the previous state-of-the-art model.Bachelor's degre
Cross-modal person re-identification (Re-lD) based on RGB-infrared fusion
No full textin the real world. Especially in the field of surveillance, such as night monitoring,
cross-camera pedestrian tracking, and smart city security systems, infrared
cameras can work in low-light conditions, which can make up for the shortcomings
of RGB cameras in low-light conditions at night. However, there is
a huge modality difference between RGB and IR data, which seriously affects
feature consistency and retrieval accuracy.
To address this problem, an enhanced SAAI framework is proposed by incorporating
the InfoNCE loss into the original architecture to achieve more effective
cross-modal feature alignment. The InfoNCE loss function is employed to draw
positive sample pairs from visible and infrared modalities closer together, while
pushing apart unrelated samples, thereby promoting the learning of discriminative
and modality-invariant feature embeddings.
Extensive evaluations on two public benchmark datasets, SYSU-MM01 and RegDB,
validate the superiority of the proposed approach. And the result shows these
improvements demonstrate the effectiveness of the proposed approach in bridging
the modality gap and enhancing retrieval robustness.Master's degre