DSpace@RPI (Rensselaer Polytechnic Institute)
Not a member yet
6809 research outputs found
Sort by
Development and application of robustness evaluation techniques for ai/ml models derived from biological data
No full textMay 2024School of EngineeringArtificial intelligence (AI) and machine learning (ML) models are frequently used to analyze large, complex biomedical datasets. These types of models are commonly used for tasks such
as disease diagnosis, biomarker identification, and network analysis. However, the data that
these models are derived from and used on are often characterized by significant amounts of
noise resulting from patient-to-patient heterogeneity, different measurement protocols, and
other commonly encountered sources of noise. This creates a problem for the robustness of
these models and one outcome is that relatively few AI/ML models have seen widespread
clinical use. As such, evaluation and subsequent improvement of AI/ML model robustness
is vital for clinical translation.
This dissertation examines methods which will allow researchers to quantify model ro
bustness, and further demonstrates how to develop more robust AI/ML models. First, this
work defines a framework which can be used to evaluate the robustness of an already-trained
biomarker-based diagnostic model. This is done by measuring the quality of the biomarkers
used to generate the classifier and observing the classifier’s performance when the data is
perturbed by several sources of noise. Next, a detailed investigation was performed that
looked at the robustness of deep learning medical image classification models in response to
being trained by data that was artificially perturbed. One key outcome from this evaluation
was that it was demonstrated that perturbing training samples results in excellent classifier
performance not only for noisy testing data but also does not sacrifice performance on unper
turbed images. This is especially important as a classifier will need to be able to perform well
on several distributions of data to truly be generalizable across multiple datasets. Finally, a
method for the creation of multi-omic co-expression networks of longitudinal biological data
was developed. The robustness of this model was assessed by noise perturbation of the data,
and further verified by comparing the model outcomes to known biological information.
By understanding how to measure and improve AI/ML model robustness, robust mod
els can be generated that perform well on diverse sets of data. In conclusion, this dissertation
lays the foundation for advancing the clinical applicability of AI/ML models by establishing
methodologies to assess and enhance their robustness in the face of inherent data noise.Ph
Towards an Ontology of Psychometric Measures
No full textThe field of psychometrics is a subdomain of psychology concerned with assessment of various social and psychological dimensions of humans. In the areas of mental healthcare and the clinical science that supports it, assessment with psychometric instruments remains a dominant strategy for discovery, understanding, and decision-making. This method predominantly relies on text-based distribution of assessment instruments (usually in the form of questionnaires), and significant limitations in terms of interoperability exist. To bridge this gap, we are developing the Psychometric Ontology of Experiences and Measures (POEM) as a domain ontology tailored to capture the relationships and semantics inherent in psychometric instruments. The central aim of this new ontology is to enable the structured representation of psychometric instruments to facilitate standardized assessment across different dimensions and contexts. Other aims are to support the accessibility and construction of instruments, and making machine-readable formats more readily available. POEM considers the connections of observable questionnaire elements and the complex set of latent (unobservable) variables that are the common targets of psychometric assessment. Further POEM supports the tracking of the provenance of assessments at the questionnaire, scale, and item levels in order to capture how assessments are created, used, reused, and translated.
The present work is the result of a collaboration of semantic engineers and clinical scientists to develop an ontology for psychometric instruments, adopting a bottom-up approach informed by competency questions and use cases generated by the domain experts in the team to ensure sufficient coverage. The developed ontology is designed to address diverse use cases spanning clinical services, research, and development of new psychometric instruments. POEM builds on the foundation of existing taxonomies and ontologies for psychological disorders like SNOMED-CT, the ICD, and the DSM, and integrates them with semantics describing the internal structure of psychometric assessments. While these existing taxonomies and ontologies include a comprehensive representation of psychological disorders and symptoms, POEM includes the semantics necessary to describe assessments, capturing items, scales, and provenance. POEM’s architecture builds upon established ontologies like the Virtual Solar-Terrestrial Observatory (VSTO), which is a semantic data framework that supports formal representations of physical quantities and their underlying representations with an instrument-focused subset (VSTO-I), and the Human-Aware Science Ontology (HAScO), which supports the description of scientific data for acquisition, preparation, and annotation across a variety of domains.
POEM facilitates the integration of observable assessment features, those that describe the architecture of an assessment instrument and its essential parts, with the unobservable semantics of the underlying concepts they represent. The majority of document-level features in POEM align with VSTO-I entities, including questionnaires, questionnaire items, and response options. POEM relates assessments to their underlying concepts on an item-level, relating items to the specific symptoms or constructs they address; more deeply, each response option for an item is shown to estimate a specific experience of the symptom in question. Crucially, POEM also supports the encoding of questionnaire items into scales and subscales, each of which assays a construct (e.g., a clinical syndrome such as depression) that may be present at some level in a human informant, further implying the existence or non-existence of a condition (e.g., a depressive disorder). Our current work focuses on the Revised Children’s Anxiety and Depression Scale (RCADS), which estimates elevations of common psychological disorders and symptoms in children and adolescents, with scales for depression and anxiety, as well as subscales for several specific anxiety disorders [2]. POEM also supports the documentation of the creation and reuse of assessments, scales, and items, including derivation, translation, and related provenance.
We evaluate POEM using our competency questions and use case scenarios with regards to the RCADS, an assessment that is suitable for evaluation due to its multiple nested scales, derived versions of different length and target respondents, many translations, and wide usage. Additionally, POEM’s utility is to be demonstrated by its integration into the Semantic Instrument Repository (SIR), a software infrastructure for managing and distributing knowledge graphs related to data acquisition instruments. SIR will harness POEM’s semantic richness to enable semantic search and retrieval functionalities, and facilitate instrument sharing in various formats, including the rendering of questionnaires into human-readable formats. SIR also supports the tracking of the provenance of assessments and their elements, in order to facilitate the creation and reuse of psychometric instruments.
We maintain a GitHub repository of the POEM ontology as well as related artifacts and documentation, available at the following: https://github.com/tetherless-world/POEMNational Institute of Mental Health (NIMH
Enhance machine reasoning via active learning with human rationales
No full textJanuary 2024School of ScienceThe main objective of this dissertation is to enhance model reasoning and explainability by introducing a new framework for Natural Language Generation (NLG). This framework generates rationales that are inspired by human reasoning, with the goal of providing insights into the underlying relationships present in the source data. By explicitly contributing to the final decision-making process, these rationales can help to improve the overall quality of the models. This process is implemented using an innovative Active Learning (AL) approach that incorporates human feedback on labels and rationales to select representative data to be annotated while minimizing the total amount of data annotation required. The proposed methodologies aim to address two limitations of current NLG advances, namely the lack of trustworthy model explainability and the scarcity of high-quality human feedback –- while transformer-based NLG models have demonstrated superior performance over various NLG tasks, they lack faithful explainability, which can lead to mistrust of their predictions. Additionally, high-quality human annotations are essential, but they are expensive and difficult to obtain in large quantities for many real-world domains. The dissertation proposes an approach to enhance the trustworthiness and explainability of NLG models. The approach is based on the human reasoning process, where intermediate rationales are generated about the source information first. These rationales are then served as guidance and explanations for the final predictions. The dissertation focuses on three crucial research questions to achieve this goal:
RQ 1. How can NLG models be guided to reason more comprehensible to humans?
RQ 2. Is the helpfulness of human rationales consistent with model predictions?
RQ 3. How to maximize the use of human feedback while minimizing annotation costs?
RQ 4. Can the proposed dual-model AL framework benefit from utilizing LLMs?
RQ 5. When do we need the proposed AL framework? As an initial step, this dissertation thoroughly evaluated the helpfulness of human rationales toward model predictions. The analysis was executed by evaluating the helpfulness in improving the prediction performance of NLG models at both the fine-tuning and inference stages, with a novel evaluation metric (RQ 1). The proposed framework was showcased through AL simulations on two NLG tasks with enhanced reasoning and explainability (RQ 2). The simulations provided evidence supporting the usefulness and efficiency, in terms of faster convergence and better performance, of incorporating human rationales in AL data selection with a novel AL sampling strategy (RQ 3). To extend the practical implication of the proposed framework with more advanced large language models (LLMs), this dissertation demonstrated the strengths and weaknesses of LLMs, such as FLAN-T5, in providing helpful rationales for predictions and resolving real-world NLG tasks. LLMs demonstrated strong potential for explanation generation in the proposed framework to further reduce human effort, despite LLMs suffering from low reliabilities for task-solving in real-world, domain-specific scenarios. The proposed framework could reduce annotation effort, maintain performance, and establish explainability in real-world tasks for NLG models (RQ 4). The final objective of the dissertation was to investigate the potential of the proposed AL framework to facilitate the development of domain-specific language models and large language models in real-world specialized domains.This dissertation executed a comprehensive benchmark and evaluation for state-of-the-art LLMs for six tasks in mental health detection via online posts. The evaluation results illustrated that traditional language models fine-tuned on specific tasks could perform comparable to generic LLMs, and multi-task fine-tuning for LLMs demonstrated promising results under low-resource scenarios. The observations shed light on the broad utility of the proposed AL framework to seamlessly support the development of domain-specific machine learning systems under low-resource scenarios, including task-specific fine-tuned traditional language models and instructional fine-tuned large language models (RQ 5). The proposed AL framework, which incorporates human feedback from labels and rationales, has the potential to facilitate a wide range of future research directions. The versatility of the proposed framework is manifested in improving the reliability of NLG models' task-solving in real-world scenarios and serving as the foundational learning framework for domain-specific NLG systems. Another promising future research direction is to develop LLM-simulated human agents and personalized assistants by actively fine-tuning LMs to simulate human behaviors and preferences. Moreover, a unified and automated annotation-learning framework could be achieved in the future, whereby the automated system automatically provides labels and rationales for unlabeled data, proactively queries humans for uncertain annotations, and iteratively learns from human feedback. This dissertation establishes the groundwork for more effective machine learning frameworks with faithful explainability, specialized NLG systems in real-world scenarios, and self-improving AI systems that can adjust to different preferences, such as personalized assistants.Ph
Understanding interfacial chemistry and electronic transport of solid electrolytes for solid-state battery development
No full textDecember 2023The development of superionic solid electrolytes (SEs) has proceeded rapidly these years, with the ionic conductivity of some electrolytes surpassing that of the conventional liquid electrolytes. However, incorporating these new materials into batteries to make solid-state batteries (SSBs) has proven difficult. The electrode/electrolyte interfacial resistance is being considered as the limiting factor, but the exam mechanisms of this resistance have not been fully explained. The first of part of my thesis focuses on understanding the interfacial chemistry between a high-capacity, low-cost conversion-type iron fluoride cathode and SEs. Two representative SEs, amorphous 75Li2S-25P2S5 (LPS) and glass-ceramic Li3YCl6 (LYC), for the sulfide- and halide-based conductors, respectively, are used in this study. The electrochemical characteristics of FeF2 in LPS SE exhibit drastic deviations from that in the liquid electrolyte batteries. Detailed characterizations by ex-situ X-ray diffraction (XRD) and X-ray absorption spectroscopy (XAS) revealed that irreversible transformation from FeF2 to FeS occurs when testing FeF2 in the sulfide-based SE. Changing the SE to halide-based materials has been shown to enable a complete conversion and reconversion of FeF2, with a specific capacity of 600 mAh/g at 0.1 C at an average potential of around 2.6 V. The characterization results from ex-situ XRD and XAS also confirmed the excellent electrochemical and chemical stability of LYC can enable a complete conversion and deconversion of FeF2. Due to the excellent interfacial stability, a halide based FeF2 cathode with an extremely long cycle life was developed. This work highlights the significant effects of SE compositions on the redox behaviour of FeF2 cathodes, and we expect the same concept of utilizing halide-based SEs can also be generalized to other conversion-type cathodes for the development of low-cost, low-toxicity and energy-dense SSBs. In addition to a long cycle life, the real application of solid-state batteries also requires the battery to have an extremely long calendar life, i.e., the life of a battery during storage. While calendar life has not been seriously considered as an issue for SSB development, it is proposed here that the long calendar life of LiPON-based thin film SSB cannot be assumed for sulfide- or halide-based SSBs due to the significant difference in the electronic conductivity of SEs (10-14 to 10 13 S/cm for LiPON vs. 10-9 to 10-8 S/cm for typical sulfide- and halide- based SEs). The second part of my thesis aims to understand the electronic transport properties in typical Li SEs. We propose that the electronic conductivities of SEs are overestimated from the conventional measurements. By revisiting direct current polarizations using two-blocking-electrode cells and the Hebb-Wagner approach, their sources of inaccuracy are provided and the anodic decomposition of SE is highlighted as the key source for the overestimated result. Modifications in the electrode selection and data interpretation are also proposed to approach the intrinsic electronic conductivity of SEs. A two-step polarization method is also proposed to estimate the electronic conductivity of sulfides that decompose during measurement. Measured by the modified approach, the electronic conductivities of all SEs are one or two orders of magnitude lower than the reported value. Despite that, the electronic conductivity of sulfides seems to be still quite high to enable SSBs with a long calendar life of >10 years, highlighting the critical need for a more careful study of electronic transport in lithium SEs.Ph
Foundations in aural architecture of jazz venues: an examination of room acoustics & multidimensional analysis of musician preference
No full textDecember 2023Architectural Acoustics has primarily focused on venues showcasing Western European Classical music. Consequently, acoustic parameters, subjective preference studies, and attribute vocabulary focus on the needs of the audience and the musician through a Western European Classical lens, excluding many other forms of music, including Rhythm and Blues, Rock, Blues, and Jazz. This research attempts to form a foundational study of Jazz venues, explicitly looking at room acoustic parameters and stage acoustics preference for the musician. An ethnographic survey and interview study were conducted to examine and rank the importance of auditory attributes of a Jazz musician. Additionally, subjects were asked to rank existing North American Jazz venues. An examination of room acoustics was conducted utilizing a 19-channel spherical microphone array, which enabled high-resolution spherical array beam processing and beamforming analysis. Standard omnidirectional and spatial room acoustic were investigated to understand the validity of applying existing acoustic parameters to Jazz venues. The measurements were also used to auralize using an Ambisonics loudspeaker. A subjective study investigated Jazz musicians' preference regarding stage acoustics of Jazz venues. A multidimensional scaling analysis revealed three factors of importance relating to timbre, clarity, early stage, and early reflection.Ph
Complex Semantic Tabular Interpretation using SDD-Gen
No full textKnowledge graphs have become an essential technology for both businesses and governments. They enable a wide variety of critical tasks, such as aligning diverse datasets, improving the capabilities of search engines, supporting error checking, and generating explanations using inference engines. However, populating, augmenting, and/or validating a knowledge graph can be challenging because developers need domain knowledge to understand their data and experience in ontology modeling to align concepts properly as well as experience with conflict detection and truth maintenance tools. Previous efforts have explored automatically integrating simple tabular data into knowledge graphs to lower the barrier to entry. These methods heavily rely on named entity overlap and require that tables are similar to relational tables in third normal form. While these methods have been successful under competition, these limitations make them impractical for general usage. In this paper, we introduce the semantic data dictionary generator (SDD-Gen), an algorithm that aligns complex tabular data to ontological terms for knowledge graph generation. Our methodology leverages context information from data dictionaries to make alignments, enabling us to align complex tables with few named entities and multiple subject columns.National Institutes of Health's Human Health Exposure Analysis Resource (HHEAR) project (NIH U2CES026555)
Air Force Research Laboratory's (AFRL) Civilian Academic Degree Program (CADP
Bergeron: Combating adversarial attacks through a conscience-based alignment framework
No full textResearch into AI alignment has grown considerably since the recent introduction of increasingly capable Large Language Models (LLMs). Unfortunately, modern methods of alignment still fail to fully prevent harmful responses when models are deliberately attacked. Such vulnerabilities can lead to LLMs being manipulated into generating hazardous content: from instructions for creating dangerous materials to inciting violence or endorsing unethical behaviors. To help mitigate this issue, we introduce Bergeron: a framework designed to improve the robustness of LLMs against attacks without any additional parameter fine-tuning. Bergeron is organized into two tiers; with a secondary LLM acting as a guardian to the primary LLM. This framework better safeguards the primary model against incoming attacks while monitoring its output for any harmful content. Empirical analysis reviews that by using Bergeron to complement models with existing alignment training, we can significantly improve the robustness and safety of multiple, commonly used commercial and open-source LLMs. Specifically, we found that models integrated with Bergeron are, on average, nearly seven times more resistant to attacks compared to models without such support
Scalable cost-efficient techniques for machine learning via sketching
No full textAugust 2024School of ScienceThe ever-increasing size and complexity of modern datasets present significant challenges for machine learning theorists and experimentalists, as the computational resources and time required to process and analyze these datasets are immense. Randomized Linear Algebra (randNLA) techniques offer a promising solution, enabling a quantifiable trade-off between accuracy and computation time. Sketching and sampling methods, central to modern algorithm design, serve as powerful techniques for compressing high-dimensional datasets into lower-dimensional representations while preserving properties of interest. This dissertation aims to exploit and leverage the advantages of randNLA techniques, with a primary focus on sketching methods, in common machine learning models to address the challenges posed by large-scale, high-dimensional datasets. This thesis investigates the following areas: (1) designing sparse graph-based sketching methods for fast numerical linear algebra computations; (2) obtaining low-rank approximations for matrix completion using sketching techniques, including the proposal of a novel two-modality sampling NoisyCUR algorithm; (3) developing convergent and efficient tensor decomposition algorithms through adaptive sketching with fast yet reliable sketching dimension computation; and (4) proposing a unified convex optimization framework for obtaining an optimal sampling probability distribution in reduced label complexity settings for Kernel Ridge Regression (KRR) models.Ph
Intelligent information provision for emerging transportation systems
No full textAugust 2024School of EngineeringThe rapid development of information transmission, vehicular technology, artificial intelligence, machine learning technologies, and computing resources is revolutionizing the transportation sector, making it more intelligent, sustainable, and equitable. These emerging technologies necessitate a paradigm shift in transportation system analysis. Leveraging these technologies is crucial to addressing traditional challenges, requiring customization guided by domain knowledge. This dissertation uniquely focuses on the transformative role of intelligent information provision in emerging transportation systems through both theoretical and practical perspectives. The first part of the dissertation focuses on the within-day dynamic traffic equilibrium with strategic information provision. The theoretical modeling and analysis pave the way for understanding the impacts of advanced and heterogeneous information, information reliability, and information design in emerging transportation systems. The second part of the dissertation focuses on mitigating the conflict between individual goals and system goals by leveraging reinforcement learning and correlated equilibrium. The dissertation proposes and customizes a two-way reinforcement learning framework for intelligent information provision in connected autonomous vehicle systems. This framework not only exemplifies the application of emerging technologies in information design but also provides innovative solutions to the longstanding challenge of achieving both individual- and system-level goals in congestion mitigation. Additionally, the research explores the theoretical foundation behind the effects of the proposed two-way learning framework, i.e., correlated equilibrium, offering a new angle and solution in traffic assignment. The dissertation contributes to advancing a more intelligent transportation system through information provision by establishing both theoretical foundations and groundbreaking applications with emerging technologies.Ph
Trend analysis of leap 2020 experimental results and validation of numerical predictions
No full textMay 2024School of EngineeringThe current landscape of geotechnical engineering benefits from abundant computational power and advanced numerical modeling platforms capable of analyzing the dynamic response and liquefaction of saturated granular soil systems. In this regard, an important step involving the validation of soil constitutive models and numerical platforms is need to foster a greater acceptance and utilization of numerical modeling in geotechnical engineering design and analysis. Recent efforts under the Liquefaction Experiments and Analysis Projects (LEAP) are aimed at promoting the validation of numerical models developed for soil liquefaction. LEAP are a series of international research collaborations aiming to produce high quality experimental data to validate soil constitutive models and modeling techniques for dynamic geotechnical systems involving soil liquefaction. The focus of this thesis is to analyze the LEAP 2020 experimental data and develop a validation methodology of numerical simulations of dynamic geotechnical systems involving soil liquefaction using centrifuge experimental data. The thesis is a collection of journal articles from chapter 2 to chapter 5. The abstracts of each chapter are presented below: Chapter 2: Miniature cone penetration tests (CPTs) were conducted during the LEAP 2017 and LEAP 2020 centrifuge experiments to characterize and quantify the soil deposit properties. All experiments were performed using Ottawa F-65 Sand and used the same CPT tool model. The CPTs showed consistent results below a critical depth, with tip resistance affected by overburden pressure and soil density. The (experiment) container widths were found to affect the CPT results when the used container is narrow. The cone (tip) penetration resistance was normalized to account for overburden pressure and corrected to address the potential effects of container width. An empirical relation was developed to estimate soil relative density from the corrected and normalized tip resistance. The obtained relation provides an estimate of relative density profile and associated standard deviation (as a measure of uncertainty) from measured CPT tip resistance of Ottawa sand deposits. Chapter 3: This chapter and compares the results of the LEAP 2020 centrifuge experiments. The centrifuge model consisted of a saturated Ottawa sand deposit supported by a cantilever sheet-pile wall that is practically rigid. Experiments were conducted at eight facilities with deposits having a relative density varying from about 50 \% to 80 \%. The models were subjected to an input base acceleration with a reference consisting of a ramped sinusoidal motion with five cycles at peak value. The achieved input accelerations varied from a (practically) monoharmonic 1 Hz motions to motions with significant 3 Hz and 5 Hz components. Peak accelerations ranged from 0.09 g to 0.27 g. The models were monitored in terms of deposit accelerations, pore pressures, surface lateral displacements and settlements, and retaining wall lateral displacements and rotations. The obtained experimental results cover a wide range of conditions and provide valuable information and insight into the response characteristics of the soil-retaining wall system. Chapter 4: The LEAP-2020 comprised a series of 23 centrifuge experiments of a saturated Ottawa sand (backfill) deposit supported by a cantilever wall. The experiments covered a comprehensive range of experimental conditions. The deposits had a relative density that varied from 49\% to 76\% and the centrifuge models were subjected to input motions that varied in amplitude and frequency content. The experimental results were consequently marked by a broad spectrum of response characteristics. A Gaussian Process Regression (GPR) was employed to investigate and assess the trends and response mechanisms of permanent lateral displacements, settlements, wall rotation and pore pressure buildups of the tested soil-retaining wall systems. The conducted analyses showed a high level of consistency among the experimental results and shed light on a number of salient response characteristics. Chapter 5: Recent efforts under The Liquefaction Experiments and Analysis Projects (LEAP) are aimed at promoting the validation of numerical models developed for soil liquefaction. LEAP are a series of international research collaborations aiming to produce high quality experimental data to validate soil constitutive models and modeling techniques for dynamic geotechnical systems involving soil liquefaction. This chapter presents a validation methodology of numerical simulations of the dynamic response and liquefaction of a soil-retaining wall systems using centrifuge experimental. The methodology includes the following steps: (1) A regression analysis is implemented to assess response characteristics and patterns in the experimental data, (2) A surrogate model of the computational model is developed based on a limited number of numerical simulations and (3) A probabilistic validation metric is implemented to assess the computational model performance. An implementation of the methodology is presented as an example for numerical simulations of a finite element model using two different soil constitutive model calibrations. The validation quantified the performance of the finite element models with respect to the centrifuge experimental data.Ph