1,721,067 research outputs found
A practical perspective on the concordance index for the evaluation and selection of prognostic time-to-event models
Full text linkDeveloping a prognostic model for biomedical applications typically requires mapping an individual's set of covariates to a measure of the risk that he or she may experience the event to be predicted. Many scenarios, however, especially those involving adverse pathological outcomes, are better described by explicitly accounting for the timing of these events, as well as their probability. As a result, in these cases, traditional classification or ranking metrics may be inadequate to inform model evaluation or selection. To address this limitation, it is common practice to reframe the problem in the context of survival analysis, and resort, instead, to the concordance index (C-index), which summarises how well a predicted risk score describes an observed sequence of events. A practically meaningful interpretation of the C-index, however, may present several difficulties and pitfalls. Specifically, we identify two main issues: i) the C-index remains implicitly, and subtly, dependent on time, and ii) its relationship with the number of subjects whose risk was incorrectly predicted is not straightforward. Failure to consider these two aspects may introduce undesirable and unwanted biases in the evaluation process, and even result in the selection of a suboptimal model. Hence, here, we discuss ways to obtain a meaningful interpretation in spite of these difficulties. Aiming to assist experimenters regardless of their familiarity with the C-index, we start from an introductory-level presentation of its most popular estimator, highlighting the latter's temporal dependency, and suggesting how it might be correctly used to inform model selection. We also address the nonlinearity of the C-index with respect to the number of correct risk predictions, elaborating a simplified framework that may enable an easier interpretation and quantification of C-index improvements or deteriorations
Predicting the Onset of Chronic Obstructive Pulmonary Disease in the English Longitudinal Study of Ageing
No full textChronic obstructive pulmonary disease (COPD) is a chronic lung disease estimated to be responsible of about 5% of all deaths worldwide. The identification of subjects at risk of developing COPD is important to reduce its global burden, as early interventions on modifiable risk factors (e.g. smoking) can delay or even prevent the decline of lung function. A few models to predict risk of COPD onset in the general population were developed, which included a small set of risk factors. The aim of this work is to develop a new predictive model of COPD onset, testing the predictive ability of a variety of variables, including socio-economic and lifestyle factors, wellbeing status, respiratory symptoms, medical history, lung function measurements and blood test biomarkers. The model was developed by applying logistic regression to a training set (n=2897) extracted from the English Longitudinal Study of Ageing. Most important variables for COPD prediction were selected by least absolute shrinkage and selection operator regularization. The analysis showed that variables not considered by the literature models, such as physical activity, depression, marital status, self-reported health, fibrinogen, C-reactive protein and cholesterol can be important predictors of COPD onset. The derived model presented good discrimination and calibration performance on an independent test set (n=724), with area under the receiver-operating characteristic curve equal to 0.81 and expected-to-observed event ratio equal to 0.93. Future works include an external validation of the model, the use of different modelling techniques (e.g. survival models) and the application of variable ranking methods
A Variable Ranking Method for Machine Learning Models with Correlated Features: In-Silico Validation and Application for Diabetes Prediction
Full text linkWhen building a predictive model for predicting a clinical outcome using machine learning techniques, the model developers are often interested in ranking the features according to their predictive ability. A commonly used approach to obtain a robust variable ranking is to apply recursive feature elimination (RFE) on multiple resamplings of the training set and then to aggregate the ranking results using the Borda count method. However, the presence of highly correlated features in the training set can deteriorate the ranking performance. In this work, we propose a variant of the method based on RFE and Borda count that takes into account the correlation between variables during the ranking procedure in order to improve the ranking performance in the presence of highly correlated features. The proposed algorithm is tested on simulated datasets in which the true variable importance is known and compared to the standard RFE-Borda count method. According to the root mean square error between the estimated rank and the true (i.e., simulated) feature importance, the proposed algorithm overcomes the standard RFE-Borda count method. Finally, the proposed algorithm is applied to a case study related to the development of a predictive model of type 2 diabetes onset
A dynamic probabilistic model of the onset and interaction of cardio-metabolic comorbidities on an ageing adult population
Full text linkComorbidity is widespread in the ageing population, implying multiple and complex medical needs for individuals and a public health burden. Determining risk factors and predicting comorbidity development can help identify at-risk subjects and design prevention strategies. Using socio-demographic and clinical data from approximately 11,000 subjects monitored over 11 years in the English Longitudinal Study of Ageing, we develop a dynamic Bayesian network (DBN) to model the onset and interaction of three cardio-metabolic comorbidities, namely type 2 diabetes (T2D), hypertension, and heart problems. The DBN allows us to identify risk factors for developing each morbidity, simulate ageing progression over time, and stratify the population based on the risk of outcome occurrence. By applying hierarchical agglomerative clustering to the simulated, dynamic risk of experiencing morbidities, we identified patients with similar risk patterns and the variables contributing to their discrimination. The network reveals a direct joint effect of biomarkers and lifestyle on outcomes over time, such as the impact of fasting glucose, HbA1c, and BMI on T2D development. Mediated cross-relationships between comorbidities also emerge, showcasing the interconnected nature of these health issues. The model presents good calibration and discrimination ability, particularly in predicting the onset of T2D (iAUC-ROC = 0.828, iAUC-PR = 0.294) and survival (iAUC-ROC = 0.827, iAUC-PR = 0.311). Stratification analysis unveils two distinct clusters for all comorbidities, effectively discriminated by variables like HbA1c for T2D and age at baseline for heart problems. The developed DBN constitutes an effective, highly-explainable predictive risk tool for simulating and stratifying the dynamic risk of developing cardio-metabolic comorbidities. Its use could help identify the effects of risk factors and develop health policies that prevent the occurrence of comorbidities
Better cardiovascular outcomes of type 2 diabetic patients treated with GLP-1 receptor agonists versus DPP-4 inhibitors in clinical practice
Full text linkBackground: Cardiovascular outcome trials in high-risk patients showed that some GLP-1 receptor agonists (GLP-1RA), but not dipeptidyl-peptidase-4 inhibitors (DPP-4i), can prevent cardiovascular events in type 2 diabetes (T2D). Since no trial has directly compared these two classes of drugs, we performed a comparative outcome analysis using real-world data. Methods: From a database of ~ 5 million people from North-East Italy, we retrospectively identified initiators of GLP-1RA or DPP-4i from 2011 to 2018. We obtained two balanced cohorts by 1:1 propensity score matching. The primary outcome was the 3-point major adverse cardiovascular events (3P-MACE; a composite of death, myocardial infarction, or stroke). 3P-MACE components and hospitalization for heart failure were secondary outcomes. Results: From 330,193 individuals with T2D, we extracted two matched cohorts of 2807 GLP-1RA and 2807 DPP-4i initiators, followed for a median of 18 months. On average, patients were 63 years old, 60% male; 15% had pre-existing cardiovascular disease. The rate of 3P-MACE was lower in patients treated with GLP-1RA compared to DPP4i (23.5 vs. 34.9 events per 1000 person-years; HR: 0.67; 95% C.I. 0.53-0.86; p = 0.002). Rates of myocardial infarction (HR 0.67; 95% C.I. 0.50-0.91; p = 0.011) and all-cause death (HR 0.58; 95% C.I. 0.35-0.96; p = 0.034) were lower among GLP-1RA initiators. The as-treated and intention-to-treat approaches yielded similar results. Conclusions: Patients initiating a GLP-1RA in clinical practice had better cardiovascular outcomes than similar patients who initiated a DPP-4i. These data strongly confirm findings from cardiovascular outcome trials in a lower risk population
Validity of Feature Importance in Low-Performing Machine Learning for Tabular Biomedical Data
Full text linkIn tabular data analysis, high model accuracy is often regarded as a prerequisite for discussing feature importance. This assumption stems from the expectation that the validity of feature importance correlates with model performance. In this work, we challenge this prevailing belief by demonstrating that even low-performing models can provide reliable feature importance on biomedical datasets. We conduct experiments to observe how feature importance rankings change as model performance progressively degrades. Using three synthetic datasets and four real-world biomedical datasets, we compare feature rankings from the full datasets to those obtained after reducing either the number of samples (samples removal) or the number of features (features removal), using different feature stability indices. Our results reveal that, in both synthetic and real datasets, feature rankings remain stable during performance degradation caused by features removal. In contrast, sample removal introduces greater discrepancies in feature importance rankings as performance deteriorates more severely. By analyzing the distribution of feature importance values and theoretically examining the probability that the model fails to distinguish importance between features, we show that models can still reliably identify feature importance despite performance degradation due to features removal. We conclude that the validity of feature importance can be preserved even at suboptimal model performance levels, as long as the degradation stems from insufficient features rather than insufficient samples. This has a considerable impact on biomedical research, where feature importance analysis plays a pivotal role in clinical decision support and translational bioinformatics
Automated control of bioreactors: An hardware-in-the-loop proof of concept test towards an experimental facility
No full textEnsuring controlled environmental conditions during bacterial growth in a bioreactor is often crucial for the success of an experiment or to grant efficiency in an industrial procedure. For instance, in some experiments, temperature, oxygen and pH levels should be kept constant to achieve a good yield. Commercial bioreactors commonly provide proprietary automation systems for the regulation of some of the above mentioned variables, but the possibility to personalize the control algorithm, to jointly control multiple variables or to modify the control task, is usually rather limited. To surpass this limitation we aim to build an Arduino-based bioreactor, controlled by a PC and capable of running customized, possibly advanced, control algorithms. As a preliminary step toward this goal, in this contribution we present the design of a simple Arduino-based reactor for the control of the concentration of a substance (coffee) in a tank, by automatically pumping a concentrated solution of the same substance or pure water. The optimal command of the pumps is computed with a nonlinear model predictive control (MPC) by using a model of the mass balance occurring within the reactor. The controller is implemented in Simulink. The system is tested via hardware-in-the-loop (HIL) simulation. The controller proved to be able to drive the simulated concentration of caffeine to the desired value, and it was verified that the pumps were correctly commanded. Similarly, sensor readings were also successfully transferred to the Simulink scheme. Future experiments with the physical system will complete the validation and demonstrate the possibility of creating controller prototypes in Matlab for this toy problem
Using Wearable and Environmental Data to Improve the Prediction of Amyotrophic Lateral Sclerosis and Multiple Sclerosis Progression: an Explorative Study
Full text linkAmyotrophic Lateral Sclerosis (ALS) and Multiple Sclerosis (MS) are chronic diseases with a severe impact on patients' lives. Both diseases create significant psychological and economic burdens due to alternating acute phases requiring hospital and home care. One possible solution could be the employment of sensor data to develop predictive models that can assist clinicians in making treatment and therapeutic decisions. In the context of the iDPP@CLEF 2024 challenge, this work aims to develop and compare different machine-learning approaches for predicting the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) scores in ALS patients, and relapses in MS patients, using wearable and environmental data, respectively. Specifically, the analysis focuses on the impact of these data and seeks to determine whether their incorporation enhances predictive performance. The results showed that there is indeed an improvement in the models' performance when sensor data are considered, in both the disease. In particular, in the case of ALS the Root Mean Square Error (RMSE) range, over the predicted twelve ALSFRS-R score, improved from [0.463-0.733] to [0.286-0.582] when incorporating the wearable data, as well as in the case of MS, where the inclusion of environmental data has improved the prediction of relapse, with the RMSE decreasing from 72.992 to 69.564
- …
