198 research outputs found
Structure Variability in Bayesian Networks.
No full textThe structure of a Bayesian network encodes most of the information about the probability distribution of the data, which is uniquely identified given some general distributional assumptions. Therefore it’s important to study the variability of its network structure, which can be used to compare the performance of different learning algorithms and to measure the strength of any arbitrary subset of arcs.
In this paper we will introduce some descriptive statistics and the corresponding parametric and Monte Carlo tests on the undirected graph underlying the structure of a Bayesian network, modeled as a multivariate Bernoulli random variable
Measures of Variability for Graphical Models
No full textIn recent years, graphical models have been successfully applied in several different
disciplines, including medicine, biology and epidemiology. This has been made possible
by the rapid evolution of structure learning algorithms, from constraint-based ones to
score-based and hybrid ones. The main goal in the development of these algorithms
has been the reduction of the number of either independence tests or score comparisons
needed to learn the structure of the Bayesian network. In most cases the characteristics
of the learned networks have been studied using a small number of reference data sets
as benchmarks, and differences from the true structure heve been measured with purely
descriptive measures such as Hamming distance.
This approach to model validation is not possible for real world data sets, as the true
structure of their probability distribution is not known. An alternative is provided
by the use of either parametric or nonparametric bootstrap. By applying a learning
algorithm to a sufficiently large number of bootstrap samples it is possible to obtain the
empirical probability of any feature of the resulting network, such as the structure of
the Markov Blanket of a particular node. The fundamental limit in the interpretation
of the results is that the “reasonable” level of confidence for thresholding depends on
the data and the learning algorithm.
In this thesis we extend the aforementioned bootstrap-based approach for the in-
ference on the structure of a Bayesian or Markov network. The graph representing
the network structure and its underlying undirected graph (in the case of Bayesian
networks) are modelled using a multivariate extension of the Trinomial and Bernoulli
distributions; each component is associated with an arc. These assumptions allow the
derivation of exact and asymptotic measures of the variability of the network structure
or any of its parts. These measures are then applied to some common learning strate-
gies used in literature using the implementation provided by the bnlearn R package
implemented and maintained by the author.Negli ultimi anni i modelli grafici, ed in particolare i network Bayesiani, sono entrati
nella pratica corrente delle analisi statistiche in diversi settori scientifici, tra cui medi
cina e biostatistica. L’uso di questo tipo di modelli è stato reso possibile dalla rapida
evoluzione degli algoritmi per apprenderne la struttura, sia quelli basati su test statistici
che quelli basati su funzioni punteggio. L’obiettivo principale di questi nuovi algoritmi
è la riduzione del numero di modelli intermedi considerati nell’apprendimento; le loro
caratteristiche sono state usualmente valutate usando dei dati di riferimento (per i quali
la vera struttura del modello è nota da letteratura) e la distanza di Hamming.
Questo approccio tuttavia non può essere usato per dati sperimentali, poiché la loro
struttura probabilistica non è nota a priori. In questo caso una valida alternativa
è costituita dal bootstrap non parametrico: apprendendo un numero sufficientemente
grande di modelli da campioni bootstrap è infatti possibile ottenere una stima empirica
della probabilità di ogni caratteristica di interesse del network stesso.
In questa tesi viene affrontato il principale limite di questo secondo approccio: la
difficoltà di stabilire una soglia di significatività per le probabilità empiriche. Una
possibile soluzione è data dall’assunzione di una distribuzione Trinomiale multivariata
(nel caso di grafi orientati aciclici) o Bernoulliana multivariata (nel caso di grafi non
orientati), che permette di associare ogni arco del network ad una distribuzione mar
ginale. Questa assunzione permette di costruire dei test statistici, sia asintotici che
esatti, per la variabilità multivariata della struttura del network nel suo complesso o di
una sua parte. Tali misure di variabilità sono state poi applicate ad alcuni algoritmi di
apprendimento della struttura di network Bayesiani utilizzando il pacchetto R bnlearn,
implementato e mantenuto dall’autore
Achieving Fairness with a Simple Ridge Penalty
No full textIn this paper we present a general framework for estimating regression models
subject to a user-defined level of fairness. We enforce fairness as a model
selection step in which we choose the value of a ridge penalty to control the
effect of sensitive attributes. We then estimate the parameters of the model
conditional on the chosen penalty value. Our proposal is mathematically simple,
with a solution that is partly in closed form, and produces estimates of the
regression coefficients that are intuitive to interpret as a function of the
level of fairness. Furthermore, it is easily extended to generalised linear
models, kernelised regression models and other penalties; and it can
accommodate multiple definitions of fairness.
We compare our approach with the regression model from Komiyama et al.
(2018), which implements a provably-optimal linear regression model; and with
the fair models from Zafar et al. (2019). We evaluate these approaches
empirically on six different data sets, and we find that our proposal provides
better goodness of fit and better predictive accuracy for the same level of
fairness. In addition, we highlight a source of bias in the original
experimental evaluation in Komiyama et al. (2018).Comment: 16 pages, 9 figure
Information theory, machine learning, and Bayesian networks in the analysis of dichotomous and Likert responses for questionnaire psychometric validation
No full textQuestionnaire validation is indispensable in psychology and medicine and is essential for understanding differences across diverse populations in the measured construct. While traditional latent factor models have long dominated psychometric validation, recent advancements have introduced alternative methodologies, such as the "network framework." This study presents a pioneering approach integrating information theory, machine learning (ML), and Bayesian networks (BNs) into questionnaire validation. Our proposed framework considers psychological constructs as complex, causally interacting systems, bridging theories, and empirical hypotheses. We emphasize the crucial link between questionnaire items and theoretical frameworks, validated through the known-groups method for effective differentiation of clinical and nonclinical groups. Information theory measures such as Jensen-Shannon divergence distance and ML for item selection enhance discriminative power while contextually reducing respondent burden. BNs are employed to uncover conditional dependences between items, illuminating the intricate systems underlying psychological constructs. Through this integrated framework encompassing item selection, theory formulation, and construct validation stages, we empirically validate our method on two simulated data sets-one with dichotomous and the other with Likert-scale data-and a real data set. Our approach demonstrates effectiveness in standard questionnaire research and validation practices, providing insights into criterion validity, content validity, and construct validity of the instrument
Hard and soft EM in Bayesian network learning from incomplete data
No full textIncomplete data are a common feature in many domains, from clinical trials to industrial applications. Bayesian networks (BNs) are often used in these domains because of their graphical and causal interpretations. BN parameter learning from incomplete data is usually implemented with the Expectation-Maximisation algorithm (EM), which computes the relevant sufficient statistics (“soft EM”) using belief propagation. Similarly, the Structural Expectation-Maximisation algorithm (Structural EM) learns the network structure of the BN from those sufficient statistics using algorithms designed for complete data. However, practical implementations of parameter and structure learning often impute missing data (“hard EM”) to compute sufficient statistics instead of using belief propagation, for both ease of implementation and computational speed. In this paper, we investigate the question: what is the impact of using imputation instead of belief propagation on the quality of the resulting BNs? From a simulation study using synthetic data and reference BNs, we find that it is possible to recommend one approach over the other in several scenarios based on the characteristics of the data. We then use this information to build a simple decision tree to guide practitioners in choosing the EM algorithm best suited to their problem
Dirichlet Bayesian network scores and the maximum entropy principle
No full textA classic approach for learning Bayesian networks from data is to select the maximum a posteriori (MAP) network. In the case of discrete Bayesian networks, the MAP network is selected by maximising one of several possible Bayesian Dirichlet (BD) scores; the most famous is the Bayesian Dirichlet equivalent uniform (BDeu) score from Heckerman et al. (1995). The key properties of BDeu arise from its underlying uniform prior, which makes structure learning computationally efficient; does not require the elicitation of prior knowledge from experts; and satisfies score equivalence. In this paper we will discuss the impact of this uniform prior on structure learning from an information theoretic perspective, showing how BDeu may violate the maximum entropy principle when applied to sparse data and how it may also be problematic from a Bayesian model selection perspective. On the other hand, the BDs score proposed in Scutari (2016) arises from a piecewise prior and it does not appear to violate the maximum entropy principle, even though it is asymptotically equivalent to BDeu
Learning Bayesian Networks with the bnlearn R Package
No full textbnlearn is an R package (R Development Core Team 2010) which includes several algorithms for learning the structure of Bayesian networks with either discrete or continuous variables. Both constraint-based and score-based algorithms are implemented, and can use the functionality provided by the snow package (Tierney et al. 2008) to improve their performance via parallel computing. Several network scores and conditional independence algorithms are available for both the learning algorithms and independent use. Advanced plotting options are provided by the Rgraphviz package (Gentry et al. 2010).
Dirichlet Bayesian network scores and the maximum relative entropy principle
No full textA classic approach for learning Bayesian networks from data is to identify a maximum a posteriori (MAP) network structure. In the case of discrete Bayesian networks, MAP networks are selected by maximising one of several possible Bayesian–Dirichlet (BD) scores; the most famous is the Bayesian–Dirichlet equivalent uniform (BDeu) score from Heckerman et al. (Mach Learn 20(3):197–243, 1995). The key properties of BDeu arise from its uniform prior over the parameters of each local distribution in the network, which makes structure learning computationally efficient; it does not require the elicitation of prior knowledge from experts; and it satisfies score equivalence. In this paper we will review the derivation and the properties of BD scores, and of BDeu in particular, and we will link them to the corresponding entropy estimates to study them from an information theoretic perspective. To this end, we will work in the context of the foundational work of Giffin and Caticha (Proceedings of the 27th international workshop on Bayesian inference and maximum entropy methods in science and engineering, pp 74–84, 2007), who showed that Bayesian inference can be framed as a particular case of the maximum relative entropy principle. We will use this connection to show that BDeu should not be used for structure learning from sparse data, since it violates the maximum relative entropy principle; and that it is also problematic from a more classic Bayesian model selection perspective, because it produces Bayes factors that are sensitive to the value of its only hyperparameter. Using a large simulation study, we found in our previous work [Scutari in J Mach Learn Res (Proc Track PGM 2016) 52:438–448, 2016] that the Bayesian–Dirichlet sparse (BDs) score seems to provide better accuracy in structure learning; in this paper we further show that BDs does not suffer from the issues above, and we recommend to use it for sparse data instead of BDeu. Finally, will show that these issues are in fact different aspects of the same problem and a consequence of the distributional assumptions of the prior
Learning Bayesian networks with heterogeneous agronomic data sets via mixed-effect models and hierarchical clustering
No full textMaize, a crucial crop globally cultivated across vast regions, especially in sub-Saharan Africa, Asia, and Latin America, occupies 197 million hectares as of 2021. Various statistical and machine learning models, including mixed-effect models, random coefficients models, random forests, and deep learning architectures, have been devised to predict maize yield. These models consider factors such as genotype, environment, genotype-environment interaction, and field management. However, the existing models often fall short of fully exploiting the complex network of causal relationships among these factors and the hierarchical structure inherent in agronomic data. This study introduces an innovative approach integrating random effects into Bayesian networks (BNs), leveraging their capacity to model causal and probabilistic relationships through directed acyclic graphs. Rooted in the linear mixed-effects models framework and tailored for hierarchical data, this novel approach demonstrates enhanced BN learning. Application to a real-world agronomic trial produces a model with improved interpretability, unveiling new causal connections. Notably, the proposed method significantly reduces the error rate in maize yield prediction from 28% to 17%. These results advocate for the preference of BNs in constructing practical decision support tools for hierarchical agronomic data, facilitating causal inference
- …
