70 research outputs found
An In-Depth Perspective on the Classical Model
No full textThe Classical Model (CM) or Cooke’s method for performing Structured Expert Judgement (SEJ) is the best-known method that promotes expert performance evaluation when aggregating experts’ assessments of uncertain quantities. Assessing experts’ performance in quantifying uncertainty involves two scores in CM, the calibration score (or statistical accuracy) and the information score. The two scores combine into overall scores, which, in turn, yield weights for a performance-based aggregation of experts’ opinions. The method is fairly demanding, and therefore carrying out a SEJ elicitation with CM requires careful consideration. This chapter aims to address the methodological and practical aspects of CM into a comprehensive overview of the CM elicitation process. It complements the chapter “Elicitation in the Classical Model” in the book Elicitation (Quigley et al. 2018). Nonetheless, we regard this chapter as a stand-alone material, hence some concepts and definitions will be repeated, for the sake of completeness.Applied Probabilit
Modelling risk in high hazard operations: Integrating technical, organisational and cultural factors
No full textRecent disasters in high hazard industries such as Oil and Gas Exploration (The Deepwater Horizon) and Petrochemical production (Texas City) have been found to have causes that range from direct technical failures through organizational shortcomings right up to weak regulation and inappropriate company cultures. Risk models have generally concentrated upon technical failures, which are easier to construct and for which there is more concrete data. The primary causes, as identified by the US Chemical Safety Board for Texas City and the Presidential Commission for the Deepwater Horizon, lie firmly rooted in the culture of the organization and determine the way in which individuals go about risky activities. Modelling human activities, especially collectively rather than individual human errors as is done in most human models, is a quite different proposition, in which complex interactions between different individuals and levels change over time as success and failure alter the pattern of payoffs. This paper examines the development of an integrated model for risk in a real-time environment for the hydrocarbon industry. It is based originally on the CATS model for commercial aviation safety, that first attempted to address some of these problems in a relatively simple way. Aviation is, however, a relatively simple activity, with large numbers of common components in a constrained environment. The Oil and Gas industry is significantly more diverse, covering the gamut from exploration, drilling, production, transport, refining and chemical production, each with its own potential for large scale disaster, but in the case of an integrated oil company all run by individuals within a common company culture. Other papers will cover the details of specific issues; this paper covers the integration of the model as a whole.Values and TechnologyTechnology, Policy and Managemen
Levee system reliability modeling: The length effect and Bayesian updating
No full textIn levee system reliability, the length effect is the term given to the phenomenon that the longer the levee, the higher the probability that it will have a weak spot and fail. Quantitatively, it is the ratio of the segment failure probability to the cross-sectional failure probability. The literature is lacking in methods to calculate the length effect in levees, and often over-simplified methods are used. An efficient (but approximate) method, which we refer to as the modified outcrossing (MO) method, was developed for the system reliability model used in Dutch national flood risk analysis and for the provision of levee assessment tools, but it is poorly documented and its accuracy has not been tested. In this paper, we propose a method to calculate the length effect in levees by sampling the joint spatial distribution of the resistance variables using a copula approach, and represented by a Bayesian Network (BN). We use the BN to verify the MO method, which is also described in detail in this paper. We describe how both methods can be used to update failure probabilities of (long) levees using survival observations (i.e., high water levels and no levee failure), which is important because we have such observations in abundance. We compared the methods via a numerical example, and found that the agreement between the segment failure probability estimates was nearly perfect in the prior case, and very good in the posterior case, for segments ranging from 500 m to 6000 m in length. These results provide a strong verification of both methods, either of which provide an attractive alternative to the more simplified approaches often encountered in the literature and in practice.Hydraulic Structures and Flood RiskDynamics of Structure
Recent Advances in the Elicitation of Uncertainty Distributions from Experts for Multinomial Probabilities
No full textIn this chapter, we consider the problem of the elicitation and specification of an uncertainty distribution based on expert judgements, which may be a subjective prior distribution in a Bayesian analysis, for a set of probabilities which are constrained to sum to one. A typical context for this is as a prior distribution for the probabilities in a multinomial model. The Dirichlet distribution has long been advocated as a natural way to represent the uncertainty distribution over the probabilities in this context. The relatively small number of parameters allows for specification based on relatively few elicited quantities but at the expense of a very restrictive structure. We detail recent advances in elicitation for the Dirichlet distribution and recently proposed alternative approaches, which offer greater flexibility at the expense of added complexity. In order of increasing flexibility, they are the generalised Dirichlet distribution, multivariate copulas and vines. An extension of multinomial models containing covariates is discussed
Structured expert judgement for decisions on medicines policy and management
No full textMany decisions related to the marketing authorisation of medicinal products as well as decisions for processes such as Health Technology Assessment (HTA), reimbursement and pricing of medicines, and the setting of clinical guidelines, are taken in the face of significant uncertainties. Moreover, decision making can be impacted by biases resulting from psychological heuristics. In other domains where decisions have to be taken with imperfect or incomplete evidence, Structured Expert Judgement (SEJ) has been found to be useful in making the best use of available evidence, and synthesising it with professional expertise, stakeholders’ values and concerns. To date, formal SEJ has only been used to a limited extent in healthcare. Aspects affecting decisions for marketing authorisation and health technology assessment, reimbursement and pricing of medicines are described and the main risks and uncertainties are identified. Some considerations and recommendations for the use of SEJ to strengthen these decisions are made
Characteristics of a process for subjective probability elicitation
No full textThe elicitation of subjective probabilities from experts can be critical in determining a course of action when making decisions under uncertainty. A sound process to elicit probabilistic judgement is necessary to ensure that good quality data are used to inform the decision-making, as well as to provide protection to those accountable for the consequences of the determined actions. We synthesise the characteristics of a good elicitation process by critically reviewing those advocated and applied. We compare the processes inherent in the guidance produced by two professional bodies to exemplify the manner in which the characteristics manifest themselves in practice. We examine whether standardisation is meaningful given the maturity of processes for the elicitation of subjective probability
Dealing with imperfect elicitation results
No full textThe trial-and-roulette method is a popular method to extract experts’ beliefs about a statistical parameter. However, most studies examining the validity of this method only use ‘perfect’ elicitation results. In practice, it is sometimes hard to obtain such neat elicitation results. In our project about predicting fraud and questionable research practices among Ph.D. candidates, we ran into issues with imperfect elicitation results. The goal of the current chapter is to provide an overview of the solutions we used for dealing with these imperfect results, so that others can benefit from our experience. We present information about the nature of our project, the reasons for the imperfect results and how we resolved these supported by annotated R-syntax
Non parametric Bayesian belief nets (NPBBNs) versus ensemble Kalman filter (EnKF) in reservoir simulation
No full textThe thesis is concerned with applying a method that uses graphical models to solve a reservoir simulation problem. The focus is on the graphical models called Bayesian belief nets. A Bayesian belief net based approach is compared with an already popular technique used in reservoir simulation, the ensemble Kalman filter.Risk and environmental modellingApplied mathematicsElectrical Engineering, Mathematics and Computer Scienc
Uncertainty Quantification with Experts: Present Status and Research Needs
No full textExpert elicitation is deployed when data are absent or uninformative and critical decisions must be made. In designing an expert elicitation, most practitioners seek to achieve best practice while balancing practical constraints. The choices made influence the required time and effort investment, the quality of the elicited data, experts’ engagement, the defensibility of results, and the acceptability of resulting decisions. This piece outlines some of the common choices practitioners encounter when designing and conducting an elicitation. We discuss the evidence supporting these decisions and identify research gaps. This will hopefully allow practitioners to better navigate the literature, and will inspire the expert judgment research community to conduct well powered, replicable experiments that properly address the research gaps identified.</p
Shrinking the Variance in Experts’ “Classical” Weights Used in Expert Judgment Aggregation
No full textMathematical aggregation of probabilistic expert judgments often involves weighted linear combinations of experts’ elicited probability distributions of uncertain quantities. Experts’ weights are commonly derived from calibration experiments based on the experts’ performance scores, where performance is evaluated in terms of the calibration and the informativeness of the elicited distributions. This is referred to as Cooke’s method, or the classical model (CM), for aggregating probabilistic expert judgments. The performance scores are derived from experiments, so they are uncertain and, therefore, can be represented by random variables. As a consequence, the experts’ weights are also random variables. We focus on addressing the underlying uncertainty when calculating experts’ weights to be used in a mathematical aggregation of expert elicited distributions. This paper investigates the potential of applying an empirical Bayes development of the James–Stein shrinkage estimation technique on the CM’s weights to derive shrinkage weights with reduced mean squared errors. We analyze 51 professional CM expert elicitation studies. We investigate the differences between the classical and the (new) shrinkage CM weights and the benefits of using the new weights. In theory, the outcome of a probabilistic model using the shrinkage weights should be better than that obtained when using the classical weights because shrinkage estimation techniques reduce the mean squared errors of estimators in general. In particular, the empirical Bayes shrinkage method used here reduces the assigned weights for those experts with larger variances in the corresponding sampling distributions of weights in the experiment. We measure improvement of the aggregated judgments in a cross-validation setting using two studies that can afford such an approach. Contrary to expectations, the results are inconclusive. However, in practice, we can use the proposed shrinkage weights to increase the reliability of derived weights when only small-sized experiments are available. We demonstrate the latter on 49 post-2006 professional CM expert elicitation studies.Applied Probabilit
- …
