Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Elusive Accountability: Evaluation in the Time of Pandemic
Full text linkBackground: When the covid 19 pandemic started spreading early 2020 Governments responded in various ways. The merits and drawbacks of national responses is not an academic concern, it was – and remains - a question of survival.
Purpose: The purpose of this paper is to analyze the evaluation of the Swedish national response to the pandemic and to assess whether the evaluation provided for accountability of the policy measures that were put in place.
Setting: The Swedish Government announced early on that its response was to be evaluated. A Parliamentary Committee was established and was given a comprehensive mandate to evaluate the process and the results of the response. The Committee was to start immediately in mid 2020, to deliver interim reports and a final synthesis report in February 2022.
Intervention: Not applicable.
Research design: We use a case study design based on a desk study of written documentation concerning the covid 19 evaluation. Our study starts with publications early 2020 and up to the final synthesis report of the evaluation and continues with events/debates through the general elections in September 2022 (when the Government responsible for the covid 19 response lost) and the months immediately afterwards.
Data collection and analysis: The key sources are the public mandate for the evaluation, its three evaluation reports, records of the debate in daily papers and professional journals, and the autobiographies of leading actors.
Findings: The political/administrative system initiated an evaluation that gave a timely, credible and comprehensive assessment of the virtues and mistakes of the Government’s response to the pandemic. Still, the question of accountability remains elusive. Structures that constrained the response were shaped long ago and the actors responsible cannot be held accountable today. Those that can be held to account made mistakes but also took brave, and in retrospect correct measures to reduce the impact of the pandemic. In addition, new information keeps changing the final judgement, for example the impact of business subsidies is better known today. In sum, the information needed to create accountability was – and is – largely available, but to establish accountability remains an elusive task
How It All started and How It Evolved: Ray’s Whip
Full text linkBackground: The chapter reconstructs the origin of the concept of “streams of evidence” in the collective work of the Inteval group, as proposed in the book From Studies to Streams (Rist, Stame, 2006). Contrary to insisting on the (scarce) utilization of single studies, answering the evaluation demands of static organizations, based on their “own incentives and reward structures”, Ray challenged the group to engage with the new scenario of the learning organizations “that embraced learning and knowledge as fundamental to their own existence and success”. Such organizations needed evaluative knowledge that was “timely, proactive and drawing on multiple sources of knowledge”, that could be provided by different sources, evaluations and research, internal or external to the organizations.
Purpose: The chapter aims to show the far-sightedness of Ray’s intuition. He was able to capture the spirit of time and to encompass different converging trends. His intellectual leadership oriented other people’s work, creating a wide space for innovation and adaptation. Moreover, in the new current conditions of pandemics and multi-crises, Ray has been able to revive and update that thrust, adding a multidisciplinary dimension to the idea of streams.
Setting: The concept of streams of evidence is a powerful and versatile metaphor. New instances of flowing stuff can be added to the original taxonomy, based on the source of the evidence and on its organization. It has entered many different fields as an autonomous cognitive tool. In evaluation methodology it has contributed to meta-analysis, realist syntheses, and a broad reinterpretation of mixed methods. In the study of evaluation systems, it has provided the basis for organized practices, from well-tempered forms of Knowledge Management to the elaboration of evaluation demands through Learning Agendas.
Intervention, N.A.
Research design, N.A.
Data collection and analysis and findings, N.A
Evaluation, Inteval, and Two National Audit Offices (U.S. GAO & Netherlands Court of Audit): Ray Rist’s Contributions and Leadership in the Early Years of INTEVAL
Full text linkBackground: This paper describes Ray Rist’s intellectual, research-focused work in the early years of Inteval (largely between the mid-1980s till around mid-1990s). It sets out what the position and roles of the (then) USA General Accounting Office (GAO) and the Netherlands Court of Audit (NCA) were with respect to evaluation.
Purpose: Differences and similarities between the GAO and NCA experience are described, with the GAO as a ‘first wave’ organization in the evaluation profession and the NCA as a ‘second wave’ organization, as it learned from the USA experiences. The labelling of first and second wave developments in evaluation was formulated by Derlien (1989). There is a particular focus on the role of leadership in shaping practice.
Setting: During the period of the mid-1980’s till mid-1990’s’, Rist and the author of this article were both working at respectively GAO and NCA in the field of policy evaluation/program evaluation and methodology. The setting was such that they worked together on a number of occasions and products, amongst others dealing with how (performance) audits and evaluation developed; the introduction (in the NCA) of government-wide (comparative) evaluations of tools of government like subsidies, information campaigns and inspections (covering all ministries), how evaluation was related to the learning capability of governments and how Inteval as a group of evaluators, auditors and other social scientists developed.
Intervention: Not applicable
Research design: Not applicable
Data collection and analysis: Not applicable
Findings: A number of lessons are set out about the role of leadership in evaluation
Transforming Evaluative Thinking: Some Ideas from Bateson
Full text linkBackground: Evaluators increasingly consider systems- and complexity-informed approaches to evaluation, especially when considering how evaluation might be transformed to evaluate current complex problems. Although Gregory Bateson was an early contributor to systems thinking, there is almost no reference to his work in the evaluation literature.
Purpose: To introduce some of Bateson’s core ideas and to pose initial questions intended to spark reflection and discussion, with the intent of contributing to further development of the concept of evaluative thinking.
Setting: Global.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Research on Evaluation Outside Journal Publications: An Analysis of Proposals Accepted for the 2019 American Evaluation Association Annual Conference
Full text linkMany evaluators may be conducting and presenting research on evaluation (RoE) without realizing it. Proposals accepted to present at the AEA 2019 conference were analyzed for whether or not they were RoE (i.e., systematic, empirical, and focused on evaluation). RoE proposals were then coded using a framework by Mark (2008) to determine the common trends across those identified as RoE. Non-RoE proposals were coded for why they were not RoE, including that they were not systematic, empirical, and/or focused on evaluation. A total of 15% of the 732 proposals analyzed were coded as RoE; most proposals examined evaluation activities using the descriptive mode of inquiry. Furthermore, most non-RoE proposals were coded as not RoE because they presented an evaluation rather than a research study on evaluation or because they were reflections on evaluation. This study presents multiple opportunities and future directions for RoE
What’s at Stake? Decolonization of Terminology in the United States
Full text linkEvery day in the United States, we use language that is oppressive and rooted in our colonial past. For this article, we specifically focus on the term stakeholder regarding evaluation research. Evaluators have defined this term as: “all of those individuals who have an interest (i.e., are somehow vested in) in the program that is to be evaluated." (Alkin & Vo, 2018, p. 51). However, little attention has been given to changing or reshaping this term to avoid perpetuating inequities to make program evaluations more culturally responsive to all parties involved. Historically, this terminology rooted in a colonial perspective is defined as “the person who drove a stake into the land to demarcate the land s/he was occupying/stealing from Indigenous territories” (Phipps, 2022). Even though the term is used frequently, many people may not understand the nature of the roots of this term and its involvement in the oppression of persons of color. Because of these points, it has been suggested that new terms be used in its stead (Sharfstein, 2016). We ask, after so many years of this country being independent of colonial rule, why are we still using terms that are oppressive and hold a connotation of being non-inclusive? This article aims to explore how terminology has been used in research and offers ideas to consider how to move forward with ensuring that the language used in program evaluation and other areas of research is culturally responsive to people and organizations affected by or show an interest in the findings of a given assessment
Visualizing Evaluation Theory: Current Reflections and Future Projections
Full text linkBackground: Among other potential benefits, evaluation theories are intended to serve as a guide to evaluation practice. Scholars have offered alternative visual representations of multiple evaluation theories, their contents and/or interrelationships.
Purpose: I offer brief commentary on the evaluation theory visualizations in other articles in this special section of the Journal of Multidisciplinary Evaluation (JMDE). I suggest evaluators should be familiar with multiple representations. I also speculate about a future generation of visual representations which might be interactive, allowing evaluators to select the level of detail most useful to them
An Iconic Evaluation Trailblazer is Gone but His Legacy Remains: A Tribute to Michael Scriven
Full text linkA tribute to Michael Scriven from a former student, Dr. Nadini Persaud
What Is This Thing Called Evaluation Theory?
Full text linkBackground: Evaluation theory has and continues to play an important role in the practice and profession of evaluation. Over the years, and as the range of evaluation theories has continued to grow, evaluation scholars and practitioners have developed various visualized classification frameworks of evaluation theory, invoking images such as trees, rivers, the periodic table, metro maps, concept maps, and most recently gardens.
Purpose: The purpose of this special issue is to explore different visualizations of evaluation theory and to discuss their benefits and implications for the practice and profession of evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
A Tree: Planted and Growing
Full text linkBackground: Evaluation theory classification systems have been developed since the field’s early stages of theory development. Many have been published with the goal of promoting further understanding of the ways in which theoretical ideas relate both philosophically and in practice. This intends to promote a better understanding of the development and evolution of one of the classification systems, the Evaluation Theory, developed by Alkin and Christie first in the late 1990s and is in its most recent version, published in 2023.
Purpose: This paper shares the primary purpose for developing the Evaluation Theory Tree, our analytic process for developing the categorization system presented as a tree, and how and why the tree has evolved over the years since its first publication