Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
The International Research Group of Policy and Program Evaluation (INTEVAL): A Celebration of Meetings, Publications, and Leadership
Full text linkBackground: A limited initiative by the International Institute of Administrative Studies in 1986 established a small Working Group on Policy Evaluation that has today became a self supporting 50-member international community of professional practice (INTEVAL) that has endured for almost four decades and published over 40 books.
Purpose: To chronicle the development of the Group and identify supportive factors.
Setting: n/a.
Intervention: n/a.
Research design: a history of the development of the Group and an organisational analysis of its leadership.
Data collection: historical records of the Group.
Analysis: Draws on psychological safety theory of organisational leadership.
Findings: The development of INTEVAL has been supported by a distinctive culture of psychological safety developed by the Group’s convenor Ray Rist, and enthusiastically adopted by members, in which Rist has set out the shared expectations and meanings of the Group’s work, and encouraged participation by demonstrating humility, practising inquiry, establishing collective ways of working, and expressing appreciation of successes and destigmatizing well intentioned misfires. Members have responded to this leadership with dedication to their contributions and to their colleagues
From Perpetuation to Disruption of Disadvantages: Learning from a Young Ray Rist and Implications for the Future of the Field of Evaluation
Full text linkBackground: Given the amplification of vulnerabilities created by the COVID-19 pandemic, the need for evaluation to more intentionally explore the intended and unintended consequences of interventions in contemporary society has increased. In this paper, we analyze one of Ray Rist's earliest papers, "Student Social Class and Teacher Expectations: The Self-Fulfilling Prophecy in Ghetto Education" to explore whether it has lessons for contemporary evaluators
Purpose: This paper seeks to learn lessons for evaluation (in identifying themes important to addressing inequities) from a seminal paper published more than 50 years ago. We explore lessons that research study conducted in a classroom in the USA might have for contemporary evaluators.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: This paper analyzes key themes from Rist's seminal paper and leverages our experience as evaluators working in multiple settings to assess the implication of his paper to contemporary evaluations. The analysis is informed by a realist lens that recognizes the importance of contexts and mechanisms in the generation of outcomes.
Findings: The following four implications of Rist's ideas are discussed: (1) the need to pay close attention to the architecture of interventions; (ii) explore how the implementation of some interventions can lead to exclusion of individuals; (iii) evaluation as a field needs to move from verdicts to explanations; (iv) evaluators need to pay attention to the dynamics of mutually reinforcing processes that exacerbate disadvantages over time
Carrots, Sticks, and Sermons – the Triad and the Book
Full text linkBackground: That evaluation results should come to use is a dogma in the evaluation community. One way of making evaluations useful (better criterion than use) is to construe their evaluands as cases of something more general. One possibility is to focus on the policy-instruments part of the evaluands, in particular their power content.
Purpose: This chapter is about Carrots Sticks and Sermons, a general framework and a book for evaluating classifying and labeling the policy-instruments part of interventions,
Setting: Not applicable
Intervention: Not applicable
Research design: Not applicable
Findings: Unexpectedly, the trifold framework and the book has been quite successful as far as sold copies and citations are concerned
Ray Rist: An Evaluator at the Government Accountability Office (GAO)
Full text linkBackground: Program evaluation and performance audit are distinct professional practices. Performance audit draws heavily on its financial accounting and auditing roots while evaluators have much more multidisciplinary backgrounds. Performance audit practice and development is associated with audit institutions such as National Audit Offices and internal audit while program evaluation is more associated with government departments and agencies in the public sector. The two practices are often working in the same group or organization but not necessarily together which could be mutually beneficial.
Purpose: The paper examines the early contribution that Ray Rist made to resolving the methodological challenge of performance auditors and program evaluators working together, early in his career at the GAO.
Setting: Independent National Audit Offices and government departments and agencies in the U.S.A. and Canada.
Intervention: n/a.
Research Design: The analysis of two early approaches to program evaluation to assess how they facilitated performance auditors and evaluators to work together.
Data Collection and Analysis: Data were collected through an analysis of documents, publications and discussions with colleagues and associates of Ray Rist.
Findings: Ray Rist worked in the group at the GAO that did evaluations and training of the multi-disciplinary staff in evaluation methodology and procedures. The approach, led by Ray, was a methodology that used multidisciplinary teams to answer specific questions. It encompassed a broader set of questions and that can more easily be used by performance auditor and program evaluators. It generally was less restricted in its application and use than other definitions of program evaluation that were based on an evaluation of a unit of analysis of a program or sub program. His early methodology transfer paper, written while Ray was at the GAO was in use for some time after he left, with numerous updates continuing with the same themes developed in the first iteration. These themes were foundational and carried into Ray’s future work, still having a strong resonance today
Moving from Studies to Streams: A More Radical Way to Avoid Floods of Evidence by Channeling Evaluative Efforts in Conditions of Complexity
Full text linkBackground: Multiple evaluations of complex problems risk creating ‘floods’ of evidence that are hard for decision makers to make sense of. Using the metaphor of ‘canalizing’ evidence to create more manageable streams of evidence, it is argued that the evaluation community can create more usable knowledge to support decision makers and strengthen accountability. ‘Downstream’ systematic reviews and clearing houses are not enough to address this issue. The problems arise ‘upstream’ from multiple, weakly connected evaluations being conducted in relative isolation, rather than focusing evaluative efforts on the key challenges facing complex policy spaces.
Purpose: It is argued that evaluation policies need to move ‘upstream’ and be consciously designed to both build on existing evaluations and take the needs of decision-makers more fully into account. This cannot be achieved by ‘one off’ improvements to individual evaluations but requires a coherent response that would support effective and meaningful retroductive realist syntheses that reveal the patterns found in evaluations of heterogeneous interventions.
Setting: Evaluations are almost never at the heart of policy debates today and we should at least be curious about why this is the case. It is argued that evaluative efforts are poorly used, unfocused and require ‘canalization’ to replace floods of evidence with coherent sense-making.
Intervention: not relevant
Research design: Review and reflection
Data collection: not relevant
Analysis and findings: Urgent action is required across the community of evaluators, commissioners of studies, and users of evaluative evidence to make better use of the resources dedicated to evaluation. Streams of evidence should be curated to promote learning and improvement, involve multiple stakeholders, in ways that require the evaluation community to take stock of our competencies and organisational forms. Whether this takes us towards a single answer, and identifies ‘best practice’, or whether it takes us towards heterogeneous insights, and identifies more ephemeral ‘good practices’, is for the evidence to decide. How this evidence is channelled will be critical
Conclusions on the Way Forward for Evaluation in Difficult Times
Full text linkBackground: The evaluation profession faces unprecedented challenges in an era of global uncertainty, with shifting political climates, rising populism, and complex societal problems requiring innovative evaluation approaches. Moments like these are opportune times to harvest learnings from the past to inform current and future evaluation practice.
Purpose: This concluding chapter reflects on Ray Rist's career and the broader implications of INTEVAL's work for the future of evaluation in uncertain times, highlighting key characteristics that can guide evaluators navigating contemporary challenges.
Setting: The article examines Ray Rist's influence across academic, governmental, and international institutions, including his work with INTEVAL, the General Accounting Office, the World Bank, IPDET, and IDEAS.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Four key characteristics define Ray Rist's approach. These characteristics guided his approach to INTEVAL over the past 40 years and remain relevant for evaluators today: (1) commitment to continuous professional development and lifelong learning; (2) dedication to addressing injustice through evaluation and audit; (3) focus on institutionalizing evaluation capacity globally, particularly in the Global South; and (4) emphasis on open collaboration and inclusive discussion. These characteristics have enabled INTEVAL to remain relevant despite changing times and can guide evaluators facing current global challenges including climate change, migration, and increasing populism
The Role of Theory in University Evaluation Coursework
Full text linkThis paper examines how theory is conveyed to novice evaluators in university coursework, providing a new perspective on the role of theory in evaluation. Using a survey of university evaluation instructors in the United States, the authors examined instructor perspectives on the role of evaluation theory as well as the evaluation content in assigned textbooks. Analysis indicated a similar lack of coherence and consistency in the use of theory to that seen in evaluation practice. The findings support three takeaways for evaluation educators, practitioners, and scholars. First, the field must continue to clarify the role of theory in evaluation. Second, an in-depth analysis of authors’ approaches to discussing theory in textbooks is warranted. Third, textbook authors and publishers could consider the themes that emerged from instructors’ responses to inform book proposals and revisions of textbook editions to better suit the range of instructor needs
Case Studies in Empowerment Evaluation: Restorative Practices Interventions in Educational Contexts
Full text linkBackground: Empowerment evaluation (EE) is a participant-led evaluation model that aligns with restorative justice (RJ) principles by centering the needs of the person or persons impacted.
Purpose: This article shares two longitudinal case studies, one using a practical empowerment evaluation approach with six U.S. Catholic institutions of higher education (IHEs), and another using a transformative empowerment evaluation approach with a cohort of 8 K-12 educators. These cases illustrate the benefits and challenges of utilizing empowerment evaluation for restorative justice programs in educational settings.
Setting: U.S. institutions of higher education (IHEs) and U.S. K-12 classrooms.
Intervention: Empowerment evaluation as a method to evaluate restorative justice implementation in multiple K-16 contexts.
Research Design: Two longitudinal qualitative case studies.
Data Collection and Analysis: Restorative circle dialogues; individual interviews; data collected during webinars; and an e-gallery walk provided data. Transcripts from circles and individual interviews were reviewed and organized by key themes to illustrate the benefits and challenges of using empowerment evaluation.
Findings: Empowerment evaluation can be conceptualized as a practical approach to evaluation, like formative evaluation, which aligns with the values of restorative justice, and focuses on program improvement. Transformative empowerment evaluation emphasizes liberation by empowering participants to step outside of typical roles, traditional structures, and assumed power relations. Due to the variation in these evaluation approaches, the case studies below vary in the emphasis placed on aspects of the evaluation process. Yet, both case studies highlighted participants’ needs for dedicated time for thinking and planning for restorative interventions; peer-to-peer support from colleagues from one’s own school or campus and from other campuses; sharing strategies for planting the “seeds” of restorative justice; and evaluation as an ongoing process of reflection and action
Capacity Development in Evaluation: The Role of the International Program for Development Evaluation Training (IPDET)
Full text linkBackground: Over the last decades evaluation has truly become a global field of practice. Of particular note has been the significant progress in the institutionalization of evaluation in low- and middle-income countries. Ray Rist, working at the Independent Evaluation Group (IPDET) of the World Bank, has been an influential champion of evaluation capacity development, among other things through establishing and leading the International Program for Development Evaluation Training (IPDET), a flagship global training program in evaluation.
Purpose: The article focuses on the role of IPDET in developing evaluation capacity. It highlights IPDET's contributions to the field, particularly in low- and middle-income countries, and discusses factors contributing to its success and future challenges.
Setting: IPDET was initially hosted by Carlton University from 2001 to 2016, offering a comprehensive training program on evaluation. In 2017, the University of Bern and the University of Saarbrücken took over, introducing a new program concept. The program has adapted to changes, including moving online during the COVID-19 pandemic and integrating into the Global Evaluation Initiative (GEI) network.
Findings: IPDET's success is attributed to its strong brand, external financial support, quality control, and community spirit. The program has trained thousands of participants globally and continues to adapt to evolving needs. Key factors for future success include differentiating itself from competition from other training programs, adapting to technological advances, and maintaining relevance in the evolving field of evaluation. The article emphasizes the importance of IPDET's holistic approach to training and its integration into the GEI network to enhance its impact
Democracy, Accountability, and Evaluation
Full text linkBackground: The focus of the article concerns the final step in the accountability chain in democratic societies, the one between the people and their elected representatives. The importance of this relationship has meant that questions regarding accountability and independent scrutiny have been important in the democratic discussions for more than 2300 years. Based on a discussion of this relationship, the article asks the question of what role evaluation can have in strengthening this relationship.
Purpose: Trust and mistrust must be balanced in a democratic society. An important prerequisite for this balance is that the citizens know that those to whom they have entrusted power are under constant and independent scrutiny.
Setting: The expansion of evaluation has meant that the amount of information, which potentially can be useful in the final step of the accountability chain, has increased. This development suggests that it is today easier than in previous times to hold the elected accountable. However, surveys and indexes show a different picture. More “old” democracies are described as flawed democracies. And it is not difficult to note developments in many democratic nations which seem to reflect increased mistrust in the elected.
We can also find indications of a lack of trust in evaluative information delivered by government offices, audit, inspections and research bodies. This seems most obvious in extreme situations, crises, breakdowns, suspicions of abuse of power and corruption or what is seen as just shocking incompetence, where routine oversight does not give trustworthy answers to the questions raised. The article points out that such situations seem to demand something extraordinary, ad hoc accountability mechanisms.
Intervention: N/a
Research design: N/a
Data collection and analysis: Systematic qualitative analysis. Particular attention is paid to the publications of Inteval in the area of accountability, and the role of investigating commissions in Sweden.
Findings: Since the 1960s accountability has been an important part of the evaluation discourse and has also been seen as an important purpose for evaluation. However, the article points out that the debate about accountability reveals tensions within the evaluation field.
The conclusion, given the background of lack of trust in many democracies, is that it is important that the evaluation community, more than earlier, emphasizes the importance of accountability and discuss how the evaluation practice can contribute to enhanced accountability.