Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Typology of Carrots, Sticks, and Sermons: Tracing Its Influence on the Discipline of Public Policy
Full text linkBackground: The 1998 book "Carrots, Sticks & Sermons: Policy Instruments and Their Evaluation" introduced a seminal typology categorizing policy tools by their mechanisms of influence. Despite its potential to shape policy theory and practice, its broader disciplinary impact and citation patterns remain underexplored. This article evaluates the book's legacy in public policy literature while testing bibliometric methods, including emerging generative artificial intelligence (GAI) tools for research synthesis.
Purpose: The study aims to (1) assess the influence of the book on the discourse in the public policy literature and (2) explore the utility and limits of AI-driven tools for tracing scholarly impact. It addresses four questions: the book's citation trends, its presence in major disciplinary texts, citation intent, and GAI's ability to reference the work accurately.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: A mixed-methods approach consisted of: (i) quantitative bibliometrics (citation counts, trends, and keyword analysis), (ii) qualitative content analysis by human researchers (major disciplinary handbooks/textbooks and articles), and (iii) generative AI semantic search and analysis.
Data Collection and Analysis: The quantitative analysis covered a corpus of interdisciplinary public policy literature from 1998 to 2024, extracted from citation databases (Scopus, Web of Science, Google Scholar). Qualitative analysis focused on 20 major handbooks and textbooks from public policy discipline (occurrence of references to book and policy instruments typology) and coding of 20 highly cited articles (classification of citation types in terms of intent). Work with GAI focused on semantic search and analysis with various AI platforms.
Findings: The findings underscore the typology's enduring relevance while advocating for methodological rigor in impact assessments and GAI-aided research. The book has achieved "classic" status in public policy, evidenced by 1,369 citations on Google Scholar, 924 citations on Scopus, 627 citations on Web of Science, as well as presence in 16 out of 20 major disciplinary handbooks/textbooks. Its typology is foundational but often superficially referenced; only 7 of 20 top articles applied it directly. GAI tools also referred to the book, although frequently misattributed or diluted the source, highlighting risks in automated literature reviews. The authors propose updating the typology of policy instruments to include behavioral tools (e.g., nudges) and stress the need for human verification of GAI outputs
From Studies to Systems: Ray Rist’s Influence on Evaluation Systems: Insights from International Research Group for Policy and Program Evaluation (INTEVAL)
Full text linkBackground: Evaluation systems aim to embed evaluation as a standard and routine practice within public organizations. By integrating evaluation into the everyday operations of government, these systems have the potential to enhance the relevance and use of evaluation findings, support organizational learning, and contribute to more transparent and accountable governance. Although evaluation systems are often promoted as tools for strengthening the connection between evaluative knowledge and decision-making, it is important to examine how these systems may both facilitate and constrain the development and meaningful use of evaluation.
Purpose: This article seeks to draw on the extensive body of knowledge developed by Ray Rist and the members of INTEVAL to better understand how evaluation systems are built, how they function, and what impacts they may have on the evaluation practice. By revisiting and synthesizing this extensive body of knowledge, we aim to extract key lessons for the design and implementation of evaluation systems that are both effective and adaptable to various contexts.
Setting: As researchers and evaluators working within academic institutions, our work is informed by interdisciplinary and international perspectives that we use to examine the development of evaluation systems across diverse political and administrative contexts. Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: This article is based on a literature review focusing on the institutionalization of evaluation and the development of evaluation systems. Particular attention is given to the Comparative Policy Evaluation series established by Ray Rist, which offers an interdisciplinary and internationally comparative body of work spanning over three decades. Drawing on this series, we examined how evaluation systems have been conceptualized, implemented, and critiqued over time and across diverse national settings.
Findings: This article highlights that evaluation systems are shaped by contextual factors, institutional arrangements, and the contributions of key individuals. The effectiveness of these evaluation systems depends not only on organizational design but also on the capacity and engagement of evaluators, commissioners, and decision makers. The effective functioning of such systems fosters an environment favorable to evaluation, achieves a balance between supply and demand, and safeguards the independence and integrity of evaluation
The Road Toward Institutionalizing Evaluation in Developing Countries: Following the Path of Ray Rist
Full text linkBackground: This paper traces Ray Rist’s many and varied contributions to evaluation capacity development (ECD) through the 2000s so as to provide a global perspective on his contribution towards institutionalizing evaluation in developing countries.
Purpose: The intent is to not only acknowledge the various ways that Ray Rist has supported countries in their institutionalizing of evaluation, but to also reflect on the various dimensions associated with building an effective and sustainable national monitoring and evaluation system (NMES), and, in so doing, draw lessons for new and emerging evaluators from the Ray Rist experience.
Setting: The early 2000s saw a major change in international development evaluation, as a new paradigm was emerging based on country ownership of evaluation and evaluation capacity that is linked to the national vision of the country, accountability, and good governance. This would require a new evaluation architecture centred around institutionalizing evaluation, and a different approach to ECD.
Intervention: Not applicable
Research design: The framework used in examining the various contributions of Ray Rist recognizes four broad building blocks to support countries develop and institutionalize their NMES – vision of leadership, enabling environment, technical capacity to supply monitoring and evaluation (M&E) information, and capacity to demand and use M&E information. All influence how far and how fast institutionalization of evaluation would occur at a country level (UNEG 2012; Lahey 2013; Lahey 2015).
Data collection and analysis: Data and information has been drawn from a review of documents, literature, formal interviews and consultations with former colleagues and key collaborators of Ray Rist, as well as first-hand experience of the author with ECD and NMES capacity development in a number of countries across the globe over the period in question.
Findings: Ray Rist was a catalytic force on his own career path, intersecting with and influencing the paths of others along the way, and all helping countries accelerate their own journeys towards country-owned and institutionalized NMES. Notable points are depicted along a non-linear path that included challenges and opportunities that Ray, collaborating with others, took advantage of to gain greater momentum
Linking Evaluation Theory and Practice: Exploring Ray Rist’s Enduring Legacy
Full text linkBackground: Over the course of his career, Ray C. Rist made substantial contributions to advancing evaluation theory and practice globally. This includes leading the International Research Group for Policy and Program Evaluation (INTEVAL) for nearly forty years, overseeing the publication of approximately forty books in the Comparative Policy Evaluation series.
Purpose: This special edition explores Ray Rist's influence and legacy over his career, and specifically through INTEVAL, examining how his achievements have impacted the domains of evaluation, audit, and learning. The aim is to identify enduring lessons for evaluators, auditors and policy makers regarding some of the issues addressed by Ray and INTEVAL.
Setting: The special edition encompasses the influence and legacy of Ray throughout his career spanning academia, government, international institutions, and voluntary service.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Contributors analyzed Ray's publication record, citation impacts, leadership approach, and collaborative methodologies. They also examine the influence of some of INTEVAL’s publications under the leadership of Ray and their influence on evaluative thinking and practice. The special edition draws on historical analysis, case studies, and theoretical frameworks to examine Ray's contributions to evaluation theory and practice, with particular focus on his collaborative work within INTEVAL and other organizations.
Findings: Ray Rist's nearly 200 publications have been cited almost 13,000 times, with significant impact both in his early academic work and later evaluation publications. His leadership style created a productive voluntary network of evaluation professionals that has sustained itself for four decades. Key contributions include pioneering work on policy instruments, results-based monitoring and evaluation systems, evaluation in accountability structures, and global evaluation capacity development. This special edition documents how Ray's collaborative approach advanced evaluation theory while maintaining strong connections to practice across diverse global contexts
Rubrics Methodology in Detail: Helping Save The Children Turn Children's Experiences of Discrimination and Exclusion into Rich, Trackable Outcomes
Full text linkBackground: Save the Children’s child sponsorship program (referred to as Sponsorship) supports children, their families and communities, and institutions in 20 countries to create social and behavior change together so that all children survive, learn, are protected, and live free from discrimination.
Purpose: To present a new rubrics-enhanced monitoring, evaluation, accountability, and learning (MEAL) system being developed with Save the Children’s Sponsorship Program. The system helps the organization track progress on key social change outcomes, especially less tangible outcomes, such as children’s experiences of inequality and discrimination.
Setting: Sponsorship’s work spans 20 countries around the world. It works through a 10-year program cycle, building community capacity to create social change that addresses the root causes of inequality and discrimination for a group ofchildren living in an impact area. The evaluation methodology showcased here was piloted in Zambia, the Philippines, and El Salvador.
Intervention: Not applicable.
Research Design: This paper presents some of the nuts and bolts of using rubrics to turn stories of change into rich, trackable outcomes, using one of the 12 rubrics developed for Sponsorship to illustrate the methodology.
Data Collection and Analysis: Focus group and interview protocols are used to gather stories and other documentation (such as health statistics and educational outcomes) from children, families, community members, educators, health care professionals, and local government officials. A detailed, step-by-step evidence interpretation guide shows how to synthesize these multiple sources of evidence and convert them into evaluative ratings using rubrics and guided evaluative reasoning.
Findings: Rubrics are a powerful tool for evaluating complex and nuanced outcomes, provided they are appropriately adapted for different contexts. For MEAL practitioners less familiar with rubrics methodology, detailed guidance and support are needed to help ensure that the framework is applied consistently across the system while also being contextually responsive. Based on Save the Children’s experience, several helpful strategies are presented for introducing rubrics into an existing M&E system: starting small and building momentum; opt-in piloting and gradual rollout; smart capacity building; providing more detailed guidance to evaluators and managers; getting buy-in to a specific measurement problem and solution rather than replacing the entire MEAL system; working with expert allies; and getting excited and innovative
Embracing Transformational Vision That Emphasizes Data Collection and Monitoring and Evaluation to Inform, Improve, and Shape Decision-Making in the Caribbean Region
Full text linkThe Caribbean encompasses an archipelago of islands with stunning beauty, tropical climates, and beautiful beaches. The region faces many developmental challenges caused by small size, undiversified economies, and regular exposure to various climate and nature induced hazards especially hurricanes which continuously impede developmental progress. The COVID-19 pandemic has worsened the region’s economic quandary, and derailed progress towards the United Nations Sustainable Development Goals (SDG) 2030 Agenda. Like many other parts of the world, the region has not yet developed a culture for monitoring and evaluation (M&E). The region’s development progress is also challenged by insufficient evidence-based data, inadequate disaggregated statistics, and dated national statistical systems. In 2019, governments of the region committed and adopted a results-based management (RBM) policy initiated by the CARICOM Secretariat, the most influential regional body. However, four years later, the implementation of this policy is still very much in an embryonic phase. This paper will summarize the current status of M&E in the region, examine the region’s progress towards Agenda 2030, and discuss why transformation thinking is important in the region’s quest for prosperity, resilience, and sustainability
Implementing Photovoice in School-based Evaluations: Navigating Challenges and Leveraging Benefits
Full text linkThis article explores the use of photovoice as an evaluation method in a high school summer bridge program (the Innovators program). Photovoice is a participatory research method where participants take photographs and share narratives in response to prompts that solicit their perspectives and experiences. With the goal of promoting critical dialogue between participants, photovoice method leverages the power of emic knowledge and visual representation to influence programmatic change. We explain how photovoice was implemented in a K-12 mixed methods evaluation and the practical and ethical considerations that evaluators must weigh. Findings indicate that photovoice not only enhances participant engagement but also serves as a valuable tool for program evaluation, yielding rich, qualitative data that traditional methods may overlook. Drawing on a multi-year evaluation of the Innovators program, the article contributes to the growing literature on photovoice by examining what the method means for young participants, in addition to describing the photovoice design and implementation processes
INTEVAL as a Positively Charged Social Network
Full text linkBackground: In this article Bastøe and Haslie seeks to employ theoretical insights from two connected bodies of literature to understand the unique characteristics of The International Evaluation Research Group (Inteval). One perspective draws on insights from social network theories about how some networks are supportive and innovative while others are not. The second perspective draws on organizational learning and Community of Practice theories which illuminate how informal networks facilitate learning, innovation, and knowledge sharing.
Purpose: This article aims to explore Inteval as a positively charged social network, employing theories on social networks and communities of practice to understand its impact.
Setting: Inteval is a multidisciplinary, multicultural, and multinational group of evaluation experts who convene annually to discuss innovative topics in evaluation, contributing to book publications in the Comparative Policy Evaluation Series.
Intervention: N/A
Research Design: The article uses a theoretical approach, drawing on social network and community of practice theories, supported by empirical observations and experiences within the Inteval network.
Data Collection and Analysis: The analysis is based on the author's extensive engagement with Inteval, supplemented by literature on social networks and communities of practice.
Findings: Inteval exemplifies the theoretical insights concerning features of a positively charged social network. Ray Rist's central role and personal connections have been pivotal in maintaining the network. Theories on organizational learning emphasize the importance of mutual engagement, joint enterprise, and shared repertoire, which are evident in Inteval's practices. The density and reciprocity within the network, evidenced by frequent and meaningful interactions, contribute to the group's longevity and effectiveness. The non-hierarchic structure further supports knowledge sharing and lowers barriers to participation. These theoretical insights underscore the significance of trust, belonging, and a supportive network culture in sustaining Inteval’s success and offer valuable lessons for other communities of practice
Ecology and Evolutionary Biology’s Unique Contribution to Evaluation
Full text linkBackground: Evaluation recognizes the need to consider three constructs – program change over time, the consequences of program action over time, and relationships between programs with their environments. Our methods for studying these constructs are home grown, i.e. they have developed almost exclusively within our field. These constructs, however, have a long and deep history in the fields of ecology and evolutionary biology (EEB). Thus it makes sense to consider how EEB might contribute to the models. methodologies, and data analysis strategies that evaluation applies to program change, outcomes, and program/environment interactions. Further, in recent years evaluation has been paying ever greater attention to how complex system behavior affects programs and their outcomes. Much in the fields of EEB can be seen as a subset of complexity.
Purpose: This article has two purposes: 1) to convince evaluators that EEB can empower their efforts to evaluate change over time in programs, outcomes, and program/environment effects, and 2) to spur the growth of a group of evaluators with an interest in further exploring EEB’s contribution to our field
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
From Ten Steps to Ten themes: Consequential Contributions in the Collective Compositions of Ray C. Rist
No full textBackground: This article is part of a collection recognizing, appreciating, and celebrating the substantial lifetime evaluation contributions of Ray C. Rist.
Purpose: This article presents 10 overarching themes from the impressive body of Ray Rist’s published works.
Setting: I write as a long-time colleague of and collaborator with Ray C. Rist.
Intervention: N/a
Research design: N/a
Data Collection and Analysis: In the introductory article, the authors offered an inventory of Ray Rist’s published works, some 34 books and 159 articles and chapters. I undertook a systematic qualitative thematic analysis of those publications as an experienced qualitative analyst. I brought to the identification of themes, and their importance, my own knowledge of evaluation as an experienced evaluation practitioner, theorist, and author.
Findings: Ten overarching themes from the impressive body of Ray Rist’s published works: (1) focusing evaluation on results not just activities; (2) taking a systems perspective on M&E; (3) engaging in comparative analysis; (4) valuing methodological diversity and rigor; (5) policy evaluation as a distinct and important focus of evaluation; (6) evaluation capacity building; (7) evaluation serving and advancing social justice; (8) editing prowess as a contribution to sharpen communications and enhance evaluation use; (9) working collaboratively; and (10) addressing and synthesizing leading edge issues. This overview concludes with the challenge of transforming evaluation to evaluate transformation. Ray Rist has long been in the forefront with prescient writings spotting, identifying, and naming transformational trends with implications for evaluation.