Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
What’s a Girl Gonna Do? Understanding Evaluation Theory from a Critical-Postcolonial Lens
No full textEvaluation theory is considered integral to good evaluation. There is, however, a lack of clarity on the distinction between prescriptive evaluation theory and evaluation approaches and perspectives. The distinction is further complicated by the central role of program theory in the praxis of evaluation. Notwithstanding, evaluation theory and theorists are popularly codified in the Alkin tree (Alkin & Christie, 2004; 2006; 2008) and presented in introductory evaluation classes in graduate programs. While the Alkin tree has seen several revisions, few female evaluators and even fewer evaluators of color are represented. In this study, we (three Black female evaluation graduate students) develop Critical-PostColonial Theory (CPT) as the analytical framework to conduct an autoethnography, interrogating and reflecting on the teaching and learning of evaluation theory in introductory program evaluation graduate classes. The paper concludes with suggestions for decolonizing the teaching and learning of evaluation theory within graduate evaluation programs and courses.
 
Using Competency-Informed Tasks to Guide Evaluation Capacity Strengthening
Full text linkBackground: EvaluATE, a federally funded evaluation hub, offers an array of resources and activities to strengthen evaluation capacity. However, there were some gaps across the full range of tasks that evaluators and evaluation users need to know how to do. The team created a framework based on evaluation tasks to guide their future evaluation capacity-strengthening work, with the aim of offering resources for all essential evaluation tasks.
Purpose: The article explains the need for a context-specific, task-based evaluation capacity strengthening framework, the framework’s relationship to the American Evaluation Association’s Evaluator Competencies, how the framework is being used, and lessons learned.
Setting: The Advanced Technological Education (ATE) program funded by the U.S. National Science Foundation
Intervention: EvaluATE, the evaluation hub for the ATE program.
Research Design: Not applicable.
Data Collection and Analysis: The evaluation task framework was validated through a task-tracking study, Delphi study, and expert review. Insights for this article are based on the authors’ experience.
Findings: The main lessons learned for developing task-based frameworks to guide evaluation capacity strengthening include: (1) Be clear about the scope and purpose of the task list (for both internal and external audiences). (2) In addition to stating what must be done for a given task, say why. (3) Verify the appropriateness of the tasks with typical practitioners and experts within the programmatic context
Theories of Change for Complex Sustainability Transformations: Mobilizing the Potential of Redefining the Relationship Between Humans and Nature
Full text linkCentral to transformations towards sustainability is the call to redefine the relationship between humans and nature. This redefinition represents a paradigm shift, with many asserting that it holds the greatest potential for transformation. However, the practical implementation of this call often remains abstract for those responsible for interventions in sustainability, such as projects and programs. This paper aims to initiate the translation of this call into actionable steps for the design of Theories of Change. It will achieve this by bringing three approaches into dialogue: Principles-focused, Complexity-aware, and Decolonization approaches.
 
From Ongoing Streams of Evidence to Final Synthesis
Full text linkBackground: Protecting the global environment requires complex program and project designs intended to produce multiple benefits in the various environmental domains (biodiversity, land degradation, water, climate change, and pollution) while benefiting the people, especially local communities living in the areas of intervention. In this context, learning is a key requirement to which independent evaluation must contribute. This necessitates bringing together streams of evidence from multiple perspectives and sources.
Purpose: The article brings out lessons learned in designing and conducting comprehensive evaluations of large-scale programs and projects funded by the Global Environment Facility (GEF) using streams of evidence. It also describes the utility and utilization of such evaluations for making decisions about future funding.
Setting; GEF is the oldest public financier for the global environment and serves as a financial mechanism to five multilateral environmental agreements in the areas of biodiversity, climate change, land degradation, and pollutants.
Intervention: Comprehensive evaluations of the GEF.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: The article draws upon the sixth and seventh comprehensive evaluations of the GEF, which were key inputs to the Fund’s replenishment negotiations. The article outlines the transition from standalone studies to continuous evaluation streams, which provided real-time insights to improve decision-making in the GEF. These evaluations assessed the GEF’s continued relevance and ability to effectively support the recipient countries in protecting the global environment. The evaluations covered project performance, governance, integration in programming, and transformational impact, as well as methodologies and addressing emerging challenges like financial constraints and the pandemic’s impact. The findings of these evaluations directly influenced GEF’s strategic direction and policy, leading to program refinements. The comprehensive evaluation experience demonstrates to the power of bringing together evidence from multiple studies into a stream that can provide both timely information on specific aspects and a comprehensive diagnostic of an organisation’s performance, strengths and weaknesses
The Interplay of Tangibles and Intangibles in Evaluation
Full text linkBackground: This paper reflects on my long-standing collaboration with Ray C. Rist, which began in 1997 and has evolved through decades of shared work in evaluation. Grounded in international development and governance, our collaboration has explored both tangible and intangible dimensions of evaluation to enhance institutional learning and policymaking.
Purpose: The study examines key lessons from this collaboration, emphasizing the role of evaluation in organizational transformation, the interplay of measurable and relational factors, and the strategic use of evaluative knowledge in decision-making.
Setting: The paper draws on experiences from diverse global contexts, including international development initiatives, university-industry partnerships, and public sector reforms. It situates evaluation within broader theoretical debates on governance and knowledge production.
Data Collection: The analysis is informed by direct involvement in evaluation projects, fieldwork across multiple regions, and scholarly contributions, including the Comparative Policy Evaluation series. Insights are drawn from case studies, interviews, and a review of evaluation practices applied in different institutional settings.
Findings: The study identifies four critical lessons: (1) the necessity of grounding evaluations in local realities to ensure relevance, (2) the importance of recognizing both tangible and intangible dimensions in program success, (3) the role of evaluative knowledge in fostering reflection and policy change, and (4) the advancement of evaluation scholarship to bridge theory and practice. These findings reinforce the evolving role of evaluation as a tool for navigating complexity, strengthening institutional capacity, and fostering inclusive governance
Complexity-Aware Evaluation for Learning: A Case Study of a Developmental Approach
Full text linkBackground: Ongoing learning is vital for steering programs in complex social-ecological systems, because it enables implementers to change course when needed, and ideally contributes to system wide insights as well. While there are many calls and guidelines for complexity-aware and learning-centred monitoring and evaluation (M&E), there are fewer case examples of attempts to implement it in practice.
Purpose: This paper reflects on a multi-year case of an attempt to design and implement a complexity-aware, developmental and learning-centred M&E framework in a complex socio-ecological landscape, highlighting the challenges, adaptations and evidence of learning.
Setting: Olifants River Basin, north-eastern South Africa and southern Mozambique.
Intervention: Resilience in the Limpopo (Olifants Basin) Program (RESILIM-O).
Research Design: Participatory action-reflection research with data collection through document analysis, interviews and focus groups
Findings: Learning among program implementers was facilitated by working in new ways with standard M&E elements including indicator-based targets and theories of change, and through opportunities for shared reflection, including novel reporting templates, reflection days, and collaborative case studies. Challenges notwithstanding, the participatory, developmental M&E approach built understanding of and competence in M&E in the organisation. The “hybrid” framework that evolved successfully combined the need for accountability with the desire for organisational learning; and associated features were subsequently adopted elsewhere
Implementation Fidelity: The Disconnect Between Theory and Practice
Full text linkThe current paper combines a conceptual analysis of major reviews of implementation fidelity studies (e.g., Dane & Schneider, 1998; Mowbray et al., 2003; O’Donnell, 2008) with reflections on fidelity measurement practice in field evaluations. It claims that practice is impoverished by the failure to recognize the existence of competing conceptualizations of fidelity, rooted in different theoretical perspectives. Different evaluation contexts may be better matched to one or the other of these perspectives. Confusion about how fidelity should be defined in a given funding program or evaluation prevents evaluators from instituting a maximally useful fidelity measurement program. Difficulties inherent to creating high-quality fidelity measures contribute to the problem. The causes and consequences of this disconnect between fidelity theory and fidelity practice are discussed herein. Preliminary suggestions for solutions are advanced
Thinking and Feeling Matter: Utilizing Impact Assessment and Photovoice to Evaluate a Community Program
Full text linkBackground The benefits of adopting mixed methods have been widely recognized in evaluation studies. However, such an methodological approach is also criticized for its lack of rigorous epistemological concerns, or simply as a-paradigmic. Therefore, in using mixed methods, how knowledge’s truth value can possibly contribute to program betterment remains uncertain.
Purpose This paper aims to provide an example of how the use of a mixed methods approach in a community program can lead to reflections on its epistemological values and possibly result in program improvements.
Setting The evaluation took place in the longest cultural walk trail in urban Hong Kong.
Intervention The evaluation method employed a combination of impact assessment, participant’s photovoice, and multiple meetings with the program managers.
Research design A mixed methods design.
Data collection & analysis Relevant statistical and thematic analyses were conducted.
Findings The data obtained from the mixed methods approach, along with its underlying epistemological foundations, revealed a unique experience for the participants, encompassing both cognitive evaluation and emotional appreciation of the program. Based on these findings, relevant suggestions were made to enhance the program
Investigating Equity and Social Justice within Programs: A Framework of Evaluative Criteria
Full text linkBackground: Evaluators have a professional and ethical responsibility to contribute to the “advancement of an equitable and just society.” A rich body of scholarship provides guidance about how evaluators can do so through social justice–oriented and decolonizing evaluation approaches, culturally responsive methods, attending to power and privilege in program and evaluation contexts, and partnering with communities. In this article, we provide guidance for examining how the program being evaluated attends to issues of equity and social justice.
Purpose: We present a framework for investigating equity and social justice within programs through the criteria that evaluations pursue. The framework is offered as a map of possibilities and a thinking tool to help evaluators surface, examine, and negotiate varying values and design evaluative lines of inquiry to address them.
Setting: We write as evaluation faculty in colleges of education. We approached the framework through the lens of our shared commitment to advancing equity and social justice in evaluation practice. We aimed to bring together our differing areas of expertise and lived experiences to develop a resource to support evaluators in advancing equity and social justice.
Data Collection and Analysis: Our work builds on a previously published model of criteria domains. Each domain reflects a broad category of program characteristics or results that can be deemed important or desirable for a given program and context. Here, we refine and expand the model by applying an equity and social justice lens to 11 different criteria domains. We draw on the social justice–oriented evaluation literature for guidance and examples about how evaluators can advance equity and social justice within each domain and associated lines of inquiry.
Findings: The framework outlines 11 criteria domains in which a program’s contribution to equity and social justice might be examined. We describe each domain and apply it to an example evaluation to illustrate. We conclude by discussing the use of the framework to advance equity and social justice through evaluation practice, education, and scholarship
Building a Mountain of Evaluative Evidence, 2004-2014
Full text linkBackground: The book From Studies to Streams was for me an eye-opener when I worked as Director of the Independent Evaluation Office of the Global Environment Facility (GEF). Right from the start in that position I was working on gathering as much evaluative evidence as we could, mix this with knowledge and insight, and look at how to deliver recommendations and insights to the GEF. From Studies to Streams inspired me to work towards a potential mountain of evidence to inform and inspire the replenishment meetings of the GEF. While the book provided an analogy of evaluation insights and evidence streaming down to the ocean, I felt that the knowledge gathered in the Overall Performance Studies would be more moving up, to reach higher levels of decision-making. At the top of the mountain of evidence and insight, the GEF replenishment meetings would decide on the goals of the Fund in the next four years.
Purpose: The chapter aims to provide a historically accurate account of how evaluations at different levels of GEF funded activities were used to inform higher-level evaluations, leading to an integrative perspective of achievements. Evaluations incorporating both scientific and national/local perspectives aimed to capture findings and insights at all levels of the GEF and its partners. While this was not the stream downwards to the ocean, it could be likened to steam that gradually wafted up to the pinnacle of GEF decision-making.
Setting: The partnership of the GEF with multilateral banks, five UN organisations and the 120 recipient countries of GEF funding. While all agencies and countries had their own evaluation policies, agreement was reached about a minimum number of common elements that would be reported on. All evaluations that touched upon GEF issues were studied to subtract insights relevant for the GEF. This was combined with knowledge generated through the Scientific and Technical Advisory Panel (STAP) of the GEF, and any relevant knowledge available through literature and expertise.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: It turned out to be possible to create a flow of evaluative evidence and other insights and knowledge regarding interventions from the many varieties of evaluation that were undertaken in the GEF and its many partners. This led to a veritable mountain of evidence that was presented every four years to the replenishment meetings for the GEF. While the GEF is a relatively unique international funding organisation, it turned out to be possible to make use of the evidence generated at various levels of the partnership and by different actors, along the lines of the Studies to Streams book