Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Rethinking Evaluator Competencies in an Age of Discontinuity – Implications of Inequities, Sustainability and the Pandemic for Training Evaluators: An Introduction to a Special Volume
Full text linkConcluding Thoughts and Reflections on the Special Issue on Program Evaluation Standards
Full text linkThis paper seeks to address the issue of relevance of the current edition of the Program Evaluation Standards published by the Joint Committee on Standards for Educational Evaluation. The paper addresses concerns related to the nature of the standards, their current applicability to practice, their comprehensiveness and completeness. The presented conclusions are that the standards are applicable, relevant, complete, and comprehensive
We Can’t Hear You – You’re on Mute: Findings From a Review of Evaluation Capacity Building (ECB) Practice Online
No full textBackground: In her presidential address to the American Evaluation Association (AEA) in 2007, Hallie Preskill (2008) highlighted the potential role of technology to promote learning from evaluation, noting the increased use of computers, the internet, and social media as untapped ways to facilitate evaluation. More than ten years later in the context of the COVID-19 pandemic, evaluators and evaluation capacity building (ECB) practitioners found themselves needing to shift to online modalities to conduct evaluation and build capacity. The COVID-19 pandemic, technological advancements, and the rapid shift to remote work have changed our way of working (Gratton, 2021; Kane et al., 2021). Building evaluation capacity is no exception to this trend.
Purpose: This study aimed to examine ways that practitioners have built evaluation capacity online or have used technology to do so, to capture lessons learned that can be applied in a COVID and post-normal context.
Setting: Findings from this study can be applied in online contexts for developing evaluation capacity.
Intervention: Not applicable.
Research Design: The study design consisted of a rapid review of the ECB literature published from 2000 to 2019 in eight academic journals focused on evaluation research and practice.
Data Collection and Analysis: Twenty-nine case applications of ECB practice that: 1) mentioned use of technology as a strategy for building evaluation capacity or 2) noted that at least one component of the ECB intervention was carried out online or virtually were reviewed for this study. Quantitative data were analyzed via descriptive statistics. Qualitative data were coded in MAXQDA using conventional content analysis (Hsieh & Shannon, 2005).
Findings: More diverse online interventions have increased over time. Less than half (45%) of ECB interventions made use of both asynchronous and synchronous strategies for building capacity while more than one-third (38%) made use of asynchronous only strategies. Key barriers to implementing ECB strategies online included lack of social connections to other participants during the capacity building activity, technical malfunctions, lack of access to or familiarity with the technology in use, and limited resources for carrying out evaluation activities. Key facilitators for enhancing implementation included facilitating participant interaction and relationship-building both on and off-line, tailoring ECB activities to participant work contexts, and providing tutorials for accessing and using the technology in play
Framing Anticolonialism in Evaluation: Bridging Decolonizing Methodologies and Culturally Responsive Evaluation
Full text linkBackground: Evaluation is grounded in academically imperialistic research methodologies, paradigms, and epistemologies, which have lasting effects on communities of study. Confronting Westernized evaluation's monoculturalism, scholars call for decolonization, to produce locally-determined, strengths-based, culturally-situated, and valid understandings. This endeavor is complicated, requiring a paradigm shift for Westernized evaluators.
Purpose: In this paper, we describe the anticolonial culturally responsive framework occurring in the intersections between culturally responsive (CRE) and decolonizing (DF) approaches. Anticolonialism honors decolonizing without displacing the authority of Indigeneity, simultaneously foregrounding the interweaving of evaluator, evaluand, and disciplinary culture. Interrogating academic imperialism through anticolonialism, confronts the social processes and cultural ideologies that produce and reproduce social inequality in evaluations.
Setting: Not applicable.
Data Collection and Analysis: We draw on scholars and scholarship who have advanced culturally responsive, decolonizing, and anticolonial evaluation and methodological fields.
Findings: The anticolonial culturally responsive framework is an invitation for evaluators trained in imperialistic Westernized approaches or who embody the colonial world through our race, language, knowledge, and culture. Our goal is not to displace the primacy and urgency of vitalizing Indigenous and decolonizing frameworks. Instead, we offer a tentative approach committed to pluriversality, justice, self-determination, and the possibility of collaboration between knowledge systems and knowers
Foreword
Full text linkIntroduction to the collaboration between the International Evaluation Academy and JMDE, a special issue on decolonization.
Fostering Values-Driven Sustainability Through an Ex-Post Capacities Lens
No full textMonitoring and evaluation (M&E) work is driven by an array of values held by funders, implementers, M&E experts, and the project participants themselves. Some are explicit, such as embedding equity or democratization values into aid projects. Some are assumed, such as the truth of “values-neutral” evaluation or that long-term sustainability will result from projects. The author espouses Quinn Patton’s “activist interventionist change-committed evaluation” in pushing for ex-post evaluation of much development aid programming’s untested hypothesis about the sustainability of results without proof post-project. Programming approaches can make development results more sustained and can be monitored and evaluated. One set of activities is the participatory involvement of national and local stakeholders from the onset and during all stages of programming. The activism includes M&E experts pushing to include listening to local participants and partners via mixed-methods evaluations, especially by hiring local evaluators, and asking not only about donor-expected results but also about locally emerging outcomes, along with sharing learning at all levels, from local to donors. Focusing on the sustainability of outcomes and impacts involves those doing M&E and those involved in explicit sustainability planning from design, with national project stakeholders at every step. This chapter focuses on a roadmap of capacities and tools needed to foster sustainability pre-exit drawn from a decade of ex-post project closure evaluations. Current evaluators can help improve development practice and the durability of results by honing their capacities to evaluate the natural system on which most projects rest and how to foster resilience to climate change via M&E. Once this is built, transformations toward locally-driven development become possible
Liberated or Recolonized: Making the Case for Embodied Evaluation in Peacebuilding
Full text linkThe quest to liberate and decolonize evaluation could create a recolonizing process in development evaluation unless practitioners pay attention to an embodied process that allows persons and communities in the global south to bring all of their epistemologies to an evaluation process. This will enable evaluation to be a learning process through which communities gain insights from their programs through their own ways of knowing. Constrained by time, resources, and limited commitment to the decolonization agenda, organizations and experts in evaluation could cause further harm to indigenous and local ways of knowing through half-baked decolonization processes . Through the lens of peacebuilding, this paper proposes an embodied evaluation as a practical way to decolonize evaluation, and reduce the risk of recolonizing the concepts and processes of project evaluation while addressing some of the power imbalances that lie beneath traditional evaluation and development. This paper suggests that monitoring , evaluation and learning of projects should be led by those most affected and closest to the problem being addressed and experts who assist in these processes should a assume a collaborative approach that requires the embodiment of the process and experiences of the affected. This approach is likely to generate reports through the world views of those affected and reduce dominance and imposition of external methodologies that distort outcomes of programmes. This paper is a reflection on practice, and a prompt for further research on embodied evaluation as a decolonizing agenda
Utilizing journey mapping to evaluate youth programs and social service systems: Case studies
Full text linkBackground: Journey mapping is a relatively new, promising method for use in the evaluation sector. In this method, individuals who have gone through a program or system provide feedback on the process in a chronological way, highlighting successes and challenges they have encountered through the process. This article provides an overview of the method and three case studies describing how journey mapping has been used to evaluate youth programs and social service systems.
Purpose: To aid evaluators in understanding journey mapping and the ways in which it can be applied to various evaluation projects with the intention to help them determine whether or not the method is appropriate in their practice.
Setting: Journey mapping can be applied in evaluations examining how individuals and groups interact with programs, organizations, and systems.
Data collection and analysis: The journey mapping method, including data collection through focus groups and interviews and thematic analysis of notes and transcripts.
Findings: Journey mapping is a method that can illuminate successes and challenges individuals and families face when interacting with a program, organization, or system. Specific recommendations for study design, data collection, analysis, and reporting are offered for evaluators’ consideration
