Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Equity “On the Sideline”: A Mixed Methods Study of New England Evaluation Practice in 2020
Get PDFBackground: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice.
Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study.
Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).
Intervention: Not applicable
Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis.
Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.
 
How should Program Evaluation Standards inform the use of cost-benefit analysis in evaluation?
Get PDFBackground: Cost-benefit analysis (CBA), like any other evaluation method, should be used in ways that uphold program evaluation standards and should be subjected to metaevaluation. In contrast to the broad remit of program evaluation standards, guidelines for economic evaluation focus mainly on technical aspects of evaluation quality, aimed at ensuring precision, accuracy, and reliability. Can CBA be conducted in adherence both to program evaluation standards and to its own methodological principles, or are there areas where expectations conflict?
Purpose: Assess the potential for CBA to be conducted in keeping with the Program Evaluation Standards (PES) of the Joint Committee on Standards for Educational Evaluation.
Setting: Analysis applies to any setting in which CBA is being considered as an evaluation method.
Intervention: N/A
Research Design: Methodological principles underpinning CBA were systematically assessed against the PES, to determine the extent to which CBA can be conducted in a manner aligned with these standards. CBA was rated according to whether it can follow each standard in principle, not the extent to which economists follow a given standard in practice.
Data Collection and Analysis: This assessment was undertaken from a theoretical perspective, through analysis of relevant literature. The ratings are evaluative; they represent the judgments of the author, made on the basis of explicit definitions.
Findings: Some ethical principles espoused in the PES are also required in CBA. On the other hand, some of the PES are not explicit requirements in CBA, though they could be applied by evaluators or economists when conducting a CBA. However, some PES logically cannot be met by CBA if it is used as a stand-alone method. All PES can theoretically be met when an evaluation combines CBA with other methods. In order to use CBA in adherence to PES, evaluators and economists must take an explicit interest in the effects of their analysis on people’s lives. This has significant implications for the way CBA should be used, including the nature and extent of stakeholder involvement, the potential use of CBA in conjunction with other methods, and decisions about when not to use CBA. As with any evaluation method, deliberation is necessary over whether, when, and how to use CBA
Building Spaces for Dialogues to Rethink Evaluator Competencies: Lessons from the Webinars Organized by the Evaluation Centre for Complex Health Interventions
Get PDFBackground: There is a need to rethink evaluator competencies given the harsh and paralyzing realities of COVID. The pandemic was a time where there was a need to balance diverse perspectives given the limited scientific evidence that existed when faced with a genuinely unprecedented time. In the Fall of 2021 (September to October), the Evaluation Centre for Complex Health Interventions in partnership with the Asia Pacific Evaluation Association organized a three-part webinar series in response to the multiple issues that surfaced during COVID-19, and specifically, the implications of the pandemic for rethinking evaluator competencies and evaluator training. The presenters were from multiple countries including India, Canada, USA, UK, and South Africa.
Purpose: The presenters pushed for more responsive evaluation approaches to address inequities and sustainability and for a decolonized approach to knowledge building. The webinar raised a number of themes that have potential implications for future discussions on evaluator competencies including: enhancing evaluation contributions to the Sustainable Development Goals (SDGs), the need to rethink evaluation criteria, the need to embrace and address varieties of uncertainties, focus on diversity and heterogeneity; understanding the role of contexts in complex programs and policies; the need to reconceptualize sustainability; being more explicit about inequities and vulnerabilities; and the need to pay attention to systems and system dynamics.
Setting: The webinars were organized by the Evaluation Centre and the Asia Pacific Evaluation Association on a Zoom platform.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Excessive Evaluation Anxiety (XEA): The Last Two Decades
No full textBackground: Excessive evaluation anxiety (XEA) refers to disproportionate or increased evaluation anxiety among those affected by evaluation (e.g., stakeholders) characterized by the sole presence of negative consequences. It can compromise evaluator-stakeholder relationships, presenting as a barrier for program evaluation. Moreover, XEA can both cause and be caused by resistance to evaluation, which is an interrelated topic that shares many common causes, characteristics, and mitigation strategies. The participatory and interactive nature of modern evaluation approaches can exacerbate the presence of XEA. However, researchers have not explored the current state of literature on XEA.
Purpose: To explore the current state of the literature on XEA over the past 20 years.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Literature review.
Data Collection and Analysis: We conducted a literature search of Academic Search Complete, Web of Science, and Scopus. We complemented the database search by a journal search of the American Journal of Evaluation, Evaluation, and the Canadian Journal of Program Evaluation. We then conducted a thematic analysis of the articles that met the inclusion criteria.
Findings: Upon review of the articles, we identified four main themes in the literature related to XEA. Specifically, XEA: leads to poor evaluator-stakeholder relationships; is influenced by cultural factors; can be mitigated through the development of interpersonal skills; and can be mitigated through a systematic and evidence-based approach to evaluation
Decolonizing Data Visualization: A History and Future of Indigenous Data Visualization
Get PDFOn the whole, the field of data visualization is white. Contemporary views of historical data visualization tend to trace back to a few iconic visuals tied to European wars and conquests. The modern explosion of the field has been centered around the ideas of white men, as if they invented data visualization. Yet, Indigenous populations world-wide have been incorporating data visualization into their record keeping for centuries before anyone had heard of Edward Tufte. In this article, three Indigenous evaluators (Mohican/Munsee, Cherokee, and Tlingit) along with a non-Indigenous co-conspirator, will discuss their journeys creating space to weave together Western notions of data visualization best practices and Indigenous ways of knowing and storytelling. The authors focus their evaluative work on the support of Indigenous communities and will reflect on what has worked in communicating data, what hasn't, and how far data visualization has to go in all four directions
Between Funding Requirements and Community Priorities: Centro Hispano of Dane County’s Transformative Approach to Program Evaluation
Get PDFEvaluation approaches that aim to support large-scale social change need to address neoliberal logic ingrained in the way evaluation has been institutionalized in the US since the early 1900s. Harmful dynamics resulting from evaluation’s institutional history include (1) a focus on accountability and effectiveness, (2) the perpetuation of deficit-based narratives about communities of color, and (3) a top-down approach to program development, in which funders define program goals and assessment criteria and outside academics are hired to provide research services. In consequence, evaluation contributes to the extraction and devaluation of community expertise rather than fostering learning, collaboration, critical reflection, and healing.
This article highlights ways of addressing these harmful dynamics through a case study that exemplifies an innovative evaluation approach focused on community strengths and values, healing ethno-racial trauma, and critical consciousness building. We call for funders to rethink their requirements for evaluation and emphasize the need to support evaluation infrastructure, time for critical reflection, and the development of community- and asset-based, culturally responsive evaluation approaches and tools
Steps Toward Evaluation as Decluttering: Learnings from Hawaiian Epistemology
Get PDFThis paper discusses one of the more contemporary challenges in development and in global health--lots of good ideas from well-meaning insiders and outsiders that end up cluttering both the physical and mental spaces of what can be loosely termed as “attempts” at development. Given the place-based nature of indigenous thought, we turn to Hawaiian epistemology at looking to insights for clarity on how one can negotiate interactions to declutter place and also confuse identity. We believe that evaluation as a field can help in bringing greater recognition of the need for models of development and learning that respect the importance of de-cluttering. Implications for a decolonized approach to evaluation are discussed
 
Editors' Note
Get PDFWith this first regular JMDE issue of 2023, we are remembering our past and looking to our future. In the Articles section, we bring you six engaging pieces: a research study focused on the relationship between employee motivation and evaluation capacity building; a methods piece examining journey mapping in youth programs; and four different reviews focused on online evaluation capacity building practice, evaluating equity-focused community coalitions, excessive evaluation anxiety, and the transnational influence of Jennifer Greene’sideas. In the Ideas to Consider section, we offer a piece on artificial intelligence and another on complexity-based evaluation of transformation. We also present a special In Memoriam section remembering Dr. Chris Coryn, who was JMDE’s executive editor and director of the Interdisciplinary PhD in Evaluation (IDPE) program at Western Michigan University
Decolonizing Evaluation of Indigenous Land-Based Programs: A Settler Perspective on What We Can Learn from the LANDBACK Movement
Get PDFThere is increasing recognition of the importance of land-based programming in promoting Indigenous cultural resurgence and community-building. This article explores the challenges associated with evaluation of on the land programs, and considers ontological and epistemological challenges associated with applying a postpositivist western evaluation approach. It is suggested that Indigenous-led evaluation should be mandated for Indigenous land-based programs, consistent with emerging recognition of Indigenous peoples’ rights to sovereignty over programs and services