Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Decolonizing Science: Undoing the Colonial and Racist Hegemony of Western Science
Full text linkDecolonization is the complicated and unsettling undoing of colonization. In a similarly simplified definition, science is a structured way of pursuing knowledge. To decolonize science thus means to undo the past and present racist and colonial hegemony of Western science over other, equally legitimate, ways of knowing. This paper discusses the paradigmatic prerequisites and consequences of decolonizing Western science. Only if Western science is toppled from its pedestal and understood in a cultural way can it engage with other sciences at eye level. Such equal collaboration that results in the co-creation of new knowledge based on the scientific method and Indigenous scientific inquiry is what decolonizing science is all about. What it looks like in practice is highly variable as there is no one-size-fits-all approach due to the fact that Indigenous knowledge is rooted in the local, the land. Therefore, decolonizing science is much more a path than a destination. This path, however, will also pave the way to a new multiparadigmatic space. A quick look into the history and philosophy of science reveals that new paradigms have always emerged after a few trailblazers started engaging in a new way of doing science
The Evaluation of Equity-Focused Community Coalitions: A Review of the Empirical Literature
Full text linkConducting evaluation and research with community coalitions involved in health equity initiatives is inherently complex. In this paper we provide a review and synthesis of the empirical literature on the evaluation of equity-focused community coalitions. We explore issues, challenges, and barriers experienced by evaluators, as well as techniques and approaches that were considered beneficial. Our review identified 11 peer reviewed articles, from which we identified seven overlapping themes: (1) framing equity in the evaluation process, (2) use of multiple theoretical frameworks, (3) use of systems-focused approaches, (4) strategic use of intersectoral partnerships and collaborations, (5) intentional communication and building trusting relationships, (6) challenges dedicating purposeful time to the work, and (7) issues of cultural and contextual clarity and responsiveness. Our findings point to a significant focus on context, history, learning, communication, relationships, and power. The cultural complexity and historical scope of each context, diversity of stakeholders, and enormity of the systemic issues involved, shape and challenge the evaluation and research process in fundamental ways, requiring a creative and kinetic thinking -- a shifting from methodological certainty to an acknowledged uncertainty, where mixing, blending and the innovative use of approaches and theories becomes a way of moving beyond the colonizing past
Sustainability, Evaluation, and Credentials
Full text link While there is clear demand for sustainability-ready evaluation in which environmental impacts are integrated, a major gap exists between this desire and reality. Stock takings show that environmental sustainability is rarely addressed by evaluations. For this to happen, a significant adaptation in how evaluation is organized and conducted is needed, and evaluators, commissioners and evaluation users must take an interest in sustainability. Competencies of individual evaluators and firms must be enhanced, but this is not sufficient. Sustainability-ready evaluation requires interdisciplinary competencies and collaboration
The Relationship Between Employee Motivation and Evaluation Capacity in a Community-Based Education Organization
Full text linkBackground: Evaluation capacity building (ECB) has gained popularity among organizations due to the increased importance of accountability and organizational effectiveness. While the ECB literature has occasionally addressed the notion of motivation, it has usually been in terms of motivation to do or use evaluation (Clinton, 2014; Taylor-Ritzler et al., 2013); this study sought to ascertain whether general overall employee motivation in an organization is itself related to evaluation capacity. By better understanding this relationship, those who are involved in administering, implementing, evaluating, or researching ECB can be better equipped to understand one of the ‘mediating conditions’ or ‘antecedent conditions’ (Cousins et al., 2014) affecting an organization’s ability to do and use evaluation, and, in turn, can more efficiently and effectively craft their ECB work.
Purpose: The purpose of this study was to explore the relationship between: (a) employee motivation and individual evaluation capacity; (b) employee motivation and evaluative thinking, and (c) evaluation capacity and evaluative thinking.
Setting: The study focused on the Cooperative Extension System, a non-formal community-based education organization linked to public land-grant universities throughout the United States. Specifically, this study drew participants from two state Extension systems, Virginia and Maryland.
Intervention: Not applicable.
Research Design: This quantitative study used a descriptive correlational design (Creswell, 2003) to uncover the relationship between the variables: motivation and evaluation capacity, motivation and evaluative thinking, and evaluation capacity and evaluative thinking.
Data Collection and Analysis: To investigate the relationship between the factors of interest (motivation, evaluation capacity, and evaluative thinking), three instruments were used: the Multidimensional Work Motivation Scale (MWMS), the Evaluation Capacity Assessment Instrument (ECAI), and the Evaluative Thinking Inventory (ETI). STATA MP 13.1 quantitative software was used to analyze the collected data.
Findings: Employees with lower overall motivation in doing their work have lower evaluation capacity, and employees with higher motivation which is triggered by no external means but driven by internal factors have higher evaluation capacity
Program Evaluation Standards for Utility Facilitate Stakeholder Internalization of Evaluative Thinking in the West Virginia Clinical Translational Science Institute
Full text linkBackground: The Program Evaluation Standards (PES) can be considered established criteria for high quality evaluations. We emphasize PES Utility Standards and evaluation capacity building as we strive for meaningful application of our work in the real world.
Purpose: We focused our methodology on understanding how stakeholders discussed utility and how their perceptions related to our evaluation work aligned with PES Utility Standards.
Setting: The West Virginia Clinical Translational Science Institute (WVCTSI) is a statewide multi-institutional entity for which we conduct tracking and evaluation since 2012.
Intervention: Sustained collaborative engagement of evaluation stakeholders with the goal of increasing their utilization of evaluation products and evaluative thinking.
Research Design: Case study.
Data Collection and Analysis: We interviewed five key stakeholders. Themes developed from analysis of PES Utility standard coding of interview data informed document analysis. Interview and document analysis were used to develop themes and illustrative examples, as well as to develop and describe a five-level Evaluation Uptake Scale.
Findings: We describe shifts in initiation, use, and internalization of evaluative thinking by non-evaluation personnel that prompted development and application of an Evaluation Uptake Scale to capture increased evaluation capacity among stakeholders over time. We discuss how focus on PES Utility and evaluation capacity building facilitated such shifts and their implications for maximizing utility of evaluation activity in large complex programmatic evaluations
Using Dissemination Research Approaches to Understand the Awareness, Adoption, and Use of The Program Evaluation Standards
Full text linkBackground: The adoption and use of effective, legally defensible, and ethically sound practices relies on the successful dissemination of evidence-based practices and professional standards. The field of program evaluation has standards, competencies, and principles, yet little is known about how these are utilized by education-focused program evaluators.
Purpose: The purpose of this study is to examine the dissemination and use of the program evaluation standards established by the Joint Committee on Standards for Educational Evaluation, relative to the dissemination and use of the American Evaluation Association’s (AEA’s) guiding principles and AEA’s evaluator competencies.
Setting: The SIGnetwork, a network of evaluators of State Personnel Development Grants (SPDGs) funded by the U.S. Department of Education, Office for Special Education Programs (OSEP).
Intervention: NA
Research Design: Descriptive research.
Data Collection and Analysis: Data collection involved administering an online survey to members designated as evaluators in the SIGnetwork directory. Descriptive statistics were used to summarize the data collected via the online survey.
Findings: Using the formative audience research approach to understanding dissemination, the results of the study support previous findings that awareness of the standards was inconsistent among a sample of AEA members. Respondents self-reported low to moderate levels of familiarity with The Program Evaluation Standards and the other two guidance documents: Guiding Principles for Evaluators and AEA Evaluator Competencies. Using the audience segmentation research approach to understanding dissemination, the results of this study indicate that participants who were AEA members were more likely than those who were not members of AEA to report being familiar with the standards and to have earned an advanced degree related to their role as an evaluator
The Integration of the Program Evaluation Standards into an Evaluation Toolkit for a Transformative Model of Care for Mental Health Service Delivery
Full text linkBackground: Stepped Care 2.0 (SC2.0) is a transformative model of mental health service delivery. This model was created by Stepped Care Solutions (SCS), a not-for-profit consultancy that collaborates with governments, public service organizations, and other institutions that wish to redesign their mental health and addictions systems of care. The SC2.0 model is based on 10 foundational principles and 9 core components that can be flexibly adapted to an organization’s or community’s needs. The model supports groups to reorganize and deliver mental health care in an evidence-informed, person-centric way. SCS partnered with evaluators from the Centre for Health Evaluation and Outcome Sciences (CHÉOS) to create a toolkit that provides evaluation guidance. The toolkit includes a theory of change, guidance on selecting evaluation questions and designs, and an evaluation matrix including suggested process and outcome metrics, all of which can be tailored to each unique implementation of the SC2.0 model. The objective of this resource is to support organizations and communities to conduct high-quality evaluations for the purpose of continuous improvement (a core component of the model of care) and to assess the model’s impact.
Purpose: The purpose of this paper is to discuss the integration of the program evaluation standards (PES) into an evaluation toolkit for SC2.0.
Setting: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation.
Intervention: Not applicable
Research Design: Not applicable
Data Collection and Analysis: Not applicable
Findings: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation
Reorienting Evaluator Competencies: Learnings from Evaluation Practice During the COVID-19 Pandemic
Full text linkBackground: The Covid-19 Pandemic has emphasized the need for evaluators to reorient their skills based on ongoing learnings from evaluations. As a result, evaluators must possess a variety of competencies to meet the challenges created by such unprecedented circumstances.
Purpose: Using existing literature about evaluators' competencies and the experiences of conducting evaluations during pandemic, this paper proposes a set of competencies that enable an evaluator to collaborate meaningfully with grassroots organizations and co-design evaluations with communities that empower evaluators to think beyond the boundaries of a project intervention towards achieving a larger goal.
Setting: Covid 19 Pandemic and evaluation practice in India
Intervention: NA
Research Design: Case study with interpretive approach
Data Collection and analysis: NA
Findings: The paper suggests that acquiring 'strategic thinking' and 'emotional intelligence and resilience' strengthens evaluators' competencies to be flexible and innovative in evolving evaluation methods
The Commitment Mural: Let’s Decolonize Evaluation Together
Full text linkIn this mural we demonstrate liberation of evaluation from normalized publication structures. The article is a challenge to publication structures, more specifically journals, to open space for diversity of expression of thought. In this mural, practitioners, academics, policymakers, development partners, VOPEs (Voluntary Organizations for Professional Evaluations), speak to the strategies to decolonize evaluation. The immediate impact of this mural is a demonstrated commitment to decolonize evaluation by more than 50 individuals and entities. Taking up the commitment will be a demonstrable change that came with the mural. The mural also provides an open space for the exchange of ideas, and strategies to decolonize evaluation. Moreover, the contributing voices are preserved in their own voices and their contribution to global knowledge production is acknowledged. This is indeed a departure from the ethics of confidentiality in research where the participants in an evaluation inquiry largely remain anonymous even in the co-creation of knowledge. We believe that when participants in an evaluation inquiry come across their commitments in evaluation journals, they are encouraged to further form coalitions with like minds and hold themselves accountable collectively to liberate evaluation from the remnants of colonialism
Paradigm-Based Evaluation for Eco-Just Systems Transformation
Full text linkSocial and ecological systems shape the extent to which humans and the rest of nature live together in a state of well-being. In this article, we work with two intertwined paradigms (i.e., worldviews) that underlie our social and ecological systems. We describe these paradigms, their historical roots, and their differing connections to clusters of system science concepts. We then explain how we are using these paradigms and system science concepts as the basis of an evaluation design process to support systems transformation toward a just and ecologically healthy civilization. The connection of this evaluation orientation to the decolonization and Indigenization of our social and ecological systems is woven throughout the article