Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Emergent Developmental Evaluation Developments
Get PDFBackground: Developmental Evaluation (DE) continues to develop. After reviewing the niche and history of DE, this article identifies five new and emergent purposes for DE to add to the original five – and forecasts future developments for diverse, innovative, and adaptive DE applications. An extended example of DE in supporting pivoting in the pandemic illustrates DE as adaptation to crisis as distinct from supporting social innovation.
Purpose: To update DE.
Setting: Global.
Data Collection and Analysis: Not applicable.
Findings: The applications and uses of DE have expanded beyond the original framing
Cultural Humility: A Collaborative Approach to Recruiting Patients with Deliberate Self-Harm into a Multi-Hospital Randomized Controlled Trial
Get PDFObjectives: The ‘SMS SOS’ Deliberate Self-Harm (DSH) Aftercare Study was conducted in Western Sydney, Australia (October 2017 to December 2020) across three large public hospitals. During this randomized controlled trial (RCT), it was observed that knowledge exchange between key stakeholders and their ‘cultural’ perspectives (for example, Mental Health Clinicians, Lived Experience Mental Health Consultants—Patient Representatives, Administrative Officers, and Researchers) was essential to effective recruitment of patients experiencing DSH. Knowledge exchange within and between cultural groups was maximised and assessed using a communication matrix. This process, transferable to other trials engaging multiple ‘cultures’, aimed to promote the early identification of wider-team strengths as well as active management of emergent issues that would otherwise impede patient recruitment, and to maximise funding and human resources.
Methods: A descriptive study was conducted with a convenience sample of team members who represented different cultures in the study. Qualitative data were elicited from a ‘know and tell’ matrix. Through an iterative process, themes were generated that encapsulated what team members needed to know from and tell to their colleagues concerning the study.
Results: Factors that impacted participation in the study included clinician workload, the level of motivation/ commitment/confidence of clinicians to recruit patients, clinician-patient engagement, perception and expectations of study involvement, inter-cultural communication, and clinician training and support. The findings of this multidisciplinary consultation informed a composite model of knowledge exchange and the development of educational briefing/ orientation modules that make explicit team members’ roles and responsibilities to foster group member participation and enhance patient recruitment.
Conclusions: It is incumbent upon multidisciplinary team members of large-scale studies to adopt a similar ‘knowledge exchange’ strategy early in the planning and design stage. Adoption of such a strategy has the potential to mitigate risk of delay in project timelines, improve project outcomes, and ensure the efficient use of research funding, particularly in newly established research teams within clinical settings and with members newer to formal research collaborations
Measuring Mental Health Literacy: Development of the Mental Health Awareness and Advocacy Assessment Tool
Get PDFBackground: Mental health literacy programs are a common community-based approach used to address the prevention of mental health issues on college campuses. Current assessment strategies used to evaluate the effectiveness of these programs often lack strong theoretical rational and psychometric rigor.
Purpose: The purpose of this study was twofold. First, based upon extant literature, theory, and standard clinical practice, we propose a process-based model of mental health literacy that includes three macro factors—identifying mental health issues, locating empirically based resources, and responding to mental health issues—and three micro processes of how they unfold—acquiring knowledge, building self-efficacy, and applying skills (behavior). The second aim was to test the psychometric properties of a new tool created to evaluate this process-based model—the Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT).
Setting: Not applicable.
Intervention: Not applicable.
Research Design: A national sample of 296 college attending participants were recruited from Amazon’s Mechanical Turk. Participants responded to a demographic questionnaire and the newly developed MHAA-AT. Psychometric properties were examined through item response theory, exploratory factor analyses, and bivariate correlations.
Findings: Results suggest the MHAA-AT is a sound measure and demonstrates appropriate item, person, and trait characteristics on declarative knowledge items, and single factor structures on self-efficacy and behavior items with moderate to high reliability and validity. While additional testing is need among other samples, results suggest that the MHAA-AT is a quality assessment tool
Evaluating the Content of Palestinian Curricula in Light of the Sustainable Development Goals 2030
Get PDFBackground: Curriculum evaluation is a dynamic and indispensable process necessary to develop the curriculum, and to support decision-makers with evidences to guide the curriculum towards the intended goals. The Sustainable Development Goals (SDGs) are the most important universal goals to be taken into consideration by curriculum. SDGs represent a call to end poverty, protect the planet and ensure that all people enjoy peace and prosperity by 2030. Because education, through curriculum, is the nations’ key to accelerate the achievement of sustainable development, this study evaluates the content of the Palestinian curriculum and assesses the degree to which SDGs are included in the curriculum.
Purpose: The study aimed to evaluate the Palestinian school curricula, through using national standards for each SDG, to examine the degree to which the SDGs are incorporated in the curricula, and to know whether there is variation in this incorporation among the different curricular subjects.
Setting: The study evaluates the content of Palestinian curricula of the main subjects based on national curriculum standards developed and published in light of the SDGs. These subjects are: Arabic Language for grades 1-12, Science for grades 1-12, Mathematics for grades 1-12, Social Studies for grades 1-12, Technology for grades 5-12, Agricultural Sciences for grades 11-12, Renewable Energy for grades 11-12, Entrepreneurship and Business for grades 11-12, Management and Economy for grades 11-12, and Smart Buildings for grades 11-12. In addition, timeframe delimit is the academic year 2018-2019.
Data Collection and Analysis: In order to evaluate content of the Palestinian curricula, descriptive-analytical methodology was used by utilizing content analysis of the guideline document for each curricular subject.
Findings: The results showed variation in the inclusion of the SDGs, and absence of essential aspects. The fourth SDG (Quality Education) obtained the highest inclusion percentage with 28.5%. While the fourteenth SDG (Life below Water) obtained the lowest inclusion percentage with 0.8%. In light of the results, the study highlighted variation of the curriculum from SDGs and recommended for the development and enrichment of the Palestinian curriculum to ensure the inclusion of SDGs, with its all dimensions, considering that education in Palestine defined as the main gateway towards progressing achievement of the SDGs
Beyond GDP: Tracking and Evaluating National Contributions to Social and Environmental Sustainability
Get PDFBackground: The Gross Domestic Product (GDP) emerged as a convenient measure of national economic activity during the Great Depression. It was subsequently adopted by international development economists to track developing countries’ progress so that, despite its severe deficiencies, it became ‘locked in’ by habit, convenience, and policy makers’ preferences.
Purpose: This article conceives of GDP as a social intervention fit for evaluation. It shows that the GDP has had a pervasive and pernicious influence on policy making. Since past strategies aimed at dethroning the GDP have failed, it proposes new, evaluator-driven approaches designed to undermine the GDP’s dominance in the global market economy.
Setting: The Stiglitz report commissioned in the wake of the 2008 financial crisis launched a ‘Beyond GDP’ movement. Since then, public alarm about the GDP growth addiction has escalated: the drawbacks of GDP as a free-market policy tool have become self-evident as the rich get richer, the ranks of the poor swell and the future of the planet hangs in the balance.
Research Design: Not applicable.
Data Collection and Analysis: For the twenty largest economies in the world, the article estimates climate change discounts to the GDP based on official CO2 emissions statistics and a social cost of carbon estimate derived from a 2015 survey of eminent climatologists. It also draws on composite indexes generated by four reputable social research organizations to rank countries for their contributions to the 5 Ps of the Sustainable Development Goals (SDGs): people, planet, prosperity, peace, and partnership.
Findings: Pending the results of on-going efforts to upgrade worldwide statistics focused on the 169 SDG targets, the proposed GDP discounts help track progress towards the SDGs. But monitoring is not enough. In a policy world dominated by vested interests, the new ‘Beyond GDP’ indicators should be combined with principled, evaluator-directed evaluations
Evaluating Environment in International Development (2nd ed.)
Get PDFA review of the book Evaluating Environment in International Development (2nd edition) edited by Juha I. Uitto, published in 2021 by Routledge
Developmental Evaluation in Theory versus Practice: Lessons from Three Developmental Evaluation Pilots
Get PDFBackground. Developmental Evaluation (DE) practitioners turn to DE theory to make design and implementation decisions. However, DE practitioners can experience difficulty in fully understanding how to implement DE using theory because it is method agnostic (Patton, 2016). Instead, DE is a principle-based approach.
Purpose. This article presents an empirical examination of how DE theory was (or was not) applied during three DE pilots. Our analysis aims to better understand how DE theory is used in practice to expand the evidence base and strengthen future DE implementation.
Setting. A consortium of three organizations implemented three DE pilots through the United States Agency for International Development (USAID) from November 2016 to September 2019. The authors—who participated in the consortium—did not implement the DEs but instead conducted a study or meta-evaluation across the DE pilots.
Data Collection and Analysis. This article focuses on the results of an ex post facto analysis of three DE pilots based on the entire DE implementation experience. For each DE studied, we used mixed methods to collect data on the effectiveness of the DE approach, to identify adaptations to strengthen DE implementation in the USAID context, and to measure its value to stakeholders. Data included more than 100 hours of interviews, 465 pages of qualitative data, and 30 surveys completed by DE participants.
Findings. We find that the ability to apply the DE principles in practice is influenced, in no particular order, by DE participant buy-in to the DE, the Developmental Evaluator’s aptitude, support and resources available to the Developmental Evaluator, and the number of DE participants. We also find that buy-in can change and this should be closely monitored throughout a DE to inform whether a DE should be paused or prematurely ended
Dialogic and Generative Reflection: An Application of the Critical Appreciative Process in Program Evaluation
Get PDFBackground: Even though the positive potential of reflective practice is widely acknowledged across professional fields, it has been recognized that reflective practice may be carried out primarily as an individual-based exercise, and at the technical or descriptive level without generative impact. Dialogic reflective processes involving both evaluators and program directors are far from being systematically implemented or examined.
Purpose: The purpose of this article is to share our experiences engaging in dialogic and generative reflections as the project director and program evaluators of a K-12 teacher education program using the critical appreciative process. Building upon the reflective practice traditions in both disciplinary areas, we introduce the use of the critical appreciative process as a promising model to guide dialogic and generative reflection to support the co-design and improvement of the program and accompanying evaluation efforts.
Setting: The project director and evaluators are engaged in a grant-funded teacher preparation project designed to prepare teachers for K-12 English learners and dual language learners. The project builds upon partnerships between the university teacher preparation program and two local school districts. The evaluation plan was designed based on culturally responsive, collaborative, and use-focused evaluation approaches and theory. In 2020, the project team faced critical decisions in the context of the COVID-19 pandemic.
Intervention: Not applicable.
Research Design: We applied self-study methodology to guide data collection and analysis in this study. The primary data source included individual written reflections and group critical friend dialogues guided by the critical appreciative process. Both the reflections and meeting notes were analyzed to identify convergent and divergent perspectives shared throughout the critical appreciative process and to highlight implications for both the evaluation and the program moving forward.
Findings: Convergent and divergent perspectives from both the project director and the evaluators were shared based on the 4-D critical appreciative process: Discover, Dream, Design, and Deliver. Based on this shared experience, we illustrate how the dialogic reflective process entails reflexivity and requires pausing; how reflective practice in program evaluation situates our dialogues as learning-oriented rather than a mere accountability discussion; and how reflective action can create a dialogic and generative virtuous cycle
Choosing an Evaluation Theory: A Supplement to Evaluation Roots (3rd Edition)
Get PDFBackground: Unlike scientific theories, evaluation theories are prescriptive: a set of actions and approaches that should be followed when conducting an evaluation. While evaluation theorists have offered a variety of writings describing their theories and approaches, few have offered a specific outline of what the theory looks like in practice. Thus, Alkin and Christie formulated a book to aid evaluators in how to apply theories in evaluations (Alkin & Christie, forthcoming). This book culminates in a series of prototypes that outline each theory’s goals, appropriate contexts, prescriptions, and observable actions in application.
Purpose: In order to aid evaluators in applying theories, this article seeks to provide a basis for comparison that can be used to help evaluators select which theory would be most appropriate in their practice.
Setting: This comparison can be applied in any setting where evaluations fit the context prescribed by each of the theories.
Intervention: Not applicable.
Research Design: Not applicablre.
Data Collection and Analysis: Not applicable.
Findings: In order for theories to influence practice effectively, theories must be displayed in a way that allows for easy comparison. This comparison of three theory prototypes demonstrates that prototypes can be an effective way for selecting a prescriptive theory when conducting an evaluation
Use of Geographic Information Systems by American Evaluation Association Members in their Professional Practice
Get PDFBackground: As geographic information systems (GIS) technology continues to develop and expand in its capacity and applications, it is becoming increasingly useful to many disciplines. Even so, little has been written about the place of GIS technology in evaluation practice, and there is a paucity of information as to the extent to and applications for which evaluation practitioners use such technology.
Purpose: In this investigation, the prevalence and common applications of GIS technology in professional evaluation practice are examined. Particularly, the study was designed to estimate what proportion of American Evaluation Association (AEA) members who self-identify as evaluation practitioners use GIS in their practice, if at all, and, if so, to what extent. For those who use GIS in their evaluation practice, the specific GIS software packages and applications used also are explored.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: A simple random sample of American Evaluation Association (AEA) members were surveyed, with an emphasis on evaluation practitioners.
Findings: Less than less than half (41.04% ±6.09%) of AEA members who consider themselves evaluation practitioners have ever used GIS in their evaluation practice and less than one-third (31.47% ±5.75%) have received some form of training in GIS methods. Data visualization is, by far, the most frequent application of GIS in evaluation practice