Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
What’s in a Scriven Number?
Get PDFBackground: Other than Leonhard Euler, one of the greatest physicists and mathematicians ever to have lived and dubbed “…c'est notre maître à tous” or “…the master of us all” by the French mathematician and astronomer Pierre-Simon Laplace, Paul Erdös was a mathematician who published more papers in mathematics than anyone else (Hoffman, 1998). Because of this feat, friends and colleagues created what has been known as an Erdös number. An Erdös number describes a person’s degree of separation from Erdös through collaborations directly with him or with others who have collaborated with him.
Purpose: In this paper, we present reasoning similar to that of an Erdös number for Michael Scriven who is widely considered to be one of the leading theorists in evaluation. With his numerous publications and influence on the theory and practice of evaluation, we present not only the hope, but also the need for a Scriven number.
Setting: Not applicable.
Intervention: Scriven number.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: A collaborative count such as a Scriven number would centralize evaluation. A Scriven number would form new connections and collaborations, thus yielding a robust connectivity
Conducting Democratic Evaluations where Democratic Principles are not always Practiced
Get PDFBackground: Turkey is a fast-developing country facing a lot of different problems. One of the problems is inadequate education. A large project was started to improve Turkish elementary education. The project includes many different purposes from physical refurbishment of school buildings, to increasing technology use in education. The author of this paper carried out a democratic study to evaluate this project. This article reports on what was learned from this evaluation study conducted in Turkey.
Purpose: The purpose of this article is to examine how democratic evaluations can be used to help understand the complex realities where undemocratic ideology has a long history.
Setting: This evaluation study was conducted in an elementary school from a lower-middle class neighborhood.
Intervention: Not applicable.
Research Design: Qualitative method of inquiry was applied.
Data Collection and Analysis: The data mostly came from the interviews. However document analysis and observations were also conducted. Qualitative data from the interviews and observations were analyzed to interpret meaningful patterns or themes.
Findings: It was concluded that democratic evaluations can serve an important mission by informing the public when the public’s role is limited in the decision making process. Further, democratic evaluators can serve as middle men who transfer information among the stakeholders, helping the evaluated program’s or project’s success. Finally, democratic evaluations can help increase democracy where democratic values are not always obeyed
Stakeholder Involvement in Evaluation: Three Decades of the American Journal of Evaluation
Get PDFBackground: Stakeholder involvement in various phases of evaluation has received increasing attention over the past three decades. Indeed, the American Journal of Evaluation (AJE) has reflected this overall philosophy and strategy through a number of publications about several theoretical frameworks and practical applications. Those AJE articles outline the primary assumptions of the stakeholder approaches to evaluation, their practical applications, constraints, and benefits for providing a new direction in evaluation.
Purpose: What lessons are there for our field concerning the way that stakeholder involvement has been conceptualized and applied in AJE? This article focuses on the ways in which AJE authors have approached the notion of stakeholder involvement over the past three decades, drawing on key articles from scholarship on this topic.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: This review shows how AJE has enhanced our understanding of the evolution of these approaches to evaluation, from a global perspective to differentiated approaches with a shared theme
Evaluating Community Outreach Efforts: A Framework and Approach Based on a National Mental Health Demonstration Project
Get PDFBackground: Community health efforts often include outreach activities designed to increase awareness and ensure uptake of services or programs. Yet, few comprehensive outreach evaluations exist, particularly those designed to improve access to and use of mental health services.
Purpose: This article summarizes the use of two established evaluation frameworks and details an approach to assessing outreach that may have broad appeal to administrators, social workers, health educators, community organizers, and others interested in exploring the results of their efforts.
Setting: Multi-site national study conducted in five states
Intervention: Community outreach
Research Design: Two existing published frameworks are applied to assess community outreach. The evaluation design included three components: 1) process, 2) outcomes and 3) context.
Data Collection and Analysis: Qualitative data were based on focus groups and key informant interviews. Quantitative data were gathered through evaluation surveys, tracking forms and other outreach protocols. The findings focus on lessons learned that may have applicability to others interested in evaluating community outreach efforts in areas beyond mental health
The Handbook of Scholarly Writing and Publishing
Get PDFA review of the book The Handbook of Scholarly Writing and Publishing edited by Tonette S. Rocco and Timothy Gary Hatcher with a forward written by John W. Creswell, published in 2011 by Jossey-Bass.
Evaluation Roots: An International Perspective
Get PDFBackground: The second edition of Evaluation Roots has expanded to more global coverage, but no chapter emerged on development theory in low and middle income countries (LMICs).
Purpose: The purpose of this article is initiate a conversation on filling this gap.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: Two important distinctions emerged in evaluation roots in LMICs. The first is that much evaluation fits effectively on the Theory Tree as it is presented, with perhaps some nuance, but we see a collective rather than individual origin of the evaluation theory writings we have uncovered. The second is the nature of the formal evaluation work in LMICs; its practical origins are not yet formalized into full prescriptive theory. The paper notes the prominence of external actors (such as the donor community) in formalizing evaluation practice, while at the same time observes important threads are developing in different regions of the world. This paper proposes a way to thinking about evaluation in LMICs that is based on practice. The paper suggests a need for evaluation analysts, especially those in LMICs, to build a more comprehensive knowledge and documentation on development evaluation and, more broadly on building the field of evaluation in LMICs
A Lesson in Carefully Managing Resources: A Case Study from an Evaluation of a Music Education Program
Get PDFBackground: A music education program with a goal of enhancing cognitive development of preschool-aged children enrolled in local preschools is evaluated by The Evaluation Center at Western Michigan University. The budget for the evaluation was small, and therefore presented several challenges to the evaluation team.
Purpose: Through a case study of a local education program, the authors explore the challenges and possible solutions of evaluating a program within time, budget, data, and political constraints.
Setting: The case study of a local music education program is implemented in a medium size Midwestern town.
Intervention: The local music education program is a participatory music program for preschool-aged children. The program provides a combination of music classes, performances, and parent education to all participants, with further musical opportunities for interested parents and children.
Research Design: The research and data collection portion of the evaluation was conducted using participatory strategies, and involved a management oriented approach.
Data Collection and Analysis: Data collection for years one and two (process) consisted of primarily qualitative methods, triangulated by exploratory quantitative data collection. Data collection for year three (impact) will consist of primarily quantitative methods, triangulated with results from the qualitative analysis done in years one and two.
Findings: Four factors, time, data, budget, and political, constrain the design and conduct of a local evaluation, yet, evaluators may implement multiple solutions that lessen the effect of the constraints on the evaluation
Assessing Implementation Integrity of a National Nutrition Education Program: A Case Study of Share Our Strength's Operation Frontline
Get PDFBackground: Treatment implementation is not just one thing but rather is a multifaceted process that includes treatment delivery, treatment receipt, and treatment adherence. As such, local variations in implementation and service delivery of interventions are an inevitable.
Purpose: To assess implementation fidelity of a multi-site experiential nutrition education program.
Setting: Multiple sites throughout the continental United States.
Intervention: An experiential nutrition education program.
Research Design: A concurrent mixed methods design was used to assess implementation fidelity.
Data Collection and Analysis: Multiple methods of data collection and analysis were used including observations, interviews, survey questionnaires, and extant data.
Findings: Although implementation fidelity varied over program sites, overall implementation fidelity was very good and when it varied, it varied to local site needs and context
‘Policing Schools’ Strategies: A Review of the Evaluation Evidence
Get PDFBackground: Schools experience a wide range of crime and disorder, victimizing students and staff, and undermining attempts to create a safe and orderly environment for student learning. Police have long established programs with schools, but there has been no systematic review of evaluations of these programs, outside of police-led prevention classroom curriculum programs such as D.A.R.E.
Purpose: This paper documents a systematic search to identify experimental and quasi-experimental evaluations that assess the effectiveness of non-educational policing strategies and programs in schools.
Setting: Included studies took place in or around K-12 schools in the United States, Canada, and the United Kingdom.
Intervention: Studies were included if they reported on a specific school-based strategy that heavily involved police and did not exclusively involve the police teaching a curriculum or program such as Drug Abuse Resistance Education (D.A.R.E.).
Research Design: Systematic review of experimental or quasi-experimental evaluations
Data Collection and Analysis: Only those impact studies that used experimental or quasi-experimental design, had at least one outcome measure of school crime or disorder, and were available through December 2009 were eligible. Electronic searches and other methods were used to identify published and unpublished evaluation reports.
Findings: The searches identified a total of eleven quasi-experimental studies. Ten of the eleven studies would likely have received a “3” on the Maryland Scientific Methods Rating Scale, a common approach to classifying studies on the basis of internal validity. If evidence rating criteria from the U.S. Department of Education’s What Works Clearinghouse (WWC) were applied, only one study would likely receive a grade of “Level 2” evidence (acceptable with reservations) and the other ten studies would likely not meet WWC evidence screening criteria.
Thinking Beyond Measurement, Description and Judgement: Fourth Generation Evaluation in Family-Centered Pediatric Healthcare Organizations
Get PDFBackground: Although pediatric healthcare organizations have widely implemented the philosophy of family-centered care (FCC), evaluators and health professionals have not explored how to preserve the philosophy of FCC in evaluation processes. Purpose: To illustrate how fourth generation evaluation, in theory, could facilitate collaboration between evaluators and families and uphold the philosophy of FCC in evaluation. Exploration focuses on describing the ways in which fourth generation evaluation is consistent with FCC and outlining a strategy for implementing it within pediatric healthcare organizations. Discussion: Current evaluation practices used in healthcare organizations reflect what some describe as the first three generations of evaluation: measurement-, descriptive-, and judgment-oriented evaluation. While these generations encourage evaluators and health professionals to use systematic and rigorous approaches and techniques, they negate opportunities to explore issues that may surface in more flexible evaluation processes and do little to promote FCC in evaluation. Fourth generation evaluation is based on the constructivist paradigm, and its hermeneutic dialectic process moves beyond these generations, as well as the problems associated with them, to reflect the FCC notions of family participation, partnership, collaboration, respect, and joint decision-making.
Conclusion: The collaborative and dialogue-oriented environment of pediatric healthcare organizations provides an ideal context for fourth generation evaluation. Although this evaluation approach is consistent with the philosophy of FCC, more research is required to understand the strengths and limitations of using it in these organizations