Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Evaluation Essentials: From A to Z (2nd ed.)
Get PDFA review of the book Evaluation Essentials: From A to Z (2nd ed.) by Marvin C. Alkin and Anne T. Vo, published in 2018 by Guilford Press
The Impact of the Safe and Successful Youth Initiative (SSYI) on City-Level Youth Crime Victimization Rates
Get PDFBackground: The physical, emotional, and financial costs resulting from youth violence are well documented.
Purpose: This article summarizes the results of a quasi-experimental evaluation study to test a youth violence intervention program in eleven cities in Massachusetts.
Setting: In 2011, Massachusetts initiated the Safe and Successful Youth Initiative (SSYI), which provides a comprehensive public health approach for young men believed to be at “proven risk” for being involved with firearms.
Intervention: The SSYI program components include: (1) Specific identification of young men, ages 14-24, at highest risk for being involved in firearms violence; (2) Use of street outreach workers to find these young men, assess their needs, and act as brokers for services; (3) The provision of a continuum of comprehensive services including education, employment, and intensive supervision. Eleven cities with the highest count of violent offenses reported to the police in 2010 were selected for SSYI funding in 2011 and began implementing the program.
Research Design: Short-interrupted time series design with a comparison group. The observed and predicted trends in monthly violent victimization rates for the 11 SSYI cities were compared to the next 23 cities (as they ranked in reported violent crime in 2010).
Data Collection and Analysis: Using police incident data, researchers examined SSYI's impact on monthly city level violent crime, aggravated assault and homicide rates for persons ages 14-24.
Findings: Results indicated that SSYI had a statistically significant and positive impact on reducing the number of victims of violent crimes, aggravated assaults, and homicides per month that were reported to the police. A city with SSYI has approximately 60 fewer victims of violence each year, ages 14-24, per 100,000 citizens over the post-intervention period
A Mechanism-Centred Approach to Evaluating Complex Aid Interventions: The Case of Accompanying Measures to Budget Support
Get PDFBackground: Current methodological debates related to theory-based evaluations (TBE) centre around questions how to improve the explanatory strength of these approaches and how to integrate mechanisms as analytic concept. Particularly in complex aid interventions, when multiple elements are expected to interact and thus create an added value, exploring mechanisms as an analytical tool can be promising.
Purpose: This paper seeks to contribute to the discussion on the use of TBE for evaluating complex aid interventions by sharing experiences from a recent evaluation of accompanying measures to general budget support.
Setting: Nine countries of sub-Saharan Africa which have received German budget support, namely Burkina Faso, Ghana, Malawi, Mali, Mozambique, Rwanda, Tanzania, Uganda, Zambia.
Intervention: Accompanying measures (mainly in the form of technical assistance and capacity development) are one element of the budget support package, which further encompasses financial contributions, policy dialogue, and conditionalities.
Research Design: We focus on interrelations between different elements of budget support and apply a mechanism-centred approach to programme theory building,
Data Collection and Analysis: After defining accompanying measures and integrating them into the intervention logic of budget support used in recent multi-donor evaluations, key mechanisms were identified on an explorative mission to Mozambique, validated in an online survey, and further discussed in expert interviews and during field research in Tanzania.
Findings: For the specific example relating to two elements of budget support (policy dialogue and accompanying measures), some of the hypothesized mechanisms were found to create an added value and thus increase the effectiveness of budget support as a package. The applied approach helped generate a more comprehensive implementation theory and provided insights into potential benefits and challenges of combining different elements in one programme. Beyond its use for future evaluations in the field of budget support, we argue that TBE of complex interventions can benefit from adopting such a mechanism-centred approach to create a better understanding of how different elements of the programme interact. Moreover, the focus on mechanisms when analysing programme implementation enables evaluators to improve their empirical inquiry on the identified mechanisms and to draw valid conclusions on the programme’s contribution to the observed outcomes
Communicating About Evaluation: A Conceptual Model and Case Example
Get PDFBackground: Despite consensus within the evaluation community about what is distinctive about evaluation, confusion among stakeholders and other professions abounds. The evaluation literature describes how those in the social sciences continue to view evaluation as applied social science and part of what they already know how to do, with the implication that no additional training beyond the traditional social sciences is needed. Given the lack of broader understanding of the specialized role of evaluation, the field struggles with how best to communicate about evaluation to stakeholders and other professions.
Purpose: This paper addresses the need to clearly communicate what is distinctive about evaluation to stakeholders and other professions by offering a conceptual tool that can be used in dialogue with others. Specifically, we adapt a personnel evaluation framework to map out what is distinctive about what evaluators know and can do. We then compare this map with the knowledge and skill needed in a related profession (i.e., assessment) in order to reveal how the professions differ.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: We argue that using a conceptual tool such as the one presented in this paper with comparative case examples would clarify for outsiders the distinct work of evaluators. Additionally, we explain how this conceptual tool is flexible and could be extended by evaluation practitioners in a myriad of ways
Examining Factors Impacting Online Survey Response Rates in Educational Research: Perceptions of Graduate Students
Get PDFBackground: In educational research, online survey has become one of the most popular methods of data collection. Academic researchers, including faculty and students, expect and require a good response rate to their research projects for reliable results.
Purpose: In this paper, the authors examine a wide range of factors related to survey response rates in academic research. Examples include email checking habits, survey design, and attitudes toward research.
Setting: An online survey environment
Intervention: Not applicable.
Research Design: A cross-sectional quantitative research method was used to analyze the factors that influence participants’ email survey response rate. Data were collected at a single point in time. The authors did not directly measure changes that come over time in this study.
Data Collection and Analysis: After receiving the Institutional Research Board’s approval, the researchers distributed the survey via the American Educational Research Association (AERA) Graduate Student Discussion List subscribers. A sample of 454 responses was used in the final analysis-- with a 78.9 % response rate. The authors used descriptive statistics (percentage, average mean) and inferential statistics (chi-square and correlations) to report the data analysis and findings.
Findings: Results indicated that research survey response rate was highly influenced by interests of participants, survey structure, communication methods, and assurance of privacy and confidentiality. The findings also suggested that male participants were more likely to respond to surveys if they received a reminder, and older participants were more likely to respond if they were promised a reward
Translating Project Achievements into Strategic Plans: A Case Study in Utilization-Focused Evaluation
Get PDFBackground: Utilization-focused evaluation (UFE) is a decision-making framework intended to design and implement evaluations that get utilized. In this case study, the UFE approach was applied to the evaluation of a youth training and employment program in Kenya.
Purpose: Analyze a case study based on empirical experience with the lens of evaluation use and influence.
Setting: Nairobi and other urban and peri-urban settings, Kenya.
Intervention: The evaluation of a youth training and employment program that provided direct training for marginalized youth as well as capacity building for employment and adapted a Basic Employability Skills Training (BEST) model from India.
Research design: Analysis of a case study to describe: how the evaluation approach provided enabling factors for funders and grantees to turn evaluation into a learning intervention; the benefit of clarifying a project’s theory of change; and learning how to combine summative and developmental evaluation.
Data collection and analysis: A case study based on an evaluation consultancy. The evaluation included site visits, extensive documentation review, qualitative and quantitative data collection.
Findings: The ‘facilitation of use’ (a step in UFE) provided a bridge between a summative and a developmental evaluation. The evaluators and program partners developed a learning relationship wherein the evidence influenced the subsequent project design and strategy
Re-Structuring Evaluation Findings into Useful Knowledge
Get PDFBackground: A long research stream has shown that when knowledge is more structured it is more likely to be effective in practical application. Building on that research, the authors applied Integrative Proposition Analysis to visualize, integrate, and assess the quality and usefulness of knowledge gained from the NMAC (formerly National Minority AIDS Council) Strong Communities evaluation.
Purpose: Demonstrate an innovative method to rigorously integrate and strengthen knowledge gained from evaluation and to encourage discussion of future directions for developing stronger theories for more effective evaluation and more effective action.
Setting: Birmingham, Alabama
Intervention: A project to identify local strategies for community-based organizations and community health centers that serve African American and Latinx gay and bisexual men and transgender women to collaboratively meet HIV-related community needs.
Research Design: The researchers applied Integrative Propositional Analysis to integrate and map concepts and causal connections emerging from the evaluation findings. The authors then analyzed the resulting map to identify top-mentioned concepts, better understood concepts, reinforcing loops, and knowledge gaps.
Data Collection and Analysis: Integrative Propositional Analysis applied to a literature review and stakeholder interview transcripts collected for the evaluation.
Findings: Integrating literature and interview results helped to identify several actions where providers of HIV-related services could increase their impact on combating the HIV epidemic among the communities they serve. The authors also identified a reinforcing loop; this shows opportunity to improve two desired outcomes by increasing one. In addition, the authors identified blank spots on the map; these show where additional research could strengthen the quality and usefulness of the mapped knowledge
You Call This Exemplary? Lessons from an Unsung International Evaluation
Get PDFThis paper reflects on the role of academic discipline and epistemic community in judging what is an exemplary evaluation. It examines the case of an evaluation that was considered ‘exemplary’ by a panel of program evaluators, but methodologically flawed by evaluators from a different evaluation tradition. The evaluation in question was carried out within an international agricultural research network (known as the CGIAR), with a rich tradition of economic impact assessment. The evaluation was carried out by a team of experienced program evaluators, who attempted to apply accepted good practices in the program evaluation community. The evaluation employed mixed methods and multiple data sources with heavy reliance on triangulated perceptual data. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. However, within the CGIAR, both the evaluation and the meta-evaluation study were rejected, as methodologically flawed. The paper closes with four propositions related to what is considered an “exemplary evaluation.”
Background: Program evaluators have reached broad agreement on principles for planning and conducting evaluations and on standards for judging their quality. However, many evaluation stakeholders, including key intended users, may judge evaluations on criteria that differ sharply from the professional standards and the criteria we commonly employ in meta-evaluations.
Purpose: This paper highlights the role of academic discipline and epistemic community in judging what is an “exemplary” evaluation, by examining the case of an evaluation that was considered exemplary by professional program evaluators, but methodologically flawed by professionals from different disciplinary traditions.
Setting: The evaluation in question was carried out within the Consultative Group on International Agricultural Research (CGIAR), a research network with a rich tradition of economic impact assessment.
Intervention: NA
Research Design: This is a case study that combines participatory action research and historical analysis.
Data Collection and Analysis: The study is based on the author’s personal involvement in the evaluation and on a review of publications and unpublished documents related to the case.
Findings: A team of experienced evaluators applied what are generally considered to be good practices in the program evaluation community. A meta-evaluation led by an experienced program evaluator considered the evaluation to be exemplary. In contrast, within the CGIAR, both the evaluation and the meta-evaluation study were considered to be methodologically flawed and biased. Three lessons related to exemplary evaluation are formulated and elaborated upon:
Lesson 1. Being exemplary is in the eyes of the beholder.
Lesson 2. Epistemic communities are hard nuts to crack.
Lesson 3. You can’t win them all.
While the early efforts with program evaluation analyzed here were experienced as failures, a number of subsequent developments have led to greater understanding of diverse evaluation approaches, and some movement toward agreement on what constitutes exemplary evaluation in the CGIAR. Nevertheless, there is still a considerable way to go
Considerations for Evaluating Evolving Organizations and Initiatives
Get PDFBackground: Evaluators are often called to be flexible in response to changing programmatic and contextual circumstances. However, the field offers little guidance around issues to consider before modifying an in-progress evaluation.
Purpose: This article describes the evaluation of an organization that underwent significant mid-evaluation changes, with a focus on factors that went into the evaluator’s recommendations about whether to modify evaluation design and instrumentation.
Setting: Community collaborative in Wake County, North Carolina
Intervention: NA
Research Design: NA
Data Collection and Analysis: The evaluator noted factors that went in to decisions about modifying evaluation design and instrumentation.
Findings: Issues around validity and sustainability push evaluators to update our evaluation designs to keep pace with changes, whereas issues around continuity and accountability pull us to back to the current course
Standards of Excellence from a Leader of Excellence: Honoring Our Professional Heritage and Remembering Daniel L. Stufflebeam
Get PDFPays tribute to Stufflebeam's leadership in developing evaluation standards and their impact on professionalizing the evaluation field