Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Using Visual Data in Qualitative Research : (2008) Marcus Banks. Los Angeles, CA: Sage.
Get PDFA review of the book Using Visual Data in Qualitative Research by Marcus Banks, published in 2008 by Sage
The World of Evaluation: Challenges Faced by Student Evaluators
Get PDFBackground: Performing a high profile evaluation in a world-class organization is a daunting experience for any professional program evaluator. As a student evaluator, it is more than just formidable it has distinctive challenges. Fortunately, professional undertakings provide student evaluators with the experience and tools to overcome these early tests with continuing practice.
Purpose: This paper discusses the challenges that student evaluators face in performing their first program evaluation project. It will draw from the experience of one student’s first major evaluation project and current, but limited, research on the subject.
Setting: N/A
Intervention: NA
Research Design: This paper will examine the broad-spectrum of challenges that student evaluators experience in their first assignment referencing as a case study an actual evaluation of a hospital risk-assessment program implementation.
Data Collection and Analysis: Literature review and documented evaluator experiences.
Findings: This paper will conclude with a discussion of possible mitigation strategies to overcome these student evaluator challenges
Measuring Agency-Level Results: Lessons Learned from Catholic Relief Services’ Beneficiary and Service Delivery Indicators Initiative
Get PDFBackground: Many NGOs have less success documenting their results at the agency level than at the program or project level. Little has been published on the challenges NGOs face in developing and measuring agency-level results. To address this issue, InterAction, an alliance of NGOs, commissioned a comparative study that drew on the existing grey literature, and a sample of 17 InterAction member organizations through case studies and interviews.
Purpose: This paper builds on that InterAction study by presenting one of the first published case studies of a successful agency-level measurement (ALM) system – Catholic Relief Services’ (CRS’) Beneficiary and Service Delivery Indicators (BSDI) initiative.
Setting: A faith-based multi-national relief and development NGO.
Intervention: N/A
Research Design: A case study approach was used to describe and document the development of the CRS ALM.
Data Collection and Analysis: The information in this study is derived primarily from CRS files and documents. Data reflecting ALM practices in other NGOs were derived from the 17 InterAction member NGOs. Data reflecting the ALM practices developed by specific NGOs and presented in tabular form in the paper were derived from official documents published by those NGOs.
Findings: The authors discuss key lessons for other large and small organizations to consider when developing their own ALM systems
Who Needs Goals? A Case Study of a Goal-Free Evaluation
Get PDFBackground: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without knowledge of or reference to predetermined goals and objectives whereas the goal-based evaluator determines merit according to the evaluand’s goal achievement.
Purpose: To examine a GFE in actual practice focusing on its operationalization as well as paying particular attention to the evaluation users’ perspective of its utility.
Setting: The evaluand was a day long training of summer camp counselors on occupational therapy (OT) related skills such as feeding, dressing, and bathing.
Intervention: GFE was the intervention however for comparison purposes an independent and simultaneous goal-based evaluation (GBE) also evaluated the evaluand.
Research Design: Case study.
Data Collection and Analysis: After the evaluation users read both the GBE and GFE reports, data collection consisted of a semantic differential questionnaires followed by a focus group. Additionally, the research team analyzed both GBE and GFE reports for relevant themes.
Findings: The evaluation users reported a slightly more positive attitude toward the GFE report on the semantic differential yet many focus group respondents stated that they found the GBE report more useful or perceived no difference between the two. Evaluation users reported the benefits of GFE to include its potential for developing or aligning goals, expanding the pool of potential outcomes, and supplementing GBE strategies
Failing Forward Quickly as a Developmental Evaluator: Lessons from Year One of the LiveWell Kershaw Journey
Get PDFBackground: Learning to be a developmental evaluator is challenging because it is relatively new and sparsely documented in the scholarly literature. Developmental evaluation is intended to support the ever-changing and adaptive nature of complex environments. In a developmental, systems-oriented evaluation framework, the evaluator is embedded in the process to support and generate learnings from ongoing findings.
Purpose: This article presents the lessons learned through one case study and how the concept of failing forward can guide the evaluator’s reflective process through a developmental evaluation.
Setting: Free clinic and community setting in Kershaw County, South Carolina
Intervention: NA
Research Design: NA
Data Collection and Analysis: The Evaluator documented mistakes and lessons learned during the beginning, planning and implementation stages of a complex community health initiative.
Findings: The evaluation team shares five mistakes made along during the journey and lessons learned. It’s important for teams to understand what differentiates developmental evaluation from other types of evaluation and the role of the evaluator. The “critical friend” boundary can be easily crossed and the evaluator needs to have a strong understanding of the needs of the initiative
Predicting Research Productivity in International Evaluation Journals across Countries
Get PDFBackground: Progress in evaluation research depends on the continuous generation of scholarly knowledge and its dissemination in the community. One way of disseminating findings is to publish in scientific journals and researchers, institutions, and even whole countries are assessed by their output in these journals. Particularly with regard to countries, there is an uneven distribution of research productivity in evaluation journals. A viable model for predicting countries’ research output in international evaluation journals, however, has not yet been developed.
Purpose: The purpose of this study was to develop and test a model for the prediction of countries’ research output in international evaluation journals by predictors from the research, economic, and social/political system.
Setting: NA
Intervention: NA
Research Design: A cross-sectional design was used for predicting research output in evaluation journals across countries.
Data Collection and Analysis: Our sample consists of 65 countries that made contributions to ten international peer-reviewed evaluation journals. We collected data for the period from 2009 to 2013 and predicted the number of authorships across countries by using boosted regression trees, a machine learning procedure.
Findings: Our model provided accurate predictions of countries’ research output. Research productivity in the social sciences had the strongest effect, followed by economic prosperity, control of corruption, and age of evaluation society. The model was generalizable to another period of time with only marginal loss in predictive accuracy
Using Graphical Perception Principles to Improve the ST Tools’ Data Visualization: Revisiting the Systems Dynamics Model
No full textBackground: Systems Thinking (ST) is the new paradigm in Evaluation. It represents a significant mind-set shift for the evaluation field and it is a powerful tool to tackle complex environments.
Heir to the systems concepts of the engineering field, and especially regarding the hard systems tools, ST in evaluation uses the same visual tools that were created many years ago. All these tools already incorporate data visualization features: they depict ideas, relationships and concepts relying in shapes and figures more than a textual explanation.
Revisiting these tools and applying the latest data visualization principles, they could be optimised in order to provide with more information within the same concept.
Purpose: To provide ST practitioners with more informative tools in order to facilitate:
- ST experts and users can optimise the application of the tools to real life models beyond the initial set up of their visual representations.
- Audiences of evaluations using ST as part of the toolkit can find the outputs more apprehensible and easy to understand.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Improving ST representations of reality and systems can help both enhance ST applications and make it more accessible and comprehensible for evaluation practitioners' and users.
Six ways for improving the understanding of the current stock and flow diagrams were identified. The tools proposed consist of: customizing the colours and shape of the variables and their relationships to make them more informative; highlighting the existing subsystems within the model; and providing the specific sequence for reading the main causal chains
Debate on the Appropriate Methods for Conducting Impact Evaluation of Programs within the Development Context
Get PDFBackground: Donors and decision-makers use impact evaluation reports to assess the effectiveness of development programs and identify ways to improve the design and implementation of projects, programs, and policies in developing countries.
Purpose: This paper will explore the existing published impact evaluation literature on development programs and provide an overview of the types of approaches and methods that are being used to conduct impact evaluations.
Setting: NA
Intervention: NA
Research Design: The paper will examine published program evaluation literature in order to shed light on issues related to appropriate methods for impact evaluations of development programs.
Data Collection and Analysis: Literature review.
Findings: The paper will conclude by suggesting a list of approaches and methods that can be used to conduct impact evaluations of programs within the development context
Adaptation and Use of a Five-Task Model for Evaluability Assessment
Get PDFBackground: Over four decades have passed since the concept of evaluability was introduced; however, the availability and accessibility of methodology, frameworks, checklists and other guidance on evaluability assessments remains limited (Smith, 2005). Evaluators who wish to conduct an evaluability assessment must adopt one of few existing models or operate without guidance. This case provides an example of one model and how it was utilized to conduct an evaluability assessment of an ongoing program intervention.
Purpose: This article provides a real-world example of how an evaluability assessment was conducted using a five-task model from the criminal justice field.
Setting: A criminal justice program operating with at-risk adolescent youth and police officers who patrol their neighborhoods.
Intervention: NA
Research Design: NA
Data Collection and Analysis: The original evaluability assessment collected data through interviews with program personnel, observations by the evaluability researchers, and analysis of survey data provided by the program (i.e., program participants’ responses to survey questions about the program).
Findings: The evaluability assessment conducted in this case study example identified multiple changes necessary for the program to be evaluable. Technical assistance was provided in order to support the efforts of the program to prepare for evaluation, though, the evaluability assessment identified risks and costs for the program to consider prior to proceeding. The findings were developed through a five-task process, which evaluators may be able to adapt for use with other types of programs undergoing evaluability assessment
Enhancing the Strategic Management Process Through the Use of Professional Evaluation Methods and the Logic of Evaluation
Get PDFBackground: The evaluation discipline has the potential to enhance other disciplines by integrating evaluation specific methodologies and logic into processes within other disciplines and improving/strengthening the manner in which evaluation of processes, programs and policies are carried out.
Purpose: This paper will highlight the evaluative nature of one of the most popular strategic management models (SMM) in corporate America, namely, Fred David’s (2013) SMM, and examine how professional evaluation logic and methodology can be used to improve David's SMM.
Setting: NA
Intervention: NA
Research Design: The paper will make a comparative analysis between Michael Scriven's Key Evaluation Checklist (KEC) and David's SMM and highlight similarities and differences between the two models.
Data Collection and Analysis: NA
Findings: The paper will conclude by providing several suggestions to enhance and strengthen David's SMM to make it more robust, valuable, and useful in the business environment