Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
The "Usability" of Evaluation Reports: A Precursor to Evaluation Use in Government Organizations
Get PDFBackground: According to the Treasury Board of Canada’s Policy on Evaluation (2009), evaluations produced by federal government departments must contribute to decision-making at an organizational level (mainly summative) as well as a program level (mainly formative). Previous research shows that although the formative objectives of evaluation are generally reached, the use of evaluation for broader, budgetary management is limited. However, little research has been conducted thus far on this issue.
Purpose: This study investigates the extent to which program evaluation is used in the Canadian federal government for budgetary management purposes.
Setting: This paper outlines the results obtained following the first component of a two-pronged research strategy focusing on evaluation use in Canadian federal government organizations.
Intervention: N/A
Research Design: Two federal agencies were recruited to participate in organizational case studies aiming to identify the factors that facilitate the use of evaluation for budgetary reallocation exercises.
Data Collection and Analysis: This report presents the findings from a detailed analysis of evaluation reports published by both agencies between 2010-2013. The data were collected from public evaluation reports and analyzed using NVivo.
Findings: The preliminary findings of the study show that instrumental use has occurred or can be expected to occur, based on the types of recommendations outlined in the reports reviewed and on the responses to the evaluations produced by program managers
Collaborative Evaluations: Step-by-Step
Get PDFA review of the book Collaborative Evaluations: Step-by-Step (2nd ed.)by Liliana Rodríguez-Campos and Rigoberto Rincones-Gómez, published in 2013 by Stanford University Press
Impact Evaluation Based on Buddhist Principles
Get PDFBackground: This is one in a series of articles in which the authors attempt to relate Eastern philosophy to contemporary programme planning and evaluation.
Setting: Not applicable.
Purpose: The authors examine impact evaluation through the lens of Eastern Philosophy.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The authors examine the basic causal statements from the approach to impact evaluation commonly used by The World Bank and from Buddhist philosophy. Second, they examine the econometric assumptions on which impact evaluation is often based and propose alternative Buddhist principles. Lastly, they speculate what impact evaluation might look like using the alternative principles that were identified.
Findings: Not applicable.
Conclusions: There is no such thing as impact in and of itself. Rather, a combination of conditions comes together in a certain way, at a certain time, and we call it an impact. Impact is, therefore, the result of conditionality (Salzberg, & Goldstein, 2001).
Evaluation that examined the conditionality of impact would be in a position to make statements about patterns of relationships. Instead of experimental and quasi-experimental approaches, the Insight Evaluation approach (Russon and Russon, 2011) might be more appropriate
A Student-Led Methodology for Evaluating Curricular Redundancy
Get PDFBackground: Curricular redundancy can be a significant problem for any educational curriculum. Redundancy can be both desirable and undesirable, but differentiating the two can be quite challenging. Further, pinpointing undesirable redundancy and quantifying it so as to produce an estimate of inefficiency is even more difficult.
Purpose: The purpose of this research is to describe a student-led strategy for evaluating redundancy in a highly integrated medical school curriculum. It is our hope that the methodology presented here will serve as a useful evaluation model for persons attempting similar work in various educational arenas.
Setting: A highly-integrated medical school at a large public university.
Intervention: This research did not require an intervention.
Research Design: We identified two advanced medical students and asked them to identify redundant material across the first two years of the medical school curriculum. The students had to operationalize ‘redundancy’, develop an evaluation plan/framework, and evaluate the extent to which undesirable redundancy was prevalent in the current curriculum.
Data Collection and Analysis: Students reviewed course syllabi, notes, and materials and documented the amount of redundant material they found in the curriculum.
Findings: A total of approximately 167 hours, or 8.35 weeks, could be eliminated from the curriculum; the vast majority of the redundancy occurred as a result of small group activities
Developing Criteria to Identify Transformative Participatory Evaluators
Get PDFBackground: The evaluation discipline continues to evolve as more and more researchers study practice. The research described in this article further defines Transformative Participatory Evaluation (T-PE) by focusing on the key elements that practitioners and theorists agree define this evaluation practice.
Purpose: A multi-stage, mixed-method approach was used to develop and examine a set of statements that serve two purposes: First, they can help identify a subset of participatory practitioners from others and, second, they further theory development by showing how T-PE practitioners differ from other evaluation practitioners on key indicators.
Setting: In the first phase of this research, three prominent evaluation theorists comprised an expert panel to develop a set of statements that would identify T-PE practitioners. The American Evaluation Association membership was used to test the statements in the research’s subsequent phases.
Intervention: NA
Research Design: A multi-stage, mixed-method approach was used to develop and test the statements.
Data Collection and Analysis: The panel was engaged in a web-based wiki to jointly edit the statements; an online questionnaire with mostly closed-ended items was used to gather AEA member input; a unique online modeling software and webinars were used to further understand findings. Analysis of variance was used to assess differences between groups and Rasch modeling and Wald tests were used to analyze the modeling data.
Findings: The eight core statements that emerged had acceptable internal reliability and limited construct validity. Though the statements’ discrimination strength was tenuous, quantitative comparisons of preferred evaluation practice models showed congruence with the predicted underlying philosophies and therefore supports the statements’ ability to discern T-PE evaluators from P-PE evaluators
Successful School Leadership: Linking with Learning and Achievement
Get PDFA review of the book Successful School Leadership: Linking with Learning and Achievement by Christopher Day, Ken Leithwood, Pam Sammons, David Hopkins, Qing Gu, Eleanor Joanne Brown, & Elpida Ahtaridou, published in 2011 by Open University Press
Institutionalization and Use of Evaluations in the Public Sector in Nepal
Get PDFBackground: Institutionalization of evidenced-based policy-making, planning and decision making practices is the pre-condition for timely demand and use of evaluations in the government. Moreover, a number of factors act as enabling environment for proper use of evaluation recommendations in the national policy making and planning processes. Reviewing various evaluation reports and periodic plan documents of Nepal, this article discusses and analyzes factors that determine uses of evaluations.
Purpose: This paper aims to review the institutionalization process and use of evaluation evidences in the planning processes in Nepal.
Setting: NA
Intervention: NA
Research Design: Triangulation of document review and Key Informant Interview (KII) was done.
Data Collection and Analysis: This study reviews evaluation reports of 29 projects or programs of various sectors that conducted engaging independent evaluations during 1995-2012 and medium-term plan documents and some policies. Information generated from the Key Informant Interviews (KII) was used to triangulate the information generated from the documentary review.
Findings: It was found that proper documentation and rigorous analysis is important to promote the use of evaluations. The use of evaluations heavily depends on the independence of the evaluation process itself and quality of reports. Key findings also include: commitment of policy makers and other important actors is important for the demand and use of evaluation; the use of evaluations also depends on the clarity of objectives; and stakeholder participation is important to use evaluations and enhance accountability
Real-Time Evaluation of Humanitarian Assistance Revisited: Lessons Learned and the Way Forward
Get PDFBackground: The real-time evaluation (RTE) approach has been applied in humanitarian assistance for two decades. Its spread seems to be as much a result of entrepreneurial evaluators who aim to demonstrate expertise to potential clients as it is a response to actual demand for immediate feedback of results to decision makers.
Purpose: As RTE has come under scrutiny recently, this study demystifies the concept of RTE and looks beyond textbook descriptions of its advantages in order to understand its practical application in humanitarian action.
Setting: NA
Intervention: NA
Research Design: NA
Data Collection and Analysis: NA
Findings: It then suggests lessons for how to improve the application of real-time evaluations and related concepts in practice.
Bioeconomic Models and the Formative Evaluation of Fisheries-Related Programs
Get PDFBioeconomics combines methods from the biological study of living resources, particularly population dynamics, with methods of economic analysis. Most applications have been in program design for resource management. Although formative evaluations often deal with potential improvements to design based on examination of the program at some point in the early or middle period of its life, there has been little interplay between bioeconomic modelling and evaluation of programs in the context of fisheries management programs
Eliminating School Fees in Low-Income Countries: A Systematic Review
Get PDFBackground: Low educational attainment in the developing world can be attributed in part to the private costs associated with sending children to public school. School fee abolition policies are supported by many development organizations and aid agencies and have been spurred worldwide by initiatives such as Education for All, the Millennium Development Goals, and the School Fee Abolition Initiative.
Purpose: We conducted a systematic review to identify and synthesize the available evidence to respond to the question, What is the evidence of the impact of the elimination of school fees in low-income developing countries?
Setting: Studies included in the review evaluated interventions implemented in low-income countries.
Intervention: Eligible studies had to meet the following criteria: the evaluation took place in a low-income developing nation as defined by the World Bank at the time of the intervention; the evaluation assessed the impact of eliminating primary or secondary public or private school fees. With the intent to conduct meta-analysis, we focused on identifying randomized controlled trials (RCT) or quasi-experimental (QED) evaluations with some evidence that the groups being compared are equivalent.
Research Design: Systematic review; narrative synthesis
Data Collection and Analysis: We identified eligible experimental and quasi-experimental studies through extensive searching, including hand searches, examining grey literature, and contacting experts in the field. Outcomes coded included impacts on primary and secondary school enrollment, gender parity in enrollment, dropout, achievement, and educational quality indicators. Although we intended to quantitatively synthesize the results from the impact evaluations in a meta-analysis, given the small number of studies that met our inclusion criteria and the variation among the studies, we elected to provide the results in a narrative fashion.
Findings: The findings of this systematic review highlight the need for more rigorous and longitudinal empirical research regarding the effects of various types of school fee elimination policies in low-income developing nations—particularly on the effectiveness of targeting policies to the most vulnerable groups, effects on education quality, and the extent to which fee abolition policies can be sustainable