Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Supplement to Theories of Change: Making Value Explicit
Get PDFAdditional notes on the definition of “Theory” as presented on page 39 in Powell, S. (2019). Theories of change: Making value explicit. Journal of MultiDisciplinary Evaluation, 15(32), 37-52. https://doi.org/10.56645/jmde.v15i32.563
What, How, and Why? A Comparative Analysis of 12 Goal-Free Evaluations
Get PDFBackground: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without reference to predetermined goals or objectives.
Purpose: The purpose of this study is to examine GFE in actual practice focusing on what GFE is, how it is conducted, and why the evaluators use it.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Document analysis.
Data Collection and Analysis: The researcher collected data from a non-random sample of 12 cases of GFE found in published and non-published papers, reports, and guidebooks. The researcher analyzed the documents using quantitative and qualitative content analysis.
Findings: The findings suggest that goal-free evaluators consider GFE an outcome evaluation that supplements GBE. These goal-free evaluators typically used an ex post facto evaluation design, non-random sampling of stakeholders, and semi-structured interviewing to collect data. The evaluators described using GFE to improve the evaluand, to find side-effects, and to evaluate highly complex evaluands
Raising Aid Efficiency with International Development Aid Monitoring and Evaluation Systems
Get PDFBackground: Since the adoption of the Millennium Development Goals (MDGs), or more specifically perhaps the Monterrey Consensus, there has been a distinct shift in the conversation around assessing the development effectiveness of international aid programs. Initially, the focus had been on establishing monitoring and evaluation systems that served the needs to bilateral and multilateral donors to demonstrate the effectiveness of their assistance vis-à-vis their own constituencies. Today, there is an increasing recognition that recipient countries should be equally concerned with the effectiveness of donor resources as they are with the use of national resources devoted to development programs.
Purpose: This article reveals the current efforts of the sovereign new and traditional donors in establishing and operationalizing the international development assistance (IDA) monitoring and evaluation (M&E) systems to raise the efficiency of IDA programs.
Setting: N/A
Intervention: N/A
Research Design: A review of previous studies and international practicies of IDA M&E.
Data Collection and Analysis: Employing game theory the authors identify the choice of particular IDA M&E system by traditional and new donors and analyze key elements and factors affecting M&E systems operationalization. They elaborate a set of policy recommendations on how to use M&E systems both in donor and recipient countries to raise aid effectiveness.
Findings: The authors conclude that in spite of donor and recipient countries having different purposes and approaches in implementing IDA projects, the use of the M&E is usually helpful to bridge their interests and to increase aid effectiveness
Big Shoes to Fill: An Evaluation Journey in the Footsteps of Daniel L. Stufflebeam
Get PDFBackground: Evaluation has evolved remarkably since the early 1960s, largely due to the innovative contributions of Daniel Stufflebeam and his colleagues. As a pioneer of evaluation methods, some of the notable achievements arising from Stufflebeam’s work include the context-input-process-product (CIPP) model, evaluation standards, and evaluation checklists.
Purpose: The purpose of this paper is to explore Daniel Stufflebeam’s journey beginning with the early days of evaluation through to his retirement and unfortunate passing at 80 years old in 2017. Key features of the CIPP model are considered within a context of other popular models for comparison with the goal of finding relevance for use of CIPP evaluation in education settings.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Literature review.
Data Collection and Analysis: Not applicable.
Findings: Stufflebeam’s CIPP model and evaluation standards remain prominent in evaluation practices and his legacy will lay the foundation for future evaluators through continued professional development
Methods for Collecting Data Based Upon an Eastern Paradigm of Evaluation
Get PDFBackground: This is one in a series of articles in which the authors attempt to relate Eastern philosophy to contemporary program planning and evaluation.
Purpose: The authors examine data collection methods through the lens of Eastern Philosophy.
Setting: N/A
Intervention: N/A
Research Design: Pre-experimental design: one-shot case study.
Data Collection and Analysis: The author attempts to "see" a photograph sealed in an opaque envelope.
Findings: The author was able to draw a picture that somewhat resembled the photograph. This was one of the author's better attempts at CRV. To be able for him to refine this technique to the point that it might have application for evaluation will no doubt take a long time -- if ever. However, it is an intriguing notion
Children’s Opinion of Retrospective Pre-Post ‘Then-Test’ Survey Validity
Get PDFBackground: Over the past forty years there have been a number of studies conducted to compare traditional pre-post surveys (pretest-posttests; administered in two stages before and after an intervention) with retrospective pre-posts (thentests or pre-then-post-tests; administered after intervention only, with participants asked to reflect back to complete the ‘pre’ retrospectively). These previous studies have been with adult respondents and overwhelmingly quantitative.
Purpose: This paper examines children’s perspectives regarding traditional and retrospective pre-post self-report subjective surveys.
Setting: A school-based program run by a community services organisation in southeastern Melbourne.
Intervention: Both pre-post survey types were administered to sixty children attending a pro-social skills group run by a community services organisation in southeast suburban Melbourne.
Research Design: Twenty children participated in eight small focus groups after completing the surveys. Each focus group was guided by three semi-structured questions, ran for 10-15 minutes, and had 2-3 participants. This research included an observation component as the researcher was present at the final session when the post surveys were completed. The research also utilised the quantitative findings from the surveys to check alignment with findings from the extant literature.
Data Collection and Analysis: Focus groups & qualitative analysis
Findings: The traditional and retrospective surveys confirm that the commonly recorded phenomenon of response shift in adults also occurs with children. Children comment that they prefer the retrospective test, identifying concerns that support theories discussed in the extant literature such as experience limitation, impression management, implicit theory of change, and memory recall.
Theories of Change: Making Value Explicit
Get PDFBackground: This article addresses two problems. The first is the Flexibility Problem: If we are to use a more flexible format for theories of change than for traditional logic models, one in which we can no longer assume that we only value things which are at the end of causal chains, nor that we intervene on all the things at the beginning of causal chains, how then can we show which things we value, and which things we intervene on? The second is the Definition Problem: What is the difference between a theory showing the causal influences within and around a project and, more specifically, a theory of change for the project?
Purpose: To solve the Flexibility Problem and the Definition Problem.
Setting: N/A
Intervention: N/A
Research Design: N/A
Data Collection and Analysis: N/A
Findings: A definition of “Theory of Change” is introduced, based upon a definition of “Theory” together with two symbols to mark variables we value (“♥”, or any suitable alternative symbol) and variables we intervene on (“▶”, or any suitable alternative symbol). These two definitions and the two symbols together answer both the Flexibility Problem and the Definition Problem, and have some interesting side-effects as follows. Firstly, they suggest that it is the task of evaluators to model how stakeholders value aspects of a project just as much as it is to model the causal chains within a project. Secondly, evaluators are able to model the fact that stakeholders may value variables which are not at the end of a causal chain, throwing a new light on the debate between results-based and principles-based programming. Thirdly, they provide a way to understand the behaviour of stakeholders and stakeholder groups in terms of their own theories of change – the way they view the world and how they can get what they want – rather than from the traditional behaviourist perspective more familiar to most evaluator
Evaluation Champions: A Literature Review
Get PDFBackground: Numerous studies call for the identification and engagement of evaluation champions in the implementation of an evaluation initiative. However, no agreed definition of an evaluation champion exists in the extant literature. Published studies on evaluation champions are limited and motivations of evaluation champions are not well understood. Understanding of how evaluation champions interact with their colleagues to generate momentum for change is lacking.
Purpose: This article explores champions in organizational settings and highlights the need for an increased understanding of evaluation champions. The research question posed is, ‘What does the literature tell us about evaluation champions in organizational settings?’
Setting: Not Applicable.
Intervention: Not Applicable.
Research Design: Relevant articles were identified through systematic searches of selected databases and reference reviews of retrieved articles from the evaluation and organizational development bodies of evidence. Theories that could assist with understanding the role of evaluation champions were also drawn upon.
Data Collection and Analysis: Not Applicable.
Findings: The analysis indicates evaluators and organizations value champions because they work to bring evaluative thinking into their practice and positively promote evaluation among their colleagues. A list of activities that may be indicative of the behaviours of champions has been compiled to contribute to the evidence base
We Now Have the Tools and Infrastructure to Hold Donors and NGOs in International Development to their Own Legal and Professional Standards
Get PDFBackground & Purpose: This article summarizes and adds to the tools and infrastructure that the author has developed to hold donors and NGOs in international development to their own international, legal and professional standards, following a call in 2008 for organizations to find objective ways to hold those organizations to compliance with international law and professional principles.
Setting: Global.
Intervention: Not applicable.
Research Design: The article presents 12 indicator tools (in the form of legal elements tests) recently published elsewhere and a new litmus test tool presented here for the first time for quick evaluations of projects using an inductive approach (looking at project logic), explaining how these tools relate to each other and how they can be used together.
After introducing these indicators, the piece then compiles and summarizes the results for several types of organizations to reveal an overall picture of which donors and NGOs are failing, which are succeeding, and what this now objectively verifies is happening in the world of international development.
Data Collection and Analysis: Not applicable.
Findings: The piece offers some reflections on the world that we live in where international standards and universal principles are not applied, where legal codifications for international development are not enforced, and where current international development approaches are leading to unsustainability, conflict, and homogenization (suppression of human diversity and adaptation) that the standards were designed to help avoid. The author’s approaches, overall, offer the larger blueprint for an infrastructure of “development” work to promote universal legal principles, as well as a larger set of reforms for changes in social and political institutions and systems in the developed world for making these changes a reality.
Keywords: sustainability, dependency, democracy, development, aid, capacity building, international relations, international law, donors, UNDP, World Bank, European Commission, NGOs, foundations
Treatment Effect Estimation Using Self-Estimated Counterfactuals Under Varying Conditions: A Meta-Analytic Exploration
Get PDFBackground: Randomized controlled trials (RCTs) are frequently not an option in evaluation practice, which is why evaluators switch to non-experimental methods–such as the “counterfactual as self-estimated by program participants” (CSEPP) for estimating intervention effects. Unfortunately, no systematic attempt has been made to test under what conditions CSEPP provides valid estimates.
Purpose: As a first step in this direction, this research compared the performance of CSEPP in terms of bias when applied in different groups of participants with different levels of education, when used for assessing the effects on different outcome variables, and when employed with different question orders within the questionnaire.
Setting: NA
Intervention: The treatment used in this research was a short educational video, in which the audience is educated about important concepts and aspects of organ donation.
Research Design: Since investigating bias in CSEPP is difficult at participant level, a series of 40 studies was conducted and bias was analyzed at study-level. For each study, the effect of the same treatment was estimated by CSEPP and compared with the effect estimated by a simultaneously conducted RCT. Afterwards, it was analyzed whether differences between CSEPP and RCT across the studies were determined by variation in the conditions under which the studies were conducted. Despite small sample sizes of the single trials, the meta-analysis was sufficiently powered to detect even small differences between CSEPP and RCT.
Data Collection and Analysis: The data was collected via online surveys on a crowdsourcing portal. For data analysis, we applied meta-analytic methods such as random-effects meta-analysis and meta-regression.
Findings: Results show that CSEPP provided accurate effect estimates, no matter under what conditions the method was applied.