Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
A Practical Framework and Model for Promoting Cost-Inclusive Evaluations
Get PDFBackground: Cost studies are an important component of any serious professional evaluation. Regardless of whether an evaluation is conducted in the public or private sector, decision-makers want answers to two fundamental questions: (1) Is the program, project, or investment worthwhile? and (2) Can funds be used in a better way? The answers to both questions rely on the use of cost analysis techniques. However, according to the literature, most evaluations do not include any type of cost study (see Christie & Fleischer, 2010) and when studies are included, very few are of high quality and rigor (see Madsen, Eddleston, Hansen, & Konradsen, 2017). This may in part be as a result of the relatively limited emphasis placed on this important topic in the evaluation-specific literature, coupled with the reality that many evaluators lack expertise to conduct cost analysis studies.
Purpose: Given the limited use of cost studies in professional evaluations to assess actual program merit/worth, this paper presents a practical framework/model to help evaluators understand fundamental issues that must be considered when thinking about some form of cost-inclusive evaluation.
Setting: Not Applicable.
Intervention: Not Applicable.
Research Design: External desk research was used to ascertain the extent to which cost analysis is discussed in the evaluation literature and the reasons highlighted for underuse. This review provided the foundation for developing the practical framework/model presented in this paper for promoting cost-inclusive evaluation.
Data Collection and Analysis: Not Applicable.
Findings: Cost analysis, which can considerably enrich professional evaluations, is currently quite underused in the evaluation profession. Notable progress has been made over the last 15 years with government agencies around the globe taking the lead to promote cost analysis. However, most independent evaluators are still failing to measure, report, and analyze costs because many do not understand the fundamentals of cost studies. This paper promotes cost-inclusive evaluation by providing evaluators with a framework/model to help them understand important issues in cost analysis
Evaluative Thinking in Practice: Implications for Evaluation from Paulo Freire's Work in Guinea-Bissau
Get PDFBackground: Evaluative thinking is rigorous analysis of evidence and reasoning to inform judgments of merit, worth, value, significance, and utility. Although formal concepts and definitions for evaluative thinking have only emerged recently, its practice has been around for many years. In-depth historical case studies of international evaluations using evaluative thinking principles are rare, especially the ones reported by those who were involved a half-century ago.
Purpose: This article describes little-known but historically significant educational reform initiative in Guinea Bissau (West-Africa) in the 1970s supported by a group led by Paulo Freire, examining what occurred through the lens of evaluative thinking.
Setting: In 1975, Freire and the Idac team were invited to develop a national adult literacy campaign in Guinea Bissau. Freire was already well known for his revolutionary ideas in “Pedagogy of the Oppressed.”
Intervention: Idac team supported the government to adopt Freire’s education and action approach comprising three core strategies: (i) critically understand reality, (ii) conduct scientific studies and (iii) application of the new knowledge. While Freire and four senior members of Idac’s team visited the country every three months, two other members lived there for four years. The decision was to invest in experimental pilot projects while reforming what was possible within the existing traditional school curricula.
Research Design: The authors conducted a retrospective developmental evaluation exemplifying what can be learned by examining past events through contemporary concepts.
Data Collection and Analysis: In-depth interviews were conducted with key informants, especially Claudius Ceccon (one of the authors) who was a senior member of the Idac team working with Paulo Freire in Guinea Bissau. Also, several project original documents and relevant publications were reviewed. Thick description and content analysis were used to revisit that educational reform and examine its implications for evaluative thinking.
Findings: Even though Freire’s most prominent ideas were developed before evaluation was considered as an established field, his approach of dialectic inquiry, critical reflection and taking action has influenced and still influences many leading scholars and practitioners within our (trans)discipline. While Freire did not use the phrase “evaluative thinking,” his process of critical engagement with local people to analyze and understand their situation and take action based on their reflections manifests core elements of what today would be considered embedded evaluative thinking. In political reforms around the world, the retrospective evaluation of what happened in Guinea Bissau reminds us of the importance of evaluative thinking and reflective dialogue – and the fragility of both in the face of political polarization and changes in leadership.
Outcome Mapping as a Monitoring and Evaluation Tool for Livestock Value Chain Interventions: The Case of imGoats
Get PDFBackground: The dynamic and complex natures of value chains added up with the multi-layered and open socio-technical systems that are affected by a range of factors and the continuous adaptation processes to changing conditions, makes measuring changes of value chain interventions a challenging task. Selection of M&E approach is dictated by the reality of multiple actors, relationships and perspectives in complex change processes. The traditional M&E approaches and tools specifically the log frame have been criticized for its failure in measuring changes in complex interventions. M&E approaches and tools that are learning focused, flexible, allow involvement of stakeholders, capture unintended results and focus on contribution of the interventions are more appropriate.
Purpose: This paper discussed the process and results of using OM as monitoring and evaluation tool for value chain interventions and reflect on the success, challenges and lessons.
Setting: imGoats project implemented in India and Mozambique with the aim of increasing income and food security in a sustainable manner by enhancing pro-poor small ruminant value chains.
Intervention: The project employed value chain (VC) and Innovations Systems (IS) approaches rather than traditional methods of technology transfer. IS approaches rely on innovation platforms (IPs), which are spaces facilitated by local innovation brokers where individuals and organizations can come together to address priority issues related to development of value chains. OM was one of the M&E tools used by the project.
Research design: Action research component was superimposed in the project implementation process where data were collected continuously on various aspects. Predominantly, the emic approach is used as most of the authors are directly involved in the action research process. The team had direct exposure in designing and implementing the tool, observing and improving (based on practical experiences and reflections) in the project implementation process. The OM process was continuously monitored and documented with a support from an external consultant.
Data collection and analysis: Data were gathered through a multi-method process including review of documents, key informant interviews, focus group discussions and participant observations. Three workshops were conducted at the beginning, midterm and end of the project to evaluate the progress and challenges of OM application. The reports and feedbacks provided by participants in these workshops are one of the data sources for this study. Furthermore, data was collected from project implementation partners on their reflections with regards to OM as M&E tool.
Findings: The findings of the study depict that Outcome Mapping has many demonstrated qualities that makes it suitable for value chain and innovation systems interventions. If properly applied, OM promotes strategic thinking and enhances organizational responsiveness due to its reflective and learning nature. Even if OM requires an environment which promotes participation, learning and flexibility, it could bring attitudinal change among those involved in its design and implementation. Due to its flexibility OM can capture unintended effects. Moreover, OM can have parallel positive effects on how partners are conducting project management and monitoring activities. In the action research it was evident that OM is adaptable to different methodologies, contexts and type of interventions. Project/intervention duration has implication to fully utilize OM. As behavioral change is a slow process and needs reasonable time, OM could not be fully utilized in terms of measuring some of the behavioral changes in short duration projects. OM is resource intensive especially when it is used for larger projects. It requires time, skilled manpower and other logistics for collecting and analyzing data. Hence, the investment needs to be carefully balanced against the use of it
The Book of Why: The New Science of Cause and Effect. Pearl, Judea, and Dana Mackenzie. 2018. Hachette UK.
No full textA review of the book The Book of Why: The New Science of Cause and Effect by Judea Pearl and Dana Mackenzie, published in 2018 by Hachette UK
Seeking Culturally Safe Developmental Evaluation: Supporting the Shift in Services for Indigenous Children
Get PDFBackground: Evaluation methods based on western frameworks that disregard Indigenous peoples’ worldviews and are imposed and implemented by outsiders are problematic for Indigenous communities.
Purpose: The article presents the experience of using developmental evaluation (DE) in supporting a shift in pre- and post-natal care programming for Indigenous mothers and their young children.
Setting: Indigenous peoples living in urban areas in Quebec often feel unwelcome mainstream services, resulting in under-use. A history of colonization in Canada has resulted in a loss of Indigenous child-rearing practices
Intervention: The study was carried out in the context of a three-year initiative aimed at strengthening the abilities of pregnant women, mothers, fathers, extended family, community, and practitioners to create conditions for the holistic development of themselves and their children. The goal was to create new knowledge through activities focused on promoting perinatal care and psychosocial adaptability. Cultural safety, an ecosystemic view of child development, and social innovation guided the approach to the intervention.
Research Design: A case study approach was used to make sense of and describe the “how to” of the DE.
Data Collection and Analysis: Multiple methods of data collection informed the case study, including observation, field notes, interviews, and participatory evaluation activities.
Findings: The article sheds light on DE as a culturally safe and participatory practice that is compatible with Indigenous perspectives and contributes to supporting the transformation in services provided to Indigenous communities. We present building relationships, creating safe spaces for reflection and dialogue, questioning fundamentals, and co-creation as critical components of culturally safe DE, enabling development and a paradigm shift
Evaluation Essentials: From A to Z (2nd ed.)
Get PDFA review of the book Evaluation Essentials: From A to Z (2nd ed.) by Marvin C. Alkin and Anne T. Vo, published in 2018 by Guilford Press
Evaluating Community Development Projects Using the OECD Evaluation Criteria
No full textBackground: Community development has over the years become very popular in the development circles. However, efforts and resources expended in improving rural communities through development projects appear to be eroded over time. Failure to integrate evaluation in the project design and periodically assess intended objectives and current results is a major contributory factor. Evaluation has become very significant especially in this era of dwindling donor support. It helps to identify strengths and weaknesses of a project and aids in averting erosion of efforts. This paper practically evaluates community development projects implemented in 4 rural communities in the Eastern region of Ghana using the OECD/DAC five project evaluation criteria which are efficiency, effectiveness, relevance, impact and sustainability.
Purpose: The study was conducted to provide a practical example of how community development projects can be evaluated using existing criteria. Also, it is intended to add to existing knowledge on evaluation and encourage project implementers to consider evaluation as an integral part of their implementation.
Setting: The study was conducted in four rural communities in two districts in the Eastern region of Ghana. Eastern region is one of the ten regions in Ghana. The projects involved in the study were a School, Clinic, Oil Palm processor, and a Soap and Cosmetics project.
Intervention: The study shows practically that evaluation is uncomplicated and can be undertaken in interventions. It outlines clearly the gains and losses that can be generated by community development project. It also outlines the threats and opportunities that exist in the implementation of community development projects. This can be applied in other settings.
Research Design: The sample size was 40 made up of 6 NGO staff, 4 local government staff, 2 health assistants and 28 community members including leaders. The community members were sampled randomly while purposive sampling was used for local government, NGO staff and community leaders.
Data Collection and Analysis: Data collection included primary methods such as interviews, focus group discussions, photographs, observations, and questionnaire administration as well as secondary methods such as reviews of relevant books, photographs, project reports, policy papers, and relevant websites. Analysis of the data collected was done qualitatively with simple statistical tools.
Findings: Some findings of the evaluation were that the school was more relevant, efficient, effective and sustainable. It also had maximum impact. The Oil Palm processor was also more relevant, effective and sustainable. It was however less efficient. The Clinic was less relevant, efficient and effective. It also had less impact on the community. The Clinic was not sustainable. The Soap and Cosmetics project was not relevant, not efficient, not effective not sustainable and did not have any impact on the community
Evaluation for an Open Society: Then and Now
Get PDFBackground: Karl Popper’s views about science and political economy remain relevant to evaluation theory and practice. His Open Society opus inspired pioneering contributions to experimental evaluation and shaped the evaluation discipline. Yet, his ideas are not widely known without the evaluation community even though populist leaders are once again threatening to undermine democracy.
Purpose: To define the Open Society, probe its epistemological tenets, confirm that they remain valid as the foundation of evaluation practice, identify the ways in which the operating environment for evaluation has changed and, against this background, propose a policy change agenda relevant to the contemporary evaluation discipline.
Setting: The Open Society is once again being undermined. Modern authoritarianism is tightening its grip. The lure of the strong man is once again gaining traction. The dominance of an international order grounded in democracy, human rights, and the rule of law is giving way to a world in which leaders are pursuing narrow nationalist and vested interests. In this troubled context, policy making has become more complex than when evaluation emerged out of the ashes of World War II. Economic and social dysfunctions have led to extraordinary concentration of wealth and privilege. Ominous environmental threats loom. The architecture of international relations designed in the mid-1940’s has become obsolete.
Research Design: To design this commentary about the prospects of the evaluation discipline, the author drew on his personal experience as evaluation academic, international development practitioner, manager of the World Bank’s Independent Evaluation Group for two consecutive five-year terms and senior independent evaluation adviser to governments and international development agencies.
Intervention: As an intervention, this article adds value to evaluation theory and practice by showing why and how the Popper/Campbell mandate for evaluation needs to be upgraded to protect the public interest in a new operating environment. Specifically, Popper’s piecemeal social experimentation concept should be refined to forge links between small scale experiments and the broader fabric of society. In addition, the ambiguity regarding the relationship between the Open Society and evaluation should be lifted through a reconsideration of the democratic evaluation model.
Data Collection and Analysis: The author conducted an extensive review of the literature and consulted with a wide range of evaluation thinkers to examine the extent to which Popper’s philosophy remains relevant to the evaluation discipline.
Findings: Popper’s Open Society ideas aimed at avoiding the rise and perpetuation of autocracy and remain highly relevant. But the current threats to democracy call for a more ambitious and detailed remit for the evaluation occupation. Beyond the promotion of evaluation in democracy and of democracy in evaluation, evaluation for democracy should be pursued. This implies putting value, ethics, and the public interest at the very center of the evaluation occupation; breaking free of Popper’s parsimonious piecemeal social engineering concept to inform systemic social reform; bringing peace to a methodologically divided house; systematic mixing of evaluation methods and models; and the promotion of evaluation independence through professionalization
Refining Theories of Change
Get PDFBackground: Despite the disparities in how they are defined and what elements are included, most Theories of Change remain consistent in one way – their visual format. Typically, Theories of Change are presented as a one-page visual in a flowchart style with lines and boxes of uniform size. In addition, Theories of Change are often created as stand-alone tools that are rarely linked effectively to other organizational tools.
Purpose: The authors: (1) propose the essential elements that contribute to robust Theories of Change and clarify the characteristics that distinguish Theories of Change from other organizational tools and formats; (2) suggest additional elements for inclusion in the Theory of Change; (3) present graphic alternatives that allow for an evolution in representing their complexity and depth; and (4) provide ways to link Theories of Change to other organizational tools to increase organizational alignment, efficiency, and, most importantly, impact.
Research Design: NA
Data Collection and Analysis: NA
Setting: NA
Intervention: NA
Findings: N