Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Complexity-Aware Monitoring and Evaluation
Get PDFBackground: Addressing today’s sustainability challenges requires adopting a systemic approach where social and ecological systems are treated as integrated social-ecological systems. Such systems are complex, and the international development sector increasingly recognises the need to account for the complexity of the systems that they seek to transform.
Purpose: This paper sketches out the elements of a complexity-aware monitoring and evaluation (M&E) system for international development programmes in the area of sustainable development.
Setting: Not applicable.
Data Collection and Analysis: The authors draw on existing literature on complexity and evaluation and on their own experience from working in the field of M&E.
Findings: An M&E system should not be seen simply as a tool to track compliance against a pre-determined theory of change. Instead, it is most useful as a real-time approach, constantly defining and re-defining narratives for change that help push systems along trajectories of interest. Dealing with complexity involves embracing uncertainty; and this challenges established notions of accountability—something which funders and implementers must begin to redefine together
Evaluation in Our New Normal Environment: Navigating the Challenges with Data Collection
Get PDFBackground: Data collection is a critical component of all evaluations. However, it often presents a number of challenges under the best of circumstances. For instance, the evaluation budget and time frame both have implications for the quality and type of data that is collected. Additionally, adherence to high quality international ethical best practices is necessary when collecting data for any purpose, methodological rigor is important for ensuring the credibility of the evaluation, improving access to important documents and stakeholders, as well as decreasing excessive evaluation anxiety on the part of critical stakeholders, when possible, is vital. These challenges have now been considerably exacerbated by the COVID-19 global health pandemic which has changed our world in fundamental ways. In what is now considered as our new normal environment, evaluators will need to make profound changes to the manner in which they plan and undertake data collection.
Objectives: This paper examines the many and varied challenges that will be encountered with data collection in our new normal environment. This new normal has had an impact on evaluation practices in all countries, developed and developing, and has significantly amplified existing challenges in countries with limited evaluation culture, budgets, technological coverage, access, and connectivity. It makes an important contribution to the literature since data collection has historically and traditionally been conducted using primarily face-to-face field work and through the freedom of movement of people to undertake this task.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Desk review was utilized for the preparation of this paper.
Findings: Evaluators need to be extremely flexible, innovative, and amendable to different approaches to data collections as our new normal environment will likely be with us for a while. This pandemic has thrown everyone a very painful curveball and introduced significant new work-related challenges for a myriad of work types and work environments. Innovation and the willingness to learn new methods have become an important necessity to help with learning, accountability, transparency. The COVID-19 pandemic has highlighted the plight of the most vulnerable and evidence-based data is the only means to assist this group. Evaluators must rise to the challenge, devise new ways to collect data that is credible and useful, and continue to promote the importance and benefits of the field of evaluation. As such, evaluators have an important role to play in the global economic recovery efforts
The Birth and Adaptation of Evaluation Theories
Get PDFBackground: Evaluation theories as we know them are prescriptions by prominent evaluators about what they believe to be an appropriate way to conduct evaluations. How do these prescriptions come about? In this paper we examine the various influences on the creation and subsequent modification of these prescribed evaluation theories. Inquiry into evaluation theories has a long history. What is new is inquiry into the evolution of theories.This makes theory formulation dynamic rather than static. Influences identified by Alkin in a National Society for the Study of Education yearbook (1989) serve as an initial guide to this inquiry. An examination of Michael Q. Patton's writings and shaping experiences provides further case study insights about the evolution of his utilization-focused evaluation theory and its offshoots.
Purpose: The purpose of this paper is to gain further understanding about the way in which evaluation theories are developed, evolve, and take new directions, and the influences that shape the theorists' understandings and prescriptions.
Setting: Interview discussion with Michael Q. Patton and synthesis of interview data.
Intervention: Not applicable.
Research design: Not applicable.
Data collection & analysis: Not applicable.
Findings: Factors that have influenced Michael Q. Patton’s initial theory development as well as subsequent modifications, adaptations, and offshoots offer insights into the connection between personal history and professional perspective. Specifically, these factors were: early personal experiences, professional training, interaction with professional colleagues, field evaluation experiences, interaction with non-evaluation academic colleagues and research conducted by Patton
Learning Lessons for Evaluating Complexity Across the Nexus: A Meta-Evaluation of Environmental Projects
Get PDFBackground: A major gap in environmental policy making is learning lessons from past interventions and in integrating the lessons from evaluations that have been undertaken. Institutional memory of such evaluations often resides externally to government, in evaluation practitioner contractors who undertake commissioned evaluations on behalf of government departments.
Purpose: The aims were to learn the lessons from past policy evaluations, understand the barriers and enablers to successful evaluations, to explore the value of different types of approaches and methods used for evaluating complexity, and how evaluations were used in practice.
Setting: A meta-evaluation of 23 environmental evaluations undertaken by Collingwood Environmental Planning Ltd (CEP), London, UK was undertaken by CEP staff under the auspices of CECAN (the Centre for Evaluation of Complexity Across the Nexus – a UK Research Councils funded centre, coordinated by the University of Surrey, UK). The research covered water, environment and climate change nexus issues, including evaluations of flood risk, biodiversity, landscape, land use, climate change, catchment management, community resilience, bioenergy, and European Union (EU) Directives.
Intervention: Not applicable.
Research design: A multiple embedded case study design was adopted, selecting 23 CEP evaluation cases from across a 10-year period (2006-2016). Four overarching research questions were posed by the meta-evaluation and formed the basis for more specific evaluation questions, answered on the basis of documented project final reports and supplemented by interviews with CEP project managers. Thematic analysis was used to draw out common themes from across the case categories.
Findings: Policy context invariably framed the complex evaluations; as environmental policy has been spread beyond the responsibility of government to encompass multiple stakeholders, so policy around nexus issues was often found to be in a state of constant flux. Furthermore, an explicit theory of change was only often first elaborated as part of the evaluation process, long after the policy intervention had already been initiated. A better understanding of the policy context, its state of flux or stability as well as clarity of policy intervention’s objectives (and theory of change) could help significantly in designing policy evaluations that can deliver real value for policy makers. Evaluations have other valuable uses aside from immediate instrumental use in revising policy and can be tailored to maximise those values where such potential impact is recognised. We suggest a series of questions that practitioners and commissioners could usefully ask themselves when starting out on a new complex policy evaluation
Equity Implications in Evaluating Development Aid: The Italian Case
Get PDFBackground: In the field of development aid, social equity is an emerging issue that concerns the evaluation community in its theoretical and practical dimensions.
Purpose: A widely held belief is that evaluators do not apply theory. In this paper, we intend to verify this statement about equity in the field of cooperation projects.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: We considered equity-focused approaches and found three common factors: stakeholder participation, attention to context, and focus on marginalized groups. These elements operate as screening criteria in identifying equity issues in a case study.
Data Collection and Analysis: The paper examines a practical experience of Italian cooperation. This involved a review of evaluations reports completed between 2013 and 2014. The reports are analyzed according to the three screening criteria.
Findings: The use of the three criteria has proved its worth in grasping the issues of equity neglected and often not recognized in reports. Once again a gap emerges between theory and practice. The availability of theoretical approaches is not sufficient. The paper, therefore, proposes a reflection on the responsibility of evaluation towards social justice
Using PhotoVoice as an Evaluation Method
Get PDFBackground: Engaging with youth through PhotoVoice is beneficial as a program evaluation method and functions as a method of inquiry to understand youths’ perceptions of a college preparation program. The students used PhotoVoice to respond to prompts about how they learn and their opinions of the college preparation program.
Purpose: This reflection of practice article provides an example of PhotoVoice as an evaluation method.
Setting: This evaluation was conducted during the summer college preparation programming.
Intervention: The combination of student photography, photo gallery walk, and group discussion as an evaluation method.
Research Design: A qualitative reflective design.
Data Collection and Analysis: The student photographs, narratives, and observational notes were analyzed thematically.
Findings: We found that the youths enjoyed the unique experience of PhotoVoice. The combination of student photography, narratives, gallery walk, and group discussion, were useful as an inquiry tool for this youth program
Opportunities and Challenges to Increase Inter- and Transdisciplinarity: A Qualitative Study of the FloodRISE Project
Get PDFBackground: The FloodRISE project, which started in 2013 in Southern California, aimed at better understanding how to promote resilience to coastal flooding. It was based on a cross-disciplinary approach, involving several research teams and local communities.
Purpose: We conducted a qualitative study of the first phase of the project (2013-2015) in order to analyze its inter- and transdisciplinary aspects.
Setting: We conducted this evaluation as a visiting postdoctoral researcher at UCI, not participating in the FloodRISE project.
Intervention: Not applicable.
Research design: We conducted 18 semi-structured interviews with members of the three project teams - modeling, social ecology and integration & impact - at UCI in 2015. Data were analyzed and interpreted to identify key aspects of the collaboration within and between project teams, as well as their relationship to local stakeholders.
Findings: The analysis showed that an intensive dialogue-based method of interaction and the presence of boundary researchers played a fundamental role in bridging the conceptual and methodological gaps between social and engineering sciences. These results thus exemplify several possibilities for developing more efficient interactions between researchers in a cross-disciplinary project. However, any cross-disciplinary project should: carefully evaluate potential for participants to become boundary researchers, since participants with multiple disciplinary expertise may be underemployed; improve researchers’ level of readiness, in order to facilitate further interaction and increase time efficiency; and clearly address remoteness issues to avoid lower collaboration between central and peripheral locations
A Critical-Historical Review of Program Evaluation and the Emerging Motif ‘Evaluation Science’
Get PDFBackground: It is important to distinguish between evaluation as an inherent, automatic, affective process and Program Evaluation (Evaluation, with capitalised ‘E’) as an institution, and equally important to consider what a good understanding of evaluation tells us about Evaluation. Evaluation is an established social institution whose modern roots can be traced back to 16th century France. Since the early 1900s the institution has developed within and across a range of scientific disciplines with interests in perceived social problems and efforts to resolve the said problems. This can be demonstrated objectively by the number and scale of relevant publications within relevant disciplines. This, in turn, helps us understand more about Evaluation as an institution. Set in this context is the question of Evaluation Science: is this simply a fashionable institutional motif or is it a potential new era for Evaluation?
Purpose: Commentary on the history and development of Program Evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research design: Not applicable.
Data collection & analysis: Not applicable.
Findings: Not applicable.
Keywords: program evaluation; evaluation; history; evaluation science
The Elements to Enhance the Successful Start and Completion of Program and Policy Evaluations: The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute: The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute
No full textBackground: Public health practitioners, including injury and violence prevention (IVP) professionals, are responsible for implementing evaluations, but often lack formal evaluation training. Impacts of many practitioner-focused evaluation trainings—particularly their ability to help participants successfully start and complete evaluations—are unknown.
Objectives: We assessed the impact of the Injury and Violence Prevention (IVP) Program & Policy Evaluation Institute (“Evaluation Institute”), a team-based, multidisciplinary, and practitioner-focused evaluation training designed to teach state IVP practitioners and their cross-sector partners how to evaluate program and policy interventions.
Design: Semi-structured interviews were conducted with members of 13 evaluation teams across eight states at least one year after training participation (24 participants in total). Document reviews were conducted to triangulate, supplement, and contextualize reported improvements to policies, programs, and practices.
Intervention: Teams of practitioners applied for and participated in the Evaluation Institute, a five-month evaluation training initiative that included a set of online training modules, an in-person workshop, and technical support from evaluation consultants.
Main Outcome Measure(s): The successful start and/or completion of a program or policy evaluation focused on an IVP intervention.
Results: Of the 13 teams studied, a total of 12 teams (92%) reported starting or completing an evaluation. Four teams (31%) reported fully completing their evaluations; eight teams (61%) reported partially completing their evaluations. Teams identified common facilitators and barriers that impacted their ability to start and complete their evaluations. Nearly half of the 13 teams (46%) – whether or not they completed their evaluation – reported at least one common improvement made to a program or policy as a result of engaging in an evaluative process.
Conclusion: Practitioner-focused evaluation trainings are essential to build critical evaluation skills among public health professionals and their multidisciplinary partners. The process of evaluating an intervention—even if the evaluation is not completed—has substantial value and can drive improvements to public health interventions. The Evaluation Institute can serve as a model for training public health practitioners and their partners to successfully plan, start, complete, and utilize evaluations to improve programs and policies
Method for Using Rubric Ratings on Fishbone Diagrams to Compare Case Studies
Get PDFBackground: In multi-case study program evaluations, the large amount of qualitative data that are generated from interviews can be difficult to utilize. This is particularly so when inference must be made as to why some cases succeed and some fail.
Purpose: This paper shows a method for comparing multiple evaluation sites by using a rubric to define ratings for relevant factors, and an Ishikawa fishbone diagram as a model to show relationships among those factors. We show how this technique identified reasons for differences in outcomes among the sites.
Setting: The evaluation setting was a large-scale safety innovation in the U.S. railroad industry. Four cases were considered—two passenger railroads and two freight railroads.
Intervention: The Confidential Close Call Reporting System (C3RS) program allowed railroad workers to confidentially submit “close calls” which were reviewed by a team made up of labor, management, and the Federal Railroad administration to determine ways to improve safety.
Research design: Multiple comparative case study, Ishikawa root and contributing cause modeling, evaluative rubric scoring, and data visualization techniques.
Data collection & analysis: Interview data were collected from four pilot railroad sites, each of which participated in a five-year test of C3RS. Testing periods overlapped, with the entire evaluation lasting about 12 years.
Findings: The method of using Ishikawa fishbone diagrams with ratings from an evaluative rubric was an effective method to summarize, analyze, and present large quantities of qualitative data. The approach succeeded in explaining degrees of success and failure across the sites. The sponsor and industry stakeholders were able to understand the analysis and the findings, and to develop deep insight into how to promote successful implementation