Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Exploring Design Thinking Practices in Evaluation
Get PDFBackground: Design thinking is the process by which the core principles of design are used to problem solve and identify innovative solutions that enahance user experience.
Purpose: The purpose of this article is to introduce the concept of design thinking and explore the basic principles of design thinking within the context of the evaluation.
Setting: The design thinking process has been used in various fields to enhance innovation and consumer experience. This article explores how the concepts of design thinking can enhance evaluation practice.
Intervention: This article did not require an intervention
Research Design: Not applicable
Data Collection and Analysis: Not applicable
Findings: We present ideas for adopting design thinking principles into everyday evaluation practic
Are All Biases Bad? Collaborative Grounded Theory in Developmental Evaluation of Education Policy
Get PDFBackground: Using two researchers as independent instruments for interpretation in education policy evaluation, this study applies a collaborative grounded theory approach to qualitative data analysis and theory generation.
Purpose: This study argues that varied perspectives should be a critical component in the methodological and analytical choices of education research, especially when the sought after outcome is deeper understanding of the impact, both positive and negative, of an education program or policy. In this study, rather than using one researcher to confirm the reliability of the other, the study explores the outcome of drawing on the positional reflexivity of two researchers, each with a distinct perspective, as a potential strength to co-generate themes and theory in the evaluation of complex policy or programs.
Setting: The data for this analysis originated from interviews of education leaders (n = 13) from two states with contrasting approaches to teacher evaluation: Kentucky and California.
Intervention: NA
Research Design: Qualitative developmental evaluation
Data Collection and Analysis: Semi-structured interviews and grounded theory coding
Findings: Results suggest that more robust theory and analysis may result from independent thematic development and converged theory generation when working in a research team, as opposed to early application of inter-rater reliability. The reflexivity of different perspectives was, in part, reflexive of the self—their own biased perspective and prior experience. When merged, their joint interpretation may have unearthed greater dimensionality. Findings from this study can inform future strategies for evaluating qualitative research data, especially within a developmental evaluation approach aimed at understanding system complexity in education policy and practice
Book Review: Evaluating Communication for Development: A Framework for Social Change: (2013) June Lennie and Jo Tacchi. London: Earthscan.
Get PDFA review of the book Evaluating Communication for Development: A Framework for Social Change by June Lennie and Jo Tacchi, published in 2013 by Routledge
On the Feasibility of Extending Social Experiments to Wider Applications
Get PDFBackground: When deciding how to allocate limited funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that rely on solid methodology, providing credible scientific evidence. The basic claim for the “social experiment”—that the “coin flip” of randomization creates two statistically equivalent groups that do not diverge except through an intervention’s effects—makes resulting estimates unbiased. Despite the transparency and conceptual strength of the experimental strategy for revealing the causal connection between an intervention and the outcomes of its participants, the wisdom or feasibility of conducting social experiments is often questioned on a variety of grounds.
Purpose: This article defines 15 common concerns about the viability and policy reliability of social experiments, in order to assess how much these issues need constrain the use of the method in providing policy evidence.
Setting: NA
Intervention: NA
Research Design: The research uses the authors’ experience designing and conducting dozens of social experiments to examine the basis for and soundness of each concern. It provides examples from the scholarly literature and evaluations in practice of both the problems posed and responses to each issue.
Data Collection and Analysis: NA
Findings: We conclude that none of the 15 concerns precludes substantially extending the use of randomized experiments as a means of evaluating the impacts of government and foundation social policies and programs.
Hirschman’s Ideas as Evaluation Tools
Get PDFBackground: Albert O. Hirschman, one of the most distinguished social scientist of the past half century, is not widely known within the evaluation community. Yet he practiced the art of evaluation without acknowledging it and the influential concepts that he generated are extremely valuable as evaluation tools.
Purpose: The purpose of this article is to demonstrate by dint of example of example that Hirschman’s ideas illuminate the development experience; inform evaluation methods and have potential as guides to evaluation practice.
Setting: Hirschman’s unique intellectual contributions to our understanding of society have recently been recognized in a flurry of publications that followed his death in 2012.
Intervention: This article did not require an intervention.
Research Design: Three popular Hirschman concepts are dissected and used to explore a variety of evaluation policy issues.
Data Collection and Analysis: Not applicable.
Findings: Hirschman’s mental constructs are interconnected; throw new light on evaluation criteria and can be used to get the most out of the evaluation function in diverse authorizing environments
Evaluating Diversity Metrics: A Critique of the Equity Index Method
No full textBackground: Evaluating diversity, inclusivity, and equity remains both a prevalent topic in education and a difficult challenge for most evaluators. Traditional metrics used to evaluate these constructs include questionnaires, focus groups, and anonymous comment solicitations. While each of these approaches offer value, they also possess a number of limitations (e.g., self-reported nature, holistic perspective, social desirability bias, varying degrees of respondent sensitivity, representative responses, etc.). Researchers at the University of Southern California have successfully utilized the Equity Index Method (EIM) as a potential approach for measuring diversity and reporting diversity-related outcomes.
Purpose: Provide a critique of the EIM and discusses how the EIM could be improved and extended to other evaluation contexts and settings.
Setting: Not Applicable.
Intervention: Not Applicable.
Research Design: Not Applicable.
Data Collection and Analysis: Not Applicable.
Findings: Despite the potential for problems with interpretations based on small samples and subgroups and some concerns about semantics involving the term “equity”, we believe the EIM possesses a number of strengths that many evaluators will find useful. We encourage other evaluators to consider this method and explore its utility in a variety of contexts
Evaluation Revolutions
Get PDFThe everyday practice of evaluation has continued for millennia, only recently sprouting an academic branch that became more sophisticated and transformed into an important discipline. More precisely, it developed into a family of sub-disciplines—product evaluation, program evaluation, personnel evaluation, policy analysis, etc. At the metalevel, the perception of it in most of the academic world has undergone some highly significant shifts, separated by what can fairly be described as revolutions. That sequence is what I try to describe below, starting with a description of the pre-revolutionary baseline state. The list extends from the past, through the present, and into the future, the latter including some suggestions about revolutions I think we need to kickstart. Most evaluators will continue to spend most of their time on applied evaluation in some specialty field, but I’m hoping they will help out part-time with the revolutions, at least by thinking and arguing about them. Better still if they are intrigued and challenged by the suggestions for future revolutions, and add their own experience to the revolutionary task—or to a counter- revolutionary reaction.
Mindful Evaluation: Cultivating Our Ability to Be Reflexive and Self-Aware
Get PDFBackground: Mindfulness, giving our full attention to what we are doing in the present moment, is perhaps best understood as training for the brain. When we are mindful, we are actively engaged with our thoughts, feelings, and sensations.
Purpose: The purpose of this article is to introduce the concept of “mindful evaluation” as a way to cultivate our reflexivity and self-awareness to improve our evaluation practice. Mindful evaluation is an invitation to be more intentional and reflexive about our ontological, epistemological, and methodological assumptions in general as well as for each evaluation we undertake.
Setting: Mindfulness has been gaining popularity both with the general public and a variety of professional disciplines thanks in large part to the growing body of research on its efficacy. As disciplines such as healthcare, economics, and education are incorporating the benefits of mindfulness in their work, we explore how we too might use the principles of mindfulness in evaluation and evaluation practice.
Intervention: This article did not require an intervention.
Research Design: Not applicable
Data Collection and Analysis: Not applicable
Findings: We present simple steps for incorporating the principles of mindfulness to how we approach evaluation
Evaluation Capacity Development through Cluster Evaluation
Get PDFBackground: The term “cluster evaluation” was first coined in 1988 by W. K. Kellogg staff in an evaluation of a Foundation-funded initiative; the concept was further developed and practiced by the Kellogg Foundation evaluation consultants and other practitioners in the evaluation community
Setting: Cluster evaluation was a model used to evaluate the programming of a small specialized UN agency in three countries (Lebanon, Jordan and Palestine) of the Arab States.
Purpose: The article explains how cluster evaluation, as originally conceived by the W. K. Kellogg Foundation, was adapted to the realities of the UN system.
Subjects: NA
Research Design: The authors present a case study of an evaluation of a cluster of programmes from the Arab States region (Lebanon, Jordan and Palestine) that was conducted by a small, specialized agency of the UN.
Data Collection and Analysis: The evaluation was designed as a series of programme reviews. Information from the reviews was to be aggregated in order to understand collective contributions to the region.
Findings: The case study demonstrates the potential benefits of the model as well as some of the challenges.
Conclusions: Analysis of the data leads to a number of interesting conclusions. First, in the philanthropic sector, cluster evaluation is a programmatic intervention. That is not the way that the model was used in the context of this evaluation. Instead of evaluation as intervention, it was evaluation as capacity development.
This evaluation took place in the Arab States, a region in which evaluation capacity is perhaps not as institutionalized as other parts of the world. The purpose of using a cluster evaluation in this context was to strengthen the capacities of the national evaluators.
Each of the national evaluators who were contracted had different areas of strength and weakness. By establishing a strong network amongst themselves, they were able to leverage each other’s strength and to compensate for any weaknesses.
The consultants developed each other’s capacity and that greatly benefited the evaluation. The higher capacity went on to benefit the collective evaluation capacity of the region. The national evaluators have all gone one to be leaders in the formation of their respective regional and national evaluation organisations
Question-Driven Methods or Method-Driven Questions? How We Limit What We Learn by Limiting What We Ask
Get PDFThe “methodologically manic-obsessive” evaluation profession and the metrics- and measures-obsessed lay users of evidence have managed to seriously limit the value of what we learn from evaluation. Evaluation questions asked at the front end are limited by the askers’ narrow understanding of what is possible methodologically at the back end. This, alongside the political and psychological forces working against real evaluation, are major drivers of the single narrative thinking that pervades the formulation and evaluation of national and local government policies and initiatives. This paper provides practical suggestions for asking the big-picture questions that really need to be asked, and suggests how real evaluation can step up to the plate, methodologically and otherwise