Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Ubuntu and Afrofeminism for Decolonizing Evaluation
Get PDFBackground: African evaluation has been at the forefront of innovating the praxis of decolonizing evaluation, especially through ‘Made in Africa Evaluation’ (MAE) and related efforts. Still, there is a wealth of additional African epistemologies and philosophical paradigms which either have not been adequately discussed in the literature, or have not yet made their way into the discourse and practice of MAE.
Purpose: The purpose of this conceptual paper is to propose a theoretical framework that can be used to further inform indigenous and decolonizing evaluation approaches in African contexts and beyond. Specifically, we address the often-cited notion of ubuntu, informed by African philosophical literature beyond the field of evaluation, and we propose Sylvia Tamale’s decolonizing and Afro-feminist lens as a complementary philosophical framing with great potential applications in indigenous and decolonizing evaluation in African contexts and beyond.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: This conceptual study draws on philosophical literature from African philosophy and political science to weave together notions of ubuntu with decolonizing Afro-Feminism.
Data Collection and Analysis: Not applicable.
Findings: We propose that a decolonizing, indigenous evaluation approach rooted in Ubuntu and Afro-Feminism would: question categorial, dichotomous, hierarchical logics (e.g., methodological hierarchical hegemonies); recognize masculinist, imperialist, modernist ideals inscribed in institutions (e.g., via government rationality, therefore also via evaluation); foreground intersectionality; and make room for “the moral economy” and other deeply communitarian framings
Special Issue Editors' Introductory Note: The Why and How of the Decolonization Discourse
No full textThe theme of the special issue decolonizing evaluation: towards a fifth paradigm, an initiative of the International Evaluation academy (IEA), was inspired by the concerns that while evaluation reports largely tell stories of success, on the ground there is minimal change, communities remain impoverished, interventions cause harm to the environment and evaluation allow that to happen
Empowerment Evaluation of Programs Involving Youth: Evaluators’ Perceptions
Get PDFBackground: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth.
Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs.
Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth.
Intervention: Not applicable.
Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1.
Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances
Quality as praxis: A tool for formative meta-evaluation
Get PDFSummative meta-evaluation is known to be more commonly practiced than formative meta-evaluation. While evaluation theorists speak to the importance of formative meta-evaluation, examples of how to do this are rarely specified in the evaluation literature. This paper aims to (1) further explore formative meta-evaluation as a means for quality assurance, with implications for both developing the capacity of evaluators and for advancing evaluation as a field of practice; and (2) to present a model with the intent to move toward a more deliberate formative quality evaluation practice. Discussion focuses on the relationship between evaluator and commissioner and how the development and use of a deliberate approach to formative meta-evaluation, through examination of the proposed model, can lead to a more egalitarian and inclusive approach to defining and promoting evaluation quality. Lastly, formative meta-evaluation is discussed as an important tool for evaluators in exercising professional judgment and for taking an active role in advancing the evaluation field
How did conservation agriculture go to scale? : A case study in utilization-focused evaluation
Get PDFBackground: The Foodgrains Bank has an established record working in agriculture and food security with resource constrained, marginalized farmers in sub-Saharan Africa. The three outcome areas of the Scaling-Up Conservation Agriculture in East Africa (SUCA) Program were: the adoption of conservation agriculture systems, an enabling institutional environment, and the promotion of enabling policies. These program areas were expected to yield intermediate outcomes that, together, would lead to the ultimate outcome of improved food security and sustainable livelihoods for smallholder farming households in East Africa. This case study reports on the end-line evaluation of the five-year program.
Purpose: To illustrate the overlap between utilization-focused evaluation (UFE) and collaborative approaches to evaluation (CAE). The case study profiles an agricultural intervention, and explores how the evaluation design accommodated the systemic nature of the program.
Setting: Scaling-Up Conservation Agriculture in East Africa (SUCA) was a five-year program of the Canadian Foodgrains Bank implemented from 2015-2020 to expand the size and scope of Foodgrains Bank’s work in conservation agriculture in East Africa. The program supported local partners with a target of 50,000 male and female farmers practicing a minimum of 2 of 3 conservation agriculture principles, and to improve food security and sustainable livelihoods for 18,000 of these farmers’ households across three countries.
Research design: The Foodgrains Bank was directly involved in the evaluation design through the definition of evaluation uses and key evaluation questions. Eleven implementing partners in East Africa were involved in primary data collection and some initial analysis.
Data collection and analysis: A mixed method approach was used combining quantitative, qualitative, and participatory / visual data collection tools. A robust, intersectional gender lens was applied to the data collection instruments in the form of gender disaggregated data collection and gender-focused questions across most data collection instruments.
Findings: The collaborative process confirmed a sense of ownership by the primary evaluation users over the evaluation design. The evaluation design combined outcome and learning uses that took advantages of the implementing organizations’ commitment to learning. The findings demonstrated the value of the program and produced a framework illustrating the multi-disciplinary approach underlying its success
Conditions to Consider in the Use of Randomized Experimental Designs in Evaluation
Get PDFDebates about the role of randomized experiments in evaluation have been heated at times, which likely has not facilitated and possibly has hindered thoughtful judgments about whether and when to use a randomized experimental design. The challenges of thoughtful deliberation may be especially great for funders and others who influence the choice of an evaluation design but are not immersed in methodological literatures. The current paper offers a non-technical summary of general factors to take into consideration when determining the appropriateness of a randomized design in a forthcoming evaluation or set of evaluations. Four general conditions are described that should be considered with respect to the specific context for the upcoming evaluation(s). These are, first, the expected value of the information that a well-implemented experiment can provide in the specific context; second, the legal and ethical issues that apply in the circumstances at hand; third, the practical constraints (or facilitating factors) that would apply to a randomized experiment in that context; and fourth, the likely value of the experimental findings in relation to and as part of a portfolio of evaluative studies in the specific context
Editor's Note
Get PDFI would like to start off this editorial note by offering my sincere thanks and appreciation to all the authors and reviewers that have entrusted the Journal of MultiDisciplinary Evaluation (JMDE) with your scholarship, especially this year as we’ve worked to transition our journal. Your patience and encouragement have meant the world to those of us on the editorial team producing this important resource
Competitive champions versus cooperative advocates: Understanding advocates for evaluation
Get PDFBackground: Evaluation offers non-profit organizations an opportunity to improve their services, demonstrate achievements, and be accountable. The extant literature identifies individuals who can enhance the uptake of evaluation as evaluation champions. However, a paucity of detail is available regarding how to identify them and the support they require.
Purpose: This research investigated the characteristics and motivations of evaluation champions and examined how they promoted and embedded evaluation in an organizational system.
Setting: Australian human and social service non-profit organizations.
Research design: Drawing upon the literature and social interdependence theory, the research took an interpretivist perspective to collaboratively generate knowledge about evaluation champions. The aim was to understand and develop a reconstruction of the characteristics of individuals. This article constitutes a component of a larger research project.
Data Collection and Analysis: This research used purposive sampling to recruit champions working in Australian non-profit organizations, who were identified via descriptive criteria gleaned from a literature review. The research involved interviewing 17 champions, four of whom also participated in organizational case studies. Analysis of the semi-structured interviews and case studies generated information about the activities, strategies, motivations, and attributes of individuals who championed and advocated for evaluation.
Findings: This article argues that evaluation advocates is a preferable descriptor when attempting to embed evaluation by cultivating mutually beneficial interactions and cooperative working relationships. This research defines evaluation advocates as individuals who motivate others and provide energy, interest, and enthusiasm by connecting evaluation with colleagues’ personal aspirations and the organizational goals to make judgements about effectiveness. This article includes a field guide to facilitate evaluation advocates’ identification, recruitment, support, and development
Intangible Outcomes: The Importance and Current Neglect Within Evaluation Practice
Get PDFHuman life – and therefore the scope of human goals – includes dimensions that are both visible and countable (e.g., money, weight, attendance or tested proficiency) as well the invisible and intangible (e.g., hope, trust, faith, love, joy, peace). Furthermore, the visible and tangible aspects of life are intrinsically connected to and dependent on the invisible and intangible aspects - much as the visible branches and fruit of a tree are connected to and dependent on an underlying and hidden root structure. While the importance of intangibles can be understood intuitively, it can also be illustrated: 73% of all charitable giving in the U.S. goes to organizations that are explicitly religious, and 118,280 nonprofit organizations are so strongly identified with the intangibles of hope, trust, faith, love, joy, peace that they included one of these words in their name. While the intangible realities of human life are explicitly relevant to a large proportion of organizations we seek to serve, it is essentially ignored by current evaluation practice: only 10 articles within the American Journal of Evaluation, New Directions in Evaluation and Journal of Multidisciplinary Evaluation included even minimal reference to the most common intangibles. New evaluation theory and methodology to address this gap will be needed, and cross-disciplinary exploration with psychology, philosophy and sociology should guide this development. In the meantime, useful questions about intangibles can be drawn from the AEA guiding principles and addressing these can provide a useful starting point for evaluators seeking to consider intangibles within their evaluations.
Concepts and Contexts of Creative Evaluation Approaches
Get PDFBackground: Creative Evaluation (CE) is an unformed constellation of evaluation approaches that is based on varied understandings of creativity. Although creativity in evaluation has been consistently valued and needed by evaluation practitioners and researchers, the use of the term CE is currently limited in its applications as it has been developed inconsistently and often in isolation to the broader evaluation practice. The authors review examples where the term CE is being used, to present this constellation of approaches and group them under concepts and contexts, aiming at fostering an exploration into CE.
Purpose: This paper examines the use of term Creative Evaluation by reviewing a set of examples from varied settings like educational programmes, cultural industries and social work and to present the reader with a better understanding about the contexts and concepts associated with CE. The authors discuss the transdiciplinary nature of evaluation along with a review of historical developments in evaluation theory and practice that help identify the emergence of responsive, flexible and problem-solving driven approaches to evaluation that can showcase a growing connection between creativity and evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Desk review was utilised for the preparation of this paper.
Findings: There are different conceptualisations of the term CE applied in a variety of disciplinary and practice contexts. The term CE is used to describe both the process of applying creative thinking and the process of employing creative methods in the evaluation process. Creative thinking indicates a broader application of creativity that shapes the evaluation process and often results in the development of novel methods while creative methods indicate a more targeted use of methods that employ creative practices to achieve a specific evaluation goal like increasing engagement in the evaluation process