Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Competencies for Evaluation as a Civic Science
Full text linkThis paper relocates the practice of evaluation from its traditional intellectual home in the applied social sciences to an interdisciplinary intellectual community that draws on concepts and practices from civic studies, political science, and studies of coproduction and citizen engagement in public administration and management. It offers an overview of the competencies for evaluation practice once relocated in this way
Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India
Full text linkThe movement towards systems thinking and complexity-informed monitoring and evaluation has been ongoing for some time. Covid-19 has accelerated this shift and increased the salience of contextually aware, adaptive forms of evaluation. Drawing from our experience over three years of providing monitoring, evaluation, research and learning (MERL) to a large international philanthropic organization’s health systems portfolio of work in India, we identify three types of evaluator capacities and associated set of competencies that derive from complexity-driven practice in a developing country.
The first type of capacity relates to deriving evidence in contexts where there are very few traditional forms of data. The MERL portfolio related to health systems outcomes of access, financial protection, and equity even as the health programs themselves were evolving in a period of rapid transformation and dynamism of policy priorities and implementation mechanisms. This required an integration of traditional performance indicators drawn from government datasets with qualitative ‘signals’ focused on drivers of change within the system. Assessment of signals in turn required synthesizing different sources of information, including informal channels of obtaining evidence such as partner meetings or government-sponsored events. The evaluating team thus consciously empowered different kinds of researchers with differential expertise with the goal of building a much more pixelated picture. The goal was not to identify a single source of truth but rather a patchwork of validated information where the relevance of different pieces of data were dependent on evolving outcomes of interest.
The second set of competencies related to the skills required to play a convening role for donors and implementing partners, supporting better understanding of the changing operating context and help inform decision-making by program officers and partners. This involved building and sustaining relationships across different stakeholders at different stages of the project – from proposal development to review. Competencies relating to effective dialogue and developing an understanding of the core interests of international, national and sub-national partners as well as international donors and experts could only be developed iteratively and over time, but this was crucial in a distributed health decision-making ecosystem like India.
The third and final set of competencies relate to operational adaptiveness, while evaluating an ecosystem with few constants. This can be the hardest competency to acquire because it is the farthest from the traditional notions embedded in the training of evaluation scientists and MERL practitioners. We found that the degree of buy-in and commitment to previously agreed upon frames of reference for evaluation can be shifted by changes in personnel or internal organizational structures. These shifts can lead to chain reactions of mismatched expectations that needed to be understood and managed in real time by MERL partners. The pandemic further created a natural experiment that on the one hand required a reexamination of program priorities and on the other depended on reliability of donor support.
Each of these three types of capacities – synthesizing nuanced evidence for adaptive action, relationship building and communication, and managing operational discontinuities are in fact inter-dependent. Building evaluator competencies isn’t simply about capacity-building but rather a recognition of the diversity of skills and worldviews that need to be encompassed within our monitoring and evaluation functions for today’s complex, discontinuous health systems
Evaluator Education for the Twenty-First Century: The Centrality of Developing Evaluators’ Interpersonal Competencies
Full text linkThis paper first traces the development of competencies for professional evaluators and identifies interpersonal competencies as a key component of evaluator education in the future. It then outlines two roles for evaluators that may affect the application of interpersonal competencies, discusses four ideas for their development, and briefly addresses the possibility of evaluation capacity building. It concludes with a brief discussion of where and how evaluator training or education might effectively teach these competencies and know that people had learned them
The Program Evaluation Standards in Evaluation Scholarship and Practice
Full text linkBackground: The Program Evaluation Standards that were developed and approved by the Joint Committee on Standards for Educational Evaluation have served as a resource to the broader evaluation field for over four decades. However, little evidence has been collected regarding the extent to which the standards have influenced the field through scholarship or professional practice.
Purpose: This study seeks to estimate the prevalence of the Program Evaluation Standards in evaluation scholarship and professional practice.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: The study combines a systematic review of evaluation literature with a survey of American Evaluation Association (AEA) and Canadian Evaluation Society (CES) members.
Data Collection and Analysis: A systematic review of articles published in 14 evaluation-specific journals from 2010 to 2020 was conducted to identify and typify articles citing the standards. Additionally, AEA and CES members were surveyed, with a focus on knowledge and use of the standards. Descriptive analyses are presented to quantify the prevalence of the standards in evaluation scholarship and practice, respectively.
Findings: The systematic review revealed that 4.48% of the 4,460 articles published in 14 evaluation-specific journals from 2010 to 2020 contained some use of the standards. Survey results show that 53.14% of AEA members and 67.12% of CES members are familiar with the standards and that, among those with knowledge of the standards, most AEA (67.67%) and CES (71.74%) members use them at least “occasionally” in their professional work, education, and scholarship activities
The Pursuit of Social, Economic, and Environmental Justice through Evaluation: Learning from Indigenous Scholars and the Fifth Branch of the Evaluation Theory Tree
Full text linkThe world faces a climate crisis that calls for increased attention to social, economic, and environmental justice. Non-Indigenous evaluators can choose to continue their work business-as-usual, or they can choose to learn from Indigenous evaluators. The exclusion of Indigenous scholars from the evaluation theory tree results in an opportunity lost to improve theory and practice across all communities. A fifth branch on the evaluation theory tree, Context and Needs, aligns with the Indigenous paradigm and serves to stimulate questions about evaluation theory and practice that inherently address issues of justice and the importance of recognizing the interconnectedness of all living and nonliving things
Beyond Ubuntu: Nnoboa and Sankofa as Decolonizing and Indigenous Evaluation Epistemic Foundations from Ghana
Full text linkBackground: Evaluation is an increasingly vital component of community and economic development projects in Africa. Yet questions remain about how relevant most evaluation approaches for the African evaluation context. Within the ‘Made in Africa’ (MAE) approach, ubuntu is frequently cited as an African philosophical concept with salience to MAE. There is a need to further expand and explicate other African philosophies that can serve as epistemological guideposts for African evaluation—and other decolonizing, indigenous evaluation approaches more broadly.
Purpose: Drawing on Ghanaian epistemologies and frameworks, the purpose of this paper is to propose the Nnoboa system of communal collaboration in farming and industry, as well as the notion of Sankofa as a traditional philosophical concept that irrupts and challenges hegemonic Eurocentric notions of the linearity of time, to yield a Ghanaian indigenous knowledge of evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: This conceptual study draws on literature on culturally responsive evaluation (CRE), MAE, and (from beyond the field of evaluation), descriptions of Nnoboa and Sankofa to propose a conceptual synthesis applicable to decolonizing, indigenous evaluation.
Data Collection and Analysis: Not applicable.
Findings: We propose that Nnoboa and Sankofa represent an addition to the decolonizing and indigenous evaluation knowledge base, building on and going beyond the reliance of CRE and MAE in ubuntu. We propose this Ghanaian approach has potential applications across MAE and CRE more broadly
Decolonizing Evaluation: Truth, Power, and the Global Evaluation Knowledge Base
Full text linkEvaluations play a critical role in shaping the way international development interventions are designed and managed. This contributes to the fact that there are intrinsic power dynamics at play during the production of evaluation knowledge. In the wake of growing commitments to deal with the repercussions of a colonial past and a prolonged history of white supremacy, there has been growing recognition of the need to decolonize evaluation theory and practice to help foster equality and ensure that evaluation knowledge is generated, shared, and understood universally. Increased efforts are being made to understand the risks involved if evaluations fail to utilize decolonized evaluation methods and approaches. We are faced with questions such as: How can we decolonize evaluation and disrupt unequal power relations? How can the evaluation process itself be transformative and an opportunity for co-liberation? What practical tools and steps facilitate power sharing? How can evaluation be used to advance decolonization and social justice?
This paper describes the connection between truth and power and how that relates to the evaluation and decolonization debates. It provides potential starting points for decolonizing evaluation practice, including some examples where this has already been tested. To conclude it focuses on the universalisation of evaluation knowledge and the need to ensure that evaluation knowledge is translated and disseminated via inclusive forms of communication to ensure that learning can be better understood and hence translated into action orientated practice from grassroot to government level.
The aim of this paper is to advance the discussion on decolonizing evaluation practice and provide some potential ways forward in terms of transforming evaluation theory and practice.
Keywords
Decolonizing evaluation, development, results, learning, accountabilit
A Complexity-Based Plan for Evaluating Transformation
Full text linkThis article presents a case for more rigorous application of complexity science in our efforts to evaluate activity that seeks to bring about transformative change. It builds on the work that is already going on in the evaluation community. Three constructs from complexity science are employed – sensitive dependence, emergence, and social attractors. The paper argues that if–then logic is recommended for small-scale change within transformation efforts, but that to evaluate transformation writ large, data from if–then evaluation must be embedded in, and interpreted in terms of, complex behavior. Methodologies for evaluating within this framework are presented. The argument is linked to a definition of transformation that is multidimensional, non-linear, and measurable. The paper is built around a generic model of transformational change and shows how that model can be customized for specific transformation scenarios. It also shows how evaluation with respect to complexity can be accomplished with methodologies that are well known and well-practiced in the evaluation community.
Background: “Transformation” is becoming an important concept in evaluation.
Purpose: Promote a complexity approach to evaluating transformation.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Tracing the Transnational Influence of Jennifer Greene’s Ideas: Research on Evaluation Theory Across Countries:
Full text linkBackground: The empirical study of evaluation theories has not been a strong focus of recent research on evaluation (RoE) efforts. Nevertheless, evaluation scholars should investigate how evaluation-related ideas are exchanged across the world in this increasingly globalizing field.
Purpose: The study examines the transnational influence of American evaluation theorist Jennifer Greene’s ideas. Simultaneously, this RoE study details a distinct methodological strategy for conducting research on evaluation theory.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Mixed-methods citation analysis—comprising both quantitative and qualitative strategies—is used to investigate how Greene’s program evaluation Theories have impacted the field across countries. The study pursues a contextualized, rich understanding of the contexts in which Greene's ideas are used to observe the imprint of her work on evaluators theorizing and practicing outside of the United States.
Data Collection and Analysis: Citation counts for five Greene publications were analyzed across three databases. The analysis included a) to what extent countries or regions were represented, b) trends in represented countries, and c) how citations changed over time. The qualitative analysis entailed review of full-text publications of international citations to understand how Greene's concepts were engaged and shaped evaluation outside the United States.
Findings: Evaluators across the world noted Greene's ideas with 42 countries represented in the citations of her five publications. The study's qualitative analysis revealed the significance of Greene's enduring argument regarding the necessity of engaging values in evaluators' practice
In Plain Sight or Just Plain Obscured?: A Review of Professional Evaluation Associations’ Frameworks for Evaluation Practice Supporting Equity, Diversity and Inclusion (EDI)
Full text linkWith an increasing focus on integrating equity, diversity and inclusion (EDI) in evaluation practice and products, there is an accompanying need to examine what structural supports exist that are provided by professional evaluation associations. This contribution systematically examines the competencies of five professional evaluation associations from Africa, Australia, Europe and North America to identify how evaluators can align integrating EDI in thei evaluation practice to professional competency domains. Also offered is a summary of thoughts on training opportunities for integrating EDI given review findings. Professional evaluation association websites were reviewed during May through to July of 2022. Content was downloaded into an Excel spreadsheet, and organized for EDI review purposes by competency domains, subdomains which occurred during August and September of 2022. The presentation of EDI content in evaluator competencies was found to be highly varied; variations were found in tone/theme, principles, and domains and subdomains