Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
What do we know about how the Program Evaluation Standards are used in public health?
Full text linkBackground: Released by the Centers for Disease Control and Prevention (CDC), Framework for Program Evaluation in Public Health prominently features the Program Evaluation Standards (1999). The Program Evaluation Standards (PES) include 30 statements in five domains: utility, feasibility, propriety, accuracy, and evaluation accountability. Despite decades of attention to the PES among framework users and others, how public health professionals apply these standards in their work is not well understood.
Purpose: The study sought to identify notable commonalities in how the PES are used in public health.
Setting: Application of the PES in evaluative work in public health and allied fields.
Intervention: Not applicable.
Research Design: The study included a search of subscription and nonsubscription sources to identify documents that included explicit content concerning use of standards in evaluative work in public health. Documents identified were screened using predetermined criteria to include or exclude each item in the study. Items included were reviewed and coded using codes developed before examining all documents. For each code, reviewers discussed data from all documents to identify commonalities and variations in application of standards.
Findings: The literature search returned 405 documents to be screened (179 from subscription and 226 from nonsubscription sources). Thirty-eight items were included in the study based on initial screening (11 from subscription and 27 from nonsubscription sources). The study revealed that authors discussed standards as a regular component of evaluation work, but precisely how standards were used was not always explained in detail. Also, authors did not always discuss standards statements but sometimes solely focused on general domains (e.g., feasibility or accuracy). When authors discussed specific statements, they were more descriptive in how they applied the PES (i.e., compared with articles that focused on general domains). Overall, authors placed far greater emphasis on Accuracy and Utility standards, compared with Propriety, Evaluation Accountability, or Feasibility. In many cases, authors used the PES in combination with other resources (e.g., checklists, guidelines, or other standards). Although program evaluation is crucial to public health practice, the mechanics of how professionals consider, integrate, or use evaluation standards is not fully understood
Evaluation Capture, Evaluator Resilience, and the Need for Competencies of Evaluators
Full text linkEvaluation capture is the situation in which evaluations and evaluators are surrounded by protocols, guidelines, standards, norms, criteria, templates, oversight and reviews from national and supranational organizations, governments, NGO’s and agencies that ask for and finance monitoring and evaluation activities; these protocols and guidelines are sometimes labelled or part of ‘evaluation policies.’ The paper describes several developments in evaluation that contribute to this capture, referred to as mechanisms. The paper also addresses the question how to tackle this issue. The focus is on the role of the evaluator’s resilience, while several competences of evaluators to realize that are also addressed
Glocal Evaluation Competencies for Learning As We Go: Zooming in and zooming out to connect system-level solutions to local beneficiaries
Full text linkIdentifying essential competencies for evaluators has received significant attention in recent years yet practical examples of how to apply competencies to real-time learning in complex environments are lacking. In particular, the experience of those at the local level - ultimate beneficiary individuals (UBIs) - can get lost when evaluations take a systems perspective. Experienced evaluators share how Learning as we go is used to describe utilization-focused developmental evaluation embedding evaluative thinking and building capacity in public sector programs, that support learning and adaptation to improve the lives of those most impacted by inequitable and unsustainable global systems
Evaluation Competencies and Evaluation Use: Some Reflections
Full text linkThis article aimed to identify the emphasis on evaluation use within evaluation competency frameworks. A review of evaluation competency frameworks shows an underlying focus on evaluation use in all frameworks. Nevertheless, specific competencies explicitly focusing on evaluation use were incorporated in more recent frameworks, reflecting the increasing attention to evaluation use. A theory of change for evaluation use is proposed depicting the role of competencies of evaluators and critical users. The article argues for more emphasis on directly use related competencies and extending the competencies beyond evaluators to users. It proposes establishing standardized and up-to-date evaluation training in academic institutions to professionalize evaluation and thereby promote the integration of evaluation in development interventions.
Keywords: competency frameworks, evaluation competencies, use of evaluation, evaluation training, development interventions
Deconstructing the Imperial Episteme: Decolonizing Knowledge Production in Program Evaluation
Full text linkScholars, practitioners, and activists have all contributed to the discussion of decolonization of evaluation practice in recent years as attention has increasingly focused on the persistent harms of colonization. While these discussions have led to the development of evaluation frameworks rooted in Indigenous and locally-situated understandings, values, and methods, little attention has been paid to the colonial origins of Western-based evaluation practices that continue to pervade the field. This article seeks to contribute to the conversation about decolonization by focusing on the ways in which Western social theory, born of colonizing nations, has been influenced by the processes of colonization. Drawing on scholars and theorists from the Global South, this article highlights specific apparatuses for dismantling imperial ways of thinking and ways of knowing, and proposes a path forward for evaluators who wish to grapple with the deeply imperial epistemological roots of our field of practice
"Best Tradition": CREATE, JCSEE and the Program Evaluation Standards
Full text linkBackground: Evaluation “is a task in the best tradition of the most abstract theoretical science as well as the most practical applied science” (Scriven, 1968, p .9). The Program Evaluation Standards of the Joint Committee on Standards for Educational Evaluation (JCSEE) operationalize the theoretical aspects of evaluation and, when used, facilitate sound evaluation methods in applied settings. Between the publications of the first and second editions of The Program Evaluation Standards, the Center for Research on Educational Accountability and Teacher Evaluation (CREATE) was funded in 1990 at Western Michigan University with federal monies of $5.2 million, and between 1990 and 1995 by the United States Department of Education, Office of Educational Research and Improvement (OERI). CREATE was established for the betterment of evaluation within the educational context (Stufflebeam, 1991; Stufflebeam & Shinkfield, 1994). CREATE’s mandate and subsequent mission furthered the work of the Program Evaluation Standards and the JCSEE by using the standards in applied settings. Keeping to Scriven’s notion of evaluation as the best tradition, the collaborative work between CREATE and JCSEE is a well-established tradition that furthers the development of theoretical aspects of evaluation and the application of the evaluation standards.
Purpose: Examine CREATE’s impact on the Program Evaluation Standards’ theoretical development and applied use.
Setting: Not applicable.
Intervention: Not applicable
Research Design: Not applicable.
Data Collection and Analysis: Systematic review of the theoretical development and applied use of the Program Evaluation Standards in the books, journal articles, monographs, special papers, meeting minutes, conference programs, and presentations associated with CREATE.
Findings: CREATE has contributed to the operationalization of the theoretical aspects of evaluation with the Program Evaluation Standards and facilitated their use in applied settings. CREATE has also furthered the work of the Personnel Evaluation Standards and the Classroom Assessment Standards (formerly the Student Evaluation Standards). Leading scholars from CREATE and the JCSEE have contributed to the standards since the 1990s. Members of CREATE have published a notable range of books, journal articles, monographs, special papers and conference presentations related to the Program Evaluation Standards. Organizational capacity and shared goals of both the JCSEE and CREATE guided the practical application and theoretical development of the Program Evaluation Standards
Introduction to the Special Issue on the Program Evaluation Standards
Full text linkThis one-page paper introduces the special issue devoted to the Program Evaluation Standards with a brief summary of the purpose of the special issue and an overview of the individual papers is provided
Entering the Ethical Space Between Epistemologies: A Step Toward Decolonizing the Heart and Mind
Full text linkAt a tipping point marked by knowledge fragmentation, the evaluation field’s ability to address the complex problems confronting the world today is threatened. Contributing to this is the field’s narrowed epistemological foundation which places limits on what counts as knowledge, alienating many engaged with it. Privileging empirical knowledge over traditional and revealed (i.e., spiritual) knowledge places the reigning evaluation paradigms at odds with Indigenous paradigms and presents numerous risks to individuals, communities, and ecosystems. Yet, if we enter the space between epistemologies (Emine, 2007), we discover that the founders of western philosophy, Aristotle and Descartes, held many ideas in common with Indigenous philosophies. Here I deconstruct Western thought to show how Aristotle and Descartes upheld an immaterial reality, with Descartes grounding all his understanding in revealed knowledge. After providing a bombardment of multidisciplinary support for revealed knowledge, I argue that embracing the Knowledge Trinity advances the decolonization of evaluator self and mind and provides a new epistemological foundation upon which to construct a Decolonizing Paradigm
Decolonizing Evaluation of Indigenous Guidance and Counseling Approaches: A Review of Selected Evaluated Programs
Full text linkThe concept of Indigenization of research has been increasingly explored in recent studies, with emphasis placed on the ontological, epistemological, and axiological perspectives of Indigenous peoples to find effective solutions to their challenges. This also applies to the evaluation of guidance and counseling approaches in Africa and other nations, where Indigenous therapies are developed based on different philosophical foundations, such as Ubuntu (Africa). Relational ontologies and epistemologies appear to be common across various Indigenous nations in Africa, Australia, Canada, and North America. This article analyzes studies from these regions on evaluations of Indigenous guidance and counseling therapies. The majority of the evaluations use conventional paradigmatic assumptions in their approach, rather than relational models that are participatory and respectful of participants’ worldviews, including the living, non-living, metaphysical, and spiritual aspects of Indigenous people. However, the Indigenous therapeutic programs analyzed in this study incorporate culturally appropriate activities and curricula that align with relational axioms. This article proposes the use of relational models of evaluation to assess Indigenous counseling programs, where researchers can draw conclusions that align with the cultural contexts of the Indigenous people being researched
Evaluation Transformation Implies Its Decolonization
Full text linkTransformational evaluation implies a decolonization process focused on relieving the misery and suffering of the oppressed. To inform such a reorientation, this article describes the challenges of social transformation; probes the links between capitalism, slavery and racism; takes stock of the post-colonial development order; examines the legacy of the evaluation occupation; and recommends new policy directions inspired by indigenous evaluation.