Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Evaluation Policy and Organizational Evaluation Capacity Building: Application of an Ecological Framework across Cultural Contexts
Get PDFBackground: Research on the role and effects of evaluation policy is limited. Some research on the policy’s role in enhancing organizational evaluation capacity (EC) is beginning to accrue but to date it has been limited largely to global Western evaluation contexts.
Purpose: We employed an ecological conceptual framework arising from our own empirical research to explore the interface between evaluation policy and EC in non-western contexts. We asked—To what extent does this framework resonate across these contexts? In the selected non-Western context, what are the salient variables moderating the relationship between policy and EC in the selected contexts? Are there differences across countries?
Setting: The present research is focused on perceptions about evaluation culture and experiences in two countries situated in the Middle East and North Africa (MENA) region, namely Turkey and Jordan.
Intervention: Not applicable.
Research design: We conducted focus groups within the respective countries with a combined total of 18 participants associated with country-level voluntary organizations for professional evaluation (VOPE). Participants worked in government, non-governmental aid agencies, universities and private sector organizations.
Data collection and analysis: We introduced the focus group participants to our ecological framework and then guided the conversation using semi-structured questions. Data were audio-recorded, transcribed and subsequently thematically analyzed using NVivo.
Findings: The ecological framework was found to resonate well but the findings were weighted heavily toward macro-level contextual variables. Even though important contextual and cultural differences between Turkey and Jordan were evident, leadership emerged as a significant meso-level moderating variable in both settings. The discussion of the results included implications for ongoing research
A Review of the DAC Evaluation Criteria: Upgrading Efficiency
Get PDFBackground: Efficiency has remained for long time the Cinderella of evaluation practice. According to several meta-evaluations only a small number of evaluations include robust and meaningful assessments in this field.
Purpose: Following the review of the Development Assistance Committee’s (DAC) evaluation criteria this article reviews the main causes of efficiency remaining in a secondary role in evaluation practices and proposes: 1) upgrading it through broadening the current economic definition of efficiency, and 2) proposing a set of four evaluation dimensions revolving around the principle of sound financial management, social perceived value, result-based budgeting and interconnectedness with other criteria.
Setting: Not applicable.
Intervention: Not applicable.
Research design: Literature review and empirical pilot testing of proposed evaluation methodology.
Data collection & analysis: Qualitative analysis.
Findings: This paper calls for a shift in focus from efficiency to a broader enriched principle of sound financial management including a strong partner-based focus and highlighting management as a pre- condition “sine qua non” for an intervention to be efficient. Efficiency assessments would then be twofold: managerial (procedures, policies and practices that lay out the requirements for efficiency to take place) and substantial (context, circumstances and reasoning for the existing budget balance between results and target groups). The key aspects of the proposal involve four proposed dimensions to be analysed under this renewed criteria that imply audits and evaluations come much closer and look at each other
Evaluating Public Participation in a Project Plan Review: A Nigerian Case Study
No full textBackground: Rooted in national and international laws regarding project planning and implementation is public participation. However, it is unclear whether public projects are enabling sufficient public input or are likely to be able to meet future management planning needs; particularly in developing countries.
Purpose: We assessed people’s experiences when contributing to a public project decision-making in order to understand the strength, weaknesses, opportunities and threat to effective public participation.
Setting: We conducted this assessment with a sample of people who contributed to a public project planning and review in a Nigerian city.
Intervention: Not applicable.
Research design: Appraisal criteria are based on the principles of public participation as laid down in the law and consists of the following elements: respondents’ profile, their involvement in the project; purpose of participation, availability of information, feedback mechanism and overall view of the participatory planning process. Information collected consists both quantitative and qualitative data and these were analysed using descriptive statistics and narrative techniques of reporting.
Findings: Findings show that public participation was far below the minimum requirement of the law and not demographically representative. The most important reason respondents participated was to protect an interest in land, although some saw participation as a democratic right. Results show that attending public hearings was the commonest way of participation in a project review. Nevertheless, three-quarters of the respondents thought the final plan did not take their observations and advice into consideration. Respondents confirmed that the process was reasonably notified with opportunities for consultation meetings. Nevertheless, findings suggest some bias actions as significant proportions of respondents held absence of transparency and political interference flawed the project planning and review process
Journey “Back Over the Line”: Critical Pedagogies of Curriculum Evaluation
Get PDFBackground: We re-trace our liberatory journey in developing a Critical Framework of Review to evaluate K-12 Filipina/x/o American curricula. Our framework is rooted in our positionality and epistemology as Filipina educational scholars engaged in confronting oppression that impacts our community. It responds to the need for evaluation methods grounded in culturally responsive and critical pedagogies.
Purpose: The purpose is to provide a critical and cultural method of evaluation to assess curriculum and pedagogy of, by, and about our communities.
Setting: The research takes place in the Filipinx/a/o American community in the United States. The authors are from three academic institutions in California, Hawai‘i and the Philippines.
Intervention: Our Critical Framework of Review attempts to counter the predominance of Eurocentric, male, objective, and uncritical models of curricula evaluation.
Research design: This research deconstructs how we developed and applied our framework, which was used to evaluate thirty-three Filipina/x/o American K-12 curricula in critical content, critical instruction, and critical impact, by asking 20 questions that reflected critical and cultural theories and pedagogies.
Data collection and analysis: We asked: Who and what informed our evaluation framework? How was it developed? How do we use it? How could our framework be further applied? We referenced diverse scholars and used critical race, feminist, indigenous, and deolonizing pedagogies as guidelines to establish our evaluation framework and standards.
Findings: The framework is an example of standards-based and responsive-based evaluation with a checklist of indicators to evaluate curricula for culture, race, positionality, and social justice. Although created for Filipina/x/o, the framework can be used to evaluate curriculum for other marginalized groups
Learning How to Learn in Sustainability Transitions Projects: The Potential Contribution of Developmental Evaluation
Get PDFBackground: Community-based sustainability transitions projects are increasingly being considered for their potential as policy delivery vehicles for the UK government Climate Change Act commitments. At the same time, project funders seek reassurances that their investments are relevant in helping communities mitigate, and adapt to, the effects of climate change. Despite this increased pressure, recent research suggests that evaluations of such sustainability transitions projects have, on the one hand variable impacts, or impacts that are of an inadequate scale, duration, or type, or on the other, that project staff lack the capacity or resources to undertake monitoring and evaluation to the degree of rigour expected by policy makers and funders.
Purpose: This article reports on an extended case study of a fully-funded five year community-based sustainability transitions project in Leicestershire, England. In particular, it reviews the deployment of developmental evaluation (DE) methods in an attempt to capture the project team’s learning about doing community-based sustainability work.
Setting: A funded community-based sustainability transitions project in a south Leicestershire market town.
Intervention: Developmental evaluation methods were used to capture project-based learning as a resource for project innovation and adaptation.
Research Design: Phronetic case study.
Data Collection and Analysis: Participant-observation, action research, focus and special issue group facilitation.
Findings: Use of a developmental evaluation method identified key learning points for the project actors; these focused on how the project had adapted to the complexities of the operating environment through innovations in second-order learning or learning how to learn. The paper makes recommendations for the design and funding arrangements of community-based sustainability transitions initiatives and developmental evaluation is endorsed as a viable and promising adjunct to more traditional impact, economic, and process evaluation methodologies
Refining and Measuring the Construct of Evaluative Thinking: An Exploratory Factor Analysis of the Evaluative Thinking Inventory
Get PDFBackground: Evaluative thinking has emerged as a key construct in evaluation, especially for evaluation practitioners and researchers interested in evaluation capacity building (ECB). Yet, despite increasing calls for more research on evaluation and, more specifically, for more research on ECB, until recently little empirical inquiry on the dimensions of evaluative thinking has been conducted.
Purpose: To address that lack, the purpose of the study presented in this paper is to refine the construct of evaluative thinking by exploring its underlying dimensions and to ascertain the internal consistency of an instrument developed to measure evaluative thinking, the Evaluative Thinking Inventory (ETI).
Setting: The ETI was developed as part of an ECB initiative focused on non-formal science, engineering, technology, and math (STEM) education in the United States, and was tested as part of a study focused on evaluating gifted education programs, also in the United States.
Intervention: Not applicable.
Research design: Survey research and exploratory factor analysis (EFA).
Data collection & analysis: The ETI was administered to participants in a study measuring the effectiveness of a tool used to conduct internal evaluations of gifted education programs. SPSS was used to conduct an EFA on 96 completed ETIs. Cronbach’s alpha was used to estimate the internal consistency of the instrument.
Findings: The analysis of the ETI revealed a two-factor model of evaluative thinking (i.e., believe in and practice evaluation and pose thoughtful questions and seek alternatives). This study also provided internal consistency evidence for the ETI showing alpha reliabilities for the two factors ranging from 0.80 to 0.82. The ETI has potentially wide applicability in research and practice in ECB and in the field of evaluation more generally
Adopting Tools from Cost and Management Accounting to Improve the Manner in Which Costs in Social Programs are Analyzed and Evaluated
Get PDFBackground: Managing programs in an environment where financial resources are limited, budget cuts are a reality, and external funding is now fiercely competitive, necessitate that both program administrators and program evaluators have a better understanding of program costs, so that financial resources can be optimized for societal good. This requires serious analysis of cost behavior and a proper understanding of the relationship between a program's variable costs and fixed costs since these costs have implications for clients fees and the number of clients that can be served. These types of analyses are quite routine in the profitability sector, but are considerably underutilized in other sectors.
Purpose: This paper will explain how several common strategic management tools from cost and management accounting can be used to present more meaningful and useful cost information, so that social program decision-making and cost-inclusive evaluations can be enhanced.
Setting: N/A.
Intervention: N/A.
Research Design: A desk review was utilized for the discussion of the cost and management accounting concepts and tools outlined in this paper. The paper illustrates how the toolkit of economic evaluation tools can be enhanced by adding tools from cost and management accounting to enhance strategic decision-making.
Findings: This paper concludes by noting that program sustainability must be the new name of the game. This necessitates that program administrators and program evaluators start to analyze and evaluate program costs differently. Much work is needed to move towards a different philosophy of thinking with regards to program costs. Program administrators and program evaluators must therefore rise to the challenge and embrace cost analytical methodologies from other disciplines since the use of such methodologies can be beneficial to all concerned
Process Evaluation to Document Crucial Moments in Development of the National Neurological Conditions Surveillance System at the U.S. Centers for Disease Control and Prevention
Get PDFBackground: Neurological conditions or disorders strike roughly 50 million Americans annually but accurate and comprehensive national estimates for many of these conditions are not available. In 2019, Congress provided $5 million to Centers for Disease Control and Prevention (CDC) to establish the National Neurological Conditions Surveillance System (NNCSS). CDC focused initial activities on multiple sclerosis and Parkinson’s disease.
Purpose: We conducted a process evaluation to document and understand multifaceted work to implement a new surveillance activity for two neurological conditions.
Setting: We conducted this evaluation with government personnel internal to the Center for Surveillance, Epidemiology, and Laboratory Services at the Centers for Disease Control and Prevention in Atlanta, GA.
Intervention: A new public health surveillance activity for two neurological conditions, multiple sclerosis and Parkinson’s disease, that uses existing data resources and systems.
Research design: The evaluation included interviews with CDC personnel and review of administrative and programmatic information. Data were analyzed and interpreted to identify crucial moments in the first year of funded work on NNCSS. The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. The findings can be used to guide work to enhance surveillance for many neurological conditions.
Findings: The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. While collaboration is a cornerstone of public health practice, it is not always well-documented in planning or implementation of surveillance or other data-related activities
Evaluating in a Fragmented Society
Get PDFBackground: Over decades American society has become increasingly fragmented, distrusting, and unequal. Distrust and inequality interact with institutions performing improperly to weaken the society.
Purpose: To suggest ways to strengthen evaluation’s role in a changing society
Setting: Evaluation has entered a post normal phase where evaluations are losing credibility and effectiveness.
Intervention: Analyze the changing society and suggest adjustments that evaluators might make.
Research design: Collate and synthesize empirical studies about society and the implications for evaluators.
Data collection and analysis: Collect and interpret seminal empirical economic, sociological, and political studies of beliefs and inequality in the United States.
Findings: To strengthen the potency of evaluations of any type, evaluators could act as moral fiduciaries, practice transparency, cultivate cognitive empathy, focus on deep stories and deep values, and mitigate inequalities in the evaluation space. They can act as critics of evaluation practices inside and outside the evaluation space. They should avoid technical, social, and situational biases, including racism, sexism, and conflicts of interest, to increase the honesty and credibility of evaluations. They should not allow career concerns to prevent them from doing the right thing. These professional ethics and practices can be applied singly or collectively to most evaluation approaches to strengthen the evaluator’s role and address major societal problems
Program Logic Foundations: Putting the Logic Back into Program Logic
Get PDFBackground: Program logic is one of the most used tools by the public policy evaluator. There is, however, little explanation in the evaluation literature about the logical foundations of program logic or discussion of how it may be determined if a program is logical. This paper was born on a long journey that started with program logic and ended with the logic of evaluation. Consistent throughout was the idea that the discipline of program evaluation is a pragmatic one, concerned with applied social science and effective action in complex, adaptive systems. It gradually became the central claim of this paper that evidence-based policy requires sound reasoning more urgently than further development and testing of scientific theory. This was difficult to reconcile with the observation that much evaluation was conducted within a scientific paradigm, concerned with the development and testing of various types of theory.
Purpose: This paper demonstrates the benefits of considering the core essence of a program to be a proposition about the value of a course of action. This contrasts with a research-based paradigm in which programs are considered to be a type of theory, and in which experimental and theory-driven evaluations are conducted. Experimental approaches focus on internal validity of knowledge claims about programs and on discovering stable cause and effect relationships—or, colloquially, ‘what works?’. Theory-driven approaches tend to focus on external validity and in the case of the realist approach, the search for transfactual causal mechanisms—extending the ‘what works’ mantra to include ‘for whom and in what circumstances’. On both approaches, evaluation aspires to be a scientific pursuit for obtaining knowledge of general laws of phenomena, or in the case of realists, replicable context-mechanism-outcome configurations. This paper presents and seeks to justify an approach rooted in logic, and that supports anyone to engage in a reasonable and democratic deliberation about the value of a course of action.
It is consistent with systems thinking, complexity and the associated limits to certainty for determining the value of a proposed, or actual, course of action in the social world. It suggests that evaluation should learn from the past and have an eye toward the future, but that it would be most beneficial if concerned with evaluating in the present, in addressing the question ‘is this a good idea here and now?
Setting: Not applicable.
Intervention: Not applicable
Research design: Not applicable.
Findings: In seeking foundations of program logic, this paper exposes roots that extend far deeper than the post-enlightenment, positivist and post-positivist social science search for stable cause and effect relationships. These roots lie in the 4th century BCE with Aristotle’s ‘enthymeme’. The exploration leads to conclusions about the need for a greater focus on logic and reasoning in the design and evaluation of programs and interventions for the public good. Science and research are shown to play a crucial role in providing reasons or warrants to support a claim about the value of a course of action; however, one subordinate to the alpha-discipline of logical evaluation and decision making that must consider what is feasible given the context, capability and capacity available, not to mention values and ethics. Program Design Logic (PDL) is presented as an accessible and incremental innovation that may be used to determine if a program makes sense ‘on paper’ in the design stage as well as ‘in reality’ during delivery. It is based on a configurationalist theory of causality and the concepts of ‘necessary’ and ‘sufficient’ conditions. It is intended to guide deliberation and decision making across the life cycle of any intervention intended for the public good