Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
An In-Depth International Comparison of Major Donor Agencies: How Do They Systematically Conduct Country Program Evaluation?
Get PDFBackground: This paper presents an in-depth international comparison of systems and procedures of aid evaluation, focusing on Country Program Evaluation among major donor agencies. The original client of this study is Ministry of Foreign Affairs, Japan (MOFAJ).
Purpose: The purposes of this paper are set as follows: (1) to understand how aid agencies conduct Country Program Evaluation; and (2) to make recommendations for improvement of the current practice of Country Program Evaluation in the aid evaluation community.
Setting: The examined donors include: the World Bank (WB), the Asian Development Bank (ADB), the Inter-American Development Bank (IADB), the United Nations Development Programme (UNDP), the U.S. (USAID), Canada (CIDA), the U.K. (DFID), the Netherlands (IOB), Germany (BMZ), France (Foreign Ministry), and Japan (Ministry of Foreign Affairs (MOFAJ)). In addition, aid agencies conducting respective project evaluation are also examined, and they are JICA (Japan), GTZ and KfW (Germany) and AFD (France).
Intervention: This study presents the result of comparative analysis among those donor agencies in terms of the following viewpoints: (1) evaluation criteria employed; (2) approaches to evaluate “effectiveness” and “impact”; (3) attribution issue; (4) the use of a rating system; and (5) overall evaluative conclusion and integrating methods. All viewpoints are focusing on Country Program Evaluation. One conclusion is that most agencies have been struggling with how to judge the degree and value of their country programs.
Data Collection and Analysis: Mixed methodologies were employed to collect data from the said donor agencies. The analysis was conducted by a systematic procedure consisting of: (i) summarizing information in a comparative table; (ii) trying to make groups/categories based on common characteristics if possible; and (iii) examining and concluding basic thoughts/philosophy which make their differences.
Findings: This study made some new knowledge about how aid agencies conduct Country Program Evaluation and identified several issues remained. Varieties of their practices are observed and it is far from the unified methods agreed. Some remarkable points identified in this study are:(1) Most aid agencies invoke the DAC five evaluation criteria for Country Program Evaluation. (Major exception was USAID); (2) “Strategic relevance” and “coherence/complementarity” are the emerging new criteria; (3) Attribution is still the issue that aid agencies have struggled; and (4) The attitude for introduction of rating system is clearly divided among aid agencies
Using Objective Stylometric Techniques to Evaluate New Testament Authorship
No full textBackground: Bible scholars often debate the authorship of certain books appearing in the New Testament. Purpose: The purpose of the present study was to evaluate New Testament authorship by using stylometric analytical techniques Setting: This research focuses on texts appearing in the New Testament. Intervention: This was an exploratory research on evaluation study with no intervention. Research Design: A powerful, state-of-the-art psychometric model was applied to Biblical text in an effort to identify correlations among word usage and writing style among each of the New Testament books. Data Collection and Analysis: Strong’s Concordance was used to provide original Greek text. Computer programming was necessary to create a worksheet that contained a list of New Testament books, each Greek word appearing in the New Testament, and a count of each word’s appearance relative to each book. Rasch-based Principal Components Analysis of standardized residual correlations was used to map stylistic similarities and differences. Findings: With regard to substantive findings, the gospels (Matthew, Luke, Mark, and John) and the narrative book of Acts were closely correlated. Other texts presented a mix of expected and unexpected findings. With regard to other findings, the technique presented in this study offers a great deal of promise to various research and evaluation practices.  
The Faster Forward Fund
Get PDFDescription and explanation of the soon-to-be-created "Faster Forward Fund," which will be founded by Michael Scriven and will aim at supporting three approaches in the evaluation field:
1. To develop ideas that show promise for significantly accelerating the field of evaluation.
2. To give particular support to efforts that are outside the natural development of well-accepted paradigms or models.
3. To support those whose development shows significant signs of benefitting people or peoples with serious unmet needs (whether physical, social, economic, or political in other ways)
The Context of Evaluation: Balancing Rigor and Relevance
No full textBackground: The context in which an evaluation is undertaken impacts not only the core evaluation activities but also the ethical standards that guide our work. Theoretical constructs and ethical decision-making frameworks often may not support us in ethical dilemmas given certain evaluation settings.
Purpose: The purpose of this article is to present an example of how a well-intentioned, responsive yet rigorous evaluation provided opportunities to expand our experiences. We conclude this paper with a set of recommendations for organizations/institutions, evaluators, project developers/ implementers, and grantors.
Setting: Not applicable
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: The complexities of conducting an external evaluation of a study that relied on a public-private partnership coupled with a complex intervention that was not fully developed resulted in unintended study limitations and necessitated adapting evaluation strategies and making ad hoc adjustments midstream. Through our recommendations we share our lessons learned and how challenges can be creatively and ethically addressed at the onset of an evaluation
Crafting a Foundation for Evaluating a Worksite Wellness Program
No full textBackground: Businesses have been exposed to many positive accounts of the benefits of employee wellness to improve employee performance as well as reduce health and injury claims costs for the employer. However, many do not have the tools or experience to effectively demonstrate the benefits of a workplace wellness program for their own management and other stakeholders.
Purpose: This paper presents a) basic components for evaluating a workplace wellness program, b) observations gleaned from evaluating a wellness project, and c) a design for a simple evaluation system that provides information about the efficacy of the wellness program and establishes a foundation for more formal wellness program evaluation.
Setting: Maine
Intervention: Evaluating a wellness program and creating a system of measures for a sustainable evaluation system.
Research Design: Steps in wellness program evaluation are delineated (common definitions, evaluation design, i.e., logic model, data collection tools, data collection, comparison analyses, and ROI) and carried out. Particulars for a sustainable wellness program evaluation are rendered.
Data Collection and Analysis: We describe how to apply the process and measurements depicted in the logic model. We illustrate a method for calculating return on investment (determine the ratio of known wellness program costs to decreases in injury claims costs and sick leave costs). We recommend components for a sustainable evaluation system based on our experience in actualizing the logic model.
Findings: Applying the analyses we found positive benefits of employee wellness in our Maine DOT case. Workers’ compensation hours claims dropped from 875 hours in 2006, to 236 hours in 2007. Strains contributed roughly 17% of the overall injury costs reported for all three years, and almost one third of the costs for 2005. We computed a four-year ROI of $2.90. To help establish regular and routine wellness program evaluation we describe and recommend additional data sources and measurement points
Story Telling and Educational Understanding*
No full textOriginally published as Paper #12, Occasional Paper Series, November, 197
Theory Creation, Modification, and Testing: An Information Processing Model and Theory of the Anticipated and Unanticipated Consequences of Research and Development
No full textBackground: Extending Merton’s (1936) work on the consequences of purposive social action, the model, theory and taxonomy outlined here incorporates and formalizes both anticipated and unanticipated research findings in a unified theoretical framework. The model of anticipated research findings was developed initially by Carifio (1975, 1977) and was followed by the addition of the unanticipated findings component by Perla (2006). This is the first formal model, theory and synthesis of anticipated and unanticipated research findings developed to date. The wide-ranging consequences and implications of the model are discussed.
Purpose: To the extent that educational researchers, philosophers and scholars reduce unanticipated findings solely to chance, whimsy, or inspiration they declare these occurrences impossible to effectively predict, model or understand. This article provides a way to conceptualize and formally model anticipated and unanticipated research findings. Many concrete examples from the history of science and research and evaluation methodology are provided to illustrate the model as well as various details of an application of the model in developing instructional materials.
Setting: Nature of science instructional materials development effort.
Intervention: Not applicable.
Research Design: Theory and model development using quantitative and qualitative methods including literature review and original model evaluations.
Data Collection and Analysis: Content analysis and modified Q-sorts of research related documents, journals, logs, literature and emails as well as various theory construction and modification techniques.
Findings: This article demonstrates that a formal model and theory of anticipated and unanticipated research findings can be developed and that such models should inform a broad range of research and evaluation efforts that are conducted daily worldwide. Nine key research and evaluation principles were derived to supplement the formal model and its operations that should be helpful to novice as well as experienced researchers regardless of the research methodology or strategies they are employing
Return on Investment: A Placebo for the Chief Financial Officer… And Other Paradoxes
Get PDFBackground: Return on investment (ROI) is one of the most popular evaluation metrics. ROI analysis (when applied correctly) is a powerful tool of evaluating existing information systems and making informed decisions on the acquisitions. However, practical use of the ROI is complicated by a number of uncertainties and controversies. The article reveals some of these controversies in an engaging and thought-provocative manner.
Purpose: The intent of this note is to highlight several of the ROI paradoxes in a format of an opinion or a viewpoint with a hope that drawing attention of the ROI practitioners and researchers to these issues will contribute to more transparent and responsible application of the ROI evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Review of current practice.
Findings: The article reveals three weaknesses of the ROI evaluations, which in the absence of the commonly accepted ROI standard, can make results of the ROI evaluations uncertain or questionable
Forms and Functions of Participatory Evaluation in International Development: A Review of the Empirical and Theoretical Literature
Get PDFBackground: Since the late 1970s participatory approaches have been widely promoted to evaluate international development programs. However, there is no universal agreement of what is meant by participatory evaluation. For some evaluators, participatory evaluations involve the extensive participation of all stakeholder groups (from donor to non-recipients) in every phase of the evaluation (from design to dissemination). For others, the participation of donors in the design constitutes a participatory evaluation approach. Participatory evaluation approaches are best considered on a continuum. In other words, there are many gradations to participation and evaluations should be classified accordingly.
Purpose: The lack of shared meaning of participatory evaluation approaches also impedes serious discussion on their use including their merits and demerits, suggestions for their improvement, and their overall effectiveness. The purpose of this article is to present an examination of the literature on participatory evaluation approaches to highlight commonalities and differences.
Setting: Not applicable
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: This article clearly demonstrates how broadly participatory evaluation is conceptualized and practiced and underscores the clear need for specification and precision when discussing what is meant by participatory evaluation. Recommendations for how evaluators should describe participatory evaluations are provided
Assessing the Impact of Planned Social Change*
No full textOriginally published as Paper #8, Occasional Paper Series, December, 1976*Reprinted with permission of The Public Affairs Center, Dartmouth College.Preparation of this paper has been supported in part by grants from the Russell Sage Foundation and the National Science Foundation, Grant Number SOC-7103704-03. A version of this paper was presented to the Visegrad, Hungary, Conference on Social Psychology, May 5-10, 1974