Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
The Contribution of Metaevaluation to Program Evaluation: Proposition of a Model
Get PDFBackground: This theoretical article points to the fundamental difference between meta-analysis and metaevaluation. A model of metaevaluation for social programs is presented based on prior practical research.
Purpose: The purpose is to present a model for metaevaluation as a tool that can be used in other studies. Theory points to the need of a qualitative framework to go beyond the understanding of meta-analysis for program evaluation.
Setting: This theoretical article is based on an empirical research conducted at a Brazilian Governmental audit agency.
Subjects: The Government agency where the practical research was conducted is responsible for the effectiveness and accountability of social programs through audits that occurred from 2003 to 2006.
Intervention: Meetings and interviews were held with auditors that participated in the evaluation process going from planning to final reports as the model proposes
Research Design: The model for metaevaluation has a qualitative approach used to evaluate prior evaluations for social programs.
Data Collection and Analysis: Data collection included structured interview with the chief manager of the agency in charge of evaluating governmental programs. Documents and reports were analyzed using qualitative method for content analysis. Synthesis of categories was applied to compare different analysis and summarize findings.
Findings: Metaevaluation and meta-analysis are different research methods with a different approach. Meta evaluation is a qualitative method useful when evaluating prior evaluations. Yet the quantitative approach of meta-analysis applies better for first evaluations. Meta evaluation may include other methods to help strengthen the evaluation results.
Conclusions: Metaevaluation aligns theory and practice for program evaluation. The proposed model for metaevaluation may hold value for future theoretical and empirical work
In This Issue
Get PDFWelcome to the first issue of the Journal of MultiDisciplinary Evaluation (JMDE) for 2009, which marks our sixth year of publication. Not too bad for an online journal with no financial resources! With this issue, we are delighted to welcome the newest member of our editorial team, Lori Wingate. Lori is the book review section editor for the American Journal of Evaluation and a doctoral candidate in Western Michigan University’s Interdisciplinary Ph.D. in Evaluation program. She has been a staff member at The Evaluation Center for almost a dozen years and has worked closely with many of the field’s luminaries, including E. Jane Davidson, Arlen Gullickson, James Sanders, Michael Scriven, and Daniel Stufflebeam. Lori’s official role on the JMDE editorial staff is as its copy editor and she was brought on to assist with assuring that papers published in JMDE meet the highest standards of quality. In this role, Lori will work directly with authors whose papers have undergone peer review and been accepted for publication to improve the formatting, style, clarity, and accuracy of their manuscripts. Please join me in welcoming Lori
From Frequent Testing to Interactive Dialogue: Diversity and Challenge
Get PDFThe articles assembled in this special issue reflect very faithfully the diversity in what is now a rich field of development. But because they do this, they also highlight a problem: what it is about this topic that leads to the contributors describing such a diverse range of approaches to the practices of formative assessment
Assessment for Learning in K-12 Science Departments: A Case Study
Get PDFThis paper discusses the efforts of science departments in two secondary schools to implement data-driven decision making at the department level. It covers the first year of the process and identifies the successes and barriers to success experienced by the two departments. The catalyst for the change at each school was the department chair studying data analysis and applying the newly learned techniques to stimulate their science departments to study assessment data. The departments were surprised by some of their findings. For example, the faculty at one school found that their students performed poorly on one strand covered in the state’s science assessment. However, their data analysis showed that their students scored about equally well on each strand. The findings led both schools to consider and implement various changes. Both departments saw the need to use shorter and timelier formative assessments to drive instruction throughout the year. The departments encountered several barriers to using data-driven decision making to guide instruction. They found it easier to identify issues that needed to be addressed than implementing solutions
An Evaluation Use Framework and Empirical Assessment
Get PDFBackground: Research on evaluation use focuses on putting evaluation recommendations into practice. Prior theoretical research proposes varied frameworks for understanding the use (or lack) of program evaluation results.
Purpose: Our purpose is to create and test a single, integrated framework for understanding evaluation use. This article relies on prior theoretical research regarding categories of utilization, typologies of recommendations, and factors affecting utilization to frame an empirical study of evaluation use that then tests the integrated theory.
Setting: The empirical part of the article draws on post- evaluation interviews with sixteen agencies that have engaged in evaluation research.
Subjects: The agencies are mostly local non-profits, but the sample also includes a state agency, a city agency, and two university-community partnerships. All agencies had undergone a program evaluation between 2003 and 2006.
Intervention: Having participated in an evaluation is the main “intervention” of interest in this article, in which we consider the relationship between evaluation use theory and empirical evidence on the topic.
Research Design: A qualitative approach, our research design involved examining each of the sixteen agencies within two years of their having been evaluated.
Data Collection and Analysis: Data collection included structured in-person interviews with at least one key informant in each agency. In addition, a short, closed- ended survey was administered to research participants. Interview data were analyzed using content analysis of themes and grouping agencies according to their evaluation’s outcomes (favorable or not) and experiences. Survey data were analyzed with simple descriptive statistics and similarly involved a subgroup analysis, according to agencies’ reported use of evaluation.
Findings: Most evaluation use is conceptual, and few agencies studied actually implemented specific evaluation recommendations. Agencies perceived recommendations as changes to rules and structure, which theory and prior research suggest suppresses use. An important human factor that influenced evaluation use was minimal post- evaluation interaction with evaluators. Some long-term influence of the evaluation is evident, but only as conceptual and not as instrumental. In fact, very little instrumental use existed in this empirical assessment.
Conclusions: Evidence suggests that evaluation use in practice aligns with theory, specifically emphasizing the conceptual use dimension. The proposed integrated model of prior evaluation use theories may hold value for future theoretical and empirical work
An Association to Improve Evaluation of Development Aid
Get PDFThis paper argues that development aid could be greatly improved if the management of development programs and projects could be governed by an effective orientation to cost-effectiveness. This in turn could substantially be achieved by changes the paper proposes to aid evaluation—through a particular kind of association—and a few lesser management reforms
Cost Analysis in Evaluation
Get PDFProgram, product, and policy evaluation can rarely be done seriously without serious attention to costs. Yet the surveys of journal literature and meeting papers show that it is still very rarely considered
The Evaluation Market in Germany
Get PDFThe United States has a long tradition in evaluation of political programs. In the 1930s and 1940s, programs were initiated to reduce unemployment and improve social security as part of the “New Deal.” In the late 1960s, somewhat comparable to the U. S. at that time, Germany’s new government started its own “New Deal.” Unemployment was modest but the growth of the economy was declining. The German government wanted to implement programs in order to follow a path of steady growth. Since most of the programs were evalu- ated, a growing market of evaluation could be observed at that time
The Concept of a Transdiscipline: And of Evaluation as a Transdiscipline
Get PDFThere are two main meanings of this term that can be disinterred from the 8,000+ references to their definition in Google (at December 12, 2007), both of them sharing the idea of a discipline that crosses over the boundaries between many other disciplines: we can distinguish them as the ‘point of view’ sense of the term and the ‘method’ sense of the term. These senses are not sharply distinct from each other, but are distinct from ‘interdiscipline’ meaning a compound approach drawing from or working in the boundary area between two or sometimes more other disciplines; and from ‘multidiscipline’ meaning a compound approach involving more than two other disciplines
Logic Modeling Methods in Program Evaluation
Get PDFA review of the book Logic Modeling Methods in Program Evaluation by Joy A. Frechtling published in 2007 by Jossey-Bass