Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Rethinking Evaluation Methodology
Get PDFMedicine, engineering, and evaluation have a highly significant common feature: they completely ignored the ban on evaluation that controlled the social sciences through most of the last century. What is it that doctors do in their core practice? They diagnose disease and malfunction, they recommend treatment, they encourage good health. What do engineers do? Amongst other tasks, they work out why the bridge failed, why the plane crashed, and how to correct the underlying errors and build better structures thereafter. And evaluators do the same with programs or policies or products or personnel—find the best, improve the flawed, report on the worst. It is the core nature of these essentially practical enterprises to be evaluative; they were not just describing or explaining or predicting how the world is, but trying to improve it. They simply didn’t take seriously that the essential nature of science had to be ‘value-free.’ Are there any lessons to be learnt from the methods used by our fellow-practitioners in these highly evaluative disciplines
Evaluation and Limitations of Social Interventions: The Case of Spain
Get PDFBackground: Although the number and practice of evaluation varies enormously among policy areas, there are very few studies about the unequal evaluation development in the different policy sectors.
Purpose: This article aims to (1) acknowledge the different evaluation development among different policy sectors in Spain (2) identify the factors and causes that provoke this disproportion, and (3) explore potential consequences of this unequal distribution of evaluation studies among policy areas.
Setting: Spain.
Intervention: Public polices in Spain.
Research Design: A sample of evaluations is classified by policy sector and the number of evaluations in each sector is analyzed and compared. Then, other significant variables are identified for explaining differences among sectors.
Data Collection and Analysis: The cases (evaluation studies) are drawn from two samples: (1) a data base of evaluation studies and (2) a survey to Spanish evaluators held in 2009. The comparison was done with difference in proportions, adjusted standardised residuals and crosstabs.
Findings: Analysis of Spanish evaluations shows that program evaluations are much more frequent in the social policies’ area than in the areas of security, defense or justice. A variable with a high ability to predict whether or not evaluations will be carried out is identified: the selective versus universal nature of the policies being evaluated. Selective interventions are more frequently evaluated than universal policies. This lack of balance makes selective interventions more prone to severe critical analysis. This evaluation bias, in turn, produces a series of perverse effects such as a greater probability of cutting down programs based on selective application strategies
Integrating Evaluation and Agent-Based Modeling: Rationale and an Example for Adopting Evidence-Based Practices
No full textBackground: While there is a great deal of discussion about complex adaptive systems in the field of evaluation, little has been written about the expression of complex adaptive systems in terms of agent-based models, the execution of those models as computer simulations, or the tight integration of agent-based modeling with traditional evaluation methods. These topics need to be explored if evaluation is to move beyond using complex adaptive systems in an exclusively heuristic fashion.
Purpose: This paper advocates for the integration of agent based modeling into traditional evaluation activities. We advance this position in order to spark a movement toward incorporating the formal application of complex adaptive systems into the theory and methods of evaluation. To make our case, we provide a hypothetical example of how interactions between an agent-based model and evaluation can provide unique and powerful understanding about the adoption of evidence-based practices in the field of addiction treatment. We advance our argument by addressing four questions: Why combine evaluation methods with simulation methods? Why use a complex adaptive systems approach? Why add agent-based modeling to the mix of evaluation tools? What is the relationship between agent-based modeling and the theories and methods of evaluation that are being developed to deal with surprise, unintended consequences, and program evolution?
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Validating Assessment for Learning: Consequential and Systems Approaches
Get PDFIn order to adequately evaluate assessment for learning, expanded approaches to validity need to be considered. The purpose of this manuscript is to explore what is necessary to evaluate claims that assessment facilitates learning. Messick’s (1994) concept of consequential validity provides one lens for determining the learning consequences of assessment. His approach suggests that learning consequences are a special case of the general concept of consequential validity and should be evaluated from that perspective. The systems approach developed by Frederiksen and Collins (1989) provides another perspective of how assessments can be designed with learning consequences in mind. Their model provides a transparent way to link assessments to learning by making teaching to the test a valid activity. The adequacy of both models for evaluating claims of learning from assessment is explored. Based on the analysis, a model for evaluating the validity of evidence for assessment for learning is outlined
Current Developments in Assessment for Learning in Universities and High Schools in Michigan: Problems and Perspectives in Mathematics and Science Education
Get PDFAssessment for Learning (AfL) or formative assessment has the potential for raising standards and student achievement. This article describes research conducted by graduate students in an NSF-funded capacity-building project with goals to increase research in the disciplines of science and mathematics education. As background, a literature review shows parallels between assessment for learning research developed over the past two decades in England with assessment reform efforts in school mathematics and science in the U.S. A series of five research projects, conducted by Western Michigan University (WMU) AfL project scholars in fall 2008, examines (1) the prominence of assessment for learning in university course syllabi at WMU and (2) at public universities across the state of Michigan, (3) its prominence in policy statements in a sample of Michigan high schools, (4) the development of benchmarking practices with preservice teachers, and (5) a comparison of existing AfL observation protocols used in classroom research. These reports represent the range of potentially important areas that may prove fruitful for moving research on assessment in mathematics and science education forward.
This material is based upon work supported by the National Science Foundation under Grant No. 0733590
Quality Criteria for Self-Evaluation in Higher Education
Get PDFThe purposes of metaevaluation go beyond the traditional functions of accountability and enhancement. It helps guide strategic organizational change and legitimizes evaluation systems. Metaevaluation results can also be used to create checklists so that the persons responsible for any evaluation can revise, monitor, and control them by themselves. In this paper, the self-evaluation stage of an evaluation process in higher education is metaevaluated. The qualitative method applied analyses to the concerns and complaints that emerged during the work sessions. The results are structured around two key elements: the input (information and self-evaluation guideline) and the throughput (the committee and the process, including the meetings and dynamics of work). The information and self-evaluation guidelines are intended to be responsive, comprehensible, and reliable. The self-evaluation committee and process should be effective, efficient, and responsible with the task. The results are discussed with regard to theoretical proposals for evaluation quality that emphasize criteria such as responsiveness and technical quality of the evaluation
Debating Professional Designations for Evaluators: Reflections on the Canadian Process
Get PDFThis paper provides a reflective account of a consultation process on professional designations for evaluators initiated and coordinated by the Canadian Evaluation Society (CES). Described are (1) the forces leading CES to generate discussion and debate about professional designations for Canadian evaluators, (2) the process of developing and implementing a request for proposals for a tentative action plan that would serve as a basis for debate, and (3) the subsequent development and implementation of a pan-Canadian, tripartite consultation process that involved CES members, CES Chapter organizations, and partners of the Society (including evaluation users). The paper concludes with reflections on the process and a summation of lessons learned
Metaevaluation as a Means of Examining Evaluation Influence
Get PDFUnderstanding how (or whether) an organization uses its evaluation findings can help that organization and others like it to evaluate more purposefully and effectively. Understanding the broader influence evaluations have on the organization and in its operating environment holds even greater potential. Metaevaluation is an appropriate approach for determining how effectively evaluations have served the purpose of program improvement. They can potentially expose other intended or unintended ways in which the evaluation influenced the organization and its environs. Using Henry and Mark’s (2003) taxonomy of evaluation influence as a platform for classification and analysis, this study metaevaluates one organization’s evaluations with an eye toward how those evaluations influenced it, whether through program improvement or other means. Metaevaluation proves a valuable means of exposing subtle forms of influence in the organization, as well as a way of revealing how it might evaluate in the future with an eye toward having even greater intentional influence
Using Online Surveys for Evaluating School-based Drug Prevention Programs
Get PDFBackground: The primary context for providing substance abuse prevention education to adolescents in the United States has been through schools and other local educational agencies. Federal and state spending for such programs is increasingly being tied to school commitments to monitor such prevention programs and evaluate their effectiveness.
Purpose: Over the past decade, access to computers and the Internet has become almost universal in U.S. elementary and middle schools. Our purpose is to outline the potential of Web-based surveys as a data collection tool that can significantly lower program evaluation and monitoring costs and to present preliminary evidence on the feasibility of online survey administration in school settings.
Setting: The empirical part of the article draws on input from teachers, administrators, and practitioners responsible for youth drug prevention and evaluation efforts in schools and communities.
Subjects: Sixteen focus group participants were recruited from counties in and around the San Francisco Bay area in Northern California. Eight of the participants were district-level prevention coordinators and county-level health administrators who administered State and Federal grants to schools within their counties. The remaining subjects were recipients of prevention funding: school teachers, health educators, and practitioners in youth drug prevention and treatment centers. In addition, telephone interviews were conducted with twenty-five Title IV coordinators and drug prevention specialists from various state education and health departments.
Intervention: The article focuses on efforts undertaken in schools to implement and evaluate drug prevention programs, how the use of online surveys can facilitate these efforts, and the feasibility of such methods in school settings.
Research Design: Our research design adopted a qualitative approach and included nationwide telephone interviews and in-house focus group discussions over a six-month period.
Data Collection and Analysis: Data collection included structured, in-person forty-five-minute interviews and discussion notes. In addition, a short, closed-ended survey was administered to focus group participants for collecting information on their school characteristics (population served, school size, Internet, and computer facilities). Survey data were analyzed with simple descriptive statistics.
Findings: The focus group discussions and telephone interviews indicated a high degree of interest in using Web-based surveys for data collection and evaluation of youth drug prevention programs. Access to computers was not viewed as an impediment. Some schools in the Bay area were already using online surveys for assessing teacher performance. Further, states like Kentucky, California, and Wisconsin have already moved to Web-based uniform reporting system that required uploading survey data online. This suggests that Web-based data collection in schools is likely to become widespread.
Conclusions: The use of online surveys in classrooms can significantly enhance the evaluation and monitoring capabilities of schools and communities by minimizing the time required for creating and administering surveys and reducing the time required for data processing. As more states move towards a central reporting database, schools and communities are likely to adopt Web-based mechanisms for collecting and reporting program outcomes
Promoting Student Ownership of Learning Through High-Impact Formative Assessment Practices
Get PDFThe most important instructional decisions, those with the greatest influence on student success, are made by learners themselves (Stiggins, 2008). Formative assessment, done well, contributes to student ownership of learning more than any other classroom-based instructional or assessment practice (Bloom, 1984). It is an economical, highly effective, and uniquely flexible method that can improve learning (Leahy, Lyon, Thompson, & Wiliam, 2005). Simply put, the teacher’s purpose in formative assessment is to give students the means, motive, and opportunity to take control of their own learning. And, through their involvement in formative assessment, students develop self-efficacy for specific learning and, more generally, they develop skills that contribute to increased self-regulation and self-assessment of learning. In order for students to be meaningfully involved in formative assessment, they must be guided by teachers who hold the beliefs, knowledge and skills that engender active student engagement in the learning process. This paper highlights interim findings from a five-year professional development initiative involving the Armstrong School District, a large, rural school district in Western Pennsylvania, and the Center for Advancing the Study of Teaching and Learning at the Duquesne University School of Education. The initiative rests on the fusion of formative assessment, teacher-student communication, and student ownership of learning. The professional development program employs online modules, peer study groups, classroom walkthroughs, and teacher inquiry into their classroom practices and the beliefs that drive them. The program explores seven formative assessment components: 1) Identifying and Clearly Communicating Learning Targets, .) Feedback that Feeds Forward, 3) Student Goal-Setting, 4) Student Self- Assessment, 6) Strategic Questioning, and 7) Formative Discourse. All components are linked to specific aspects of student motivation: intrinsic motivation, self-efficacy, self-regulation, goal-setting, and student attributions. The paper describes the impacts of formative assessment on student ownership of learning, student achievement, motivation, and active engagement as well as provides insights into teachers’ experiences with student involvement. Findings show that not only have the teachers come to value and promote student ownership of learning using high impact formative assessment strategies, but that their efforts have resulted in high student engagement in learning and increased student achievement