Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Toward a General and Unified View of Educational Research and Evaluation: Bridging Philosophy and Methodology
No full textThis paper addresses an important problem that may really be a pseudoproblem perpetuated by a current artificial or even political need to isolate researchers, evaluators, and theorists and put them in conceptual-methodological boxes. In this paper, we argue that scholars of all kinds should focus on the nouns (i.e., research, evaluation, theory) versus the adjectives (i.e., quantitative, qualitative, or mixed methods) used to qualify these nouns and core constructs. In a word, researchers, evaluators, measurement professionals, and theorists of all persuasions should strive to be neither qualitative nor quantitative, as that is like striving to be a single leg in a triangle. Following Hacking (1965), our view is that all forms of research methodology and evaluation methodology represent a “patchwork quilt” that is essentially “one fabric” with two connected but often unacknowledged sides to the quilt and that all methodology, analysis, and evaluations are essentially “qual-quantification” and “quant-qualification” and usually both at the same time. This general and unified or “one fabric” view of these debates and issues is discussed from a historical-philosophical perspective, and the need for a more general and inclusive evaluation and research epistemology is elucidated. Having outlined an epistemic framework and argument for a more comprehensive and complete view and model of evaluation and research that transcends a specific research paradigm, the paper concludes by outlining eight general key points that most (if not all) researchers and evaluators should consider
Ethical Issues in Agency Evaluation from the Viewpoint of Activity Theory: A Basis for Inter-Organizational Learning?
Get PDFThis article focuses on ethical issues faced in evaluation practice from the viewpoint of third generation of activity theory, which gives a constructive perspective on how contradictions can be a driving force behind interorganizational learning and development in multiactor networks. The problem field is firstly addressed through an illustration of the problematic position of evaluators in situations where cooperational relationships and professional networks are close. This perspective is then extended by an analysis of a reflection model designed to initiate discussion about the principles of evaluation. From an activity theoretical perspective, the ethical issues reflect contradictions, which can be a starting point for development, whether the actors can get oriented collectively in the analysis of a contradictory situation, and modelling, implementing and examination of a new solution. The analysed model for ethical reflection can be used, despite its limitations, as a heuristic framework for this kind of collaborative ethical reflection in a multivoiced network of people involved in the evaluation process
Adapting the Utilization-Focused Approach for Teacher Evaluation
Get PDFMandated teacher evaluations are of little use to educators for a variety of reasons, including lack of reliability and validity of evaluation instruments and methods, failure of evaluation systems to adhere to the Personnel Evaluation Standards (Joint Committee, 2008), and the inherent bias in using classroom observations as a method of evaluating teachers. As a result, teacher evaluations fail to improve teaching or impact student learning. This paper suggests that the utility, and ultimately the impact, of teacher evaluations can be improved by adapting the utilization-focused approach to teacher evaluations. Specific suggestions for implementing the Utilization-focused Evaluation Checklist (Patton, 2008) are also provided
The Insight Evaluation Approach
Get PDFProjects and programmes that are often the objects of our inquiry are actually no more than mental constructs. They are the result of numerous conditions that happen to come together for one brief moment. Their funding, activities, and results arise, dwell in a state of constant flux for a period of time, and then pass away. Insight Evaluation provides an approach to see projects and programmes clearly, without false impression.
The views expressed in this manuscript are those of the authors and do not necessarily reflect the policy of the International Labour Organization, the United Nations, or the Evaluation Capacity Development Grou
A Job Analysis for K-8 Principals in a Nationwide Charter School System
Get PDFBackground: Although no single technique on its own can predict job performance, a job analysis is a customary approach for identifying the relevant knowledge, skills, abilities, and other characteristics (KSAO) necessary to successfully complete the job tasks of a position. Once the position requirements are identified, the hiring process is faster and more effective because job candidates are evaluated on a common set of objective criteria.
Purpose: The purpose of this study was to conduct a job analysis of the principal position for elementary and middle school public charter schools. Moreover, the final outcome of the job analysis was a linkage of KSAOs to their respective job tasks that could be used during the candidate selection process.
Setting: The job analysis was conducted in a national K-8 charter school system.
Subjects: The information collected during the job analysis came from six subject matter experts (SME) who were employed as a principal or as a supervisor of principals by the charter school system at the time of the study. The SMEs represented a balanced blend of perspectives of the principal position and they possessed a wealth of knowledge, job experience, and skill level that qualified them as experts of the principal position.
Intervention: During the job analysis, SMEs provided information about the principal position.
Research Design: The job analysis used a qualitative case study design with a convenience sample of SMEs.
Data Collection and Analysis: Subject matter experts participated in three phases of data collection. The first two phases collected information on the essential job tasks and the third phase collected information on the KSAOs required to successfully complete the job tasks.
Findings: In total, 314 essential job tasks were identified across 8 functional job categories. The job analysis condensed redundant job tasks and eliminated job tasks that were not endorsed by half of the SMEs, which totaled a final set of 84 essential job tasks. Then, SMEs rated the essential job tasks on several dimensions that were used to measure job task importance. Of the 84, 46 essential job tasks met the criterion of being the most important in terms of achieving organizational goals. A collaborative group that included the job analyst, principal recruiters and one SME identified the KSAOs necessary for completing the 46 essential job tasks. During the final phase of data collection, SMEs rated the KSAOs on the following dimensions: necessary for principal applicant to possess, practicality of finding a principal applicant with the KSAO, and consequence if KSAO is ignored in the selection process. Forty-nine of the 53 KSAOs met the inclusion criteria based on the ratings of the dimensions above.
Conclusions: The results from the job analysis revealed the diverse set of KSAOs required to successfully complete the essential job tasks of the principal position. Thus, using objective criteria established from the job analysis is a critical step in selecting the candidate who is best suited to succeed in the principal position
Analyzing an Active Labor Market Program in Germany: A Regional Approach: An Attempt to Use Propensity Score Matching for the Estimation of Causal Effects on the Level of Counties and Independent Cities
Get PDFThe Institute for Work, Skills and Training was assigned to evaluate a labor market program aimed at the integration of long-term unemployed individuals aged 50 or older. The integration should have been achieved not only by training and coaching of individuals, but also by building regional networks between labor market stakeholders within a region. To appraise the success of the action undertaken on the regional level in the sense of causal effects, an observational study was used. The experimental- and control-groups were built using propensity score matching. The matching was done using not individual-level data, but data on the regional level because of missing individual data and the aims of the program. The mean growth of the number of employees subject to social insurance contributions over time was chosen as the outcome of the program. The findings are that observational studies are suitable to estimate the causal effects of active labor market programs on the regional (macro) level if individual data are missing or if the aims of the program cannot be observed on the individual level
Evaluation Theory, Models, & Applications
Get PDFA review of the book Evaluation Theory, Models, & Applications by Daniel L. Stufflebeam & Anthony J Shinkfield, published in 2007 by Jossey-Bass
Metaevaluation in Practice: Selection and Application of Criteria
Get PDFThis paper examines the practice of metaevaluation as indicated by the Metaevaluation standard of the Program Evaluation Standards, as the evaluation of a specific evaluation to inform stakeholders about the evaluation’s strengths and weaknesses. The findings from an analysis of eighteen metaevaluations, including a description of the data sources and methods used to come to conclusions about the evaluation and the criteria of quality employed, are reported. A diverse set of practices were identified, ranging from the use of emergent criteria in a narrative review of information about an evaluation to the structured application of the Program Evaluation Standards using a checklist. The paper concludes that the evaluation field does not have a common understanding of metaevaluation practice
The Development of Evaluation Culture in Spain: A Historical Approach Based on the Biographical Method
Get PDFThis paper presents a retrospective explanation as to how evaluation culture has been slowly incorporated into politics and public institutions in Spain during the last three decades. Evaluation, quality, and excellence are relevant concepts embedded in the design of current political programs. At this moment in history, it is important to ask, what has been the evaluation history in Spain? Where do influences on research designs come from? I address these questions in order to describe the evaluation process in the public sphere based on the biographical method, specifically on the testimony of five Spanish evaluators with recognized experience in this field. The aim is to set up a historical vision that represents the evolution of evaluation culture in Spain, analyzing and linking the individual perceptions of evaluators with the specific characteristics of a particular social context