Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Evaluating the Due Process and Crime Control Perspectives Using Rasch Measurement Analysis
Get PDFBackground: The biases jurors possess may influence everything from the interpretation of case evidence to impressions of the defendant to, ultimately, verdict and recognition of this has led to a number of juror attitude scales attempting to tap into important biases. A common ideology discussed in legal research is that individuals attitudes toward the law and the legal system differ along a continuum moving from due process (a concern for the preservation of individual rights) to crime control (a focus on swift and harsh punishment for those who break the law) although an agreed upon assessment of these perspectives has yet to be created.
Purpose: The current research addresses due process and crime control perspectives and uses the ideology as a source for a new measure of juror bias: The General Attitudes toward the Legal System (GALS) scale.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: The GALS scale was constructed based on existing theory and administered to nearly 700 undergraduate psychology students at a large Midwestern university. The psychometric properties of the instrument were then evaluated to determine instrument quality.
Data Collection and Analysis: The Rasch Rating Scale Model (RRSM) was used to evaluate the psychometric properties of the GALS. Evaluation focused on six characteristics of the instrument: dimensionality, reliability, rating scale quality, item quality, item hierarchy, and measure quality.
Findings: Results indicate the GALS is a psychometrically sound instrument for measuring juror bias.
“Standard Assessment Framework”: A Proposal to Fill a Gap in the “Paris Declaration”
No full textBackground: The “Paris Declaration” (PD), is a statement by which, in 2005, leading representatives/actors of International Development Cooperation agreed on a series of ways and means with a view to improving aid effectiveness. Among those ways and means, the need for the establishment, by partner countries, of national “Assessment Frameworks” was stressed without, however, alluding to the nature and content of such frameworks. This must be considered a serious gap.
Purpose: The purpose of this article is to make a proposal with a view to filling in this gap and thus to provide Policy and Decision Makers, Program and Project Planners, Implementers and Evaluators as well as Parliaments and other professionally interested quarters, with an operational instrument allowing to seriously enhance chances of realizing the objective of the PD: improving aid-effectiveness.
Setting: Developing countries anywhere and their developed partner countries and partner organizations.
Intervention: The article is addressing the problems and the possibilities of any systematic intervention: Policies, Programs and Projects, designed to further the development of developing countries.
Research Design: The article undertakes to address the problems and opportunities mentioned above by establishing a “Standard Assessment Framework” (SAF) designed to concentrate the minds of all actors on the creation of sustainable benefits for the target groups of development interventions; welding Planning and Evaluation into one single mutually reinforcing system, always to be kept up to date.
Data Collection and Analysis: The SAF has been developed over time by numerous inputs from a great variety of sources and presents, thus, a depository of lessons of experience still open to further inputs and improvements. The templates of members of the “Development Assistance Committee” (DAC) of the Paris based OECD were, i. a., systematically taken into consideration.
Findings: The SAF represents, thus, in a nutshell, the most important findings of interventions too numerous to be mentioned individually or to be tracked. It therefore presents simultaneously, however, a general grid that should structure findings in a much more detailed way, to be consigned to a SAF Data-Base that can be built up on the basis of experiences made in the past and on those accumulating further under current evaluation work
Performance Standards
Get PDFOriginally published as Paper #11, Occasional Paper Series, December, 1977* This paper was prepared under a grant from the Carnegie Corporation of New York
Introduction to the Occasional Paper Series
No full textIntroduces the Occasional Paper Series, provides its history, and explains the importance of the series to the history of evaluation.
Benefits of the Person-Oriented Perspective for Program Evaluation: Analyzing the Differential Treatment Effects of the Vienna E-Lecturing Program
Get PDFBackground: In program evaluation interventions are usually examined for global effects but not always for differential effects. The reasoning behind the focus on global effects is that most of the concepts applied in data analyses, and most of the hypotheses being assessed, are variable-oriented. Consequently, existing differential effects will not be detected through global analyses based on a variable-oriented perspective. Accordingly, the intervention under evaluation will be wrongly considered to be ineffective.
Purpose: This contribution, therefore, argues for the application of person-oriented analyses in addition to the common variable-oriented approach in order to identify the differential effects of an intervention during evaluation.
Setting: Both approaches will be applied in the evaluation of a showcase program conducted in real life setting. The results of the two perspectives will be compared.
Intervention: An internet supported program for the promotion of self-regulated learning in an academic context called Vienna E-Lecturing (VEL).
Research Design: The evaluation of the VEL was conducted with a treatment-control-design. Data was collected at the beginning as well as at the end of the intervention.
Data Collection and Analysis: Data regarding motivation as well as factual knowledge was collected. All motivational variables were assessed using previously published scales consisting of 3-4 items. Factual knowledge in statistics and methodology was tested regarding the aspects of reproduction, comprehension and the production of methodological content.
Findings: The application of a variable-oriented perspective showed small effects with low practical relevance. Through the application of a person-oriented approach, and taking into account the theoretical framework of the intervention, three students groups which differed systematically in their learning motivation could be identified. For two of these groups the intervention was very successful. The discussion provides recommendations on how person-oriented approaches can be used systematically in program evaluation
Integrating the Funder-Fundee Relationship into a Model of Evaluation Utilization
Get PDFBackground: Evaluation is a tool that can promote accountability and enhance organizational improvement. For these reasons funding entities from government to foundations are increasingly relying on program evaluation as a key instrument to determine effectiveness and hold recipient organizations accountable. What has ensued is an environment of increasing evaluation efforts, where findings often weigh heavily on future funding decisions.
Purpose: The purpose of this article is to examine how the nature of the funder-fundee interorganizational relationship influences evaluation utilization by the fundee organization. Within this context, attention is paid to the role the evaluator plays in the evaluation process and the skills needed to promote utilization.
Setting: A statewide multi-tier public/private initiative addressing early childhood education.
Intervention: Not applicable.
Research Design: A model is designed to test the influence of the nature of the funder-fundee interorganizational relationship on factors associated with utilization, and ultimately on use itself.
Data Collection and Analysis: A survey was administered online and data analysis was conducted using principal components analysis (PCA) and structural equation modeling (SEM).
Findings: The results suggest that the funder-fundee relationship plays a critical role in the evaluation process, and consequently influences utilization. However, the findings indicate that the relationship among the mediating factors is more complex than originally hypothesized
From Theory to Practice: A Path Through Imperfection-Lessons from Policy Oriented Environment and Health Research in Belgium
No full textBackground: Dealing with complex issues per definition bears the burden of imperfection. Whatever comforting theoretical concepts may promise, real life complexity will take its messy toll once travelling from conceptual ambition to real life practice. We specifically reflect on the social scientific contribution to these inter- and transdisciplinary endeavours.
Purpose: We reflect on how social scientific research has tried to find its way in complex real life research practice in the field of environment and health and specifically focus on the challenges posed by the imperfections we encountered.
Setting: Two case studies in the field of environment and health in Belgium. One case study focuses on policy interpretation of research results. The other focuses on a selection of research priorities.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: inter- and transdisciplinary research and multi-criteria analysis
Findings: Resulting from our practical experiences we present a typology of imperfections: imperfect information, imperfect expert assessment, imperfect processing of incommensurable data, imperfect socio-political weighing/deliberation and reflection on imperfection. As easy as it seems to design processes of structuring complex issues and decision making in which involvement of a relevant diversity of actors and factors is considered praiseworthy, as complicated it is, as we have shown, in practice to organize and live up to expectations. In judging the quality of this endeavour, diverse theoretical yardsticks maybe applied that will not necessarily do justice to the practical complications of research practice. The concepts of abduction, intersubjectivity and transferability, offer interesting qualitative notions of pragmatic approaches that seem relevant to our work
Evaluation Essentials: From A to Z
Get PDFA review of the book Evaluation Essentials: From A to Z by Marvin C. Alkin published in 2011 by Guilford Press
Program Evaluation Particularly Responsive Evaluation*
No full textOriginally published as Paper #5, Occasional Paper Series, November 1975.
* Paper presented at a conference on “New Trends in Evaluation”, Goteborg, Sweden, October, 1973
Empathy Training: Methods, Evaluation Practices, and Validity
Get PDFBackground: Empathy is an individual’s capacity to understand the behavior of others, to experience their feelings, and to express that understanding to them. Empathic ability is an asset professionally for individuals, such as teachers, physicians and social workers, who work with people. Being empathetic is also critical to our being able to live with others in general, and ultimately to leading happier lives. Subsequently it seems imperative to examine if and how it is possible to enhance people’s empathic ability.
Purpose: The purpose of this article is to use narrative review method to analyze studies of empathy training in human service and social science disciplines over the past thirty years to address the questions: “How have people been trained in empathy and what are the findings?” and “How was empathy training evaluated and how valid are these evaluation findings?”
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Narrative review.
Findings: Twenty-nine articles pertaining to empathy training evaluation research were identified based on an advanced computer search on the following databases: “Education Full Text,” “ProQuest Education Journals,” “Web of Knowledge” and “Educational Resources Information Center (ERIC). Seven types of training methods were noted in these 29 evaluations with the most popular being didactic related (42%). All but two studies (93%) reported positive findings, mainly in regard to learning (86%), or the cognitive component of empathy. These findings suggest that regardless of the training method, individuals can learn about the concept of empathy. Unfortunately, information pertaining to the effects of training on individuals’ feeling for others, and their ability and propensity to take the perspective of others and to demonstrate it in the natural environments is lacking. Consequently, very little is known about the trainability of the affective and behavioral components of empathy. Also, some of the findings were moderated by gender, age, education level, and time of measurement. Regarding evaluation research designs, most of the studies used self-reporting to collect trainees’ knowledge about empathy and most of the quantitative studies used a control group and pretesting to examine training impact. Construct validity of both empathy measurement and training is very problematic. A majority of the studies did not clearly define empathy, provide training as defined, and/or measure what is being trained; conceptualization of empathy across studies was not consistent either. In sum, data from the studies reviewed were neither complete nor valid enough to provide a clear and full understanding of the trainability of empathy. More research is apparently needed and hopefully lessons learned from our review will be considered in designing future studies.