Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
The Use of Technology in Evaluation Practice
Get PDFBackground: Evaluation practice is no longer limited to pencil and paper questionnaires, today technological advances allow evaluators to collect data with handheld devices, visualize information in interactive ways, and communicate instantaneously with stakeholders across the globe. These advances have changed how we conduct our practice and they will continue to redefine how we design our evaluations, interact with stakeholders, and communicate our findings. There are few published articles that examine the interface between evaluation and technology and this study represents an initial attempt at examining the technological tools that evaluators use in their practice, the reasons they are adopted, and future technological interest of practicing evaluators.
Purpose: This research on evaluation study attempts to (1) identify the types of technology tools evaluators use in their practice, (2) describe the factors that predict technology adoption, and (3) understand the tools that evaluators are interested in learning more about. This inquiry offers the evaluation community a broader perspective on the technologies that evaluators are implementing in their practice, offers insights on future technological trends within the field, and introduces the evaluation community to tools that can potentially enhance practice.
Setting: Virtual on-line community of evaluation practitioners.
Intervention: This was an exploratory research on evaluation study with no intervention.
Research Design: A panel of experts on technology and evaluation were recruited to brainstorm a comprehensive list of technologies that could be adopted by evaluators as part of their practice. The comprehensive list of technology tools was then embedded within a larger survey instrument that was distributed to a sample of evaluation practitioners from the American Evaluation Association. The survey asked evaluators to select the technology tools they have or currently use in their practice, how often each were used, their satisfaction with each tool, and why they were utilizing each tool in their practice.
Data Collection and Analysis: Data were collected through a web-based survey from members of the American Evaluation Association. The analysis utilized descriptive statics to represent trends in technology use and adoption. Multiple statistical comparisons using ANOVA were also utilized to examine the relationship between technological adoption and use and evaluator background characteristics. Open-ended responses in the survey were also presented as part of the analysis.
Findings: Analyses revealed that technological tools were adopted by evaluators because they helped to produce quality products, increased timeliness, reduced errors, and increased cost efficiencies, and the most adopted tools tended to aid in quantitative data analysis, project management, and productivity. Many evaluators expressed interest in learning more about the use of qualitative analysis tools, web-based data collection tools, and relational database creation and management
Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers
Get PDFBackground: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assigned are very reliable whereas others suggesting that they are so unreliable that random assignment of grades would have been almost as helpful.
Purpose: The primary purpose of the study was to investigate the reliability of grades assigned to written reports. The secondary purpose was to illustrate the use of Generalizability Theory, specifically the fully-crossed two-facet model, for computing interrater reliability coefficients.
Setting: The participants for this study were 29 undergraduate students enrolled in an introductory-level course on Political Behavior in Spring 2011 at a Midwest university.
Intervention: Not applicable.
Research Design: Students were randomly assigned to one of nine groups. Two-facet fully crossed G-study and D-study designs were used wherein two raters graded four assignments for 9 student groups—72 evaluations in total. The universe of admissible observations was deemed to be random for both raters and assignments, whereas the universe of generalization was deemed to be mixed (random for two raters but fixed for four assignments).
Data Collection and Analysis: The semester-long project was assigned to groups consisting of an annotated bibliography, survey development, sampling design, and analysis and final report. Four grading rubrics were developed and utilized to evaluate the quality of each written report. Two-facet generalizability analyses were conducted to assess interrater reliability using software developed by one of the authors.
Findings: This study found a very high interrater reliability coefficient (0.929) for only two raters who received no training in how to use the four grading rubrics
Investigating the Efficacy of a Professional Development Program in Formative Classroom Assessment in Middle School English Language Arts and Mathematics
Get PDFBackground: Teachers who engage in formative classroom assessment using practices that accurately measure student learning should be better positioned to diagnose the instructional needs of their students and to act on that information. For this reason, there has been increased interest in formative classroom assessment in recent years. Although some researchers have found indications that some assessment practices may raise student achievement, evaluations of professional development programs designed to increase teacher assessment skill have not found differences in student performance. However, few studies of this type have been performed. Purpose: The purpose of this study was to determine the efficacy of a professional development program in formative classroom assessment on teacher assessment knowledge and their students’ achievement. Setting: The professional development program was implemented in state-identified, low performing middle schools from November 2005 through April 2008. Intervention: Researchers investigated a professional development program for teachers designed to increase their skill in creating and using assessments to support student learning. In Year 1, the professional development was implemented by an assessment coach in the treatment schools. No professional development was provided in the control schools. In Years 2 and 3, levels of treatment were investigated such that the professional development was implemented by an assessment coach or a relatively untrained facilitator. Research Design: Year 1 involved a multi-site, cluster randomized trial where schools were randomly assigned to either the treatment or control group. Year 2 and Year 3 involved a quasi-experimental design. Data Collection and Analysis: Researchers collected pretest and posttest teacher measures and analyzed the data using a split-plot ANOVA each year. Summative large scale assessment data was collected for students and analyzed using hierarchical linear modeling (HLM). Findings: Findings from this study indicate that the professional development program increased teacher assessment skill regardless of whether the program was implemented with a trained assessment coach or a relatively untrained facilitator. However, students of teachers participating in the professional development tended to demonstrate lower achievement than a matched set of students whose teachers did not receive the professional development. Implications for how teachers use assessment data to guide reteaching are discussed. Keywords: formative assessment; classroom assessment; student achievemen
Crowd-Sourced Evaluation: A Qualitative Study of User-Generated Product Review Videos on ExpoTV.com
Get PDFBackground: While user-generated videos are typically associated with humorous or shocking videos far removed from the rigorous world of evaluation, this paper explores the potential for utilizing user-generated videos as evaluation data. This topic is addressed through a qualitative pilot study of product review videos available on the ExpoTV.com website. The findings from this important subset of the evaluation field are analyzed with the goal of identifying themes and insights that could be useful to the broad field of evaluation.
Purpose: The primary purpose of this study is to identify characteristics or insights about the user-generated content that could serve as a guide for future studies and the development of new theory or methodology to enlarge the scope and relevance of data used in evaluations.
Setting: The ExpoTV.com website
Intervention: Not applicable.
Research Design: Grounded theory.
Data Collection and Analysis: User-generated videos downloaded from the ExpoTV.com website. Analyzed in three stages – In Vivo coding, Axial coding and theme identification.
Findings: This article finds intriguing strengths to user-generated video as an evaluation data source because user-generated product review videos consistently demonstrate three important elements of evaluation: they provide a description of the product, explain the broader context for both the product and nature of the specific review, and provide an evaluative conclusion that is logically related to specific evaluative descriptions.
Principles of Methodology: Research Design in Social Science
Get PDFA review of the book Principles of Methodology: Research Design in Social Science by Perri 6 and Christine Bellamy, published in 2012 by SAGE
Social Psychology and Evaluation
Get PDFA review of the book Social Psychology and Evaluation edited by Melvin M. Mark, Stewart I. Donaldson, and Bernadette Campbell, published in 2011 by Guilford Press
Analysis of Paired Dichotomous Data: A Gentle Introduction to the McNemar Test in SPSS
Get PDFBackground: Although McNemar Test is the most appropriate tool for analyzing pre-post differences in dichotomous items (e.g., “yes” or “no”, “correct” or “incorrect”, etc.), many scholars have noted the inappropriate use of Pearson’s Chi-square Test by researchers, including social scientists and evaluators, for the analysis of related or dependent dichotomous variables.
Purpose: The goal of this paper is to promote the use of McNemar Test among evaluators by providing a gentle introduction to the method.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Using data from 506 6th grade students’ responses to a pre-post science test; this contribution illustrates how to conduct McNemar Test in SPSS.
Findings: This contribution provides a non-technical introduction to McNemar test and illustrates its use in an applied research/evaluation context
Formative, Preformative, and Proformative Evaluation
Get PDFMichael Scriven defines the terms 'preformative' and 'proformative' in the context of his thinking on evaluation types, such as in the Key Evaluation Checklist. The discussion works through formative, summative, and ascriptive evaluations, eventually introducing and providing examples of 'proformative evaluation' as an addition to the set of three
Responding to the UNDP Evaluations Unit
Get PDFThe author of an indicator that tests evaluation system compliance with good governance principles addresses the UNDP’s response to his article by offering an empirical test of the UNDP’s commitment to reform. While the UNDP Evaluations Office claims to be working on reforms, the test exposes the unwillingness of the UNDP Evaluations Office to take even a simple step that would correct the problems they admit exist. When presented with the chance to support reform on a real case – a de-politicized evaluation contract that protects professional standards and offers public accountability for the Spanish government’s MDG-Fund to the UNDP – the Vice Director of UNDP’s Evaluations Unit failed to support a single change. The author believes that public oversight of international organizations has been so weakened that not even their evaluation units understand anymore what democratic, public control means or how it works and they have now become part of an embedded culture that is unable to even conceive of how to apply simple public solutions
Evaluating the Cost Effectiveness of Heifer International Country Programs
Get PDFBackground: In the 1950s and ‘60s cost effectiveness was the primary focus of evaluation of international development aid, but in recent decades this has been increasingly neglected. The most recent decade saw great interest in impact evaluation, but the step from impacts to cost effectiveness was often neglected. This article explains how a multi-year evaluation of a major international NGO that was designed to estimate country program impacts was expanded to include estimates of cost effectiveness.
Purpose: The article explains the importance and illustrates the practicability of evaluation for cost effectiveness. It describes the evaluation’s methodology and presents its major finding, that Heifer International is highly cost effective in improving the income, assets and nutrition of poor households in Albania, Nepal and Uganda.
Setting: The article focuses mainly on the 2011 evaluation of the Heifer International country program in Uganda.
Intervention: The evaluations focused on usually three to five year projects of usually a few dozen to a few hundred households, but the evaluation itself did not conduct interventions (beyond its interviews).
To add evaluation of cost effectiveness, the focus was changed from the project to the household level and evaluators estimated changes in income, assets and nutrition due to the project. Estimates of income impacts were then used as the primary basis for estimating the cost effectiveness of the respective country program (although assets and nutrition were also considered).
Data Collection and Analysis: Data were collected through two and a half to three week country program evaluations by two or three evaluators and their translators. Group and household interviews were based on questionnaires, but they also required evaluators to pursue lines of inquiry to logical conclusions. Quantitative and qualitative factors were considered as a basis for impact estimates on scales from zero to five in the original evaluation and in terms of economic values for income and assets. Analysis was carried out largely with Excel spreadsheets.
Findings: Due to Heifer International’s expenditure of about 8.5 million a year on an ongoing basis and asset gains of about 1 expenditure by country programs in Albania, Nepal and Uganda, households can be expected to gain about 1.19 and $1.25 in the respective country programs on an ongoing basis once the projects reach the maturity profile of those included in the evaluation