Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Mixed Methods Research in Designing an Instrument for Consumer-Oriented Evaluation
Get PDFBackground: The educational product market has been gradually shifting from primarily print to primarily digital content. Educators must make quick decisions when selecting materials that will assist students in their learning.
Purpose: Purposes of this study were to describe the application of a two-stage sequential mixed-method, mixed-model design in designing an instrument for consumer-oriented evaluation and to describe implications of using mixed methods research in developing a rubric to evaluate prekindergarten through Grade 12 digital content.
Setting: The Ohio State University, Columbus, OH.
Intervention: N/A.
Research design: A two-stage sequential mixed-method, mixed-model design.
Data collection & analysis: In Stage 1, a modified electronic Delphi survey technique was implemented with US geographically dispersed subject matter experts. In Stage 2, cross-sectional focus group interviews were conducted with local teachers, administrators, and textbook publishers.
Findings: Inclusion of multiple perspectives and viewpoints from teachers, administrators, textbook publishers, and experts on importance, clarity, and appropriateness of criteria to evaluate digital content resulted in a final version of the rubric that can be used by teachers and administrators to evaluate digital content that supports students’ learning in prekindergarten through Grade 12
Towards a Complexity Framework for Transformative Evaluation
Get PDFBackground: Complexity ideas originating in mathematics and the natural sciences have begun to inform evaluation practice. A new wave in evaluation history is about to break. A new mindset, new methods, and new evaluation processes are being summoned to explore and address the challenges of global pandemics, growing inequities, and existential environmental risks. This is part of a broader paradigm shift underway in science where interdisciplinarity has become the norm rather than the exception.
Purpose: This article explores the utility of a complexity framework for a more effective evaluation function. It unearths the antecedents of complexity thinking; explores its relevance to evolving knowledge paradigms; provides a bird’s eye view of complexity concepts; uses the logic of complex adaptive systems to unpack the role of evaluation in society; and draws the implications of contemporary social challenges for evaluation policy directions.
Setting: Not applicable.
Intervention: Not applicable.
Research design: Not applicable.
Findings: The evaluation complexity challenge coincides with an urgent imperative: social transformation. The on-going pandemic has brought to light the disproportionate effects of health emergencies on disadvantaged groups and emphasized the urgency of improving the interface between humans and nature. It has also demonstrated the importance of modelling for policy making – as well as its limitations. Evaluation, a complex adaptive system, should be transformed to serve society.
Keywords: complexity; computers; disciplines; emergence; modelling; paradigm, system
Using Cognitive Interviewing to Test Youth Survey and Interview Items in Evaluation: A Case Example
Get PDFBackground: Cognitive interviewing is a pretesting tool used by evaluators to increase item and response option validity. Cognitive interviewing techniques are used to assess the cognitive processes utilized by participants to respond to items. This approach is particularly appropriate for testing items with children and adolescents who have more limited cognitive capacities than adults, vary in their cognitive development, and have a unique perspective on their life experiences and context.
Purpose: This paper presents a case example of cognitive interviewing with youth as part of a national program evaluation, and aims to expand the use of cognitive interviewing as a pretesting tool for both quantitative and qualitative items in evaluation studies involving youth.
Setting: Youth participants were located in four regions of the United States: Northeast, Central, Southern, and Western. Interviewers were located at Montclair State University.
Intervention: Not applicable.
Research design: A cognitive interview measure was designed to include a subset of survey items, interview questions, and verbal probes, to evaluate if these items and questions would be understood as intended by both younger and older youth participants. An iterative design was used with cognitive interviewing testing rounds, analysis, and revisions.
Data Collection and Analysis: The cognitive interview was administered by phone to 10 male youth, five from the 10-13-year-old age range and five from the 15-17-year-old age range. Interviews were audio-recorded, transcribed, reviewed, and coded. Survey items and interview questions were revised based on feedback from the participants and consensus agreement among the evaluation team. Item revisions were included in further testing rounds with new participants.
Findings: As a result of using cognitive interviewing to pretest survey and interview items with youth, response errors were identified. Participants did not understand some of the items and response options as intended, indicating problems with validity. These findings support the use of cognitive interviewing for testing and modifying survey items adapted for use with youth, as well as qualitative interview items. Additionally, the perspective of the youth participants was valuable for informing decisions to modify items and helping the evaluators learn the participants’ program culture and experiences. Based on the findings and limitations of the study, we give practice recommendations for future studies using cognitive interviewing with a youth sample
An Exploratory Study on Public Sector Program Evaluation Practices and Culture in Barbados, Belize, Guyana and Saint Vincent and the Grenadines: Where Are We? Where Do We Need To Go?
Get PDFBackground: Little is known about the status of program evaluation culture and practice in the English Speaking Commonwealth Caribbean (ESCC). This study examined the extent of evaluation culture and practice in four small Caribbean nations: Barbados, Belize, Guyana, and Saint Vincent and the Grenadines.
Purpose: The objective of this study was to learn about public sector program evaluation practices and culture in these four countries.
Setting: Public sector agencies in the four countries.
Intervention: NA
Research Design: Cross-sectional survey
Data Collection and Analysis: A total of eighty-eight public sector officials participated in a cross-sectional survey study.
Findings: The research highlighted that while program evaluation is being practiced in the public sector in all four countries, the extent of practice varied among the countries. One noticeable factor that affected program evaluation practices was the financial costs associated with program evaluations. This problem was particularly evident in Guyana where program evaluations were conducted primarily to satisfy funding requirements. However, despite the variation in practices, all four countries recognized that program evaluation is a valuable and important activity. Solutions proposed by the study participants to build a more vibrant public-sector program evaluation culture included making program evaluation culture a part of organizational culture, building organizational monitoring and evaluation (M&E) capacity, and making program evaluation a regular mandatory public sector activity
Using Action Research to Build Evaluation Capacity in Public Health Organizations
Get PDFBackground: New practice standards in Ontario require the ongoing evaluation of public health programs. However, public health units have limited capacity to conduct and use evaluations.
Purpose: The purpose of this study was to assess an action research approach as a means to build evaluation capacity in public health units.
Setting: 36 Canadian public health units in Ontario.
Intervention: Action research for evaluation capacity building.
Research Design: Multiple-case study.
Data Collection and Analysis: Qualitative, semi-structured interviews were held with study participants after the design and implementation of evaluation capacity building strategies in their organizations. Analysis was conducted using the general inductive approach (Thomas, 2006).
Findings: Evaluation capacity building is well-supported by an action research approach
Being an Evaluator: Your Practical Guide to Evaluation
Get PDFA review of the book Being an Evaluator: Your Practical Guide to Evaluation by Donna R. Podems published in 2019 by Guilford Press
Donald T. Campbell’s Evolutionary Perspective and its Implications for Evaluation
Get PDFBackground: Donald T. Campbell, a scientist, humanist, and generalist, left an indelible mark on the evaluation discipline through his methodological work. He is less well known within the evaluation community for his landmark contributions to biology and philosophy. Yet, the evolutionary epistemology that he pioneered has significant implications for evaluation.
Purpose: This article examines the relevance of Donald T. Campbell’s blind variation and natural selection approach to evaluation theory, including an elucidation of its basic logic, its social remit as a discipline and trans-discipline, and its summative and formative functions. It also sketches in broad strokes the implications of evolutionary thinking for evaluation practice, including natural and artificial selection, ontogeny, phylogeny, co-evolution, and feedback. Finally, it comments on Campbell’s Experimenting Society vision and the ongoing craze for randomised evaluations in development through an evolutionary lens.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Communication for Social Change: Seldom a Stand Alone, and Rarely Verified: Seldom a stand alone, and rarely verified
No full textBackground: Communication for social change is rarely a stand-alone initiative. More often it is combined with several communication purposes such as networking, organizational visibility, information dissemination, or behavioural change.
Purpose: This article reports on an inter-disciplinary, capacity building experiment that combines communication strategy development with Utilization-Focused Evaluation (UFE).
Setting: The analysis stems from close to a dozen case studies where we tested a hybrid approach of UFE and communication strategy development. Our partners were research teams in a variety of areas including open education, open and collaborative science, Internet privacy, cyber-security, and open data. The Networked Economies Program of the International Development Research Centre (IDRC, Ottawa) funded each one of the research teams. The partners were based in different countries and had a global reach.
Intervention: The authors are members of a research project entitled “Designing Evaluation and Communication for Impact” (DECI) that provides mentoring in evaluation and communication to partners. This article focuses mainly on lessons from DECI-2, the second phase of the project that was operational from 2012-2017. DECI is led by a team in Canada and has engaged regional mentors based in Latin America, Asia and East Africa, who have provided much of the capacity building support to partners in their regions. At the end of each mentoring cycle, the DECI team produced a case study summarizing the experience. The collection of these case studies is the basis for this article.
Research Design: This article is a meta-evaluation of the experiences gained from the mentoring. It brings the findings from the grounded work and seeks to find theoretical insights from the evaluation and communication literature. Existing family trees in evaluation and communication are reviewed in search for commonalities that underlie the hybrid decision-making framework.
Data Collection and Analysis: The article leans on the findings of the case studies and the hybrid framework. Our analysis builds on earlier work by the authors in communication for social change. In particular, we analyze a common pattern where communication strategies tend to encompass several purposes in tandem. We refer to the planning steps in utilization-focused evaluation as a structured decision-making process that can help organize communication planning. Finally, we reflect on the benefits of formulating communication objectives that can be tracked or measured.
Findings: The hybrid decision-making framework allows communication planners to add some rigor to their strategies. At the same time, it invites evaluators to introduce evaluation questions about the outcomes of a communication intervention. An external evaluation of the DECI-2 project concluded that the combined decision-making process enabled partners to become better at adaptive management. The process introduced reflection spaces and helped teams adjust their projects as research findings emerged, and as conditions shifted in the policy arenas that they sought to influence.
Retrospective Pretest and Counterfactual Self-Report: Different or Same?
Get PDFPurpose: To examine discriminant validity of treatment participants’ self-report of the state they would be in had they not received treatment (counterfactual); specifically, the distinction between self-report of counterfactual and self-report of preintervention state (retrospective pretest).
Setting: An education department of a large University in North America.
Intervention: Methods of self-reporting research self-efficacy with counterfactual items and with retrospective pretest items.
Research design: A randomized comparison group design with two treatments that were defined by the version of the survey used in each. In the survey for the counterfactual condition, items about research self-efficacy without the influence of their program of studies were included. The survey in the retrospective pretest condition contained items regarding research self-efficacy before participating in their program of study. The same items about research self-efficacy at the current time (posttest) were included in both treatment conditions.
Data collection & analysis: Participants were graduate students recruited via email who answered an online survey about research self-efficacy. These students were randomly assigned to one of the two aforementioned treatments. Responses were analyzed using a mixed 2 by 2 randomized factorial ANOVA design with self-report method (counterfactual or retrospective pretest) as the between-subjects factor and time (pre and post intervention) as the within-subjects factor.
Findings: Our findings show that counterfactual and retrospective pretest scores and treatment effects computed based on these two sets of scores are virtually identical, casting doubt on participants’ ability to differentiate between a state of no treatment and a state at treatment commencement after they have received treatment
Using Program Theory to Evaluate a Graduate College Student Development Program
No full textBackground: The “3 Minute Presentation” is a graduate student competition based off the more popular “3 Minute Thesis” competition. The program aims to help graduate students learn to inform others of their research in a quick and accessible manner. Programs to engage graduate students more deeply in their education require evaluation to determine if they are useful and effective at meeting their intended goals. Evaluation literature in graduate educational programs is currently limited, but increasingly needed for both the field and the students served.
Purpose: Development and testing of a program-theory evaluation to understand participation, recruitment, preparation, training, skills, and confidence of graduate students engaging in a “3 Minute Presentation” competition at a state university.
Setting: Institution of Higher Education
Intervention: 3 Minute Presentation competition
Research Design: Mixed-method program-theory evaluation
Data Collection and Analysis: Direct observations and closed-ended survey analyzed through qualitative coding, descriptive statistics, group comparisons, and correlation analysis.
Findings: Overall, the program evaluation found, with a possible lack of diversity in participants, that the program components of recruitment, preparation, and skill development work as expected. Additionally, engagement in preparation was associated with competition scores and the perceived helpfulness of preparation was related to students’ confidence in their presentation skills. This evaluation was deemed useful for program improvement and capacity building in the program’s continuation at the university.