Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Evaluating Application of Knowledge and Skills: The Use of Consensus Expert Review to Assess Conference Abstracts of Field Epidemiology Training Participants
Get PDFBackground: Often evaluations of training programs are limited — with many focusing on the aspects that are easy to measure (e.g., reaction of trainees) without addressing the important outcomes of training, such as how trainees applied their new knowledge, skills, and attitudes. Numerous evaluations fail to measure training’s effect on job performance because few effective methods are available to do so. Particularly difficult is the problem of evaluating multisite training programs that vary considerably in structure and implementation from one site to another.
Purpose: NA
Setting: NA
Intervention: NA
Research Design: We devised a method of a consensus expert review to evaluate the quality of conference abstracts submitted by participants in Field Epidemiology Training Programs – an approach that can provide useful information on how well trainees apply knowledge and skills gained in training, complementing data obtained from other sources and methods. This method is practical, minimally intrusive, and resource-efficient, and it may prove useful for evaluation practice in diverse fields that require training.
Data Collection and Analysis: NA
Findings: N
Evaluability Assessment: Clarifying Organizational Support and Data Availability
Get PDFBackground: Evaluability assessment (EA) emerged in the 1970s as a way to ensure a program was ready for summative evaluation. The primary purpose was assessing the presence of measurable program objectives (Trevisan, 2007), yet evaluators conducting EA encountered difficulty with unclear, ambiguous methods (Smith, 2005).
Purpose: The purpose of this qualitative study was to clarify two aspects of evaluability assessment, organizational support and data availability. In practice, organizational stakeholders must support the evaluation project to ensure it is pursued to completion. In addition, the availability of operational data facilitates analysis of the evaluand effect.
Setting: Participants from both human services and corporate organizations participated in interviews. Participants worked on evaluation projects serving in three roles: organizational stakeholder, program evaluator, and information technology personnel.
Intervention: NA
Research Design: A qualitative research design was selected to best understand the experiences with regard to organizational support and data sufficiency of individuals who have engaged in evaluation studies and to understand how these domains affected their ability to conduct an evaluation.
Data Collection and Analysis: This study consisted of purposive sampling of 13 participants serving various roles to add breadth to the data. The researchers conducted semi-structured interviews and analyzed the data using thematic analysis.
Findings: The findings indicate the importance of specific organizational and data related considerations that affect evaluability. The researchers recommend considerations that elaborate upon the existing EA framework. The recommended evaluability considerations assist evaluators in identifying ill-advised evaluations and enhancing the likelihood of success in ongoing studies
The Role of Random Allocation in Randomized Controlled Trials: Distinguishing Selection Bias from Baseline Imbalance
Get PDFBackground: This paper addresses one threat to the internal validity of a randomized controlled trial (RCT), selection bias. Many authors argue that random allocation is used to ensure baseline equality between study conditions in a given study and that statistically significant differences at pretest mean that randomisation has failed.
Purpose: The purpose of this study was to clarify the role of random allocation in an RCT study. Is the role of random allocation to protect against selection bias? And does it have a further role, namely to ensure baseline equality and the absence of statistically significant differences between study conditions at pretest?
Setting: The participants for this study were 229 children in 1st and 2nd grade and data were collected as part of an RCT evaluation of a volunteer reading programme piloted in Ireland, Wizards of Words (WoW).
Intervention: Not applicable.
Research design: The allocation procedure adopted in this study was stratified and blocked random allocation.
Data collection and analysis: Data were collected using standardised and criterion-referenced tests of reading ability. Data were collected by qualified Speech and Language Therapists. Independent-samples t-tests were used to analyse pretest data.
Findings: The role of random allocation is to protect against selection bias, and statistically significant baseline differences can result even when random allocation has been successful. Whether or not random allocation has been successful is determined by the generation of the random allocation sequence and the steps taken to ensure its concealment. The size of differences between study conditions at pretest can be important for the analysis of posttest data but does not by itself determine whether random allocation was successful. In addition, there are serious concerns about the appropriateness of tests of significance when comparing two study conditions at pretest
How Micro-Cultures of Evaluation Practice Shape Evaluation Design
Get PDFBackground: Not applicable.
Purpose: To consider how employment settings influence evaluation design.
Setting: International conference on culturally responsive evaluation.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Not applicable
Examining the Relationship between Community Participation and Program Outcomes in a Metaevaluation
Get PDFBackground: The salience of stakeholder participation in community development is not disputed. However, there is a paucity of evidence that clearly links participation with program outcomes.
Purpose: We examined the link between participation and program outcomes.
Setting: The article discusses data collated from World Vision (WV) program evaluations. WV is a faith based, grass-root community engaging, child focused relief and development organization that works in close to 100 countries to improve and sustain child well-being. Community participation is very central to WV’s program.
Intervention: 92 community development programs evaluated between 2005 and 2010.
Research Design: We used a metaevaluation design (Meta-analysis Summaries [DeCoster, 2004]), which involves identifying the prevalence of certain effects (such as child well-being outcomes) and the strengths of relationships among those effects and certain explanatory variables (such as community participation).
Data Collection and Analysis: A document review tool comprising 327 variables was used to review program design documents and evaluation reports. Each review item generated a score whenever a positive response was checked. The maximum possible score for a program was 200. These scores were used to measure correlations among major variables of participation and program outcomes. After the program documents were reviewed, the data was then manually entered into PASW statistics for analysis.
Findings: Programs that mostly used empowering approaches engaging a higher level of stakeholder participation were more than fifteen times more likely to exhibit improvements in child health, community health, education, and protection outcomes than those that which used direct service delivery (give a fish) approaches engaging less stakeholder participation. Also, participation of vulnerable groups like children and women had a stronger relationship with program outcomes than other forms of participation that did not involve vulnerable groups.
Evaluation Independence in Organizations
Get PDFBackground: Evaluation independence is a neglected issue in the literature. Yet, it is a critical aspect of evaluation governance in organizations.
Purpose: The article draws on organizational theory, institutional economics and international development evaluation practice to define evaluation independence in organizations, outline principles geared to the design of evaluation processes within organizations and trigger a debate on evaluation independence in the evaluation community.
Intervention: Not applicable.
Research Design: Literature review informed by personal exposure to management of the World Bank Independent Evaluation Group.
Data Collection and Analysis: Not applicable.
Findings: Independent and self-evaluation are complementary and synergistic. Organizational theory demonstrates that a judicious combination of independent and self-evaluation contributes to managerial accountability and double loop learning
Descriptive, but not Injunctive, Normative Appeals Increase Response Rates In Web-based Surveys
Get PDFBackground: In an earlier experiment Misra, Stokols, & Marino (2012) found that participants who received a descriptive normative prompt in the message requesting them to complete an online survey were more likely to comply with the request compared to participants who did not receive any normative prompts.
Purpose: Building on that earlier study, the present field experiment compared the separate and additive effects of descriptive and injunctive norm- based persuasive messages on response rates of online surveys. We also investigate the influence of email reminders on response rates.
Intervention: Participants in an interdisciplinary conference were assigned to one of four groups. The three experimental groups received one of the following messages asking them to complete an online survey that highlighted: (1) a descriptive social norm indicating typical response rates among attendees of prior similar conferences; (2) an injunctive norm appealing them to join fellow participants in completing the survey; or (3) both social norms. The control group received a generic request to complete the online survey without any norm-based appeals.
Research Design: This study used an experimental design which afforded a within subjects replication of participants’ assignment to each of two treatment groups and the control prompts for two successive surveys.
Data Collection and Analysis: One hundred and twenty nine conference participants were requested to complete an online questionnaire about their experiences of an interdisciplinary conference in Fall 2011, immediately following the conference (Time 1) and then three months later (Time 2). At both time points, participants were given three weeks to complete the online surveys. For both time points, participants who had not completed the survey one week after they were sent the initial request to complete the survey received an e-mail reminder. One week after the first email reminder, participants who had still not answered the survey were sent a second email reminder. Once the online surveys were closed at both time points, response rates were calculated. To evaluate the differences in response rates among the various treatment and control groups at both time points, we conducted one-tailed z-tests for proportions to ascertain the z-scores.
Findings: Participants receiving a message highlighting the descriptive social norm when asked to complete an online survey were more likely to comply with the request compared to all the other groups. Additionally, one and two email reminders were found to be effective in improving response rates of online surveys.
Bhagavad-Gita: History’s First Manual on Results-Based Management
Get PDFBackground: This is one in a series of articles in which the author attempts to relate Eastern philosophy to contemporary programme planning and evaluation.
Purpose: The author compares and contrasts the Bhagavad-gita with contemporary guidance on Results-based Management.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The author used the four pillars identified as being the base of RBM from a UNDP technical note as the framework for the analysis.
Findings: Not applicable.
Conclusions: In RBM strategic goals are defined that initiate a causal process in which results are specified to achieve the goals, and these, in-turn, determine programmes, processes and resources.
One of the central ideas of the Bhagavad-gita is the performance of activities without attachment or any expectation of results. This is a concept to which the tome refers as renunciation. Furthermore, the Gita states that renunciation enables us to enter into a conscious alignment with the higher order
Rubrics: A Method for Surfacing Values and Improving the Credibility of Evaluation
Get PDFBackground: The challenges of valuing in evaluation have been the subject of much debate; on what basis do we make judgments about performance, quality, and effectiveness? And according to whom? (Julnes, 2012b). There are many ways identified in the literature for carrying out assisted valuation (Julnes, 2012c). One way of assisting the valuation process is the use of evaluative rubrics.
This practice-based article unpacks the learnings of a group of evaluators who have used evaluative rubrics to grapple with this challenge. Compared to their previous practice, evaluative rubrics have allowed them to surface and deal with values in a more transparent way. In their experience when evaluators and evaluation stakeholders get clearer about values, evaluative judgments become more credible and warrantable.
Purpose: Share practical lessons learned from working with rubrics.
Setting: Aotearoa (New Zealand).
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: They have found that while evaluative rubrics look beguilingly simple they are hard to do well. However, when done well, evaluative rubrics can substantially increase the use and credibility of evaluation
Methodological Pathways to Improvements of Evaluation Approaches: The Case of Irrigated Agriculture Performance Evaluation
No full textBackground: Irrigated agriculture is often evaluated but few reviews of evaluation methodology adapted to this object are available in the literature. Besides, recommendations to improve evaluation in this field are lacking.
Purpose: The purpose of the paper is to contribute filling this gap.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: This review shows the evolution of evaluation methodology in the field of irrigated agriculture pointing out a trend towards more comprehensive methodologies. The review also suggests some methodological tools to improve evaluation process