Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Democratic Evaluation & Crowdsourcing: It's a Match!
Get PDFBackground: After attending professional evaluation conferences, and having the privilege of listening to the founders of the discipline of Evaluation, the author reflects on how the new generation of evaluators should aspire to upgrade their role in society by making the most of the tools and social dynamics of new technologies, and exploring ways in which evaluation could operate in both present and future scenarios.
Purpose: To foster democratic evaluation as a powerful tool for more democratic government, by expanding traditional ways of reaching the community, either as evaluation participants or traditional funders, leveraged through new technologies.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: There are powerful potential synergies that the evaluation community could explore between participatory evaluation approaches –such as democratic evaluation– and the democratization (democratic: for all those individuals who have internet access) of participation facilitated by new technologies –in terms of voting, giving opinion, donating or contributing in some way via internet. .
One of these possibilities is known as Crowdsourcing: asking services, ideas, or content to a large group of people, and especially from an online community. This alternative has started to be used in many disciplines. In particular, two crowdsourcing modalities have been found to be directly applicable:
- Crowdvoting: asking the public's opinion regarding certain matters, not only in the data collection phase of the evaluation, but in the phases of analysis and judging, or even in the evaluation design, as a way of introducing other voices into the evaluation process.
- Crowdfunding: asking citizens to contribute with small amounts of money to support the evaluation of public services and programmes, as an alternative to depending solely on the funding decisions of traditional decision-makers. This can be very pertinent in cases where decision-makers are not following the general interest and democratic evaluation appears to be the best approach to follow for citizens to try to induce a change of policy.
However, further research is needed to explore these and other modalities and synergies, with special emphasis on experimentation to test such hypotheses
Evaluation Theory, Models & Applications (2nd ed.)
No full textA review of the book Evaluation Theory, Models & Applications 2nd edition by Daniel L. Stufflebeam and Chris L. S. Coryn published in 2014 by Jossey-Bass
Conference Evaluation Focusing on Learning and Transfer
Get PDFBackground: The article presents the results of using a model for evaluation of conferences; a model focusing on learning and transfer. Many conferences are evaluated using participant-satisfaction surveys, but satisfaction is a diffuse concept and is linked to many factors other than learning.
Purpose: The purpose of the present study was to examined how participants of four national conferences of adult learning assessed the relevance of the content of a number of workshops, how much they had learned from the workshops and whether they had used what they had learned. And to test a tool to measure this.
Setting: The subject matters of the study is four evaluations based on surveys of national conferences in Denmark in 2010, 2011, 2012 and 2013. The overall purpose of the conferences was to communicate and discuss new knowledge within the area of general adult education and vocational education and training, and to help ensure that this knowledge is subsequently used in the participants' daily practices at work.
Intervention: Testing a tool to measure learning and transfer, which is not too resource-intensive to use for managers of workshops as well as conferences to strengthen the potential for learning and transfer by participants.
Research Design: Using the same survey-based evaluation-tool in four different conferences in order to compare the results and try to find general knowledge.
Data Collection and Analysis: Collecting and analysing data on the basis of survey-data of the participant’s self-reported perception of relevance, learning and use (transfer) of the content of the workshops.
Findings: The study shows that the percentage of conference participants who experience relevance, learning and transfer from the conferences only varies marginally from year to year, while this percentage varies particularly between the individual workshops. Another finding is that participants’ assessment of relevance is more associated than satisfaction with learning and transfer
The Quality of Mathematics Education Technology Literature
Get PDFBackground: The present study evaluated the quality of 1,165 scholarly literature papers about mathematics education technology literature.
Purpose: The purpose of the present study was to determine the extent to which mathematics education technology literature reports the information needed to support the scientific basis of a study.
Setting: N/A
Intervention: N/A
Research Design: A systematic review was used to organize the data collection and analysis processes
Data Collection and Analysis: A literature search was conducted to identify scholarly papers that addressed the use of technology in mathematics education. A coding process was developed to record descriptive information about each paper. The Quality Framework developed for this process provided a structure to identify key information across research types based on types of analyses conducted, assigning a certain number of possible points based on the type of research conducted.
Findings: Dissertations accounted for a surprisingly high portion of the literature and research: 39.7% of the available literature and 57.0% of the research studies. The overall quality of the mathematics education technology literature was lower than we expected, averaging only 48.9% of the points possible. We noted that the quality of research papers, with respect to possible point values averaged 54.6% over four decades. For mathematics education technology researchers, manuscript reviewers, and editors, these results suggest that more attention is needed on the information being included and excluded from scholarly papers, especially with regard to connections to theoretical frameworks and research designs
The Promise of Participatory Evaluation in Family-Centered Rehabilitation Settings: A Qualitative Study
Get PDFBackground: Family-centered service philosophy (FCS) is an important contextual aspect of many pediatric rehabilitation programs. It recognizes the importance of supporting family relationships and the benefits of active family participation in all aspects of programming. Unfortunately, many professionals often overlook FCS philosophy when designing and implementing evaluations. Given the emphasis that participatory evaluation places on collaboration and the engagement of stakeholders, it appears to be theoretically compatible with FCS philosophy. Purpose: To explore parents’ and staff members’ perceptions of participatory program evaluation, including its congruence with family-centered service (FCS) philosophy, as well as the feasibility and practicality of using participatory program evaluation within pediatric rehabilitation centers.Setting: The study was conducted at two urban pediatric rehabilitation centers in Ontario, Canada.Intervention: Not applicable.Design: Qualitative exploration.Data Collection and Analysis: The study included qualitative interviews, focus groups and a thematic analysis. Findings: Participants described how participatory evaluation, in congruence with FCS philosophy, would increase the relevance of program evaluations for families, help support program interventions, assist in the development of clinician-parent relationships, and facilitate the empowerment of families. They also described how a lack of time, funding, and training, as well as variations in the priorities and interests of families presented challenges for using participatory evaluation within their centers
Relating Outputs, Outcomes and Impact in the Evaluation of International Development Projects
Get PDFBackground: The recent controversies on the pages of the JMDE regarding UNDP evaluations, as well as the DAC criteria, are discussed in the light of two UNDP/GEF evaluations in Latin America for which the author was primarily responsible.
Purpose: The author defends the utilization of all five evaluation criteria of the Development Assistance Committee of the OECD (the DAC criteria), and their integration via Theories of Change.
Setting: The pine/oak forest of Honduras and the mangrove regions along the coast of Brazil.
Intervention: Two projects of the Global Environment Facility (GEF) and the United Nations Development Programme (UNDP), with the respective governments.
Research Design: Mid-term evaluations with site visits and interviews.
Data Collection and Analysis: In the Brazilian case, longitudinal analysis of available data was conducted and related to the findings of the interviews and observation, as well as published reports and studies. A Theory of Change (action model) of the Honduran project was structured and graphically portrayed based on desk review of the project document and other documentation, adapted following initial interviews, and field tested.
Findings: In Brazil, preliminary evidence derived in part from Management Effectiveness Tracking Tool (METT) questions with reference to results indicates a relative lack of project effectiveness and of preliminary signs of impact. The state of Pará offers one possible exception, related in part to the fact that a certain momentum had already been built up in that state prior to project start-up and in part to the early adoption through the project of a new method of transportation of the fiddler crab (Ucides cordatus), in baskets covered with wet sponges rather than in sacks. The Honduran project, with the strong support of the national government and the UNDP field office as well as GEF, successfully adapted the provisions of the project document to pursue a more community- and community organization-centered approach, rather than relying on the outside consultants originally specified. Project-supported gathering of impact data raised environmental awareness, strengthened the local university and established a baseline for future ex-post impact evaluation
Utilization Focused Developmental Evaluation: Learning Through Practice
Get PDFBackground: Utilization-focused evaluation provides an overall decision-making framework with the intention of ensuring evaluation products and processes are actually used. Developmental evaluation provides a structure to learn from an experiment or pilot in the making and provide feedback to course-correct and improve the ongoing effort. In this paper we report on a project where we combined both into a utilization-focused developmental evaluation (UFDE).
Purpose: To determine the theoretical complementarities and the practical advantages of combining UFE with DE, by reflecting on a practical experience. We include a synopsis of the methodology along with a sample of findings, followed by a reflection of the overall process. We emphasize the conditions that enabled this experience to evolve to guide other practitioners interested in this learning approach to evaluation.
Setting: The context was the piloting of a social and financial education curriculum for youth called Aflateen that was developed by Aflatoun Child Savings International in Amsterdam and test-driven by over forty partners around the World.
Intervention: The evaluation experience took place during a ten-month period between December 2012 and October 2013. This paper provides a summary of the context and a justification for the approach.
Research Design: Not applicable.
Data Collection and Analysis: In additional to conventional data collection instruments, we applied participatory inquiry tools from Social Analysis Systems (www.sas2.net) as a means of engaging stakeholders in real-time data collection and analysis.
Findings: Examples are provided to demonstrate how some developmental evaluation findings informed organizational strategic and operational decisions during the life of the evaluation
Propensity Scores: A Practical Introduction Using R
Get PDFBackground: This paper provides an introduction to propensity scores for evaluation practitioners.
Purpose: The purpose of this paper is to provide the reader with a conceptual and practical introduction to propensity scores, matching using propensity scores, and its implementation using statistical R program/software.
Setting: Not applicable
Intervention: Not applicable
Research Design: Not applicable
Data Collection and Analysis: Not applicable
Findings: In this demonstration paper, we describe the context in which propensity scores are used, including the conditions under which the use of propensity scores is recommended, as well as the basic assumptions needed for a correct implementation of the technique. Next, we describe some of the more common techniques used to conduct propensity score matching. We conclude with a description of the recommended steps associated with the implementation of propensity score matching using several packages developed in R, including syntax and brief interpretations of the output associated with every step
How Good are Our Measures? Investigating the Appropriate Use of Factor Analysis for Survey Instruments
Get PDFBackground: Evaluation work frequently utilizes factor analysis to establish the dimensionality, reliability, and stability of surveys. However, survey data is typically ordinal, violating the assumptions of most statistical methods, and thus is often factor-analyzed inappropriately.
Purpose: This study illustrates the salient analytical decisions for factor-analyzing ordinal survey data appropriately and demonstrates the repercussions of inappropriate analyses.
Setting: The data used for this study are drawn from an evaluation of the efficacy of a drama-based approach to teaching Shakespeare in elementary and middle school.
Intervention: Not applicable.
Research Design: Survey research.
Data Collection and Analysis: Four factor analytic methods were compared: a traditional exploratory factor analysis (EFA), a full-information EFA, and two EFAs within the confirmatory factor analysis framework (E/CFA) conducted according to the Jöreskog method and the Gugiu method.
Findings: Methods appropriate for ordinal data produce better models, the E/CFAs outperform the EFAs, and the Gugiu method demonstrates greater model interpretability and stability than the Jöreskog method. These results suggest that the Gugiu E/CFA may be the preferable factor analytic method for use with ordinal data. Practical applications of these findings are discussed
Using Impact Evaluation Tools to Unpack the Black Box and Learn What Works
Get PDFResearchers and policy makers are increasingly dissatisfied with the “average treatment effect.” Not only are they interested in learning about the overall causal effects of policy interventions, but they want to know what specifically it is about the intervention that is responsible for any observed effects. This discusses Peck's (2003) approach to creating symmetrically-predicted subgroups for analyzing endogenous features of experimentally evaluated interventions and then it identifies several possible extensions that might help evaluators better understand complex interventions. It aims to enrich evaluation methodologists’ toolbox, to improve our ability to analyze “what works” in addressing important questions for policy and program practice