Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
Viewpoint on Ethical Reflection in Evaluation Practice in Multiactor Networks
Get PDFBackground: In the evaluation process, it is not only the organising of different views and action plans so as to reach the mutual understanding that advances evaluation utilization through argumentation. It is above all complicated to compose a framework that can ensure clear guidelines for ethical evaluation practice in specific contextual situations and in complex operational environment with conflicting role expectations.
Purpose: The aim of this article is to suggest guidelines for ethical reflection in evaluation practice in multiactor networks. For this purpose, we study the normative features of the use of language, illustrate the complexity of ethical decision-making, and discuss the importance of professional virtues in ethical reflection.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: The results indicate that from speech acts it may be impossible either to logically derive moral duties or obligations to act, or to present idealising suppositions of such rules for dialogical situations as would ensure the production of universal norms for participants in a conversation. However, the argumentation process is fruitful expressly when the participants can set mutual understanding as a goal and commit to aspire to that goal—although it will be impossible to reach it perfectly in practice. The use of neither extensive principles nor reflection on several theories can ensure a clear view of the situation.
Conclusions: The ethics of evaluation is mostly concerned with balancing the conflicting principles and values. Therefore in ethical reflection, the focus should be on commitment to a certain reflective professional way of life in which the identification and acquirement of professional virtues have an important role
Why Government and Non-Governmental Policies and Projects Fail Despite ‘Evaluations’: An Indicator to Measure whether Evaluation Systems Incorporate the Rules of Good Governance
Get PDFBackground: While much has been written in the evaluation literature on the theory of evaluations and on specific cases, there is still no comprehensive and easy to use indicator that can be used to hold organizations to the principles of effective evaluation, to score their quality in several areas, and to offer an immediate diagnostic for improvements.
Purpose: The article offers an easy-to-use indicator for measuring whether organizational evaluation policies for government and non- governmental organization spending actually protect the public interest in line with good governance and management principles or whether they serve, instead, to protect bureaucracies and hide wrongdoing under the cover of an “outside” evaluation.
Setting: The primary focus of the piece is on international development evaluations but the author shows how the same indicator and model can be used for other government agencies as well as businesses, with modifications.
Intervention: The article examines failures of evaluation systems in light of the principles for quality and shows how an indicator can be used to measure and prevent those failures.
Research Design: The piece defines the principles of evaluation systems and accountability using both the frameworks of international agencies, themselves, and professional texts, places them in a model framework for effective evaluations, and then turns this framework into a set of questions to derive an indicator.
Data Collection and Analysis: The article offers a sample detailed test of the indicator using the United Nations Development Programme (UNDP) as a case study. Several other organizations of different types are also tested briefly to yield a variety of results on the quality of their evaluation systems.
Findings: Use of this indicator on several organizations including those of the UN system, the EC, U.S.A.I.D., reveals that a number of governmental agencies and contractors (and particularly in the field of international development) are actually failing to protect the public interest and are using evaluation processes as tools to cover up abuses and mistakes and to advocate or advertise for more funding
Being Blind in a World of Multiple Perspectives: The Evaluator’s Dilemma Between the Hope of Becoming a Team Player and the Fear of Becoming a Critical Friend with No Friends
Get PDFBackground: A large number of evaluation theorists have debated the issue of objectivity and bias in evaluation over the last five decades. In particular, the degree to which distance from the evaluand enhances the validity and reliability of evaluation findings has been a prominent topic of discussion.
Purpose: This article has two primary objectives. First, it intends to present some of the positivist and post-positivist theories on distance that have dominated the evaluation discourse since the late 1960’s, by also showing the limitations of their respective assumptions. Second, it describes a more recent evaluation theory on distance that is proving to help evaluators rapport with their evaluand more effectively, especially in the case of complex programs involving a large variety of stakeholders.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The paper is the result of both a desk review and a series of interviews with some major evaluation theorists aimed to compare and contrast some of the most relevant ideas on distance in evaluation expressed over the last five decades.
Findings: The author shows that evaluators today are still facing the dilemma of whether they should seek proximity to or distance from the evaluand. However, the author identifies an increasingly popular evaluation approach (herewith referred as the pluralist approach as opposed to the niche approach) that promises to overcome the issue of distance in evaluation more successfully than any other earlier theory. The author dismisses the idea of absolute distance, predicated by both Scriven and Campbell. In doing so, he also shows that evaluators who are closer to the evaluand and the context contiguous to it tend to have a deeper understanding of the issues at stake and therefore enhance the overall quality of their evaluation. In addition, the author acknowledges that evaluators today have a new important role to play vis-à-vis their evaluand: mediating stakeholders’ competing values and agendas for the sake of equity and social justice.
Conclusions: The author concludes that the proximity to the evaluand and the integration of multiple perspectives in an evaluation represent two of evaluation’s most enriching—rather than detrimental—factors. The author also asserts that a truly participatory approach can effectively coexist with advocacy, so long as evaluators are able to clarify their stances vis-à-vis the social, economic, and political issues associated with their evaluand.
Inclusiveness and Social Justice in Evaluation: Can the Transformative Agenda Really Alter the Status Quo? A Conversation with Donna M. Mertens
Get PDFBackground: Evaluation is sometimes viewed as a professional practice rather than a discipline corresponding to a well defined set of theories. However, Shadish, Cook and Leviton (1991) were able to demonstrate that evaluators’ work does have theoretical foundations. In particular, the authors identified five main elements for evaluation theory and described the contribution made to each of them by seven of the most influential scholars in the field over the last five decades. Purpose: This paper intends to further the discussion on evaluation theory, by examining some of the contributions made by Donna Mertens. The main focus of the paper is on her innovative ideas on each of the five main elements of evaluation theory. Setting: Not applicable. Subjects: Not applicable. Research Design: Not applicable. Data Collection and Analysis: The paper is the result of both a desk review of her work on transformative evaluation, inclusiveness, and social justice, and a phone interview with her. For the sake of accuracy, the text of the interview and the corresponding analysis were submitted to Mertens for review prior to publication. Findings: The author demonstrates how Mertens has incorporated the five principles into her own work and practice. According to Mertens, evaluators need first to be cognizant of the plurality of values held by the communities where they work and second to let those very same values guide the design, implementation, and use of evaluation. Based on her belief that knowledge is not neutral but is influenced by human interests, Mertens calls upon evaluators to play a more active role in the construction of new and “socially compensatory” knowledge. In doing so, Mertens particularly stresses how evaluation practice could address and challenge the social, cultural and economic inequities perpetuated by the status quo. Conclusions: Pursuing inclusiveness and social justice in evaluation is possible. Transformative evaluation is capable of altering the status quo. Transformative evaluation is instrumental in bringing about not only social change but also social transformation. This article also shows that the debate on evaluation’s main theoretical foundations is still relevant. Such theoretical categories are especially beneficial in that provide common ground for understanding. Keywords: transformative evaluation, social justice, cultural competence, inclusiveness, valuing, social programming, knowledge, practice, us
Feminist Evaluation and Gender Approaches: There's a Difference?
Get PDFBackground: Feminist evaluation and gender approaches offer evaluators distinct ways of thinking and applying evaluations. A Namibian case narrative demonstrates how feminist evaluation and gender approaches, among others, resulted in a useful and used evaluation.
Purpose: The purpose of this article is to provide readers with a historical overview and description of feminist evaluation and gender approaches and is intended for those who are interested in understanding these approaches to evaluation.
Setting: Southern Africa.
Intervention: A nonprofit that advocates for sex workers safety and the decriminalization of sex work.
Research Design: A comparative framework is used to describe feminist evaluation and gender approaches. The evaluation employs qualitative methods that explored the reality of sex work and sex workers through both semi-structured and exploratory questionnaires. The approach was guided by feminist evaluation, gender approaches, and to a lesser extent drew on several other evaluation approaches.
Data Collection and Analysis: Interviews and document reviews were used to collect data and content and thematic analyses were used to analyze data.
Findings: Feminist evaluation and gender approaches should be viewed as distinct approaches. Their use should be of interest both to evaluation scholars and to those who design, implement and/or use evaluations
Quantitative Methods for Estimating the Reliability of Qualitative Data
Get PDFBackground: Measurement is an indispensable aspect of conducting both quantitative and qualitative research and evaluation. With respect to qualitative research, measurement typically occurs during the coding process.
Purpose: This paper presents quantitative methods for determining the reliability of conclusions from qualitative data sources. Although some qualitative researchers disagree with such applications, a link between the qualitative and quantitative fields is successfully established through data collection and coding procedures.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Case study.
Data Collection and Analysis: Narrative data were collected from a random sample of 528 undergraduate students and 28 professors.
Findings: The calculation of the kappa statistic, weighted kappa statistic, ANOVA Binary Intraclass Correlation, and Kuder-Richardson 20 is illustrated through a fictitious example. Formulae are presented so that the researcher can calculate these estimators without the use of sophisticated statistical software
Comments on the 2009 Claremont Debates on “The Promise and Pitfalls of Utilization-Focused and Empowerment Evaluation” Journal of MultiDisciplinary Evaluation Vol. 6, No. 13 (2010)
Get PDFThis piece includes general and specific comments on "The Promise and Pitfalls of Utilization-Focused and Empowerment Evaluation.
Using Readability Tests to Improve the Accuracy of Evaluation Documents Intended for Low-Literate Participants
Get PDFBackground: Readability tests are indicators that measure how easy a document can be read and understood. Simple, but very often ignored, readability statistics cannot only provide information about the level of difficulty of the readability of particular documents but also can increase an evaluator’s credibility. Purpose: The purpose of this article is two-fold: (1) to provide readers with logical reasons for using readability tests and (2) how to choose the right test for a project. Setting: United States. Research Design: A comparative framework is used to present the need for readability testing. Keywords: Readability tests, evaluation instruments, survey research, low-literacy surve
Theory Construction and Model-Building Skills: A Practical Guide for Social Scientists
Get PDFA review of the book Theory Construction and Model-Building Skills: A Practical Guide for Social Scientists by James Jaccard & Jacob Jacoby published in 2010 by Guilford
Using Abductive Research Logic to Construct a Rigorous Explanation of Amorphous Evaluation Findings
No full textBackground: Two kinds of research logic prevail in scientific research: deductive research logic and inductive research logic. However, both fail in the field of evaluation, especially evaluation conducted in unfamiliar environments.
Purpose: In this article I wish to suggest the application of a research logic—abduction—the logic of discovery—which is powerful and very effective in constructing and validating explanations of new phenomena (evaluation findings, in particular).
Setting: The primary focus of the article is theoretic with a case example illustrating the practice of using the logic of discovery.
Intervention: Not applicable.Research Design: Not applicable.Data Collection and Analysis: Not applicable.Findings: Not applicable