Journal of MultiDisciplinary Evaluation (JMDE)
Not a member yet
528 research outputs found
Sort by
The Interdisciplinary Ph.D. in Evaluation: Reflections on Its Development and First Seven Years
Get PDFBackground: The Interdisciplinary Ph.D. in Evaluation (IDPE) program at Western Michigan University (WMU) was designed and developed to address the increasing need and demand for highly trained and skilled evaluators. After only six and a half years, the program’s graduates are serving in important evaluation leadership positions throughout the world.
Purpose: The purpose of this article is to provide readers with a historical overview and up-to-date description of the IDPE and is intended for those who are interested in evaluation curriculum and advances in the profession.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: In this article each of the program’s current and previous leaders reflect on their role in the IDPE.
Findings: The concept, design, and operations of this program should be of interest both to evaluation scholars and to those who administer graduate education programs in evaluation
The 2009 Claremont Debates: The Promise and Pitfalls of Utilization-Focused and Empowerment Evaluation
Get PDFBackground: Hundreds of evaluators visit the Claremont Colleges in southern California each year to discuss a wide range of topics related to improving the quality of evaluation practice. Debates between thought leaders in the field have been one of the most popular and informative ways to advance understanding about how best to practice evaluation in contemporary times.
Purpose: The purpose of this article is to provide a written transcript of the 2009 Claremont Evaluation Debates. The first debate is between Michael Quinn Patton and Michael Scriven on the promise and pitfalls of utilization-focused evaluation. The second debate is between David Fetterman, Michael Quinn Patton, and Michael Scriven on the promise and pitfalls of empowerment evaluation.
Setting: The debates occurred at the Claremont Graduate University on August 23-24, 2009. Several hundred evaluators from around the world also viewed and participated in the debates via a live webcast.
Intervention: Not applicable.Research Design: Not applicable.Data Collection and Analysis: Not applicable.Findings: Not applicable
Evaluating Faculty Perceptions of Student Learning Outcomes: A Rasch Measurement Analysis
Get PDFBackground: The importance of assessing student learning outcomes has demanded attention from most everyone involved in the higher education enterprise, as accreditation and funding implications are often linked to the results. Faculty, however, are often critical of the assessment process because outcomes assessment is costly with regard to time, energy and other resources, and evidence of its effectiveness is not always noticeable.
Purpose: The purpose of this article is to evaluate faculty perceptions of various student learning outcomes in order to determine which types outcomes are most valued by faculty.
Setting: United States.
Intervention: Not applicable.
Research Design: The HERI Faculty Survey was utilized to capture a nationally representative snapshot of faculty perceptions of student learning outcomes.
Data Collection and Analysis: The Rating Scale Model, a Rasch measurement model, was used to analyze survey data of 7,356 respondents.
Findings: Faculty from virtually all disciplines are primarily concerned with the intellectual growth of students. All other types of student learning outcomes (i.e., social, emotional, cultural growth) are of lesser concern. These findings suggest higher education institutions seeking “faculty buy-in” may want to consider focusing more on intellectual types of outcomes as these outcomes appear to best resonate with faculty, and in turn may result in less faculty resistance.
Credibility and Independence of Evaluation in UNDP: A Response to David Lempert
Get PDFThe paper contains a response to the article by David Lempert on 'Why Government and Non-governmental Policies and Projects Fail Despite Evaluation' (JMDE, Vol. 6, No. 13, 2010). The paper focuses on the role of the evaluation function in public organizations, using the example of the United Nations Development Programme. It makes broader observations regarding the independence and credibility of evaluation. Finally, the role of evaluation in oversight is discussed
The Calorie Count of Evaluation
Get PDFWe all know one kind of answer to the question of the value of evaluations; they can help to improve the evaluands, and/or establish accountability for the expenditure that created or supports or buys the evaluands, and/or increase our knowledge about the evaluands’ merit, worth, or significance, etc. Those answers are about the useful functions of evaluation; they’re like saying that food can help build bone structure and/or muscle mass and/or brain mass; these are all functions that food performs. But there’s a more fundamental and more specific level at which we say that a particular food has a certain calorie count, a certain fat content, a certain sugar content, has some iron or peanut oil or salt in it, etc. That level of analysis—a kind of nitty-gritty or component level—is very valuable for the nutritionist and for many consumer concerns. What is the equivalent of this kind of answer for evaluations? Is this a level of analysis we have been overlooking
Moving to Genuine: Credible Cultural Competence and Stakeholder Believability
Get PDFBackground: Cultural competency is becoming an increasingly important concept in evaluation. The developing discussions tend to revolve around how to become culturally competent, why it is important, and how to know when it is attained. More problematically, cultural competency seems bound by dimensions of race, even though culture represents a broader scope of characteristics.
Purpose: We review the current usage of cultural competence to point out its limitations and we suggest alternative ideas that can better facilitate communication about this essential topic. We look beyond the evaluation field to learn how cultural competence is handled in other disciplines. Particularly seeking to support communication between evaluation clients and evaluators, we offer strategies to engage in a dialogue about cultural issues.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Analytic essay.
Data Collection and Analysis: Not applicable.
Findings: Cultural competence, as a term, is inherently limiting in its connotations, implications, and ability to bring about change. Cultural humility may be a more appropriate term. Regardless of the semantics, the basic need to communicate about the topic remains. Asking for cultural competency, or humility, as we suggest, in Requests for Proposals may be one way to start the conversation. Positioning statements and focused interview questions also may serve to generate discussion between client and evaluator
What Does it Take to do Evaluation in Communities and Cultural Contexts Other Than Our Own?
Get PDFBackground: This paper builds on the growing body of evaluation literature around the importance of culture and cultural context in evaluation (e.g. Greene, 2005; Hood, Hopson and Frierson 2005; Hopson, 2009; Kirkhart, 1995 and 2005; La France, 2001).
Purpose: The place of language, culture, cultural context, and leadership roles in evaluation is explored through consideration of the question, “What does it take to do evaluation in communities and cultural contexts other than our own?”
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Findings: Attention to the location of power and privilege in evaluation, and to community engagement and ‘sense-making’ processes are the conversational starting points to begin to explore what it takes to do evaluation in communities, where the language, culture, and cultural context are different from one’s own.
How do Evaluators Communicate Cultural Competence? Indications of Cultural Competence through an Examination of the American Evaluation Association’s Career Center
No full textBackground: Cultural competency is an important but under-adopted skill among professional evaluators. Yet in the transactions around job seeking and hiring in evaluation, cultural competency is a practical and common concept. How cultural competency gets communicated in those transactions may provide insights for the field.
Purpose: The purpose of this article is to identify ways job seekers and employers discuss cultural competency in order to move toward a more widely accepted way of operationalizing the concept.
Setting: The American Evaluation Association’s (AEA) Career Center Webpage.
Intervention: Not applicable.
Research Design: A nonexperimental design was used in the study.
Data Collection and Analysis: Document review was the main form of data collection, where resumes and job postings on AEA’s webpage were systematically collected. They were then rated by the two study authors on cultural competency and interrater reliabilities were calculated. Content analysis was also used to identify themes in ways through which cultural competency is expressed or communicated in the documents.
Findings: Indicators of cultural competency are identified, in the context of employment seeking. The study also highlights the conditions which may have contributed to low interrater reliabilities and a larger need to develop a practical, operationalized definition of cultural competency, despite inherent flaws
How the I Ching or Book of Changes can Inform Western Notions of Theory of Change
Get PDFBackground: This article is the third part of a trilogy. The Eastern paradigm article (Russon, 2008) dealt with ontological issues (the nature of reality). The insight evaluation article (Russon & Russon, 2009) dealt with epistemological issues (how we can know reality).
Purpose: This article deals with a methodological issue—specifically it explores how ancient Chinese philosophy might influence the way in which modern day evaluators think about theories of change.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The paper is the result of a desk review that compared and contrasted modern theory of change literature with the writings contained in the I Ching or Book of Changes.
Findings: The authors believe that the theories of change expressed in the systems diagrams of the I Ching would be considered to be conducive to recurrent change that occurs due to cycles in the natural world. They further believe that the change which harnesses the power of naturally occurring cycles offers the best hope for long-term sustainability. This is because such change takes advantage of an intuitive cooperation with the natural order.
Conclusions: The authors believe that using the diagrams of the I Ching as a theory of change template offers modern day evaluators a number of advantages. The advantage of greater sustainability has already mentioned. The authors believe that organic theories of change also encourage evaluators to think about the contribution from many factors instead of attribution of a few factors.
Theorists’ Theories of Evaluation: A Conversation with Jennifer Greene
Get PDFBackground: Evaluation is sometimes viewed as a professional practice rather than a discipline corresponding to a well defined set of theories. However, Shadish, Cook and Leviton (1991) were able to demonstrate that evaluators’ work does have theoretical foundations. In particular, the authors identified five main elements for evaluation theory and described the contribution made to each of them by seven of the most influential scholars in the field over the last five decades.
Purpose: This paper intends to further the discussion on evaluation theory, by examining some of the contributions made Jennifer Greene, one of the most influential figures in contemporary evaluation. The paper mainly focuses on Greene’s innovative ideas on each of the five main elements of evaluation theory.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The paper is the result of a desk review of Jennifer Greene’s most relevant work on bias, objectivity, and advocacy in evaluation and a phone interview with her. For the sake of accuracy, the text of the interview and the corresponding analysis were submitted to Greene for review prior to publication.
Findings: The author shows how Greene has incorporated the five principles into her own work and how this eventually influenced her practice. Greene sorts evaluation approaches based on the interests they serve and the values they promote. However, she seems to have developed her theory on valuing further over the years and today she claims that evaluators should never privilege anyone’s specific side in the course of their assignments. Second, as knowledge is mediated by evaluators’ perceptual frames, Greene believes that an unfiltered (objective) view of the world is not feasible. Third, she views evaluation as a force for democratizing public conversations about important issues. Fourth, although the evaluator’s relationship with program staff could be collegial, Greene believes that evaluators have no authority or responsibility for the program design and implementation. Fifth, Greene declares that advocacy in evaluation is inevitable and, as a result, evaluators should play a socially enfranchising role today.
Conclusions: The debate on evaluation’s main theoretical foundations is still relevant. For this purpose, the author recommends that the five main theories applied by this article to examine Jennifer Greene’s work should be used more systematically in the future to describe and analyze evaluators’ practice. Such theoretical categories would be especially beneficial in that they will provide some common ground of understanding among both practitioners and scholars on evaluation concepts and practices which experience has shown to be in constant evolution