28 research outputs found
Rubric Authoring Tool for Supporting the Development and Assessment of Cognitive Skills in Higher Education
No full textThis paper explores a method to support instructors in assessing cognitive skills in their course, designed to enable aggregation of data across an institution. A rubric authoring tool, ‘BASICS’ (Building Assessment Scaffolds for Intellectual Cognitive Skills) was built as part of the Queen’s University Learning Outcomes Assessment (LOA) Project. It provides a workflow for assessment choices and generates an assessment rubric that can be tailored to individual needs based on user input. The dimensions and criteria in BASICS were adapted from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics, and drew on annotations from over 900 work samples from the LOA project. This paper summarizes the development of the tool, and presents initial reliability and validity data from a pilot study. The pilot found that the BASICS developed rubric was consistent for the assessment of critical thinking and problem solving. The pilot compared assessment data derived from course Teaching Assistants with that of trained Research Assistants. Analysis found moderate intraclass correlation coefficients between the BASICS rubric and corresponding VALUE rubric dimensions, suggesting that the BASICS rubric aligned with the VALUE criteria. Preliminary findings suggest that BASICS is an effective tool for instructors to author rubrics, tailored to their own specifications for assessment of cognitive skills in a course. It is also promising as a method for aggregation of data across the institution. Researchers are conducting further investigation to evaluate the reliability of BASICS rubrics over multiple work samples from a range of disciplinary contexts
Engagement in Assessment Change and the Role of SoTL
No full textThis study follows a network-based Assessment Redesign Project at a Canadian university to investigate engagement and sustained implementation. The following strategies were employed in the project: mini-grants, embedded support, a community of practice, and social networks. Assessment facilitators worked in discipline clusters to achieve mutual goals for assessment reform targeted at the authentic assessment of critical thinking and problem-solving. Interviews were conducted with nine of the 25 project members one-year post-implementation. The study adopted a motivational theoretical lens to investigate how the experience of the Assessment Redesign Project affected motivation and the continued adoption or propagation of assessment strategies. Participants commented on how helpful the embedded support had been in building their assessment skills or knowledge. The mini-grants were used (in some cases) to fulfil scholarship of teaching and learning (SoTL) goals. All of those engaged in SoTL demonstrated intrinsic motivation for assessment change and had propagated assessment techniques or activities into other courses. In the few cases where motivation was purely extrinsic, there was no SoTL or continuation of assessment activities. This study highlights the links between SoTL and the longer-term impact of the Assessment Redesign Project. Suggestions are provided for institutions wishing to replicate outcomes from the project
Leveraging curriculum data using business intelligence to promote constructive alignment and continuous improvement at an Australian university
No full textBackground/context. The Higher Education Standards Framework 2021 [1] requires Australian higher education institutions todemonstrate their processes for quality assurance and alignment of learning outcomes across courses and programs. Lecturers, often unfamiliar with other courses’ outcomes, may not be able to identify gaps that evidence the need for change.The initiative/practice. The presentation introduces a digitised curriculum management system designed to aid in identifying gaps in the evaluation of assessment and learning outcomes. It also presents a case study which demonstrates its application to evaluate the processes of constructive alignment[2] in a specific department of an Australian university.Methods of evaluative data collection and analysis. Research Methods, employed by authors representing different contextualperspectives, include quantitative research methods providing information about the completed constructive alignment projects and project outcomes. Qualitative methods employ case study interviews and principles of critical friends [3] for data collection and analysis.Evidence of outcomes and effectiveness. Participants will be presented with a digitised curriculum tool for constructive alignment and the case study to demonstrate its effectiveness. They will reflect on the role of alignment in evaluating assessment and learning outcomes. Participants will identify strategies for engaging with colleagues at their own institutions about constructive alignment.References. [1] Tudge, A. (2021, October 8). Higher Education Standards Framework (Threshold Standards) 2021. Australian Government.Federal Register of Legislation. http://www.legislation.gov.au/Details/F2021L00488. 36 [2] Biggs, J. (1996). Enhancing teaching throughconstructive alignment. Higher education, 32(3), 347-364. [3] Handal, G. (1999). Consultation using critical friends. New directions for teaching and learning, 1999(79), 59-7
Network-Based Approach to Assessment of Cognitive Skills
No full textCognitive Assessment Redesign (CAR) project is an institution-wide, network-based approach to the development of cognitive skills in undergraduate education. This project aims to encourage first and fourth-year instructors to align skill development through the design of course assessments, to enhance cognitive skill acquisition and provide a measurement of learning. The learning outcomes for the project are framed and operationalized using the language and dimensions from the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics. An assessment redesign network was created, matching assessment facilitators who have disciplinary and educational expertise with instructors to develop authentic assessments of student learning. One of the goals of the network is to encourage sustained participation and collaboration, and to build progression in teaching and learning throughout the institution. The project also includes a standardized test for comparison to course assessment outcomes. Testing at the fourth-year level has been dependent on the use of incentives for student participation. Although recruiting instructors from the faculty of Engineering and Applied Science was initially a challenge, course instructors have reported various successes stemming from participation in the project
Informal academic networks and the value of significant social interactions in supporting quality assessment practices
No full textThis research investigated social interactions within small significant networks across a range of higher education settings to determine their role in supporting improvements to assessment. Thirty-four academic staff from three higher education settings (Australia, Canada and Sweden) provided assessment change examples and drew network diagrams to explain their interactions. Significant social interactions were defined as engaged exchanges between people who trust and respect each other, around topics that hold common value. They led to an emotional response, promoted reflection and resulted in action and/or a shift in thinking. Significant social interactions were demonstrated to be effective in supporting changes in assessment practices. The qualitative findings were supplemented with quantitative investigation of the relational ties within the networks. The most significant relational ties related to changes in the assessment were the value of the interactions (d =.64) and the similarity between individuals (d =.50). Authors recommend that leaders in higher education heed lessons learned about how value was generated within networks and utilized for improvement activities. It is suggested that the following positive change-oriented behaviours be developed and actively encouraged: Building of diverse networks; appreciating reciprocity; forging trust; creation of time and space for significant social interactions; and external recognition of the shift toward quality assessment practices. This study builds on existing literature for improving teaching and assessment in higher education, and particularly highlights the benefits of informal academic networks and the potential for significant interactions as a mechanism for change toward a quality agenda
Formative feedback and scaffolding for developing complex problem solving and modelling outcomes
No full textAN ENGINEERING DESIGN COURSE TO DEVELOP AND ASSESS CRITICAL THINKING AND PROBLEM SOLVING
No full textA challenging new engineering design course is developed as part of the Engineering Design and Practice Sequence in the Civil Engineering program. This course engages students in a cyclical design process where they plan, build, test, and evaluate a model-scale tidal current turbine. They then use their own observations and analysis to iteratively inform, improve and re-test their design.The two objectives of this paper are to provide a description of the development and structure of this design course, and to assess student learning. The Final Design Reports were externally evaluated using the Valid Assessment of Learning in Undergraduate Education rubrics. Students also completed a standardized test called the Collegiate Learning Assessment as an objective evaluation of longitudinal learning gains. The Civil Engineering students demonstrated significant improvement in critical thinking, problem solving, and written communication skills
Triangulated authentic assessment in the HEQCO Learning Outcomes Assessment Consortium
No full textThe Higher Education Quality Council ofOntario (HEQCO) has established a consortium ofinstitutions committed to the development of usefullearning outcomes assessment techniques and to theirwide-scale implementation in their institutions. Queen'sUniversity is one of three universities and three collegesof the consortium, and the Faculty of Engineering andApplied Science (FEAS) is participating due to familiaritywith assessing learning outcomes as part of accreditation.The specific learning outcomes that are of interest toQueen's are Critical Thinking, Problem Solving,Communication and Lifelong learning.The goal of this three-year project is to assess theaforementioned general learning outcomes and cognitiveskills using three assessment methods simultaneously:embedded course assessment, using meta--rubrics toscore student artifacts, and using standardizedtests/surveys. The study will document cost and timerequired to access each of these methods in specificcourses, analyze correlation between scores from thethree methods, and evaluate developments of the genericlearning outcomes over the duration of a program. Weaim to ensure that the work of outcomes assessment issustainable, works within standard course contexts, andcan be integrated into regular course activities. Thepaper identifies the goals of the project, currentapproach, and an example of data collection in one firstyearengineering design course.</jats:p
Employing a digitised curriculum management system to address the risks of contract cheating through continuous course improvement
No full textFormat of the roundtable. The roundtable discussion will focus on current approaches to addressing risks related to contract cheating, identifying assessment modes through business intelligence (BI) reporting and redesigning assessments for assurance of learning. Participants will consider how their curriculum management strategies can mediate risk for academic dishonesty and other issues of poor constructive alignment.Context/background. Assessment needs to provide an authentic, trustworthy representation of student achievement (Bond & Docky, 2010). The referenced academic essay poses a most salient risk to academic integrity (Brown, 2010). Online “paper mills” remain problematic and largely unresolved in policing compliance. Access to assessment data is necessary to identify and review academic integrity risks. Point for debate/focus of the work-in-progress/topic for discussion. A digitised curriculum management system can provide considerable assistance in identifying those courses assessed through traditional methods, placing a university at high risk for contract cheating. Accessing BI provides university administrators instant access to data relating to the threat from non-authentic, traditional assessment tasks. The presentation will then suggest concrete strategies for modifying assessment and exploring the constructive alignment of learning outcomesmore broadly.Intended outcome. Participants will explore the availability of rich data within their curricular systems to mediate these risks and will be encouraged to explore these options by engaging with colleagues to facilitate a community of practice.References. Boud, D., Dochy, F, 2010. “Assessment 2020. Seven Propositions for Assessment Reform in Higher Education.” Australian Learning and Teaching Council. www.assessmentfutures.com.Brown, G. (2010). The Validity of Examination Essays in Higher Education: Issues and Responses. Higher Education Quarterly, 64(3), 276-291
How We Know They're Learning: Comparing Approaches to Longitudinal Assessment of Transferable Learning Outcomes
No full text
