AUT’s approach to assessment design in an AI landscape

In the current assessment landscape, the key challenge for universities is how to assure that the learning that has taken place reflects the expected outcomes of a student graduating in a particular discipline.

AUT’s response to this challenge is to take a whole-of-institution approach to assessment redesign. This work is grounded in the assessment principles, policy and procedures that provide a foundation on which to build a sustainable approach to the integration of AI into assessment and feedback practices.

Here, we outline the key features of AUT’s approach to assessment and AI.

The assessment policy and the two-channel approach

The definition of assessment from the AUT policy is "The ongoing and systematic process of gathering evidence of what a student knows, understands, and is able to demonstrate”.

AUT has adopted the University of Sydney’s two-lane model to the use of AI in assessment and has reframed this so that assessment sits within one of two channels:

• Channel 1: Assessment tasks that take place in an invigilated, live setting, in which AI cannot be used by students during the summative judgement of the learning outcomes(s). Assessments are designed to allow the observation of expected learning in settings that are not AI-enabled.
• Channel 2: Assessment tasks that take place in a range of non-invigilated settings (in-person, online, live, asynchronous) in which the use of AI is permitted. Assessments are designed to allow the observation of expected learning in AI-enabled settings.

Importantly, both channels are valuable and contribute to developing a well-rounded, future-ready graduate. Assessments in both show evidence of different types of valuable learning.

Assurance of learning at AUT

• AUT is framing the assurance of learning as the “understanding, observing and evidencing student learning”
• At AUT, assessments in both channels will assure learning we do not, therefore, use the terms secure/open or controlled/uncontrolled
• For AUT assessments in both channels assure different types of valuable learning. Both channels are therefore controlled by design
• If we can't understand, observe or evidence the learning occurring in an assessment - which is the role of assessing – we can’t assure the learning, and the assessment will need to be redesigned so that it sits within channel 1 or channel 2
• What assurance of learning means for a particular assessment task will depend on a range of factors – including its place in the programme, the learning outcomes it is measuring
• Whether an assessment task assures the intended learning is a decision made by the programme team and the faculty Board of Studies, within AUT’s Academic Quality Guidelines

Points of observation of learning

• To assure learning, we need to understand, observe and evidence the student learning that takes place as part of an assessment task
• In invigilated settings, this assurance takes place at a single point in time (a performance, an exam, a supervised practical test)
• In non-invigilated settings, this assurance will include evidence of student learning that has taken place in preparation for the assessment task
• To support assurance of learning AUT has developed the Points of Observation of Learning (POOL) model
• The POOL model is the mechanism that surfaces the expected learning that a student should complete during the development of an assessment task
• POOLs are evidenced through the assessment rubric and carefully weighted to ensure they are valued and become part of the final submission

Supporting programme and course teams

AUT is taking a systematic approach to working with teaching staff to redesign assessments. This includes workshops that take a programme view of assessment and shorter design sprints that are focused on revising assessment tasks.

At the programme level, teams:

• Work together to create and evaluate a current state assessment map
• Consider the appropriate placing of channel 1 and channel 2 assessments, in the context of assuring the programme learning outcomes
• Redesign assessments that sit within channel 1 and channel 2

At the assessment task level, teaching staff:

• Review a current assessment task to evaluate how learning is assured
• Consider potential points of observation of learning and how they could be implemented
• Redesign of the assessment to include these points of observation of learning