A more accurate measure of the off-campus student experience

The Commonwealth Department of Education and Training commissioned the Australian Council for Educational Research (ACER) to review the Learner Engagement Scale (LES), a key measure of student experience. It is administered as part of the Student Experience Survey, an annual survey of a sample of the 400 000 first and final year undergraduates studying at Australian Higher Education institutions – 80 per cent on-campus and around 8 per cent off-campus.

Over the years, LES results have consistently shown substantially lower engagement levels in off-campus students – far in excess of the variation between teaching modes that might be expected – giving rise to concern that the survey itself was biased in favour of on-campus students. The review, which reported in late 2017, made several recommendations, including:

• The inclusion of five new items specifically targeting the off-campus student experience
• The revision of existing LES items to make them more relevant to off-campus students
• The development of a new scale to more accurately measure learner engagement.

These recommendations were reviewed by the Department and the revision of existing items were accepted are now in use for the 2018 survey. Other recommendations are still under consideration by the Department.

The review process

The literature review, stakeholder consultation and psychometric analysis concluded that the existing LES primarily measured interactions and was therefore prejudiced in favour of on-campus students. It also found that often time-poor off-campus students valued different aspects of engagement and tended to privilege the relationship with their teacher, for example, over connections with other students. A redeveloped scale was piloted to test whether it more accurately measured student engagement.

Analysis of the pilot survey results showed that it worked well for both on- and off-campus students; however, it elicited more positive responses from off-campus students than the original, proving that these students are indeed engaged with their learning when questioned about factors they consider important.

Options for further redevelopment

Taken individually, the new and redeveloped survey items performed strongly for both on- and off-campus students, but psychometric analysis revealed that the survey results emphasised teaching quality rather than learner engagement. This suggests that separate scales – one measuring interactions with other students and the other measuring engagement with teachers and course delivery – may be appropriate in future iterations of the tool. The recommendations for new items to be added to the scale is being considered and reviewed further by the Department because questions around whether they accurately measure student engagement still remain.