Lessons learned on research methodologies from assessing teaching quality

In response to the Australian Government’s evaluation of its teacher development aid investments in 2015, it began supporting a multi-year study of teacher development investments in Timor-Leste, Vanuatu and Lao PDR. As part of the Education Analytics Service, the Australian Council of Educational Research (ACER) was commissioned to investigate: To what extent does the Australian investment produce improved teaching quality and improved student learning?

Student learning outcomes are an important source of evidence regarding improved teaching quality. However, a key challenge for the multi-year study series was the availability of valid and reliable student learning outcomes data.

A systematic approach to monitoring learning achievement over time is important for any education and learning system because it enables governments to make the changes required to lift student achievement levels and equity. Sustained support is needed to implement national cycles of collection of student outcomes data. However, in the absence of consistent data approaches across the 3 countries, the study team found innovative solutions that provided a holistic view of teaching quality.

With an understanding that data about teaching needs to come from a range of sources, the team designed and implemented solutions on a country-level basis, taking lessons learned over time to enhance data quality. Importantly, these approaches acknowledge the complex nature of teacher learning and that sustained change in teaching practice and student learning takes time.

ACER embarked on a mixed-methods data collection approach to provide a more holistic view of teaching quality. In Vanuatu, existing national and regional data was used to provide a reference point for student learning in literacy and numeracy, but in Laos, where no Grade 1 national data was available, a project-specific student assessment was designed by the study team. In Timor-Leste, a range of student data sets collected by different stakeholders was examined.

In-depth case studies involving teacher and education stakeholder interviews and classroom observations enabled deeper insights into teaching practice and teachers’ experience of professional learning support. A fit-for-purpose classroom observation tool designed for the Lao PDR study was later adapted for the Timor-Leste study, and extended to Vanuatu.

Classroom observations provided important evidence of changes in teaching practices to supplement stakeholder reports, and offer an example of the adaptive approach of the study series. Each cycle of data collection was improved based on previous lessons learned across the 3 countries.

A deliberate approach to engaging in-country research partners early enabled each study to benefit from deep knowledge and understanding of local contexts, thereby strengthening the quality of sampling processes, instrument development and accuracy of data analysis and reporting. For example, a key consideration in the design of the classroom observation tool was that it realistically needed to be implemented by local researchers with no classroom-based experience.

The confidence of Lao local researchers with the classroom observation tool led to their involvement in the data analysis process, which enabled the collection of rich observation data and promoted researcher capacity building.

Their knowledge and experience was both valued and utilised, and the initial time and investment was worth tenfold when the pandemic hit.

In fact, in all 3 countries local research teams were able to continue the data collection process to a high standard with remote support from ACER. Where possible, this capacity-sharing approach will be used as a model moving forward.

Read all the lessons learned in the 3 papers below.

Teacher development multi-year studies. Insights on the challenges of data availability for measuring and reporting on student learning outcomes

Teacher development multi-year studies. Using classroom observations to investigate and understand teaching quality: Initial lessons learned

Teacher development multi-year studies. Using case studies to investigate and understand teaching quality and student learning: Initial lessons learned

Further reading

Read more research conducted as part of the Education Analytics Service.

Learn more about ACER’s education and development work.