Comparing school learning outcomes internationally

Known as the UIS Observatory of Learning Outcomes, the project will compile indicators of achievement among primary and secondary students in more than 200 countries. As a partner in the project, ACER was involved in a pilot study of the Observatory in 2011. ACER Deputy Director of International Surveys Dr John Cresswell is leading ACER’s involvement in the project.

The Observatory of Learning Outcomes contributes to the ongoing collective efforts by nations and the international development community to emphasise the importance of learning outcomes. It forms a key part of UNESCO’s Education for All agenda; a global commitment to provide quality basic education for all children, youth and adults. At the 2000 World Education Forum, 164 governments pledged to achieve Education for All and identified six goals to be met by 2015. Goal six seeks to improve the quality of education so that recognised and measurable learning outcomes are achieved by all, especially in literacy, numeracy and essential life skills.

The design of the Observatory includes two components:

A catalogue of all measures used by countries to understand the learning levels of their primary and secondary school children (i.e. from national assessments, including exams, and international sources) as well as selected attributes of the approaches countries choose to generate and express these indicators.
A global database of internationally comparable indicators of learning outcomes by the end of primary school from existing and/or new assessments, with a focus on reading, numeracy and writing skills.

In 2011 ACER worked with the UIS during the pilot study of the catalogue component. The pilot involved collecting information about educational assessments from 24 participating countries.

The data collection process was centred around a template that allowed data collectors to record information about the attributes of each assessment—for example, if it was a national or international assessment, whether it was norm referenced or criterion referenced, if it was administered as a sample or to a census of students.

Following completion of the individual country reviews, ACER combined the information and designed two different ways to report on each country’s assessments. The design of the first style of report from the database emphasised each assessment and the results obtained by the country. Some of the attributes of the assessment were recorded in detail; for example, the grade and language of assessment.

The second style of reporting the information from the database was to focus on a particular subject area and display results for that area in the different assessments undertaken by the country. This allowed a side-by-side comparison of results that a country obtained in national and international assessments, allowing countries to get an indication of the level of their own national assessment compared to other countries. This was found to be very valuable in some countries where national results were very high, but international ones were much lower.

ACER prepared an evaluation report on the pilot study and submitted it to the UIS and the Advisory Board of the Observatory of Learning Outcomes in late 2011. The design of the project has now been endorsed and the UIS is in the process of considering the implications of scaling up the project from the 24 countries in the pilot study to all 200 countries in UNESCO. This will be a complex task and ACER is preparing a discussion paper to help provide direction.