Building assessment expertise in Armenia

Having already achieved near-universal enrolment in Grades 1 to 9, development efforts in Armenia are now focused on improving the quality of education. The World Bank reports that Armenia has made considerable progress in this area over recent years, with such reforms as the development of a national curriculum framework, mandatory teacher certification and professional development policies.

However, results from the Trends in International Mathematics and Science Study (TIMSS) demonstrate that Armenian student performance is not improving. Between 2003 and 2011, there was no significant change in Grade 4 mean scores in mathematics and significant declines in scores for Grade 8 mathematics, while scores in science declined at both Grade 4 and Grade 8 level.

To build Armenia’s capacity to measure the quality of its education system, the World Bank funded the delivery of a five-day workshop based on its publication, National Assessments of Educational Achievements – Volume 2: Test Development and Questionnaire Design for a National Assessment of Educational Achievement. Co-author of the publication, ACER Principal Research Fellow Prue Anderson travelled to Armenia in October 2013 with ACER’s Director of Systemwide Testing, Chris Freeman, to deliver the workshop.

The workshop’s 20 participants included senior education department staff and representatives from universities and teacher education facilities, and from the Armenian Assessment and Testing Centre, which was recently established to enhance capacity to assess student performance.

As Ms Anderson explains, Armenian education had traditionally tended to match assessments to clearly stated information in prescribed text books. In contrast, contemporary international trends in education are towards valuing higher-order thinking skills such as the ability to interpret and reflect over basic information retrieval.

The workshop covered the stages of test development and questionnaire design, from the selection of stimulus materials to the construction of multiple-choice and open response items, the preparation of marking guides and data analysis.

To demonstrate how higher-order thinking skills can be measured in an assessment of reading, ACER prepared an assessment that incorporated items similar to those found in large international surveys like the Programme for International Student Assessment (PISA) and the Progress in International Reading Literacy Study (PIRLS). The assessment was translated into Armenian and administered to 50 Grade 5 and 50 Grade 9 students prior to the workshop, in order that the responses could be marked and the data analysed.

‘A major concern of the workshop participants was that in Armenia, as in many countries, education data is collected but it is not analysed in any depth or used to inform learning,’ Ms Anderson said.

‘Class reports were therefore examined to show how teachers might identify areas of strength as well as areas of weakness they needed to address.’

The workshop participants’ analysis suggested that Armenian students are not used to answering open items, supporting an opinion with evidence from the text and interpreting inferred ideas, and would need to develop these skills if Armenia intends to participate in PISA or PIRLS in the future.

The final day of the workshop saw participants develop a draft questionnaire, with ACER input, about a new Armenian social studies course recently introduced in primary schools.

Post-workshop feedback from the participants was overwhelmingly positive, with particular interest in a follow-up workshop to further build expertise in data analysis.