Citizen-led assessments: Evaluation reports

In 2014 ACER through the Centre for Global Education Monitoring (GEM) worked with Results for Development (R4D) to conduct a broad evaluation of four citizen-led learning assessments: The Annual Status of Education Report (ASER) in India, Beekunko in Mali, Jàngandoo in Senegal, and Uwezo in Kenya, Tanzania, and Uganda. The evaluation was funded by the William and Flora Hewlett Foundation, which is a main funder of these four citizen-led assessments.

A key message from ACER’s involvement in evaluating these citizen-led assessments is the essential place of a technically skilled and stable in-country project team to manage large-scale assessment programs. ACER’s approach to supporting developing countries’ educational improvement places the building of local technical capacity at the centre of any learning assessment program.

The findings of the evaluation of the four citizen-led assessments are published by R4D in:

The technical review

Assessment experts from ACER’s Centre for Global Education Monitoring conducted a technical review of the ASER, Beekunko, Jàngandoo and Uwezo citizen-led assessments, comprising analysis of:

  • the quality of the assessment tools (cognitive tests and contextual data collection instruments);
  • sampling;
  • data collection procedures;
  • data analysis; and
  • reporting and dissemination.

For each learning assessment a detailed desk review was produced. The individual desk reviews were shared with the grantees to support further development of the assessments. Several follow-up projects have since been initiated between the grantees and ACER. These projects focus on implementing recommendations resulting from the desk reviews – evidence that the grantees have welcomed the technical advice, and that they recognise the need for further development to improve the quality and impact of their assessment programs.

One example of how the recommendations from ACER’s review are realised is the action plan for Jangandoo by Laboratoire de Recherche sur les Transformations Economiques et Sociales – LARTRES. View the Janagndoo action plan in French at the bottom of the LARTRES/Jangandoo program website.
 

The quasi-experimental studies

As part of the technical review, ACER designed and implemented two small quasi-experimental studies, published by (R4D):

  • A concurrent validity study explored the relationship between performance on Uwezo and performance on Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) instruments that themselves have confirmed validity and reliability.
  • An inter-rater reliability study of Uwezo explored the agreement in the scores assigned by Uwezo volunteers and by an expert rater to children’s responses to the Uwezo tasks.

Both quasi-experimental studies were conducted in Kenya with the support of Qdata Enterprises (the local implementation partner), the Uwezo regional office, Uwezo district coordinators and volunteers, RTI-Kenya personnel and individuals who had been involved in previous RTI-led implementations of EGRA/EGMA.