CSPA development

CSPA content development

There are challenges in effectively addressing LLN within VET in the 21st century as literacy and numeracy are no longer just the three Rs, but are sets of complex skills. The Australian LLN standards, the Australian Core Skills Framework (ACSF), tries to represent this complexity across five core skills and five wide levels of performance.

Valid, reliable and effective methods of assessing and reporting on the LLN skills of learners, especially as part of any pre-training assessment, is crucial to improved learning outcomes and completion rates alongside meeting compliance requirements. We believe that some existing LLN assessments and related tools are not reliable, valid and therefore fair because, for example, you cannot assess an ACSF core skill across five widely spread levels using only five short, simple questions.

From a compliance and quality assessment perspective it is important to guarantee that any LLN assessment tool has been written and validated against the breadth and depth of the ACSF. ACER has a rigorous assessment development process that guarantees the quality of all test items. Multiple review mechanisms are put in place to ensure our assessments are relevant, reliable and fit for purpose.  This process is highlighted below.

Test development process

 

Quality in the CSPA design

The CSPA was designed in response to the needs our current TAFE and VET clients who asked for an assessment tool that would accurately assess the core skills of learners across the five levels of the ACSF. 

The CSPA was developed based on an agreed framework.  Key test constructs were identified in conjunction with clients to ensure the relevance of test content and an appropriate spread of item difficulty.

Six steps to test construction - Why the CSPA works for VET and adult learners  

  1. A team approach to development - CSPA items were written by experienced test developers and moderated through peer review.  Items were subject to ongoing review and improvement, with only the best items considered for the final test.
  2. Pilot testing - all prospective items were trialled with a representative group of learners allowing for important feedback to influence the final test design (see more about Learner Feedback below).
  3. Client consultation - the CSPA user group (made up of adult education providers) also played an active role in reviewing items and their relevance for learners.
  4. Trial test analysis - upon analysis, poor performing items were removed from consideration for selection into the final test.
  5. Final test form creation - all statistically sound items were reviewed against the assessment framework, an additional quality check prior to final test item selection.
  6. A test measurement scale - developed to provide a meaningful ACSF alignment for all learners.            

Learner feedback

The CSPA test items were trialled with approximately 1435 TAFE and VET students across four states of Australia throughout 2011. The feedback from learners during the trials indicated that they were able to access and answer the questions and engage with the assessment. 

Average item success rates equalled 20-40%, even on some of the more difficult items (ACSF Levels 4 and 5).  At the lower end of the scale (ACSF Levels 1-2), the success rates vary from between 90-99% of learners being able to successfully answer questions.  

Feedback from the trials and from live assessments indicates that the CSPA is accessible for learners operating at all ACSF levels.