Improving health professions admission testing

Recent developments include a test development study, improvements to essay marking, online registration system upgrades and the delivery of candidate seminars.

A new study, conducted by ACER researchers Dr Luc Le, Mr Sam Hambur and Dr Sean Pywell, aims to find ways to improve the validity and the test development efficiency of the Graduate Australian Medical School Admissions Test (GAMSAT), by reviewing the design of the test of knowledge and reasoning in the biological and physical sciences.  The study will determine if knowledge-based or reasoning-based items are more appropriate, or whether both item styles have merit. It will also explore whether chemistry, biology and physics results should be clustered together for reporting, as is currently the case.  This has implications for item development, and for the overall look and feel of the instrument.

Since 2010 the consortium that manages GAMSAT has allocated funding for research that enhances the knowledge and understanding of the GAMSAT itself, and the role of GAMSAT in the selection of university graduates wishing to pursue a career in the health professions. Since 2009 the Undergraduate Medicine and Health Sciences Admission Test (UMAT) consortium has had an equivalent program in place. Research funds are allocated on a competitive basis to ACER and consortium universities.

The current study is the third ACER proposal to be funded.  The first – Establishing the criterion validity of UMAT: A multi-cohort study of Australian medical students – was completed in 2011.  The second project, a study investigating the understanding of people assessment and measures of interpersonal aptitude, is still underway.

Further quality improvements have been made to the marking of the written component of GAMSAT. ACER moved away from traditional hand marking to onscreen marking in 2010, and has since continued to refine the system to give a greater level of reporting on marking and marker statistics. Onscreen marking enables ACER to randomly supply markers with candidates’ essays and have each of the two writing tasks marked by three independent markers drawn from a larger pool.

Part of the process of ensuring fairness and comparability of measures of candidate writing is to find raw scores awarded by individual markers that may be implausible.  Where the three scores awarded to each task differ by more than two points, a fourth marker is used to replace the most discrepant score. New reports which identify discrepant scores awarded by markers are generated immediately, ensuring all scores are reviewed throughout the marking cycle, rather than at the end.  This has significantly improved the efficiency of the marking process.

Registrations for the 2012 sitting of GAMSAT in Ireland and Australia were taken through the new ACER Online Assessment and Reporting System (OARS). Incorporating registration through this system has afforded greater management of candidate information as well as the opportunity to offer candidates the chance to purchase and access preparation materials securely through their OARS accounts. Other benefits include candidate access to admission tickets and to results statements. Candidates can request refunds and deferrals, manage their registration details, and apply for special testing requirements directly through the portal.

In May 2012 ACER gave presentations to potential candidates at consortium universities in Australia and New Zealand. In total, around 2000 people attended the UMAT seminars held in May in Perth, Dunedin, Hobart, Sydney, Melbourne and Newcastle. A further 800 people attended the GAMSAT seminars held on the Gold Coast in May and in Melbourne last year. Presentations are published on university websites.

Seminars reviewed the test history, information on test content and preparation, an overview of the new online registration system, and information on test results.  The UMAT seminars looked at worked examples of the practice questions from the UMAT Information Booklet, which was a great success with potential candidates. Candidates were able to ask questions at each seminar, which was beneficial for them and allowed ACER to discover what candidates want to know more about.

These high-stakes tests are often unlike anything the candidates have done previously, so the seminars enabled potential candidates to see the human face of ACER and have some myths dispelled and some uncertainties clarified. Feedback from universities and also from candidates was very positive and indicated the usefulness of repeating the seminars for future test cycles.

Find out more:
For further information on health professions admissions testing visit <gamsat.acer.edu.au>