Research

Summary page

Research

Project no: nr1F02

Publication title: The development of quality online assessment in vocational education and training - Volume 1

Introduction

The use of online technology for learning and assessment in the vocational and other education sectors is increasing. However, online assessment raises many challenges for the vocational education and training (VET) sector, particularly because of the specific requirements of a competency-based assessment system driven by industry standards.

The view has been expressed that there has been little use of the ‘online’ medium to construct reliable, evidence-based final assessments. Given the growth of interest in online course development, it is essential for practitioners to have access to information about the design and conduct of quality online assessment to expand their repertoire.

The research questions

This study investigated the use and potential of online assessment with the aim of identifying:

• What are the principles of quality assessment in an online learning environment?

• What assessment purposes can be supported through the use of online technologies?

• What assessment methods and/or tools work well in an online learning and assessment environment?

• What factors influence the choice and design of online assessment methods (for example, learner characteristics)?

• What are the challenges, barriers and possible pitfalls in achieving quality online assessment?

The research approach

The study drew on literature and interviews with Australian and international online designers and deliverers from the VET and university sectors. Interviews were conducted with 41 respondents to identify the key issues encompassed by the research questions. A project website was developed to assist communication with respondents. From the literature and interviews, issues were identified which, when solved, would expand the potential for quality assessment in the online environment. These issues have been explored and possible solutions identified. Finally, specific scenarios were developed for application to the VET context which highlight how the issues can be dealt with in this specific context.

The assessment context

There are different demands and emphases in delivery and assessment in the education sectors included in this study. The need in the VET sector for the demonstration of competence against industry standards focusses on assessment which collects evidence of competence across skill areas. On the other hand, the traditional emphasis in the university sector is on the assessment of the body of knowledge which underpins the discipline area. Nevertheless, the lessons learned from the experience of those in different education sectors can be instructive in solving issues and problems and expanding the potential of the use of the online environment for assessment.

The factors that define quality in assessment in VET have been identified in the Australian Quality Training Framework. These standards must be applied to any mode of delivery and assessment. While the online environment offers great potential for the development of a range of assessment activities, research indicates that a ‘blended’ delivery strategy (involving a mix of online and faceto-face learning and assessment) is generally favoured by experienced designers and deliverers. Consequently, it may not always be appropriate to conduct all assessment activity online. It is important therefore, that assessment strategies are carefully planned from the outset.

Before examining the assessment process in isolation, it must also be acknowledged that assessment is an integral part of the whole learning process. By investigating online assessment separately from the broader learning process, there is a risk that the issues involved in the design and conduct of assessment become isolated and thus distorted.

The principles of quality assessment

Principles of quality assessment are the same for online as they are in any form of delivery. These principles, set out in the Australia Quality Training Framework (ANTA 2001), include the need for assessment to be valid, reliable, fair and flexible. Assessment design also needs to be conducted in consultation with industry bodies. The online environment with its use of technology adds a layer of complexity and exaggerates the difficulties and issues associated with assessment in the VET sector generally.

Assessment purposes

The study and the literature suggest that no single assessment activity is necessarily more ‘achievable’ than another. This project found examples of all types of assessment; however, it was apparent that there is a greater degree of both experience and confidence in the use of online technology for formative assessment activity; that is, assessment which is comprised of assessment events which contribute to an overall ‘mark’.

In some cases respondents did not discuss formative assessment separately but as part of the overall learning strategies. The place of formative assessment is critical in the learning process and is accentuated in flexible and online learning arrangements. In these delivery contexts assessment needs to be structured and consciously designed into the learning process. In the face-to-face learning environment formative assessment can be included intuitively by the teacher.

Assessment methods and tools

Interviews for this study revealed that a range of methods are being used for online assessment activity. However, respondents identified that a wider range of possibilities are still to be explored. These methods tend to be in the areas of self and group assessment, using chat and bulletin boards, and group and collaborative approaches. In assessing skills, greater use of simulation and project-based assessment could expand the potential range of methods, providing evidence from ‘authentic’ environments.

The potential of the online environment to support a student- or learner-centred approach is identified by a number of respondents in the study. The assessment methods that could be further expanded are those where the learner is central to the learning process. These methods include self-assessment and project-based assessment.

Considerations for online assessment design

The choice and design of online assessment methods are influenced by the learners’ needs, their access to technology, the available resources, and to some extent, the discipline or industry area. It is important to examine what is to be learned and assessed to identify appropriate methods to demonstrate these skills. In most situations it is also important to use a range of methods to collect critical evidence to ensure valid assessment.

Issues in achieving quality online assessment

Issues identified from the literature and respondents fell broadly into pedagogical and technological areas. Technological issues, such as access to appropriate technology and sufficient bandwidth, were considered beyond the scope of this project.

The issues selected for further development were:

• using the online environment to expand the learning and assessment possibilities through selfassessment, peer assessment and group/collaborative assessment

• measuring the quality of online interactions/contributions between learners and between learners and teachers

• creating methods to ensure assessments done online yield evidence that is relevant to the skills being assessed and indisputedly from the learner; that is, assessments that are valid and authentic

• assessing learner readiness for online learning and assessment

• time and resource management for online assessors

• providing online support for online assessors

• cheating—ensuring authentic learner performance.

As well, experts consulted for this study were asked for their predictions about the future of online assessment. Advances in information and communication technology will change the face of delivery of learning programs, leading to more sophisticated diagnostic and self-assessment tools and the development of customised learning programs based on an assessment of specific learners’ needs. Standardised large-scale tests will provide solutions to time and consistency issues. For the VET sector, online technologies will play a stronger role in the assessment and recognition of current competence.