Research

Summary page

Research

Project no : nr7037
Publication title: Not just falling over the line? A snapshot of competency-based assessment

Evaluation overview

The overall aim of this evaluation was to provide indicators for improvement in competency-based assessment (CBA) policy and practice. A model was used to evaluate its effectiveness for a range of users.

What is competency-based assessment?
Under Australia's National Training Framework (NTF), competency-based assessment is defined as demonstrating:

whether a person has the skills, knowledge and experience required to perform specific tasks in the workplace, or to gain credit towards a vocational education and training qualification or course. Assessment is based on industry determined competency standards.
[ ANTA (July 1998) About competency-based assessment, The Australian Training Information Network (TRAIN)]

This evaluation was framed on the premise that the implementation of the NTF (including CBA) continues to evolve. Our interpretation of CBA acknowledges this evolution. The research therefore includes some respondents within vocational education and training (VET) who were not yet assessing against industry standards, because they had not yet been developed and/or endorsed.

Thus, the view of the researchers was that, in order to answer the research question ‘CBA—is it meeting needs? ’, it was necessary to adopt a broader definition of CBA practices based upon a wider interpretation than nationally-endorsed industry competency standards. Under this definition, assessment may be based on industry standards, competency-based curriculum outcomes or enterprise standards.

Surveys were sent to over 300 users of CBA. Case studies were conducted from three industry sectors. They covered 45 site visits, interviewing approximately 310 people from across all States and Territories. This data provided an evaluation, with judgements about the value, of CBA. Although our respondents were generally satisfied with CBA, they nonetheless considered that certain aspects required improvement.

Strategies to improve competency-based assessment
The following is addressed to the main players, and suggests strategies to:

strengthen the quality of CBA's impact on VET
increase its uptake with a broader range of users

Policy makers

Competency standards
Competency standards are the foundation of CBA. In some industry sectors, competency standards were considered to reflect good minimum standards but did not reflect the best standard. Respondents in other sectors did not think that competency standards accurately reflected the complete range of skills required at work. Additionally, some standards did not adequately include essential underpinning knowledge, making the CBA process less relevant to the needs of those industries.

Some respondents found means of adding value to existing competency standards in their CBA policy and practice. This value often took the form of enterprise standards or customised assessment resources.

A review of competency standards for their value and relevance to industry was suggested as a strategy that would improve CBA practice.

The language of assessment
A number of respondents expressed concern with the language of CBA. Difficult terminology, constant changes and an increased use of jargon impressed upon them the need for an overhaul of assessment language.

The language of assessment is an important starting point to participation, and it is at this point that people are seeking a more refined, user-friendly strategy.

Marketing of CBA
Most respondents in this evaluation confirmed that there was a wealth of information about assessment in circulation. In fact, for most, the amount of material available was overwhelming. Yet, despite such a plethora of resources, information and suggestions, many still felt the need for more targeted marketing. For many, CBA could be improved if information was more clearly focused on the needs of particular industry users and learners.

It is therefore suggested that there be a creative, national marketing/communication strategy aimed at promoting the benefits of VET (and thus CBA) to users and potential users—for example employers, employees, schools and the wider community.

Industry Training Advisory Boards (ITABs)/Industry and Registered Training Organisations (RTOs)

Establishing training and assessment partnerships
Respondents identified establishment of training and assessment partnerships between industry and RTOs as being very important. Those who had already developed successful partnerships suggested that their CBA practice had improved because both parties were aware of industry realities impacting on assessment. As a result, CBA tended to be more appropriately tuned to meet industry needs.

From the RTO perspective, the evaluation revealed that those who were able to use existing resources in a more flexible way were capturing more of the VET market. This was particularly evident among private RTOs and segments of TAFE, who had successfully adapted existing systems and resources to meet the needs of their market.

Reflecting on current CBA practice
Information-sharing opportunities were seen as an effective way for industry to reflect on their current CBA approaches. Many participants suggested that a large part of their current CBA system was working well, but suggested that components of their CBA practice would be improved through an increased opportunity to network ideas, practices and policy with each other. This is already happening informally in some areas; however, respondents have suggested that more extensive formalised approaches would increase the usefulness of networking as a means of refining CBA.

CBA and Occupational Health and Safety (OH&S)
Respondents from the Construction and Community Services and Health industries linked improvements in OH&S to their use of effective CBA. For these respondents this was a major benefit of participating in VET. Such an industry/workplace benefit should be more widely communicated as a possible outcome of effective use of CBA.

Key Issues

Throughout this evaluation, four issues related to CBA emerged consistently:

grading
skills versus knowledge
quality
resources

Grading

Grading of CBA was an issue that emerged primarily from the ITABs survey and the Teacher/Trainer/ Assessor survey. The major challenge facing respondents was the need for strategies to bridge the gap between competence and excellence.

In this evaluation many employers said that grading provided them with a way to define strengths and weaknesses of their employees/learners. They said they needed more than an award of 'competent' or 'not yet competent' when making staff selections and placements.

Learners saw CBA as not recognising additional effort. Although CBA removed the fear of failure for lower achievers, learners felt that it often ignored the efforts of higher achievers. Many learners, ITABs and teachers, trainers and assessors considered graded assessment as a way to motivate employees/learners.

A number of respondents suggested that percentage marks did not have to be used in grading CBA. Rather, they were interested in exploring other ways to measure and report levels of performance.

Skills versus knowledge

The key concern of respondents was ensuring that assessment was inclusive of underpinning knowledge, skills, attitudes and ethics.

A number of participants feared that they were losing the teaching and learning of knowledge through competency-based training (CBT) and CBA. This group suggested that competency standards should adequately reflect the need for assessment of underpinning knowledge, skills, attitudes and ethics. Competency standards of all three industries covered in this evaluation were identified as having gaps in their consideration of underpinning knowledge and assessing ethics and attitudes.

Practitioners suggested that effective CBA incorporated assessment of attitudes, ethics and underpinning knowledge. Although many indicated that this was happening in an informal way, it was suggested that the CBA process and documentation needed to specifically seek assessment of underpinning knowledge, ethics and attitudes.

The term 'quality' was used to discuss processes and outcomes of CBA that were perceived to be beneficial either for an individual, an organisation or an industry as a whole. These included:

ensuring CBA systems meet user needs
using the right assessors
getting the paperwork right
quality of competency standards

Participants in this evaluation considered that significant improvement could be made to their current practice of CBA. Most were able to identify specific areas for improvement, knowing what was already working well and what needed changing. This ability to evaluate their own CBA practices and processes marked a significant shift for users of CBA. Many suggested that, previously, they would not have been able to make such evaluations, indicating that time and experience with CBA provided them with the ability to distinguish between effective and ineffective practices.

For this user group improvements in 'paperwork,' including competency standards, and in the quality of assessor training, would result in beneficial change.

Resources

Resources were defined broadly as including people, assessment materials, time and money. Respondents stressed the need to review current use of resources in order to make them more efficient and effective.

Evaluating existing resources was seen by participants as a worthwhile exercise. Teachers, trainers and assessors indicated that they were aware of the large number of resources available to them, but felt that very few were appropriate and usable.

Similarly, employers and ITABs suggested that a consideration of how people are used in CBA systems would be a valuable task. Interestingly, most respondents did not request huge amounts of additional funding or a large development of resources. Instead, they felt that a revamp of what was currently available, and a careful consideration of how people and paper resources were being used, would add value to existing CBA practice.

While previous research on workplace assessment carried out by the Centre for Vocational Assessment Research (CVAR) identified a reasonably high level of satisfaction, those respondents were less able to identify either the issues or areas for improvement than were the present respondents.

It could thus be said that, in general, today's users of CBA are more aware of their needs in this respect, and how to go about meeting them.

Developments in CBA

An overall impression derived from the evaluation was that respondents have acquired a higher level of understanding and more sophisticated uses of CBA, which has enabled them to articulate their concerns and suggest strategies for improvements.