The authentic performance-based assessment of problem-solving

A new approach to assessing problem-solving

In this project, titled the Authentic Performance-based Assessment of Problem-Solving Project, a new approach to the assessment of problem-solving has been developed. This approach is embodied in an assessment tool, the problem-solving assessment instrument, and in the method of its administration. The instrument was trialled in the study reported in this document.

The approach is authentic, as it is undertaken on participants' attempts to solve problems that occur routinely in their courses and on tasks that simulate tasks that are expected to be encountered in the workplace.

The assessment is performance based. In attempting routine tasks, learners are evaluated on the extent to which they use and can demonstrate identified problem-solving processes.

The approach is also evidence based. Participants are required to present evidence to show that they have employed identified problem-solving processes, and the assessment made on the quality of the evidence that participants can show.

The assessment is criterion based, since performance criteria are specified in advance for each of the component processes that are identified. The task of both the learner and the assessor is to interpret the evidence against specified performance levels for each process indicator that is described in the problem-solving assessment instrument.

Development of the problem-solving assessment instrument

The problem-solving assessment instrument was developed on the basis of several convergent theories of problem-solving. Its development followed four distinct phases:

• Theoretical conceptions of problem-solving were explored in order to arrive at a coherent description of problem-solving. This provided a basis for the claim of construct validity for the concept as it was implemented.
• Five major component processes in problem-solving were identified from a variety of theoretical positions. These major processes served to delineate the scope of problem-solving and to establish a basis for the content validity of this implementation.
• For each of the five major processes identified, a set of indicators was proposed. These indicators operationalised the major processes and linked the theoretical foundation of the problem-solving assessment instrument to its practical implementation.
• Finally, for each indicator, a set of performance levels was described. These levels provided a basis for scoring the evidence that learners presented to support their claims of the use of problem-solving processes.

Scope and structure of the report

The purpose of this study was to demonstrate that developments in cognitive theories of problemsolving, and in assessment, measurement and reporting, can form the basis of valid assessments of problem-solving performance. A further purpose was to investigate the relationship between demonstrated problem-solving performance and learning within a course.

The report begins with an introduction to the context in which the trial of the problem-solving assessment instrument was conducted (chapter 1). The project was carried out in the Electronics and Information Technology Program at Torrens Valley Institute of TAFE, Adelaide.

The literature review (chapter 2) of this report is extensive. It begins with a review of the emergence of key competencies in Australia. The processes that led to the definition of key competencies are an important factor in understanding some of the issues that remain problematic in this area of public policy. Key issues that have emerged from this analysis are the definition of key competencies and, in particular, their assessment, reporting and certification. Four main approaches that have been taken to the assessment of problem-solving are reviewed. Each is shown to have advantages and to meet particular needs. A broad policy approach that endorses the complementary use of several of these approaches is suggested. Several unresolved issues in their evaluation include the validity, reliability and precision, and forms of reporting, of the assessment of key competencies. This project is an attempt to suggest possible solutions to some of these issues, specifically in the area of problem-solving.

The methods employed in this research (chapter 3) have been strongly influenced by relatively new approaches to measurement. Past practice has deemed as satisfactory the assignment of numerical grades to levels of performance. However, current methods, most notably item response theory, permit the measurement properties of assessment instruments to be tested and linear measurement scales of known precision to be established. Data collected from the administration of the problem-solving assessment instrument were analysed using the Rasch measurement model. These analyses revealed that the instrument does yield satisfactory measures of problem-solving ability. Knowing the precision and the distribution of learners' problem-solving abilities enabled a number of discriminable levels of performance to be established.

The assessment method employed in this project involved two stages: self-assessment by students using the problem-solving assessment instrument, followed by validation by lecturers, also using the instrument. This approach is suggested as a means both of assessing learner performance on problem-solving and also of enhancing this generic ability and learners' explicit knowledge about it.

In addition to the primarily quantitative data that were collected, qualitative data arising from an evaluation by students of the problem-solving assessment instrument and its method of administration were also gathered and analysed. These suggest that the instrument achieved an important purpose—that the assessment process was also a significant learning activity for students.

The analysis of the data was necessarily technical, and its details are presented in chapter 4, Results. However, the results have clear implications for policy makers and practitioners in the field. These matters are discussed separately, in chapter 5, Discussion of results.

In chapter 6, Conclusions and future directions, a consolidated summary of the results of this project is presented. The instrument trialled in this project, the problem-solving assessment, shows considerable promise as a general-purpose device for the reliable assessment of problem-solving ability. Its wider use is suggested on a trial basis. Such a trial will enable more detailed analyses of the instrument itself, and also its refinement. Further, the process by which the problem-solving assessment instrument was developed has potential for the development of other instruments to measure other generic skills. A suggestion is made that instruments to assess other key competencies are developed using the development methodology that produced the problemsolving assessment instrument. The development of comparable instruments for other key competencies would allow the reporting of key competencies profiles which may prove to be attractive to graduates of the vocational education and training (VET) sector and their potential employers.

Key findings

In the main study the problem-solving assessment instrument was shown to be a reliable instrument for the assessment of problem-solving performance across a range of tasks within the Electronics and Information Technology program at Torrens Valley Institute of TAFE.

The instrument was also shown to work well in a validation study undertaken in the Certificate IV in Workplace Assessment and Training course, also at Torrens Valley Institute.

The assessment processes that were employed in trialling the problem-solving assessment instrument, which involved both self-assessment and lecturer validation, not only led to the assessment of the key competency of problem-solving, but also to its development among participants.

A strong relationship was found between problem-solving performance and educational achievement in the units of competency that students undertook.

Future directions

As a result of the research conducted in the Authentic Performance-based Assessment of Problem-Solving Project, several suggestions are made for the further development of the problem-solving assessment instrument and for the extension of the methods used in the project to other generic skills domains. These include:

• that more extensive trials of the problem-solving assessment instrument and its associated assessment processes be conducted in a wider range of courses and with a greater number of participants
• that the problem-solving assessment instrument be revised on the basis of feedback from further trials with diverse samples of learners
• that other key competencies assessment instruments be developed using the methodology that gave rise to the problem-solving assessment instrument
• that robust analytical techniques, such as those employed in this project, be used to ascertain the number of performance levels that can be discriminated for each of the key competencies that are to be assessed
• that forms of reporting based upon robust analytical procedures of reliable data be developed to better inform all stakeholders of the key competencies achievements of VET sector graduates.