The effects of different modes of delivery: Student outcomes and evaluations

About the study

The principal aims of this study were to determine the effects of different modes of delivery on pass rates and module outcomes, and to determine from students, their experiences and evaluations of the mode of delivery undertaken.

In the first instance pass and completion rates in six discipline groupings (accounting, commercial cookery, computing, hospitality, civil engineering and electronic engineering) were computed for each mode of delivery employed. These rates were then placed into bands of performance to show the relative ranking of the different delivery strategies. Similar rates were also computed for all vocational education and training (VET) students regardless of discipline grouping.

The analysis of student experiences and evaluations of the mode of delivery were based on their responses to a survey questionnaire.

Because the survey gathered information from only those students who had successfully completed their studies, it was considered important to access information on those students who had not completed their studies. The data on student outcomes from the national VET data collection assisted with information about this group of students.

Major findings—student outcomes

Module pass rates

This analysis has been unable to provide definitive answers about which strategy should be used to best ensure consistent successes in terms of module pass rates (MPRs) for all clients. However, it has shown that non-traditional delivery strategies are generally able to record solid performances. MPRs comprise the number of module enrolments in which a student is assessed and is awarded a pass taken as a percentage of the number of such enrolments which result in a pass, a fail and a withdrew-failed outcome. These findings show that module pass rates for all but one of the strategies were generally high. In addition, all strategies across all disciplines were able to produce pass rates which were over 80 per cent and, with the exception of one, were capable of producing pass rates over 90 per cent.

The external/correspondence mode of delivery, consistently produced MPRs for almost all discipline groupings which were typically about half the rate of those produced by other strategies, and often well below the 50 per cent mark.

It must be acknowledged, however, that the percentage of withdrew-failed outcomes for the external/correspondence delivery strategy for five out of the six discipline groupings represented about a third of all outcomes. This inflated the MPR. For this reason it was decided to develop what we have called a student-assessed pass rate (SAPR) which takes into account only the results obtained from an assessment.

Student-assessed pass rates

The student-assessed pass rate describes the number of modules in which students are assessed and are awarded a pass taken as a percentage of all modules in which students are assessed. When this pass rate is computed, the SAPRs for the external/correspondence delivery strategy produced pass rates which are similar to those produced by other strategies.

Module completion rates

An examination of module completion rates (MCRs) for the different strategies showed that all strategies were able to produce completion rates at about the 75 per cent level, and all but one able to produce pass rates which were 82 per cent or over. However the external/correspondence delivery strategy consistently produced completion rates which were in most cases substantially below those produced by other strategies.

Major findings—student evaluations

About the students

In this study all students who received instruction delivered in a predominantly face-to-face teacher-directed manner were described as traditional delivery students. All students who received instruction via alternative strategies were described as flexible delivery students.

A total of 769 students provided responses to the questionnaire survey. Traditional delivery students slightly outnumbered flexible delivery students. Flexible delivery students tended to be older than traditional delivery students. They had a median age of about 31 years as compared to a median age of about 25 years for traditional delivery students.

Reasons for choosing method of delivery

Flexible delivery students were more likely than traditional delivery students to say they had chosen the method of delivery because it fitted in with their lifestyles. Traditional delivery students were more likely to indicate that they had chosen the method because it was the only one offered.

How students learn best

Flexible delivery students were far more likely than traditional delivery students to say that they learnt best when studying individually with texts and study guides to help them, doing their own research and interacting on-line with a computer. Traditional delivery students were more likely than flexible delivery students to say that they learnt best from a lecturer in a traditional classroom, and practising doing things in a practical workshop. They were also more likely to say that they learnt best when working on a problem with other members in a group, and looking at pictures or diagrams which help explain concepts and processes.

Student self-ratings of skill levels

There were no statistically significant differences between the two groups on students' self-ratings of their literacy, language, and problem-solving skills. However the 'well above average' ratings showed that flexible delivery students were far more likely to rate themselves as 'well above average' in all cases apart from mechanical skills, than traditional delivery students. The 'below average' ratings, showed traditional delivery students to consistently rate themselves at this level at a slightly higher rate than flexible delivery students.

Preparation and support for learning

These findings indicate that students generally valued the training they had undergone whatever the delivery strategy. Students believed that they had been provided with the assistance they required prior to commencing their studies as well as that required for the duration of their studies.

Problems and concerns

The most common problems for flexible delivery students included making enough time to study and fitting in family obligations with their study commitments. Completing assignments to deadlines, finding enough time to study, and following instructions for practical sessions were the most common problems for traditional delivery students.

Perceived advantages of the delivery methods

Flexible delivery students cited advantages which related to flexibility; that is, they appreciated the opportunity the method had given them to self-pace their study program and fit in study times with work and family obligations. For traditional delivery students the most frequently cited advantages related to contact with others. This included face-to-face contact with teachers and students, and the support received from teachers.

Perceived disadvantages of delivery methods

The most common disadvantage associated with flexible delivery methods, related to lack of interaction with others. The lack of instant access to teachers when experiencing difficulties, and the lack of general interaction with other students and teachers were the most frequently reported problems. The second most frequently cited disadvantage concerned personal issues such as the self-discipline required to get things done. For traditional delivery students the most common disadvantage, identified by well over a third of students, related to time pressures.

Perceived effectiveness—student evaluations

Students were positive in their evaluations of the extent to which the method they had utilised had suited the content of the module studied and had enabled them to understand subject material, practise skills and complete course requirements. Because there were no major differences between the two groups in how they evaluated the effectiveness of their strategy, it was not possible to determine whether the traditional method helped students do better than the flexible delivery method or vice versa.

However, flexible delivery students were far less likely to claim that their method allowed them to have ready access to instructors than did traditional delivery students. Although not unusual, this finding provides us with a basis for comparing the two groups. Ready access to instructors at the time of learning is one of the central differentiating factors between flexible and traditional delivery methods.

Student satisfaction

Flexible delivery students were more likely than traditional delivery students to report that they looked forward to their study sessions, preparing for assignments and doing assessments. In addition, they were also more likely to report that they would recommend this method of learning to other students.

Suggestions for improvement

The most common suggestions for improvement for both groups related to altering the way training was delivered by adding more structure to the course. This included alterations like changing the time at which courses started, spending more time on revision, organising classes in a different way etc.

Conclusion and recommendations

Student outcomes

Although it is relatively straightforward to produce rankings of delivery strategies in terms of module pass and completion rates and student-assessed pass rates, it is more difficult to determine a direct relationship between the strategy and the outcomes produced; that is, it is difficult to relate the strategy to the outcomes. The influence of other variables needs to be taken into account. However, it is encouraging to note the strong performances of alternative methods of delivery in producing student-assessed pass rates. The success of any strategy in delivering pass rates or completion rates is strongly dependent on those who are delivering the training and the assessment as well as on those who are receiving the training and undertaking the assessment. Any evaluation of the effects of different delivery strategies needs to take into account the experiences of students and teachers.

The information on module outcomes that we have examined in part 1 of this study is based on information provided by training providers to national authorities. Another consideration that must be taken into account when evaluating the effects of different modes of delivery is the reliability of this information. If providers do not have sufficient staff and other resources to implement systems that will enable them to provide accurate information according to the AVETMIS Standard, it is difficult to determine the extent to which module outcomes are affected by different delivery strategies. Furthermore, training providers may have other reasons for reporting outcomes in certain ways.

Student evaluations

Information from students has provided a number of possible explanations for why students find difficulty in completing work or passing assessments, and while there is little in these findings to suggest that one method is substantially better than another, it is quite clear that the completion rates are poorest for the external/correspondence delivery method. Information from students has been valuable in highlighting the advantages of each strategy and the problems that can occur.

The findings also indicate that students have chosen delivery methods which harmonise with the way they believe they learn best. They also suggest that students are generally prepared to accept the responsibility for their own shortcomings.

The findings suggest that, in the long run, unless students are strongly motivated to follow a disciplined study routine, the more flexible methods of instruction may not be the most efficient for them. Busy people may need to be mindful of the fact that, although increased flexibility may bring certain advantages in terms of when, where, and how they study, this increased flexibility may not automatically translate into better pass or completion rates.

In summary, based on the findings from the student survey and on the analysis of student outcomes we can conclude that certain learning principles should guide the structure of the learning whatever the delivery method chosen. These are the need for supportive instructional activities, clear instructional materials, opportunities to discuss problems or issues with teachers and peers, availability of teacher support, timely feedback, practical examples and enough time and willingness to practise skills and meet requirements.

Recommendations for action

Because delivery strategies on their own do not produce module outcomes, the focus of these recommendations needs to be directed to students and their teachers and administrators. They address the following requirements:

1. The provision of information sessions for students. These sessions (delivered on campus, or via the internet etc.) can alert students to the particular problems associated with some methods of delivery. Students should be made aware that the external/ correspondence method may not be the only way to achieve flexibility in how, when, and where to study.

2. The provision of regular opportunities for face-to-face interaction between teachers and students and students with their peers. This can be accomplished by providing workshops prior to the commencement of the course, midway through the course and, where appropriate, prior to the assessment process. The purposes of introductory workshops would be to introduce students to teachers and other students, to explain the basic requirements of the course and to provide some helpful hints on study skills and time-management issues.

3. The provision of professional development opportunities for teachers. This can be accomplished by enabling teachers to attend special workshops designed to provide them with the information and skills required for assisting students. During these workshops teachers could share their experiences with successful strategies. For teachers in flexible delivery modes the emphasis can be on helping students to remain focussed so that they complete their work on time.

4. Appropriate and well-maintained facilities and equipment. Arrangements should be made to ensure that electronic equipment is fully maintained or kept up to date to enable students and teachers to maintain regular contact. Where students are dependent on this equipment for completing assignments, arrangements should be made to ensure their easy access to this equipment during or after hours.

Recommendations for further research

In this study there has been no attempt to control for variations in level or ability of students, and ability and experience of teachers. There has also been no attempt to control for level of course, or subject content. An experimental study in which students taking the same level of course are randomly assigned to delivery strategies which are provided by teachers of similar ability and experience, would further increase our knowledge of the effects of different modes of delivery on module outcomes.

Another area worthy of further research is the actual status of those outcomes which are reported as unknown.