Evaluation of web-based flexible learning

Context and overview

With both teachers and students having increased access to the web from home and from the workplace, there is a trend to extend technology-based learning. Furthermore, the web is seen as a means of efficiently delivering education and training when and where it is needed, and communicating with large numbers of students who may be separated from each other and from the learning centre by time or distance. It also enables learners to access information at times and places of their choice. This suggests that the new web-based technologies offer good prospects for implementing flexible delivery options. Not only do web-based learning solutions offer effective responses to requirements for efficient information transfer, they also offer powerful alternatives for obtaining effective learning outcomes.

Coinciding with the implementation of flexible delivery methodologies has been the onset of the information technology revolution and the development of digital communications technologies, such as email and the world wide web (www). In the context of educational services, web-based communications technology is not only seen as offering a supplement for traditional delivery methodologies, but also as being capable of revolutionising distance teaching and bringing on- and off-campus teaching modes closer into alignment.

Anecdotal evidence appears to suggest that web-based approaches to flexible learning have been taken up enthusiastically within the Australian vocational education and training (VET) system. Nevertheless, little of a factual nature is known about the variety or effectiveness of these new practices, particularly from a student and learning point of view.

In this report the focus is on the nature of the uptake of web-based technologies and the quality of teaching and learning emerging, since, while digital technologies can offer access to an enormous volume of information, this does not in itself translate into learning; nor does it ensure the development of expertise required for workplaces in a state of change.

Scope and objectives of report

This report provides an overview of selected online VET delivery sites in the Australian scene and, in addition, proposes a model for evaluating web-based flexible learning practices and illustrates the application of the model in two case study sites.

The broad goal of this report is to build a knowledge base in order to ensure that web-based technologies translate into effective learning practices. In addressing this challenge, the focus will be specifically on evaluation issues arising from web-based technology uptake. In achieving this, the objectives of the report are to:

• survey and describe online VET sites in a national context
• evaluate web-based flexible learning practices at selected VET sites using new purpose-built evaluation tools

Information obtained from surveys and evaluations of learning practices facilitates the development of concepts and tools useful for further evaluations of web-based learning.

In summary, the report consists of:

• a theoretical framework for web-based flexible learning evaluations derived from a 'conversational' framework for teaching obtained from research on what constitutes good teaching and learning practice
• arising from the theoretical framework, a profile of teaching/learning exchanges critical to quality learning outcomes that occur in web-based modules
• arising from these profiles and data obtained from the study of VET practices, information relevant to:
  - the improvement of web-based teaching processes and outputs
  - meeting VET quality assurance requirements relating to web-based pedagogy
  - the need for further research and refinement of evaluative tools and web-based course offerings

Outcomes of this study provide important information for shaping future policy directions and guiding program and delivery choices in flexible learning, especially those involving the world wide web.

Research methodology

This project is descriptive and analytical in nature and has hence adopted quantitative and qualitative methodologies. A theoretical framework, conceptualised in terms of 'conversational' exchanges in the context of web-based flexible learning, underpins the data collection and analysis of the study. Data were gathered through survey and case study methods.

The theoretical framework underpinning the evaluation was developed from Laurillard's 'conversational' theory (1993, 1997) in which the main types of reciprocal exchanges are idea to idea, action to action and idea to action. This research facilitates the mapping of those exchanges for particular modules, clusters of modules and courses.

Good practice in web-based learning settings will be driven by teachers and/or designers paying attention to effective use of 'conversations'. As in traditional classrooms, good practice is more than just stringing together a variety of 'conversations' per se. It is important that practitioners understand that, like the teaching strategies adopted in traditional classrooms, 'conversations' need to be appropriate and linked to the learning outcomes they are seeking to achieve. The framework developed for this research can assist in promoting a set of coherent 'conversations'. Moreover, the general nature of this framework ensures that it is applicable in a wide variety of learning contexts in vocational education.

Because of the grounded-theory approach also adopted in this research, the 'conversational' framework was also enhanced by the incorporation of relevant context variables. This revision is referred to as a descriptive framework but the 'conversational' framework is embedded within it. This enhanced framework was used for the case studies and is the basis of the tools associated with the gathering and analysis of interview data.

Key findings

Web-based technology uptake in 1998

• There was a high level of interest in the evaluation of web-based flexible learning amongst providers and teachers.
• The number of modules in the survey was surprisingly small because many advertised modules did not yet have enrolments, and because almost half of providers (8 out of 20) were reluctant to disclose detailed information for commercial reasons.
• At the time the survey was undertaken there were 74 modules from 12 providers eligible for inclusion in the survey. These were predominantly from TAFE (public provider) but also included private providers. By projecting from these figures and the number of refusals to participate in the study, it can be estimated that approximately 120 modules were offered in a web-based flexible learning mode in 1998. The median number of students in the modules surveyed was 31, with a range of 6150. These modules involved a total of about 2300 students.

Module characteristics

• The modules surveyed varied widely and covered a wide range of subject domains, including many technical areas and spanning a broad range of skills. A considerable number of modules dealt with less tangible areas such as the development of teamwork and interpersonal relationships.
• For most modules the nominal duration was 40 hours or less (median=40) as is typical of VET provision. However, two 'modules' were reported to be of over 200 hours in duration.

Evaluation of web-based learning tools

• The development and trialling of an evaluative methodology for gauging online learning in VET is indeed timely, as outcomes from this project can have a direct impact on future web-based modules on offer.
• The 'conversational' framework and its enhancements offer an effective process for the evaluation of web-based flexible learning. The tools developed provide an efficient means of data-gathering and analysis, leading, for example, to 'snapshots' which map learning activity.
• Teacher-centred approaches are dominant in current offerings despite the great potential of web-based flexible learning to engage learners in problem-solving, responding to change and improvement through self-monitoring and self-reflection.

Future directions for web-based learning environments

• The nature and frequency of interactions promoted through web-based programs have the potential to support and enhance lifelong learning with an emphasis on learner-directed learning and adaptability.
• Instructional designers of web-based programs need to take advantage of the capabilities offered by technology in ensuring that content materials encourage rich 'conversational' interactions and that student self-directedness and reflection are encouraged.

Tools to aid evaluation of web-based courses

This research has led to the development of tools for the evaluation of web-based flexible learning in VET. The tools are of two types, those that assist in the gathering of data for evaluations and those that assist in the analysis and display of results.

Email surveys facilitate an efficient means of gathering quantitative data about teachinglearning, which can then be analysed through cluster analysis and displayed using the snapshot graphing tools.

In addition, the interview and data coding tools facilitate a more detailed analysis of qualitative data. The tools indicated are helpful in displaying the range and quality of learning activity and therefore assist in making judgements about the appropriateness of instructional design and teaching methods.

Research implications

As a result of this study, various implications for VET research, context and practice have been identified. These are as follows:

• 'conversations' in web-based flexible learning
  From this research, it is clear that there is great interest and potential in web-based flexible learning for the VET sector. While the internet is central to this teaching learning mode, it should be noted that complementary offline activities are also important for good practice. What is needed for good learning is a wide range of deep 'conversational' exchanges which involve authentic practice and rich discussions among teachers and groups.
• implications for delivery
  This research covered a wide range of modules and encompassed an extremely diverse range of topics, illustrating that web-based flexible learning can make use of 'conversational' exchanges both on- and offline in the pursuit of learning outcomes. Being extremely flexible, the conversational model allows many forms of interactions-there are no best 'conversational' methods. Certain methods and combinations of methods are better than others at achieving different learning outcomes. Designers and teachers can establish the best combinations through monitoring and evaluation of practice. What is important is that good teaching in web-based flexible learning will involve engaging learners in rich 'conversations'.
• implications for the knowledge economy
  Web-based flexible learning can support the kinds of learning needed for the knowledge economy of the information society, such as lifelong learning, learner-directed learning, learning to learn, contextualised learning, customised learning, transformative learning, collaborative/co-operative learning and just-in-time learning. The evaluative tools provided in this research can help practitioners explore and expand the kinds of flexible learning 'conversations' to extend the breadth of experiences for adult learners.
• implications for course design
  The 'conversational' nature of teaching and learning provides a useful way for course designers and teachers to conceptualise their practice. Such 'conversations' should be thought of as including exchanges other than just face-to-face, and so are inclusive of all types of interactions inherent in resource-based and web-based learning. For learners to achieve expertise, learning must be deep, conceptual and reflective, with strong links between concepts and practice. Teachers need to fashion learning experiences and 'conversations', or combinations of 'conversations', which lead learners to expertise. This is similar to the ways in which teachers align their teaching strategies to the learning tasks at hand.
• implications for evaluation
  In attempting to determine the effectiveness of alternative web-based flexible learning systems, there is a need to categorise and compare groups of users as well as to describe characteristics of interactions of individuals within the same system. The analytic tools provided in this report assist this task.
• implications for further research
  An understanding of how web-based flexible learning can be developed effectively is far from complete. As the development of web-based flexible learning systems continues, it is important to develop appropriate evaluation tools. Such tools need to take account of how to document and analyse learner interactions with such systems. The tools must also take account of the fact that non-linearity and associativity can be represented in numerous ways, and accommodate the importance of both the knowledge representation and interface issues.