National Evaluation of Adult Learners' Week 2001 and 2002

Summary of needs assessment

The Australian National Training Authority (ANTA) commissioned ACNielsen to conduct an independent national evaluation into the impact of Adult Learners' Week (ALW) as a promotional activity for adult and community education (ACE) and lifelong learning. This project was subsequently managed by the National Centre for Vocational Education Research (NCVER). This evaluation was undertaken in 2001 and 2002.

Study context

In reading these findings, it should be noted that the evaluation of Adult Learners' Week was conducted prior to Adult Learners' Week in August 2001 and 2002, and after the event in September 2001 and 2002. Each of these stages involved a random telephone survey of approximately 1000 residents aged over 18 years throughout Australia. An initial pilot test was conducted to confirm questionnaire structure, wording and length. Figures refer to the findings of the survey conducted after the 2002 Adult Learners' Week unless otherwise noted.

Qualitative research was also conducted in late October 2001 and 2002 to explore issues and responses to Adult Learners' Week communication. In 2001, this involved one focus group in each of Sydney, Adelaide, Perth, Melbourne and Brisbane. In 2002, groups were undertaken in Campbelltown, Melbourne, Bendigo, Charleville and Brisbane. Comments are made if/where applicable.

Campaign context

The 2002 Adult Learners' Week campaign targeted two distinct learning markets: an internal/industry program aimed at adult learning providers and learners using the message It's time to share the value of learning and an external or 'outreach' program aimed at older rural men, using the message Never stop learning (seeking to encourage members of the community to seek out the learning opportunities available to them).

As a result of this shift in direction, the 2002 evaluation sought to explore learning issues and campaign responsiveness among three types of learners.

• Committed learners: people who have recently done/intend to continue adult learning, including some who have done or would consider teaching or organising classes/courses for adult learners
• An outreach group: males aged 45 to 60 living in regional/rural areas who are not regular learners
• Those with barriers to learning (to assist in developing future directions for the campaign): those who have not done any adult learning and don't intend to do any in the future, including a mix of people who did not finish school, who are currently unemployed, or for whom English is a second language

Conclusions - What is adult learning? (qualitative research)

The qualitative research clearly identified three tiers of adult learning, at the core of which are structured programs (short duration/part time and primarily institutionally based and self-interest). The next level involves 'serious' or qualification/vocation-based long-term learning, followed by 'big picture' learning which incorporates life and do it yourself learning (for example, research/reading, informal learning, travel). The latter level can be classed as adult learning but not as 'proper' learning for most.

Two schools of thought emerge on what is involved in adult learning-one involves both vocational and personal interest, while the other focusses on vocation-related learning (found for some in the 'outreach' target).

Committed learners see past the immediate benefits of adult learning to life-enhancing benefits and applications of gained knowledge-it is part of life and benefits have been reinforced by positive outcomes in the past. Those with some barriers to learning or in the target outreach audience tend to focus on short-term gains such as qualifications or earning potential. They are wary of the perceived effort that learning is seen to involve, and while they have had positive experiences, other experiences have undermined their value of learning (deficiencies with trainers, poor course material, early schooling).

Conclusions - Who is the Adult Learners' Week audience?

The base audience of the campaign comprises all Australian adults: 'Adult Learners' Week is a national celebration and promotion of all forms of adult learning'. The year 2002 was the first that Adult Learners' Week has divided its focus, targeting two distinct learning markets: an internal/industry program aimed at adult learning organisations and an 'outreach' program aimed at older rural men.

As noted in the benchmark research in 2001, there is often a segment of the population with greater take-up of messages surrounding the campaign. This generally includes people who are already open to messages about adult learning, and hence, more likely to be inspired by such messages.

Seven distinct learning segments emerge in the latest study, some with surprising similarities to previous studies, and some with a stronger disposition toward learning messages.

• Three-quarters of Australians feel positively about adult learning regardless of their intention to integrate learning as part of their life.
• Within the wider population, four segments can be classed as learning advocates (even if learning is not a current activity), one segment as indifferent to learning, while two express negative opinions of learning.
• Interestingly, a direct match has been found between two particular segments in the 2000 Australian National Training Authority publication, A national marketing strategy for VET: Meeting client needs and the current Adult Learners' Week evaluation. These are the 'learn to earn' (focus on job/qualification-oriented learning and very likely to learn in the future) and 'forget it' (least value learning and unlikely to undertake more) segments. Little change has been found in their size during this period with 17% being 'learn to earners' (17% in 2000) and 10% being 'forget it' learners (8% in 2000).
• Importantly, the evaluation also identified segments which closely relate to the target audiences for the 2002 communications:
• 'Passionate all-rounders' who are positive about learning whether job or personal in nature, positive about adult learning messages, and very likely to learn in the future (11% of the community). This segment is more likely to be skewed towards an older (55+) female, and not necessarily employed (part time, professional, or retired).  
• The 'not for me' segment is positive about adult learning but is unlikely to learn for work and less likely to learn in the future (16% of the community). They are more likely to be older (45+) and male, and not necessarily employed (retired or pensioners).

Awareness of the term 'Adult Learners' Week' has been taken up by a higher than average proportion within the higher appeal group:

• In the study undertaken after Adult Learners' Week, the 'passionate all-rounders' adult learning segment shows strongest awareness (41% vs. the overall average of 29%), while the 'forget it' segment shows lower awareness (17%).
• The 'not for me' learning segment shows 'average' awareness (30%).

Knowledge of the term and awareness of associated publicity record similar demographic profiles. Recall is more likely among females, those aged over 45 and those in regional Australia, as well as those from the Northern Territory, Tasmania and the Australian Capital Territory. This pattern is evident in the both 2001 and 2002 studies.

Conclusions - Has the Adult Learners' Week audience changed?

Significant changes in awareness levels have been identified between the surveys undertaken before and after Adult Learners' Week in both 2001 and 2002. Awareness of the term rose from 23% to 28% in 2001 and from 21% to 29% in 2002, while awareness of publicity rose from 19% to 24% in 2001 and from 20% to 26% in 2002. This suggests that the changed focus in 2002 has not eroded the campaign's reach. Further, some parts of Australia are more likely to show increases:

• In relation to the term 'Adult Learners' Week', in 2002, New South Wales, Queensland and Victoria are the only states to show a significant increase in awareness between the survey periods. In 2001, Queensland and the Australian Capital Territory showed significant changes.
• In relation to publicity about Adult Learners' Week, in 2002, Tasmania is the only state to show a significant increase in awareness, with Northern Territory also showing a large increase. In 2001, Victoria and Queensland showed significant changes.

In 2002, both of the campaign target groups demonstrated significant increases in awareness from pre- to post-campaign:

• In terms of the outreach group (males aged over 45 in regional areas), awareness of the term, 'Adult Learners' Week' increased from 16% to 28%, and awareness of publicity increased from 15% to 29%.
• In terms of the internal group (those currently undertaking learning, very likely to continue learning and strongly agree that learning is important for all), awareness of the term increased from 23% to 32%, and awareness of publicity increased from 22% to 27% (not statistically significant).

As always, the achievement of a change in awareness must be balanced by the level of investment and resources used to achieve this improvement. This is one of the aims of the tracking campaign.

Conclusions - What approaches is Adult Learners' Week using?

Of those aware of some publicity or information, the main avenue is through stories or editorials (newspaper, television, radio) with a significant increase in 2002 from 47% before the week to 57% after the week.

Television announcements or advertising were mentioned by 25% both before and after the event in 2002, lower than in 2001 (42% before with 34% after).

Another significant difference between the two years is the drop in recall of print materials from 25% post-Adult Learners' Week 2001 to 1% post-Adult Learners' Week 2002.

• Slightly more of those in the 'not for me' segment (similar to the outreach target group) show awareness of stories and editorials.
• More 'passionate all-rounders' show awareness of an adult community education centre.
• More of those in the 'enough for now' segment show awareness of television and radio announcements and advertising.

There have been significant drops in the proportion of residents who have not seen any activities or promotion related to Adult Learners' Week in both 2001 (86% to 81%) and 2002 (83% to 78%). Both the outreach (88% to 74%) and internal (84% to 72%) target groups are significantly less likely in 2002 to be unaware of activities or promotion post-Adult Learners' Week. These findings suggest that the targeted approach taken in the 2002 campaign has proven effective, given the response among both target groups.

Recall of the 1300 telephone number, the Collins writing competition and local area activities appear slightly higher than noted for other activities.

Conclusions - How is Adult Learners' Week being marketed?

Of those aware of some publicity or information, there is consistently greater recall of messages that adult learning is available for adults of any age and that it doesn't matter how old you are. In 2002, the only message to show a significant increase in recall from pre- to post-Adult Learners' Week is that adult learning is for everyone.

For the segment identified as being passionate about adult learning, messages more often recalled are learning is something you can do every day/part of life and keep on trying. The 'not for me' segment (similar in profile to the outreach group) tend to recall tell people/get involved/share the value of learning and all types of learning are important. Perhaps not surprisingly, those in the 'forget it' segment didn't take any particular message away from the campaign.

The follow-up qualitative research suggests relatively clear messages are associated with the 2002 campaign. These can be expanded and their familiarity easily enhanced.

• The internal campaign is seen as aimed at learning providers (rather than individuals) and incorporates messages which encourage them to celebrate and promote their product, to 'come and teach', and to take advantage of advertising and resources. There is also a feeling that it is aimed more at self-interest and that the poster's visual cues suggest a youthful learning target.
• The outreach campaign is seen as aimed at the older male demographic with a strong message of keep learning, go out and communicate, fulfil your life, you're never too old to learn, and I did it, you can do it. It does however, obtain a mixed reaction dependent upon openness toward adult learning.

It is clear that the types of messages used in the campaign appeal to people. Consistent with last year, the most favoured message is that learning is a lifelong activity.

• It is clear that the two most negative adult learning segments, the 'forget it' and 'unenthusiastic learners' segments, are significantly less likely to find appeal in all/most of the statements related to adult learning.
• Of interest, however, is that the 'not for me' segment (similar in profile to the target outreach group and somewhat indifferent in their view of adult learning) is as likely as the general population to find appeal in the messages, suggesting they are not averse to hearing about learning per se.
• The three more positive segments ('passionate all-rounders', 'learn to earn' and 'learning on hold') show higher-than-average appeal in the statements about adult learning.

The campaign messages encourage positive feelings-people tend to strongly agree that the messages remind them of the good things about adult learning. They are prompted to think that learning could be for them, and that learning is something they'll be doing in ten years' time.

The messages are more likely to connect with the internal campaign target group than the external group.

As was found in 2001, the follow-up qualitative research identified many positive outcomes of the campaign. The messages and stories are appealing and credible, and the use of real and 'down-to-earth' people is a particular strength-those who portray learning as a positive undertaking and who help to make learning look attainable. There is a much clearer call to action in the latest campaign, and regardless of personal affinity with the message, there seems to be little difficulty in suggesting the desired outcome from the material.

In terms of areas of opportunity, the 2002 campaign has moved past some of the issues raised in 2001. Participants no longer stress changes to the delivery of the message. In 2001 more concern was directed towards reflecting the objective of celebrating learning-a desire for those used as case studies to be inspiring and enthusiastic in the way they spoke about their learning experience. Future campaigns could incorporate an 'older' face for the internal campaign, a message for individuals (if this is desired), small changes to the Adult Learners' Week website to improve user-friendliness, and an incentive other than competitions for the outreach program.

Recommendations

Focus on quality not quantity

Where a limited budget and campaign effort applies, the aim of communications should be to improve the quality of recall rather than the quantity of recall. In other words, the aim should be to obtain small but quality increases in awareness rather than seek big increases in awareness but degrade or dilute the offer/cause.

Importantly, the communication does not seem 'tired' (that is, it hasn't shown signs of wear, such as reduced awareness, inability to present a clear message) and hence, is not implying an obvious need for change. We would suggest that Adult Learners' Week campaign continue with the type of activity used in the latest campaign, including a more specific focus, which relies on a supportive network of state and territory co-ordinators and a network of learning providers and information providers.

Who is the target audience?

The change in focus for the 2002 campaign, which targeted particular segments of society (rather than offering a message directed at the entire population), has yielded a positive outcome. Awareness levels have been maintained with the target audience reflecting a responsiveness to the message and its relevance. Further, the message used in the internal and external campaigns is easily interpreted (and consolidated).

Consideration could be given to widening the internal campaign to provide greater relevance to specific individuals (that is, committed learners). The existing material is seen to relate primarily to learning providers, whether these are organisations or teachers, and hence learners see the message as less relevant to them. There is a feeling that sharing the value of learning is someone else's-the learning provider's-responsibility and area of interest.

Changing the external segment target may have varying impacts on overall awareness and response to communication. The outcome will depend on the size of the segment, the level of resistance this segment has to receiving messages about learning, and the triggers surrounding learning. The segmentation analysis undertaken in the latest study may facilitate a greater understanding of the relative value in considering particular segments. The follow-up exploratory research will provide insights for developing messages for particular groups, such as those with barriers to learning, older workers etc.

Who is the target audience?

If encouraging committed learners to become learning providers is a future priority for the Adult Learners' Week, the internal campaign may need to be diversified and expanded to demonstrate the transition from student to teacher to committed learners. It is recommended that qualitative research is first undertaken to investigate this process. Expanding the campaign in this manner may, however, be outside the scope of this campaign given its broadness of purpose.

Key performance indicators and return on investment

The obvious key performance indicator from a campaign such as Adult Learners' Week is what is happening to take-up of the message.

A possible indicator could be course enrolments; that is, at the end of the day, how much has been spent on communications for an incremental increase in enrolments? However, an indicator such as this has a number of inherent problems:

• Enrolments cannot accurately address all forms of adult learning, given the structured and unstructured nature of learning.
• Enrolments will be affected by activity outside Adult Learners' Week and, hence, direct impact will be difficult to attribute. This includes independent promotions by learning providers, media activity, other 'weeks' etc.
• There is a lag effect in take-up at this level as people may be inspired but not act on their motivation until some time later. This is because the campaign message may be only one part of the learning encouragement process, or may not 'fit' at the exact period of time that the message was received.
• This relies on reporting of enrolments and changes over time by a representative cross-section of learning providers.
• It fails to recognise that the communications can only aim to achieve so much-to reach the right people/target group, generate interest and create the environment for conversion to enrolment. What occurs after conversion is an external issue and may still disappoint/fail to meet expectations (for example, how it is handled, course content, course experience).

Another indicator could be the growing recognition and acceptance of lifelong learning by communities. While this measure has not been quantified in this study, anecdotal comments in the qualitative research (2001 and 2002) suggest it is a common and growing belief. As with enrolments, however, this is strongly influenced by aspects outside Adult Learners' Week and cannot be directly attributed to the campaign. One possibility is for future evaluations to gauge and track opinion on this issue among those aware of Adult Learners' Week and the wider population.

We suggest that a more useable Key Performance Indicators under these circumstances is a more immediate call to action which can be directly attributed to the Adult Learners' Week promotional activity. This relates to enquiries by telephone and website visitation and queries (the latter once this facility is available), and year-by-year comparisons. While this may prove the most direct measure, as noted by Adult Learners Australia, it would also need to be adopted with some caution.

As yet, there is no single call to action resource available in Australia (unlike the UK who have the Learn Direct call centre and web-based database as a single-point reference service). It should also be noted that, while the national website and 1300 are promoted on nationally produced materials, alternative websites and phone numbers are also often promoted on ALW materials produced by States or local providers. It should be recognised too that some demographics are uncomfortable using either a phone or website to gather information and may prefer to seek information in a face to face environment such as at their local learning provider, public library or at a learning provider display at a shopping mall. Finally, it should be noted that our aim with the ALW campaign is not to have people call a number or visit a website, but to actively seek out and make contact with the various learning providers in their local community.

Adjunct to campaign

To add power to the campaign, and to provide editorial content which will be of interest to media gatekeepers (for example, editors/program managers etc. with newspapers and radio/ television stations), Adult Learners' Week could create an editorial environment to enable receipt of advertising. This will facilitate the known synergy between print/visual media and editorial content (advertising is at its most powerful when able to tap into the target audience profile as well as higher audience viewing/listening/readership).

It could take the form of a syndicated interview with a respected 'knowledge' person that would attract the interest of not only the communication channel but the target audience. This could then be 'top and tailed' by the relevant community service announcements to bring the campaign or campaign 'face' to life. In essence, this will seek to create a more favourable environment for receiving the message.

In this way a low-cost supplement (using a 'real' person with 'real' life experiences) is provided to the main communications which can act as 'filler' for media providers. It is likely that non-metropolitan providers would be the main thrust of this additional communication.

This could be tied to/leveraged off the type of editorials/stories and press coverage of events by state and territory co-ordinators.

To 'week' or not to 'week'?

The use of a week has been an ongoing area of some controversy. A 'week' implies a use-by date and relevance during a certain time. Adult learning is clearly more than a one-week event, and more than a set time period-it is a lifelong activity.

Further, the adult learning message has to stand on its own, it is a long journey which should extend and stay relevant. It is critical that there is no complacency in the value of the offer or the message presented.

Despite this, there are many quite significant issues in favour of continuing with an Adult Learners' Week, to the point that a 'week' works for adult learning rather than against it.

• There is little ability to control the use and distribution of the Adult Learners' Week message and materials following its initial distribution.
• The use of a week provides a device to concentrate attention and raise the relevance of adult learning-to cut through the clutter and place value on the material, given the strong use of networking and below-the-line activity.
• A week can focus the end user (for example, learning organisations, media) who has no vested interest other than to feel good and be a good corporate citizen.

In light of this, the week focus (but with a campaign which commences early in the year and extends beyond the week with support/follow-up) is probably the best and most cost-effective way to control the end user and cut through competing messages. This still allows for innovation in developing new ways or messages to present to and reach target audiences.

Recommendations-Overview

As noted in 2001, the Adult Learners' Week campaign has created a platform for the future, and perhaps most importantly, a foundation for the communication to evolve in a workable framework over time. This is not always the case with communication campaigns, and is an important and positive step for the future development of Adult Learners' Week.

Any action points noted in this document need to be tempered by the funds available on the campaign and the support available from states and territories (including co-ordinators, learning providers, sponsors etc.).

Recommendations-National evaluation objectives

At the beginning of the 2001 evaluation, the Australian National Training Authority provided four primary objectives upon which to evaluate the Adult Learners' Week campaign. If future evaluations are undertaken, these broad objectives will remain relevant as they focus on content, delivery, audience and behaviour.

Additional criteria may provide further insight, although the low level and indirect nature of the campaign suggests that these may need to be supported by both ANTA and Adult Learning Australia. Further evaluation of Adult Learners' Week could include consideration of:

• An examination of community understandings of 'adult learning' (in terms of whether this is primarily vocational, primarily personal interest, or encompasses both), and community acceptance of the importance/need for adult learning. This can then be assessed in relation to the public awareness of Adult Learners' Week and the wider population.
• Identifying and measuring people's expectations of adult learning before and after the campaign; that is, ideally, what is the role of adult learning in the community?
• The reporting phase should include feedback from stakeholders involved in the campaign's development and delivery, such as learning providers, the advertising/communications contractor and state/territory co-ordinators. The research agency could, for example, obtain campaign performance measures during contact with state/territory co-ordinators (for example, simple ratings of their perception of campaign co-ordination, support/advice provided, openness of learning providers etc.). Adult Learning Australia in its end-of-campaign report to ANTA could include additional evaluations of the success of Adult Learners' Week sourced from learning providers or utilise results from the learning provider phone survey conducted by Adult Learning Australia staff.