A stocktake of the Longitudinal Surveys of Australian Youth

The paper assesses the fitness for purpose of the existing Longitudinal Surveys of Australian Youth (LSAY) instruments as a source of information on the determinants of youth transitions in Australia. The paper is a comparative analysis of other longitudinal studies, a review of the current direction of youth policies and programs in Australia, and a review of the literature on the school-to-work transition.

This paper was written in 2009 and all material was accurate at the time of writing.

Key messages

• The paper concludes that the LSAY survey instruments have stood the test of time. They provide a reasonable and consistent array of information to assist our understanding of the school-to-work transition. But the emerging literature and the direction of other comparable surveys suggest that, to fully understand the school-to-work transition in a changing policy context, data collection needs to be extended in both directions. That is, data on earlier circumstances are required to better understand the effects of early childhood learning on school-to-work transitions and the post-school evaluation window needs to be lengthened beyond 25 years.
• LSAY would also benefit from the collection of a richer source of information on the socioeconomic background of young people to better understand the processes whereby this background influences youth outcomes. Capturing standardised information on the health and wellbeing of young people would provide measures of successful transitions beyond economic and educational outcomes.
• Matching the LSAY data with administrative datasets should be investigated as a way of improving data quality.
• Attrition bias is an issue for most longitudinal surveys, and LSAY is no exception. However, in LSAY this is particularly problematic because the young people we are most interested in are those most likely to drop out of the survey. Consideration should be given to a focused incentive for this group of young people.
• These suggested survey design and content changes come at a cost—whether it is an additional burden on the respondents or cost implications for the overall program. Hence, some trade-offs need to be made to implement them successfully, such as increasing questioning in a certain subject area (for example, health) at the expense of reduced questioning in another area (for example, school subject information).

Tom Karmel
Managing Director, NCVER

The transition from school to work and adulthood remains an important policy and economic focus, as it has since the inception of the Longitudinal Surveys of Australian Youth (LSAY) program and its predecessors. This paper assesses the fitness for purpose of the existing LSAY survey design and questionnaires in generating data that enable us to best understand the determinants of successful youth transitions.

LSAY¹ is a research program that tracks young people from around age 15 to 25 years and captures information on how they move from school to post-school destinations. It uses large, nationally representative samples of young people and covers a wide range of education, employment and social aspects of the school-to-work transition.

In assessing the fitness for purpose of the existing LSAY instrument as a source of information on the determinants of youth transitions, we first conduct a comparative analysis of four other relevant longitudinal studies (Canada’s Youth in Transition Survey; Australia’s Youth in Focus; Longitudinal Study of Young People in England; and the United States National Longitudinal Survey of Youth) to explore their questionnaire content and survey design. Secondly, current youth policy settings at national and state and territory levels are summarised to assess the relevance of LSAY. Finally, we review the national and international research literature on youth transitions to assess the suitability of the LSAY data for youth transition research. The paper concludes with an evaluation of the fitness for purpose of LSAY and outlines key areas for future development.

The findings from this review portray LSAY as an informative source of information on the determinants of successful youth transitions in Australia. Against key longitudinal studies on youth transitions, LSAY measures up as a world-class survey which has largely kept pace with the changing landscape of youth policy in Australia. However, the review also identifies areas where LSAY lacks depth and coverage.

The following key areas for future development come with resource implications for the survey program. Hence, opportunities to explore the trade-offs that can be made to develop these areas should be investigated.

The first two areas for development relate to the survey design. The emerging literature and the direction of other comparable surveys suggest that, to fully understand the school-to-work transition in a changing policy context, data collection needs to be extended in both directions. That is, data on earlier circumstances are required to better understand the effects of early childhood learning on school-to-work transitions, and the post-school evaluation window needs to be lengthened beyond 25 years to reflect the recognition that an age of 25 years is too young to determine the success of transition. The third area for development relates to questionnaire content, while the fourth and fifth are concerned with improving data quality.

1. Capturing early childhood learning: Parent questionnaire

There is compelling evidence in this paper from the literature review, survey review and current policy direction that a matched parent or guardian survey administered at least once per cohort would greatly enhance LSAY, since it would capture a richer array of information on young people’s early childhood and background. This would provide detailed and high-quality information on parental education, occupation, household income and the sources of income. The quality of such data, which are reported by the young people themselves, is a major limitation for previous waves of LSAY. A parent questionnaire could also collect data on important family attributes (for example, family structure and parental unemployment), which clearly play an important part in youth transitions.

2. Extend the survey beyond 25 years

The increasing trend for young people to take longer to complete their school-to-work transition means that the LSAY cohorts are terminated too early. Many young people at age 25 have only recently left or may still be enrolled full-time in higher education, which is a considerable limitation on the overall quality of the LSAY program. Wage rates at age 25, for example, are a poor measure of the return from education.

Extending the survey period for respondents beyond 25 years would enable more visible ‘end-points’ to become clearer and allow for better estimates from returns on education and training. Of course such an extension would increase the cost, and sample attrition would be further increased by lengthening the survey.

A possible trade-off is to consider surveying respondents less frequently (biennially) after the critical years of transition, for example, after 21 years.

3. Review the questionnaire content

LSAY is very strong in its measures of employment outcomes. This assists our understanding of how young people navigate the education pathways through to the labour market. The survey also identifies some events outside the education and employment domains that are known to be related to outcomes, such as marriage. However, the review of key studies and the literature highlighted the limited coverage on health, wellbeing and other individual factors that affect education and employment outcomes, or are important outcomes in their own right. These items are becoming significant in current policies for all groups, particularly young people.

The LSAY questionnaire has been modified from year to year to meet societal and policy changes. This needs to continue to ensure that LSAY remains relevant to current policy initiatives and keeps pace with the current behaviours of young people. The review recommends consideration of the following content changes:

• a review of the life satisfaction questions and limited health identifiers to ensure a more comprehensive exploration of young people’s health and wellbeing
• the inclusion of:
  • measures of social capital to assist in understanding how socioeconomic background impacts on educational outcomes and transitions
  • measures of other personal characteristics such as deviant behaviour and personality traits associated with young people’s decision-making and their impact on later outcomes
• the implementation of minor improvements to keep pace with technology changes, such as the role of information communications technology (ICT) in assisting young people to seek careers advice and employment.

Trade-offs may need to be made to increase or introduce questionnaire items in a certain area, such as health, at the expense of reduced questioning in another area (for example, school subject information).

Any changes or modifications to the survey instruments also require a planned approach that involves cognitive testing and analysis of previous questionnaires. A history of questionnaire changes should be kept to document why changes have occurred from year to year.²

The creation of an ‘ideal topic map’ would assist in developing the questionnaire content, allowing for easier adaptation for future cohorts and maintenance of consistency in the data collection between cohorts. It would also assist in the placement of questions relating to ‘non-core’ areas. The lag time for implementation also needs to be considered, and changes should be determined well in advance of the questionnaires being pilot-tested.

4. Integrate LSAY with administrative datasets

The connection between the Programme of International Student Assessment (PISA) and LSAY is one of the strongest features of the survey design and provides a robust measure of academic achievement at the age of 15 years. Of the key studies discussed in this review, only the Canadian Youth in Transition Survey (YITS) integrates with PISA. However, the other key studies benefit from sampling frames that can provide additional or supplementary information, which allows data quality to be improved where responses from individuals are missing for particular sections. Several of the measures obtained in LSAY are potentially obtainable from administrative datasets, such as tertiary entrance rank (TER) scores and information about the receipt of government payments. The matching of school administrative records and test scores has been achieved with other key longitudinal surveys. Technical studies should be undertaken to investigate the reduction in measurement error that would accrue through the capture of some information from other datasets, and privacy issues would need to be explored.

5. Attrition

Attrition bias is an issue in most longitudinal surveys, and LSAY is no exception. However, in LSAY this is particularly problematic because the young people we are most interested in are those most likely to drop out of the survey. Analytical methods can account for some of this bias, but a focused incentive for respondents could improve the quality of data for this group of young people.

Incentives are a way to keep attrition in check and are used in some other longitudinal surveys, but they add significant costs to the survey program. Further investigation should be undertaken to examine the trade-offs that could be made to offset the introduction of respondent incentives in the LSAY program. Creating a respondent’s webpage on the LSAY website is a low-cost strategy to encourage respondent involvement and potentially increase retention.

1 LSAY is managed and funded by the Australian Government Department of Education, Employment and Workplace Relations (DEEWR), with     support from state and territory governments.

2 An example of a well-documented survey history can be found in the History of Employer Survey by NCVER, accessed at: <http://www.ncver.edu.au/publications/1794.html>.