Enhancements to the Longitudinal Surveys of Australian Youth

The Commonwealth Department of Education commissioned the National Centre for Vocational Education Research (NCVER) to assess the value and implications of eight enhancements to the Longitudinal Surveys of Australian Youth (LSAY). 

The objective of LSAY is to track young Australians as they move from school into further study, work and other destinations to provide a meaningful dataset through which to understand youth transitions. Enhancements to LSAY are considered in this paper in the context of continuing to enable researchers to track young people over time and examine relationships between the variables that impact youth transitions.

Key messages

• All enhancements are interrelated, with a change to one aspect of LSAY affecting other aspects of LSAY. 
• There are options to alter the sample design (including the frequency of starting new cohorts and changing the sample size) within the current survey design, whereby the sample is selected from school students who participate in the Programme for International Student Assessment (PISA). 
• Addressing attrition from the first wave (PISA) to the second wave will improve the value of the LSAY dataset and, in particular, improve the ability to analyse sub-populations. 
• In order to make a case for gathering any new information, a relationship between the additional measures and the success of youth transitions needs to be demonstrated. There is support for improvements to information in areas such as outcomes beyond age 25, wellbeing and parental background. 
  • Transitions are taking longer, providing support for extending the survey beyond age 25. 
  • Higher levels of wellbeing are associated with more successful transitions to adulthood, providing support for improving the breadth of wellbeing information. 
  • Variation between students on educational outcomes is related to family background but the quality of parental background information in LSAY is currently questionable because it is missing or inaccurate.
• Collecting or improving information on outcomes beyond age 25, health and wellbeing and parental background will incur costs and the accuracy of the information will be dependent on response rates. Various strategies can be undertaken to minimise costs and encourage participation. 
• Linking LSAY with administrative collections allows for the inclusion of information from other time dimensions and the improvement of information in areas identified as weak, including health and wellbeing and parental background. 

The Commonwealth Department of Education  commissioned NCVER to examine and report on areas in which LSAY could be enhanced. The findings from this report will be considered by the department in the 2013 review of LSAY. The remit of the department’s review of LSAY is to consider how the survey has been used, whether it has provided value for money and how it could be improved and made more useful for the evolving policy environment. (See the terms of reference in appendix A). 

Background information

The LSAY program commenced in 1995 and was based on two other annual surveys: the Australian Youth Survey (AYS; 1989—97) and the Youth in Transition survey (YIT; 1978—1996). Survey participants (known as a 'cohort') enter the study when they turn 15 years, or as was the case in earlier studies, when they were in Year 9. Individuals are contacted once a year for ten years. Since 2003, the initial survey wave has been integrated with the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA). Over 10 000 students start out in each cohort.

A longitudinal survey obtains information from the same respondents at multiple points in time (Bureau of Labour Statistics 2013). This enables researchers to:

• account for the unobserved differences amongst cohorts and to investigate and make inferences about the relationships between the variables of interest;
• isolate the influence of policies and practice from confounding influences such as social background and context;
• track patterns of development and change over time;
• identify sequences and pathways; and
• identify critical periods in human development for exposures and risks and inform the timing of preventive measures (Logie, Hogan & Puet 2004; Bureau of Labour Statistics 2013; Howieson, Croxford & Howat 2008).

The purpose of LSAY is to track young people as they move from school into further study, work and other destinations to provide a meaningful dataset through which to understand youth transitions. It is in this context that the advantages and disadvantages of possible enhancements to LSAY are considered. 

In 2009, Nguyen et al. (2010) assessed the fitness for purpose of LSAY in generating a data source that could be used to understand youth transitions. This ‘stocktake’ review resulted in five broad recommendations to improve the usefulness of LSAY, ranging from reviewing the survey content to improving survey attrition. This discussion paper builds on the stocktake report by Nguyen et al. by providing an assessment of the value and implications of the recommended enhancements within the remit: that the main purpose of LSAY is to provide an information source to enable young people to be tracked from school into further education and work. 

This paper assesses the followings enhancements:

• reconsidering the frequency for starting new LSAY cohorts
• evaluating and changing the sample size
• adopting measures to reduce attrition
• extending the age to which LSAY cohorts are followed to beyond 25 years 
• introducing a parent questionnaire to collect more comprehensive background information on respondents
• reviewing the survey questionnaires to improve data collection on health and wellbeing, resilience and adaptability
• linking to other educational and administrative datasets such as the NAPLAN
• introducing supplementary topical surveys, interviews, focus groups or other means of enhancing the usefulness of LSAY to policy-makers.

The methodology involved assessing the advantages and disadvantages as well as the feasibility of each enhancement through data analysis, technical knowledge and a literature review. For the purpose of this paper, the changes that come with substantial costs have been noted, although we have not allowed costs to constrain our thinking. 

Findings

As noted, the purpose of LSAY is to track young Australians as they move from school into further study, work and other destinations. LSAY is not a whole of youth study, and, as such, the content of the data and questionnaires should be restricted to the elements that measure and impact on education and employment. 

In their earlier review, Nguyen et al. (2010) noted that this ability to track the population of interest and examine the relationships between variables that influence transitions makes LSAY a valuable dataset for researchers and policy-makers. Our assessment confirms that the survey design produces robust estimates of the population at the national, state and territory and school sector levels. There are, as always, some limitations. The options for addressing these limitations are considered in this paper but in the context of continuing to enable researchers to track young people over time and examine relationships between the variables that impact youth transitions.

The body of the report provides an assessment of each enhancement. It should be noted that all enhancements are interrelated, with a change to one aspect of LSAY affecting other aspects of LSAY. For example, extending the age to which cohorts are followed must be weighed against the impacts of attrition over a longer timeframe. It should also be noted that a major change to the LSAY survey design may involve a prioritisation of other improvements, have cost implications and require strategies to reduce respondent burden. 

Sample design and maintenance

LSAY survey participants are currently selected from school students who participate in PISA. The options for changing the sample design (including the frequency of starting new cohorts and changing the sample size) and addressing attrition are more limited within the PISA arrangement. 

On the other hand, the PISA sample design does offer a number of options for altering the sample size and reducing the frequency of introduction of new cohorts, with different PISA options available to the participating countries. For example, there is scope to extend the time between cohorts, to every six years, which would free up funds to enhance the survey in other areas. While this would not detract from the key purpose of LSAY, a disadvantage would be a reduced capacity to examine the effects of economic downturns and other potentially influential national events that occur between cohorts. There is also scope to alter the sample size. Australia chooses to sample above the PISA minimum requirement (5000 students), resulting in a LSAY sample size of approximately 14 000 students. Although the current sample design works well in providing reliable estimates, there is scope to reduce its size for the larger states and increase the sample size for the smaller states, which would improve analyses at the jurisdictional level. If the LSAY—PISA link remains, a review of the sampling options within the PISA framework could be considered. 

If LSAY is separated from PISA, there will be more control over sampling and the adoption of methods to reduce attrition. A key benefit would be the opportunity to change the sample design to make the existing sample more efficient. This could include improving the ability to further stratify schools and boosting the number of LSAY participants from equity groups. It is noted from the research that oversampling sub-populations could reduce the representation of the overall population of Australian youth. This potential consequence needs to be weighed against the main purpose of LSAY. Best practice also suggests it is preferable to examine specific sub-populations, such as the Indigenous population, via specialised surveys.

The key obstacles to breaking the LSAY—PISA link include costs, recruitment challenges, and information gaps on schools, students’ background and academic performance. The information gaps would need to be addressed as a matter of priority through additional survey questions and testing. Of particular importance is the requirement for a reliable measure of academic performance, as research consistently demonstrates that literacy and numeracy are strong predictors of education and labour market outcomes. The most cost-efficient option for collecting information on academic performance is to obtain data from the National Assessment Program — Literacy and Numeracy (NAPLAN) through data linkage or to link the sampling of LSAY participants to NAPLAN testing. One of the first steps in assessing whether to maintain the PISA link would be to explore the accuracy of data linkage between NAPLAN data and LSAY and the reliability of NAPLAN scores as a predictor of youth transitions. 

Addressing attrition is fundamental to improving the value of the LSAY dataset and is linked to several of the enhancements considered in this paper. LSAY currently suffers from a loss of sample members, which is a common problem in longitudinal surveys. The most substantial sample loss occurs from the first wave (PISA) to the second wave, and represents a sample loss of between 20 and 40%. Field reports indicate that as much as 25% of a cohort can be lost because incorrect contact details are provided by students when sitting PISA. Addressing attrition from the first to the second wave is the area of most priority. This could be addressed by offering financial compensation to schools to provide accurate contact details, interviewing participants in the first wave or exploring options to provide incentives at this time. 

The use of incentives is a method adopted by other comparable longitudinal surveys to encourage participation. The cost of providing incentives will be high if incentives are used for all LSAY participants (approximately 14 000 per cohort on commencement of the survey). The costs could be offset by reducing the sample size to a level where reliable and accurate estimates would still be produced — at least at the Australian and state and territory levels. Alternatively, incentives could be targeted to sub-populations, such as those suffering from the highest rate of attrition, noting that this raises equity issues.

Other strategies aimed at reducing attrition in all waves include establishing a strategic communication plan targeted to young people and their parents and rebuilding the sample or re-sampling sub-populations with large attrition. Furthermore, the implementation of some of these strategies has wider benefits. For example, the introduction of an information pack for participants and their parents has the capacity to promote the value of the survey and provides a means to administer parental surveys and letters of consent for data linkage and participation in ad-hoc surveys and studies. 

Other enhancements

The remaining enhancements considered in this paper are related to improvements to the information in areas such as outcomes beyond age 25, health and wellbeing and parental background. The evidence suggests that improved information in most of these areas is warranted and would result in a dataset that would further enhance research on youth transitions. Research confirms that transitions are taking longer, supporting the value of extending the survey beyond age 25. There is evidence that higher levels of wellbeing are associated with more successful transitions. There is less evidence of a relationship between health measures, such as dietary intake and physical activity, and successful youth transitions. Finally, research shows that approximately 80% of the variation in educational outcomes is linked to individual and family background factors. Data on parental background are collected in the first wave of LSAY. The quality of this information is questionable because it is reported by the young people themselves, whom may not be fully aware of their primary care giver’s qualifications or occupation.

There are several options for collecting or enhancing information on outcomes beyond age 25, health and wellbeing and parental background. As noted, extending the survey beyond age 25 is likely to impact on attrition and will increase survey costs. A possible approach to minimising costs would be to survey people on a biennial basis after age 20. This would result in an increase of only one extra survey wave, but extra costs would be incurred through increased sample maintenance. Information gaps on health and wellbeing could be addressed by the inclusion of a health and wellbeing module in the questionnaire, either through a personal interview or via the current computer-assisted telephone interviewing (CATI) mode. Other less costly approaches include asking a sub-sample of participants to participate in a separate study or investigating data linkage. This approach is particularly recommended for options to improve information on health because there is less evidence that diet and activity is related to the success of youth transitions. Response rates, testing burden and costs are the main barriers to the adoption of a parental questionnaire. A marketing plan and communication strategy may boost response rates, and costs could be reduced by exploring data-linkage options. 

Data linkage could enable the information areas identified as weak, including health and wellbeing and parental background, to be improved by linking to existing administrative collections, such as Medicare and Centrelink. The key benefits to enhancing information through this approach are reduced costs and no impact on respondent burden. Privacy concerns and legal issues can be minimised by ensuring that consent is obtained and the use of an official data-integration authority. The conclusion is that it is technically possible to link LSAY with other datasets, which suggests many future possibilities for enhancing LSAY. 

Summary

LSAY provides robust estimates of Australian youth and from this perspective is considered a valuable dataset for tracking young people over time. There are some limitations to the data (and longitudinal surveys in general). LSAY suffers from attrition, particularly from the first wave (PISA) to the second wave. It is limited in its ability to provide in-depth analyses of health and wellbeing, to capture accurate information on parental background and, due to high attrition, to provide reliable estimates of the Indigenous population below the national level. 

The specific enhancements considered in this paper (for example, changes to the sample design, reducing attrition, linking to other datasets, and improving questionnaire content) are related, and all have merit, but the options and priority areas may vary depending on whether LSAY continues to be linked to PISA. Other key considerations to enhance the value and usage of LSAY include addressing attrition and linking to other datasets. The cost of the options will be an important issue, with some options bearing significant costs, without guarantee that the benefit to the dataset will be of the same magnitude. Priority areas can be identified by considering the findings from this paper and the wider review of LSAY being conducted in parallel to the preparation of this paper.

The remainder of the document considers each enhancement, in detail, in a separate chapter. Based on the literature, analyses of LSAY data and technical opinion, the following elements are considered for each chapter:

• advantages and disadvantages of the change;
• technical and/or practical feasibility;
• influence on analysis and reporting; and
• approximate costs.

The paper ends with concluding comments.

Dr Craig Fowler
Managing Director, NCVER