Description
This technical paper provides an evaluation of three alternative methodological approaches for calculating completion rate projections for apprentices and trainees. Projected rates are calculated for recently commencing cohorts because many contracts are still active, and the actual completion rate is not yet known. Completion rate projections were calculated for cohorts commencing in 2013, 2014, and 2015, and evaluated against actual completion rates for these cohorts.
All three methods generated projections within five percentage points of actual completion rates. However, the work revealed differences in the consistency of error rates, data requirements, and the timeliness with which projections could be generated for new cohorts.
Summary
About the research
The purpose of this technical paper is to summarise exploratory work investigating the effectiveness of the life tables methodology, which is currently used to calculate completion rate projections for apprentices and trainees, against two alternative methodological approaches − Markov chains and machine learning.
The National Centre for Vocational Education Research (NCVER) publishes projected contract completion rates for recent commencing cohorts of apprentices and trainees. Reporting projected rates is necessary because actual completion rates cannot be known until enough time has passed for contracts to be completed and for the outcomes to be reported to NCVER.
This report includes:
- an overview of the three methodologies − life tables, Markov chains and machine learning − that were applied to calculate projected completion rates for apprentice and trainee contracts of training
- a comparison of the accuracy of the projections generated by the three methodologies
- an evaluation of the relative strengths and limitations of the three methodologies.
Key messages
- Between 2013 and 2015 − the three years of data for which projections were examined − the completion rate projections calculated using Markov chains and machine learning were generally more accurate than the rates achieved using life tables. When evaluated against observed actual completion rates:
- For non-trade cohorts, the error rates were the lowest and most consistent for machine learning projections.
- For trade cohorts, the error rates were the lowest and most consistent for Markov chains projections.
- Each method has strengths and limitations:
- The life tables approach calculates completion rate projections for December quarter commencing cohorts, which are taken as proxy rates for annual commencing cohorts. This may account for the lower accuracy of life tables projections when evaluated against observed actual completion rates, which are calculated for annual cohorts.
- The Markov chains approach produced the most accurate projections overall. However, this methodology generates projections with a 12-month delay, in contrast to the life tables and machine learning approaches.
- The machine learning methodology produced highly accurate projections for non-trade cohorts, and the projections for trade cohorts were comparable with those generated by Markov chains. Unlike the other methods, strategies such as adding additional attributes or improving the data quality of existing attributes can be used to improve the performance of the model.
- All three methodological approaches draw on a five-year window of recent historical data to calculate projections. Due to the significant disruption to apprentices and trainees resulting from the COVID-19 pandemic, it is not clear whether the assumptions underlying the methodologies remain valid for years where the reference period includes disruption from COVID-19. For this reason, it is not considered appropriate to implement changes to the projection methodology used in NCVER reporting at this time.