Description
This exploratory piece of work aims to illustrate the potential of matching methods in vocational education and training (VET) research. It applies propensity-score matching to the Student Outcomes Survey to estimate the causal effects of completing a VET qualification. Outcomes are examined for both graduates and module completers. The author provides a detailed account of the methodology used. Overall, it was found that, for graduates, higher-level qualifications yielded better outcomes. However, more robust data are needed to fully analyse the effects for module completers. This research was funded through the NCVER Fellowship Program.
Summary
About the research
This paper's primary focus is methodological. Its purpose is to show how matching methods can be used to estimate the effects of a treatment, such as completion of a VET qualification.
The experimental sciences have a huge advantage over the social sciences. Experiments can be carefully designed to isolate the effect of a treatment. Such an approach is rare in the social sciences, where typically a group of people is observed, some having been subjected to a 'treatment' and others not. Given such data, a simple comparison of the outcomes between the treated group and the untreated group can be quite misleading because the characteristics of the two groups can be quite different. To overcome this, multivariate statistical models can be built. These models can become very complicated and incorporate assumptions which may or may not be reasonable. The complexity of the models is also limited by the number of observations.
Matching methods offers an alternative to multivariate modelling. This approach is intuitively attractive and involves comparing the outcomes of the treated group with a comparison group. The statistical rigour is obtained from the construction of the comparison group, such that for each member of the treatment group there is an individual in the comparison group with very similar characteristics. The method by which the comparison group is constructed involves estimating a probability that an individual is in the treatment group on the basis of background characteristics. An individual in the treatment group is matched with an individual not treated on the basis of having the same probability of being in the treatment group.
In some cases, there is a background variable that is so important that a match on this characteristic is also required. For example, if the outcome variable is being employed, then it is critical that the treated individuals have the same employment status as their untreated peers, for the simple reason that the best predictor of being employed after training is being employed before training.
Using Student Outcomes Survey data to look at the effect of qualifications on outcomes after training, the matching techniques are illustrated. The main point to emerge was that, for vocational educational education and training (VET) graduates, higher-level qualifications, on average, increase earnings, improve employment outcomes and are considered more relevant to jobs than lower-level qualifications.
With regard to module completers, however, it is shown that the method cannot be used to estimate: the relative effects of module completions at different levels; or, the effects of completing a full qualification, as opposed to only completing some modules. This is because the background characteristics do not provide a good prediction of who will complete.
This research was funded through the NCVER fellowship program, which encourages researchers to use NCVER datasets.
Tom Karmel
Managing Director, NCVER