An aid to systematic reviews of research in vocational education and training in Australia

• A systematic review of research is a decision-making tool for policy and practice. It is a piece of research in its own right, using explicit and rigorous methods that follow a standard set of stages. These methods identify, critically appraise and synthesise relevant research (both published and unpublished) around a specific research question.

• The review process allows for different studies to be weighted for relevance and quality of findings to answer a given question. The ultimate effect of this is that research can influence a review’s conclusion only when based on agreed guidelines, and when the reviewers have confidence in the research.

• In undertaking the first systematic review of research in vocational education and training (VET) in Australia on the mature-aged and skill development activities, the National Centre for Vocational Education Research (NCVER) was required to also establish a model and infrastructure for future reviews. NCVER’s proposed eight-step model is outlined in this report.

The National Centre for Vocational Education Research (NCVER) was contracted by the Australian National Training Authority (ANTA) to undertake a first systematic review of research related to the topic of mature-aged workers. The contract included the development of a replicable framework and infrastructure for further systematic reviews of research.

A systematic review of research is a decision-making tool for policy and practice. It is a piece of research in its own right, using explicit and rigorous methods that follow a standard set of stages. These methods identify, critically appraise and synthesise relevant research (both published and unpublished) relating to a specific research question.

In undertaking the first systematic review of research in vocational education and training (VET) in Australia, NCVER learnt many lessons. Our eight-step model and developed infrastructure for future reviews includes:

Step 1: Identify the review question

• A steering group will be established and include high-level representation from state, territory and Commonwealth authorities, from industry and from the research community, or specific experts for the particular topic.

• A consultation group will be established and include external reviewers and potential reviewers or those with expertise, or who have expressed interest in the topic (including international advisors).

• Policy-makers and other stakeholders will be involved in defining the review question by focusing on a very specific population, intervention and outcome.

• Sufficient time will be allowed to consult widely with key groups and individuals for refinement of the question.

• The key reviewer and second reviewer will be selected at the beginning of the review process. In this way they will fully understand the development of the question, be involved in screening the studies for in-depth review, and be familiar with all included studies before synthesising the evidence and compiling the final report.

Step 2: Develop a framework document

The framework document takes the review question and defines keywords, search strategy, review and appraisal criteria, and contents of final report.

• The advice of the consultation group will be used to guide the development of the framework that arises from and supports the question.

• The key reviewer will contribute to the management of the review process (for example, communicating with consultant reviewers).
  
• Once the framework is established, a database will be developed to contain the results of searches, critical appraisal and selection (using inclusion and exclusion criteria) of materials, and the relevant findings and evidence from the included studies to answer the review question.

Step 3: Search for all relevant research

• At least two searchers will undertake the extensive and thorough searching process for each review.

• The initial selection will be done by the searchers, using titles and abstracts, and should include all material that appears to meet the inclusion criteria established within the framework, and if in doubt, they should include the material.

• The steering and consultation group will be provided with lists of the excluded and included studies and consulted to ensure that no key studies are missed. However, the final decision on inclusion will remain with the project team.

• Screening is an iterative process and a further screening stage (see next step) using full documents will be undertaken by the key reviewer and second reviewer to arrive at the final selection of studies for in-depth review.

Step 4: Select studies to be included

• The inclusion criteria will be applied more strictly to the full documents by the key reviewer and second reviewer who will be involved in the final screening stage. This will include an initial appraisal of the relevance of findings to the review question and of the quality of the research.

• Only research studies which provide evidence to answer the review question and which meet the quality criteria will be included in the in-depth review. There should be no more than 20 studies included for in-depth review (there may be a set of ‘reserve’ documents or lower rated studies kept for contextual information).

• The reviewers who will be synthesising the evidence for the report will be familiar with all the included studies and aware of those excluded from the in-depth evaluation.

• Sufficient time will be allowed so that references within the selected studies can be followed up, in order that these may be considered for inclusion in the review process.

• All selected reviewers will attend a training workshop before commencing the in-depth appraisal and review of evidence from the selected studies. Detailed guidelines will be provided to the reviewers.

Step 5: Appraise the studies

• Reviewers should be allocated studies, to some extent, according to their expertise in both the topic and the research and analytical techniques, as required (for example, quantitative research, including statistical analysis and economic modelling or qualitative research).

• Each study included in the in-depth review will be allocated to two reviewers who, working independently, will enter into the electronic template their appraisal of the relevance and quality of the findings to the review question. The reviewers will then reach a consensus decision.

• The project team may moderate ratings given by reviewers where consensus is not possible or some inconsistency is noted.

• The reviewers will add to the database the details of study aims, methods, population, intervention, outcomes and findings and the best examples to illustrate the findings.

Step 6: Synthesise the evidence

• A database will be used in future to enable electronic sorting and amalgamation of evidence from the studies to assist in the synthesis of findings and in checking the evidence trail for the final report.
  
• The key reviewer and a second reviewer will synthesise the evidence found to answer the review question into categories, pooling material from the studies in whose findings we can have confidence.
  
• Using other members of the project team to provide feedback, the key reviewer will compile a draft final report of the evidence to answer the review question, and implications for policy, practice and research.

Step 7: Present findings to stakeholders

• The draft final report will be distributed to the steering group members and reviewers for comments before finalising for publication.

Step 8: Disseminate the findings

• The final report will be published on the NCVER website.

• Presentation of the findings of systematic reviews will be made to stakeholders through research forums, conferences and other channels as appropriate.