Assessing the impact of NCVER's research

The National Centre for Vocational Education Research (NCVER) has undertaken this project to examine whether its research has an impact in the vocational education and training (VET) sector and, if so, the nature of the impact.

This was not a straightforward exercise. There are challenges associated with defining what is meant by impact, with connecting the outcomes of research to policy and practice, and with identifying measures which will provide robust but not unrealistic levels of evidence.

This project also developed a model to help in assessing the impact of NCVER’s research and in identifying practices that will further enhance the influence that research has on policy and practice. The model allows for identification of impact across four categories: knowledge production, capacity building, informing policy and informing practice.

Key messages

• The case studies provided a positive assessment of NCVER’s research impact, with examples of impact in each of the four categories.

• Bibliometric analysis is useful, but does not capture every aspect of research impact.

• Dissemination is key to research impact, suggesting that NCVER’s emphasis on this aspect of research is appropriate. Harnessing researchers in this endeavour is worth considering.

Tom Karmel
Managing Director, NCVER

The National Centre for Vocational Education Research (NCVER) is committed to ensuring that its research is useful and achieves a practical outcome related to policy implementation and practice in the vocational education and training (VET) sector. In the context in which NCVER operates, its foremost interest is in applied research. To this end it is important that we evaluate the impact of research and its alignment with policy and practice initiatives.

There are challenges associated with connecting the outcomes of research to the judgements and behaviours of professionals and practitioners. Evidence suggests that this process is not straightforward and much debate ensues about the extent to which it is indeed possible to accomplish effectively and robustly. Certainly, there is not usually a linear relationship between one research project and one consequent policy change.

The purpose of this report is to assess the impact of NCVER’s research. The project was undertaken through a combination of a review of the appropriate literature, a workshop, citation analysis and case studies of three individual projects and one suite of projects based on a particular theme. The project builds on previous work in this area, but it has been some time since NCVER considered it directly.

As part of this project, we considered current thinking on assessing impact to develop a model suitable for NCVER’s purpose. Models for assessing the impact of research help in the understanding of the processes involved and the parameters needed to ensure that realistic goals are maintained. The final model proposed allows for identification of impact across four categories:

• producing knowledge (knowledge production), which is related to dissemination, general awareness of and engagement with the research, contributions to the literature and ability of the research to inform future activities in policy, practice and research

• building capacity (capacity building), which supports the abilities of researchers to undertake fit-for-purpose research, improves the skills of relevant stakeholders to engage with the research and enhances their decision-making abilities. It also supports and encourages the training of early career researchers

• informing policy, encompassing research that is used to guide decisions or actions

• informing practice, broadly encompassing the behaviour, actions and knowledge of how things are done.

Importantly, the model acknowledges a number of key elements on which NCVER places high importance and which the organisation has successfully implemented over many years. These include:

• establishing the priorities and purpose of the research with some involvement from end-users

• adopting multiple dissemination strategies acknowledging the diverse needs of the sector

• engaging with stakeholders throughout the research cycle.

For each category the model identifies the types of impact and various sources of information as indicators of impact. We found significant examples of where research has made a difference across the various categories.

In terms of knowledge production there are a number of sources of information, including bibliometric and citation analysis in other published research and in government reports, parliamentary documents and journals, as well as citations in the media.

Across all case studies (eight research reports), a total of 142 citations, 30 media citations, based on NCVER’s records, and around 120 000 web hits on those reports were identified. While these figures were somewhat resource-intensive to collect, the results are important indicators of the extent to which the research has been accessed and used.

Establishing purely quantitative measures of impact is difficult and we did not intend to do this. Instead we embraced a broad scope for what could be included in the citation analysis and did not place any emphasis on scoring or ranking the results. This is because NCVER’s audience does not rely on scholarly journals as their principal source for VET research outcomes. In addition, a lower citation count may result, depending on the purpose of the research and the nature of dissemination activities. This does not necessarily equate to less impact.

There were a number of capacity-building impacts identified in the case studies. Some of these were not planned in the research purpose but did lead to further work and professional development opportunities for the researcher.

Notably, this project identified examples in all but one of the case studies where research has made some contribution to policy development, and it would appear to be at considerable levels. Nevertheless, we need to be realistic about the scale of that impact, given the nature of policy development, which draws on a diverse range of sources. There are real difficulties with proving direct cause and effect.

With respect to informing practice, we were limited by the nature of the research projects selected for the case studies, emphasising again the importance of recognising the purpose or intent of the research within any impact analysis. While some specific examples were identified, many of these were anecdotal.

In conclusion, our assessment of the impact of NCVER’s research is positive. In addition, a number of points emerged:

• in creating impact, the importance of knowledge translation; that is, the iterative process of synthesising and exchanging research outcomes

• the potential of stakeholders as catalysts for enhancing the impact of research

• in making an impact, the efficacy of a suite of research reports rather than a single report.