Issue 46,

October 2008

PDF

Organizations that fund research are under intense pressure to show that the research they support brings value to their communities. i In fact, in Canada, demonstrating the impact of research is a requirement of the federal government’s science and technology strategy. ii Due mainly to the applied nature of their research, health services and policy researchers in particular face high expectations to describe the benefits of their work. iii,iv

Measuring research impact can justify requests for increased funding of health services research and also drive improvements in research methodology, the research management process and, for funding organizations, the design of research funding programs. Ultimately, being clear about the objectives of a funding program or individual grant application is critical to informing the methodological approach for evaluating the outcomes of the program or grant. v

Demonstrating value from research is complicated by many factors: research impacts are often not easily quantifiable; there is a time lag between when research is funded and when its benefits are realized; and there are challenges in attributing an impact to a specific research finding rather than to an accumulated body of knowledge with multiple sources. i Also, there is often ambiguity between the terms outcome (a mid-term and intermediate effect) and impact (a long-term and ultimate effect).vi And there are two types of impacts:vii instrumental impacts show a clear connection between a particular study and a specific outcome, such as a change in clinical practice; while conceptual impacts are less tangible, advancing knowledge and informing decisions, but without showing a direct link.

Buxton and Hanney have identified five categories of research impact (Table 1), which collectively define “payback” from health research.iii The Canadian Institutes of Health Research has drawn heavily from the payback model for its own impact framework (Table 1). viii, ix

Table 1. Categories of payback or impact from health research

Other researchers have also offered helpful frameworks for measuring impact. John Lavis and colleagues provide a tool that describes various health research impact measures, related data sources, and steps to measure the impact of research. x The authors argue that “impact measures can be categorized according to whether the active role in promoting the research is played by the researchers (producer-push measures), decision-makers (user-pull measures) or both researchers and decision-makers (exchange measures).”x Shyama Kuruvilla and colleagues’ Research Impact Framework describes the impact of health services and policy research.xi Their framework includes four impacts: research-related; policy; service; and societal. This approach has shown promise in developing “research impact narratives” with researchers at the London School of Hygiene and Tropical Medicine.xi

While it’s important to define impacts, there are challenges in selecting appropriate indicators to measure these impacts and in developing methods to consistently and reliably capture the data content of each indicator. RAND Europe, a non-profit research organization, has developed a research measurement toolkit, which, although not comprehensive, includes a range of evaluation methods, along with their advantages and disadvantages, for measuring the benefits from research (Table 2).i

Table 2. Methods for measuring the benefits from research, as defined by RAND Europe i

Based on its work, RAND Europe suggests taking a multi-indicator, multi-method approach to measuring research impact.i By drawing on these methods and combining them with what is known about defining the benefits of research and the emerging evidence base, health services researchers and funding agencies should be well-positioned to describe the value their work brings to communities.

*Measuring the impact of research: What do we know? (Part II) will be published in early 2009.

References

The literature for this issue of Insight and Action was gathered and analyzed by CHSRF summer intern Amanda Hancock, who also contributed to the writing of early drafts.

i. RAND Europe. 2006. “Measuring the benefits from research.” Policy Resource.

ii. Government of Canada. 2007. Mobilizing science and technology to Canada’s advantage.

iii. Buxton M. & Hanney S. 1996. “How can payback from health services research be assessed?” Journal Health Services Research and Policy; 1(1): 35-43.

iv. Kuruvilla S., Mays K. & Walt G. 2007. “Describing the impact of health services and policy research.” Journal of Health Services & Research Policy; 12(1): 23-37.

v. Brutscher P-B., Wooding S., & Grant, J. In press. Health research evaluation frameworks: An international comparison.

vi. Hovland, I. 2007. “Making a difference: M&E of policy research.” Working Paper 281 Results of ODI research presented in preliminary form for discussion and critical comment.

vii. Davies, H., Nutley, S., & Walter, I. 2005. “Assessing the impact of social science research: conceptual, methodological and practical issues.” Research Unit for Research Utilisation (RURU), University of St. Andrews.

viii. Canadian Institutes of Health Research. 2005. “Developing a CIHR Framework to Measure the Impact of Health Research – Synthesis Report of Meetings.”

ix. Canadian Institutes of Health Research. 2008. “A framework to measure the impact of health research.” Personal communication, Laura McAuley, Manager Impact Assessment, Knowledge Translation and Commercialization.

x. Lavis J., Ross S., McLeod, C. & Gildiner, A. 2003. “Measuring the impact of health research.” Journal of Health Services Research and Policy; 8(3): 165-170.

xi. Kuruvilla S., Mays, N., Pleasant, A. & Walt, G. 2006. “Describing the impact of health research: A Research Impact Framework.” BMC Health Services Research; 6: 134.

For Further Reading

Canadian Health Services Research Foundation. A summary of Kuruvilla et al.’s 2007 “Describing the impact of health services and policy research.” Insight and Action; Issue 33.
   

Canadian Institutes of Health Research. 2007. It’s payback time: New international study to assess impact of heart and stroke research.
 

Canadian Academy of Health Sciences. 2007. Prospectus for a major assessment: The return on investments in health research.
  

Canadian Academy of Health Sciences. 2007. Forum summary: The return on investments in health research: Defining the best metrics.
  

Organisation for Economic Co-operation and Development (OECD). 2007. Science, Technology and Innovation Indicators in a Changing World: Responding to Policy Needs is a selection of the papers discussed at the OECD Blue Sky II Forum (Ottawa, Canada, 25-27 September 2006).