124,778 research outputs found
Yes, research can inform health policy; but can we bridge the 'Do-Knowing It's Been Done' gap?
No full textCopyright @ 2011 Hanney and Gonzalez.Provisional Abstract: This editorial introduces a new Supplement in Health Research Policy and Systems and highlighs the importance of assessing the impact of health research by examining whether we can move from 'Know-Do' to 'Do-Knowing It's Been Done'This article is made available through the Brunel Open Access publishing fund
Assessing the impact of England's National Health Service R&D Health Technology Assessment program using the "payback" approach
No full textCopyright © Cambridge University Press 2009Objectives: This study assesses the impact of the English National Health Service (NHS) Health Technology Assessment (HTA) program using the "payback" framework.Methods: A survey of lead investigators of all research projects funded by the HTA program 1993-2003 supplemented by more detailed case studies of sixteen projects.Results: Of 204 eligible projects, replies were received from 133 or 65 percent. The mean number of peer-reviewed publications per project was 2.9. Seventy-three percent of projects claimed to have had had an impact on policy and 42 percent on behavior. Technology Assessment Reports for the National Institute for Health and Clinical Excellence (NICE) had fewer than average publications but greater impact on policy. Half of all projects went on to secure further funding. The case studies confirmed the survey findings and indicated factors associated with impact.Conclusions: The HTA program performed relatively well in terms of "payback." Facilitating factors included the program's emphasis on topics that matter to the NHS, rigorous methods and the existence of "policy customers" such as NICE
Going Beyond Counting First Authors in Author Co-citation Analysis
No full textThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Engagement in research: An innovative three stage review of the benefits for health-care performance
No full text© Queen's Printer and Controller of HMSO 2013. This work was produced by Hanney et al. under the terms of a commissioning
contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and
study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement
is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be
addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre,
Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.Background - There is a widely held assumption that research engagement improves health-care performance at various levels, but little direct empirical evidence.
Objectives - To conduct a theoretically and empirically grounded synthesis to map and explore plausible mechanisms through which research engagement might improve health services performance. A review of the effects on patients of their health-care practitioner's or institution's participation in clinical trials was published after submission of the proposal for this review. It identified only 13 relevant papers and, overall, suggested that the evidence that research engagement improves health-care performance was less strong than some thought. We aimed to meet the need for a wider review.
Methods - An hourglass review was developed, consisting of three stages: (1) a planning and mapping stage; (2) a focused review concentrating on the core question of whether or not research engagement improves health care; and (3) a wider (but less systematic) review of papers identified during the two earlier stages. Studies were included in the focused review if the concept of ‘engagement in research’ was an input and some measure of ‘performance’ an output. The search strategy covered the period 1990 to March 2012. MEDLINE, EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science and other relevant databases were searched. A total of 10,239 papers were identified through the database searches, and 159 from other sources. A further relevance and quality check on 473 papers was undertaken, and identified 33 papers for inclusion in the review. A standard meta-analysis was not possible on the heterogeneous mix of papers in the focused review. Therefore an explanatory matrix was developed to help characterise the circumstances in which research engagement might improve health-care performance and the mechanisms that might be at work, identifying two main dimensions along which to categorise the studies: the degree of intentionality and the scope of the impact.
Results - Of the 33 papers in the focused review, 28 were positive (of which six were positive/mixed) in relation to the question of whether or not research engagement improves health-care performance. Five papers were negative (of which two were negative/mixed). Seven out of 28 positive papers reported some improvement in health outcomes. For the rest, the improved care took the form of improved processes of care. Nine positive papers were at a clinician level and 19 at an institutional level. The wider review demonstrated, for example, how collaborative and action research can encourage some progress along the pathway from research engagement towards improved health-care performance. There is also evidence that organisations in which the research function is fully integrated into the organisational structure out-perform other organisations that pay less formal heed to research and its outputs. The focused and wider reviews identified the diversity in the mechanisms through which research engagement might improve health care: there are many circumstances and mechanisms at work, more than one mechanism is often operative, and the evidence available for each one is limited.
Limitations - To address the complexities of this evidence synthesis of research we needed to spend significant time mapping the literature, and narrowed the research question to make it feasible. We excluded many potentially relevant papers (though we partially addressed this by conducting a wider additional synthesis). Studies assessing the impact made on clinician behaviour by small, locally conducted pieces of research could be difficult to interpret without full knowledge of the context.
Conclusions - Drawing on the focused and wider reviews, it is suggested that when clinicians and health-care organisations engage in research there is the likelihood of a positive impact on health-care performance. Organisations that have deliberately integrated the research function into organisational structures demonstrate how research engagement can, among other factors, contribute to improved health-care performance. Further explorations are required of research networks and schemes to promote the engagement of clinicians and managers in research. Detailed observational research focusing on research engagement within organisations would build up an understanding of mechanisms.National Institute for Health Researc
Dispelling the Myths Behind First-author Citation Counts
No full textWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: The case of Asthma UK
No full text© 2013 Hanney et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.This article has been made available through the Brunel Open Access Publishing Fund.BACKGROUND: Debate is intensifying about how to assess the full range of impacts from medical research. Complexity increases when assessing the diverse funding streams of funders such as Asthma UK, a charitable patient organisation supporting medical research to benefit people with asthma. This paper aims to describe the various impacts identified from a range of Asthma UK research, and explore how Asthma UK utilised the characteristics of successful funding approaches to inform future research strategies. METHODS: We adapted the Payback Framework, using it both in a survey and to help structure interviews, documentary analysis, and case studies. We sent surveys to 153 lead researchers of projects, plus 10 past research fellows, and also conducted 14 detailed case studies. These covered nine projects and two fellowships, in addition to the innovative case studies on the professorial chairs (funded since 1988) and the MRC-Asthma UK Centre in Allergic Mechanisms of Asthma (the ‘Centre’) which together facilitated a comprehensive analysis of the whole funding portfolio. We organised each case study to capture whatever academic and wider societal impacts (or payback) might have arisen given the diverse timescales, size of funding involved, and extent to which Asthma UK funding contributed to the impacts. RESULTS: Projects recorded an average of four peer-reviewed journal articles. Together the chairs reported over 500 papers. All streams of funding attracted follow-on funding. Each of the various categories of societal impacts arose from only a minority of individual projects and fellowships. Some of the research portfolio is influencing asthma-related clinical guidelines, and some contributing to product development. The latter includes potentially major breakthroughs in asthma therapies (in immunotherapy, and new inhaled drugs) trialled by university spin-out companies. Such research-informed guidelines and medicines can, in turn, contribute to health improvements. The role of the chairs and the pioneering collaborative Centre is shown as being particularly important. CONCLUSIONS: We systematically demonstrate that all types of Asthma UK’s research funding assessed are making impacts at different levels, but the main societal impacts from projects and fellowships come from a minority of those funded. Asthma UK used the study’s findings, especially in relation to the Centre, to inform research funding strategies to promote the achievement of impact.This study was funded by Asthma UK
Who needs what from a national health research system: Lessons from reforms to the English Department of Health's R&D system
No full textThis article has been made available through the Brunel Open Access Publishing Fund.Health research systems consist of diverse groups who have some role in health research, but the boundaries around such a system are not clear-cut. To explore what various stakeholders need we reviewed the literature including that on the history of English health R&D reforms, and we also applied some relevant conceptual frameworks.
We first describe the needs and capabilities of the main groups of stakeholders in health research systems, and explain key features of policymaking systems within which these stakeholders operate in the UK. The five groups are policymakers (and health care managers), health professionals, patients and the general public, industry, and researchers. As individuals and as organisations they have a range of needs from the health research system, but should also develop specific capabilities in order to contribute effectively to the system and benefit from it.
Second, we discuss key phases of reform in the development of the English health research system over four decades -
especially that of the English Department of Health's R&D system - and identify how far legitimate demands of key stakeholder interests were addressed.
Third, in drawing lessons we highlight points emerging from contemporary reports, but also attempt to identify issues through application of relevant conceptual frameworks. The main lessons are: the importance of comprehensively addressing the diverse needs of various interacting institutions and stakeholders; the desirability of developing facilitating mechanisms at interfaces between the health research system and its various stakeholders; and the importance of additional money in being able to expand the scope of the health research system whilst maintaining support for basic science.
We conclude that the latest health R&D strategy in England builds on recent progress and tackles acknowledged weaknesses. The strategy goes a considerable way to identifying and more effectively meeting the needs of key groups such as medical academics, patients and industry, and has been remarkably successful in increasing the funding for health research. There are still areas that might benefit from further recognition and resourcing, but the lessons identified, and progress made by the reforms are relevant for the design and coordination of national health research systems beyond England.This article is available through the Brunel Open Access Publishing Fund
Lessons from the evaluation of the UK's NHS R&D Implementation Methods Programme
No full textBackground: Concern about the effective use of research was a major factor behind the creation
of the NHS R&D Programme in 1991. In 1994, an advisory group was established to identify
research priorities in research implementation. The Implementation Methods Programme (IMP)
flowed from this, and its commissioning group funded 36 projects. In 2000 responsibility for the
programme passed to the National Co-ordinating Centre for NHS Service Delivery and
Organisation R&D, which asked the Health Economics Research Group (HERG), Brunel University,
to conduct an evaluation in 2002. By then most projects had been completed. This evaluation was
intended to cover: the quality of outputs, lessons to be learnt about the communication strategy
and the commissioning process, and the benefits from the projects.
Methods: We adopted a wide range of quantitative and qualitative methods. They included:
documentary analysis, interviews with key actors, questionnaires to the funded lead researchers,
questionnaires to potential users, and desk analysis.
Results: Quantitative assessment of outputs and dissemination revealed that the IMP funded useful
research projects, some of which had considerable impact against the various categories in the
HERG payback model, such as publications, further research, research training, impact on health
policy, and clinical practice.
Qualitative findings from interviews with advisory and commissioning group members indicated
that when the IMP was established, implementation research was a relatively unexplored field. This
was reflected in the understanding brought to their roles by members of the advisory and
commissioning groups, in the way priorities for research were chosen and developed, and in how
the research projects were commissioned. The ideological and methodological debates associated
with these decisions have continued among those working in this field. The need for an effective
communication strategy for the programme as a whole was particularly important. However, such
a strategy was never developed, making it difficult to establish the general influence of the IMP as a
programme.
Conclusion: Our findings about the impact of the work funded, and the difficulties faced by those
developing the IMP, have implications for the development of strategic programmes of research in
general, as well as for the development of more effective research in this field
Project Retrosight. Understanding the returns from cardiovascular and stroke research: Case Studies
No full textCopyright @ 2011 RAND Europe. All rights reserved. The full text article is available via the link below.This project explores the impacts arising from cardiovascular and stroke research funded 15-20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This report presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.This study was initiated with internal funding from RAND Europe and HERG, with continuing funding from the UK National Institute for Health Research, the Canadian Institutes of Health Research, the Heart and Stroke Foundation of Canada and the National Heart Foundation of Australia. The UK Stroke Association and the British Heart Foundation provided support in kind through access to their archives
Assessing the payback from health R & D: From ad hoc studies to regular monitoring
No full textChapter 1 : Introduction
• The increasing demands for the benefits of payback from publicly funded R&D to be assessed are based partly on the need to justify or account for expenditure on R&D, and partly on the desire for information to assist resource allocation and the better management of R&D funds. The former consideration is particularly strong in relation to the R&D expenditure that comes out of the wider NHS budget.
• In this report a range of categories of payback will be identified along with a variety of methods for assessing them.
• The aim of the report is to make recommendations as to how the outcomes from health research might best be monitored on a regular basis. The specific context of the report is the NHS R&D Programme but many of the issues will be relevant for a wide range of funders of health R&D.
• The introduction sets out not only a plan of the report but also suggests that readers familiar with the general arguments and existing literature may choose to jump to Chapter 6.
Chapter 2 : Review of Existing Approaches to Assessing the Payback from Research
• Existing work describes various approaches to valuing research. Some are ex ante and attempt to predict the outcomes of research being considered, others are ex post or retrospective.
• The five categories of benefit or payback from health R&D that have been identified involve contributions: to knowledge; to research capacity and future research; to improved information for decision making; to the efficiency, efficacy and equity of health care services; and to the nation’s economic performance. These are shown in Table 1 of the report
• The process by which R&D generates final outcomes can be modelled as a sequence. This includes primary outputs such as publications; secondary outputs in the form of policy or administrative decisions; and final outcomes which comprise the health and economic benefits. Feedback loops are also introduced and mitigate the limitations of a linear approach.
• Qualitative and quantitative approaches can be used but there are immense problems with time lags and attributing outcomes, and sometimes even outputs, to specific items of research funding.
• Four common methods of measuring payback can be used. Expert review, by peers or, sometimes, users is the traditional way of assessing the quality of research. Bibliometric techniques can involve not only counting publications but also using datasets such as the Science Citation Index and Wellcome’s Research Outputs Database (ROD). The various methods of economic analysis of payback are difficult to undertake given the costs and problems of acquiring relevant information and estimating benefits. Social science methods include case studies, which can provide useful information but are resource intensive, and questionnaires to researchers and potential research users.
Chapter 3 : Characteristics of a Routine Monitoring System
• In moving from ad hoc or research studies of payback towards a more regular monitoring it is noted that whereas there has always been a tradition of evaluation of research, in the public services in general there is now a greater emphasis on audit and performance measurement and indicators. A review of these various systems suggests we should be looking to develop a system of outcomes monitoring that incorporates performance indicators (PIs) and measurement rather than an audit system that is trying to monitor activities against predetermined targets.
• Standard characteristics of performance measurement systems do not necessarily apply to research where, for example, there are non-standard outputs. Difficulties have arisen in the USA in attempting to apply the Government Performance and Results Act to research funding agencies. It is shown that because the findings of basic research, in particular, enter a knowledge pool in which people and ideas interact, it is difficult to use a PIs’ approach to track eventual outcomes. However, for some types of health research it has proved more feasible to trace the flow between research outputs and outcomes.
• An outcomes monitoring system could be useful if it met the following criteria: relevant to, with as comprehensive coverage as possible of, the funders objectives; relevant to the funder’s decision making processes; encourages accurate compliance; minimises unintended consequences; and has acceptable costs.
Chapter 4 : Differences Between Research Types
• The range of differences between types of research can be relevant for the design of a routine monitoring system. The OECD distinguishes between basic research, applied research and experimental development. Most DH/NHS research is applied. There might be more of a tradition of publication of findings in applied research in health than in other fields. Nevertheless, the publication and incentives patterns operating in basic research mean that it would be inappropriate to use bibliometric indicators in a simple way across all fields even in health research.
• Despite having some differences from health research in publication patterns and in the detailed categories of payback, the broad approach proposed in Chapter 6 could be applied to social care research.
• Research that is commissioned, especially by the government, has some of the minimum conditions built into it that are associated with outcomes being generated, in particular because the funder has identified that a contribution in this area will be valuable.
Chapter 5 : What Units of Research?
• The term programme has various meanings including being used to describe a collection of projects on a common theme and to describe a block of funding for a research unit.
• Three main streams or modes of funding can be identified: projects, which are administratively grouped into programmes including a responsive programme; institutions/centres/units; individual researchers. These 3 streams are displayed in Figure 1. It is probable that the regular data-gathering for a monitoring system would operate at the basic level of each stream or mode.
• Previous work demonstrates that the full range of benefits can sometimes be applied at the level of projects, either in the responsive mode or in programmes, through the use of questionnaires to researchers. Expert and user review and user surveys have also been applied.
• Institutions and centres increasingly have experience not only of traditional periodic expert review but also of producing annual reports, although there are debates about what dimensions to include in such reviews and reports.
• Individuals in receipt of research development awards have completed questionnaires during and after the awards. These concentrate on the development of research capacity but can go wider.
Chapter 6 : A Possible Comprehensive Outcomes Monitoring System
• The proposed system is intended for DH/NHS to monitor the outcomes from its R&D in order to justify the R&D expenditure and assist with managing the portfolio. More detailed information is required for the latter purpose.
• We propose a multidimensional approach be adopted to cover all the dimensions of payback and that information be gathered from three sets of sources and Table 3 shows which methods would cover which output/outcome categories.
• Firstly, possibly annually, a questionnaire (possibly electronic) covering most payback categories should gather data from the basic level of each funding stream ie. from lead researchers of projects, from research institutions/centres, and from individual award holders.
• Secondly, supplementary information should be gathered from external databases (including the citation indices and Wellcome’s ROD).
• Thirdly, a range of approaches ie. user surveys, reviews by experts and peers, case studies including economic evaluations, and analysis of sources used in policy documents such as NICE guidelines, would be undertaken on a sample basis. They would provide not only supplementary information but, as with the external databases, would also verify the data collected directly from researchers.
• These proposals can be evaluated against the criteria set out in Chapter 3:
• The system is relevant to DH’s objectives of generating payback in a range of categories.
• Various problems have to be overcome before the system could be fully decision relevant. Firstly it might be necessary to ask researchers to apportion the contribution made to specific outputs from various funding streams. Second, to be decision relevant the information would have to be analysed and presented in a manner consistent with funders’ decision making processes. This would involve a) showing how for each outcome and output, for example publications, data from one project or stream could be compared with those from another and b) demonstrating how different outputs and outcomes could be aggregated.
• The questions of accuracy of data, minimisation of unintended consequences and the acceptability of the net costs are also addressed.
Chapter 7 : Research and Monitoring
• Whilst this report is primarily concerned with moving from ad hoc studies towards a routine monitoring system there are issues that need further research.
• Before embarking on full implementation the feasibility needs to be tested of items such as on-line recording of data and asking researchers to attribute proportions of research outputs to separate funding agencies.
• Once the system is implemented the value of some items can be better assessed, for example the additional value provided by self reporting of publications beyond that gained from relying on external databases.
• The data provided by the system would provide opportunities for further payback research on, for example, links between publications and other categories of payback.
• Some items such as network analysis could potentially be added to the monitoring system after further examination of them.
• Finally the benefit from the monitoring system itself should be assessed.Department of Health; Wellcome Trus
- …
