Innovations in services and the appliance of science

Managers and doctors should both seek evidence of effectiveness

Everyone now accepts that new drugs should be tested extensively before their introduction. A similar consensus is developing over health technology.1 2 Broadly defined, health technology includes methods of organising care,3 but currently innovations in how health services are organised and delivered often seem to be unregulated and unevaluated. Several factors drive innovation in services. These include perceptions of improved cost effectiveness on the part of policymakers or local purchasers and providers, perceptions of demand from the public and patients for changes in the delivery of care, and enthusiastic promotion of new technologies by commercial interests. Perceptions of usefulness are, however, all too often based on subjective impressions rather than established facts.

Many current innovations are concerned with the interface between primary and secondary care because of the (largely untested) belief that shifting care to the community will be more cost effective and more acceptable to the public than current patterns of provision of hospital services. Such innovations include hospitals at home, specialist outreach schemes, primary care emergency centres, and the development of discharge planning. Other innovations include changes in skill mix and professional roles, such as the introduction of practice counsellors, nurse practitioners, and health advocates for ethnic minority groups. Research on several of these topics is now being commissioned4 as part of the NHS research and development programme. Developments in services, however, have frequently gone ahead independently of plans for research.

In the case of specific medical treatments, the gold standard for evidence of effectiveness is a systematic review, and these reviews are being coordinated through the Cochrane Collaboration and the NHS Reviews and Dissemination Centre in York.5 Systematic reviews of randomised trials are much superior to standard medical review articles, which have been heavily criticised for using information selectively.6 Randomised trials may not always be feasible in evaluating innovations in how services are organised. In this case, systematic reviews need to take account of evidence from other types of studies, whose limitations must be borne in mind. Although systematic reviews and meta-analyses of specific medical interventions are becoming more widely available, such reviews of innovations in the organisation of care are much rarer. A Cochrane collaborative review group on structural and organisational changes in the delivery of health care is being planned, but currently purchasers and providers frequently make decisions without reliable evidence. The challenge will be to ensure that the findings of such reviews are used.

A range clearly exists in the strength of evidence concerning the cost effectiveness of new innovations, from little evidence (specialist outreach7), through modest evidence from non-randomised studies (hospital at home schemes8), to good evidence from a randomised trial (the provision of care attendants for elderly patients discharged from hospital9). Even at the stronger end of the range, however, the strength of the evidence rarely approaches that available for specific treatments for which large multicentre randomised trials or exhaustive meta-analyses have been undertaken. In addition, questions arise about the generalisability of the results. Reports should include details of staffing, skill mix, resource use, and other factors necessary for replication. The outcome of innovations in services may vary considerably according to the location (whether in rural areas, small towns, or inner cities) and may be influenced by subtle changes in how services are organised. Complex learning processes may take place while innovations are being set up and the enthusiasm of innovators is unlikely to be emulated in routine care. To compound the problem, those who are engaged in innovation are not necessarily well equipped to evaluate their own work. The skills of research into health services are poorly distributed in the NHS.

In practice, evaluation may be no more than a simple feasibility study, but even if an innovation can be set up within local resources there is a danger that the broader issues of cost effectiveness and potential generalisability may remain unanswered with such an uncoordinated approach. The risk is also that many innovations will fail; in a market environment in which the features that constitute successful innovation are well understood, 70% of new products fail to survive.10

We propose, therefore, that significant innovations in services should be registered—for example, by inclusion on the register of NHS research projects.11 The definition of an innovation would need clarification with local purchasers and providers. The register would give information about the proportion of schemes that were planned but never took place and the number of patients who entered a scheme—thereby giving an early warning of those that are not likely to be generalisable because of low uptake. It would also allow people planning innovations to learn from the experience of those who have some experience of developing schemes. Furthermore, it would signal potential topics for systematic reviews by the NHS Reviews and Dissemination Centre.

Additionally, the register would be a resource for health service researchers wishing to study specific innovations in cooperation with purchasers and providers and for potential funders of such research. As far as possible, multicentre trials of innovations should be promoted to determine whether innovations are generalisable and, if they are not, in which circumstances they are most appropriate. Entry to the register could be by notification to the regional research and development directorate, which could also try to ensure that funds were made available to evaluate innovations consuming substantial NHS resources. This should act as an incentive for notification.

Innovations have the potential for both good and harm. They entail the expenditure of scarce resources that could be used in other ways. A strong case therefore exists for avoiding premature dissemination, improving coordination, and ensuring that effort is focused where it is most likely to benefit the NHS. This could be achieved by a mechanism such as that of the current Standing Group on Health Technology, which takes stock of existing, proposed, and possible future innovations and advises on where development and research efforts might best be concentrated. Such a mechanism should not stifle innovation but channel it to maximise the experience of those who innovate. It could promote the dissemination of successful examples to the rest of the NHS without delay.12

Doctors are under increasing pressure to take account of evidence of clinical effectiveness when treating patients. Similarly, managers should seek out and heed information about the cost effective organisation of health care.

People interested in taking part in the Cochrane Collaboration's proposed review group on structural and organisational changes in the delivery of health care should contact Nicholas Mays, King's Fund Institute, 14 Palace Court, London W2 4HT.