Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial
BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e3874 (Published 21 June 2012) Cite this as: BMJ 2012;344:e3874
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We thank Dr Innes and his colleagues for their response. We are in the process of analysing data for admissions and mortality for subgroups defined by the three long-term conditions and predictive risk score. The trial was not powered for these subgroup analyses, although they were pre-specified. We will apply a checklist to assess the credibility of any effects [1].
References
[1] Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, et al. Credibility of claims of subgroup effects in randomised controlled trials: a systematic review. BMJ 2012;344:e1553 doi:10.1136/bmj.e1553
Competing interests: No competing interests
We thank Professor Greenhalgh for her response. We agree with her about the importance of non-randomised studies. Yet at the same time, many commentators agree that randomised trials have an important role to play, and we hope that this trial will provide valuable information about telehealth.
Our analysis was conducted and written up in line with published recommendations [1] and our original protocol. Dissemination policies were governed by the standard contractual terms for projects funded by the DH Policy Research Programme [2]. The terms specified that permission to submit findings for publication cannot be withheld. Draft copies of proposed publications were sent to Department of Health in advance of submission for publication and clearance was given in line the contract. We favoured publication of the articles in the peer-reviewed press, so that the draft articles could be extensively examined.
As you note, the Department of Health published several documents during the peer review process [3]. The research team was not involved in these interpretations of the findings and the resulting documents. Our role has been to design and conduct a relevant and high-quality evaluation and to report the findings clearly and transparently. We believe we have done this with the peer-reviewed material.
References
[1] Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension to cluster randomised trials. BMJ 2004; 328:702-708
[2] Department of Health (2012). Policy Research Programme Guidance for Applicants. Available at: http://prp.dh.gov.uk
[3] Department of Health (2011). Whole system demonstrator programme: Headline findings – December 2011. Available at: www.dh.gov.uk/health/2011/12/wsd-headline-findings.
Competing interests: No competing interests
We welcome this high quality randomised control trial1 as it fulfils the need for more robust evaluation of telehealth interventions with evidence to inform our commissioning decisions.
The outcomes of the research reducing secondary care, reductions in mortality are pertinent to primary commissioning outcomes. However the evidence for actual clinical effect of the hospital day’s difference on its own is reported to be not significant enough. Also, the conclusion that the study is not generalisable puts a limitation on local use of the findings to transfer the knowledge directly into our telehealth work.
Two things we noted of relevance to our work locally are firstly, for pathway design that the mechanism for avoiding emergency admissions is not yet clear and there is no indication of which end of risk stratification yields the best outcomes. Secondly, the study is not powered to pick up anything meaningful regarding dementia and their specific carers. We know dementia is going to be one of the biggest Long Term Conditions we need to deal with in the coming years. Early indications from local evaluation work with service providers and patients using telehealth interventions shows benefits for people with long term conditions including dementia. The main problem is the need to have the findings from cost effectiveness and quality of life studies to give us evidence on the outcomes for patients receiving telehealth and whether the studies collectively strengthens the evidence; we need the complete picture.
1. Steventon et al, Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ, 2012. 344.
Competing interests: No competing interests
Steventon et al present interesting data that raise more questions than are answered.
Telehealth takes many forms, and outcomes should not merely be in terms of mortality and pounds saved. In the North of Scotland we have used a combination of video conferencing and telephone consultations to consult with patients in our dispersed population who have chronic kidney disease.
We have seen an improvement in quality of service we can provide, and little or no reduction in our cost, but taking a wider view a reduction in costs to society in terms of patient travel, and time off work for patients and or carers.
We can see more patients in their locality by selecting patients for VC and telephone consultation which frees up out patient clinic space. In our practice this frequently prevents the need for a patient from making a 250 mile round trip, saving their time and reducing CO2 emissions. The feedback from our patients has been positive, and as a sceptical bunch of clinicians we have been pleasantly surprised to find that the quality of our consultations is not diminished.
The bigger picture needs to be seen and rural areas may have the most to benefit from from telehealth solutions.
BMJ2012;344:e3874
Competing interests: No competing interests
In Steventon et al ‘s report on the effect of telehealth on use of secondary care and mortality the apparent benefits in the intervention arm appear to be driven largely by an unexplained increase in adverse events in the control arm following entry to the study.1 A possible explanation for this is that although this was intended to be a “usual care” arm the resources available to deliver usual care had in fact been diverted to provide the trial intervention. The methods state that the telehealth monitoring centres were “staffed by specialist nurses and community matrons from local health organisations” which must mean that they had been diverted from previous roles.
A particular anxiety in COPD has been that resources will be diverted towards unproven telehealth interventions at the expense of high-value interventions with a strong evidence base, specifically pulmonary rehabilitation which remains significantly under-provided in the UK. 2 3 Preliminary findings from the Whole System Demonstrator program presented at the King's Fund suggested the cost per quality adjusted life year (QALY) was in the region of £80,000. As outlined in the accompanying editorial,4 the evaluation of telehealth interventions is complex and needs to be considered within the wider health context. This must include the opportunity costs of investment of finite healthcare resources in this area.
1. Steventon A, Bardsley M, Billings J, Dixon J, Doll H, Hirani S, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ 2012;344.
2. Lacasse Y, Goldstein R, Lasserson TJ, Martin S. Pulmonary rehabilitation for chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2006(4):CD003793.
3. Dodd JW, Hogg L, Nolan J, Jefford H, Grant A, Lord VM, et al. The COPD assessment test (CAT): response to pulmonary rehabilitation. A multicentre, prospective study. Thorax 2011;66(5):425-29.
4. Car J, Huckvale K, Hermens H. Telehealth for long term conditions. BMJ 2012;344.
Competing interests: No competing interests
We read with interest the recent report of the whole system demonstrator randomised controlled trial (WSD)[1]. This was a much needed study of the impact of an increasingly frequent intervention which until now has been deployed with relatively little evidence of efficacy. The headline reduction of an approximate halving of mortality within the intervention group is both impressive and surprising. This is mirrored by the reduction in hospital admissions of 5.3%. Figure 2 reveals this apparent intervention-related reduction in admissions is in fact an early increase in admissions in the control group when compared to baseline. As inferred by the authors this may be due to an increase in anxiety-related requests for admission as a consequence of the denial of telehealth in the control group. It is known that hospital admission can be associated with an increase in subsequent mortality[2, 3]. In particular, in patients with COPD (the largest group included in the trial) the recent Cochrane review comparing hospital at home with inpatient care suggested that the risk ratio for death is 0.65 in those managed at home[4]. Hospital admission appears to be bad for your health!
If hospital admission is the main reason for the greater mortality seen in the WSD control group then this should occur irrespective of the original inclusion diagnosis. We challenge the authors to present the relative risk of admission and death in their three diagnostic categories. If there is no difference then whilst the WSD demonstrates a significant impact on admissions, whether this is due to the intervention, or the unfulfilled expectation of telehealth, is a moot point.
1. Steventon et al, Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ, 2012. 344.
2. Shepperd, S.D., H. Angus, RM. Illiffe, S. Kaira, L Ricauda, NA. WIlson, AD., Hospital at home admission avoidance. Cochrane review, 2010.
3. Bell, C.R., MD,Mortality of patients admitted to hospital on weekends as compared with weekdays. New England Journal of Medicine, 2001. 345(9): p. 663-8.
4. Jeppesen E, B.K., Vist GE, Wedzicha JA, Wright JJ, Greenstone M, Walters JAE, Hospital at home for acute exacerbations of chronic obstructive pulmonary disease. Cochrane review, 2012.
Competing interests: No competing interests
Limitations listed for this trial did not include close involvement of the funder in its design and execution. Under ‘Finances’, the authors state: “The Department of Health reviewed the protocol … and provided project manager support”.[1]
The Department of Health makes greater claims for its own involvement in the trial. In January 2012 it signed a ‘concordat’ with the technology industry, which referred to “a randomised controlled trial funded and run by the Department of Health” (paragraph 1).2
The authors have not commented formally on the substantial mismatch between their findings and conclusions (which were measured and cautious[1]) and those used by the Department of Health to inform policy (which were one-sided and sensationalist[2,3]), though individual WSD researchers have expressed misgivings.[4]
Randomised trials, which ‘control for’ context, have limited purchase for evaluating politically driven e-Health programmes.[5] The Department of Health’s cherry-picking of unanalysed data to put on its website before the trial had finished recruiting was scientifically inappropriate but politically expedient.[6]
The BMJ has led the field in exposing how the pharmaceutical industry’s conflicts of interest distort research. In failing to require the WSD’s authors to consider conflicts of interest by the state (whose intention to implement telehealth was enshrined in policy before the WSD results were analysed[7]), and in privileging randomised trials over study designs that allow analysis of political influences,[8] the BMJ has let itself to be used as a pawn by an increasingly powerful industrial-political complex.
1. Steventon A, Bardsley M, Billings J, Dixon J, Doll H, Hirani S, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ 2012;344:e3874.
2. Department of Health. A concordat between the Department of Health and the telehealth/telecare industry. http://www.3millionlives.co.uk/pdf/Concordat - FINAL.pdf. London: Department of Health, 2012.
3. Department of Health. Whole Systems Demonstrator: Headline findings December 2011. London: Stationery Office, 2011.
4. Praities N. Telehealth 'unlikely to be cost effective', admit researchers leading DH pilot. Pulse 2012;http://www.pulsetoday.co.uk/newsarticle-content/-/article_display_list/1....
5. Greenhalgh T, Russell J. Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles. PLoS Med 2010;7(11):e1000360.
6. Greenhalgh T. Whole System Demonstrator: Desperately seeking peer-reviewed papers (rapid response). BMJ 2011:http://www.bmj.com/rapid-response/2011/11/03/whole-system-demonstrator-d...
7. Department of Health. Operating Framework 2012-13. London: Stationery Office, 2011.
8. Greenhalgh T, Procter R, Wherton J, Sugarhood P, Shaw S. The organising vision for telehealth and telecare: discourse analysis. BMJ Open 2012;in press.
Competing interests: No competing interests
Why did the Whole Systems Demonstrator report reductions in emergency hospital admissions? Insights from new analyses.
The Whole Systems Demonstrator (WSD) trial found that telehealth was associated with lower rates of emergency hospital admissions than usual care amongst patients with long-term health conditions.[1] However, admission rates increased amongst the control group shortly after recruitment, leading to concerns about whether the estimated treatment effect reflected telehealth, or was an artefact of the trial. We have conducted further analyses of this issue,[2] and summarise these below as they have implications for the generalisability of the WSD trial. Attention to generalisability is important because trials typically estimate treatment effects for the sample of individuals recruited, rather than for the whole target population. The two might differ due to differences in the patients, centres or treatments.[3]
Possible explanations for the patterns observed
Like many trials of behavioural interventions, the WSD trial could not be fully blinded,[4] and we previously thought this might explain the increase in admissions amongst the control group.[1] For example, as the trial was cluster randomised, allocations could not always be concealed from those recruiting patients. Thus, health care professionals might have identified unmet need during the recruitment process and have changed the management of patients assigned to the control (usual care). Also, although allocations were concealed from patients until after they had consented to participate, patients then knew whether or not they were receiving telehealth. A range of artefacts can occur when patients decide to join a trial out of a desire to receive new treatments, but find themselves allocated to less desirable care. For example, patients might become demoralised,[5–7] and their utilisation of unscheduled health care might consciously or unconsciously be altered. Finally, it is possible, but unlikely, that there was differential selection of patients into the two arms of the trial.
In the new analysis, we exploited the large, person-level data sets collected for the original evaluation. We contrasted the outcomes of control patients with those of local patients who had the same conditions, and were eligible for the WSD trial, but did not participate. If we assume the trial control patients had similar clinical management to the non-participants (i.e., both had ‘usual care’), their outcomes should also be similar, after adjusting for differences in their baseline characteristics. To assess whether this was indeed the case, we conducted novel placebo tests, which unlike traditional statistical hypothesis tests are designed to examine evidence of similarity rather than of difference.[2] We adjusted for baseline characteristics by matching trial control patients to non-participants.[8] This approach ensured good balance of 65 baseline characteristics between the trial participants and non-participants.
We found that, for categories of health care that are planned using an appointments system, namely outpatient attendances, general practice contacts and elective admissions, trial control patients had similar utilisation patterns to what was expected (see Table 1, column A). However, trial control patients visited emergency departments more than non-participants, and experienced correspondingly higher numbers of emergency admissions. In light of these findings, it no longer seems likely that health care professionals changed the management of patients assigned to the control. However, it is possible that the trial recruitment processes led patients in the control group to change aspects of their health-care seeking behaviour. For example, some patients assigned to this group may have anticipated that they would have received improved care with telehealth, leading to unease in the period immediately following recruitment and a higher proportion of these patients to seek help at emergency rooms than previously.
The placebo tests might have failed due to differences between the non-participants and the RCT control group in prognostic variables that were not observed but had an effect on hospital admissions in the period immediately following recruitment. We note that WSD control patients experienced more deaths than matched non-participants, which could reflect unobserved differences between the trial participants and non-participants that were of specific importance for this endpoint (Table 1). Although we cannot determine the reason for the failure of the placebo tests, the conclusion is the same: we could not confirm that the trial was generalisable with respect to emergency admissions. This is because the placebo test indicates that either the study population differed from the target population in significant ways that were unobserved, or the care received by control patients differed from usual care. Either possibility would make it difficult to apply the trial findings to routine practice.
Sensitivity analyses
To examine the implications of the failure of the placebo tests and the lack of generalisability of some of the initial trial results, we compared patients who were allocated to receive telehealth with a matched subset of local non-participants (Table 1, column C). This comparison produces an alternative estimate of the effect of telehealth compared with usual care, which is non-randomised (and thus susceptible to selection bias from confounding) but does not require the implausible assumption that control patients in the trial received usual care. It can be contrasted with the estimate from the RCT data alone, which benefits from randomisation but assumes that control patients received usual care (Table 1, column B).
For endpoints where the placebo tests passed (such as general practice contacts), estimated treatment effects were similar regardless of whether the telehealth patients were compared with RCT controls or non-participants. Thus, we can be reasonably confident that telehealth led to small increases in general practice contacts compared with usual care.[9] However, while the original study reported that telehealth was associated with fewer emergency admissions than usual care, the alternative analysis reports that telehealth was associated with a trend towards more emergency admissions (not statistically significant, at the 5% level).
Conclusions
These placebo tests are an example of a growing body of methods that assess aspects of the generalisability of RCTs quantitatively.[10–14] Ideally, RCT investigators would conduct placebo tests and accompanying sensitivity analyses to assess the generalisability of their findings, particularly when trials are embedded within large databases.[15,16]. In WSD, the placebo test indicates that we cannot generalise the findings for emergency admissions to routine practice. This should be reflected in future systematic reviews.
References
1 Steventon A, Bardsley M, Billings J, et al. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ 2012;344:e3874. doi:10.1136/bmj.e3874
2 Steventon A, Grieve R, Bardsley M. An approach to assess generalizability in comparative effectiveness research: a case study of the Whole Systems Demonstrator cluster randomized trial comparing telehealth with usual care for patients with chronic health conditions. Med Decis Mak
3 Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”Lancet 2005;365:82–93. doi:10.1016/S0140-6736(04)17670-8
4 Day SJ, Altman DG. Blinding in clinical trials and other studies. BMJ Br Med J 2000;321:504. doi:10.1136/bmj.321.7259.504
5 Cook TD, Campbell DT. Quasi-experimentation: Design and Analysis Issues for Field Settings. Boston: : Houghton Mifflin 1979.
6 Ward E, King M, Lloyd M, et al. Conducting randomized trials in general practice: methodological and practical issues. Br J Gen Pract 1999;49:919–22.
7 McCambridge J, Sorhaindo A, Quirk A, et al. Patient preferences and performance bias in a weight loss trial with a usual care arm. Patient Educ Couns 2014;95:243–7. doi:10.1016/j.pec.2014.01.003
8 Diamond A, Sekhon JS. Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies. Rev Econ Stat 2013;95:932–45. doi:10.1162/REST_a_00318
9 Bardsley M, Steventon A, Doll H. Impact of telehealth on general practice contacts: findings from the Whole Systems Demonstrator cluster randomised trial. BMC Health Serv Res 2013;13:395. doi:10.1186/1472-6963-13-395
10 Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. Am J Epidemiol 2010;172:107–15. doi:10.1093/aje/kwq084
11 Stuart EA, Cole SR, Bradshaw CP, et al. The use of propensity scores to assess the generalizability of results from randomized trials. J R Stat Soc Ser A (Statistics Soc 2011;174:369–86. doi:10.1111/j.1467-985X.2010.00673.x
12 Hartman E, Grieve R, Ramsahai R, et al. From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects. J R Stat Soc Ser A (Statistics Soc 2015;Epub Ahead. doi:10.1111/rssa.12094
13 Pressler TR, Kaizar EE. The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Stat Med 2013;32:3552–68. doi:10.1002/sim.5802
14 Frangakis CE. The calibration of treatment effects from clinical trials to target populations. Clin Trials 2009;6:136–40. doi:10.1177/1740774509103868
15 Fröbert O, Lagerqvist B, Olivecrona GK, et al. Thrombus aspiration during ST-segment elevation myocardial infarction. N Engl J Med 2013;369:1587–97. doi:10.1056/NEJMoa1308789
16 Van Staa T-P, Goldacre B, Gulliford M, et al. Pragmatic randomised trials using routine electronic health records: putting them to the test. BMJ 2012;344:e55. doi:10.1136/bmj.e55
Competing interests: No competing interests