Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study
BMJ 2007; 335 doi: https://doi.org/10.1136/bmj.39376.447211.BE (Published 06 December 2007) Cite this as: BMJ 2007;335:1202
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
The finding of no connection by Yank, Rennie and Bero1 between
results in meta-analyses and financial ties is surprising, given the
finding by Chan and Altman2 of “greater deficiencies for reporting of harm
outcomes among trials that were solely funded by industry (median 56% per
trial) compared with those that were not (27%).” Evidently the Oxman and
Guyatt3 measure of scientific quality of research reviews used by Yank,
Rennie and Berro in their study is not sensitive to the capture of primary
research protocol-to-publication discrepancies and the selective reporting
of outcomes, which Chan and Altman found to be prevalent in a large sample
of PubMed-indexed randomized trial results.
In context, three matters need emphasis. First, as indicated by Chan
and Altman,4 “outcome reporting bias acts in addition to and in the same
direction as publication bias of entire studies to produce inflated
estimates of treatment effect.” Second, the antidote is to require
registration of all trials and protocols in the public domain before study
completion and to assure that they be made available along with any
manuscript undergoing peer review for journal publication.
Third, it is important to correct the mistaken belief that somehow
the collected raw data are unaffected by the artifacts of research design,
sampling, and measurement. Epstein’s5 dismissal of the importance of the
Yank, Rennie and Bero1 findings, along with his strained argument to
justify problematic drug and medical device industry practices, are based
on such misunderstanding. Epstein’s5 assertion that “nothing in the work
of Yank and colleagues suggests that the raw data from the drug sponsored
studies were defective,” overlooks the authors’ use of the admittedly
subjective Oxman-Guyatt3 measure of research quality as a statistical
control variable. Meta-analysts, when combining the results of primary
reports, are unlikely to run to the ground, as Chan and Altman2 did,
research protocol-to-publication discrepancies and selective reporting of
outcomes.
Given the high stakes for the public health, Epstein’s5 choice
between “fewer studies of presumably better quality” and “more studies
whose quality may be more biased” in favor of the latter makes no sense
except from the self-interested perspective of the drug and medical device
industry. In biomedical as in all research, research quality is judged by
(1) the importance of the question addressed and (2) the reliability of
the answer. Indeed, Epstein’s argument that government intervention in the
form of legal restrictions would be economically dysfunctional asserts the
interests of industry. It is the argument of an unabashed industry
apologist,6 who glosses over the fact that in welfare economics regulatory
intervention is sometimes the solution for market failure.7 Widespread
premeditated bias8 in published and unpublished reports of clinical trial
results linked to industry sponsorship is certainly “smoking gun” evidence
of market failure.
1 Yank V, Rennie D, Bero LA. Financial ties and concordance between
results and conclusions in meta-analyses: retrospective cohort study. BMJ
2007; 335; 1202-1205.
2 Chan AW, Altman DG. Identifying outcome reporting bias in
randomized trials on PubMed: review of publications and survey of authors.
BMJ 2005; 330; 1-6, doi:10.1136/bmj.38356.424606.8F, available at
http://bmj.bmjjournals.com/cgi/content/full/330/7494/753[accessed on September 7, 2006].
3 Oxman AD, Guyatt GH. Validation of an index of the quality of
review articles. J Clin Epidemiol 1991; 44; 1271-1278.
4 Chan, Altman, ‘Identifying outcome reporting bias’, op. cit, p. 5.
5 Epstein RA. Influence of pharmaceutical funding on the conclusions
of meta-analyses. BMJ 2007; 335; 1167.
6 Epstein RA. Pharma furor: why two high-profile attacks on big drug
companies flunk the test of basic economics. Legal Affairs 2005 (Jan/Feb),
http://www.legalaffairs.org/issues/January-February-
2005/review_epstein_janfeb05.msp
7Market failure: http://en.wikipedia.org/wiki/Market_failure
8 Noble JH. Detecting bias in biomedical research: looking at study
design and published findings is not enough. Monash Bioethics Rev 2007;
26; 24-45.
Competing interests:
None declared
Competing interests: No competing interests
Yank and colleagues' recommendation that pharma sponsored drug trials
should be interpreted with caution, is well made (1). If the authors’
analysis is correct, healthcare commissioners should be even more cautious
when it comes to interpreting industry sponsored palliative coronary
interventional studies and advice given by professional bodies who have
close links with industry.
Most percutaneous coronary intervention (PCI) procedures involve the
implantation of a coronary stent. The majority of stent studies are funded
by equipment manufacturers and are designed and conducted by researchers
who believe in coronary intervention despite the lack of hard evidence of
cost effectiveness or clinical superiority over optimal medical therapy
(2,3). Therapists’ irrational faith in intuitive based practice adds an
extra dimension to the ‘positive spin’ effect described in the paper.
Given the paucity of independently funded coronary stent studies and the
total lack of a placebo controlled study of this palliative therapy,
healthcare commissioners have a hard time unravelling spin, especially
when professional bodies weigh in with their spin on the evidence.
The RITA 2 study showed that whilst palliative PCI was associated
with a small and transient improvement in symptoms, it increased the
incidence of MI/death by nearly 80% and was £2,684 more costly than
medical therapy (4,5). It is worth remembering that the highly effective
anti inflammatory agent Vioxx was withdrawn when a near identical excess
incidence of MI/death was uncovered. Despite RITA 2’s worrying results the
British Cardiovascular Intervention Society (BCIS) strongly lobbied for an
expansion of PCI. What grounds did BCIS have for supporting the rapid
expansion of palliative PCI despite the its poor showing in RITA 2?
The present study suggests that it would be prudent to take account
of the relationship between BCIS and industry when assessing BCIS’s
recommendations. A search through UK professional bodies’ websites reveals
that BCIS appears to be the only UK professional body to admit to having
two industry representatives on council (6).
References
1. Yank V et al. Financial ties and concordance between results and
conclusions in meta-analyses: retrospective cohort study. BMJ,
doi:10.1136/bmj.39376.447211.BE (published 16 November 2007)
2. Griffin et al. Cost effectiveness of clinically appropriate
decisions on alternative treatments for angina pectoris: prospective
observational study. BMJ 2007;334:624 (24 March)
3. Boden et al. Optimal Medical Therapy with or without PCI for
Stable Coronary Disease. N Engl J Med 2007; 356:1503-1516
4. RITA-2 Trial Participants. Coronary angioplasty versus medical
therapy for angina: the second Randomised Intervention Treatment of Angina
(RITA-2) trial. Lancet 1997;350:461–468
5. Sculpher et al. Coronary angioplasty versus medical therapy for
angina. Eur Heart J 2002;23:1291-300.
6. http://www.bcis.org.uk/about/council
Competing interests:
Professor Chester provides advice to PCT/PBC Commissioners on patient centred angina service redesign
Competing interests: No competing interests
How valid were the assessments of concordance between results and conclusions?
Although the study by Yank et al appears for the most part to be
carefully conducted, there is an important deficiency in their methods. I
note that the information on whether results and conclusions were
favourable or otherwise was assessed in an unblinded manner, mostly by a
single rater. Given their a priori hypothesis that “financial ties to one
drug company would be associated with favourable results and conclusions”
and the subjective nature of assessing whether conclusions are
“favourable”, there is a clear potential for bias in assessing outcomes. I
do not find the moderate inter-rater agreement of 0.6 for assessment of
conclusions reassuring in this respect.
I therefore decided to look at a small number of the papers rated as
having favourable conclusions in the absence of favourable results to see
if those assessments appeared valid. The second such paper I found (Conlin
et al, reference w18) was rated as having conclusions that were “In favour
of study drug”. The drug first mentioned was losartan, which is also the
drug marketed by the company with financial ties to the paper, and so I
assume was considered the “study drug”. However, the conclusions of the
paper do not in any way favour losartan. The discussion section of the
paper begins “This comprehensive analysis shows the comparable efficacy of
losartan, valsartan, irbesartan and candesartan”. Hardly “in favour of”
losartan.
The conclusions of Yank et al’s analysis depend on a valid and
unbiased assessment of whether results and conclusions favoured the study
drugs or not. The validity of those assessments appears open to question,
which therefore calls into question the extent to which we can trust the
conclusions.
Competing interests:
My company provides manuscript writing services to pharmaceutical companies.
Competing interests: No competing interests