The finding of no connection by Yank, Rennie and Bero1 between
results in meta-analyses and financial ties is surprising, given the
finding by Chan and Altman2 of “greater deficiencies for reporting of harm
outcomes among trials that were solely funded by industry (median 56% per
trial) compared with those that were not (27%).” Evidently the Oxman and
Guyatt3 measure of scientific quality of research reviews used by Yank,
Rennie and Berro in their study is not sensitive to the capture of primary
research protocol-to-publication discrepancies and the selective reporting
of outcomes, which Chan and Altman found to be prevalent in a large sample
of PubMed-indexed randomized trial results.
In context, three matters need emphasis. First, as indicated by Chan
and Altman,4 “outcome reporting bias acts in addition to and in the same
direction as publication bias of entire studies to produce inflated
estimates of treatment effect.” Second, the antidote is to require
registration of all trials and protocols in the public domain before study
completion and to assure that they be made available along with any
manuscript undergoing peer review for journal publication.
Third, it is important to correct the mistaken belief that somehow
the collected raw data are unaffected by the artifacts of research design,
sampling, and measurement. Epstein’s5 dismissal of the importance of the
Yank, Rennie and Bero1 findings, along with his strained argument to
justify problematic drug and medical device industry practices, are based
on such misunderstanding. Epstein’s5 assertion that “nothing in the work
of Yank and colleagues suggests that the raw data from the drug sponsored
studies were defective,” overlooks the authors’ use of the admittedly
subjective Oxman-Guyatt3 measure of research quality as a statistical
control variable. Meta-analysts, when combining the results of primary
reports, are unlikely to run to the ground, as Chan and Altman2 did,
research protocol-to-publication discrepancies and selective reporting of
outcomes.
Given the high stakes for the public health, Epstein’s5 choice
between “fewer studies of presumably better quality” and “more studies
whose quality may be more biased” in favor of the latter makes no sense
except from the self-interested perspective of the drug and medical device
industry. In biomedical as in all research, research quality is judged by
(1) the importance of the question addressed and (2) the reliability of
the answer. Indeed, Epstein’s argument that government intervention in the
form of legal restrictions would be economically dysfunctional asserts the
interests of industry. It is the argument of an unabashed industry
apologist,6 who glosses over the fact that in welfare economics regulatory
intervention is sometimes the solution for market failure.7 Widespread
premeditated bias8 in published and unpublished reports of clinical trial
results linked to industry sponsorship is certainly “smoking gun” evidence
of market failure.
1 Yank V, Rennie D, Bero LA. Financial ties and concordance between
results and conclusions in meta-analyses: retrospective cohort study. BMJ
2007; 335; 1202-1205.
2 Chan AW, Altman DG. Identifying outcome reporting bias in
randomized trials on PubMed: review of publications and survey of authors.
BMJ 2005; 330; 1-6, doi:10.1136/bmj.38356.424606.8F, available at http://bmj.bmjjournals.com/cgi/content/full/330/7494/753[accessed on September 7, 2006].
3 Oxman AD, Guyatt GH. Validation of an index of the quality of
review articles. J Clin Epidemiol 1991; 44; 1271-1278.
5 Epstein RA. Influence of pharmaceutical funding on the conclusions
of meta-analyses. BMJ 2007; 335; 1167.
6 Epstein RA. Pharma furor: why two high-profile attacks on big drug
companies flunk the test of basic economics. Legal Affairs 2005 (Jan/Feb), http://www.legalaffairs.org/issues/January-February-
2005/review_epstein_janfeb05.msp
Rapid Response:
Surprising Non-Finding, Predictable Criticism
The finding of no connection by Yank, Rennie and Bero1 between
results in meta-analyses and financial ties is surprising, given the
finding by Chan and Altman2 of “greater deficiencies for reporting of harm
outcomes among trials that were solely funded by industry (median 56% per
trial) compared with those that were not (27%).” Evidently the Oxman and
Guyatt3 measure of scientific quality of research reviews used by Yank,
Rennie and Berro in their study is not sensitive to the capture of primary
research protocol-to-publication discrepancies and the selective reporting
of outcomes, which Chan and Altman found to be prevalent in a large sample
of PubMed-indexed randomized trial results.
In context, three matters need emphasis. First, as indicated by Chan
and Altman,4 “outcome reporting bias acts in addition to and in the same
direction as publication bias of entire studies to produce inflated
estimates of treatment effect.” Second, the antidote is to require
registration of all trials and protocols in the public domain before study
completion and to assure that they be made available along with any
manuscript undergoing peer review for journal publication.
Third, it is important to correct the mistaken belief that somehow
the collected raw data are unaffected by the artifacts of research design,
sampling, and measurement. Epstein’s5 dismissal of the importance of the
Yank, Rennie and Bero1 findings, along with his strained argument to
justify problematic drug and medical device industry practices, are based
on such misunderstanding. Epstein’s5 assertion that “nothing in the work
of Yank and colleagues suggests that the raw data from the drug sponsored
studies were defective,” overlooks the authors’ use of the admittedly
subjective Oxman-Guyatt3 measure of research quality as a statistical
control variable. Meta-analysts, when combining the results of primary
reports, are unlikely to run to the ground, as Chan and Altman2 did,
research protocol-to-publication discrepancies and selective reporting of
outcomes.
Given the high stakes for the public health, Epstein’s5 choice
between “fewer studies of presumably better quality” and “more studies
whose quality may be more biased” in favor of the latter makes no sense
except from the self-interested perspective of the drug and medical device
industry. In biomedical as in all research, research quality is judged by
(1) the importance of the question addressed and (2) the reliability of
the answer. Indeed, Epstein’s argument that government intervention in the
form of legal restrictions would be economically dysfunctional asserts the
interests of industry. It is the argument of an unabashed industry
apologist,6 who glosses over the fact that in welfare economics regulatory
intervention is sometimes the solution for market failure.7 Widespread
premeditated bias8 in published and unpublished reports of clinical trial
results linked to industry sponsorship is certainly “smoking gun” evidence
of market failure.
1 Yank V, Rennie D, Bero LA. Financial ties and concordance between
results and conclusions in meta-analyses: retrospective cohort study. BMJ
2007; 335; 1202-1205.
2 Chan AW, Altman DG. Identifying outcome reporting bias in
randomized trials on PubMed: review of publications and survey of authors.
BMJ 2005; 330; 1-6, doi:10.1136/bmj.38356.424606.8F, available at
http://bmj.bmjjournals.com/cgi/content/full/330/7494/753[accessed on September 7, 2006].
3 Oxman AD, Guyatt GH. Validation of an index of the quality of
review articles. J Clin Epidemiol 1991; 44; 1271-1278.
4 Chan, Altman, ‘Identifying outcome reporting bias’, op. cit, p. 5.
5 Epstein RA. Influence of pharmaceutical funding on the conclusions
of meta-analyses. BMJ 2007; 335; 1167.
6 Epstein RA. Pharma furor: why two high-profile attacks on big drug
companies flunk the test of basic economics. Legal Affairs 2005 (Jan/Feb),
http://www.legalaffairs.org/issues/January-February-
2005/review_epstein_janfeb05.msp
7Market failure: http://en.wikipedia.org/wiki/Market_failure
8 Noble JH. Detecting bias in biomedical research: looking at study
design and published findings is not enough. Monash Bioethics Rev 2007;
26; 24-45.
Competing interests:
None declared
Competing interests: No competing interests