How valid were the assessments of concordance between results and conclusions?
Although the study by Yank et al appears for the most part to be
carefully conducted, there is an important deficiency in their methods. I
note that the information on whether results and conclusions were
favourable or otherwise was assessed in an unblinded manner, mostly by a
single rater. Given their a priori hypothesis that “financial ties to one
drug company would be associated with favourable results and conclusions”
and the subjective nature of assessing whether conclusions are
“favourable”, there is a clear potential for bias in assessing outcomes. I
do not find the moderate inter-rater agreement of 0.6 for assessment of
conclusions reassuring in this respect.
I therefore decided to look at a small number of the papers rated as
having favourable conclusions in the absence of favourable results to see
if those assessments appeared valid. The second such paper I found (Conlin
et al, reference w18) was rated as having conclusions that were “In favour
of study drug”. The drug first mentioned was losartan, which is also the
drug marketed by the company with financial ties to the paper, and so I
assume was considered the “study drug”. However, the conclusions of the
paper do not in any way favour losartan. The discussion section of the
paper begins “This comprehensive analysis shows the comparable efficacy of
losartan, valsartan, irbesartan and candesartan”. Hardly “in favour of”
losartan.
The conclusions of Yank et al’s analysis depend on a valid and
unbiased assessment of whether results and conclusions favoured the study
drugs or not. The validity of those assessments appears open to question,
which therefore calls into question the extent to which we can trust the
conclusions.
Competing interests:
My company provides manuscript writing services to pharmaceutical companies.
Rapid Response:
How valid were the assessments of concordance between results and conclusions?
Although the study by Yank et al appears for the most part to be
carefully conducted, there is an important deficiency in their methods. I
note that the information on whether results and conclusions were
favourable or otherwise was assessed in an unblinded manner, mostly by a
single rater. Given their a priori hypothesis that “financial ties to one
drug company would be associated with favourable results and conclusions”
and the subjective nature of assessing whether conclusions are
“favourable”, there is a clear potential for bias in assessing outcomes. I
do not find the moderate inter-rater agreement of 0.6 for assessment of
conclusions reassuring in this respect.
I therefore decided to look at a small number of the papers rated as
having favourable conclusions in the absence of favourable results to see
if those assessments appeared valid. The second such paper I found (Conlin
et al, reference w18) was rated as having conclusions that were “In favour
of study drug”. The drug first mentioned was losartan, which is also the
drug marketed by the company with financial ties to the paper, and so I
assume was considered the “study drug”. However, the conclusions of the
paper do not in any way favour losartan. The discussion section of the
paper begins “This comprehensive analysis shows the comparable efficacy of
losartan, valsartan, irbesartan and candesartan”. Hardly “in favour of”
losartan.
The conclusions of Yank et al’s analysis depend on a valid and
unbiased assessment of whether results and conclusions favoured the study
drugs or not. The validity of those assessments appears open to question,
which therefore calls into question the extent to which we can trust the
conclusions.
Competing interests:
My company provides manuscript writing services to pharmaceutical companies.
Competing interests: No competing interests