Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses

I am troubled by the suggestion by Sun et al that a subgroup effect
consistent with a pre-specified direction will increase the credibility of
a subgroup analysis, and that getting the direction wrong weakens the case
for a real underlying subgroup effect. 1

Freedman has suggested that the concept of clinical equipoise in the
conduct of a study requires that there is uncertainty, not necessarily on
the part of the individual investigator, but within the clinical
community. 2

Greenland argues that the design and analysis of studies may be
biased towards results desired by an investigator. Investigator bias may
arise from many sources and can be a major source of uncertainty about
study effects. 3

While it may be appropriate for an investigator to reveal a bias in
one direction or another, it should not be necessary to accurately predict
what is going to be discovered in any sub-group analysis. If a bias is
revealed or a prediction made, it is important that neither influence the
conduct of the study, nor the validity or credibility of the results.

1. Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect
believable? Updating criteria to evaluate the credibility of subgroup
analyses BMJ 2010; 340: c117

2. Freedman, B. Equipoise and the ethics of clinical research. N Engl
J Med. 1987; 317:141-145.

3. Greenland, S. Accounting for uncertainty about investigator bias:
disclosure is informative: How could disclosure of interests work better
in medicine, epidemiology and public health? J Epidemiol Community Health
2009; 63:593-598

Competing interests:
None declared

We read with great interest Sun et al’s paper about evaluating the
credibility
of subgroup effects. We agree with the third point “is the significant
subgroup effect independent” although we would raise one concern
regarding the authors’ final suggestion that in a small sample size it is
acceptable to pre-specify a limited number of important interactions to be

considered.

In analyses of non-randomized interventions it is important to adjust
for all
potential confounders since otherwise the estimated intervention effect
may
be biased. In an RCT with randomized allocation to interventions there
should be no confounders and it is not typically necessary to adjust for
other
factors (although doing so may increase precision in the estimated
intervention effect). However in an RCT, an analysis concerning
differences in
intervention effects in a subgroup should initially use a full model
including
all interactions between all possible confounders and that subgroup,
because
otherwise the interaction of interest may be confounded. This is well
noted
by the authors in their smoking and fracture-type example.

The authors suggestion that it is acceptable to pre-specify a limited
number
of important interactions to be considered is perhaps analogous to, and as

valid as, only including a pre-defined subset of possible confounders in
an
analysis of observational data. It would be better to create a full model
and
use a standard model selection process, such as change in estimate or
model
fit, to create a parsimonious model. If the sample size is too small to
permit
detailed consideration of all possible confounders to a subgroup effect,
it
may be more appropriate not to attempt to identify subgroup effects.

In our own work we have found a simple rule of thumb to be helpful in

judging whether subgroup effects might be confounded. If the coefficient
for
the subgroup interaction of interest is very similar to the difference
between
the coefficients from subgroup specific (i.e. stratified) analysis, then
the
model may be adequate and the subgroup difference inferred credible.
Conversely, if there is a discrepancy it suggests that the model is mis-
specified, additional interaction terms are needed and the subgroup
difference inferred may not be credible. Sun et al’s paper provides a very

helpful way forward for assessing the credibility of subgroup analyses,
which
need not be confined to randomized control trials and could be further
improved by being both more rigorous and more pragmatic.

Competing interests:
None declared

Interesting paper - thank you for attempting to clarify subgroup
analyses and effect modification in general. I do feel however that some
of your suggestions are not entirely convincing.

1) You note, correctly, that the presence of an effect modification
depends on the model that is used for combining risks, i.e., additive or
multiplicative. But then you affirm that the multiplicative model is the
correct one, on the basis of an example that merely shows that an additive
interaction may correspond to the absence of a multiplicative interaction.
How does that make the multiplicative model better? In real life, results
fall often inbetween models - e.g. the risk of disease may be 10% in
female nonsmokers, 30% in female smokers, 20% in male nonsmokers, and 50%
in male smokers. So the effect of smoking is STRONGER in men on the
additive scale (+20% in women but +30% in men), but WEAKER on the
multiplicative scale (x3 in women but x2.5 in men). How can that be (or
rather: what can that mean in biological terms)? In fact the "effect
modification" is an artefact caused by the choice of the statistical
model. I would suggest that interpreting statistical interactions in
biological terms is hazardous unless the baseline risks are nearly
identical (e.g. the risks of disease in male and female nonsmokers). More
discussion of this problem in Rothman and Grenland's Modern epidemiology.

2) You submit that consistency (of the presence of an interaction)
across closely related outcomes lends credibility to the effect. That is
circular logic. If two outcomes are closely correlated within a sample,
their analysis MUST yield similar risk models. There is no corroborating
evidence in that observation.

Thank you for your thoughts on this, and apologies for taking up so
much space...

Competing interests:
None declared