Time to improve the clarity of clinical trial reports by including estimands

In clinical trials, as in practice, treatments are not always taken exactly as prescribed for various reasons. As a clinical trial statistician, I have rarely seen a trial dataset where all participants have been able to follow the desired treatment protocol. When this happens, by looking at trial data in different ways, different questions can be unpicked, such as: “Does the treatment improve health outcomes for all patients even if it is not taken as instructed?” or “Does the treatment improve health outcomes only for patients who take it as prescribed?” The answers to these different questions may lead to different conclusions on treatment benefit; therefore, when evaluating trial results, it is important to understand precisely what question has been addressed.

Clearly defining the treatment effect being investigated in a trial has recently come to the forefront with the publication of new international trial regulatory guidelines (ICH E9(R1)).1 These call for trialists to clarify the precise questions being addressed in trials by using estimands. An estimand is a clear description of precisely what treatment effect a trial is aiming to investigate (for examples see23). Another helpful way to think about an estimand is a definition of precisely what the trial demands to find out (as demand nicely rhymes with estimand). By defining the estimand of interest at the initial planning stages of a trial, a trial can be subsequently designed, conducted, and analysed to answer the key clinical question of interest.

An estimand is quite separate to the statistical method, which is referred to as the estimator and is used to compute the trial result. The numerical trial result is called an estimate.

Even with statistical knowledge I have found it is not always possible to understand exactly what question a clinical trial addresses from the statistical methods alone. As this is a new area of interest, while medicine regulators worldwide are adopting the ICH E9(R1) guidelines,4 my statistical colleagues and I conducted a review of published trials to assess how often the precise question being investigated could be unravelled from the reported methods.2 This was to establish whether the reporting of estimands is necessary in trial reports to understand what question has been addressed.

We identified that, worryingly, most often it is not possible to understand precisely what question has been investigated in trials from the statistical methods. There is an urgent need to include estimands in trial publications to avoid misinterpretation of results.

I discussed selected results of the review with the Public Advisory Panel (PAP) for the National Institute for Health and Care Research funded HEALTHY STATS research project, which aims to improve the information reported from clinical trials for patients. Public partners were aged between 20 and 70 and of mixed ethnicities and sex. The panel was surprised that the precise question being investigated was not always clear from trial reports, and highlighted the need for the description of the question being addressed in a trial to be reported alongside the numerical result. This to avoid confusing healthcare professionals and patients.

The use of estimands in trials will ensure the numerical results being estimated are relevant and useful. In our review we found the statistical methods used in some trials led to the quantification of the effect of treatment for all patients if they had remained on treatment despite adverse events, or had not died. It is unclear if this is what the investigators really wanted to know, or a consequence of the selected methods. The new guidelines on estimands reflect wide acknowledgment that clarifying what is of interest to know should come first (the estimand). Trial design and statistical analysis should then be aligned.

Estimands provide us with an opportunity to ensure the needs of different trial stakeholders are taken into account, including those of patients. To do so, more than one question of interest, hence estimand, may need to be investigated in a trial since the question of most relevance to address will likely vary across different healthcare practitioners, policy makers, and patients. For example, a policy maker, evaluating whether a medical intervention should be introduced for an entire population, will typically be interested in the effect of the intervention for all patients, regardless of adherence. For patients, this is not always the outcome that matters to them. My conversations with the HEALTHY STATS PAP indicated that in a trial where not all the treatment is taken as prescribed, the panel would also like to know the effect of the intervention for those that take treatment as prescribed. The PAP stressed the importance and opportunity of involving patients early on in trial planning discussions to conduct trials that tackle what matters to patients and healthcare professionals.

Trial reporting must be improved by including estimands to clarify the precise questions that have been investigated to remove any chance of misinterpretation.