External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges

The authors Riley et. al. [1] described each and every step, very efficiently and effectively regarding the issues in the analysis of big data and their validation in clinical prediction models from e-health records. We read all the related articles given in the reference portion, a majority of which are the published articles by the authors themselves. We just wanted to add few points in this regard along with a few concerns. First of all, that these online e-health data sets are very easy to use, let us take the example of data sets of Demographic and Health Surveys (DHS) [2] Program. If you want to access the data for your research project (your topic should fall in the category of research conducted by DHS), you simply have to register and login through a link.

https://dhsprogram.com/data/dataset_admin/login_main.cfm?logout=&CFID=16...

Then submit your research proposal and mention the region or a specific country, of which you need the data. If your proposal is accepted, you are allowed to access and download the corresponding data. Suppose you want to access further data sets of the same/or other regions, you make another request, make necessary changes in your application and DHS will modify your application. Then they even assist you in providing with step by step guide from entering the data to its processing and: (a) to select the surveys (of your research interest by type, country, year) for analysis, b) review the questionnaires, c) you can edit/delete or modify your variables d) you can use sample weights and deem special values etc. They even guide you how to open the data sets and how to put the variables SPSS, SAS and Stata through the following links.

https://dhsprogram.com/data/Data-Processing.cfm
https://dhsprogram.com/data/Using-DataSets-for-Analysis.cfm

In our country, Pakistan, if you want to conduct a clinical research, the access to a data of few hundred patients could lead to a wait of several months (and sometimes years), it is often not free of charge and needs approval from several authorities before they could handle you the original data, along with a list of “ghost writers” to be added in the manuscript, making it literally impossible to conduct even a small research project. Then the analysis of the data is much more difficult and time-consuming as the majority of the data is in hard form, not legible at times (if you need older data for longitudinal studies), also with a lot of missing variables or missing data. And so is the-the data entry of all these variables; with a huge amount of errors.

In such scenario, these online data sets are a blessing. Even for categorical analysis of prediction or for complex indicators, the Guide to DHS Statistics is also available – it is free to use. You can download the final reports of each dataset and also order it in hard form – yet again, free of charge.

So, I think that the chances of biases, mistake or miscalculations are minimized while using such datasets. You can compare your results with that of the DHS results for proper validation. Now considering the analysis of these datasets a “challenge”, is an overstatement in this particular scenario on which hundreds of research articles have already been published.

Secondly, many e-health data set providers help you in the analysis of the data. Sometimes they charge (for few services) and sometimes they don’t. The charges are nominal but somehow a relief to handle the big sets of longitudinal studies – especially if one is writing for a high impact factor journal like BMJ.

We agree with the authors regarding the fact that the proposed models for the prediction research are too much and the reliability of these models depends on the external validation. [3, 4] But mostly in medical research, before applying a specific model for prediction, its reliability is often measured; not only for the whole model but for the individual component analysis through various statistical testing procedures. So, a proper knowledge of statistical analysis, use of the softwares can be of great help.

Thirdly, we already have the guidelines for the validation and re-evaluation of the proposed predicting models by the same team members [5-7], known as Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) for prognostic and diagnostic prediction modeling research [8, 9]. They had already discussed the challenges and opportunities in their previous papers. We scrutinized all their research papers, but somehow, we could not find a novel approach in this paper. It is simply, a summary of their previous findings.

Lastly, in one other research, your team mentioned [10] that for IPD research, the meta-analysis can offer some unique prospects for prediction research through discrete model intercept terms for each and every research and for a different set of the population, for the sake of generalizability and also validate their model, one can utilize ‘internal-external cross-validation’. So you have already concluded with the solution of this problem, then why it was necessary to reproduce another article by combining all the previous results?

And as far as the errors in the methodology are concerned, it can be reduced by the collaborations with those study groups that are actively involved in IPD predictor research.

Despite the challenges mentioned by the authors of the study, e-health data sets are a blessing for young researchers like us for prediction research. We, cannot afford to travel to several countries to conduct our research project; it is very time-consuming and costly for us - especially for the third world countries where we have to self-fund our education and so as the research project. Although Riley and his colleagues, described the challenges and opportunities related to big data sets, we just wanted to add few things for the information of the BMJ readers regarding the ease and usage guidelines for these big data sets.

References
1. Riley Richard D, Ensor Joie, Snell Kym I E, Debray Thomas P A, Altman Doug G, Moons Karel G M et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges BMJ 2016; 353 :i3140
2. DHS implementing partners and ICF International. Demographic and Health Surveys 1988-2015 [Dataset, modify years as appropriate]. Data extract from DHS Recode files. IPUMS-Demographic and Health Surveys (IPUMS-DHS), version 3.0, Minnesota Population Center and ICF International [Distributors]. Accessed from http://idhsdata.org on 6.23.2016.
3. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med2012;9:e1001221. doi:10.1371/journal.pmed.1001221 pmid:22629234.
4. Antoine E, Susan M, Marie D, Jean-N P, Mike C. Prognostic models in acute pulmonary embolism: a systematic review and meta-analysis. BMJ Open 2016;6:e010324 doi:10.1136/bmjopen-2015-010324.
5. Collins GS, Moons KG. Comparing risk prediction models. BMJ2012;344:e3186. doi:10.1136/bmj.e3186 pmid:22628131.
6. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med2016;35:214-26. doi:10.1002/sim.6787 pmid:26553135.
7. Collins GS, Altman DG. Predicting the 10-year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ2012;344:e4181. doi:10.1136/bmj.e4181 pmid:22723603.
8. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med2015;162:W1-73. doi:10.7326/M14-0698 pmid:25560730.
9. Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. BJOG 2015; 122 (3): 434–443.
10. Ahmed I, Debray TP, Moons KG, Riley RD. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Med Res Methodol2014;14:3. doi:10.1186/1471-2288-14-3 pmid:24397587.