Effect of school closures on mortality from coronavirus disease 2019: old and new predictions
BMJ 2020; 371 doi: https://doi.org/10.1136/bmj.m3588 (Published 07 October 2020) Cite this as: BMJ 2020;371:m3588Read our latest coverage of the coronavirus outbreak
Linked Editorial
Predicting the pandemic
Linked Opinion
Covid-19: Modelling the pandemic
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Editor,
The paper comments that the "original code used for Report 9 has not been released", going on to state that the team used the re-implementation of the Imperial model called CovidSim.
On reading this, I contacted the corresponding author Graeme Ackland on 08/10/2020 and mentioned that the original code was available as early as 9th July (1). In response, he commented that the paper was submitted in June. On asking why they were unable to use the original code I was informed that:
"Ben and Victoria's group were heavily involved in the github rewrite
'CovidSim' which was carefully verified to give the same outputs as the
original, but was much faster, nicer to use and easier to
compile. There didn't seem to be any point in using the original when
we had a better version we'd already worked with." (2)
Based on his comment, the paper is misleading in that it states the original code had not been released implying that is why they subsequently used CovidSim. It had not been publicly released, but their choice appears to have been based on convenience and performance instead.
This is important as Report 9 was not produced with what he refers to as the "better version". Although stating that CovidSim has been carefully verified to give the same outputs as the original, the reference for this in their paper is the Codecheck (3). This only compares the CovidSim output to Report 9. The Codecheck has no tests with the original code which it refers to as "private code", noting that CovidSim gives the same trend but not the same absolute values as Report 9.
On the other hand, the readme.txt file Imperial has supplied with the original code released under FOIA gives details on how to exactly reproduce the results of Report 9:
"This code will not work without an input population file which is commercially licensed. We will release that underlying population file to individuals who provide written proof that they have a license for Landscan 2018 - see https://landscan.ornl.gov/landscan-data-availability.
The code will only exactly reproduce report 9 results if run on a 32-core Windows machine with the network file provided."
As there appears to have been a means for the Edinburgh team to use the most accurate method for reproducing Report 9 i.e. the original code, doing further analysis with it, and had knowledge of using it due to their involvement in the rewrite; their paper should have had a clearer explanation why they chose not to do so.
I followed-up with Graeme Ackland on 11/10/2020 and 15/10/2020 seeking clarification of his comments but have not received a response.
1) Provided by Imperial College London in response to an Internal Review of FOIA request IMPFOI-20-293, 09/07/2020
2) Email correspondence with Graeme Ackland, 08/10/2020
3) Stephen J. Eglen. (2020, May 29). CODECHECK certificate 2020-010. Zenodo. http://doi.org/10.5281/zenodo.3865491
Competing interests: No competing interests
Dear Editor,
The authors have highlighted the importance of data related to care home residents and how it was not available to modellers at the outset. This lack of data for residential institutions, such as care homes and prisons, was also raised in 2006 (1) in the last publication prior to Report 9 (2) that used results based on the original Imperial software. Although being identified as important pre and post Report 9, Report 9 makes no mention of this gap in knowledge yet the age-dependent risk is recognised in the report.
If government policy is to be informed by a computer model then perhaps the limitations should clearly accompany the results. This could include references to how the code i.e. the implementation of the model, has been validated.
The Edinburgh team has included such a reference, the external validation of CovidSim against the results of Report 9 (3). However, CovidSim is the post-Report 9 implementation of the Imperial model which has been developed using practices that give a better quality of code. Although the model may not have changed, a new implementation could add or remove behaviours from a previous implementation which may only manifest under very specific conditions.
Is there a broader set of test scenarios and results that the authors could have referenced comparing the CovidSim implementation against the original implementation as used for Report 9 to better evidence consistency between versions?
(1) Ferguson NM, Cummings DAT, Fraser C, Cajka JC, Cooley PC, Burke DS. Strategies for mitigating an influenza pandemic. Nature 2006;442(7101):448–52.
(2) Neil M Ferguson, Daniel Laydon, Gemma Nedjati-Gilani et al. Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imperial College London (16-03-2020), doi: https://doi.org/10.25561/77482.
(3) Stephen J. Eglen. (2020, May 29). CODECHECK certificate 2020-010. Zenodo. http://doi.org/10.5281/zenodo.3865491
Competing interests: No competing interests
Dear Editor,
In his accompanying opinion piece, Graeme Ackland poses the question, "Why was this advice ignored?"; the advice being, presumably, to leave schools open, because, in the modelling done by Imperial College, London, replicated by the team in Edinburgh, school closures increased the total number of deaths(1). "Reading it [Imperial's Report 9] carefully", did Ackland get as far as the discussion (2)? It states:
"Overall, our results suggest that population-wide social distancing applied to the population as a whole would have the largest impact; and in combination with other interventions - notably home isolation of cases and school and university closure - has the potential to suppress transmission below the threshold of R=1 required to rapidly reduce case incidence. A minimum policy for effective suppression is therefore population-wide social distancing combined with home isolation of cases and school and university closure" (p14, 15).......
"We therefore conclude that epidemic suppression is the only viable strategy at the current time"(p16)
Imperial's accompanying predictions, for an unmitigated epidemic, of 510,000 deaths in Great Britain and 2,2 million in the United States, make it clear that one of the main impulses for suppression policies, in the UK, the US and France, rather than "interventions.... which focus strongly on protecting the vulnerable(1)"(an approach favoured by many even then) came from the Imperial team themselves. This makes Ackland's conclusion that,
"With hindsight the Imperial Model has proved remarkably accurate. It turned out that the experts really are expert",
seem generous to the point of re-writing history.
Nevertheless, those scientists and politicians receiving the advice, in March, might well have shown a greater degree of scepticism for both technical and historical reasons (3,4). Far from it being known that, "Detailed models of individual interactions, which can take many hours of supercomputer time to run, are a reliable way to predict the course of an epidemic and investigate counterfactual scenarios"(5), the utility of this real time epidemic modelling, from Imperial and more generally has been, frequently, called into question. (6,7) Indeed it might be regarded as extraordinary, with at least six university groups in the UK, engaged in infectious disease modelling (Imperial, London School of Hygiene and Tropical Medicine, Oxford, Warwick, Edinburgh and Glasgow), all often better staffed and funded than the public health offices in the four UK nations that have to support investigation on the ground, that the Royal Society saw it as necessary to recruit yet more modelling groups, from other disciplines and with the public finances in free fall, that UK Research and Innovation saw fit to fund them.
What has been most informative in managing the epidemic in the UK, although it has attracted much less media attention, has been the empirical epidemiology, often making use of the digital storage of much clinical data, that has variously documented the risks for death in hospital(8), admission to critical care (9) and recording a positive SARS CoV 2 test(10). Questions remain, however, that require empirical data;
the incidence, severity and duration of "Long Covid",
risk factors for transmission in hospitals,
the role of various community facilities including schools in giving rise to cases
to name but some.
It is unfortunate, perhaps, that the the perception of epidemiologists and their work has moved, in the UK at least, from "disease detectives" to mathematicians. "Epidemiology can be regarded as a sequence of reasoning concerned with biological inferences derived from observations of disease occurrence and related phenomena in human population groups" (11). Mathematical modellers, invariably use others' observations and often lack lived experience of the limitations of those observations. This distance from the problems on the ground and the empirical data may go some way to explaining how the Edinburgh and Imperial teams, basically using the same data and code could, without acknowledging the fact, state different conclusions, with very different policy implications. It is a further indication that it is time, in the UK, to review the approach to the pandemic and the science on which it is based. .
Roland Salmon
1. Ackland GJ. Modelling the pandemic. BMJ 2020 October 7 https://blogs.bmj.com/bmj/2020/10/07/covid-19-modelling-the-pandemic/
2. Ferguson NM., Laydon D, Nedjati-Gilani G, Imai N, et al. Report 9:Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. https://doi.org/10.25561/77482
3. Salmon R. Les donnees pour soutenir la politique du confinement font defaut. Le Monde 2020 April 8
4. Salmon R. Modelling the epidemic (letter). London Review of Books 2020. June 4.
5. Rice K, Wynne B, Martin V, Ackland GJ. Effect of school closures on mortality from coronavirus disease 2019: old and new predictions. BMJ 2020;371:m3588 doi: https://doi.org/10.1136/bmj.m3588
6. Taylor N. Review of the use of models in informing disease control policy. A report for DEFRA. University of Reading 2003 May 26
7. Hine D. The 2009 Influenza Pandemic. An independent review of the UK response to the 2009 influenza pandemic. London 2010 July. The Cabinet Office
8. Williamson ES et al. OpenSAFELY: factors associated with COVID-19-related hospital deaths in the linked electronic health records of 17 million adult NHS patients. MedXRiv preprint doi: https://doi.org/10.1101/2020.05.06.20092999 this version posted May 7, 2020.
9. Docherty AB et al. Features of 20,133 patients in hospital with covid-19 using the ISARIC WHO Clinical Characteristics Protocol:prospective observational cohort study. BMJ 2020;369m1985|doi:10.1136/bmj.m1985
10. de Lusignan S et al. Risk factors for SARS-CoV-2 among patients in the Oxford Royal College of General Practitioners Research and Surveillance Centre primary care network: a cross sectional study. Lancet 2020.https://doi.org/10.1016/S1473-3099(20)30371-6.
11. Lilienfeld DE, Stolley PD. Foundations of Epidemiology. Oxford 1994. Oxford University Press
Competing interests: Medical Referee (Authority to Cremate) Cardiff Thornhill Crematorium, remunerated on a fee for service basis
Dear Editor,
In their excellent and timely paper Professor Rice and colleagues (1), have shown how vital it is to put all the right questions into any modelling system. The decision to close schools and universities in March may have been an error, but it is vital that we do not continue compounding this error as we enter the winter. In particular, universities and schools should not be closed every time new and often asymptomatic cases occur, but rather their contribution towards herd immunity seen as part of the solution.
Moreover, schools should return to the normal environment that is vital for children’s education and mental health. Analysis from PHE data from June and July showed very little evidence of transmission in schools (2). However, since September, the parent group UsforThem, campaigning to get schools back to a semblance of normality, has received numerous reports of increasingly restrictive measures in schools (3). Sport and music are still seriously restricted, science often taught with no practical lessons. Increasing numbers of primary schools are requiring parents to wear masks at drop off and pick up, giving a graphic message to children that they and their parents are a danger to each other, despite clear evidence to the contrary. We have reports of children being refused entry to school without a mask and those with exemptions being required to wear badges or lanyards to mark them out. Schools also vary in how they deal with a positive test result in a child, ranging from sending home just close contacts through to whole year groups.
Children have already suffered disproportionately through the lockdown (4). Surely now is the time to give them back their childhood.
1. Rice K, Wynne B, Martin V, Ackland G. Effect of school closures on mortality from coronavirus disease 2019: old and new predictions. BMJ 2020;371:m3588
2. Ismail SA, Saliba V, Lopez Bernal J, Ramsay ME, Ladhani SN. SARS-CoV-2 infection and transmission in educational settings: cross-sectional analysis of clusters and outbreaks in England. available from https://assets.publishing.service.gov.uk/government/uploads/system/uploa...
3. Kingsley M. Time for parents to stand against the scandalous treatment of schoolchildren. https://www.telegraph.co.uk/news/2020/10/04/time-parents-stand-against-s...
4. Creswell C, McCrory E, Viding E, Goodyer I. COVID-19 Lockdown: A Child and Adolescent Mental Health Crisis. https://reachwell.org/2020/07/09/covid-19-lockdown-a-child-and-adolescen...
Competing interests: No competing interests
Dear Editor
As a retired GP with no expertise in modelling I was struck by how this paper might mislead on superficial reading. My understanding is that the modelled second wave outcomes are based on all mitigating interventions (social distancing, etc) being stopped after 91 or 120 days, and not being reintroduced. This is implied but not explicitly stated. So the predicted second wave outcomes do not apply to the current situation where mitigations have been reintroduced.
To expert readers this is probably obvious but this paper has potential to cause confusion and I imagine the headline conclusion may be open to misinterpretation.
Richard James
Competing interests: No competing interests
Dear Editor,
You will no doubt be aware that this paper has been ceased upon by elements of the British press more concerned with selling opinions and newspapers than informing their readers. “Major study reveals Covid rules may INCREASE deaths,” screamed the Daily Mail. “Herd immunity ‘could have saved more lives than social distancing’,” states the Telegraph’s. By perpetuating the myth that herd immunity is a suitable response to this pandemic you have exacerbated the government’s failure to manage the pandemic and to clearly communicate the dangers of COVID-19 and the need to adhere to lockdown measures. It will now be even longer until my elderly grandfather may finally feel safe to leave his flat.
The paper ignores numerous unknowns about COVID-19. Has anyone proven that infection results in long-term immunity? Has anyone studied the long-term effects of COVID-19 infection on individuals’ health? Has anyone found a cure for “long-covid”? Has anyone excluded serious complications occurring years after infection (cf eg measles)? Has anyone been able to predict which “normally fit and well” people will die as a result of infection? Although Mr Trump may regard his infection as a blessing I suspect he may be in the minority of sufferers. Publishing research which is being used to advocate a herd immunity strategy while failing to acknowledge these serious potential harmful effects of COVID-19 infection is as negligent as consenting a patient to surgery without discussing any risks 70 million times over.
There are also many practical issues with the modelled course of action. For example, how can those most vulnerable to COVID-19 be shielded if they live with others, particularly multigenerational households? Or require domiciliary care? Or live in a care home? Or need to see a doctor? Or are admitted to hospital? Presumably anyone living with a vulnerable person or working as a carer or in a care home or medical facility would also need to be shielded? And what of their household members?
Furthermore, the model does not seem to account for the effect of a large surge in cases on the health service. During those months of unrestricted transmission amongst the population what medical care will be available? Given ITUs were close to being overrun during the lockdown, what would have happened to the additional patients needing intensive care if the virus had been allowed to spread more quickly? And what of those with other illnesses? This is particularly important as those currently advocating herd immunity are doing so as we enter winter when the NHS always becomes overwhelmed even without a pandemic.
Finally, the model does not acknowledge the greatest known unknown. Viruses mutate. We have no way of predicting when COVID-19 will mutate to become more contagious, more lethal or to overcome any existing immunity. Allowing the modelled unrestricted transmission would lead to much higher rates of virus replication and potential mutations.
Rather than put our confidence in computer models we should, as a caring and scientifically trained profession, acknowledge the limitations of our understanding, abilities and treatments. We must communicate the uncertainty and potential harms clearly and consistently to the population we claim to serve. Let us show an appropriate and necessary respect to the power of nature. We should publicly support the only strategy we can confidently state will prevent morbidity and mortality, that is suppression of the virus. And let us leave filling the media with fake news to the current President of the USA.
Competing interests: No competing interests
Dear Editor
The Imperial model's many limitations - e.g. as noted in Shen, Taleb, Bar-Yam in March 2020 (link below) are not referenced in this paper - a serious omission.
This paper seems to reinforce the dangerous polarisation of the present public debate between 'lockdown' vs 'herd immunity'. The debate ignores the strategy which has had proven success in S Korea, Taiwan, New Zealand, etc - namely, 'find, test, trace, isolate, support ' - without lockdown. The model is not able to simulate this kind of contact finding and isolating.
As Chen et al note 'Focusing on details but using incorrect assumptions, makes for bad policy advice. Where lives are at stake, it is essential for science to adhere to higher standards'
Chen Shen, Taleb, Bar-Yam
https://www.academia.edu/42242357/Review_of_Ferguson_et_al_Impact_of_non...
(Covid science discussion group
https://drive.google.com/file/d/1gaVwqDAHwhMzvq6GW45hahc6Ky7gIj8F/view?u...)
Competing interests: No competing interests
Re: Effect of school closures on mortality from coronavirus disease 2019: old and new predictions
Dear Editor,
We thank the contributors who have taken time to respond to our article wherein our objective was “To replicate and analyse the information available to UK policymakers when the lockdown decision was taken in March 2020 in the United Kingdom.” from the Imperial model. We are well aware of the range of reasonable and unreasonable criticisms aimed at the model, and the danger that the numerical precision with which results can be computed can suggest a misleading level of accuracy in the assumptions. It should not have required a supercomputer to recognise that a disease with around 1% mortality spreading unchecked in a population of 70 million would result in hundreds of thousands of deaths in a short timespan.
Jackson notes that the model used in “Report 9” does not explicitly include care homes and prisons as specific places. These settings have proved critical in the high fatality rates, and are now incorporated in Imperial’s model. Crucial though care homes proved in the wider context, they are not central to the more limited scope of our paper which was to investigate what information was available from the model at the time of Report 9.
We discussed at great length which code version and which parameterization to use to implement the model. Thanks to Eglen’s external validation, the equivalence of the original Imperial code and the CovidSim rewrite was not at issue. As more data became available, Imperial included better parameter estimates. Since our study was about what was known in March, we decided to use the model and parameters as used for policy in March, and to not include new features like care homes. We stated that the original code was "not available" - we accept that we could have obtained it via an FOI request: had we done so none of the conclusions of the paper would be different.
Eglen S. CODECHECK certificate 2020-010; 2020. https://zenodo.org/record/3865491#.XuPW_y-ZPGI.
A broader set of test scenarios than presented in our paper had already been carried out during CovidSim rewrite, as documented by those authors. This was not our work, so we simply provided a reference.
Ferguson NM, Gilani GLN, Laydon DJ, et al. COVID-19 CovidSim Model. GitHub. 2020. https://github.com/mrc-ide/covid-sim.
Jackson’s comments raise some interesting questions about how science should proceed during an epidemic. Arguably, our paper is simply a detailed peer review of the results of Report 9, which was not possible on the timescale of the epidemic, and even then we did not attempt to critique the model. We can assume that other groups on SPI-M and SAGE provided this critique before using the model for policy.
The letter from Dr. James highlights that, following report 9, we stop interventions after 91/120 days and did not include mitigating interventions for the second wave: . This is an important point. Of course, we had no idea what interventions would be in place – indeed at the time of the study we didn’t even know when first-wave interventions would be lifted. Furthermore, we could not anticipate the effects of lockdown fatigue, arrival of vaccines and new variants. However, we could (and did) run many speculative scenarios. Some results, notably speed of onset of second wave, the peak ICU demand and the timescales for the epidemic, are very sensitive to such details. For that reason we did not present those results as conclusions of the paper. Careful readers of Report 9 may even notice that in some scenarios the general lockdown itself leads to more long-term deaths, but this outcome is highly dependent on details of later interventions. The conclusion we emphasised about school closures is robust, as was the conclusion that a second wave was predicted in all simulations where interventions are applied. The second wave was upon us when the paper was published, but it was still a matter of debate when we submitted the paper.
Report 9 states that “suppression is the only viable strategy”, but suppression here does not mean elimination: in all Report 9 scenarios, second and subsequent waves result in hundreds of thousands of deaths. They did not report calculations where interventions were in place until infection was eliminated, so we reinvestigated this. We found that a 211 day initial lockdown was sometimes sufficient to avoid a second wave, but even 300 days gave no guarantee. These numbers should not be taken literally - they come from a model with now outdated parameters - but they illustrate how challenging a suppression to elimination strategy would have been.
Very early in the epidemic Dr Salmon drew attention to the crucial lack of public data, https://www.bmj.com/content/368/bmj.m918/rr. This affects clinicians and modellers alike, and he notes recent studies where this has improved. We have no control over headline writers at the Mail and Telegraph, but in speaking with the press we emphasized that we did not, do not, and will not offer any policy advice. He rightly points out that epidemiology is only one aspect that guides policy.
Dr. Jones advances further reasons why school closures are damaging for children. We recognise that these factors are indeed important to consider. However, they do fall outwith the model and, as Jones is a paediatrician, we are confident she understands these issues better than we do.
Broadbent stated that the Imperial model cannot describe Track and Trace. This is not quite correct: the model incorporates “Case Isolation” as one of the interventions. Track and Tracing which does not lead to self-isolation can have no effect of spread – what matters is the subsequent isolation. Thus, epidemiologically, Track and Trace is a method of improving case isolation and is included via the effectiveness of that intervention. He further claims that the New Zealand, Taiwan and Korea followed a policy of “find test trace isolate support – without lockdown”. In fact, compared with UK, far more stringent lockdowns, strict quarantine and travel bans were implemented in New Zealand (https://www.thelancet.com/journals/lanpub/article/PIIS2468-2667(20)30237-1/fulltext ) and in Daegu, the location of the majority of Korea’s cases (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7592145/). Early intervention, strict quarantine and travel bans in Taiwan kept infections below 30 per day, a level at which their Track and Trace system was able to cope, but irrelevant to the UK situation in March. The effectiveness of the UK system was illustrated in a “natural experiment” (Fetzer https://www.medrxiv.org/content/10.1101/2020.12.10.20247080v1.full) comparing the differential outcomes in areas where data was “lost” in a spreadsheet error: analysis of their data suggests that the current system reduces transmission by some 30% - significant but not enough to prevent an infection. Most importantly, the UK system did not exist in March.
Dr Davis notes how the paper attracted press attention from people advocating certain policy positions contrary to his own. School closure is only one of many facets which policymakers must weigh up, and we were careful in all press interviews to be clear that we did not and do not advocate any policy position. It was chastening to see this message never reported in print.
The biggest uncertainty in our study was when and how the vaccine would appear. Report 9 suggested that even the most effective non-vaccination interventions would not limit UK deaths below 200,000. It now appears that safe and effective vaccines were already developed in March, but probably more than 2 million people died before they were deployed. As we appear to be heading towards vaccine-enhanced herd immunity it is interesting to speculate how history will view the COVID epidemic. We are confident that your readers will have a broad range of opinions on this matter for many years to come.
Competing interests: No competing interests