SAGE uses a set of assumptions called the “reasonable worst-case scenario” in its pandemic planning. In this post, Jonathan Birch looks at the group’s minutes and documents from early 2020 and argues that over-reliance on these assumptions led to costly delays.
What unfolded in the UK in the spring of 2020 was a national tragedy within the global tragedy of COVID-19. The course the epidemic took was not inevitable: it was affected by policy choices made in a state of severe uncertainty. Can we learn from the way those decisions were made to develop better norms for scientific advising in extremis?
The advisory process in the UK has been impressively transparent. In particular, SAGE (the Scientific Advisory Group for Emergencies) has published minutes and other documents soon after its meetings. This gives us a rich set of resources on which to draw. In a draft paper, I discuss several issues arising from these minutes. Here, I want to zoom in on one specific issue: the role played by “reasonable worst-case scenarios” in strategic planning.
The reasonable worst-case scenario
The “reasonable worst-case scenario” (RWCS) has been at the core of SAGE’s approach to the pandemic from the beginning. An RWCS is a set of assumptions that reflect one way in which the epidemic in the UK may unfold. The set of assumptions is “reasonable” in the sense of being regarded by SAGE as a serious possibility. It is “worst-case” in the sense of being at the pessimistic end of the range of serious possibilities.
The implicit principle behind the use of RWCSs seems to be this: if you assume you are in the RWCS, and plan accordingly, then you will be as well prepared as possible for less severe scenarios.
What was the RWCS that guided the UK’s initial response to COVID-19? It was, in some respects, truly grim. SAGE’s RWCS planning assumptions, dated 6 March, set out a scenario in which 80% of the population gets COVID-19 over a period of about nine weeks, with 50% displaying symptoms, and an infection fatality rate (IFR) of 1%. The result would have been around 520,000 excess deaths within three months. In an earlier draft dated 4 March, there is an explicit assumption that no effective treatments or vaccines will become available either before or during the epidemic. This line is deleted from the version dated 6 March, but seems to tacitly guide strategic planning.
SAGE also made some background assumptions that, while not formally part of the RWCS, are important for understanding it. First, SAGE assumed that contact tracing would cease to be effective once there were more than 50 cases per week. Second, SAGE appears to have assumed that, as a realistic maximum, interventions (such as social distancing and shielding) could be sustained for 13 weeks. Some modelling was done of a scenario in which they are sustained for 26 weeks, but 13 weeks is the assumption in the summary papers produced by SAGE. Third, SAGE assumed partial compliance. For example, it was assumed that 50% will comply with household quarantine. This is described in a 3 March paper as “high levels of compliance”.
Yet in other respects, the RWCS assumptions are excessively optimistic. SAGE (in the 4 March draft) assumes that R0, the basic reproductive rate of the virus in the absence of mitigation, is 2.4, leading to a doubling time of 4-6 days. Estimates varied a great deal at the time, and still do, but this was, even then, towards the lower end of serious estimates for R0.
A study published in The Lancet on 31 January had estimated R0 at 2.7. On 11 February, researchers at the Theoretical Biology and Biophysics unit at the Los Alamos National Laboratory, USA, released a preprint estimating that R0 was between 4.7 and 6.6.
There was another optimistic background assumption, introduced on 25 February, that surveillance “should provide evidence of an epidemic around 9-11 weeks before its peak”. Just as R0 being higher than expected was not part of the RWCS, surveillance being poorer than expected was also not part of the RWCS. In short, the planning assumptions at this time were a mix of bleak pessimism and excessive optimism.
Consequences
What were the consequences of these choices? The costly delays between 2 March and 23 March, when cumulative infections in England surged from about 13,500 to about 1.6m (to use the latest MRC-BSU estimates), can be seen as consequences of planning assumptions made around the beginning of this period. The combination of an optimistic estimate for R0, optimism about surveillance, and an assumption that 13 weeks was a realistic maximum duration for any intervention, made it seem optimal to delay the introduction of social-behavioural interventions that were in fact urgently needed.
This problem was compounded by continued scepticism, up to 16 March, about the idea of maximally aggressive suppression. For a time, SAGE’s advice was explicitly against maximally aggressive suppression and in favour of more moderate measures. There was a lot of talk in the media, in early March, about “herd immunity”. SAGE was perceived by its critics to be following a “herd immunity strategy”. The Chief Scientific Adviser (Sir Patrick Vallance) vehemently denied that there was any such strategy in private emails subsequently released to the BBC.
What was really going on? The reality is clear from the documents released by SAGE. Up to 9 March, there was no explicitly recommended strategy. SAGE presented various different options, without formally endorsing one. The recommended strategy between 9 and 16 March was a “high transmission reduction strategy” in the sense of Figure 1: a strategy which aimed to flatten the curve without suppressing it completely. In the week of 16 March, the recommendation changed abruptly to one of maximally aggressive suppression, including school closures.
Figure 1: A sketch that features prominently in a paper dated 9 March, intended to highlight the superiority of “high transmission reduction” to “very high transmission reduction, later lifted”.
This initially sceptical attitude towards maximally aggressive suppression can be traced back to the RWCS planning assumptions. In a reasonable worst-case, no effective treatment, contact tracing system or vaccine becomes available before measures are relaxed, and no long-term behavioural changes are instilled, so total suppression leads robustly to the epidemic returning with unmitigated force in the autumn, infecting 80% of the population and overwhelming the health service. This corresponds to the green line in Figure 1. In that bleak scenario, we come to bitterly regret the aggressive measures adopted in the spring.
In the actual world, by contrast, we have been left bitterly regretting our failure to take these measures soon enough. If SAGE had unambiguously recommended maximally aggressive suppression (including school closures) on 2 March, when its SPI-M subcommittee first reported that “It is highly likely that there is sustained transmission of COVID-19 in the UK at present”, then many lives would have been saved and the epidemic would have taken a different course.
Reflections
Let’s return to the implicit principle guiding the use of RWCSs: if you assume you are in the RWCS, and plan accordingly, then you will be as well prepared as possible for less severe scenarios. This case, I suggest, shows us some important exceptions to this principle.
The principle is not true if your RWCS is pessimistic in most relevant respects but not all of them. I will introduce the term “globally pessimistic” for a scenario that is pessimistic in all relevant respects. If the RWCS is not globally pessimistic (e.g. its estimate for R0 is too optimistic), then there is a serious risk that reality will be worse than the RWCS in some respects. This is what happened in relation to R, which was almost certainly above 2.4 before the spring lockdown in England, according to more recent MRC-BSU modelling, despite substantial attempts at mitigation.
Of course, a globally pessimistic scenario may well seem unreasonable. It is tempting to think: it is very unlikely that we will be unlucky in all respects! This brings out a tension in the concept of an RWCS. To avoid reality catching you out, the RWCS must be globally pessimistic, and may therefore look unreasonable when viewed as a whole. But what matters is that each individual assumption represents a serious possibility.
More subtly, there can be circumstances in which assuming you are in the RWCS justifies actions, delays, or omissions that will impair your response significantly if you are in a less severe scenario.
For example, it makes sense to say that, in a reasonable worst case, no effective treatment or vaccine will become available, contact tracing will never become effective, and no long-term behavioural changes will be instilled, even if you delay the epidemic by several months. In this case, maximally aggressive suppression of transmission is likely to make things worse in the long run, as depicted in Figure 1.
But suppose you are in a less severe scenario, in which one or more of these pessimistic assumptions is false. In such a scenario, maximally aggressive suppression is likely to be far superior to more moderate action, in terms of both its public health consequences and its long-term economic consequences. If you plan for the RWCS, and are thereby led to adopt a strategy that involves aiming for the brown/orange line in Figure 1, you will not be as well prepared as possible for a less severe scenario.
Proposals
This leads me to some proposals concerning the use of RWCSs in scientific advice. First: RWCSs, if used in any form of planning, should be globally pessimistic, which is to say at the pessimistic end of scientific opinion in all potentially relevant respects, not just some.
Second, although they are a useful guide for day-to-day operational planning (e.g. how many pieces of PPE to purchase), RWCSs should not dominate strategic planning. For strategic planning, it is important to consider a wide range of possible scenarios, including but not limited to the reasonable worst case (which, if globally pessimistic, will be unlikely).
The apparent inevitability of a large wave as soon as measures were relaxed was sensitive to a specific set of worst-case planning assumptions, which assumed that measures could not be sustained until an effective treatment, vaccine, or contact tracing system was implemented. These assumptions dominated strategic planning up to 16 March.
Yet as soon as modellers dared to relax one of those pessimistic assumptions, the strategic picture suddenly changed. A paper by modellers at Imperial College, dated 16 March, considered the possibility of maintaining maximally aggressive suppression measures (including long periods of school closure) indefinitely, until a vaccine is achieved (Figure 2). The paper concludes, dramatically, that aggressive suppression is “the only viable strategy at the current time”. This paper produced a swift change in the direction of the UK’s response.
Figure 2: A graph from Ferguson et al. (2020), in which aggressive suppression measures are maintained indefinitely. The blue-outlined blocks represent periods of school closure.
This leads me to another proposal: scientific advisers should highlight, as part of their advice, the circumstances under which their recommended actions might lead to serious regret. When SAGE recommended a “high transmission reduction” strategy over maximally aggressive suppression on 9 March, it should have highlighted the potential for this strategy to lead to serious regret – in the form of tens of thousands of excess deaths which could, in retrospect, have been prevented by pursuing a more aggressive suppression strategy at an earlier stage.
These three proposals (make the RWCS globally pessimistic, do not allow it to dominate strategic planning, and communicate the circumstances under which recommendations arising from it may lead to serious regret) are not a recipe for effective scientific advising. A lot can still go wrong. But I do think they would have helped in this case – and might help in future crises on a similar scale.
Part 2 →
Dr Jonathan Birch is an Associate Professor of Philosophy at the LSE and Principal Investigator (PI) on the Foundations of Animal Sentience project. In addition to his interest in animal sentience, cognition and welfare, he also has a longstanding interest in the evolution of altruism and social behaviour.
Jonathan
Very interesting argument about the RWCS. I’d like to share a few thoughts on the pandemic and its handling in the UK.
What has always been so troubling to me is that China had already shown that there was a way to entirely suppress and move to eliminate the virus. This at a time when it was assumed such measures must be impossible in the UK (and elsewhere in Europe too). China was swiftly followed by other territories – Taiwan, and then South Korea – who showed a determination to suppress and eliminate the virus in a more open society. It is interesting to note that the East Asian countries had had experience of SARS and never ever contemplated living with the new SARS-CoV-2 virus. I still find it incredible that scientists and politicians in the UK and elsewhere thought the experience and strategies of East Asia were not applicable.
I well remember my feeling of horror and amazement when I saw Johnson interviewed (I think by Andrew Marr) and he said – we think that others have got it a bit wrong… there is an argument for letting it ‘run through the population’. I shuddered, because China had already shown that the virus could be controlled and more or less eliminated – as had happened with the first SARS virus.
It seemed an extraordinary kind of exceptionalism and arrogance, to think that our scientists and politicians knew better, that the British population would never accept restrictions, that borders could never be controlled. The myth developed that there was some trade off between ‘the economy’ and ‘health’, whereas what has been shown is that these are completely linked aims – what is good for health is good for the economy. Failing to prioritise health led to the notion that the epidemic could be managed – ‘flattening the curve’. The only bar the UK ever seemed to set as a goal was ‘do not overwhelm the health service’. This explains the evident assumption in the later Imperial model – Figure 2, that the virus would not actually be eliminated, merely ‘controlled’ through a series of lockdowns or at least school closures.
It is clear that some scientists desired an epidemic – Graham Medley made this explicit – ‘a nice big epidemic’. It is also clear that the ‘herd immunity’ scientists had the ear of the government – which is the implication of Johnson’s letting it ‘run through the population’.
The assumption behind that must have been that reinfections were not possible, or if they were, they must be mild. Neither of which has been shown to be the case.
The UK ended up, tragically, adopting the influenza pandemic model, based upon the wrong assumption (already proved wrong by China) that the virus could not be stamped out, that test, track and trace was hopeless and not worth continuing once ‘community transmission’ was established. It was the wrong model. Australia originally followed the UK / Europe model but realised their mistake and then pivoted to the East Asian model.
It is such a monumental tragedy.
Thanks Adrian! I’m sorry your comment got lost in the filter for a while. One comment: SAGE’s view in mid-February, based on what Public Health England was telling them, was that contact tracing would simply become impossible once there were more than 50 cases per week. They were ready for 5 cases per week, and thought they could scale up to 50 per week – but that was the limit.
We can see from other countries that it is certainly possible to handle hundreds of cases per day, but whether we could achieved that kind of capacity in time for the “first wave”, had we tried, is another matter.
The sad situation now is that although we do have a substantial, very expensive test-and-trace system, we have never had infection levels low enough to give it a realistic shot at working effectively – infection levels have never reached mere hundreds per day. “Monumental tragedy” is a fair comment.
Very nice post, and your advice does make sense on the whole. I wonder though if you are failing to give the SAGE enough credit regarding their assumptions. Please correct me if I am wrong, but I have not heard of any cases *in a non-Asian country* where test-and-trace has proven useful with hundreds of cases per day. And that *maximal* suppression cannot be maintained more than 13 weeks is also looking like a pretty good estimate – no country has done it longer, as far as I know, (though maybe Victoria in Australia has). The low R0 and low compliance estimates, however, do look like poor choices with hindsight.
Thanks for the comment! I think some interventions have proved sustainable far beyond 13 weeks – especially social distancing and mask-wearing. With full lockdowns, people’s mental health is probably the limiting factor. We have had lockdown since early November, apart from a disastrous break in December, and I think it has tested everyone’s endurance. Could test-and-trace have worked effectively in this country if cases had ever been in the hundreds per day? That’s just a great unknown – we had the opportunity to try last summer and declined the opportunity.