We've been focusing on all-cause mortality as the key indicator of vaccine safety and effectiveness: if, for a particular age group over a reasonable period of time, the number of all-cause deaths (i.e. covid and non-covid) are lower in the vaccinated than the unvaccinated then the benefits of the vaccine outweigh the risks for that age group.
As a hypothetical example, imagine 20,000 people with Covid, 10,000 of whom are vaccinated and 10,000 are not. After 6 months we find the following:
Then the all cause mortality of the vaccinated is 400 (40 per 1000), while the all-cause mortality of the unvaccinated is 480 (48 per 1000).
So the vaccine benefits outweigh the risks: it saves more people (100) from Covid than it kills through adverse reactions (20)
But we have been missing something critical. For people infected with covid, the vaccine is not the only relevant medical intervention. There is very strong evidence that early treatment protocols are effective in reducing covid deaths - and they are extremely safe. An infected person (vaccinated or not) may or may not get early treatment. So, in comparing all-cause mortality of the vaccinated against the unvaccinated we must condition the analysis on whether early treatment was provided.
Because we know that only a small minority of vaccinated people have received early treatment, the observational data like that above might show that vaccine benefits out weigh the risks, but it may be hiding the fact that early treatment is far more effective than the vaccine. Suppose that 1000 of the 10,000 in each group got early treatment and that the full data are as follows:
Then in this hypothetical example, despite the aggregate data showing the vaccine benefits outweigh the risks, the mortality rate for those unvaccinated but who had early treatment is lower (31) than the vaccinated who did not have early treatment (41), while those who were vaccinated and got early treatment (31) do no better than those who were unvaccinated and got early treatment (31). This is an example of Berkson's paradox.
So, while the vaccine benefits clearly outweigh the risks in this hypothetical example, given the choice between early vaccination and early treatment, the latter is clearly the better option.
Of course, as we have previously made clear, any risk-benefit analysis should be undertaken separately for different age groups and any rigorous data analysis must take account of the following causal structure of the problem (additional risk factors such as comorbidities should also ideally be added):
This is an example of a Bayesian network and in a future part of this post I will show the model in action. Berkson's paradox can also be demonstrated in the model by adding a node called 'in sample' whose parents are the nodes 'Early treatment?' and 'vaccinated?':
The key point is that when 'in sample' is set to True - whereby we replicate the sample bias in which more people who are vaccinated than received early treatment are in the sample - we observe the aggregated results.