In his book The Black Swan, Nassim Nicholas Taleb presents the “ludic fallacy”: the mistake of applying unvarnished theoretical probabilities to real-world scenarios.
Here is his example:
You have a fair coin which, on the last 99 flips, had come up “heads”. Assuming it is a fair coin, what is the probability of it coming up “heads” on the 100th flip?
Professor of statistics: Easy: 50%.1
Professional gambler: Come on, chump, it is 100%. What are the odds of it being a fair coin?
We are looking at a highly unusual outcome. The odds of getting it with a fair coin are something like a googol to one against.2
That means this did not happen by chance. We can rule out the “null hypothesis”, therefore. It can’t be a truly random event: either the coin is loaded, or the flipper is somehow rigging the game.
This is an effective logical argument:
If:
The coin is fair, and
The flipping method is fair, then
The coin will not land heads 100 times in a row.Therefore, if the coin does land heads 100 times in a row, then either:
The coin is not fair, or
The flipping method is not fair.
So, we can try to get to the bottom of this.
We can ask to inspect the coin. We can test it out for ourselves. If, on a thorough inspection: its weight, shape, balance, density, magnetism and so on seem fair, and if we flip it a few times for ourselves and it yields expected something like the expected 50/50 outcome, we can rule out the coin as the problem.
We can ask to inspect the flipping process. We will need to be wary: an experienced hustler may be skilled at conjury. She may engineer “heads” each time through misdirection and sleight of hand. Especially if the flipping happens “off-stage”. If we can’t see the coin being flipped, we don’t know how it may have been edited or fixed to omit all the “tails” outcomes, or force the heads result.
In any case, if we are not allowed to inspect the coin, or we are not allowed to observe the flipping method, alarms should sound immediately. This is what a conjurer would do. We should not let ourselves be misdirected here.
Ludic fallacy applied
We have the potential for a “ludic fallacy” in Ms. Letby’s case. We know the probability of a nurse being rostered on for a given shift. Depending on how hard she works, it is between one-in-five and one-in-three. Here, there were 25 “events” — collapses for which she was charged. She was on duty for all of them.
The chances of the same nurse being present at 25 collapses in a row is a bit like flipping 100 heads in a row: the odds against it are not quite a googol to one, but they are still so vanishingly small — in the billions to one against — as to be impossible.
So we can reframe our logical argument:
If:
Ms. Letby is innocent, and
The sampling method for the collapses is fair, then
Ms. Letby will not be on duty for 25 collapses in a row.Therefore, if Ms. Letby was on duty for 25 collapses in a row, then either:
She is not innocent, or
The sampling method for the collapses was not fair.
The “null hypothesis” — that the collapses were caused by unrelated natural causes and Ms. Letby’s presence was a pure coincidence — is off the table.
We have two things to consider: firstly, what are the “base rate” probabilities for these alternative explanations, and secondly, what evidence do we have to support the alternative explanations that might change those?
Base rates
What is more likely? That someone, perhaps inadvertently, cherry-picked some statistics, or that a nurse tried to murder 25 premature infants?
I don’t have hard data, but I hope you won’t find the suggestion that people being a bit casual with information is a bit more likely than nurse serial murderer, if you had to choose.
Here, we do have to choose. We should be prepared to change our minds if the evidence requires it, but until we have a good reason, we should assume an unusual coincidence of collapses is more likely to be sampling bias than serial murder. Because, all else being equal, sampling bias is common, and serial murder isn’t.
So is all else equal? There are things we can check.
We can observe the nurse: her activities, her logs, hospital monitor outputs, swipe card data, recollections of other nurses, her behaviour, her social media, her apparent psychiatric state — is there anything that suggests malice? Did anyone see her attack an infant? Did she stockpile insulin? Was she spotted fiddling with feedbags? Did she Google “venous air embolus” or “killing an infant with milk”?
We can also inspect the collapses. Were these patients unusually susceptible to collapse? How, generally, was the hospital performing? Were there other collapses? Were there other deaths? How thorough was the process of going through the hospital records over the 15-month suspect period to find other suspicious episodes? Have any collapses been ruled out? Can we have a look at those?
And we can inspect the charge selection process. There ought to be a lot of data about that. Police, CPS and the hospital will all have copious records of their discussions, conversations and what investigations they took. There will be post-mortems and tests. We can see how the suspicion arose, how the suspicious events were tested, what has been filtered out and on what criteria.
In this case, the “coin” is Ms. Letby. The “master magician” flipping it is the Crown Prosecution Service and its beautiful assistant, the Cheshire Police. How much information do we have about each, and what does it tell us?
The nurse
The nurse has been working at COCH for approximately 8 years full time, she is a Cheshire resident, and a single parent. The staff member has since placed a grievance against COCH. There has been no formal investigation of misconduct and no motive identified. There are no mental health issues known and nothing has been highlighted by occupational health. There are no management issues.3
If a coin is loaded, you can often tell immediately. You do not always need to flip it to know. It will be misshapen, shaved, magnetised or counterweighted. At the limit, you can flip it for yourself and see how it performs. The key is to be able to freely and fully inspect it.
If a nurse is repeatedly murdering patients in an intensive care unit, likewise, you should be able to tell. You should not need to look at a roster of attendance to work this out. The key is to be able to freely, fully inspect her behaviour.
So, have we had an opportunity to freely test the nurse?
Yes.
Ms. Letby was under suspicion for the best part of a decade. She stood trial for ten months. She took the stand for fourteen days, much of it under the glare of withering cross-examination. Her every movement has been the subject of deep-sea dives by an army of keyboard warriors: your correspondent is but one.
What behaviour would we look for? We might expect to catch her in the act, or in possession of medicine or instruments she cannot explain. We might expect incriminating CCTV footage. She may be overheard saying incriminating things. We might expect some documented mental illness, psychiatric issues or some criminal propensity in her background. We might expect her immediate colleagues to harbour suspicions about her. We might expect them, at the very least, to be uneasy around her.
What have we seen?
In the round, a striking lack of the characteristics you would expect of someone committing serial murder. No direct evidence. No possession of incriminating implements. No CCTV footage. No history of mental illness, emotional abuse or criminal propensity. And her immediate colleagues, to a person, attest they had no concerns about her behaviour. Even the hospital consultants, in their first meeting with the police, conceded as much:
As part of the review staffing was looked at, there was a notable high statistical relationship between a member of the nursing staff and babies deteriorating in the unit. There is no evidence, other than coincidence.4
Now, it is not necessarily fatal if there are none, or few, of these things — one thing a master criminal must do is camouflage her incriminating traits — but it makes our base rate case — “the identified nurse is a serial murderer” — even more improbable.
The charge selection method
[...] This is after all the basis of our concerns and I think for the police to have their interest piqued we need to have that with all of the cases (it’s not going to be in the info Ian H/Stephen C have given to the police). I have attempted to do this as belwo [sic] for the ones I was involved with but hopefully more In a stating the facts way than a subjective finger pointing way.
If everyone is in agreement we should all do something similar for the ones we know about. Time is of the essence, I would like to get this to Superintendent Wenham by the middle of next week at the latest.
Ravi
—Email from Ravi Jayaram to Consultants at the Countess of Chester Hospital, 4 May 20175
What about the Crown’s charge selection method, then? We start by estimating the chance of a nurse being on duty for a given shift. Let’s give Ms. Letby the benefit of the doubt and say it was ⅓. It is from here that the improbability of Ms. Letby’s coincidental presence flows — multiply ⅓ by itself to get the cumulative probability for each successive collapse. By the time you have done this 25 times, it is a very small number indeed.
But only if you selected collapses and without reference to Ms. Letby.
If your “sample space” of shifts from which you are selecting suspicious events comprises only shifts on which Ms. Letby was working, her chance of being on duty for a given shift is not ⅓, but 100%. You can multiply 100% by itself as many times as you like, and you still get 100%.
So it is important to be sure the sampling was done in a way that did not pre-judge Ms. Letby. If the sample excluded collapses or deaths where she was not present, then that, too, is problematic: this is like your coin hustler omitting all the times he flipped his coin and got tails.
Now, no one is suggesting the Cheshire police are hustlers. But we should be able to see their sampling methodology to see whether it has properly, randomly sampled the available data. This is how we check that we are not being misdirected, even inadvertently.
This information should be freely available. Nor should the Crown have any reason to withhold it.
But they did withhold it. The Crown declined to share that information. Mr. Myers asked for it, and was denied it. Mr. Myers applied to the court for an order that the Crown share it, and the court declined.
We can assemble some of the surrounding context from court transcripts and matters inadvertently disclosed to the Thirlwall Enquiry. From the outset, the investigation was heavily focussed on the concern that there was a malicious nurse. If there was not, there was no need to involve the Police. Ms. Letby was named in the first meeting with police, well before any decision was made to engage experts or proceed with an investigation.
It seems that the Crown Prosecution did not realise the significance of this confirmation bias: by way of response to Mr. Myers’ application for disclosure, Mr. Johnson admits it in a sort of vaguely double negative way: he cannot apprehend how collapses with which Ms. Letby wasn’t being charged could possibly be relevant to collapses for which she was:
...it would be fairly odd if in investigating what else she may or may not have been up to, they weren’t looking at cases where she was there.
Of course we were only looking at Ms. Letby’s shifts, in other words. And the Court accepted it and upheld the Crown’s refusal to disclose it as irrelevant:
4. [...] Whilst the information sought, or the basis for further police inquiries into the defendant may be of great interest to the defendant herself, they are irrelevant to the issues in this case. So there is no material to disclose, there being no material that “might reasonably be considered capable of undermining the case for the prosecution against the accused or of assisting the case for the accused ... that has not been disclosed to the accused” (CPIA s. 7A). Inevitably, the investigators were looking at cases where the defendant was there, but that has nothing to do with disclosure; [...].
So, back to our logical calculation. It can’t be chance: either Ms. Letby is guilty, or the Crown’s statistics are lousy.
On a simple comparison of base rates, we should start by presuming “lousy statistics are more likely than a serial murderer nurse”. We have a lot of information about the nurse which we can use to update that assessment. None of it strongly suggests she is a serial murderer. We have a lot less information about the Crown’s statistical methodology, which it has refused to disclose, but what information we do have suggests it was subject to material sampling bias.
This makes things look a lot less suspicious.
We are back to our base rate comparison: what is more likely? Lousy statistics, or a serial murdering nurse?
See also
References
This is an outrageous straw man: only someone without much grasp of advanced probabilities — Bayesian reasoning in particular — would say such a thing.
A googol is 10100, or 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000. The odds here of flipping heads 100 times in a row are 0.5 * 10100, so technically one in half a googol.
Cheshire Police briefing note of meeting with COCH, 5 May 2017. The reference to Ms. Letby being a single parent appears to be an error.
Cheshire Police briefing note of meeting with COCH, 5 May 2017.
Not disclosed to the defence during either trial. Disclosed as part of the appeal on the second trial.
Share this post