Less likely than thought

People should remember the claimed accuracy rates of medical tests do not convey the true probability of disease when results are positive.

Name four creatures beginning with the letter "S". What is 4 plus 15 minus 17? Why is a carrot like a potato? If these questions cause you trouble, don't worry - chances are you're just having an off day. Certainly no one in the prime of life should lose sleep over the fact that they're part of a new test for Alzheimer's disease, the most common form of dementia.

But what if you're not as young as you once were? With some media organisations putting the entire test online and claiming it is 93 per cent accurate, it is tempting to try it out or give it to an elderly relative you are concerned about. Yet these reports should carry a big health warning: don't try this at home. The test may be quick and simple, but it is also potentially highly misleading. That is not because the researchers behind the test have exaggerated its reliability: the problem lies in a phenomenon which often wrong-foots even experts.

At its heart is a mathematical result called Bayes's Theorem, which shows how to update our beliefs in the light of new evidence. While the formula itself is somewhat involved, its implications are straightforward: be wary of vague talk about "accuracy". In the case of the new test for Alzheimer's, the media reports make much of an "accuracy rate" of 93 per cent. But what, exactly, does that mean? Reporting the research behind their new test in the current issue of the British Medical Journal, the neurologist Jeremy Brown and his colleagues at Addenbrooke's Hospital, Cambridge, UK, are clear: the 93 per cent is the percentage of patients correctly identified by the test as having Alzheimer's disease - or, in the jargon, the "true positive" rate

But Bayes's Theorem shows this figure alone does not capture the reliability of the test: after all, it might be effective at detecting the disease, but only at the expense of falsely detecting Alzheimer's in many healthy patients. We therefore also need the "false positive" rate - that is, how often it leads to the wrong diagnosis. Again, Dr Brown and his colleagues have this covered, and state that their test wrongly diagnosed Alzheimer's in 14 per cent of healthy patients. Ignored in media reports, this figure is vital in gauging the test's effectiveness. Bayes's Theorem does this via the so-called "Likelihood Ratio" (LR) - the true positive rate divided by the false positive rate, which for the Addenbrooke's test is 93/14, or around 7. Thus whatever the original level of belief about a patient having Alzheimer's, a positive result from the test means that belief should increase seven-fold. Levels of belief can be captured via the odds we would give on their proving correct. So, for example, if we initially thought there was a one per cent chance of a patient having Alzheimer's, a positive result from the test would increase those chances to around seven per cent.

That's far short of the 93 per cent chance implied by that "accuracy" figure quoted in the media. But as Bayes's Theorem shows, that is because the prior probability that the person had Alzheimer's was already very low, and not even the seven-fold increase in weight of evidence supplied by the new test can boost that to a high probability. And that is pretty crucial for anyone under the age of 80 who takes the test. While Alzheimer's disease becomes more prevalent with age, even septuagenarians face only around a 1 in 20 chance of developing it. Bayes's Theorem then shows that even the "93 per cent accurate" Addenbrooke's test is more likely to be wrong than right among septuagenarians - because of the low prevalence of Alzheimer's within their age group.

The rule of thumb is this: a positive result from any test is most likely to be wrong if its false positive rate exceeds the prevalence of whatever is being detected. Does this mean the Addenbrooke's test is useless? Not at all. With its false positive rate of 14 per cent, the rule of thumb means a positive result is likely to be correct for anyone over the age of 80 - as the prevalence of Alzheimer's in this age group exceeds 20 per cent. Even among younger age groups, the test will be helpful in screening prior to further tests. This is how Dr Brown and his colleagues want to see it used - and in this they are backed by Bayes's Theorem, which shows that further tests rapidly boost the weight of evidence.

It may come as little surprise that the Addenbrooke's team has a better grasp of Bayes's Theorem than the media - but perhaps it should. Research by Professor Gerd Gigerenzer and his colleagues at the Max Planck Institute, Berlin, has shown that many medics struggle with its implications. In one study, a group of doctors was asked to estimate the chances that a patient really does have breast cancer - affecting one per cent of women - given a positive result from a test with a true positive rate of 80 per cent, and a false alarm rate of 10 per cent.

Most of the doctors put the chances as at least 70 per cent. Yet just knowing the rule of thumb shows this can't be right. With a false alarm rate 10 times greater than the prevalence of the disease, the positive result is far more likely to be wrong than right. The correct answer, found by just two doctors - who presumably knew something about Bayes - is 8 per cent. With home diagnostic kits for everything from allergies to cancer now coming on the market, doctors should brace themselves for increasing numbers of distraught but perfectly healthy patients - and mug up the basics of Bayes.

Robert Matthews is a Visiting Reader in Science at Aston University, Birmingham, England