Statistics can be dangerous
As a race, we humans do seem pretty ill-prepared to deal with the sophisticated judgments needed in todays hi-tech society. We aren't, yet, savvy enough, nor are we automatically educated well enough to correctly deal with 'statistics'.
Here's a case in point as an illustration. Let's imagine a sophisticated A.I. driven detector that has been designed to use facial recognition technology to identify terrorists - not a too far-fetched scenario. Now, let's say it's really good and will detect the bad person correctly 99% of the time. So, if we examined 100 terrorists we would get 99 of them. It does have a bit of a flaw though in that is will identify an innocent person as a potential bomber 1% of the time, so if we looked at 100 innocents it would find one 'terrorist' who wasn't there. Still a pretty accurate machine wouldn't you agree?
So, let's release it into a city of 1,000,000 people in which, we know, there are 100 terrorists (don't ask me how we know, we just do - OK?). The question I pose is, if it is used in the city centre and identifies my friend Bloggs as a terrorist, what is the chance that Bloggs actually is a terrorist? Go on have a think about that. What would you say - almost certainly? 90% certainly? 50-50? unlikely ... ?
The actual answer is that there is only around a 1% chance that he actually is a genuine terrorist - one chance in a hundred - highly unlikely. Hopefully, the subsequent typically thorough police investigations will show this and he won't spend the rest of his life in an orange onsey (? how is it spelled ?).
Most of us instinctively just don't believe this - it can't be that low surely? The machine is, after all, 99% 'accurate'!
Well let's 'prove' the figures. We'll put everybody through the machine and see what the statistics work out as.
We already know that there are 999,900 innocents and 100 terrorists in the city.
Th machine therefore identifies 99 terrorists (and one will go free). Of the remaining, ordinary 999,900 we'll actually wrongly identify 1% of these (9999) as terrorists. Therefore at the end of the experiment we have identified 9999+99 = 10,098 'terrorist'. But actually, only 99 of these are really bad guys and the rest aren't. So the chance that one of the 'identified' group is an actual extremist killer is 99/10098 which is only 1 in 102 or just less than 1%. This can, of course, all be calculated much more elegantly using Bayes' Theory - and I might do a bit on this at another time.
Ok, this is just an artificial, hypothetical scenario but it does expose the fact that we can't just take 'accuracy' figures at face value. We also need to know the incidents of false positives, false negatives and something about the prevalence of what we're looking for to get anything like a good idea of how really 'certain' an identification is.
Many people think the establishment of a national 'DNA database' is a no-brainer as we'll always catch be able to identify and catch the bad guys then. But, in light of the above would you really be happy 100% to rely on an identification? If you would be then do some further research - for a start have a look at a paper by William Thompson and colleagues (Thompson et.al, 2003)
Thompson, W.C., Taroni, F., Aitken, C.G.G., 2003. How the probability of a false positive affects the value of DNA evidence. J. Forensic Sci. 48, 47–54.
Here's a case in point as an illustration. Let's imagine a sophisticated A.I. driven detector that has been designed to use facial recognition technology to identify terrorists - not a too far-fetched scenario. Now, let's say it's really good and will detect the bad person correctly 99% of the time. So, if we examined 100 terrorists we would get 99 of them. It does have a bit of a flaw though in that is will identify an innocent person as a potential bomber 1% of the time, so if we looked at 100 innocents it would find one 'terrorist' who wasn't there. Still a pretty accurate machine wouldn't you agree?
So, let's release it into a city of 1,000,000 people in which, we know, there are 100 terrorists (don't ask me how we know, we just do - OK?). The question I pose is, if it is used in the city centre and identifies my friend Bloggs as a terrorist, what is the chance that Bloggs actually is a terrorist? Go on have a think about that. What would you say - almost certainly? 90% certainly? 50-50? unlikely ... ?
The actual answer is that there is only around a 1% chance that he actually is a genuine terrorist - one chance in a hundred - highly unlikely. Hopefully, the subsequent typically thorough police investigations will show this and he won't spend the rest of his life in an orange onsey (? how is it spelled ?).
Most of us instinctively just don't believe this - it can't be that low surely? The machine is, after all, 99% 'accurate'!
Well let's 'prove' the figures. We'll put everybody through the machine and see what the statistics work out as.
We already know that there are 999,900 innocents and 100 terrorists in the city.
Th machine therefore identifies 99 terrorists (and one will go free). Of the remaining, ordinary 999,900 we'll actually wrongly identify 1% of these (9999) as terrorists. Therefore at the end of the experiment we have identified 9999+99 = 10,098 'terrorist'. But actually, only 99 of these are really bad guys and the rest aren't. So the chance that one of the 'identified' group is an actual extremist killer is 99/10098 which is only 1 in 102 or just less than 1%. This can, of course, all be calculated much more elegantly using Bayes' Theory - and I might do a bit on this at another time.
Ok, this is just an artificial, hypothetical scenario but it does expose the fact that we can't just take 'accuracy' figures at face value. We also need to know the incidents of false positives, false negatives and something about the prevalence of what we're looking for to get anything like a good idea of how really 'certain' an identification is.
Many people think the establishment of a national 'DNA database' is a no-brainer as we'll always catch be able to identify and catch the bad guys then. But, in light of the above would you really be happy 100% to rely on an identification? If you would be then do some further research - for a start have a look at a paper by William Thompson and colleagues (Thompson et.al, 2003)
Thompson, W.C., Taroni, F., Aitken, C.G.G., 2003. How the probability of a false positive affects the value of DNA evidence. J. Forensic Sci. 48, 47–54.
