Of course Charles Manson is in his cell. The computer said so.

Bruce Schneier links to an Ars Technica post that reports on a demonstrated ability for a hacker to open a prison cell door from a remote location.

Schneier was stuck by this paragraph in the Ars Technica post:

"You could open every cell door, and the system would be telling the control room they are all closed," Strauchs, a former CIA operations officer, told the [Washington] Times.

(See the Washington Times article here.)

Think of the ramifications of this - not just with prisoners, but with everything. In the prison case, you have a computer screen telling you that Charles Manson is safely in his cell, muttering about George Harrison or whatever Manson does in his cell. But perhaps if you actually went down to the cell, you'd see that he had actually been freed (or, in Strauchs' scenario, killed).

Now extend that to other computerized systems that tell you something that may not be true. I don't believe that we can categorically state that every computer system is 100% accurate. For example, I'm sure that they put some incredible design into traffic lights, but it's probably possible for a situation to arise in which green lights appear in all directions. The possibility may be extremely small, but it's possible.

So if I'm driving down the street and approaching a green light, and I see a car on the other street approaching the intersection without slowing down, it would not be wise for me to say, "Of course he'll stop. He has to have a red light since I have a green light." (Obviously the more likely scenario is that the other driver DOES have a red light, but is ignoring it.)

But what do you do in other cases in which a computer is telling you something that might not be true?