Rise Of The Machines

I love the Terminator films, and I’m a big fan of the current Sarah Connor Chronicles. I do realize that they are ficti0n. But the nut truth about the franchise, that we should be very wary of unmanned warfare, remains. Skynet is not real. But our military’s  devotion to developing robotic warriors is. If you think I’m just being delusional, check out ‘Robots at War’ in the latest Wilson Quarterly. The possible tragedies of reliance on computer have already occurred. Twice.

The most dramatic instance of a failure to override occurred in the Persian Gulf on July 3, 1988, during a patrol mission of the U.S.S. Vincennes. The ship had been nicknamed ­“Robo-­cruiser,” both because of the new Aegis radar system it was carrying and because its captain had a reputation for being overly aggressive. That day, the Vincennes’s radars spotted Iran Air Flight 655, an Airbus passenger jet. The jet was on a consistent course and speed and was broadcasting a radar and radio signal that showed it to be civilian. The automated Aegis system, though, had been designed for managing battles against attacking Soviet bombers in the open North Atlantic, not for dealing with skies crowded with civilian aircraft like those over the gulf. The computer system registered the plane with an icon on the screen that made it appear to be an Iranian F-14 fighter (a plane half the size), and hence an “assumed enemy.”

Though the hard data were telling the human crew that the plane wasn’t a fighter jet, they trusted the computer more. Aegis was in semi-­automatic mode, giving it the least amount of autonomy, but not one of the 18 sailors and officers in the command crew challenged the computer’s wisdom. They authorized it to fire. (That they even had the authority to do so without seeking permission from more senior officers in the fleet, as their counterparts on any other ship would have had to do, was itself a product of the fact that the Navy had greater confidence in Aegis than in a ­human-­crewed ship without it.) Only after the fact did the crew members realize that they had accidentally shot down an airliner, killing all 290 passengers and crew, including 66 ­children.

The tragedy of Flight 655 was no isolated incident. Indeed, much the same scenario was repeated a few years ago, when U.S. Patriot missile batteries accidentally shot down two allied planes during the Iraq invasion of 2003. The Patriot systems classified the craft as Iraqi rockets. There were only a few seconds to make a decision. So machine judgment trumped any human decisions. In both of these cases, the human power “in the loop” was actually only veto power, and even that was a power that military personnel were unwilling to use against the quicker (and what they viewed as superior) judgment of a ­computer.

Expect more of these type of horrors as we continue to develop even more deadlier and exacting ways of killing each other. We need a military. True. But do we need to continue to be the leader in efficient killing systems? We thought the nuclear bomb would be the ultimate weapon. That hasn’t turned out so swell for us. Why would we think robots will be any different?

The point is not that the machines are taking over, Matrix-style, but that what it means to have humans “in the loop” of ­decision ­making in war is being redefined, with the authority and autonomy of machines expanding. There are myriad pressures to give ­war-­bots greater and greater autonomy. The first is simply the push to make more capable and more intelligent robots. But as psychologist and artificial intelligence expert Robert Epstein notes, this comes with a built-in paradox. “The irony is that the military will want [a robot] to be able to learn, react, etc., in order for it to do its mission well. But they won’t want it to be too creative, just like with soldiers. But once you reach a space where it is really capable, how do you limit them? To be honest, I don’t think we can.”

No known limit. I’m already looking up Sarah Connor’s name in the phonebook.