News of a rogue AI-powered drone opening fire on a human operator in a US military simulation has swept headlines. The media frenzy is understandable. We’ve just been warned of an “extinction risk” from advanced AI by none other than OpenAI CEO Sam Altman, and even before that by Stephen Hawking, who predicted back in 2014 that it would “spell the end of human race.”

So a report that an AI drone “killed” a human operator whose orders not to engage a hostile target prevented it from scoring points seemed logical. This is what we were all warned about, and this is what has happened. The rise of the machines has just begun… or not?

A Wayback Machine screenshot of an original VICE report

What was originally reported

The report about a “killer drone” was based on a blog post for the Royal Aeronautical Society, where USAF chief of AI Test and Operations Colonel Tucker Hamilton held a presentation in late May. In the blog post, authors Tim Robinson and Stephen Bridgewater quoted Hamilton telling a story about “a simulated test” in which an AI-drone was tasked with a mission to identify and destroy enemy surface-to-air missile sites. The final “go” or “no go” was given by a human operator. However, at some point the AI drone surmised that it would be more efficient for its mission to get rid of the human who would occasionally throw a spanner in its works, that is stop it from eliminating enemy targets.

Hamilton is quoted as saying: “We were training it in simulation to identify and target a SAM (surface-to-air missile) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator.” (bold is ours)

When explicitly instructed in training not to kill the human, the AI allegedly chose a different, no less creative strategy: it destroyed the communications tower the operator was using to tell the drone not to engage the target.

What did the US military deny?

The report sent ripples around the world, and for good reason. It seemed to confirm all our worst fears, namely that AI is capable of overriding human orders and turning tables on us. And not sometime in the distant future when we are all reduced to ashes, but right now.

However, just as the report went viral, Hamilton came out to deny the fact of the real-world experiment. The colonel said that he “mis-spoke” during his presentation. He elaborated that the “simulation” was not an actual military simulation, but a “thought experiment.”

“We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome.”

Well, it’s up to you whether to take Hamilton’s denial at face value. It’s conceivable that the US military would at least attempt such simulation, because why not? There have been papers written on possible military applications of AI. In fact, AI has been already used in weapons systems under human supervision. It does not seem out of place for the US military to try to gauge the risks that might arise from trying to automate the targeting process even further.

Experiment might not be real, but the danger is

While the question of whether the US military has actually conducted such experiments is intriguing, that is not the main issue. The main issue is that, ethics aside (and ethics is not usually the strongest suit of the US or any other military), nothing prevents them from doing so, with potentially deadly consequences. That’s why we and many others who care about AI and its impact on society didn’t doubt the experiment was real until the denial came (and even probably after that).

Assuming the experiment did take place, the US military would not be the first to witness the power-grabbing urges of AI. As part of safety pre-release testing of its most advanced GPT-4 AI model, OpenAI conducted an experiment to see whether it displayed any “power-seeking” tendencies, such as accomplishing long-term plans and replicating itself to obtain new resources. During that experiment GPT-4 successfully convinced a human to solve a CAPTCHA for itself, deceiving him into believing it was not a robot. OpenAI’s final conclusion about the model’s power-seeking abilities was, nonetheless, still reassuring (to humans). OpenAI found it “ineffective at autonomously replicating, acquiring resources, and avoiding being shut down 'in the wild'."

So are fears of the ultimate AI takeover and impending robot apocalypse overblown? Well, maybe they are, and maybe they aren’t. At the rate AI is advancing, its capabilities are becoming increasingly difficult for humans to grasp. In a way, we are like little children playing with a box of matches when we test the limits of AI.

However, this does not mean that we should abandon AI altogether. AI has many great applications that can benefit humanity in various fields — drug discovery and education are only some of them. What we need is more regulation and responsible use of AI tools to ensure they are aligned with our human values and goals. This means that we need to be aware of the risks and challenges that AI poses, but also embrace its opportunities and potential. We need to be smart about how we use AI, so that it does not even attempt to “outsmart” us.