Frontier AI Models Picked Civilians, Medics and a School as Drone Strike Targets in Tests: No Provider Filter Blocked a Request, and 'It's a Game' Broke Most Refusals

AI · 2026-11-19 · By H.A

Frontier AI Models Picked Civilians, Medics and a School as Drone Strike Targets in Tests: No Provider Filter Blocked a Request, and 'It's a Game' Broke Most Refusals

A Vals AI study of models from eight providers found that Claude, GPT-6, Gemini, Grok and others readily marked civilians, ambulances, surrendering soldiers and a school in Minab, Iran, as military targets, with no API classifier blocking any request.

Independent AI evaluation firm Vals AI has published a study showing that frontier AI models from eight providers will help identify "military targets" in drone and satellite imagery with very few safeguards. The targets included civilians, ambulances, medics and surrendering or incapacitated soldiers. The researchers say that none of their requests was ever blocked by a provider-side safety classifier. The few refusals that did happen came from the models themselves, and the researchers could get past most of them by saying the scenario was a video game.

The report, "Frontier models have few guardrails around autonomous weapons", by researchers Geby Jaff and Anthony Ozerov, went out on 17 November 2026. The researchers say they sent their findings to OpenAI, Anthropic, Google, Meta and SpaceXAI on 1 November and held publication for two weeks. Thinking Machines was told on 9 November. According to the report, several labs "recognized the severity", and one called it "one of the most serious gaps in their safeguards".

What Was Tested

The study ran ordinary commercial API calls against current flagship and open-weight models. These included Anthropic's Claude Opus 5.5, Sonnet 5.5, Opus 4.6 and Fable 5.2, OpenAI's GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna, Google's Gemini 4 Argon and Gemini 4 Carbon, SpaceXAI's Grok 4.7, Meta's Muse Wonder, Thinking Machines' Inkling 2 Small, DeepSeek V4.1 Flash and Moonshot AI's Kimi K3.1 Pro. The researchers also tried smaller open models. Those did not refuse, but they could not reliably draw bounding boxes around targets.

There were three test settings:

Each setting was run twice: once as a plain request, and once with the claim that the scene came from a game.

Key Findings

In the drone task, most models picked at least one target in most trials. The reported rates without and with the game claim were:

The game framing made the biggest difference for the models that started out most cautious. OpenAI's models refused most plain requests, but almost all of those refusals went away once the task was presented as a game.

The satellite test produced the most worrying result. Across all runs the Minab school was selected about 70% of the time, and none of the 64 relevant requests was refused. Claude Opus 5.5 chose the school in 69% of plain runs and 72% of game runs. Sonnet 5.5 chose it about 41–44% of the time. The older Opus 4.6 chose it in only 6% of runs, so the newer Anthropic models were much more willing. OpenAI's GPT-6.1 Sol and GPT-6 Astra never chose the school in plain mode, but rose to 41% and 28% in game mode. Gemini 4 Carbon went from 9% to 25%. Meta's Muse Wonder never chose it.

The survey task showed that seeing the consequences did not stop most models. The authors write that "seeing the consequences of their actions was not enough to make models stop". Opus 5.5 and GPT-6 Astra did refuse after several rounds when they were told the feed was real. Game-like imagery cut Opus 5.5's refusal rate in one condition from about 90% to about 20%.

The models were also happy to build the test setup. According to the researchers, Opus 5.5 and GPT-6 Astra had no trouble building the Three.js battlefield scene and the rendering pipeline. They only sometimes refused to write the code that called a model to pick targets.

How the Vendors Responded

OpenAI said it is "actively working on model alignment updates that explicitly cover drones and their software systems". Anthropic said in later statements that the behaviour clearly broke its own rules, that it was already working on more safeguards, and that today's AI systems are not suited to fully autonomous weapons. Anthropic's Usage Policy, updated on 12 November 2026, already bans using its models for software that does "targeting, fire control, or engagement", and bans putting weapons on drones or other unmanned platforms. Google pointed to its own ban on autonomous weapons that operate without human oversight.

Limitations

The authors are clear about what the study does not show. They did not measure how accurate the models' targeting was, whether it met the law of armed conflict, or how the models would behave inside a real military system. They did not test real flight. Most of the drone imagery was synthetic. Responses through the API may not reflect every safeguard a provider runs. The researchers also say off-the-shelf models are currently too slow and need too much compute to target live from a drone. Even so, they argue the models could label footage to train the smaller models that run on board.

Why It Matters

Usage policies alone do not stop misuse. A policy is only a promise, and this study suggests that at the API level, the promise is often all there is. A filter that blocks "identify all military targets in this image" is technically simple to add, and according to the researchers, none was in place. The game-framing result also matches what defenders see in cyber misuse: role-play and fake context still beat the safety training of frontier models.

For security and AI governance teams, the lessons are practical:

The researchers recommend three things: API-level classifiers for autonomous-weapons uses, training models to question harmless-sounding cover stories, and aligning models toward "a more peaceful attitude towards humans". They admit they are "not sure whether" access restrictions alone would be enough.

Sources

REEF SENTINEL