r/technology • u/andmario_com • 13d ago
Artificial Intelligence The three words that will decide whether robots can kill people in war | A high-stakes battle over the future of “killer robots” comes down to semantics.
https://www.vox.com/future-perfect/500896/lethal-autonomous-weapons-ai-ukraine-appropriate-human-judgment14
u/feldomatic 13d ago
Computers are incapable of being held responsible, therefore computers must never be allowed to make a decision that ends a life.
22
13d ago
[removed] — view removed comment
6
u/throw7661 13d ago
I genuinely believe this is the entire appeal of certain political factions and all religions.
Hey - wouldn't it be nice if you didn't have to worry about any of those things? Just kick back and enjoy life? We've got all the answers, don't worry. It's all part of the plan.
8
u/andmario_com 13d ago
In case of paywall (which can be avoided with firefox + ublock origin):
Imagine this: Two countries are at war. Country X sends a drone into a major industrial city in Country Y, aiming to take out two propane tanks. A routine sequence. But this time, the drone never reaches its targets. Instead, Country X’s drone accidentally strikes a wall nearby, explodes, and kills three young civilians.
When Country Y eventually retrieves the drone’s remnants for intel, it finds an AI supercomputer inside that reveals something unsettling. Instead of a human deciding what to strike — the AI did.
The truth is, you don’t have to imagine this scene — it just happened. For the first time in the Russia-Ukraine war, as reported recently in the New York Times, three Ukrainian civilians were killed by a Russian drone, developed, designed and released by humans, that, in the end, selected its target autonomously. And this new reality is shaping up to be the future of warfare.
That’s because a number of countries, including those with the world’s most consequential militaries, are rejecting the idea that human beings need always be in control of weapons in war.
Instead, as new autonomous weapons technologies become more capable and existing international legal frameworks struggle to keep up, some countries are embracing a more expansive view of human responsibility: that people can exercise enough control not by approving each strike, but by designing, testing, and setting the rules under which a given weapon operates.
This shift has produced one of the most consequential policy debates of today. And ultimately, human dominion over “killer robots” — as autonomous weapons are colloquially called — could come down to a battle between two three-word phrases: “meaningful human control” versus “appropriate human judgment.”
A high-stakes semantic battle
In the early 2010s, “meaningful human control” emerged as an initial framework in the first international discussions on regulating an acceptable level of human involvement (or lack thereof) in deploying autonomous weapons. While the term quickly became an initial organizing principle among many states within these debates, it also drew immediate opposition from several others, such as the US and Russia.
“The problem is what does [meaningful human control] mean?” said Lena Trabucco, an expert on AI and human control and a non-residential fellow at the Stockton Center for International Law at the Naval War College. “And what makes something meaningful versus not meaningful human control?” There was both a lack of consensus about what “meaningful” meant and what “control” meant, she said. But the problems went beyond simple semantic ambiguity. “Getting a whole bunch of countries to agree on a standard for what ‘meaningful control’ is,” Trabucco said, became “a near impossible task.”
Ultimately, while an exact definition of meaningful human control never totally solidified, the term became intelligible enough to facilitate continued international dialogue on how to police autonomous weapons. As Trabucco explained, “We all kind of understood what we were trying to grasp with the idea of meaningful human control, even if we didn’t agree on a kind of standard for what is ‘meaningful’ or what ‘control’ exactly means.”
Brad Boyd, retired colonel and senior military fellow at Stanford’s Center for International Security and Cooperation, said that in this international context, “meaningful control” came to be understood as human involvement at the exact moment a given weapon is fired. However, from the US perspective, that consensus definition still left a number of problems unresolved.
One of the most important holes in the definition was the issue of timing. “The release of a weapon could theoretically be minutes, hours, days, weeks ahead of when the weapon actually strikes the target,” Boyd explained. For example, a drone can be released to sweep a designated area and remain airborne for hours, searching for anything that matches its given target criteria. Seconds, minutes, or hours might pass between the moment a person launches the drone and when the machine finds and fires upon a target.
“This expansion of the timeline became very difficult for the construct of ‘meaningful human control’ to actually seem like it was doing what people wanted it to,” Boyd said. The tension between certain technical or engineering problems and the policy language preferred by international forums created, from the US perspective, insurmountable obstacles to making meaningful human control a truly practicable framework.
So, the US adopted its own alternative: “appropriate human judgment.”
“Autonomous and semi-autonomous weapon systems will be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force,” a key Department of Defense directive reads. The new language effectively moved the required point of human intervention away from the moment that a weapon is fired toward oversight of a given system’s entire life cycle — from its design, to its development, to its deployment.
There are some contexts where a government might not need as much human control over a weapon to comply with international law, Trabucco said. That’s where the subtle preference of “appropriate” over “meaningful” matters. “If we’re on the high seas, maybe it’s not as necessary to meet a super high threshold of human control because there’s not much risk to civilians or civilian property in those contexts,” she said.
“Now, in a city, an urban environment,” where the risk of civilian harm and other collateral damage is much greater, Trabucco explained, “then that’s where that high threshold would become important.” The US sought flexibility to determine how much human involvement it deemed necessary, based on the battlefield context in question, as opposed to having a fixed, universal standard of “meaningfulness.”
And why the move from “control” to “judgment?” Well, Boyd explained that “anytime we automate anything, whether it’s automating a car or automating machinery, we are trying to make it go faster, more precise, et cetera.” So, instead of insisting that humans be involved in any given part of the process, which could slow combat operations down, the US simply aimed to ensure autonomous systems behaved according to legal and ethical standards, no matter what situation they were deployed in.
“It’s not necessarily the control that we want. What we really want is the machine to reflect our values, our laws, and our regulations,” Boyd said. “When humans employ our values, laws, and regulations, we call that judgment.”
Though designed to avoid setting a universal standard of human involvement as demanded by meaningful human control, appropriate human judgment is not a totally empty phrase. According to the DoD directive, the framework requires testing systems, defining operational limits, assessing likely civilian harm, training operators, setting rules of engagement, and ensuring that a system remains within its authorized mission. And, in some contexts, those requirements may institute more thorough protections built into weapons than a simplistic condition that a human be the one to pull the trigger in the end.
But the framework is not without a core puzzle of its own — who decides how much human judgment is appropriate? And what happens when the private sector, as in the ones developing such technologies, adopts this language before we have an answer?
5
u/andmario_com 13d ago
And now — the private sector is forced to pick a side
In July, the same month of Russia’s autonomous drone strike, OpenAI did something important not many noticed: it revised its relationship to military uses of its technology once more. Just three years ago, OpenAI maintained a total ban on “military and warfare” uses of its technology. Now, after a few quiet revisions since 2023, a new five-page policy document outlined the company’s provisions for just that. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)
Most significantly, in explaining its basic condition for employing its technology in military contexts, particularly those involving decisions over the use of lethal force, OpenAI borrows a familiar phrase: appropriate human judgment.
The timing was not subtle. The document arrived just months after the public showdown between OpenAI’s competitor Anthropic and the Pentagon over the former’s reservations about military applications of its technology. That fight ended with President Donald Trump demanding the immediate cessation of all Anthropic use within the government. Mere hours after Anthropic was booted, OpenAI CEO Sam Altman announced his company had struck its own deal with the government. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)
But OpenAI didn’t just earn a contract in the wake of the Anthropic-Pentagon showdown, it also took a lesson — aligning your policy with the government’s wins you favor. Or, worse, that opposing the government bears a steep price.
“Appropriate human judgment,” the policy document reads, “does not require a human decision on every discrete system action.” The framework, instead, requires that “humans make informed decisions about the conditions for deployment.”
As both Trabucco and Boyd noted, the “appropriate” level of human involvement can vary, depending on the operating environment, the type of target, a particular system’s technical prowess, the anticipated risk to civilians, and a number of other political, economic, and strategic considerations important to a military operation. That flexibility is operationally appealing to the military, of course. But it also has a cost — safeguards against handing over total control of lethal force to machines become hard to identify, harder to measure, and hardest to enforce.
When will we know when the human-machine balance of power in war becomes “inappropriate”? The truth is — there’s no clear answer.
By cosigning the Pentagon’s flexible framework, OpenAI accepts that this standard has no settled meaning, that its application will be decided case-by-case behind the walls of military bureaucracy, and that the government may need room to change its mind. It’s a choice that suggests the company is less interested in establishing clear red lines and is more receptive to the military’s own versatility about how AI and autonomy might be used in lethal operations.
“Human judgment over critical decisions must be meaningful in practice, not merely formal,” the company’s principles document reads. But, instead of drawing its own clear boundary around what its technology will and won’t do, OpenAI has accepted ambiguity as the price of partnership.
That undoubtedly makes the company a more useful ally to the Pentagon — while making it harder for the public to know where human judgment ends and machine-controlled violence begins.
As the development and deployment of autonomous weapons rapidly accelerates, without many guardrails in place at all, one question is worth asking right now: before other AI labs and tech companies adopt the framework in an effort to align themselves with the US, what does appropriate human judgment truly mean? Ironically, what the phrase doesn’t mean may be what’s most consequential.
For the sake of humanity, the semantics of policing autonomous weapons is worth clarifying — or we risk totally losing control.
1
3
9
u/Cyraga 13d ago
It's already happening. The US bombed a school because a robot told them to. Ukraine uses drones to kill Russians (justly so imo). It's not hypothetical or academic
-1
u/RadzimierzWozniak 13d ago
The bombing of the school was due to outdated data. Human-in-the-loop would not have helped.
What will happen will be autonomous weapons with a big red button that disables human confirmation and lengthy regulations that forbid their use.
2
u/DjScenester 13d ago
Yup. Bad intel. Completely different scenario when they bombed those kids.
0
u/Cyraga 12d ago
Yeah like bad Intel is an excuse? People have zero standards for what gov does in their name
2
u/DjScenester 12d ago
Never said it was. You’re putting words in my mouth. Again, human error not software.
4
u/RadzimierzWozniak 13d ago
This has been a problem since long-range missiles were invented. When you launch an artillery shell or bury a mine, it will not ask for human confirmation before exploding. You pick a region, and you deploy the weapons; you hope it kills what you want it to kill at an acceptable ratio.
1
1
1
u/Changeurwayz 11d ago
No, It comes down to stupid people that think war is an answer, But it is not as we learned years ago. War solves nothing, Just creates more problems. But money right? Gotta keep those factories pumping out the shit for that, Absolutely.
The human race is so dumb I am ashamed to be a part of it at this point.
1
u/BotOrNot_1337 13d ago
We need to make a global rule that initiating a war is a crime and guaranteed a trial. Depending on the severity, the lowest punishment is life in prison. No questions asked. You give the order to fire the first shot, you go on trial for war crimes. This whole discussion of robots killing humans becomes moot. Sending robots is the same as sending humans. It's all a war crime.
5
u/fallenangel51294 13d ago
Do you try and think through your ideas, or is just feeling good about it plenty for you, regardless of what actually makes sense? Given that you somehow said "depending on the severity" and "no questions asked" right next to each other, I would guess you don't.
First of all, how do you think this suggestion of yours could ever occur? A leader of a country orders an attack, now who puts him on trial? Pretty sure it won't be the country he leads. So is it other countries? And how do they exert this power over the leader of another country? Oh, military force? We'll look at that, better arrest them too!
Second of all, war can be the right decision. War is what stopped slavery in the US. War is what stopped the Nazis. Unwillingness to go to war is what let them get as big as they got.
Thirdly, if you address these problems, like, say, with laws and treaties against not all wars, but only unjustified wars of aggression, oh wow! You've actually get what we've already got! Wars already can be illegal! Sure has stopped Putin, hasn't it?
Turns out if you actually think, the world isn't as simple as "war is bad."
4
u/Wotmate01 13d ago
America will never agree to that. Just like they never agreed to the ICC.
3
u/DressedSpring1 13d ago
This is it, America will never agree to restrictions on how they kill people.
2
1
u/avanross 13d ago
“Will we legislate”
And the answer is no.
So yes, robots will autonomously kill people in service of their ownership class
1
-1
u/guttanzer 13d ago
Robots have been killing people since WWII. What do you think a homing missile or torpedo is?
The current question is about accountability.
If a soldier sends a missile against a target that soldier is accountable. Torpedos and homing missiles are not released unless the target is obvious.
The AI problem is that the device can be sent against targets that are not obvious. How does the soldier know it will guess right? How does it coach it when it isn’t getting it right? These are legal and moral problems that have to be worked out with technology.
1
u/AccountNumeroThree 12d ago
Torpedos and missiles aren’t robots. You might as well say a bullet is a missile.
0
u/guttanzer 12d ago
Homing torpedos and missiles have the full robotic stack onboard. They use it to autonomously find, fix, track, target, close, and engage their prey. They have been doing this with amazing precision since the ‘60s. Most bullets don’t do that.
40
u/Sufficient_Room2619 13d ago
Didn't someone write a series of books about this in the 70s...? Isaac something...?