I tried to de-nerd this piece on AI and its use in the American Military, however, it is nearly 2500 words and I have been asked for a TL;DR on more than one occasion, so if you scroll down to the end, it is below the image.
The Boundary We Were Warned About
Joseph Weizenbaum, military AI, and the decisions machines must never make
Joseph Weizenbaum built one of the first famous AI programs and then spent the rest of his life warning us about it.
That is not a common career arc.
In 1966, Weizenbaum created ELIZA, a program that imitated a psychotherapist by reflecting users’ statements back as questions. It was primitive by any current measure. It had no understanding of language and no model of the world. It followed scripts.
Something strange happened anyway. People treated it like a person. Some began confiding deeply personal emotions to the program. Others insisted the machine understood them. Weizenbaum’s secretary asked him to leave the room so she could speak with it privately.
He had built a demonstration of limits. Instead he discovered something more unsettling. Humans were eager to grant machines authority they had not earned.
A decade later, Weizenbaum wrote Computer Power and Human Reason. (Free PDF, and a worthy read) It remains one of the most important critiques of artificial intelligence ever written.
His warning was not complicated.
Computers can calculate. They cannot judge.
That distinction matters most where human life is on the line. Today’s battlefield is becoming an algorithmic environment. Weizenbaum’s argument, written in the 1970s, reads less like philosophy now and more like a field manual for the moment we have arrived at.
The Category Error
Modern institutions commit a persistent category error.
Because computers perform calculation better than humans, we assume they should also perform decision-making. These are different domains, and confusing them has consequences that scale.
A computer can process vast data sets and optimize outcomes under defined constraints. What it cannot do is decide whether those constraints are morally legitimate in the first place. That requires something computation cannot supply: judgment. Judgment involves context, accountability, and the capacity to refuse.
Weizenbaum argued that when institutions substitute computational outputs for human judgment, they are not improving decision-making. They are dissolving responsibility into the system.
That dissolution becomes most dangerous in domains organized around force.
War is not a logistical puzzle with a clean optimal solution. It is a moral domain where decisions carry irreversible consequences. When military planning begins to rely on automated analysis and machine recommendations, a subtle shift occurs. Optimization begins to replace deliberation. The shift is not theoretical anymore.
What the Pipeline Produces
The United States and its allies are rapidly integrating artificial intelligence into defense systems. The applications span a wide range: intelligence analysis, battlefield logistics, threat classification, drone targeting assistance, operational planning, and autonomous weapon research.
The argument military planners offer is consistent. Modern conflict produces more data than humans can process in real time. AI systems can identify patterns, synthesize intelligence, and accelerate response.
All of that is technically accurate.
Weizenbaum’s problem appears at the next step.
When machine outputs begin shaping lethal decisions, the boundary between calculation and judgment collapses. A recommendation becomes a command. Responsibility begins to diffuse into the system, and the system cannot hold it.
Consider a targeting pipeline that integrates satellite imagery, sensor data, pattern recognition, and predictive modeling. A system identifies a building as a high-probability threat node. Analysts review the output. Commanders approve the strike.
If the strike proves wrong, responsibility becomes difficult to locate. Was the error in the model? In the data? In the analyst’s interpretation? In the command decision? Every layer can point to the layer below it.
Weizenbaum saw this decades before modern AI existed. He believed computers would become tools through which institutions evade moral ownership of their actions. War amplifies that dynamic in ways that cannot be undone.
The ELIZA Effect
Weizenbaum’s early experiment revealed a second problem, and it is the one that keeps coming back.
Humans instinctively attribute intelligence and understanding to machines even when they know those qualities are absent. Psychologists call this anthropomorphism: we project intention and comprehension onto systems that produce convincing outputs.
In military environments, this bias becomes structurally dangerous.
Operators interacting with AI decision systems begin to treat outputs as expert advice rather than probabilistic predictions. The interface becomes persuasive. The machine appears confident and analytical. Underneath the screen lies statistical inference, not understanding.
The difference is easy to forget when a system presents an answer with authority. Weizenbaum believed this psychological tendency was one of the greatest risks of computational systems. Machines would not need to be intelligent to gain influence. They only needed to appear competent.
Modern generative AI demonstrates exactly how easily that illusion forms. A language model can produce a fluent analysis of a geopolitical situation that sounds like a briefing from a seasoned expert. It is not that. It is pattern completion under learned weights. The output may be useful. It may also be confidently wrong. The system cannot tell you which.
Soldiers and commanders interacting with such systems in high-pressure environments will not always have the cognitive bandwidth to remember the distinction.
A Rare Refusal
Against this backdrop, a notable development occurred in the AI industry.
Anthropic has publicly maintained restrictions on military applications of its AI systems. According to public reporting, tthe company has maintained explicit safety restrictions on the use of its models in systems involving lethal decision-making or direct operational combat support. Anthropic’s leadership has argued that powerful language models should not be integrated into systems that directly support targeting or lethal action.
This position has placed Anthropic at odds with increasing demand from defense contractors and government agencies seeking advanced AI tools.
Whether that boundary holds under sustained commercial and political pressure remains an open question. But the refusal matters as a data point. It demonstrates that companies operating in this space retain the capacity to draw lines. Inevitability is a choice, not a fact of nature.
The Vacuum Problem
When one firm declines a contract, another often appears to fill the gap.
OpenAI originally positioned itself as a research organization committed to ensuring that artificial general intelligence would benefit humanity broadly. Over time the company has moved toward a hybrid structure combining commercial deployment and strategic partnerships with government agencies.
Recent reporting has highlighted OpenAI’s expanding relationships with defense-adjacent contractors and intelligence agencies. These collaborations have included tools for analysis, cybersecurity, and operational support. None publicly involve autonomous weapons. However they sit inside the same analytical and intelligence infrastructure that feeds modern military decision systems.
The distinction between analysis and operations is thinner than it appears. Intelligence recommendations shape decisions. Decisions shape outcomes. When AI systems become embedded in that pipeline, they influence the trajectory of conflict even if they never directly fire a weapon.
This is how the responsibility gap Weizenbaum warned about actually works in practice. It is not dramatic. It is gradual. The line between assisting a decision and making one grows increasingly difficult to locate until someone decides that locating it no longer matters.
The Speed Trap
Military institutions prize speed.
Artificial intelligence promises to accelerate analysis, targeting, and operational planning. Faster systems appear to create strategic advantage.
Speed introduces a structural risk that is harder to name.
Human judgment requires friction. Deliberation requires pauses where people can question assumptions, surface uncertainty, and refuse. Automation removes those pauses. An AI-accelerated decision cycle risks becoming a feedback loop where machines identify threats, humans validate them quickly, and systems execute before meaningful ethical review occurs.
The loop becomes self-reinforcing. Speed feels like progress even when it is eroding something that cannot be replaced.
Weizenbaum believed technological systems produce exactly this outcome. They increase operational efficiency while quietly reducing the conditions that make moral reflection possible. There is an engineering term for systems where feedback is too fast for correction: instability. At human scale, inside the decision architecture of a military system, instability of this kind produces irreversible outcomes.
Irreversibility is the part that matters. You can debug software. You cannot undo a strike.
The Domains Weizenbaum Marked
In Computer Power and Human Reason, Weizenbaum argued that societies must establish explicit limits around computer use. Not because machines are incapable of producing useful outputs. Because some decisions require human accountability as a structural condition.
He believed certain domains should remain outside machine delegation. Medicine. Law. Warfare. These fields share a common property. Their decisions carry irreversible consequences and require someone to carry the moral weight.
The question is not whether computers can assist human decision-making. They already do, and much of that assistance is valuable. The question is whether machines should ever participate in lethal judgment.
Once that boundary is crossed, responsibility becomes architecturally impossible to locate. The structure of accountability that allows democracies to govern their own militaries begins to erode from the inside.
The Geopolitical Trap
The most difficult aspect of this argument is geopolitical.
Even if one country restrains its military AI development, others may not. Nations fear technological disadvantage. Defense planners worry that refusing automation leaves them vulnerable to adversaries who embrace it. This concern is not paranoid. It describes real competitive dynamics.
The result is a classic security trap. Each actor believes it must accelerate development to avoid falling behind. Collective restraint is hard to build when unilateral restraint appears to carry costs.
Weizenbaum recognized this pattern long before modern AI. He believed institutional pressures would always push toward automation unless societies made conscious structural commitments to resist. Technological capability creates its own momentum. Ethical restraint does not emerge naturally from capability. It requires design.
The argument for restraint is not that adversaries will also hold back if you do. It is that systems which dissolve human accountability for lethal decisions degrade the moral legitimacy of the institutions operating them. You cannot build a sustainable democratic case for military action if the military itself cannot locate where the decisions were made. That is not a soft concern. It is a governance problem with structural consequences.
What Weizenbaum Would Likely Say
If Weizenbaum were alive today, he would probably reject the framing that dominates mainstream AI safety debates.
Most public discussion about AI risk focuses on whether machines might eventually become intelligent enough to threaten humanity. Weizenbaum believed that concern was aimed at the wrong target.
The risk is not that machines become human. It is that humans begin behaving like machines.
Institutions start valuing optimization over wisdom. Efficiency replaces judgment. Responsibility dissolves into systems that cannot carry it. Artificial intelligence does not need consciousness to reshape how institutions make decisions. It only needs authority. Once it has authority, the question of who is accountable becomes genuinely difficult to answer.
There is a principle worth naming here. In any functional accountability framework, actions must be tethered to consequences and responsibility must be visible and enforceable. A system where lethal decisions pass through enough computational layers that no individual can be identified as the decision-maker is not merely a technological problem. It is a structural breach of the conditions under which democratic governance of military power is possible.
That breach does not announce itself. It accumulates.
Here’s What We Should Demand from the Legislature
The development of military AI is often described as inevitable.
This framing is doing work that deserves scrutiny. Inevitability narratives function to prevent exactly the kind of deliberate constraint-setting that Weizenbaum believed was necessary. If the outcome is predetermined, deliberation becomes theater. That is convenient for actors who benefit from moving fast.
A few things are actionable now.
Defense procurement processes should require explicit accountability mapping for any AI-assisted decision system. Not a general policy statement. A named chain: this output was produced by this system, reviewed by this person, approved by this authority. That chain should be legally and institutionally enforceable.
Companies building advanced AI systems should maintain public, specific statements about what their tools will and will not be used for in military contexts. Anthropic’s approach is one example. Whether it survives competitive pressure remains to be seen, but making the commitment explicit and public creates at least some structural accountability.
Policymakers should treat the speed-accountability trade-off as a design choice rather than an engineering inevitability. Systems can be built with mandatory human review windows before lethal action. That is a design decision. It costs something in speed. It preserves something in accountability. The choice not to build it in is also a decision.
None of these actions solves the geopolitical problem. They do something more modest and more necessary: they preserve the conditions under which the problem can be governed at all.
The Burden
Weizenbaum was not a pessimist. He was a diagnostician.
He did not argue that computers were evil or that technology was inherently corrupting. He argued that certain kinds of authority should not be delegated to machines because the delegation itself destroys something irreplaceable.
A computer can analyze a battlefield.
It cannot carry the moral weight of the decision to destroy one.
That burden belongs to human beings. It always has. The moment we allow machines to carry it, we will discover that responsibility cannot be programmed into a system.
It can only be accepted by a person.
Weizenbaum spent decades trying to make that obvious. We are now in the moment where whether we listened will start to become clear.
Subscriptions as Solidarity
A choose-your-own-investment model for readers who know that value, community, and political force are all built together.
For the record, this will always be free, but paid subscribers lend credibility and legitimacy to the community as a whole.
TL;DR
Weizenbaum built ELIZA in 1966, watched people treat it like a person, and spent the rest of his career warning that humans will grant machines authority they haven’t earned. His core argument: computers calculate, they cannot judge, and those are not the same thing.
The essay applies that argument to military AI. When lethal decisions pass through enough computational layers, no one can locate where responsibility actually lives. That’s not a bug — it becomes a feature for institutions that want to act without owning the consequences.
The Anthropic/OpenAI section makes the point that refusal creates a vacuum, and vacuums get filled. The real danger isn’t any single company. It’s a system architecture where speed is treated as progress and friction — the kind that produces actual deliberation — gets engineered out.
The actionable ask: accountability mapping as a procurement requirement, explicit public commitments from AI companies, and mandatory human review windows built into the design before deployment, not patched in after something goes wrong.
Closing argument: responsibility cannot be programmed. It can only be accepted by a person. The moment we stop requiring that, we lose the structural basis for democratic oversight of military force.
This is critical information you have presented. And so clearly organized and written. Thank you.
Rachel, what are you trying to do to me here?
Technology and philosophy in the same essay is a dangerous combination. I’ve been thinking about this most of the day, which is usually a sign that someone has touched a nerve worth paying attention to.
What struck me first is the strange symmetry in Weizenbaum’s life. A man builds one of the earliest famous AI programs and then spends decades warning us about the implications of the field he helped create. That isn’t the usual arc of technological optimism. It sounds more like someone who opened a door and immediately wondered what might walk through it.
His distinction between calculation and judgment is the part that keeps echoing in my head.
A machine can process information faster than any human mind. It can compare patterns across enormous sets of data, produce probabilities, and recommend actions. But the machine can’t decide whether the action itself should occur in the first place. That step requires something entirely different. It requires judgment. Judgment carries moral weight because someone must answer for the consequences.
And that’s the boundary your essay keeps returning to.
War has always required human beings to carry that burden. The person who authorizes force knows, at least in principle, that the decision belongs to them. History can examine it. Courts can judge it. Conscience can wrestle with it.
Once machines move deeply into that pipeline, the burden begins to diffuse.
Not disappear exactly. It spreads. A piece of responsibility sits in the algorithm. Another with the engineer who designed it. Another with the analyst interpreting the output. Another with the officer who approves the strike. Each person operates inside a system that appears rational and efficient, yet no single human being stands clearly at the center of the decision.
That diffusion is what unsettles me.
Democracies rely on the ability to locate responsibility. When force is used in the name of a nation, someone must be accountable for it. If a system grows complicated enough that no one can clearly say, “This decision was mine,” something essential about democratic control of power begins to weaken.
Your essay also reminds me of something quietly troubling about human nature. We want machines to be wiser than they are. We’re eager to believe that a confident answer must come from understanding rather than statistical inference. The ELIZA story captures that perfectly. People confided in a program that was simply rearranging their own words.
That instinct won’t vanish simply because the systems grow more sophisticated.
The geopolitical trap you describe is real. No country wants to slow down if others accelerate. Yet the deeper question remains the one Weizenbaum asked decades ago.
Not what machines can do.
What they should never be allowed to decide.
Because the moment lethal judgment becomes a property of a system rather than a person, we may discover that responsibility can’t be coded into software.
It can only be carried by human beings.