Weizenbaum warned that computers calculate but cannot judge. As AI enters military systems, the responsibility gap he named in the 1970s has become a structural emergency.
Some of my other pieces on AI in the military are more histrionic - honestly, most media has been treating this as a simple procurement announcement. I tried really hard to keep the emotion and concern tightly framed for this one.
It’s so clear, the variable is the human. One human can use AI for good and another human can use the same AI for bad. And humans, as you point out, can easily deceive themselves.
Technology and philosophy in the same essay is a dangerous combination. I’ve been thinking about this most of the day, which is usually a sign that someone has touched a nerve worth paying attention to.
What struck me first is the strange symmetry in Weizenbaum’s life. A man builds one of the earliest famous AI programs and then spends decades warning us about the implications of the field he helped create. That isn’t the usual arc of technological optimism. It sounds more like someone who opened a door and immediately wondered what might walk through it.
His distinction between calculation and judgment is the part that keeps echoing in my head.
A machine can process information faster than any human mind. It can compare patterns across enormous sets of data, produce probabilities, and recommend actions. But the machine can’t decide whether the action itself should occur in the first place. That step requires something entirely different. It requires judgment. Judgment carries moral weight because someone must answer for the consequences.
And that’s the boundary your essay keeps returning to.
War has always required human beings to carry that burden. The person who authorizes force knows, at least in principle, that the decision belongs to them. History can examine it. Courts can judge it. Conscience can wrestle with it.
Once machines move deeply into that pipeline, the burden begins to diffuse.
Not disappear exactly. It spreads. A piece of responsibility sits in the algorithm. Another with the engineer who designed it. Another with the analyst interpreting the output. Another with the officer who approves the strike. Each person operates inside a system that appears rational and efficient, yet no single human being stands clearly at the center of the decision.
That diffusion is what unsettles me.
Democracies rely on the ability to locate responsibility. When force is used in the name of a nation, someone must be accountable for it. If a system grows complicated enough that no one can clearly say, “This decision was mine,” something essential about democratic control of power begins to weaken.
Your essay also reminds me of something quietly troubling about human nature. We want machines to be wiser than they are. We’re eager to believe that a confident answer must come from understanding rather than statistical inference. The ELIZA story captures that perfectly. People confided in a program that was simply rearranging their own words.
That instinct won’t vanish simply because the systems grow more sophisticated.
The geopolitical trap you describe is real. No country wants to slow down if others accelerate. Yet the deeper question remains the one Weizenbaum asked decades ago.
Not what machines can do.
What they should never be allowed to decide.
Because the moment lethal judgment becomes a property of a system rather than a person, we may discover that responsibility can’t be coded into software.
Like Pandora, he lifted the lid just enough to see what was inside. And unlike most engineers of his era, he did not immediately convince himself that the contents were harmless.
What you describe about responsibility is the core of it. Democracies function because power can be traced. Someone signs the order. Someone answers for the consequences. History can examine the decision and assign judgment to a human being.
Once decision systems become layered enough that responsibility spreads across software, engineers, analysts, and commanders, that clarity begins to blur. Not through malice. Through architecture.
That is the danger Weizenbaum saw before almost anyone else.
The machines themselves are not the moral actors. They cannot be. The risk is that institutions begin to behave as though the system itself has authority. Once that happens, the people inside the system start to experience decisions as outputs rather than choices.
And when choices stop feeling like choices, accountability becomes very hard to hold onto.
Responsibility cannot be coded into software. It can only be carried by human beings who are willing to say, without qualification, “This decision was mine.”
The moment that sentence becomes impossible to say clearly, something foundational about democratic control of force begins to erode. That was Weizenbaum’s fear.
Just sent this link off to my congressional rep, and senators. I think our extinction odds are growing grimmer by the day; but my latest t-shirt purchase states: Read Books, Walk Dogs, Fight Fascists. Onward.
This is critical information you have presented. And so clearly organized and written. Thank you.
Some of my other pieces on AI in the military are more histrionic - honestly, most media has been treating this as a simple procurement announcement. I tried really hard to keep the emotion and concern tightly framed for this one.
It’s so clear, the variable is the human. One human can use AI for good and another human can use the same AI for bad. And humans, as you point out, can easily deceive themselves.
Rachel, what are you trying to do to me here?
Technology and philosophy in the same essay is a dangerous combination. I’ve been thinking about this most of the day, which is usually a sign that someone has touched a nerve worth paying attention to.
What struck me first is the strange symmetry in Weizenbaum’s life. A man builds one of the earliest famous AI programs and then spends decades warning us about the implications of the field he helped create. That isn’t the usual arc of technological optimism. It sounds more like someone who opened a door and immediately wondered what might walk through it.
His distinction between calculation and judgment is the part that keeps echoing in my head.
A machine can process information faster than any human mind. It can compare patterns across enormous sets of data, produce probabilities, and recommend actions. But the machine can’t decide whether the action itself should occur in the first place. That step requires something entirely different. It requires judgment. Judgment carries moral weight because someone must answer for the consequences.
And that’s the boundary your essay keeps returning to.
War has always required human beings to carry that burden. The person who authorizes force knows, at least in principle, that the decision belongs to them. History can examine it. Courts can judge it. Conscience can wrestle with it.
Once machines move deeply into that pipeline, the burden begins to diffuse.
Not disappear exactly. It spreads. A piece of responsibility sits in the algorithm. Another with the engineer who designed it. Another with the analyst interpreting the output. Another with the officer who approves the strike. Each person operates inside a system that appears rational and efficient, yet no single human being stands clearly at the center of the decision.
That diffusion is what unsettles me.
Democracies rely on the ability to locate responsibility. When force is used in the name of a nation, someone must be accountable for it. If a system grows complicated enough that no one can clearly say, “This decision was mine,” something essential about democratic control of power begins to weaken.
Your essay also reminds me of something quietly troubling about human nature. We want machines to be wiser than they are. We’re eager to believe that a confident answer must come from understanding rather than statistical inference. The ELIZA story captures that perfectly. People confided in a program that was simply rearranging their own words.
That instinct won’t vanish simply because the systems grow more sophisticated.
The geopolitical trap you describe is real. No country wants to slow down if others accelerate. Yet the deeper question remains the one Weizenbaum asked decades ago.
Not what machines can do.
What they should never be allowed to decide.
Because the moment lethal judgment becomes a property of a system rather than a person, we may discover that responsibility can’t be coded into software.
It can only be carried by human beings.
Like Pandora, he lifted the lid just enough to see what was inside. And unlike most engineers of his era, he did not immediately convince himself that the contents were harmless.
What you describe about responsibility is the core of it. Democracies function because power can be traced. Someone signs the order. Someone answers for the consequences. History can examine the decision and assign judgment to a human being.
Once decision systems become layered enough that responsibility spreads across software, engineers, analysts, and commanders, that clarity begins to blur. Not through malice. Through architecture.
That is the danger Weizenbaum saw before almost anyone else.
The machines themselves are not the moral actors. They cannot be. The risk is that institutions begin to behave as though the system itself has authority. Once that happens, the people inside the system start to experience decisions as outputs rather than choices.
And when choices stop feeling like choices, accountability becomes very hard to hold onto.
Responsibility cannot be coded into software. It can only be carried by human beings who are willing to say, without qualification, “This decision was mine.”
The moment that sentence becomes impossible to say clearly, something foundational about democratic control of force begins to erode. That was Weizenbaum’s fear.
It is also why his warning still matters now.
That is what we are missing in our presidency at this moment. We have no way to hold Trump accountable.
Just sent this link off to my congressional rep, and senators. I think our extinction odds are growing grimmer by the day; but my latest t-shirt purchase states: Read Books, Walk Dogs, Fight Fascists. Onward.