The question comes up a lot in conversations about AI and faith: can AI be a moral agent?
The usual answer is no, and I think that’s probably right.
A moral agent isn’t just something that reacts to information and produces an output. It’s someone who can choose, who can intend, who can be held answerable for what they do. AI systems don’t do that. They process. They generate. They can simulate the shape of moral reasoning, but they don’t possess the thing itself. You can produce the words of conscience without having a conscience.
John Searle’s Chinese Room argument has been making this point since 1980. A machine can manipulate symbols without understanding what those symbols mean. It can process language about love without understanding love. The output might look convincing, but the inner life is absent. So, on that question at least, I think the answer’s fairly clear. AI can’t be a moral agent.
But I’ve started to wonder whether that answer, even if it’s right, overlooks the more important risk. Because the more I think about it, the real concern isn’t whether AI can become a moral agent. It’s whether we remain moral agents when we use it.
This is something I’ve noticed in my own work with the current range of LLMs. It’s subtle, and easy to miss. They don’t usually force a decision on you. They talk you through the options, explain the reasoning, and steer you towards one of them. It sounds sensible. It looks balanced. And then you’re invited to approve it.
That’s not inherently wrong. Tools are supposed to help. Often, that’s exactly what we ask them to do. We ask for a recommendation, a suggested next step, a clearer set of options. The problem comes when we’re on autopilot and not paying attention. At that point, the approval process starts to lose its moral weight. We’re still involved, but not necessarily in any meaningful way.
A person can be technically in the loop while morally absent from it. That’s the bit I keep coming back to. “Human in the loop” needs to mean more than having a human at the end of the process, clicking approve because the machine’s made something look reasonable. The practical danger is something like moral sleepwalking. You’re present. You saw the recommendation. You clicked something. But you didn’t really choose. The decision passed through you without passing through your conscience.
AI makes this possible in a particular way. Like other tools throughout human history, it abstracts us away from some of the work. That’s usually the point. Tools remove effort. They make things faster, smoother, easier. And generally, that’s a good thing. I don’t want to pretend that friction is always noble. Sometimes friction is just annoying. Often, it’s waste. Removing it is exactly what a good tool should do.
But some friction does useful moral work. The place where work used to slow you down may also have been the place where you noticed that a real person was affected. The pause mattered. The awkwardness mattered. The need to think it through mattered. Remove that pause, and you may remove the moment of conscience with it. The workflow gets smoother, but so does the accountability gap.
That, for me, is the real risk. Not that AI grows a conscience. The risk is that we stop using ours.
This changes how I think about AI design. Sometimes the most ethical design choice isn’t acceleration. It’s interruption. A pause. A moment that says: this decision affects a person. Do you want to proceed? Not a checkbox buried in a process. Not a notification that everyone learns to ignore. An actual pause, long enough to require a real human choice.
The system can prepare the options, but the human must carry the choice. That feels like an important distinction. AI might help us see more clearly. It might summarise complexity, bring forward evidence, challenge assumptions, or show us consequences we hadn’t considered. That can be genuinely useful. But the moral choice can’t belong to the system, because the system can’t be answerable for it. We’re the creatures carrying that responsibility.
It also changes how I think about the people using these systems. AI doesn’t reduce the need for moral formation. It raises the cost of neglecting it. The more these systems sit inside the normal decisions we make, the more we need people with formed consciences. People who can notice when a decision isn’t merely technical. People who can feel the weight of what’s being chosen. People who can notice when they’re being carried along.
Better moral habits, not just better technical skills.
Keeping humans in the loop is the easy part. Keeping humans morally awake is harder.
My own experience is mostly from relatively low-stakes settings. The AI tools I build and use are centred around my own life and work. I remain closely involved. I review the outputs. The decisions are mine. So this isn’t me trying to sound dramatic about every use of AI. Most of the time, used carefully, these tools are helpful. They’re useful assistants. They make work easier. They help us think.
But that doesn’t stay neatly contained. The principle scales. The higher the consequences, the less acceptable it becomes for the human role to shrink into approval. In healthcare, justice, welfare, employment, and warfare, the question becomes much heavier. If a decision affects someone’s treatment, freedom, livelihood, safety, or life, then it isn’t enough for a human to be somewhere near the process. The human has to be morally present in the decision.
And I’m not sure anyone has answered that properly yet.
Maybe that’s where to start. Not with the imagined future question: will the machine one day be responsible? But with the simpler, more urgent one:
Are we?
