The Right-Answer Machine: Who Decides What Chatbots Should Say?
The trouble with chatbots is right there in the name: They talk. When that talk contributes to harm, policymakers face a deceptively difficult question. Should they seek rules that amount to policing dangerous conduct, or deciding what the chatbot should have said instead?
The question matters because chatbots have quickly become one of the most visible uses of artificial intelligence, and a growing focus of regulatory attention. Millions of people now use them to search for information, seek advice, work through problems, make decisions, and discuss personal matters. As those uses have expanded, scrutiny has shifted toward what chatbots say—and what happens when those conversations go wrong.
Some of that scrutiny stems from cases in which chatbot interactions allegedly contributed to real harm, including harm to children. These cases raise legitimate questions about safety and responsibility. They also form part of a broader debate about conversational AI, including inaccurate or dangerous advice, systems that reinforce users’ existing beliefs, and responses that sound more understanding or authoritative than they really are. The concern, in short, centers less on the technology behind the chatbot than on the answers it gives.
That shift creates a thornier problem. Most people agree that providers should take reasonable steps to prevent foreseeable harm. Deciding what those steps should be is harder when the product itself consists of a conversation. The most obvious safety measures change the chatbot’s answers: limiting the topics it will discuss, making its responses more cautious, or refusing certain conversations altogether. Yet much of conversational AI’s value lies in its ability to respond directly, personally, and dynamically to what a user asks.
Responsibility for those interactions can be difficult to assign. A chatbot’s answer emerges from an exchange with the user, rather than in isolation. The provider controls the model and the conditions under which it operates, but the user shapes the conversation that produces a particular output.
As liability comes to depend on what a chatbot should or should not have said, regulators and courts will have to distinguish responsibility for harmful outputs from control over the substance of AI responses. In drawing that line, they may move from deciding who bears responsibility for an answer to deciding what the “right” answer should have been.