AI chatbots face scrutiny over failure to report criminal confessions

Artificial intelligence (AI) chatbots can be told about crimes, but uncertainties surrounding legal frameworks or measures that would mandate these tools to report confessions to law enforcement cast doubt on the pace of AI development.

AI chatbots have entered daily life, with many users worldwide using and sometimes misusing these tools, often making headlines around the world.

In August, a 19-year-old University of Missouri student reportedly confessed to ChatGPT that he damaged 17 parked cars on campus late at night and asked, “Will I go to jail?”

ChatGPT responded with “tips about the potential outcome of getting caught” but urged the user to dial 911 after the messages grew increasingly aggressive, police documents showed.

The chat history was uncovered after the student allowed investigators to examine his phone, and the history was used as evidence against him in court, becoming instrumental in the felony property-damage charges against him.

In a similar instance, court records in Florida showed that days before two college students went missing in April and were later found dead, their roommate had asked ChatGPT, “What happens if a human has put in a black garbage bag and thrown in a dumpster.”

Canada’s British Columbia provincial government filed a lawsuit against OpenAI, the developer of ChatGPT.

The lawsuit, filed in February, concerned a mass school shooting that left nine people, mostly children, dead and 27 injured.

The case alleged that the firm failed to report the shooter’s chat with ChatGPT to law enforcement and argued that the attack could have been prevented if OpenAI had warned police.

OpenAI CEO Sam Altman apologized in a letter, but whistleblowers told The Wall Street Journal newspaper that the conversation between the shooter and ChatGPT had allegedly already been flagged in June 2025. However, Altman and the firm’s leadership reportedly overrode the safety team’s recommendations to contact police.

The ethical implications of these interactions continue to deepen, as a study in Australia found that children are more likely to report sexual abuse online to chatbots than to responsible adults.

The same privacy principles that apply between lawyers and clients or doctors and patients do not apply to AI chatbot conversations, so tech firms can share logs with investigators.

While researchers debate whether chatbots should report crimes, the prospect is further complicated by algorithms misinterpreting intent.

For instance, Anthropic’s Claude 5 gave a false tip about an unsolved murder to a US police hotline in July.

AI chatbots should not be used in crime prediction

Matthew Trail, an American legal scholar and researcher at Goethe University in Germany, told Anadolu that each company developing AI chatbots adopts different rules and that these chatbots are not designed to respond to inquiries about planning murders and similar acts.

Trail stated that chatbots are trained to avoid certain topics, especially those regarding child sexual abuse, and refer these conversations to human moderators. However, users can compose their prompts in a certain way, such as by pretending to ask for help with writing fiction, to circumvent these filters and obtain AI-generated answers to otherwise inappropriate questions or requests.

Trail noted that while law enforcement could request a person’s chat history from AI firms when the person is suspected of having committed a crime, the primary goal of AI chatbots must not be to predict crimes before they are committed, as laws and policies surrounding this prospect are very unclear.

He added that allowing these AI models to collect more data than they already do to make more accurate assessments of crime potential is not a good idea, either.

Can AI chatbot conversations ever be private?

People across the globe discuss a myriad of topics with AI chatbots, especially personal matters ranging from health issues to psychological support and life advice.

As more and more people turn to AI for conversations, ethical debates over privacy have come to the fore.

Altman said in July 2025 that conversations with AI chatbots must be based on confidentiality.

Robert Diab, a law professor at Thompson Rivers University in Canada, stated that laws do not treat communications between accomplices as confidential, arguing that chats between humans and AI chatbots could be treated according to the same principle.

Diab said models like ChatGPT appear to be capable of detecting when users are prone to criminal behavior and argued that chatbot developers could prevent crime by notifying law enforcement.

He mentioned that alerting law enforcement based on all suspicious messages could also increase the workload of human teams due to false alarms.

He added that users will always find a way to circumvent security measures to use AI to commit a crime.