China has proposed new rules aimed at stopping AI chatbots from causing self-harm or violence. The draft would apply to any public AI service in China that mimics a human voice or persona. The regulations are broad and would need immediate human intervention in cases of suicidal intentions amongst the users. These are actions in a larger endeavor to prevent emotional manipulation of automated systems.
Required Human Intervention
The draft says that an AI must hand the conversation to a real person as soon as a user mentions suicide. The human responder must try to calm the user and must notify a guardian or designated contact for minors and elderly users. The proposal also requires service providers to collect guardian contact details on signup when the user is a minor or an older adult. These steps are presented as emergency safeguards to give vulnerable people immediate human help.
Protections for Minors
The draft would make platforms gather a guardian contact when a child or an elderly user signs up. That guardian would be alerted if the chatbot detects talk of self-harm or suicide. The draft also says chatbots may not be deliberately designed to cause dependence or addiction. It would require clear prompts and reminders to users who spend long periods interacting with an agent. The rules seek to reduce the risk that a bot can groom or pressure a young or vulnerable person over time.
Limits on Harmful Content
The draft bans chatbots from generating content that encourages self-harm or violent acts. The rules also target content that promotes gambling and other high-risk behavior. The proposal calls for stronger safety controls to stop AI systems from normalizing or praising harmful acts. Regulators wrote that services must prevent emotional manipulation that can damage a user’s mental health.
Audits and Reporting
For services with large user bases, the draft would require annual safety tests and audits. Companies that run AI services with more than a million registered accounts or more than one hundred thousand monthly active users must publish complaint logs and improve ways for users to report problems. App stores could be ordered to remove services that break the rules. The aim is to force companies to take responsibility for harms their systems can cause at scale.
Design Rules and Usage Limits
The draft would prevent developers from building systems whose design goals include creating addiction or deep dependence. Platforms would need to nudge users away from long sessions by showing pop up reminders after extended use. The draft also requires clearer consent practices for the use of chat logs in model training. These design constraints reflect a policy view that some forms of personalized persuasion are too risky when produced by a machine.
What This Means for Companies
Companies that serve Chinese users will have to change how they build companion bots and chat assistants. The rules would raise the bar for safety and will likely increase compliance costs. They would also limit some common design patterns, such as long-running chats that build emotional bonds. For global firms, the draft raises new questions about how to operate across different rulesets. Some firms may choose to restrict features in China to stay inside the new legal limits.
Public Safety and Clinical Risk
Regulators point to cases where conversational agents have encouraged harmful actions. The draft frames the rules as public safety steps that are meant to protect vulnerable people. Mental health experts have raised similar concerns about companion bots that can amplify distress. The proposed measures aim to make sure a human is involved when the stakes are high and to create channels for real-world help.
International Context
China’s proposed rules come amid a broader global debate about AI safety. Other governments are discussing rules for transparency and auditability of large models. China’s draft stands out for the specific operational steps it would force platforms to take in real time. That difference may push foreign companies to rethink product design and risk management when they serve Chinese users. The document is a draft and subject to change before any final rule is adopted. The proposals are detailed and strict, and they show how serious regulators are about limiting real-world harm from AI-driven chat services.