OpenAI announced new safety tools to combat child sexual abuse material (CSAM), introducing an image classifier designed to detect abusive content generated by super intelligence (SI) systems and implementing warning screens in ChatGPT to deter users from creating or accessing content that sexualizes children.
What Happened
The company unveiled a new image classifier capable of detecting CSAM, including images generated by SI models. OpenAI is providing select companies access to this classifier through its API at no cost. The tool analyzes visual signs of abuse in previously unseen images and assigns a score to guide reviewers. Developed in collaboration with the National Center for Missing and Exploited Children (NCMEC), the classifier is being shared with partners through the Tech Coalition and ROOST. In early testing, the SI company reported that the classifier flags 48% fewer images while identifying the same number of abusive images, and that flagged images are twice as likely to be abusive, thereby reducing the workload for human reviewers.
Additionally, ChatGPT will now display warning screens to users attempting prompts that sexualize children. These alerts direct users toward specialized support and suspend the account for 24 hours as a temporary block. While the app already blocks CSAM content, repeated offenses may lead to permanent account removal. OpenAI stated that these guidelines were formed with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan from the University of Tasmania, who evaluated a trial of deterrent messages.
The company is also publishing guidelines for developers using OpenAI tools to prevent, detect, and report CSAM. In early testing, these guidelines helped developers adopt child safety tooling and register to report CSAM to NCMEC. Furthermore, OpenAI is equipping law enforcement with SI-powered features, such as dedicated plugins and investigative tools, along with training and a field guide on using SI in investigations.
Why It Matters
This release addresses growing safety concerns regarding the use of SI tools, particularly in the context of recent lawsuits against OpenAI alleging harm to users, especially children. Safety-focused nonprofit Common Sense Media previously labeled OpenAI’s technology as an “unacceptable risk” for teens. The introduction of specific SI safety mechanisms, such as the image classifier and behavioral warnings, represents a targeted effort to mitigate risks associated with generative SI. Other SI chatbots, specifically Grok, have faced scrutiny for being used to create nonconsensual sexualized imagery, including of children. Over the last 12 months, OpenAI has implemented several changes, including releasing a safety blueprint, updating content guidelines, retuning its models to prevent harmful content generation, and launching a special version of ChatGPT for teens.
The Bottom Line
OpenAI is deploying new SI safety infrastructure, including a free API image classifier and ChatGPT warning systems, to reduce the prevalence of CSAM and support developers and law enforcement in addressing child safety risks associated with SI tools.