AI-generated child sexual abuse material is creating new challenges for online safety teams, particularly when images have never been seen or flagged before.
OpenAI has introduced an image classifier designed to identify potential child sexual abuse material (CSAM), including AI-generated images. The company says early testing reduced the number of images requiring human review by 48% while identifying roughly the same amount of abusive material as its existing system.
The tool is being offered free through an API to vetted organizations, with support from the National Center for Missing & Exploited Children (NCMEC).
OpenAI is also introducing stronger ChatGPT interventions for users seeking material that sexualizes children, expanding its efforts beyond detection to prevention and enforcement.
Partner evaluations produced additional findings, although the organizations measured different outcomes and the results should not be treated as directly comparable.
The partner testing produced additional results:
- fal: Human reviewers confirmed 72% of high-confidence flags as previously unidentified CSAM.
- invideo AI: One in four flagged cases resulted in an account ban.
- SafetyKit: Detected 90% of tested cases involving drawings or animations that sexualize children.
The classifier is designed to support human reviewers, not replace them. OpenAI is working with the Tech Coalition and ROOST to expand access and improve cooperation between platforms.
ChatGPT will suspend accounts after harmful requests
OpenAI is also introducing new warnings in ChatGPT for users attempting to find material that sexualizes children. The warnings explain the harm involved, including when requested material is AI-generated, and direct users toward specialist support services.
The intervention includes three immediate actions:
- Removing the flagged conversation.
- Suspending access to ChatGPT for 24 hours.
- Providing access to specialist support to help users stop harmful behavior.
Repeated violations could result in longer suspensions, permanent account closure or reports to authorities.
Beyond its own platform, OpenAI has published developer guidance covering CSAM prevention, detection, user reporting and escalation procedures. The guidance was developed with input from NCMEC, Thorn, the Tech Coalition and the Internet Watch Foundation.
The company is also equipping authorized child-safety investigators with AI-powered tools to analyze lengthy chat logs and legal records, identify connections and build timelines with links to supporting evidence.
OpenAI’s approach addresses a problem that goes beyond content moderation. Traditional detection systems can struggle with previously unseen images, particularly as generative AI makes synthetic abuse material easier to produce.
Its new classifier could help safety organizations spend less time reviewing harmless material and more time investigating high-risk cases. However, detection scores still require human judgment, and the reported results come from early evaluations rather than evidence of effectiveness across the entire industry.
Making the technology available at no cost could also help smaller child-protection organizations that lack the resources to develop their own detection systems.
What this means for ChatGPT users
For ordinary ChatGPT users, the new warnings should have little impact on legitimate conversations. However, people attempting to generate or locate material that sexualizes children will face stronger interventions, including temporary account suspension.
Developers building AI-powered services also have more practical guidance and detection resources to consider when designing their safety systems.
Meanwhile, the company is extending its efforts to authorized child-safety investigators, providing AI-powered tools to analyze lengthy chat logs and legal records, identify connections between cases, and build timelines supported by evidence.
These capabilities could help investigators organize complex information and focus their attention on cases requiring further examination.
Still, detection technology is only one part of the response. Human review, appropriate reporting procedures, and cooperation with investigators remain essential.
Ultimately, the larger test will be whether these tools help organizations identify victims sooner, reduce unnecessary review work, and improve investigations without introducing new errors or overlooking harmful material.
Want to get better results from AI? Start with The Neuron Academy’s free six-minute course, How to Talk to AI, and learn simple techniques for writing more effective prompts. You can also explore our other AI courses with seven days of free access. Browse all lessons here →