Some ChatGPT conversations are being reviewed by human contractors as part of OpenAI’s model-improvement work, according to leaked internal materials reported by 404 Media.
Under an internal project reportedly called “Project Lily,” reviewers grade model responses, summarize user intent, and flag behaviors such as robotic phrasing or excessive agreement. The materials also indicate that OpenAI uses automated privacy filtering before conversations reach reviewers.
Consumer AI chats should not be treated as private correspondence, especially when conversations may be used for model improvement.
Inside Project Lily
According to 404 Media, contractors earn over $50 an hour through intermediary firms like Crossing Hurdles and Mercor to review selected conversation streams and score generated replies from 1 to 7.
Documents show reviewers are trained to flag “AI-speak,” weed out sycophancy, and eliminate unnecessary emojis, such as lists packed with green checkmarks.
OpenAI scrubs usernames and routes text through an automated tool called Privacy Filter to remove identifying details before conversations reach reviewers, although the system may not catch every rare identifier or ambiguous reference.
OpenAI acknowledges the filter can fail on rare identifiers or ambiguous phrasing, and contractors can still view a “user memories summary” displaying an individual’s past interests and approximate location.
Asked whether users realize human reviewers may see their conversations, one contractor told 404 Media, “No, I don’t think they would imagine some contractor somewhere […] is analyzing the conversations.”
The regulatory fault line
The disclosure could intensify privacy and regulatory scrutiny around how consumer AI conversations are processed and disclosed.
As The Next Web noted, the Court of Justice of the European Union held last September in EDPS v SRB that a data controller’s duty to inform users applies at the exact moment of collection, regardless of whether a downstream recipient can directly identify someone.
OpenAI—already fined 15 million euros by Italian data regulators for processing without a proper legal foundation—faces compounding scrutiny because consumer tiers leave data sharing active by default. The “improve the model for everyone” toggle is enabled automatically on Free, Plus, and Pro tiers, while corporate Enterprise, Business, and Edu accounts are opted out by default.
Why conversational AI creates a privacy blind spot
The privacy tension comes from how conversational AI is designed. Chatbots can feel more personal than traditional software, encouraging users to share context they might not enter into a search box or form, while some conversations may still be used in model-improvement workflows.
When an interface simulates human empathy, consumers naturally lower their guard, treating the dialogue box like a personal diary or confidential advisor.
This creates a sharp structural divide. OpenAI applies different data-handling defaults across its products. Enterprise, Business, and Edu accounts are not used for model training by default, while consumer users may need to disable model-improvement settings themselves.
That distinction matters for companies deciding whether employees should use consumer AI accounts for work involving confidential or regulated information.
Turning off model-improvement sharing affects future conversations, but users should not assume the change retroactively removes data that has already entered existing processing or review workflows.
Industry norms and consumer impact
OpenAI is not alone. Rivals Anthropic and Google Gemini also employ human reviewers for chat optimization under specific account settings. However, critics argue the lack of explicit, direct prompts warning users before they hit send creates an avoidable trap.
For regular users, the practical rule is simple: do not enter passwords, financial details, health information, confidential company data, or other sensitive material into a consumer AI service unless you understand how that data will be handled.
ChatGPT users should also review their model-training settings. For workplace use, businesses should rely on approved AI products and organizational policies designed for sensitive information rather than assuming a consumer chatbot is private by default.
In other AI news, Anthropic CEO Dario Amodei called for slower frontier AI development, with Sam Altman and Elon Musk backing the broader push as safety and cybersecurity concerns grow.