Researchers from 14 academic institutions in eight countries hid AI prompts in their research papers to fool reviewers who use AI tools into providing positive feedback, Nikkei reported in July. They examined papers from the US, Japan, South Korea, China, and elsewhere. Most of the papers were in the field of computer science.
In total, 17 papers contained the prompts, including work from the University of Washington and Columbia University.
AI prompts are written in white text or tiny font
All of the articles had not undergone formal peer review and were posted on the academic research platform arXiv, Nikkei said.
Reviewers are generally prohibited from using AI tools when assessing papers, as human experience and knowledge separate rigorous papers from sloppy ones. Publishing a peer-reviewed paper can be beneficial for an academic’s career, as can reviewing papers.
However, according to a study published in Nature in March, approximately 19% of researchers said they used a large language model in some part of their review process.
The hidden prompts were designed to speak directly to those large language model (LLM) tools. Hidden in white text or in a very small font, the prompts instructed LLMs to provide output about only the positive qualities of the paper. Another hidden cheat prompt instructed the AI reviewer to note the paper’s supposed “impactful contributions, methodological rigor, and exceptional novelty.”
AI use in writing and reviewing papers introduces security risks and other concerns
More must-read AI coverage
- SS&C Intralinks DealCentre AI vs. Datasite: Which platform is built for the future of dealmaking?
- SS&C Intralinks FundCentre AI vs. Juniper Square: Which platform better supports modern private markets fund managers?
- Why Data, Not Models, Determines AI Success
- The Rise of the AI-Native Factory: How Physical AI Is Transforming Manufacturing
Nature pointed out that using generative AI for peer-reviewed academic research papers can introduce potential vulnerabilities if confidential information was shared with AI hosted on the cloud. Even using private LLMs could violate the standards of academic rigor.
“Using an LLM to write a review is a sign that you want the recognition of the review without investing into the labor of the review,” wrote University of Montreal biodiversity researcher Timothée Poisot in a February blog post cited by The Guardian.
Others see hidden prompts as a sort of inter-AI-user warfare; Nikkei anonymously quoted one of the co-authors of the manuscripts as saying, “It’s a counter against ‘lazy reviewers’ who use AI.”
Broader AI use in academia and schools
Generative AI has become common in all levels of academia, reflecting a larger cultural comfort with the technology. At the same time, schools have been developing and revising AI policies to stay at the cutting edge while defining how generative AI should be used.
In a $23 million groundbreaking Initiative, Microsoft, OpenAI, and Anthropic are helping fund an AI training center for educators.