A 16-year-old student in Baltimore, Maryland, was handcuffed by police after an artificial intelligence (AI) system wrongly identified a bag of chips as a firearm.
The incident has reignited debate over the accuracy, reliability, and ethics of AI-based weapon detection systems in US schools.
“Police showed up, like eight cop cars, and then they all came out with guns pointed at me talking about getting on the ground,” said student Taki Allen, in an interview with local media outlet WMAR-2 News.
False alert, real consequences
The event underscores the risks of deploying untested or overconfident AI surveillance tools in sensitive public spaces.
According to the Baltimore County Police Department, officers “responded appropriately and proportionally based on the information provided at the time.”
However, a later review revealed the alert was a false alarm — the system had confused Allen’s chip bag for a firearm.
BBC reported that the AI alert, provided by Omnilert’s technology, was reportedly sent to human reviewers who found no threat. Yet the school principal failed to see the “no threat” update and contacted the school resource officer, who, in turn, called local law enforcement.
The miscommunication led to the arrival of armed officers on school grounds, escalating a non-event into a traumatic experience for the student.
Procedural failure, not system flaw?
In a statement to BBC News, Omnilert said it “regrets this incident occurred and wishes to convey our concern to the student and the wider community affected.”
The company emphasized that its system “operated as designed” and that its human verification process worked correctly. The failure, it said, came later in procedural handoff.
While Omnilert defends its technology, the company admits that “real-world gun detection is messy.”
AI models rely on training data that may not encompass every lighting condition, object shape, or color variation. In this case, the system’s visual model apparently could not distinguish the reflective surface of a chip bag from a firearm.
More must-read AI coverage
- SS&C Intralinks DealCentre AI vs. Datasite: Which platform is built for the future of dealmaking?
- SS&C Intralinks FundCentre AI vs. Juniper Square: Which platform better supports modern private markets fund managers?
- Why Data, Not Models, Determines AI Success
- The Rise of the AI-Native Factory: How Physical AI Is Transforming Manufacturing
Beyond just privacy and compliance
The misidentification highlights a growing problem with AI in safety and law enforcement: false positives can lead to dangerous or traumatic consequences in the real world.
Cybersecurity governance now extends beyond data privacy and system security. It must also ensure ethical deployment of AI. This includes auditing algorithms for bias, testing for real-world accuracy, and establishing transparent escalation procedures.
Without proper oversight, the rapid rollout of AI surveillance tools could amplify human error rather than reduce risk. AI ethicists argue that systems intended to protect should undergo the same level of scrutiny as cybersecurity defenses.
How schools can mitigate risk
To prevent similar incidents, school districts and organizations adopting AI detection tools should apply a layered approach that balances safety with ethical responsibility:
- Implement human-in-the-loop validation: Ensure all AI alerts are reviewed by trained personnel before police involvement and require a second set of human eyes before contacting law enforcement.
- Regularly audit AI models: Test systems under varied real-world conditions to evaluate false positive rates and bias.
- Establish clear escalation policies: Define communication chains between AI system operators, school staff, and law enforcement to prevent missteps.
- Enhance transparency: Share AI accuracy metrics and review findings with parents and the community to build trust.
- Adopt ethical AI frameworks: Incorporate accountability, fairness, and explainability requirements into vendor contracts and governance policies.
Together, these measures help ensure AI-driven security systems operate responsibly, minimizing harm while maintaining trust.
When automation outpaces accountability
As AI technologies expand into policing, hiring, and education, their errors can carry disproportionate consequences. Baltimore’s chip incident illustrates how a system meant to prevent violence can instead inflict harm through misinterpretation and procedural failure.
The rapid adoption of AI in schools and public safety sectors demands stronger regulatory frameworks, standardized accuracy testing, and third-party auditing. Mistakes like this highlight why ethical oversight is a fundamental requirement of safe AI deployment.
The Baltimore incident serves as a cautionary tale for all organizations integrating AI into decision-making and security processes. As AI systems grow more autonomous, human accountability must keep pace.
Editor’s note: This article first appeared on our sister publication, eSecurityPlanet.com.