AI has seen a renaissance over the last year, with developments in driverless vehicle technology, voice recognition, and the mastery of the game "Go," revealing how much machines are capable of.
But with all of the successes of AI, it's also important to pay attention to when, and how, it can go wrong, in order to prevent future errors. A recent paper by Roman Yampolskiy, director of the Cybersecurity Lab at the University of Louisville, outlines a history of AI failures which are "directly related to the mistakes produced by the intelligence such systems are designed to exhibit." According to Yampolskiy, these types of failures can be attributed to mistakes during the learning phase or mistakes in the performance phase of the AI system.
Here is TechRepublic's top 10 AI failures from 2016, drawn from Yampolskiy's list as well as from the input of several other AI experts.
1. AI built to predict future crime was racist
The company Northpointe built an AI system designed to predict the chances of an alleged offender to commit a crime again. The algorithm, called "Minority Report-esque" by Gawker (a reference to the dystopian short story and movie based on the work by Philip K. Dick), was accused of engaging in racial bias, as black offenders were more likely to be marked as at a higher risk of committing a future crime than those of other races. Another media outlet, ProPublica, found that Northpointe's software wasn't an "effective predictor in general, regardless of race."
2. Non-player characters in a video game crafted weapons beyond creator's plans
In June, an AI-fueled video game called Elite: Dangerous exhibited something the creators never intended: The AI had the ability to create superweapons that were beyond the scope of the game's design. According to one gaming website, "[p]layers would be pulled into fights against ships armed with ridiculous weapons that would cut them to pieces." The weapons were later pulled from the game's developers.
3. Robot injured a child
A so-called "crime fighting robot," created by the platform Knightscope, crashed into a child in a Silicon Valley mall in July, injuring the 16-month-old boy. The Los Angeles Times quoted the the company as saying that incident was a " freakish accident."
4. Fatality in Tesla Autopilot mode
As previously reported by TechRepublic, Joshua Brown was driving a Tesla engaged in Autopilot mode when his vehicle collided with a tractor-trailer on a Florida highway, in the first-reported fatality of the feature. Since the accident, Telsa has announced major upgrades to its Autopilot software, which Elon Musk claimed would have prevented that collision. There have been other fatalities linked to Autopilot, including one in China, although none can be directly tied to a failure of the AI system.
5. Microsoft's chatbot Tay utters racist, sexist, homophobic slurs
In an attempt to form relationships with younger customers, Microsoft launched an AI-powered chatbot called "Tay.ai" on Twitter last spring. "Tay," modeled around a teenage girl, morphed into, well, a " Hitler-loving, feminist-bashing troll"—within just a day of her debut online. Microsoft yanked Tay off the social media platform and announced it planned to make "adjustments" to its algorithm.
SEE: Big data can reveal inaccurate stereotypes on Twitter, according to UPenn study (TechRepublic)
6. AI-judged beauty contest is racist
In "The First International Beauty Contest Judged by Artificial Intelligence," a robot panel judged faces, based on "algorithms that can accurately evaluate the criteria linked to perception of human beauty and health," according to the contest's site. But by failing to supply the AI with a diverse training set, the contest winners were all white. As Yampolskiy said, "Beauty is in the pattern recognizer."
7. Pokémon Go keeps game-players in white neighborhoods
After the release of the massively popular Pokémon Go in July, several users noted that there were fewer Pokémon locations in primarily black neighborhoods. According to Anu Tewary, chief data officer for Mint at Intuit, it's because the creators of the algorithms failed to provide a diverse training set, and didn't spend time in these neighborhoods.
8. Google's AI, AlphaGo, loses game 4 of Go to Lee Sedol
In March 2016, Google's AI, AlphaGo, was beaten in game four of a five-round series of the game Go by Lee Sedol, a 18-time world champion of the game. And though the AI program won the series, Sedol's win proved AI's algorithms aren't flawless yet.
"Lee Sedol found a weakness, it seems, in Monte Carlo tree search," said Toby Walsh, professor of AI at the University of New South Wales. But while this can be considered a failure of AI, Yampolskiy also makes the point that the loss "could be considered by some to be within normal operations specs."
9. Chinese facial recognition study predicts convicts but shows bias
Two researchers at China's Shanghai Jiao Tong University published a study entitled "Automated Inference on Criminality using Face Images." According to the Mirror, they "fed the faces of 1,856 people (half of which were convicted violent criminals) into a computer and set about analysing them." In the work, the researchers concluded that there are "some discriminating structural features for predicting criminality, such as lip curvature, eye inner corner distance, and the so-called nose-mouth angle." Many in the field questioned the results and the report's ethics underpinnings.
10. Insurance company uses Facebook data to issue rates, shows bias
And, finally, this year England's largest vehicle insurer, Admiral Insurance, set out to use Facebook users' posts to see there was a correlation between their use of the social media site and whether they would make good first-time drivers.
While this isn't a straight AI failure, it is a misuse of AI. Walsh said that "Facebook did a good job in blocking this one." The endeavor, which was called "firstcarquote," never got off the ground because Facebook blocked the company from accessing data, citing its policy that states companies can't "use data obtained from Facebook to make decisions about eligibility, including whether to approve or reject an application or how much interest to charge on a loan."
As evidenced by these examples, AI systems are deeply prone to bias—and it is critical that machine learning algorithms train on diverse sets of data in order to prevent it. As AI increases its capabilities, ensuring proper checks, diverse data, and ethical standards for research is of utmost importance.
- Why Microsoft's 'Tay' AI bot went wrong (TechRepublic)
- AI can predict outcome of human rights trials, but should it? (TechRepublic)
- #ILookLikeAnEngineer: How women are using social media to bust stereotypes and redirect the STEM conversation about gender (TechRepublic)
- Can AI really be ethical and unbiased? (ZDNet)
Hope Reese has nothing to disclose. She doesn't hold investments in the technology companies she covers.
Hope Reese is a journalist in Louisville, KY. Her writing has been featured in The Atlantic, The Boston Globe, The Chicago Tribune, Playboy, Undark Magazine, VICE, Vox, and other publications.