Malware

Am I a man or a machine? My role in the hunt for AI

How taking part in a Turing Test to find the first thinking computer made Nick Heath realise he is more like a machine than he thought.

Yesterday, I spent two hours trying to act like a person.

Should be pretty easy - after all, I've had 34 years' practice. But it turns out that acting like an authentic human being is far trickier than it sounds.

I was one of four people chosen to be human testers in this year's Loebner Prize, the annual event where artificial intelligence software tries to pass the Turing Test, invented by the brilliant British mathematician Alan Turing in 1950 to identify a machine that can think. The 2012 event took place at Bletchley Park in the UK, where Turing worked as one of the codebreakers who helped the Allies win World War II.

During the event each judge has simultaneous instant message conversations with a remote person and software designed to chat like a human - and then has to decide which is which. The contest has been run for 22 years, and in that time the chat software hasn't managed to fool a third of the judges, which is the threshold that Turing set for identifying an intelligent machine.

My only way of convincing the judges that I was a fully paid-up member of the human race was through what I typed into a chat window. If I failed to make my answers relevant or my prose distinctive, I risked being written off as a chatbot, mechanically spewing out canned responses with little regard to the question.

The Loebner Prize, and to some extent the Turing Test itself, has been criticised by AI academics for lacking rigour and structure, and is considered by some to be more of a sideshow than a serious proving ground.

But even if the prize's intellectual credentials are in doubt, the event poses a fascinating question: how do we distinguish between human and artificial communication? It's a distinction that I found far trickier to make than I first thought.

As soon as the judge's "Hello" or "Hi" popped up on my screen, I was faced with a dilemma: do I go with a stock greeting or is that too predictable and exactly what some faceless bot would choose.

Because every word you type is broadcast online and people are milling around the human test room reading your messages over your shoulder, I found that the tone of my communications was more guarded and less natural.

Throughout the conversations I kept questioning my own responses. I repeatedly asked myself whether I was being spontaneous enough and whether I should signal my humanity by dropping in a colourful fact or quirky turn of phrase.

But often the easiest response to the judges' barrage of questions, including, "What was India like?" or "How did you find the journey here?" was the sort of generic blah that could emerge from a machine gluing together relevant subjects, verbs and objects.

As the contest continued, it struck me that, contrary to my preconceptions, much of what people say to each other isn't a pure expression of human individuality, but a sprinkling of fresh thoughts on a bed of reheated phrases and sentiment. A style of discourse that many people would describe as robotic.

While my conversations with other humans sometimes felt laboured, it was the bots that provided the truly crazy flights of fancy. What generally gave the chatbots away wasn't predictable comebacks or stilted tone, but their off-topic and outlandish replies.

One bot insisted that it was a cat, while another offered a judge condolences on the death of his pet dragon. At times the bots' whacked-out patter read like a bad parody of drug-addled conversation.

Typically within 15 minutes of starting the half-hour discussions, the judge would despair of the nonsense from my digital counterpart, telling me, "The other guy is terrible" or "You're clearly the human".

In short bursts the bots were fine - and could answer certain questions plausibly, without lapsing into nonsense.

One of the judges told me that if you edited together snippets of the conversations with the bots they would look perfectly plausible but, unedited, the whole artifice comes crashing down.

Admittedly, the bots are at a disadvantage. I could immediately tip off the judges to my authenticity by mentioning the tour of the venue that preceded the contest or the garish colours of organiser Hugh Loebner's Hawaiian shirt. Also, it takes only one slip-up by a bot to ruin an otherwise perfect dialogue and reveal its true identity.

Perhaps, as suggested by Brian Christian, who wrote a book inspired by his experience at an earlier Loebner Prize, a fairer test would be to have the humans remotely logging in over the internet.

Being a human tester at the event opened my eyes to how much human dialogue is predictable, but also to how a truly human-sounding chatbot needs to overcome far greater challenges than just coming off as robotic.

My idea of how bots communicate has been shaped by bad science fiction, shouting film names at automated cinema booking services and, more recently, Apple's pocket assistant Siri.

But the reality of the chatbot, unfettered and free to discuss any topic, is an entity struggling to even understand the question put to it, sinking in a sea of associated meanings and grasping at whatever floats by.

I was worried about sounding like a robot when it seems that not even the bots can manage that.

About

Nick Heath is chief reporter for TechRepublic UK. He writes about the technology that IT-decision makers need to know about, and the latest happenings in the European tech scene.

10 comments
turing_allenturing
turing_allenturing

Brian Christian, the guy you mentioned in your article, wrote the book on being the "The Most Human Human", and won that particular award in 2009. Did you win this year? if not, who best champion humanity for us real humans, in this year's contest?

LedLincoln
LedLincoln

...because you don't know whether I'm a human or a bot! :)

l_e_cox
l_e_cox

And encouraging, I think, for those who don't look forward to a machine takeover. If our understanding of "what makes a human" does not improve markedly, neither the robot takeover fanatics nor the skeptics who think it's impossible will get their way. They are both likely to fall prey to beings from somewhere who understand life better than we do. The real trick that the enemies of humanity could pull would be to pose as humans. The robot suit would be a bit of a giveaway.

Kevin917
Kevin917

Is probably one of the most human things we do. The very difficulty expressed in "trying" to type like a human is a very human thing. Watching the whole course of a typed conversation from start to end would probably show a shift from more stilted to less stilted as you become more comfortable. That alone would indicate a human participant IMHO. KJ

don.howard
don.howard

I know I don't have the whole context of the conversation, but am I the only one who thought, "The country or the ink?"

Gemmz
Gemmz

Computers can beat grand masters at chess. Now, I have to admit that I am not very good at chess. However, I can enjoy losing a game to someone who is better than me. I can enjoy the unfolding of the game and the demonstration of my opponent's skills. This is not as unconnected as it might seem. A computer will never enjoy the satisfaction of a well presented game that whilst lost, was closely fought. At the end of the game, the computer is switched off and put in a cupboard until needed. There is a lot more to thinking than a yes:no/right:wrong dichotomy.

dogknees
dogknees

The Turing Test is a lot more subtle than is being made out. The machine has to be able to fool any person and to do so on an ongoing basis. Turing didn't put a time limit on it, or specify the sort of things that could be discussed. It's not just question and answer. My approach would be to use phychologists, psychiatrists, linguists and others with similar expertise. Try to develop a friendship with the machine, push a few emotional buttons, try to rile it up,..... The other side of the question is whether the Turing Test is relevant. Are we trying to create a clone of a human mind, or something new? What if we don't want it to have human sensibilities, but to still be as intelligent, or more so? Which of course raises the question of whether something can have "intelligence" without the human traits. It's common for writers to describe an entity that acts in every way like a human, but doesn't have "true intelligence" or a true sense of self. I'm not sure that this is even possible. Not because we could not eventually build such a thing, but because the sort of intelligence we recognize includes our human foibles. For example, we assume all people will make mistakes under certain conditions. Will we require our creations to have this sort of built-in flaw before we consider them "intelligent"? Personally, I think we should be looking to define what intelligence is when it's not human-like. What intrinsic characteristics would an intelligent entity have before the average person would agree that it is intelligent? I'm not really interested in creating robot-slaves, but in the more esoteric area of general intelligent and consciousness. But that last word is whole other, possibly even more fraught, discussion.

wizard57m-cnet
wizard57m-cnet

"At times the bots??? whacked-out patter read like a bad parody of drug-addled conversation." Yep, sounds like some of the comments in the discussions here at TR! ;)

LedLincoln
LedLincoln

As to the Turing test being relevant, I can easily imagine a scenario of a company wanting to offer online chat tech support without paying someone to do it. The "intelligent" software could run through scripts as well as some of the tech support people I have had to work with. Maybe this is already being done.

mad tabby
mad tabby

Day after day I am deleting spam from my answering machine now, something I didn't have to do even 2 years ago. Pressing 0 on the phone no longer automatically connects you with a human. I think we're starting to see the beginning of it with the popups on certain sites "Hi, I'm Cindy, how can I help you today?" Although if Cindy started telling me she was a cat, I would probably log off.

Editor's Picks