Enterprise Software

Five things you need to know before building an IBM Watson app

Developers give tips on what steps you should take when building a service on top of the quiz show-winning cognitive computing platform.

IBM's Watson may have schooled two quiz show champs in the art of winning Jeopardy but how can it help your business?

Big Blue wants firms to employ Watson as a pocket expert - one that can imbue new shop assistants with the knowledge of a lifelong employee or help doctors choose personalised therapies for patients.

Companies wanting to tap Watson's ability to learn from natural language in documents and questions can plug applications into the seven APIs IBM has made available. These interfaces expose Watson's core ability to provide credible answers to everyday questions, alongside services ranging from machine translation to user modelling, which categorises people based on their email or public posts.

TechRepublic had a chance to ask developers at IBM and Red Ant, one of the first companies to build a service on top of Watson's APIs, about how firms can get the most out of the system.

Feed Watson everything you can

To answer a question with any certainty Watson needs to be fed relevant information and lots of it.

"If you think of how we did Jeopardy it wasn't a hard-coded set of rules or answers to look up. We didn't pre-empt 'Here are all the questions that Jeopardy might ask', put them in a big container and pull them out," said Dale Lane, IBM Watson software developer.

"To do general knowledge Watson had to extract the knowledge from a corpus of text, so we gave it access to encyclopaedias and dictionaries. All of Encarta, all of Wikipedia, the Oxford English Dictionary, books and magazines - the New York Times, Time magazine and so on."

For businesses these documents, rather than encyclopaedias, are likely to be press releases, technical manuals, staff training booklets - anything that contains information that might help Watson understand what a business does.

This information - which can be in form of Word docs, PDFs and web pages - can be uploaded via a browser-based tool.

"You have to start with a good wide selection of documentation but the good thing is you can be quite indiscriminate and say 'Give me every single document you can get your hands on, from training manuals to customer reviews to product descriptions'," said Alex Sbardella, product strategy director with Red Ant.

Red Ant has created a Watson-based tablet app that works alongside its RetailOS service to help shop assistants answer customer queries. The app will underpin an in-store Q&A service for a telecoms retailer in the UK. Assistants in 20 of the retailer's stores will be able to defer customer questions to Watson as part of a trial, ahead of the system being rolled out to every outlet early next year.

Training can also take the form of giving Watson sample questions and answers, in the case of preparing for Jeopardy the system was fed 20,000 questions from previous shows.

"Like with a child, you need to give lots of examples of getting things right and getting things wrong," said Lane.

watson-jeopardy-940px.jpg
IBM Watson defeated two of Jeopardy's greatest champions.
Image: IBM

"Think of the type of questions you think the system is going to get and train the system how to answer those questions by giving it examples [of those questions] and the right answer."

Ideally organisations should feed Watson questions and answers based on real-life exchanges, he said.

"It's better like that because you are getting questions that are representative instead of someone trying to guess what might someone ask.

"We have self-serve, cloud-hosted tools that allow you to write the questions and give Watson what the right answer would be. It's not about building a rules engine, what you're giving it is experiences to drive its learning."

Provide expert coaching

Feeding Watson information is only the first stage of its training. The machine also needs help working out which parts of that text is relevant.

This stage requires a subject matter expert - for example an experienced call centre rep - to highlight to Watson which sections of the training material are most relevant when answering the questions businesses want Watson to handle.

"You're teaching it 'When I get this sort of question, this where I would look and this is the bit I would pick out of this source'," said Lane.

"We're giving clients the tools to be able to do that for themselves, because obviously they're the subject matter experts."

While it took about five days for Red Ant to build a prototype service on top of Watson's APIs, it took months to train that service Watson to answer questions at the telecoms retailer.

"The skill and the challenge, the interesting bit, is the training, that's the lion's share of the work," said Sbardella.

"It's very different to what we're normally used to, where development is normally 80 percent of the work now it's about 30 to 40 percent and the rest is in the training. We've probably spent about two to three months training our first instance.

"The good thing about retail is they already have a load of training materials because every single new employee they have goes through a training process and Watson is basically just the new guy.

"He comes in and looks at the documentation you have available, he asks questions and you tell him the right answer, and it's exactly the same for Watson."

Training up Watson to the point where it is fit to answer questions requires both time and people.

"Depending on how many resources you have you can scale up very easily, you just need lots of people inputting questions or lots of people looking through documents," said Sbardella.

"The interesting thing about Watson is that normally a system is best on the first day you put it in. With ours it's the worst. We're very early on in that journey and within your top five answers you'll have your right answer 80 -90 percent of the time. Obviously we want to get that up to 99.9 percent of the time and for it to always be the first answer."

Watson is only as good as the trainer

As tedious as it may be to help a know-nothing computer learn the ropes, this isn't a role that can be palmed off to the intern.

To an extent, Watson will only be able to answer questions as well as the person that trained it, and the trainer needs to be able to identify which parts of a corpus contain useful information and which are marketing fluff.

"A non-subject matter expert is going to write me 2,000 questions that is sucking stuff out of the press release [for the answer]," said Lane.

"Watson will learn whenever I get asked a question press releases are the place to go, never go near these technical documents as I never get anything useful out of them. Even if it does try to use those documents it won't use them very well because it won't have learned how to.

"Someone with expertise in the subject needs to identify from that corpus what bit is the right answer.

"We ask them [the trainer] to say 'This is where I would look for the answer, maybe highlight the passage they think is appropriate and so on'. With enough experiences like that it will teach the system to be able to do that role."

"Make sure you've got someone who really is an expert and make sure you are driving them to not just write the questions that are easy to write, but the questions you want the system to implement."

"This is not a one-time exercise, it's really an ongoing iterative approach," said Watson CTO and chief architect Sridhar Sudarsan.

Train Watson on the job

Watson's manual training shouldn't finish as soon as its deployed in a business, but should continue with employees providing feedback to the system about how good a job it's doing.

"The training doesn't have to be one-off, Watson can learn from the responses it gets from users," said Lane.

The retailer using Red Ant's service has put in place a simple thumbs up or down rating system that the store assistant can use to provide feedback on the quality of Watson's answer.

"If you're a retailer you already have customers coming in and asking lots of questions," said Sbardella.

"Every answer that comes back you can vote whether it's correct. In retail if it's reporting an out-of-date price it's very important we can react to that quickly.

"If you deployed this to 200 store staff around the country you've now got 200 people voting or down voting, marking answers as correct or incorrect."

It's crucial to make it as easy as possible for the users to question Watson and to give feedback on its performance, he said.

"It's really important to design it in such a way that it feels natural and it's not a huge ask," referencing the simplicity of the voice recognition system that allows store assistants to ask questions of its Watson-based service verbally, alongside the thumbs/up down rating system for the answers Watson provides.

"It's about smart ways of scaling it up, so the impact on the business is essentially negligible. Theoretically it's a huge change in the way the business operates and we want to try and minimise that disruption as much as possible."

Buy in outside help

Eventually IBM hopes businesses will be able to select much of the content needed to train Watson from an online repository it calls the Content Marketplace. This will host domain-specific knowledge Watson needs, but at present is limited to a small amount of content related to the healthcare and travel industry.

As well as providing an online repository for documents, IBM also plans to link businesses up to individuals with the skills they need to get the most out of Watson, via an online service known as the Talent Club.

"Various companies will have skills gaps in these areas so the Talent Club provides a set of companies with skills in areas ranging from user experience, to data analysis, to application development and training," said Sudarsan.

About Nick Heath

Nick Heath is chief reporter for TechRepublic. He writes about the technology that IT decision makers need to know about, and the latest happenings in the European tech scene.

Editor's Picks

Free Newsletters, In your Inbox