Artificial Intelligence

How deep learning and artificial intelligence power Comcast's voice remote

Jeanine Heck, vice president of AI product at Comcast, discusses how Comcast has integrated deep learning into their own AI voice remote, taking their product to a new level of success.

Comcast's vice president of AI product, Jeanine Heck, speaks with TechRepublic's Tonya Hall about the success of an AI voice remote product, integrated with deep learning. The following is an edited transcript of the interview.

Tonya Hall: It's push to talk, and not always listening. Welcome, Jeanine.

Jeanine Heck: Thank you, Tonya.

Hall: So, what is your role entail, exactly?

Heck: My role is really to be a product manager, and that entails working with the engineering team to ensure that we're building products that are valuable to our customers, and so it's important for me in my role to understand the customers needs, how they use our products today, how they may want to use our products tomorrow, and also making sure that the products are very competitive in the marketplace. So I'm looking at all of that and interpreting it for our engineering teams so that we can prioritize our engineering work according to what will meet the customer needs the best.

Hall: How have you deployed machine learning and artificial intelligence throughout Comcast?

Heck: Well, in many ways, actually. But I would say the most exciting one has been our voice remote. So we are using our own artificial intelligence on the remote to do voice recognition and natural image processing. And then we have also used deep learning in the past few years to really enhance the accuracy of our voice remote results, and it's taken them to a whole different level of accuracy, because of what you can do with deep learning.

Hall: Now, your voice remote, is that called X1?

Heck: Yes, our TV platform, our entertainment platform, is called X1, and the voice remote is called X1 voice remote.

Hall: What makes it so special?

Heck: It flattens the UI, kind of the way we express the benefit. It's one button. It can do everything on the TV. So you don't really need to know any other tricks, except press the button and say what you want. And our engineering team has spent years developing the natural-language processing components so that it really can understand what your intent is, and you don't have to now punch around in the guide if you know exactly what you want. It can take you directly there without a lot of button pressing or jumping through a bunch of screens.

SEE: AI changing the world? Maybe eventually, but definitely not today (TechRepublic)

Hall: What kind of problems were you trying to solve, or what kind of enhanced experience were you trying to deliver?

Heck: Well, when we set out, we really thought we were building a better way to find a movie or a show when you knew exactly what the program is. And that's still really the top benefit that we can give. So if you know that you want to watch Modern Family, you say "Modern Family," and it takes you directly there. Or if you know you want to watch comedies on HBO, you can say, "comedies on HBO," and it takes you directly there.

One of the things we've actually been surprised about though, that we didn't set out to do, is the ability to change channels just by saying the name. That was something we weren't really sure about until we put it in front of customers in trials, and we saw that when people said, "CNN," they wanted the channel to change right to CNN. It's actually become one of our biggest-use cases, and I think those two problems, not needing to remember channel numbers, and not needing to go punch in letter-by-letter the name of a show on a keyboard on the TV, those two problems being solved are probably the biggest things that we've accomplished with the voice remote. I've been working in the TV industry for a long time, and especially on the content discovery side of the house. Those are problems that existed in the guide, for as we've been making guides, and the voice remote really solves those problems and helps people get their content more quickly. It's kind of just a huge leap ahead from where we were.

Hall: What other types of things did you learn through user experience that you weren't expecting?

Heck: I would say one really cool thing was that people appreciate their words showing up on the TV screen. It's kind of a small usability feature, but when you see your own words ... So if I say, "watch ESPN," and I see the words "watch ESPN" on the TV ... Seems very simple, but for us it was ... It helped make the product very usable but I think, also, it created a bit of magic. For the first time, you could see in an instant that your TV heard you. So it really was this new, exciting experience, and there was some magic to it as well as some real utility, because if it misheard you or misunderstood you, you could see right away what it heard.

Hall: Is the voice analysis AI performed in the cloud, or on the network, or is it somehow stored locally on the device?

SEE Special report: How to implement AI and machine learning (free TechRepublic PDF)

Heck: It's all happening in the cloud. Actually, X1 is a cloud-based platform for TV, and that was something that gets all of us on the engineering side really excited, because once we went to a cloud-based vibe, we could deploy whenever we wanted. We could deploy, launch a new guide overnight for all of our customers. And prior to that, we had to go market by market, because all of the code on a set-top box was literally downloaded to the box itself, so it was a very QA intensive, quality assurance intensive process, just to make sure that the code was ready to go on the box, because it was kind of a difficult thing to roll back. So nowadays, with cloud, cloud solutions plus the ability to role back really easily, we can take a lot more risks.

Hall: What were some of the biggest challenges you came across in deploying this?

Heck: Biggest challenge, I would say, getting it onto a physical remote device, was one of the bigger challenges. It was really straightforward to launch voice control on an app, so we did that early. We got this thing working in 2012, we launched it right away in the start of 2013 as a iOS and Android app, so you could go onto the app and tap the microphone and talk to your TV, and do all the things that you can do today on the remote. The remote took a bit more time, and I think we were ahead of our time with the need to send audio through a remote to the set-top box, so not all of the protocols were there for using radio frequency, or RF, to send the audio to the box, and so our timing was a little bit early when we built the first version of the remote, but by the time we built the second one, we had this really great protocol called RF4CE that allows to send audio from the remote directly to the set-top box and then out to the cloud.

Hall: Yeah, you were first to market with this, right?

Heck: We were. Certainly the first cable company, the MSO as we call ourselves, to launch a voice remote.

Hall: And is it more important to be first to market, or is it more important to have a kind of wait, applied research analysis, and then deploy?

Heck: I think it's more important to get it right. So I wouldn't launch anything just to be first. But when you're developing something inside a company and you see other companies launch, maybe before you, you kind of feel like, "Aw, wish that was us." But at the same time, anytime I saw anyone launching a voice product, before we launched our voice remote, it really reinforced how excited I was about our product, because I think ours is superior to what else has been launched.

Hall: So do you say Comcast then maybe takes on big risks, or does it employ more a kind of wait and see approach?

Heck: Oh, I absolutely think Comcast takes big risks. With the voice remote, our leadership team decided to put a voice remote with every set-top box that went out. From the day we launched voice remote, that became the default remote, and the other remote, we stopped deploying it. So it was a big decision, a big financial investment by the company to put a remote out there that has voice on it. We took a big risk because we felt that the product was up to par, and that we had a really high accuracy with our voice-recognition, so we really did take the bet on it, and we said right away, "This is the remote for everyone."

SEE: Why human-AI collaboration will dominate the future of work (Tech Republic)

Hall: How many units have you placed in-home so far?

Heck: We have more than 20-million voice remotes in homes today.

Hall: How are you approaching security and privacy?

Heck: Well, we take privacy very seriously. Our engineers actually ... We don't marry together account information with usage information, so from our point of view, everything's anonymous, and if we're using it, it's generally as an aggregate, and we really try to separate out any information that would be personal. There's really nothing in the voice remote that is ... We totally separate out any information from your account or your bill from what you watch on TV. We actually don't care too much about who says what. It's more looking at the trends and the bigger aggregate usage patterns, and actually the machines can do so much of the analysis there that we typically are ... We take it very seriously with privacy, and we do make sure that we treat accounts separately.

Hall: How are you addressing competition?

Heck: I would say that we're always trying to stay ahead of the competition. I think Comcast has been really great with being competitive with our products. With X1, we have been ahead. We took a big bet on making a cloud-based guide, starting to incorporate Internet or IP TV on the guide really early. With voice, I would say that we have definitely made sure to, number one, we see how people are using the remote. We can understand if our features aren't meeting their needs by keeping a close look at what people are saying and how we can maybe make their experience better. I would say that we, because of our scale and because of our usage today, because we have so many people using this remote on a daily basis, that we really are very proactive about making the experience better, and we think that we actually learn more by looking at what our users are doing than what the competition is doing.

Hall: Okay.

Heck: That helps us stay competitive.

Hall: Does it talk back to you? Will it ever talk back to you?

Heck: It doesn't talk back to you, and that's actually by choice. We did test the ability for the guide to talk back to you. So I say, "Show me movies on HBO," and it would say, "Showing movies on HBO," and it kind of just seemed redundant, and we realized that it was mostly annoying. It wasn't really adding to the experience, it was kind of just getting in the way, and so what we've done with our voice remote, and I think we've really differentiated the product in many ways, but one of them is that it really gets out of your way. It's very much a voice interface that takes you directly to what you want, and then gets quiet again. So we've done a bunch of things like that that make it different, so we don't give our voice remote a personality. It's just called Voice Remote, and it's not trying to be your friend or connect with you emotionally, it's just trying to let you interface with your TV using voice, because we think that's the best way to get to your content.

Hall: You know, I might want it to connect with me emotionally. Why can't it just be a robot, right?

Heck: Well, the great thing about TV and X1 is you connect with your content. So for us, one of the things we did was you could talk to Minions with the voice remote, so when the Minions movie was launching, you could say, "Banana," and the voice remote would respond with, first of all, a list of content that was all about food, because to Minions banana means that they're hungry. And also you would hear the Minions talk back to you and say, "Banana!" And it was pretty funny, but when we make the voice remote interactive, it's usually getting the content or the brand to interact with the user, not so much the remote itself.

Hall: Okay, Comcast acquired Fandango in 2007. You've recently launched something to enhance the experience of voice. Talk about that.

SEE: IT leader's guide to the future of artificial intelligence (Tech Pro Research)

Heck: Yeah, so we have this really cool feature where you can watch a movie trailer, and you can say, "Get tickets," and we will bring up a Fandango app on the TV that allows you to buy movie tickets right there on your TV with an app. You will also log into the Fandango app on your phone, but we make it really easy to make the transaction happen on your TV. This is something we've actually heard from users that they wanted, and so we were really excited to be able to offer it.

Hall: Being competitive and staying competitive is an issue for anybody in this industry. How many employees does Comcast have focused on emerging technology?

Heck: We have many employees focused on emerging technology. I would say that we have our whole X1 team, which is a pretty large size. I usually don't talk about numbers, exactly, publicly, but I think we have a pretty competitive team, certainly in the cable industry and the TV industry in terms of size. So a pretty large team, and we're able to recruit from all different areas, so we have a lot of ... Our team works on AI, and so we are able to attract people right out from their PhD programs who want to go to a company where they can solve really cool problems with a lot of data, because we know with AI, a lot of times it's, sure the algorithms are exciting, but the data is what makes the algorithms, it kind of completes the algorithms. At a company like Comcast, we do have enormous scale, so we find that our engineers get really excited about the scale that we have with our products, and with the amount of data we're able to train on when we're building algorithms.

Hall: Tell us about Lab Week and Science Fair.

Heck: Oh, I love Lab Week. Yes. Three times a year, Comcast's TPX, is what our department is called, Technology Product and Experience, we have three Lab Weeks a year. The engineers are encouraged to work on anything that they're excited about, or a new feature or a new product that they want to see built. They have time for that full week to focus on that product, and then we have a science fair at the end of the week where the engineers showcase the work that they've done that week, and our executives all the way up to the CEO come to the Lab Week Science Fair and walk the floor and see what the new ideas are. I love it, because I really do believe that the best ideas come from the engineers, especially the engineers working on products. They know where to take the product next, and so I think that Lab Week event really just reinforces that belief as a company that we want to encourage more ideas to come out of our engineering teams.

Hall: Okay, Jeanine, what are some of the future goals related to machine learning, artificial intelligence, or Internet of Things, that you guys have planned?

SEE: Research: Companies lack skills to implement and support AI and machine learning (Tech Pro Research)

Heck: We are, in addition to voice recognition, we are also ... We have a lot of work in the recommendation space. So when you move around the X1 guide, you do get recommendations for content and I always say, until we know exactly what you want to watch right now, our AI work is not done there. We still have a lot of efforts around the home that we're pretty excited about. So one of them is with cameras. Our AI team has built some of our own computer vision capabilities, and we launched that last year on Xfinity Home Cameras. One of the things that we launched is the ability to build a better smart thumbnail, so we zoom into the moving object in your thumbnail, rather than you just always seeing the same frame, so it makes it a lot easier to find the video that you're interested in from your home camera. So yeah, we have a lot of ... Our team is focused on the broad array, or I would say portfolio, of Comcast's products, and any product that has the Xfinity brand or that you can use from Comcast, is a product that we are applying AI to.

Hall: Well, thank you so much, Jeanine, for giving us some insight to how you're focused on machine learning and artificial intelligence, and what we can expect to see from Comcast in the future. If somebody wants to connect with you, maybe they want to find out more about these products, how can they do that?

Heck: Yeah, I think the best way is to find me on Twitter. You can find me with the Twitter handle @j9heck and I would love to hear from you, so please do reach out.

Hall: Thanks again so much.And if you guys want to follow me, you can. You can see more of my interviews right here on TechRepublic, or ZDNet. Or maybe if you want to connect with me on social media, you can go to TonyaHall.net, and I've got links to my Facebook, my LinkedIn, Pinterest even. In fact, if you want to chat, I'd love it if you followed me on Twitter. I'm at @TonyaHallRadio on Twitter. I'd love to hear from you. Thanks for watching.

Also see

Jeanine Heck, vice president of AI product at Comcast

About Tonya Hall

Tonya Hall is a pioneer in new media broadcasting. She hosted a daily broadcast radio show on social media, worked as a producer for one of the largest tech podcasting networks, and continues to bring compelling guests and stories to tech fans across...

Editor's Picks

Free Newsletters, In your Inbox