Social Enterprise

Sentiment analysis: Understanding customers who don't mean what they say

Marketers face challenges in peeling back layers of meaning in language, like slang and sarcasm. Sentiment analysis could be the key.

sentiment06014.jpg
iStock/kvkirillov

If a co-worker has ever turned to you and deadpanned "I sure do love Mondays," and you retorted, "Right? They're the worst," then you can thank the little built-in sarcasm detector in your brain for keeping you from responding with incredulity.

For your average human, this is easy stuff. For a brand looking to understand the many complex meanings embedded in the ways people use language — sentiment analysis — in places like social media, it's a challenge they're gradually conquering.

Even the Secret Service announced earlier in June that it's looking for software to detect sarcasm online. For the guys looking to keep the President and his close associates safe, it's a matter of parsing what they should be worried about, and what they shouldn't, in the interest of keeping people alive.

For a brand, the stakes are a bit lower. But in their realm, it can provide a chance to learn about things like product issues, service issues, unexpected uses of a product, areas where the brand is spending too much time - or not enough, or general feelings from customers and potential customers.

Given that human to human, we don't always understand each other, finding increasingly reliable sentiment analysis remains somewhat difficult.

What's out there

According to Altimeter Group analyst Susan Etlinger, there are a few approaches to sentiment analysis including keyword-based analysis (think turning up every reference, whether innocuous or not, to "bomb" on Twitter), as well as natural language processing.

"What they can do is analyze those chunks of language for similarities with other known chunks of language and see to a certain statistical probability, it probably means one thing or the other," she said. This helps find the real meaning behind a phrase like "outstanding debt," which is not a positive phrase in anyone's book.

Rob Key, CEO of Converseon, a company Forrester Research recently ranked highest overall for data processing and sentiment analysis, said that they use a combination of natural language processing and machine learning. While machine learning can be limiting in the sense that it is only as good as the data it's seen before, Converseon shores up the difference by keeping human analysts in the mix in a "semi-supervised" setup where they'll double check anything that sits below a certain threshold of confidence.

"The gold standard in sentiment analysis is essentially how well can your text analytics achieve human-level perfection," Key said. Converseon's accuracy rate is about 95%, which is higher than the average accuracy rates for individual techniques.

Keyword analysis can typically yield 60-65% accuracy, natural language processing comes in a little higher at 80-85%, Etlinger said.

As for that lingering hypothetical 15%, for a brand, Etlinger said, it depends on what they deem acceptable.

"It's one thing if you're trying to figure out if people are really enjoying your TV show or are going to buy your shoes, it' another thing if you're talking about the health and welfare of American citizens," she said.

According to Max Kalehoff, senior vice president at SocialCode, companies in this field seem to do best when relying on a combination of approaches, rather than just one. And for SocialCode, a sentiment score is part of several metrics used to gauge if content is effective and likely to have a positive impact on the brand.

On Converseon's end, Key also discussed the importance of achieving granularity, citing three different levels:

1. Record - Taking something like a blog post and categorizing is as generally positive, negative, or neutral.

2. Sentence - Applying the same idea to sentences.

3. Facet - Being able to read for multiple layers of meaning within a sentence.

He explained the two key measures a marketer wants to focus on are precision and recall (how many points of analysis can you analyze and unearth within the conversations).

"You want a really high precision level and a really high recall level; and, historically, social monitoring tools simply aren't good enough to get to that level of precision and recall that you need for high demand use cases," he said.

Challenges

A classic example of where language can elude a computer is a word like "sick." In reference to a guitar solo, perhaps, "sick" is a positive descriptor. In reference to illness or a particular shade of "wrong," sick is negative. According to Etlinger, if you're just relying on keywords, this is where some of the results can get muddy.

Spoken language doesn't exactly experience stasis as culture is continually shifting and expanding. Consider the words that have entered our vernacular in the past few years. "The word 'Tebowing' didn't exist, the word 'selfie' didn't exist. There's all this new slang, so you have to continually feed the system with human language.

"It's the subtle nuances around context, which is the biggest area where sentiment mining has challenges, which is why in the brand marketing world you'll typically see sentiment mining technologies used in conjunction with human analysts," Kalehoff said. The upshot is "an analyst armed with technology," can be very prolific and very productive.

"You see a combo of tech as well as human, and sometimes humans reside within a brand or sometimes the humans will reside at the actual service or offering, but for very sophisticated, very precise, you'll see them hand in hand," he said.

What's next

While sentiment analysis is still imperfect, it's likely to branch into other areas in the future, like interpreting not just text but images and audio, whether they be emoji, strings of emoji, or video. Consider a system that's able to interpret a picture of someone holding a can of Coca Cola, smiling, as positive sentiment, Etlinger said.

From Key's perspective, in the future, not only will there be high numbers in terms of precision and recall, but sentiment analysis will get more specific. The next generation will be tied to different verticals, whether for companies selling pharmaceuticals, chocolates, or shoes, and the language will be tuned to those.

"For example, the word 'small' is good if you're selling smartphones, but it's really bad if you're selling hotel rooms," Key said.

And beyond sentiment, emotion. "We use Plutchik's wheel of emotions, and there's eight different emotions we analyze in the data- joy, anger, happiness- so sentiment is going to get more sophisticated and integrated with emotional analysis," he said, "That's going to be very critical because there's a lot of research that shows that a lot of the decisions that consumers make are really driven by emotion."

Where that leaves the Secret Service is hard to say. Etlinger summed up the situation like this: "I think this is going to be a pretty rich problem to solve over the next several years."

Also see:

About

Erin Carson is a Staff Reporter for CNET and a former Multimedia Editor for TechRepublic.

Editor's Picks

Free Newsletters, In your Inbox