The Hidden Meaning in Text – Sentiment Analysis

“I think of these tools like a metal detector, sure it beeps when it finds something, but it’s still up to you to expend the energy to dig in up to your elbows and find the nugget. “ – Anonymous

Everyone has an opinion. When the Internet was commercialised in the late 1980’s, it was only a matter of time before the landscape of information as we know it changed forever. “More information will be created in the year 2009 than in all of recorded history up until that year” [1] and by the mid 1990’s, with the advent of blogs, email and online discussion forums, it became clear that we were no longer in a world that needed to take a back seat as big corporations continued to make all the rules.

The consumer has truly become king on an unprecedented scale and they are taking to the Internet to air their satisfaction or dissatisfaction with everything from government to product to service delivery – often in overwhelmingly huge numbers. In fact, the last year has seen a rise in the uptake and infiltration of social networks, blogs, microblogs and forums amongst businesses. A global survey conducted by Regus [2] revealed that 52% of businesses globally and in particular 55% of businesses in South Africa are using web sites such as Twitter to engage, connect with and inform existing customers.

Consider this – of the 6 billion people on the planet, 4.8 billion have a mobile phone. That means that 4.8 billion customers can potentially express their opinion, literally at the click of a button. Now consider a more shocking statistic – only 4.2 billion own a toothbrush [3]. The priorities of an entire generation of consumers are irrevocably tied into technology and online presence. Can any company or brand really afford to ignore the endlessly changing and churning online discussion and opinion of that many people with that kind of spending power?

The Internet has exploded with sentiment, and the voice of the customer has become one of the most powerful weapons humanity has ever unleashed. As a result, businesses are increasingly being made, or buried, with sentiment. So how does this tie in with Business Intelligence and the like? Well, once considered to be the ugly stepsister of Advanced Analytics, Sentiment Analysis has suddenly taken the world by storm and, quite literally it seems to have gone viral. The near instantaneous sentiment gleaned from unstructured, unsolicited sources means that companies now potentially have their fingers on the pulse of the collective customer. Reaction times to marketing campaigns, new offers or addressing unsatisfactory service can be immediate.

Consider the words of the man considered by some to be the father of modern advertising – John Wannemaker: “Half the money I spend on advertising is wasted; the trouble is, I don’t know which half.” With a well trained, robust sentiment model you will know exactly which half.

Definition

Sentiment analysis is the method of automatically locating and analysing digital content in real time from across multiple channels such as websites, emails, forms and surveys in an attempt to hone in on customer sentiment buried within textual data.

The goal of Sentiment Analysis or opinion mining is to determine the attitude, opinion, emotional state, or intended emotional communication of the message.

In South Africa, some of the more sophisticated models and solutions are being offered by Fuseware – Offering real time online reputation management, comprehensive analysis, brand monitoring and comparison; Cision – Specific social media monitoring, theme and topic identification, key Influencer identification; Media House– Creates and monitors a social media presence on behalf of a company, monitoring and analysis are part of the more advanced packages.

PBT uses a hybrid machine learning / human input model which becomes more accurate the larger the sets, the better the human input and the longer they run. They are taught to identify relevant comments and then classify them according to why they feel positive or negative towards something. What are they unhappy about? Why do they prefer the competitor? The model is ‘taught’ to identify slang, sarcasm, double negatives, irony, shorthand and punctuation.

Imagine being able to measure the impact of a marketing model in close to real time. Or being able to address customer satisfaction the moment it is verbalised. Flame detection (bad rants), product perception and brand reputation management are all applications of sentiment analysis.

PBT recently performed sentiment analysis for a South African cellular provider, analyzing free text survey data, collected via SMS, to determine customers’ sentiments around their brand / service as well as a new ‘chat’ service offering. The question was asked “Do you have any comments, complaints or suggestions about Product A?” Using IBM SPSS Text Analytics the analysis was repeated over 2 cycles.

The analysis clearly showed where the cellular company needed to focus its drive to improve customer satisfaction, and over time this was measured and the shifting priorities were highlighted.

Managing the positive and the negative

Facebook is quite possibly the best example of consumer to consumer transfer of opinion and information. Say Company A creates a company profile on Facebook. I ‘like’ the profile and become part of their fan base. I encourage my friends to join. The platform is primed to take advantage of my expression of positive sentiment and that expression encourages the like-minded fan base to also express themselves. However, what happens when I display negative sentiment? Unchecked and unmanaged – this could quickly develop into a PR nightmare for Company A, especially if I have a large and loyal group of friends.

Consider that there are over 800 million active Facebook users, the average Facebook user has 130 friends and is connected to 80 pages, events and groups. There are over 3.5 billion pieces of content (web links, news stories, blog posts etc.) shared on Facebook each week [3].

Yet, somehow, in the face of this overwhelming customer presence, 95% of Facebook Wall posts are not answered by brands [3].

It must be understood that managing the positive and, quite crucially, the negative has become a full time job that does not keep regular working hours. We live in a world that is as small as the time it takes me to press enter on my keyboard. A world that is always awake. And being able to keep a finger on the pulse of a very fickle consumer base will become one of the biggest challenges facing companies this century. This is where Sentiment Analysis really comes into its own. The near instantaneous, automated analysis of the sentiment that is being expressed at any given moment at any given point on the Internet is vital for the success of any company that is hoping to be successful.

Many companies only become aware of the power of consumer opinion when it turns against them. That is the potential problem when organisations are reacting to consumer sentiment rather than proactively managing it. And there is massive risk in that approach. Recently, a much loved and well thought of brand was accused of “deliberately and intentionally” copying a brand of soft drinks. A regulatory body found them guilty and the social media backlash left the company involved reeling and on the back foot. For better or worse, this is not an isolated occurrence either. Nor is it regionally contained. Last year a multinational company was forced to recall a popular brand of baby food after a single piece of glass was found in one bottle. This only affected the food produced and sold in France, yet the social media backlash was global.

The Good News and the Bad News

More and more organisations are using sentiment analysis to successfully increase their return on investment and using the models to make a difference in the areas where it matters most – the bottom line and brand reputation. A perfect example of this is the approach FNB took to social media during their recent re-launch. Michael Jordaan, CEO of FNB is so passionate about FNB’s ‘always on’ approach and the advent of their ‘face’ of FNB, @RBJacobs, that he has been tweeting since July 2009, giving the customer unprecedented access to a major decision maker. He believes that their pioneering approach to social media has given them a competitive advantage and allowed their organization to collect rich data about their communities and customer segments. This has enabled them to develop more effective communication channels and strategies and has significantly impacted on their marketing strategies [4].

However, Sentiment Analysis has developed a bad reputation through no fault of its own. Due to the very nature of trying to classify text and compensate for the intricacies of vernacular – it is incredibly difficult to set up a model that delivers accurate sentiment with minimal effort. All too often, many people are utilising the most basic of tools available on the web to try and glean some useful information relevant to their brand with less than encouraging results. As such, the everyday user of these tools is expressing their dissatisfaction with the very tool that is needed to find and classify that very sentiment – creating a bit of a paradox.

When it’s all said and done, it must be remembered that above all, we are still at the beginning of this journey. Best practice, appropriate tools, true understanding – are all still a work in progress. With the astounding increase of blogs, tweets and solicited and unsolicited social commentary, not only is the sheer volume that needs to be mined staggering, but certainly the world has never been more primed for this specific kind of analysis.

For the longest time, the rules of marketing were etched in stone. Corporations were happy with, and had fully embraced, the monologue that flowed from business to consumer, ensuring that the consumer only ever knew exactly what business wanted them to know. However, with the staggering amount of unsolicited consumer opinion being aired, that monologue has out of necessity become dialogue. And possibly for the first time in human history, the word or social pen of the consumer has become more powerful than the marketing sword wielded by big business.

Companies now need to make the connection: