Extracting Insights from Consumer Reviews

Here is one more example on how we can extract Insights from Consumer Reviews. This time we will use Reviews that were given for several Supplement Brands of Omega-3 Fish Oil.

For this example we analyze 4018 Reviews of Consumers who bought Omega-3 Supplements.  Keep in mind that in most cases each Product Review has an associated Rating (usually given as 1-5 stars) which signifies the overall satisfaction of each Consumer . Therefore, after data collection of the Reviews and Ratings we have a file with the following entries per row :

[Text of Review,Rating]

The fact that a Customer gives also a Score can be especially helpful because we can identify the words and Phrases that differentiate Positive experiences (ie those having 5 Star Ratings) from the Negative Ones (We assume that any Review having a Rating of  4 stars or less is Negative). So for example, Positive Reviews may contain mostly words and phrases such as "Great", "Happy" and "Will buy again" whereas Negative Reviews may contain words and phrases such as "Never buying again","not happy" or "damaged".

The tools used for this example are NLTK and Python. The code simply reads the reviews and associated text and creates a Matrix with the same representation as the file it read.

Next, we want to identify which Insights we can extract from this representation. For example :

-Identify which words commonly occur in 5-star reviews
-Identify which words commonly occur in Reviews with a rating of 4 Stars or Lower.
-Identify potentially Interesting Phrases and Words
-Extract term Co-Occurrences

We start with terms occurring more frequently in Negative Reviews for Omega-3 Supplements. Here is what we've found :

So it appears that people tend to give negative Reviews when the Taste (and possibly After-Taste) is not quite right. A lot of people complain about a Fishy odor. Notice also that the 3rd Term is sure which we can assume that it originates from customers saying that they are not sure if the Product works or not (Notice also that the 4th term is yet). Some more terms to consider :

however
rancid
krill (a type of Oil which is alternative Product to Omega-3 Supplementation)
soy
stick


Now let's look at the Terms associated with Positive Reviews :

great and excellent are terms that were expected to be found in Positive Reviews.  Some terms to consider are :

price
quality
brain
triglycerides
cholesterol

We move on to identifying potentially interesting terms and Phrases. Here is a Screenshot from the Software that i used  :

I added a Red Rectangle wherever sensitive information (such as Company Names) appears which for the purpose of this post is not relevant (but it certainly is relevant in a different setting).

We immediately see some interesting mentions, for example : Heavy Metal poisoning, Upset Stomach incidences, Cognitive Function , Joint Pains, Panic Attacks, Reasonably Priced Items, Postpartum Depression, Allergic Reactions, Speedy Delivery and Soft Gels that Stick together.

Recall that in a previous example we found that the term however is a term that occurs frequently within Negative Reviews. Some analysts may have chosen to treat this term as a stopword which in this case would be a serious mistake. The reason for this is that the term however shows us very often the reason for which a product or service is not receiving a perfect rating and vice-versa. Therefore, If a Data Scientist would have chosen to exclude this term from the Analysis (stopwords are typically removed from the text), potentially interesting insights would have never surfaced.

Ideally, we would like to know what is the context that occurs after the term however whenever this term occurs withing a negative review. That will help us to focus on all occurrences of however with negative sentiment. To do this, we only take into account all reviews containing the term however and having a Rating of 3 stars or less. It appears that the most common terms occurring after the term however was Fishy odor and After-taste. In other words, fishy odor is the cause that keeps Customers from giving a 5-star Rating.

On the other hand, phrases such as highly recommend are interesting because we may use co-occurrence analysis to see which terms co-occur with a highly recommended product.

Of course this is -by no means- the end on what we can do. To extract even better insights we have to spend significantly more time to do proper Pre-processing, use Information Extraction and use several other techniques to analyze Text Data in novel and potentially interesting ways.

Becoming a Data Scientist : A RoadMap

I receive a lot of questions regarding which books one should read to become a Data Miner / Data Scientist. Here is a suggested reading list and also a proposed RoadMap (apart from the requirement of having an appropriate University degree) in becoming a Data Scientist. 

Before going further, it appears that a Data Scientist should possess an awful lot of skills : Statistics, Programming, Databases, Presentation Skills, Knowledge of Data Cleaning and Transformations.
 

The skills that ideally you should acquire are as follows :

1) Sound Statistical Understanding and Data Pre-Processing
2) Know the Pitfalls : You must be aware of the Biases that could affect you as an analyst and  also the common mistakes made during Statistical Analysis
3) Understand how several Machine Learning / Statistical Techniques work.
4) Time Series Forecasting
5) Computer Programming (R, Java, Python, Scala)
6) Databases (SQL and NoSQL Databases)
7) Web Scraping (Apache Nutch, Scrapy, JSoup)
8) Text Data

Statistical Understanding :  A good Introductory Book is Fundamental Statistics for the Behavioral Sciences by Howell. Also IBM SPSS for Introductory Statistics - Use and Interpretation and IBM SPSS For Intermediate Statistics by Morgan et al. Although all of the books (especially the two latter) are heavy on  IBM SPSS Software they are able to provide a good introduction to key statistical concepts while the  books by Morgan et al give a methodology to use with a practical example of analyzing the High-Scool and Beyond Dataset.

Data Pre-Processing : I must re-iterate the importance of thoroughly checking and identifying problems within your Data. Data Pre-processing guards against the possibility of feeding erroneous data to a Machine Learning / Statistical Algorithm but also transforms data in such a way so that an algorithm can extract/identify patterns more easily. Suggested Books :

•  Data Preparation for Data Mining by Dorian Pyle

• Mining Imperfect Data: Dealing with Contamination and Incomplete Records by Pearson

• Exploratory Data Mining and Data Cleaning by Johnson and Dasu

Know the Pitfalls : There are many cases of Statistical Misuse and biases that may affect your work even if -at times- you do not know it consciously. This has happened to me in various occasions. Actually, this blog contains a couple of examples of Statistical Misuse even though i tried (and keep trying) to highlight limitations due to the nature of Data as much as i can. Big Data is another technology where caution is warranted. For example, see : Statistical Truisms in the Age of Big Data and The Hidden biases of Big Data.

Some more examples :

-Quora Question : What are common fallacies or mistakes made by beginners in Statistics / Machine Learning / Data Analysis

-Identifying and Overcoming Common Data Mining Mistakes by SAS Institute

The following Book is suggested :

• Common Errors in Statistics (and how to avoid them) by P. Good and J. Harding

In case you are into Financial Forecasting i strongly suggest reading Evidence-Based Technical Analysis by David Aronson which is heavy on how Data Mining Bias (and several other cognitive biases) may affect your Analysis . 

Understand how several Machine Learning / Statistical Algorithms work : You must be able to understand the pros and cons of each algorithm. Does the algorithm that you are about to try handle noise well? How Does it scale? What kind of optimizations can be performed? Which are the necessary Data transformations? Here is an example for fine-tuning Regression SVMs:

Practical Selection of SVM Parameters and Noise Estimation for SVM Regression 

Another book which deserves attention is Applied Predictive Modelling by Khun, Johnson which also gives numerous examples on using the caret R Package which -among other things- has extended Parameter Optimization capabilities.


When it comes to getting to know Machine Learning/ Statistical Algorithms I'd suggest the following books  :

• Data Mining : Practical Machine Learning Tools and Techniques by Witten and Frank

• The Elements of Statistical Learning by Friedman, Hasting, Tibishirani 

Time Series Forecasting : In many situations you might have to identify and predict trends from Time Series Data. A very good Introductory Book is Forecasting : Principles and Practice by Hyndman and Athanasopoulos which contains sections on Time Series Forecasting. Time Series Analysis and its Applications with R Examples by Shumway and Stoffer is another book with Practical Examples and R Code as the title suggests.

In case you are interested more about Time Series Forecasting i would also suggest ForeCA (Forecastable Component Analysis) R package written by Georg Goerg -working at Google at the moment of writing- which tells you how forecastable a Time Series is (Ω = 0:white noise, therefore not forecastable, Ω=100: Sinusoid, perfectly forecastable).

Computer Programming Knowledge: This is another essential skill. It allows you to use several Data Science Tools/APIs that require -mainly- Java and Python skills. Scala appears to be also becoming an important Programming Language for Data Science. R Knowledge is considered a "must". Having prior knowledge of Programming gives you the edge if you wish to learn n new Programming Language. You should also constantly be looking for Trends on programming language requirements (see Finding the right Skillset for Big Data Jobs). It appears that -currently- Java is the most sought Computer Language, followed by Python and SQL. It is also useful looking at Google Trends but interestingly "Python" is not available as a Programming Language Topic at the moment of writing. 

Database Knowledge : In my experience this is a very important skill to have. More often than not, Database Administrators (or other IT Engineers) that are supposed to extract Data for you are just too busy to do that. That means that you must have the knowledge to connect to a Database, Optimize a Query and perform several Queries/Transformations to get the Data that you want on a format that you want.

Web Scraping: It is a useful skill to have. There are tons of useful Data which you can access if you know how to write code to access and extract information from the Web. You should get to know  HTML Elements and XPath.  Some examples of Software that can be used for this purpose : 

-Scrapy
-Apache Nutch
-JSoup

Text Data: Text Data contain valuable information : Consumer Opinions, Sentiment, Intentions to name just a few. Information Extraction and Text Analytics are important Technologies that a Data Scientist should ideally know.

Information Extraction :

-GATE
-UIMA

Text Analytics

-The "tm" R Package
-LingPipe
-NLTK

The following Books are suggested :

• Introduction to Information Retrieval by Manning, Raghavan and Schütze

• Handbook of Natural Language Processing by Indurkhya, Damerau (Editors)

• The Text Mining HandBook - Advanced Approaches in Analyzing Unstructured Data by Feldman and Sanger

Finally here are some Books that should not be missed by any Data Scientist :

• Data Mining and Statistics for Decision Making by Stéphane Tufféry (A personal favorite)
• Introduction to Data Mining by Tan, Steinbach, Kumar 
• Applied Predictive Modelling by Khun, Johnson
• Data Mining with R - Learning with Case Studies by Torgo
• Principles of Data Mining by Bramer

Analyzing PubMed Entries with Python and NLTK

I decided to take my first steps of learning Python with the following task : Retrieve all entries from PubMed and then analyze those entries using Python and the Text Mining library NLTK. 

We assume that we are interested in learning more about a condition called Sudden Hearing Loss. Sudden Hearing Loss is considered a medical emergency and has several causes although usually it is idiopathic (a disease or condition the cause of which is not known or that arises spontaneously according to Wikipedia). 

At the moment of writing, the PubMed Query for sudden hearing loss  returns 2919 entries :

  

In order to be able to collect this information from PubMed we can use Entrez library from BioPython which enables us to save to a csv file the Title (Column A) and the Abstract (Column B) of each PubMed entry :

Now that we have the PubMed entries in place we may start Text Analysis with the  NLTK Toolkit. The Python code simply reads the csv file which was created, it removes stop words and uses a simple function to search and replace specific words. For example for this type of Data it is a good idea to replace occurrences of : 

 genes to gene,

 induced, induces,inducing to induction,

antagonists to antagonist,

agonists to agonist,

..etc.

This pre-processing work will help for more efficient retrieval of (possibly) interesting findings. For this example we want to find Collocations of terms and to do this we will use the BigramCollocationFinder from the NLTK Toolkit. After running the Bigram Collocator the program prints the top 100 most-important word-pairs scored using Pointwise Mutual Information :

Let's try to "relax" our requirements by increasing the amount of words that are fed to our analysis. Here are our new results :

We immediately notice the differences between the first analysis and this one since on the second instance we see much more potentially interesting word pairs (more Medical Conditions, Substances, etc are shown) as opposed to the first set of results.

Let's suppose that we are interested in finding which gene(s) could be involved. To do this a Python function is used which scores the co-occurrence of a word of interest (in this case 'gene') with other words.

Here are the results :

The result is not very useful. Nevertheless, it reminds us that we should probably replace polymorphisms with polymorphism. 

We may then decide to relax again the way with which bigrams are created and we increase the number of subsequent words that are searched and re-run the code :

The Top-Rated results from 'gene' analysis return a term named MTHFR which actually is a gene called Methylenetetrahydrofolate reductase. The same happens with the occurrences of the genes in the bigram co-occurrence analysis just before our 'gene' inspection. We also notice that Co-enzyme Q10 ( a well known and popular supplement) shows on the top of the list. After a bit of searching within PubMed entries it was found that CoQ10 is used for treatment of Sudden hearing Loss and also that CoQ10 was found in low concentrations of people having this condition. 

We can use BioGraph to submit a query for sudden hearing loss and see which concepts are associated with this Condition :

So MTHFR was found on Biograph as well, however at the moment of writing CoQ10 was not in the list (not shown because of its length).  We submit a query to the same engine  for CoQ10 and also filtering specifically for diseases:

Again - at the moment of writing - Sudden hearing loss was not found on Biograph as an associated condition. Of course it is not suggested here that Biograph entries are incomplete. Different types of Analysis may be used and the data that i used for this example were much more targeted (which in its own right should warrant extreme caution) to the specific problem.

Biograph is a wonderful resource (more on this later) which enables researchers to form several hypotheses (Notice Known, Inferred keywords in the results) with which new solutions to medical problems may be found.

The subject of the analysis was not random. For more than 2 years a person who i will not disclose had several incidents of Sudden Hearing Loss which -luckily- were not permanent. Several ENTs have consulted him and dismissed this event as "Too much Stress" and "Idiopathic" after making sure that no other problems (e.g acoustic neuroma) were present.

Upon further investigation the person found to have an MTHFR C677 homozygous polymorphism and additional testing revealed elevated Homocysteine levels. After administration of 5-Methyltetrahydrofolate (an activated form of Folic Acid - Levomefolic Acid)  there were no further incidences of Sudden Hearing Loss.

The solution was originally found using BioGraph.

Venture Capitals in an Age of Algorithms (Revisited)

Some time ago i wanted to explore the idea of analyzing several kinds and sources of Information (e.g TechCrunch, TheNextWeb,  News sites and Twitter) to identify promising Investment opportunities in Technology and more specifically Startups. 

Here is a snapshot of a Webpage from TechCrunch :

In many posts in this Blog it was discussed how our Reactions for almost any kind of information are recorded. This was not possible when everyone was reading newspapers in its paper form whereas  now any kind of Text is associated with a number of Views, Re-Tweets, Number of Comments and FaceBook "Likes".

The second important information that is being generated is our Emotions for any Topic as these are expressed within Comments, Twitter and FaceBook posts. The intensity of our emotions is also captured and this information is very important since  whatever we associate with intense emotions really stays within our psyche, fuels our interest and (usually) drives our purchase decisions.

We may then continue with some Exploratory work as follows : We can collect Posts from various Tech sources and their associated Reactions, annotate the text with Sentiment, Events and Topics and analyze this information to understand which Topics and/or Events appear to have an affinity for a high number of Reactions or High Sentiment intensity for Startups or Tech Topics .

As an example, 10K posts from various Tech sources were collected and each one of the posts was marked as generating either HIGH or LOW interest based on the amount of Reactions (Re-Tweets, FB Likes, Comments)  that each post generated. Special filtering is applied for the frequencies of the words that appear in each post :

Then this information is fed to KNIME for further analysis. The implementation which  is shown here is rather naive and simplistic for many reasons :  Only keywords are used as input -as opposed to Topics, Events- and many other parameters that are involved and which will be discussed later but for our example we will keep things simple.

The workflow uses 3 algorithms namely PART (so that some rules are generated), SMO and Random Forests :

 
This -again- is a very naive approach which gave a result of 61.9% (F-Measure) in identifying keywords that commonly appear with posts that generate Interest vs posts that do not. We keep in mind that this knowledge alone is not enough with which a decision can be made but we decide to explore things a little further.

We may find that some words that we expected do appear in posts of High interest (such as Google, Apple, Pinterest). There could be however some words that deserve more of our attention such as Education and Schools which during the analysis appeared to exist more frequently in High Interest posts.

So how can this information be used for a potential investment on a Startup and is there really a way to model new ideas and predict their performance?  Again, it is not suggested here that if you come across a startup aimed in Education you should immediately put your money in but this observation could be one parameter to consider. There are so many other considerations such as whether the idea is novel or not, how many competitors exist, who are the people behind the Startup, whether its founders have created a successful Startup in the past, which people have already invested in the particular Startup, what is the "buzz" that this Startup has generated so far and so on.

Whenever we read about a new startup there are some immediate thoughts going through our minds : Does this sound like a good idea? Is it applicable to me and would it make my life easier? Is this idea truly disruptive or not? What does our "gut feeling" tells us?  

We should always keep in mind that there are limitations to what Predictive Analytics can do but perhaps we can extract some hints that we may then use to make better decisions.

It was also interesting to read this post (hence the use of word "Revisited" in this Post's Title) on Gigaom regarding the same Subject. This is a fascinating area that i started looking at and there will be similar posts in the future on this Subject.

Finding the Right Skillset for Big Data Jobs

Perhaps one of the key skills of a Data Scientist is the ability to be able to collect and access data that are not readily available. 

I was wondering about the trends in Job Postings and more specifically which skills and qualities employers (or agencies) search for on a candidate for a job in "Big Data" so i decided to use R to answer this question.

Of course, one must first find Data (in this case Job Postings) so that they may be analyzed. This is possible by using the library scrapeR of R to scrape content from websites that contain Job Postings. Once this is done, tm package can be used to analyze thousands of Job Advertisements so we may extract useful knowledge.

The analysis which you will see below is based at around one thousand Job Postings that contain the phrase "Big Data". Better pre-processing could help in getting better term co-occurrences but here i aimed in presenting the application.  Once the data are collected we can start by looking at the Frequency distribution of the words found (after removal of stop words) :

Note that the word 'big' is removed from the bar chart. Notice also how the term "experience" (which also includes occurrences of term "experienced") was frequently found in Big Data Job Postings.

Interestingly, the term "skill" (which also counts the term "skilled") is found way below in the frequency diagram.

Next we can use Text Analytics to find which words co-occur with topics of interest. We start by looking at which terms co-occur in Job Postings where Hadoop is mentioned :

Suppose that one wishes to better understand which skills are discussed along with the Java programming language :

 When it comes to skills, it appears that communication skills are those which are important  (as expected):

In the same manner we can  :

-Find the frequencies of skills of interest (e.g Java, Python, Ruby, NoSQL, Oracle DB) and generate trend charts for each of them.

-Run term co-occurrence analysis on the skills which are "good to have" or "preferred".

-Capture early trends on emerging skills (in the "Big Data" case, this could be Pig)

The idea of analyzing Job postings and CVs using Information Extraction (and then using Predictive Analytics once this information becomes structured) is quite interesting. The ability to extract inferred knowledge is also quite challenging : For example could we infer from the text found in CVs  :

-The total number of years of experience in Project Management of an Applicant in case that this has not explicitly been stated in his/her CV?

-Whether an Applicant shows a coherent Career growth through the years ?

-The years needed for an Applicant to move to a Managerial Position?

Food Data : The Next Target of Massive Analytics

It has been a very busy period since my last post but also a very interesting one.  At the Social Media Analytics panel of the European Text Analytics Summit there was a question on "What would you suggest to new Entrepreneurs when it comes to Text Analytics". The answer from most of us was "Specialize" :  Build an Exceptionally Good vertical solution.   

Text Analytics has been put to use for several verticals : Finance, Telecommunications, Pharmaceuticals to name a few. Perhaps the next important vertical for Text Analytics is something as Basic -but necessary- as Food. 

Using Analytics for the Food Market is not just about analyzing millions of Tweets to understand and detect Trends on Food consumption, identifying ingredient associations that are liked by Consumers (e.g Olive Oil => Garlic) and the sentiment that a Food experience creates.

Food Sector is a tremendous Market  : Super Markets, Restaurants, Chefs, Books, Magazines, Television Series and Consumers. So Insights from Food Data Analytics could be used by all the "knowledge consumers" mentioned above.

In other words :

- Can we identify emerging trends on the Food Market? And if we can, who are the possible recipients of this knowledge?

- Can we understand and suggest new Food Experiences according to several metrics found whenever Food is discussed in Social Media?

-What other potential sources can be used to collect and then analyze Food Data?

- Can we understand how consumers make choices when it comes to Food?

-Can we Predict Popular Recipes? And how can we monetize from this knowledge?

Text Analytics is  a key technology for transforming all the unstructured information on Food found on the Web. Predictive Analytics can be put to use if we can combine unstructured information with a target variable that we wish to predict.

One of the interesting tasks of a Data Miner is to be able to identify several -actionable and interesting- applications of both Data Mining and Text Mining given some Data. Of importance is also to find and/or to create new Data sources that can help in making better predictions. This is a challenging task but with careful considerations and lots of testing it may well prove to be a worthwhile and rewarding experience.

Coming back to Food Data we could potentially use mentions from Tweets, FB Posts and "Likes", Blog and  Website Posts to capture unstructured information. The hardest part is to be able to somehow incorporate more information about Consumer Behavior as this knowledge -and also to be able to predict Consumer Behavior - would be particularly interesting.

There is a limitation on what Analytics can do especially when we are talking about Predicting Consumer Behavior. As always, proper Data Collection, Pre-processing and thorough Testing is required to reach consistent results.

Text Analytics in Telecommunications - Part 3

It is well known that FaceBook contains a multitude of information that can be potentially analyzed.  A FaceBook page contains several entries (Posts, Photos, Comments, etc) which in turn generate Likes. This data can be analyzed to better understand the behavior of consumers towards a Brand, Product or Service.

Let's look at the analysis of the three FaceBook pages of MT:S, Telenor and VIP Mobile Telcos in Serbia as an example.  The question that this analysis tries to answer is whether we can identify words and phrases that frequently appear in posts that generate any kind of reaction (a "Like", or a Comment) vs words and topics that do not tend to generate reactions . If we are able to differentiate these words then we get an idea on what consumers tend to value more : If a post is of no value to us then we will not tend to Like it and/or comment it.

To perform this analysis we need a list of several thousands of posts (their text) and also the number of Likes and Comments that each post has received. If any post has generated a Like and/or a Comment  then we flag that post as having generated a reaction. The next step is to feed that information to a machine learning algorithm to identify which words have discriminative power (=which words appear more frequently in posts that are liked and/or commented and also which words do not produce any reaction.)

After performing this analysis we essentially come up with a list of words and a metric which tells us the discriminative power of each word. Here is an example of identifying these words :

(Note : Results based on a very limited Data Sample)

Keeping in mind that results shown are extracted from a very limited amount of data, the decision tree depicted above shows us that :

The presence of word Dragi (which means "Dear" in Serbian) means that a post usually does not receive reactions. This makes sense as many posts that reply to subscriber questions start with the word "Dear" and then the first name of the subscriber is added.

novo (="new") is a word that receives a lot of reactions along with hocu (="i want") and dopuna (=recharging credit for prepaid subscriptions). In the same manner we identify more words that are selected to be important in discriminating interesting vs non-interesting posts.  Note that we have to identify the correct context. For example we have to identify what the word novo refers to most of the time : A new cell phone or a new promotion?  From the sample analyzed It appears that :

 1) Subscribers "like" posts that discuss  New devices such as Cell phones and tablets (The Next Step could  be the identification of these devices)

2) Subscribers want new promotions (but we then need to find which types of promotions exactly)

3) Issues with incorrect re-charging are creating a very negative sentiment (but then we need to find  which operator co-occurs with this sentiment and for which cases)

In this way we are able to better understand subscribers, extract the Topics that they are interested in and take all this information into account when creating future initiatives. Note that with this way we can have hints on several potential "hot" topics such as Cell Phone and Tablet Brands, Tariffs, Services, Marketing campaigns, and that this can be performed for each Telco Provider page which means that we can analyze and identify the "hot topics" applicable for each Telco provider.

All the above along with several other uses of Predictive and Text Analytics for Telecommunications i will present in the upcoming European Text Analytics Summit in London, UK.

In the event that a Marketing or PR Agency uses -as in the ways shown above- Social Media Analytics to identify hot topics in News, Sports, TV, Banking and Consumer Goods  a Knowledge Base is created which has many uses : Imagine a scenario where a Telecommunications provider wishes to use a Sport event for a Marketing campaign. We could take into account the hot topics found from a "Sports" analysis and suggest ideas in a much more informed way.  More for this on the next post.

Text Analytics for Telecommunications - Part 2

In the previous post we have seen the problems that a highly inflected language creates and also a very basic example of Competitive Intelligence. The Case Study that i will present in the forthcoming European Text Analytics Summit is about the analysis of Telco Subscriber conversations on FaceBook and Twitter that involve Telenor, MT:S and VIP Mobile located in Serbia.

It is time to see what Topics are found in subscriber conversations. Each Telco has its own FaceBook page which contains posts and comments generated by page curators and subscribers. Each post and comment also generates "Likes" and "Shares". Several types of analysis can be performed to find out :

1) What kind of Topics are discussed in posts and comments of each Telco FaceBook page?
2) What is the sentiment?
3) Which posts (and comments) tend to be liked and shared (=generate Interest and reactions)?

For each FaceBook page post, an identifier is added to the post text which designates the origination page (either MT:S, Telenor or VIP Mobile) of the post. Prior the analysis of  a FaceBook Post which says "We want more promotions" we need to be aware that this text originated -for example- from the MT:S FaceBook Page and not Telenor's.

Identifying the topics discussed in Telco subscribers posts and comments has a number of benefits. We gain a better understanding in the areas that a Telco should focus on. If we find that the topic of INTERNET is on the top  list of discussions repeatedly then this is where a Telco should pay attention. If we find Network mentions  to be associated with a competitor Telco repeatedly (which most likely is not good) we can choose the right time to air commercials implying that we are constantly working for better Network coverage.  We can also identify how much "buzz' was created from a new marketing campaign or  a new phone offer and the sentiment associated with it.

Let's have a look at the first type of analysis, namely Topic Detection. Using Information Extraction we  identify the Topics mentioned in thousands of FaceBook posts and comments for a particular period. Here are the results :

In terms of user engagement on FaceBook, MT:S is the winner since in absolute numbers, its FaceBook page contains more posts and comments than the other two FaceBook Telco Pages. Notice the frequencies of other topics found such as SMILEYs, INTERNET, PROMOCIJA (= Promotion), MREZA (=Network), POSTPAID and ANDROID.

With the chart shown above we become aware of the distribution with which Topics are discussed on all three FaceBook Pages. We do not know what is being discussed for now but we know that subscribers  talked more about Internet, then Promotions (=PROMOCIJA), then Network (=MREZA) and so on.

Let's look at  what Topics exist with PROMOCIJA (=Promotion). In other words, which other Topics are found in FaceBook Posts when Promotions are mentioned? Here are the results :

Most posts collected that discuss about Promotions are actually posts found on the MT:S FaceBook page. Notice also that the presence of Topic HOCU (=I want) which tells us that subscribers simply state that they want new promotions. Here is what the picture looks like for topic INTERNET :

So Telenor is found more frequently in INTERNET mentions. However caution is required since we do not know if all of  these Topic distributions found and their associations with specific Telcos can be attributed to pure chance or not.

It is very important to be confident enough to communicate to any Telco that being associated with Network mentions or any other Topic  is -or is not- simply a random event.

Text Analytics for Telecommunications - Part 1

As discussed in the previous post, performing Text Analytics for a language for which no tools exist is not an easy task. The Case Study which i will present in the European Text Analytics Summit is about analyzing and understanding thousands of Non-English FaceBook posts and Tweets for Telco Brands and their Topics, leading to what is known as Competitive Intelligence.

The Telcos used for the Case Study  are Telenor, MT:S and VIP Mobile which are located in Serbia. The analysis aims to identify  the perception of Customers for each of the  three Companies mentioned and understand the Positive and Negative elements of each Telco as this is captured from the Voice of the Customers - Subscribers.

By analyzing several thousands of Tweets and FaceBook posts and comments we can have a first glimpse of Competitive Intelligence. For example when we wish to identify which words frequently occur with mentions about postpaid packages this is what we find  :

Red boxes show Telco Brands - notice "mts" and "mtsa" which point to the same Telco, namely mt:s.  Blue boxes indicate similar words that should be merged.  From a first look at the results above we see that : 

a) mt:s is found more frequently when users mention PostPaid packages.

b) Telenor and VIP Mobile are not found as frequently as MT:S in PostPaid package conversations.

c) We see several  problems from insufficient pre-processing : Kredit and Kredita (=credit) should merge into one word, the same applies for telefona - telefon, internet - interneta and mts - mtsa.



Notice that we can perform the same High-level analysis for several Telco Topics such as Network, Billing, Customer Care, Promotions, Questions of subscribers and so on. The next task is to identify the reason(s) why MT:S was found to have more mentions about PostPaid packages. Note that at this point we do not know why this is so : It could be the fact that MT:S prices of prepaid packages are high, very cheap or something else is happening that needs to be identified.


The Serbian Language poses extra work because it is a highly inflected language : Even the ending  of  Brand names change according to the usage.  Consider the following :

U mts-u (at mts)
Sa mts-om (With mts)
Bez mts-a (Without mts)


It is evident that a highly inflected language explodes our feature space and for this reason R can come to the rescue with some success. We can use R for changing several synonyms to one word, removing (Serbian) stop words, removing URLs and performing several other pre-processing steps that are necessary prior to an extensive analysis. More on the next post.

Case Study : Competitive Intelligence for Telecommunications

Telcos are a good example of a fast moving business environment and a good candidate for using Competitive Intelligence analysis from Social Media sources. The Case Study involves three major Telcos located in an Eastern European Country and shows the results from the analysis of thousands of Tweets and FaceBook wall posts to understand the following :

- How subscribers perceive each Telco Brand? 

- Which information do subscribers tend to Re-Tweet and "Like" on FaceBook Wall Posts? 

- Which words and Topics are commonly found with Intense feelings / thoughts?

- Which topics are mostly discussed when subscribers compare two or more Telco operators?

- What do subscribers discuss about  Network Quality and Speed, Billing, Promotions, Marketing Events, Customer Care, TV Commercials etc.

- How do they prioritize these topics and which of them are interesting and why?  

- What do subscribers talk about in general (i.e without any Telco Brand being mentioned) regarding Internet speed, Charges and what would they expect to see more?

I will present the Case Study mentioned  above in the forthcoming 9th Annual European Text Analytics Summit in April in London - UK. The Case Study is an example of application of Text Analytics to a language for which currently no tools exist and thus all difficulties and possible solutions will also be discussed. Examples will be also given on analyzing information to different conceptual levels and how this technique provides even more insights in consumer behavior.

The following tools were used for the analysis : 

- GATE to annotate all Topics that occur within Telco conversations (such as "sms", "internet", "dropped call", "network","promotion") and for setting up Conceptual Levels.

- R for pre-processing Text and performing Text Classification, Topic Detection and Cluster Analysis.

- WEKA  for Feature Selection and Text Classification.

- Finally,  Java is used to manage the information that is generated from GATE such as  understanding how subscribers prioritize various Telco Concepts and Topics and also identify important phrases and/or words that frequently occur when these Topics are being discussed.  

New Insights from Text Analytics

Text Analytics has gained the attention it deserves in the past few years. Sentiment Analysis is perhaps the most frequently discussed type of analysis but there will be always new ways to analyze and gain insights from text data.  

Examples of new types of analysis -and they have a vast potential- are in my opinion two :  Sequence Detection and Concept Mining. I am not aware whether  these types of analysis are currently being implemented by any Text Mining practitioner at the moment and if there is one, feel free to add your comments below.

So what is Sequence Detection and Concept Mining ? Some examples :

Suppose that you receive several similar e-mails sent from customers as the one seen below :

"I have been trying repeatedly to solve my billing problem through customer care. I first talked with someone called  Mrs Jane Doe. She said she should transfer my call to another representative from the sales department. Yet another rep from the sales department informed me that i should be talking with the Billing department instead. Unfortunately my bad experience of being transferred through various representatives was not over because the Billing department informed me that i should speak to the......"

Currently Text Analytics software will identify key elements of the above text but a very important piece of information goes unnoticed. It is the sequence of events which takes place :

 (Jane Doe => Sales Dept =>Billing  Dept =>...)


Being able to detect the sequence of events is an important element in understanding customer interaction. In our example above, imagine the possibility of detecting similar sequences through thousands of e-mails or call center transcripts and running a sentiment analysis, a process which then could correlate sentiment with specific event sequences.

Next,  is the usage of Concept Mining (this is just a phrase i coined for this post) : Being able to analyze information to different conceptual levels. A very powerful technique indeed and let's see why this is so.

People that have attended the 7th annual Text Analytics Summit in Boston had the opportunity to listen to several presentations regarding Semantics. The discussions between experts from the Semantics Panel and the attendees revealed that people could not find Semantics practical for several reasons. Yet, in Semantics lies the power of being able to find patterns on different conceptual levels.

As a -very basic- example, if we use Information Extraction to annotate -say- the Tweets containing mentions of American Telcos we can tag each one as a more general category called TELCOS. We can also tag individual prepaid packages as a more general category called PREPAID_PACKAGES. By doing that we can then search for patterns in a more general conceptual level than searching for patterns only at a Telco Brand level or a specific Telco's prepaid package. As an example we can run  a sentiment analysis on all prepaid packages mentions,  identify patterns of negative or positive sentiment and see which Telco is the winner of positive sentiment at a conceptual level.

The possibilities are endless.

Big Data : Case Studies, Best Practices and Why America should care

We know that Knowledge is Power. Due to Data Explosion more Data Scientists will be needed and being a Data Scientist becomes increasingly a "cool" profession. Needless to say that America should be preparing for the increased need for Predictive Analytics professionals in Research and Businesses.

Being able to collect, analyze and extract knowledge from a huge amount of Data is not only about Businesses being able to make the right decisions but also critical for a Country as a whole. The more efficient and fast this cycle is, the better for the Country that puts Analytics to work.

This Blog post is actually about the words and phrases being used for this post : All words and phrases on the title of the post (and the introductory text) were carefully selected to produce specific thoughts which can be broken down in three parts :

•  Being a Data Scientist has high value. 

• "Case Studies" and "Best Practices" communicate to readers successful applications and knowledge worthwhile reading.

• "America should". This phrase obviously creates specific emotions and feelings to Americans.

"Case Study" and "Best Practices" were phrases found to be commonly associated with posts of high visibility. You might also get many views if you create a post which proves that whatever concept you are writing about is the right thing to do (for example write a post that clearly demonstrates yet another reason to use Social Media and have this post shown to Social Media Professionals).  Regarding our example : It is very probable (and logical) for Data Miners to look at and then re-tweet (or otherwise share) information which is a "proof" about Data Mining being useful  and also a "cool" profession. The higher concept / motive which works behind the scenes is that "I am doing the right job and this post proves it".

You might also get many views by submitting a post which disproves well-accepted concepts or posts that demonstrate the difficulties that well-accepted concepts face : For example, if you were a Data Scientist or a BI Professional, you would be inclined to read a post titled "Big Data is a Big Hype".  Whether you will re-tweet or share the post is of course under your discretion. At this point it should be noted that there is a big difference between number of clicks of a post and the number of shares it got (by Retweeting it, Liking it, etc) because sharing a post means that this post is considered worthwhile to read.

All of the above (and much more) have been found by analyzing thousands of Blog posts along with their number of clicks and shares they got (either by RT's , FaceBook "Likes", etc) and this is what i will be presenting in Text Analytics World in New York this October. It was also very interesting to see that some findings are in tandem with findings discussed by Joseph Carrabis during the Text Analytics Summit 2011 in Boston back in May. 

Of course it is not suggested  that   by using specific words and phrases you are guaranteed a successful post being re-tweeted from thousands of people and there are many reasons for this which i will not get into here. Additionally, Text Analytics cannot infer the higher meaning and concepts suggested within Text and this problem deserves a post on its own. This analysis however identifies concepts and/or phrases that point Bloggers and Marketers to look at a specific direction and with this knowledge to have increased probabilities for a successful Web presence. Again, this is an example of true Social Media Intelligence. Not (just) Reports.

So, in case that this post title immediately got your attention from other posts, you've just had a little taste of Predictive Analytics in action.

Apple Products on Twitter - A Text Analytics example

My presentation on the 7th annual text analytics summit was a tutorial in one of the methodologies one could use to analyze unstructured text. The sample consisted of 365000 tweets that contained keywords of Apple products and concepts such as iPad, iPhone, iPod, Apple Store, Mac, Steve Jobs and the goal was to get an understanding of what people where tweeting about each product or concept. The first step is to use a text analysis toolkit (i used GATE) to annotate the tweets and identify which concepts and keywords occur within the tweets. But this is not always easy. Take the word Mac for example. According to the context, Mac could be a computer type,  a burger type, the MAC beauty products or Mac Arthur airport. So when a query sent to Twitter API that contains the word  Mac we end up with lots of erroneous information. So one of the things that have to be done to ensure good results  is word sense disambiguation. We know for example that if a tweet contains a word such as fries, lettuce and/or salad then quite likely the word Mac that was also found within this tweet was about the Big Mac (even though the word Big may not be present). If we find the word Arthur next to the word Mac then the tweet is about the Mac Arthur airport, etc. Here is GATE in action, identifying different keywords and concepts in Tweets :

Now we can see which concepts and keywords appear frequently in Re-Tweets ('USER' denotes that a '@' was present in the Tweet, 'URL' that a URL link was found in the Tweet,etc)

 We can also see which words frequently occur with iPhone5 :

Event Detection: Analytics becoming more personal

Sentiment Analysis is a hot technology at the moment. Marketers are interested in the perception that consumers have about  a specific brand, product or service as this is found in unstructured text. Some people claim that Sentiment Analysis does not meet their expectations but also that it is not straightforward for a company to find the "right" solution.  Comparing different Sentiment Analysis solutions could prove a difficult task.

Marketers and Decision makers need insights with which they can make better decisions - They need both Reports and Intelligence. Therefore the question that always follows the finding that "Your product has a 35% negative sentiment in the past 10 days" is "Why".  Social Media Monitoring tools must also  provide actionable Intelligence. 

All this is important information as it shows why your Brand / Product / Service could be losing customers. You monitor what is being said, identify whether a negative or positive Sentiment Trend is declining or rising and take necessary actions accordingly.

One of the questions i often get is what other applications can emerge from using Text Analytics and Data Mining. With Text Analytics and Data Mining we can find behavior patterns on many levels and -assuming that information such as Tweets will keep coming- the understanding of consumers can  go to the next -and sometimes more personal- level.

One of these applications is Event Detection. I am not aware if Event Detection is provided by any tool at the moment but i believe that this type of analysis could become a next major source of consumer insights. But what exactly is "Event Detection"?

Since we are able to have a computer automatically identify whether a phrase contains positive, negative or neutral sentiment, perhaps we could use Text Analytics and Machine Learning to detect that a specific event has occurred to an individual from the Tweets that someone posted such as "i've just returned from holidays". But that's not all. We can mine for patterns of consumer behavior given the fact that an event has occurred. And that potential knowledge from such an analysis could be very powerful. Because apart from the emotions that a product / service / person generates, the same applies for events happening in our lives. These events and the emotions they create can sometimes change our lives and also drive our decisions. A logical next step is to collect several behavioral Data and use Data Mining to analyze this information.

I will discuss an example of using Event Detection towards the end of my presentation on the 7th Annual Text Analytics Summit in Boston this May along with the reasons for such an analysis being important and i am looking forward to the reactions.

The fact is that with more insights, privacy issues arise even more and I get an increasing number of people asking me about privacy. I was also interviewed by a major British newspaper last month on what companies can learn by applying "Super Crunching" on Tweets. I tried to show both worlds of "Super Crunching" but the truth is that consumer insights become more personal as companies understand the value of structured and (more recently) unstructured information. 

7th Annual Text Analytics Summit

I would like to say a few words about an upcoming major event for all of those interested in Text Analytics and its various uses in Social Media, Marketing and Business Intelligence. Starting on May 18th, the annual Text Analytics summit (the only conference dedicated completely to Text Mining) will take place in Boston, MA with a total of 28 speakers presenting material on applications of Text Analytics including :

• Social Media Analytics

• Sentiment Analysis

• Voice of the Customer

• Marketing

• Semantics

Well-known names in the industry will be there ( Seth Grimes, Tom Anderson, Gregory Piatetsky-Shapiro, Ronen Feldman) as well as experts from companies such as SAS, IBM, Forrester Research, Attensity, Adobe, J.D. Power&Associates, Clarabridge and others.   

My presentation will be about Behavior Mining in Social Media using Text Analytics and i will be giving a step-by-step tutorial  on the analysis of data originating from Twitter regarding a major Electronics Brand in USA. More specifically :

• I will show how Tweets can be transformed and then analyzed using various statistical NLP techniques and software. 

• Discuss the various problems that are found when one wants to analyze Text data

• Discuss and introduce new ways of seeking for valuable information and extracting insights when it comes to Mining Behavior.

Although the Case Study will be using data from Twitter, the techniques shown can be applied to any other Text Data such as those found in FaceBook, Blog posts, User comments, etc.

I am looking forward to seeing the work done by others, learning about successful applications of Text Analytics and the knowledge gained and also seeing the issues that professionals come across and how these are faced by them.