Summarization of Blog posts with "Web Pulse" Reports

In the past couple of months i was looking for a way to best capture and understand what happens on the Web -and more specifically what people write in blogs- in terms of sentiment and emerging trends. The first thing that i came up with was the the idea of creating a "Web Pulse" Report : A way to summarize what people are discussing on the web. Although the implementation was not as complex as i expected, i was pleased to find that the knowledge that can be extracted is -to say the least- very useful and interesting. Before looking at an actual Report examples here are the elements that comprise it :

1) Concept Frequencies : Identifies the concepts that bloggers most frequently write about

2) Global co-occurence Matrix : Identifies most frequent word bigrams

3) Keyword Associations for Concepts : Which keywords tend to co-exist with a specific concept?

4) Most frequent n-grams associated with a given Concept (where n=2,3,4,5)

As an example we will identify what bloggers were discussing in Greek blogs on July 27th, 2010 and specifically the Blog titles in more than 300 Greek blogs.

Here are the concept frequencies found (in descending order) on that date :

[Turkey]=178
[Politics]=128
[Economy]=101
[International Monetary Fund - IMF]=62
[Banking]=61
[Public Sector]=50
[Negative Characterizations]=30
[Political Parties]=29
[George Papandreou]=29 (=Prime Mininster of Greece)
[Loans]=22
[Society]=20

The first interesting fact was that "Turkey" appears to be in the top of the list of Greek blog articles, even though Greek mass media did not place so much weight in the latest Turkish behavior in the Aegean sea on that day. The second concept is Politics with the Economy following next.

Here is the top part of the Global Co-occurence Matrix found (in Greek) :

ΣΚAΦΟΣ,ΤΟΥΡΚΙΚΟ : 25
ΡΕΙΣ,ΤΟΥΡΚΙΚΟ : 25
ΠΙΡΙ,ΤΟΥΡΚΙΚΟ : 25
ΡΕΙΣ,ΣΚAΦΟΣ : 24
ΕΛΛAΔΑ,ΧΩΡΑ : 23
ΥΠΟΥΡΓΕΙΟΥ,ΟΙΚΟΝΟΜΙΚΩΝ : 22
ΡΕΙΣ,ΕΡΕΥΝΗΤΙΚΟ : 22
ΠΙΡΙ,ΕΡΕΥΝΗΤΙΚΟ : 22
ΡΕΙΣ,ΠΙΡΙ : 21
ΑΝΑΜΕΝΕΤΑΙ,ΣΥΜΦΩΝΑ : 21

ΠΟΛΙΤΙΚΗ,ΧΩΡΑΣ : 19
ΟΙΚΟΝΟΜΙΑ,ΕΛΛΗΝΙΚΗ : 19
ΜΟΝAΔΩΝ,ΔΕΗ : 19
ΚΥΒΕΡΝΗΣΗ,ΠΑΠΑΝΔΡΕΟΥ : 19
ΗΓΕΣΙΑ,ΠΟΛΙΤΙΚΗ : 19
ΕΡΕΥΝΗΤΙΚΟ,ΤΟΥΡΚΙΚΟ : 19
ΥΠΟΧΡΕΩΣΕΙΣ,ΜΝΗΜΟΝΙΟ : 16
ΧΩΡΑ,ΜΝΗΜΟΝΙΟ : 15

The top 4 frequent keyword associations is -again- about the latest problems of Greece with Turkey and more specifically with the fact that a Turkish boat named "Piri Reis" (in Greek : ΠΙΡΙ ΡΕΙΣ) has been repeatedly entering without a permission a Greek part of Aegean Sea.

Let's look at the Associations frequencies found between specific Concepts : The following is an example of concepts associated with "Giorgos Papandreou" (Greek Prime Minster)

International Monetary Fund - IMF=32
Politics=28
Political Reform=6
Nea Dimokratia=3 (=Oppositional Political Party)
Politics, International Monetary Fund, Loans,Political Parties=2
Negative Sentiment=2
Public Sector=2

Uncertainty=2

It appears that George Papandreou is frequently mentioned where the IMF is involved and also a political reform might be on its way.

The fourth element of the report shows phrases that are commonly found in Blog posts. Since many blogs tend to use the same titles, with this functionality one is able to look at the distribution of the information from one blog to another.

The report can be enhanced in various ways : For example by tokenizing Blog posts in sentences i have added the option of performing chi-square tests to identify co-occurences in a more concise way, rather than using strictly absolute term frequencies. Through different types of analysis and knowledge representation we are able to look to our subject(s) of interest in different ways, which -hopefully- leads us to better insights.

From my experience so far, this type of report is a simple but efficient way to summarize the content of Blogs and also show what is 'hot' at the moment and why.

Concept Trending : A Glimpse into the future?

In the previous post some ideas were presented on the trends of Text Analytics. Analyzing and extracting knowledge from text is a hard thing, whether this involves Sentiment Analysis, Text Classification, Cluster Analysis or Information Extraction.

A particularly interesting application of Text Analytics is the identification of trends for specific concepts. In contrast with simple keyword trending, this type of trending attempts to disambiguate keywords according to their context and use co-reference resolution to identify the subjects for which the sentiment relates to.

To better understand concept trending let's look at an example : Suppose that one wishes to identify the trend of negative characterizations -and even swear words- that exist on the Greek web. The first step would be to collect the information from various blogs and forums whenever a negative keyword is found. A Text analysis toolkit could then provide the means of identifying the subject(s) of negative characterizations on the Greek web such as Politicians, the Economy or the International Monetary Fund which recently came in to the rescue.

From a post dated December 28th, 2009 :

"Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness."

Although not stated on purpose, the country which the article addressed was Greece and the trend increase on negative sentiment was found to be starting in the beginning of December 2009. This is a photo of a Greek newspaper taken on February 4, 2010

The title shown writes about the "Fear of Social Explosion". On May 6th 2010 after clashes in the center of Athens, mentions about "Social Explosion" in Greece started appearing on the Web. The following Google search uses a timeline for "Social Unrest". The increase of mentions appears to be starting on February 2010.

Although concept trending has significant challenges it is a process which in my experience has proven itself many times. A recent article at NewScientist suggests that by capturing the sentiment of the crowds we are able to predict the moves of S&P 500 or by looking at keyword searches such as "job search engine" we can predict coming changes of the US unemployment rate.

The future and trends of Text Analytics

I recently attended a GATE seminar on the University of Sheffield. Having used GATE for quite some time now, i was happy to see that the GATE team is well committed to developing the GATE Text Analysis Workbench by constantly adding more functionality.

Although many of the participants were PhD students i was also happy to see people from companies that now wish to leverage the hidden knowledge that exists in unstructured text. Whether it was analysis on text of Patents information, intelligent search on Text of Photo Captions for a large News Agency or understanding what a customer wants, Text Analytics are becoming an important tool for making better decisions.

I also had the opportunity to speak with several people about the future of Text Analytics. What are we likely to see happening in the next years on Information Extraction and Text Analytics?

First we have to understand how Text Analytics deliver results. In order for a computer to 'understand' unstructured text, it should be 'taught' that the word 'Dollar' is a currency of a country that is called 'US' and also that US, United States, USA and U.S.A is the same concept. This means that hundreds of thousands of concepts and synonyms have to be specified so that a computer identifies them in unstructured text. This process is called Text Annotation.

The Golden Standard of Text Annotation is annotations done by humans : A computer sifts through the text of a web page, annotates it with concepts and then these annotations are checked against annotations made by humans on the same text to assess the accuracy with which a computer 'understands' this text and the concepts and entities that exist in it.

So what does the future hold? First of all, since unstructured text becomes more available there will be a greater need for 'annotation farms' : Groups of people who will be manually annotating free text, identifying an ever-growing number of Companies, Managers, Politician names, or anything else that has to be 'taught' to a computer. Note that Annotation Farms already exist but the need for this service will become greater.

The second trend on Text Analytics could be something equivalent to what we have seen happening with NetFlix. Suppose that you own a company that produces Brand 'X' and you wish to track the reputation of your product online. You would then submit a sample of your product's mentions to various companies that analyze text and have them compete against each other in terms of -for example- Precision and Recall. The one that produces consistently the best metrics (whether Precision - Recall, Kappa statistic or F-Measure) will also get the job.

A third trend could be the development of text analytics for specific concepts : Sentiment Analysis and Named Entity recognition is hard work if one wants to produce sound and accurate results. So it could be probable that Text Analytics experts will choose a specific concept -For example reputation of Banks- and then work in the analysis of this -very specific- concept so that they achieve better metrics.

Predictive Analytics and Politics - Part 2

In the previous post we have seen an example of analyzing messages sent from citizens regarding a new taxation plan. We identified some correlations between keywords and concepts but there are more ways to gain knowledge from such unstructured information.

By using Cluster Analysis we can extract groups of similar concepts among thousands of comments written by citizens but also presenting an order within them. Let's assume that Cluster Analysis reveals the following clusters (or similar concepts) within submitted messages :

- battling tax fraud

- requests for a fair tax plan

- requests for less taxation for large families

- various incentives for citizens

Our problem is finding the order of importance that people place on the various concept categories shown above : Is battling tax fraud considered more important (=discussed more frequently by citizens) than requesting a fair tax plan? How about taxation for larger families?

A cluster analysis can reveal to us the size of each cluster and -as a consequence- how important each cluster is :

We make the assumption that in the text representation shown above Cluster 5 (which contains 329 citizen messages) is about requests for a fair tax plan and Cluster 10 contains messages with requests that tax fraud should be minimized. It appears that significantly less people are concerned with a battle against fraudulent activity but they request -more immediate- benefits through a fair tax plan.

Collecting and analyzing information found in blogs and forum entries is another area of analysis that could prove very interesting. Let's see an example with the Political / Social / Economic situation in Greece : The goal is to identify and extract trends and co-occurences of key concepts from blog titles and forum posts such as :

- Names of major Political parties

- Names of Politicians

- Economy (words/phrases such as "austerity plan")

- Negative characterizations

- Company Names

...etc

For this kind of data several applications can emerge. We could track specific concepts through time and see their trends. We can also identify which concepts are discussed together. As an example we could identify the reasons on why Giorgos Papandreou (PM of Greece) is characterized in a bad way in blog posts. (= what other concepts are found in Blog posts containing keywords 'Giorgos Papandreou' AND Bad Characterizations?) :

(Note : PASOK = Governmental Political Party )

Politics = 120

Economy=72

Economy, Politics=40

PASOK=24

Politics, PASOK, Referendum=8

Economy, Politics,PASOK,Referendum, Immigrants=8

Economy, Politics, Society=8

Society, PASOK=4

In other words : Giorgos Papandreou is criticized mainly for his Political decisions and the Economy followed by criticism on PASOK. Negative sentiment also exists because of the fact that a percentage of Greek citizens require that a referendum should take place concerning the latest decision of the Greek government to give to a large proportion of Immigrants the Greek citizenship.

Predictive Analytics and Politics - Part 1

One of the most interesting applications of Data/Text Mining and Information Extraction is Politics. I started collecting information from various blogs, websites and forums and applying Information Extraction and Data/Text Mining techniques to extract potentially useful knowledge in this area. By combining different pieces of information one could come up with trends that may tell us what lies ahead of us.

The latest developments in Greece are more or less known to most of people that read International News. The situation is difficult and the voice of citizens in various blogs and forums could give us the sentiment of Greek Web Users. For example :

- Which are the most frequently occurring words?

- Which are the most frequently occurring thoughts?

- What are the things that have to be changed by Greek politicians?

To answer these questions i have started collecting information found on the top 120 Greek blogs, the OpenGov website (a state-run website where Greek citizens express their opinions) and a couple more Greek sites of economic content. For blogs and forums a Java program scans every 20 minutes for new information :

This information is then sent to an annotation engine which analyzes the textual content. Once the text is analyzed we can -for example- produce a keyword vector that we can later use to understand what citizens are saying on the Web. We can then find out answers to many interesting questions such as :

- With which words is Mr George Papandreou (PM of Greece) associated with?

-When there are some very negative words (such as swearing) what other words are found in the same text?

- What does keyword trending tell us? (For example, we identify an increasingly number of swear words in citizen posts)

First let's see some examples regarding the OpenGov website where thousands of citizens have expressed their opinions on the tax policy of the Greek state. The following chart shows us a number of pairwise correlations between written words in these comments :

Under the red rectangle appear two words (dikigoros,iatros) which in Greek mean "Lawyer" and "Medical Doctor" respectively. This essentially tells us that these two professions are used together frequently in citizen discussions. By looking closely at these messages one can reveal that professionals in these two sectors are said to avoid taxes by not issuing receipts.

Next we could use association rule learning to look for some more -potentially interesting - rules :

The highlighted rule although one of low support it could prove interesting : A subset of citizens are requesting that freelancers and the self-employed should be more closely monitored for tax fraud.

Apart from rule learning, it is interesting to identify the proportion of the total dataset for which each rule holds. That also gives us a sense of order with which different ideas and thoughts exist on the mind of citizens.

In the next post : What is the Voice of the Citizen tells us in Blogs and forums?

Detecting Novelty in Twitter posts

A question one could come up with is the following : How can we easily identify and extract novel information from the web? Although we could apply this "novelty detection" into many areas i would like to discuss for now the idea of semi-automatically identifying novelty among posts on Twitter.

Let's take for example the IPhone. Thousands of Tweets are generated every day regarding the Apple IPhone. These tweets mainly discuss about :

• Which new apps are available / used / liked.
  
• New accessories (cases, chargers, etc)
• User Experiences and sentiment (such as blaming IPhone's short battery life)
• Pros and cons of the IPhone vs other similar devices
• Upgrading / hacking etc.

So the problem is : How can we identify novel information among thousands of tweets? Some would argue that we should first define what is "novelty" such as finding a new application or a new accessory for the famous mobile device. Others might argue that novelty is a customer idea that not many people about the IPhone thought about and for which Apple would be interested in identifying among thousands of Tweets. As an example consider the following Tweets :



A subset of users experiences problems with the automatic orientation of the IPhone : This subset of IPhone users is perhaps very small but identifying these tweets could give Apple some ideas to work on.

Here is another subset of Tweets that talk about the charger's cable length :



In the example shown above notice that using just "iphone cable" as search terms would return a large number of Tweets, making it hard to identify novelty among all these Tweets.

Searching for novelty and identifying new ideas among Tweets is not an easy task. The problem is that we do not know what we are looking for in the first place : We can define the general context -such as wanting to identify novelty in user experience- but then we come to a halt in terms of what techniques to use (with an exception perhaps being cluster analysis).

The potential of using semi-automatic novelty detection on Twitter and other websites -such as delicious links- is very big. Although this is work still in progress, the general methodology of novelty detection in Twitter could be to :

1) Collect a large subset of Tweets mentioning IPhone and a keyword that identifies context (such as the word charger).

2) Identify keyword frequencies

3) Generate search queries using a subset of keywords chosen in an "intelligent" way, otherwise the number of search queries would be practically impossible to be evaluated.

4) Test these combination of keywords by submitting them to Twitter search and evaluating the results.

Steps (3) and (4) shown above are the key to success of course. In our example about the IPhone cable being too short we had results returned because the combination of keywords submitted could make sense. Trying out IPhone, cable, snow tells us that such keyword combination is not a valid one and -hence- not an "intellligent" keywords subset :

Building a Knowledge Hub

The web is a huge source of information. It stores facts, thoughts, feelings and intentions of people. It also records what people like and what they don't in an indirect way - something that we are going to be looking at shortly . Some of the examples on harnessing this information were shown previously in this blog, such as :

• Extraction of user opinions, beliefs and values from Twitter

• Prediction of popular stories on Digg

• Prediction of popular Tweets

Consider the following snapshot from a BBC webpage :

The table above shows a representation of the most popular business stories on BBC on the 22nd December 2009. Even though we do not have specific metrics, we intuitively understand that the order with which the stories are listed also tell us the popularity of each post. Notice that the first post on the most read stories talks about the British economy while the last one is a title regarding football.

This is knowledge that we can harness. No doubt it is a very specific kind of knowledge because it tells us only what -mostly- British readers of BBC have found interesting. In other words this is knowledge for a specific population : Most likely in another country -say France- the title about UK being still in recession would not be so interesting but a title about France being in the same situation would. Subject, Time and Location are all important parameters that need to be captured and taken into account.

Let's consider the idea of creating a Knowledge Hub : This could be done by collecting massive amounts of information from Social Media, blogs, comments from forums and news titles (and their popularity). Techniques such as Information Extraction with concept annotation, Data and Text Mining could be used to extract knowledge by combining incidents, opinions, intentions and emotions found from different sources.

I have been monitoring and collecting for the past 3 months news and forum posts generated from/for a specific country. The information collected is then annotated in such a way to extract concepts. This text annotation is matched with keywords of concepts, incidents and intentions. Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness.

It is a very interesting fact that our behavior is recorded -up to a point- by the web. Again, the key is the way that we are able to organize this information into logical chunks and then use this representation to find possible insights.

2009 has been a year of big changes. Best wishes for a Happy and Prosperous New Year for everyone.