Sequence Data Mining for Health Applications

An often overlooked type of Analysis is  Sequence Data Mining (or Sequential Pattern Mining).


Sequence Data Mining is a type of Analysis which aims in extracting patterns sequences of  Events. We can also see Sequence Data Mining as an Associations Discovery Analysis with a Temporal Element.

Sequence Data Mining has many potential applications (Web Page Analytics, Complaint Events, Business Processes) but here today we will show an application for Health. I believe that this type of Analysis will become even more important as wearable technology will be used even more and therefore more Data of this kind will be generated.

Consider the following hypothetical scenario : 

A 30-year old Male patient complaints about several symptoms which -for simplicity reasons- we will name them as Symptom1, Symptom2, Symptom3,etc.

His Doctor tries to identify what is going on and after the patient takes all necessary Blood work and finds no problems. After thorough evaluation the Doctor believes that his patient suffers from Chronic Fatigue Syndrome. Under the Doctor's supervision the patient will record his symptoms along with different supplements to understand more about his condition. Several events (e.g a Visit to the Gym, a stressful Event) will also be taken under consideration to see if any patterns emerge.

-How Can we easily record Data for the scenario above?

-Can we extract sequences of events that occur more frequently than mere chance?

-Can we identify which sequences of Events / Food / Medication may potentially lead to specific Symptoms or to a lack of Symptoms?

Looking the problem through the eyes of a Data Scientist, We have :

A series of Events that happen during a day : A Stressful event, A sedentary day, Cardio workouts, Weight Lifting, Abrupt Weather Deterioration, etc

A Number of Symptoms : Headaches, "Brain Fog", Mood problems, Insomnia, Arthralgia, etc.

Let's begin with Data Collection. We first suggest to the patient to use an Android app called MyLogsPro (or some other equivalent application) to easily input information as this happens :

  

So if the patient feels a specific Symptom he will press the relevant Symptom button on his  mobile device. The same applies for any events that have happened and any Food or Medication taken. As the day passes we have the following data collected :

The snapshot shows what happened starting on the 20th of August 2014, where our patient has logged the intake of Medication (at 08:22 AM) and/or Supplements upon waking up then a Food entry was added at 08:47. At 11:06 the patient had a Symptom and immediately reached his phone and pressed the relevant Symptom (Symptom No 4) button.

After many days of Data Collection we decide that its time to analyze this information. We export the data from the application as a csv file which looks as follows :

We will use KNIME to read the csv file, change the contents of the entries accordingly so that an Algorithm can read the events and then perform Sequence Data Mining. We have the following layout :

 The File Reader reads the .csv file, then during the Pre-processing block (shown in yellow), a String Manipulation node which removes colon (:) from time field (e.g 12:10 becomes 1210). The Sorter sorts the data according to date then time as the second field and a Java snippet uses replaceAll() function to remove all leading zeros from Time field (e.g 0010 becomes 10).

The R Snippet loads the CSPADE Algorithm and then uses this Algorithm to extract pattern of sequences.

After executing the stream we get the following output :

The information consists of two outputs : The first one is a list of sequences along with their support and the second one contains the output from rule induction which gives us two more useful metrics (namely the lift and the confidence for each rule).

We immediately notice an interesting entry on the first output :

Medication1->Symptom2

and on the second output we see that this particular rule has a lift of 1.4 and 0.8 confidence.

However, as Data Scientists we should always double-check the extracted knowledge and must be aware of pitfalls. Let's see some examples (list not exhaustive) :

1) The algorithm does not account for time as it should : As an example, consider the following entries :

10/09/14,08:00,Medication1
10/09/14,08:05,Symptom2

We assume that Medication1 is taken by mouth and needs 60 minutes to be properly dissolved and that these entries occur frequently enough in that order in our data set. Even though the algorithm might show a statistically significant pattern , it is not logical to hypothesize that Medication1 could be related to Symptom2. The Analyst should first examine each of these entries to see which proportion of the records has a time difference of at least -say- or greater than 60 minutes.

Apart from the example shown above we must consider the opposite effect. Consider this entry :

10/09/14,08:00,Medication1
...
...
...
10/09/14,21:05,Symptom2

In other words : Is it possible that a Medication taken in the morning to generate a Symptom 12 hours later?


2) The algorithm is not able to account for the compounding effect of a Medication. For example, the patient might have low levels of Taurine and for this level to be replenished, an x amount of days of Taurine supplementation is needed. The algorithm cannot account for this possibility.


 3) The patient should also input entries of "No Symptoms". It is not clear however when this should be done (e.g at the end of each day? assess every 6 hours and add 2 entries accordingly?)


However, this does not mean that a Sequence Mining algorithm should not be used under these circumstances. This technique can generate several potentially interesting hypotheses which Doctors and/or Researchers may wish to pursue further.
 

Becoming a Data Scientist : A RoadMap

I receive a lot of questions regarding which books one should read to become a Data Miner / Data Scientist. Here is a suggested reading list and also a proposed RoadMap (apart from the requirement of having an appropriate University degree) in becoming a Data Scientist. 

Before going further, it appears that a Data Scientist should possess an awful lot of skills : Statistics, Programming, Databases, Presentation Skills, Knowledge of Data Cleaning and Transformations.
 

The skills that ideally you should acquire are as follows :

1) Sound Statistical Understanding and Data Pre-Processing
2) Know the Pitfalls : You must be aware of the Biases that could affect you as an analyst and  also the common mistakes made during Statistical Analysis
3) Understand how several Machine Learning / Statistical Techniques work.
4) Time Series Forecasting
5) Computer Programming (R, Java, Python, Scala)
6) Databases (SQL and NoSQL Databases)
7) Web Scraping (Apache Nutch, Scrapy, JSoup)
8) Text Data

Statistical Understanding :  A good Introductory Book is Fundamental Statistics for the Behavioral Sciences by Howell. Also IBM SPSS for Introductory Statistics - Use and Interpretation and IBM SPSS For Intermediate Statistics by Morgan et al. Although all of the books (especially the two latter) are heavy on  IBM SPSS Software they are able to provide a good introduction to key statistical concepts while the  books by Morgan et al give a methodology to use with a practical example of analyzing the High-Scool and Beyond Dataset.

Data Pre-Processing : I must re-iterate the importance of thoroughly checking and identifying problems within your Data. Data Pre-processing guards against the possibility of feeding erroneous data to a Machine Learning / Statistical Algorithm but also transforms data in such a way so that an algorithm can extract/identify patterns more easily. Suggested Books :

•  Data Preparation for Data Mining by Dorian Pyle

• Mining Imperfect Data: Dealing with Contamination and Incomplete Records by Pearson

• Exploratory Data Mining and Data Cleaning by Johnson and Dasu

Know the Pitfalls : There are many cases of Statistical Misuse and biases that may affect your work even if -at times- you do not know it consciously. This has happened to me in various occasions. Actually, this blog contains a couple of examples of Statistical Misuse even though i tried (and keep trying) to highlight limitations due to the nature of Data as much as i can. Big Data is another technology where caution is warranted. For example, see : Statistical Truisms in the Age of Big Data and The Hidden biases of Big Data.

Some more examples :

-Quora Question : What are common fallacies or mistakes made by beginners in Statistics / Machine Learning / Data Analysis

-Identifying and Overcoming Common Data Mining Mistakes by SAS Institute

The following Book is suggested :

• Common Errors in Statistics (and how to avoid them) by P. Good and J. Harding

In case you are into Financial Forecasting i strongly suggest reading Evidence-Based Technical Analysis by David Aronson which is heavy on how Data Mining Bias (and several other cognitive biases) may affect your Analysis . 

Understand how several Machine Learning / Statistical Algorithms work : You must be able to understand the pros and cons of each algorithm. Does the algorithm that you are about to try handle noise well? How Does it scale? What kind of optimizations can be performed? Which are the necessary Data transformations? Here is an example for fine-tuning Regression SVMs:

Practical Selection of SVM Parameters and Noise Estimation for SVM Regression 

Another book which deserves attention is Applied Predictive Modelling by Khun, Johnson which also gives numerous examples on using the caret R Package which -among other things- has extended Parameter Optimization capabilities.


When it comes to getting to know Machine Learning/ Statistical Algorithms I'd suggest the following books  :

• Data Mining : Practical Machine Learning Tools and Techniques by Witten and Frank

• The Elements of Statistical Learning by Friedman, Hasting, Tibishirani 

Time Series Forecasting : In many situations you might have to identify and predict trends from Time Series Data. A very good Introductory Book is Forecasting : Principles and Practice by Hyndman and Athanasopoulos which contains sections on Time Series Forecasting. Time Series Analysis and its Applications with R Examples by Shumway and Stoffer is another book with Practical Examples and R Code as the title suggests.

In case you are interested more about Time Series Forecasting i would also suggest ForeCA (Forecastable Component Analysis) R package written by Georg Goerg -working at Google at the moment of writing- which tells you how forecastable a Time Series is (Ω = 0:white noise, therefore not forecastable, Ω=100: Sinusoid, perfectly forecastable).

Computer Programming Knowledge: This is another essential skill. It allows you to use several Data Science Tools/APIs that require -mainly- Java and Python skills. Scala appears to be also becoming an important Programming Language for Data Science. R Knowledge is considered a "must". Having prior knowledge of Programming gives you the edge if you wish to learn n new Programming Language. You should also constantly be looking for Trends on programming language requirements (see Finding the right Skillset for Big Data Jobs). It appears that -currently- Java is the most sought Computer Language, followed by Python and SQL. It is also useful looking at Google Trends but interestingly "Python" is not available as a Programming Language Topic at the moment of writing. 

Database Knowledge : In my experience this is a very important skill to have. More often than not, Database Administrators (or other IT Engineers) that are supposed to extract Data for you are just too busy to do that. That means that you must have the knowledge to connect to a Database, Optimize a Query and perform several Queries/Transformations to get the Data that you want on a format that you want.

Web Scraping: It is a useful skill to have. There are tons of useful Data which you can access if you know how to write code to access and extract information from the Web. You should get to know  HTML Elements and XPath.  Some examples of Software that can be used for this purpose : 

-Scrapy
-Apache Nutch
-JSoup

Text Data: Text Data contain valuable information : Consumer Opinions, Sentiment, Intentions to name just a few. Information Extraction and Text Analytics are important Technologies that a Data Scientist should ideally know.

Information Extraction :

-GATE
-UIMA

Text Analytics

-The "tm" R Package
-LingPipe
-NLTK

The following Books are suggested :

• Introduction to Information Retrieval by Manning, Raghavan and Schütze

• Handbook of Natural Language Processing by Indurkhya, Damerau (Editors)

• The Text Mining HandBook - Advanced Approaches in Analyzing Unstructured Data by Feldman and Sanger

Finally here are some Books that should not be missed by any Data Scientist :

• Data Mining and Statistics for Decision Making by Stéphane Tufféry (A personal favorite)
• Introduction to Data Mining by Tan, Steinbach, Kumar 
• Applied Predictive Modelling by Khun, Johnson
• Data Mining with R - Learning with Case Studies by Torgo
• Principles of Data Mining by Bramer

Venture Capitals in an Age of Algorithms (Revisited)

Some time ago i wanted to explore the idea of analyzing several kinds and sources of Information (e.g TechCrunch, TheNextWeb,  News sites and Twitter) to identify promising Investment opportunities in Technology and more specifically Startups. 

Here is a snapshot of a Webpage from TechCrunch :

In many posts in this Blog it was discussed how our Reactions for almost any kind of information are recorded. This was not possible when everyone was reading newspapers in its paper form whereas  now any kind of Text is associated with a number of Views, Re-Tweets, Number of Comments and FaceBook "Likes".

The second important information that is being generated is our Emotions for any Topic as these are expressed within Comments, Twitter and FaceBook posts. The intensity of our emotions is also captured and this information is very important since  whatever we associate with intense emotions really stays within our psyche, fuels our interest and (usually) drives our purchase decisions.

We may then continue with some Exploratory work as follows : We can collect Posts from various Tech sources and their associated Reactions, annotate the text with Sentiment, Events and Topics and analyze this information to understand which Topics and/or Events appear to have an affinity for a high number of Reactions or High Sentiment intensity for Startups or Tech Topics .

As an example, 10K posts from various Tech sources were collected and each one of the posts was marked as generating either HIGH or LOW interest based on the amount of Reactions (Re-Tweets, FB Likes, Comments)  that each post generated. Special filtering is applied for the frequencies of the words that appear in each post :

Then this information is fed to KNIME for further analysis. The implementation which  is shown here is rather naive and simplistic for many reasons :  Only keywords are used as input -as opposed to Topics, Events- and many other parameters that are involved and which will be discussed later but for our example we will keep things simple.

The workflow uses 3 algorithms namely PART (so that some rules are generated), SMO and Random Forests :

 
This -again- is a very naive approach which gave a result of 61.9% (F-Measure) in identifying keywords that commonly appear with posts that generate Interest vs posts that do not. We keep in mind that this knowledge alone is not enough with which a decision can be made but we decide to explore things a little further.

We may find that some words that we expected do appear in posts of High interest (such as Google, Apple, Pinterest). There could be however some words that deserve more of our attention such as Education and Schools which during the analysis appeared to exist more frequently in High Interest posts.

So how can this information be used for a potential investment on a Startup and is there really a way to model new ideas and predict their performance?  Again, it is not suggested here that if you come across a startup aimed in Education you should immediately put your money in but this observation could be one parameter to consider. There are so many other considerations such as whether the idea is novel or not, how many competitors exist, who are the people behind the Startup, whether its founders have created a successful Startup in the past, which people have already invested in the particular Startup, what is the "buzz" that this Startup has generated so far and so on.

Whenever we read about a new startup there are some immediate thoughts going through our minds : Does this sound like a good idea? Is it applicable to me and would it make my life easier? Is this idea truly disruptive or not? What does our "gut feeling" tells us?  

We should always keep in mind that there are limitations to what Predictive Analytics can do but perhaps we can extract some hints that we may then use to make better decisions.

It was also interesting to read this post (hence the use of word "Revisited" in this Post's Title) on Gigaom regarding the same Subject. This is a fascinating area that i started looking at and there will be similar posts in the future on this Subject.

Finding the Right Skillset for Big Data Jobs

Perhaps one of the key skills of a Data Scientist is the ability to be able to collect and access data that are not readily available. 

I was wondering about the trends in Job Postings and more specifically which skills and qualities employers (or agencies) search for on a candidate for a job in "Big Data" so i decided to use R to answer this question.

Of course, one must first find Data (in this case Job Postings) so that they may be analyzed. This is possible by using the library scrapeR of R to scrape content from websites that contain Job Postings. Once this is done, tm package can be used to analyze thousands of Job Advertisements so we may extract useful knowledge.

The analysis which you will see below is based at around one thousand Job Postings that contain the phrase "Big Data". Better pre-processing could help in getting better term co-occurrences but here i aimed in presenting the application.  Once the data are collected we can start by looking at the Frequency distribution of the words found (after removal of stop words) :

Note that the word 'big' is removed from the bar chart. Notice also how the term "experience" (which also includes occurrences of term "experienced") was frequently found in Big Data Job Postings.

Interestingly, the term "skill" (which also counts the term "skilled") is found way below in the frequency diagram.

Next we can use Text Analytics to find which words co-occur with topics of interest. We start by looking at which terms co-occur in Job Postings where Hadoop is mentioned :

Suppose that one wishes to better understand which skills are discussed along with the Java programming language :

 When it comes to skills, it appears that communication skills are those which are important  (as expected):

In the same manner we can  :

-Find the frequencies of skills of interest (e.g Java, Python, Ruby, NoSQL, Oracle DB) and generate trend charts for each of them.

-Run term co-occurrence analysis on the skills which are "good to have" or "preferred".

-Capture early trends on emerging skills (in the "Big Data" case, this could be Pig)

The idea of analyzing Job postings and CVs using Information Extraction (and then using Predictive Analytics once this information becomes structured) is quite interesting. The ability to extract inferred knowledge is also quite challenging : For example could we infer from the text found in CVs  :

-The total number of years of experience in Project Management of an Applicant in case that this has not explicitly been stated in his/her CV?

-Whether an Applicant shows a coherent Career growth through the years ?

-The years needed for an Applicant to move to a Managerial Position?

Personal Data Mining - (Part 2)

On the previous post i described the way that i used to capture a 1-year worth of personal data using my Smartphone with the purpose of identifying trends with my immune system which at times gave me perennial conjunctivitis and also swollen lymph nodes. Now it was time to analyze all of this data in hope that some useful knowledge could be found.

 I had to make a decision for which tools to use. I used WEKA and also decided to give KNIME a try so here is an example of a KNIME workflow :

I first use the File Reader to read in my 1-year worth of life data, then an R Script which is used for several data transformations. I then send the streams of Data first to an R Script which runs the FSelector package with which several Feature Selection algorithms (about 10 of them) are applied to get an understanding of what are the important Features for the problem at hand.

Then another stream sends the Data to an R node which creates dummy Variables and then sends the transformed Features to a Linear Correlation node for further inspection.

A third stream (not shown) sends the data to 3 Machine-Learning algorithms (namely an SVM, Decision Tree and Random Forest) and the Scorer shows how each algorithm performed.

I first executed the FSelector node using 10-fold Cross-Validation because i wanted to get a first feel of the features that are important in identifying some patterns about my perennial conjunctivitis.  7 out of 10 of  FSelector algorithms agreed that :

1) Vitamin D3
2) Garlic
3) Yoghurt

..Appear to have the most predictive power. The problem is that at this point we do not know if any of the features actually help or aggravate my condition. However, the output of FSelector gives an idea on which features should be looked at more closely.

Then the second stream was run, namely the one which sends the data to 3 machine learning algorithms so that i could get a first feel of how the algorithms perform. All three algorithms gave an F-Score of around  59 - 62%.

By looking at the results some patterns appeared to arise (Note the word "appeared")

1) A rather large daily dose  (>5200 IU) of Vitamin D3 appears to be associated with smaller incidences of conjunctivitis
2) Garlic consumption appears to increase my conjunctivitis incidences.
3) Yoghurt consumption appears to increase my conjunctivitis incidences.


For Pattern (1) we need to be aware that Vitamin D3 dosage has a compounding effect so it is rather naive to think that boolean logic applies (see previous post for more).

Next i had to look at patterns (2) and (3). One of the things that i realized when searching the web for the effect of various nutrients in functions of the human body is the fact that you can find for any several entries that some times contradict  each other. My very brief web search has found Garlic and Yoghurt to be "immune boosters". Of course caution should be exercised in drawing any conclusions because of the way the data have been collected and also the problematic origin of the analysis. Moreover, i am not a doctor and i cannot possibly know whether Garlic or Yoghurt can aggravate an immune response in such a way. 

 I began taking Vitamin D3 and eliminating Garlic and Yoghurt from my diet. The result was that over a period of one month i stopped getting bouts of conjunctivitis and incidences of swollen lymph nodes. So has Vitamin D3 acted as an "immune response regulator" and Garlic - Yoghurt as "immune boosters"?

Although my bouts of conjunctivitis have ceased, I am not in any position to make any claims because there are a lot of uncontrolled variables :

- It could be a placebo effect.
-There may be unknown hidden variables that are important
- (My) Genetics
- Environment
- Variations in Dosage and Nutrient Content
- Interactions between nutrients

and lots of others that could not possibly be accounted for under these circumstances.

What i can say (and this is the reason for writing this post) is that analytics may help us to identify several patterns that may then be used to guide a sound knowledge discovery process. If people had the ability to collect data on a daily basis (see Quantified Self) and then analyze them on a massive scale, several unknown patterns that call for closer investigation could emerge.

Personal Data Mining

I believe that i have an overactive immune system. I get recurring bouts of Perennial Conjunctivitis and i also experience at times pain on my neck lymph nodes and my right maxillary sinus. I always believed that all of these symptoms were somehow related. Because of my conjunctivitis i was not able to wear contact lenses. My ophthalmologist confirmed that my eye problems were "allergy-related".

Since September 22nd, 2011 i began a personal experiment. I decided to keep a detailed record of various elements of my everyday life : Whether i had a good night sleep and spent time outdoors, what i ate and how much stress i felt. 

I carry almost always my smartphone with me. So I used a Text-Editing application to record every day as much detailed information as i could. Here is an example of two consecutive dates as these appear in my daily log :

10/11/12, slept/bad, vitamin_c/0, coffee/1, self/ok, stress/low, sausages,cholesterol_food, sugar/5, pasta, tomatoes, mushrooms,  next_sleep/ok 

10/12/12, slept/ok, vitamin_c/500, coffee/2, self/ok, stress/high, bread, honey, milk, sugar/10, meat, garlic, yoghurt, conjunctivitis,  icecream,  next_sleep/ok  

So on the first example date, i did not sleep well the previous night. I had one coffee and roughly 5 teaspoons of sugar the whole day. I was feeling ok with myself, i had sausages and eggs (tagged as cholesterol_food) for breakfast and pasta with mushroom (red) sauce for lunch but no dinner. I managed to sleep well at night. I did not have any signs of an overactive immune system. However, the next day i had conjunctivitis.

I then had to somehow transform the entries to a suitable format - a .csv file- which could then be used by Data Mining Software (such as R and WEKA) for analysis. To do that, a simple Java program was used to transform all log entries to a .csv format using the following rules :

1) Each line represents a day.

2) Each entry is separated by comma (",")

3) If an entry does not contain a forward slash character ("/") then it is treated as a Boolean feature. 

4) If an entry contains a forward slash then it is treated as a Numerical or Categorical feature.

So our two example dates, are transformed like this ( Not all features are shown) :

R was used to perform several pre-processing steps such as coding a function called addfeature which i use to derive new variables from old ones :

data.df<-addfeature("fiber",c("beans","stringbeans","oats","okra","lentils"),data.df)

data.df<-addfeature("cholesterol_food",c("eggs","mayo","octopus","squid"),data.df)

data.df<-addfeature("nuts",c("hazelnuts","walnuts","peanuts","cashews","almonds"),data.df)

data.df<-addfeature("immunity",c("itchyeyes","lymphpain","sinuspain","conjunctivitis"),data.df)   
  

So if on any day i had eggs, mayo(nnaise), octopus, squid or any combination of these foods an entry of cholesterol_food will be used to replace these entries.

 Having the log transformed to the format shown above, i was ready to analyze a 1-year worth of data (in this case IMMUNITY is the target), extract patterns and several hypotheses - for example that "there appears to be a connection between high stress and over-activity of my immune system."

However we must be aware of the dangers that might lead us to incorrect findings. For instance we must take into account the fact that conjunctivitis usually lasts more than one day and also that some features -like Vitamin-C intake- are special in the sense that the representation shown above does not take into account the compounding effect of Vitamin intake. In other words, i might have to take for  n number of days,  an x amount of Vitamin C consecutively to see any effect. Furthermore, this analysis does not take into account the sequence of events. Should we remove foods/ingredients that  normally co-occur or not? How would that affect results? The list of questions and considerations goes on (and then when we finally have some results from the analysis, the first thing to do is to question them).

Using Data Science i was able to almost stop getting sinus and lymph pain and wear again my contact lenses ( i still get symptoms but very-very rarely). Two foods  appeared to be moderately correlated with  signs of an overactive immune system - with one of them being garlic. One would probably argue that i could find that using a simple food diary -  i  doubt about it since things were not so evident. Once this information was found these two foods were eliminated from my diet to see the outcome.

Analysis has also identified a particular Vitamin that was able -in my opinion- to regulate my immune system response so that i could have no food restrictions. Several other patterns (or hypotheses) emerged that could be used for further evaluation by specialized personnel. Whatever i tried, i tried it under the close supervision and consent from Doctor specialists.

It's a logical next step to imagine the potential knowledge and hypotheses extracted by implementing the same experiment on a wider scale (for example by using Kaggle) .

On the next post : More thoughts, results and warnings    

Do Social Media Monitoring tools provide True Intelligence?

Having recently read a report from WebLiquid one of the interesting facts to consider is that around 70% of Marketers replied finding the insights gleaned from Social Media Monitoring tools "Somewhat Valuable".  Slightly more than 20% of them found these insights "Extremely valuable". The report also shows that most Marketers plan to invest more in SMM tools with few of them retreating from any further investment.

This is Big News. 70% of Marketers finding insights gleaned from SMM tools "Somewhat" valuable is not a good thing and perhaps there are reasons for this. It would be very interesting to know what do Marketers consider Insights, how they prioritize those Insights and how easily they can act once they have those insights . The problem can be summarized in one sentence:


- Marketers do not want (just) Reports.

There is a lot of useful information provided by many Social Media Monitoring tools  : The number of mentions of a Brand (or Product or Service) per channel, which users talk frequently about your Brand  (and which of them are considered influential). Sentiment Analysis provides Marketers with the perception of a Brand but also the perception about competitive Brands leading to what is known as Competitive Intelligence. Perhaps Social Media Monitoring platforms have many types of metrics still to offer : For example, a potentially useful metric could be the ability to identify Consumer  Intentions ("I will definitely buy...") and how these intentions differentiate - such as "I would buy 'ABC' if it was cheaper" or "I would buy 'ABC' if i hadn't  purchased 'XYZ' already".

Notice that SMM tools provide metrics  : Number of mentions per channel, Top influential users, percentage of positive / negative / neutral sentiment and sentiment intensity, how mentions of a new product disperse through different social media channels, etc. 

But what is considered Intelligence in Social Media? Would someone identify as intelligence the fact that during the past 2 months there was an increase in specific Brand Mentions on Twitter but not on YouTube? Or is it Intelligence when we notice that there has been a decline in positive sentiment about a product?  All of this information is Reporting and Feedback. It is not meant that this is not useful information :  It is important to know what is happening and why.

So what True Intelligence is all about?

True Intelligence is about knowing how to successfully Promote and Market a Brand, Product or Service. To do that a Marketer wants to know the Best Practices : With Social Media Reports, Marketers know what is happening (a decline in positive mentions on our new smartphone) and why this is happening (a potential hardware problem). Social Media Analytics can identify the right strategies to make things happen. True Social Media Intelligence is about knowing which parameters (channels, number of mentions) are important in achieving a result. Is it important to have a product associated with intense (positive) sentiment? Or could it be more important to have a Product being highly associated with Rumors?

There is still a long way to go in terms of Insights from Social Media Monitoring tools. There are many processes and parameters that will eventually used for deriving more Insights and better Strategies. The answer to true Social Media Intelligence is the use of Predictive Analytics (Data and Text Mining)  applied to Social Data : One area that is currently untouched by most Social Media Monitoring tools.

Forex Trading with R : Part 2

In the previous post the first steps were given for building the basis for trading forex. Now it is time to build the actual classifiers that  can give us future buy / hold / sell signals.

Assuming that everything is in working order and the instructions given in  the previous post were followed we can start building these classifiers.

First let's train a Neural Network. The following command trains a Neural Network and then applies the trained model on our test data and outputs the predictions for buy/sell/hold signals :

set.seed(134)
nn <- nnet(class~.,traindata, size = 3, rang = 0.1,decay = 0.001, maxit = 3000,trace="F")
table(actual=testdata$class,predicted=predict(nn,newdata=testdata,type="class"))

Note that a seed number was used.  You should either try different seed numbers (so that network weights are re-initialized) or omit the set.seed() directive. You should also experiment with other Neural Net parameters such as the number of iterations (maxit), the learning decay (decay), etc.


The confusion matrix shows us the necessary information for calculating TP, FP, TN,FN rates for each class (ie for each signal type).


Similarly we can train and test a Random Forest :


 rf.model<-randomForest(class~.,data=traindata,nodesize=40,importance=FALSE,mtry=3,ntree=100)
table(actual=testdata$class,predicted=predict(rf.model,newdata=testdata,type="class"))

Now let's train an SVM for our data. We can issue the following command : 

###train SVM

sv<-svm(class~.,traindata,gamma=0.01,cost=5,kernel="radial")

To see how the classifier did on the test set, we enter :

table(actual=testdata$class,predicted=predict(sv,newdata=testdata,type="class"))

Next we can try to optimize parameters of the SVM classifier as follows :


#find optimal values of Gamma and Cost for an RBF- SVM classifier
tuned <- tune(svm, class~., data = traindata,ranges = list(gamma = c(0.0001,0.001,0.05,0.1,0.2,0.3), cost = c(1,5,10,20,50,100,120,130)),tunecontrol = tune.control(sampling = "cross"),cross=10)


tuned


The first command uses 10-fold cross validation to identify the best gamma and cost parameters among some predetermined values. We then issue the command tuned to see which combination of parameters  gives us the lowest classification error. Knowing these parameters we can then use these parameters to train an SVM classifier and see how this model performs (as was shown previously).

Be aware of the following key points :

1. Three sets of data should be used : Training, Test and Validation. The Validation set should not be a part of the optimization (=finding the best algorithm parameters) process.
2. Make sure that you create classifiers for several time periods. Test the performance of any classifier according to the percentage of available data you use for training / testing / validation and the number of periods you use for the sliding window.
3. Make also sure that once you have chosen your model, you use a correct way to test your system by simulating buy / hold / sell signals and taking under consideration all associated trading costs.

Forex Trading with R : Part 1

I recently started learning R - probably something i should have done a long time ago - and since learning by doing is the best way to learn something i decided to  use  R to generate buy/see/hold signals for the EUR/USD Pair. For those that wish to use R for making Trading decisions, this series of posts is a short introduction with which one can pursue the subject further. By no means it is implied that this post's methodology is the one that  you should use  to trade :   different response variables, signal thresholds, technical indicators and classifiers than the ones presented here should be tried. Then, elaborate testing methods should be put to use to assess the  performance of  each classifier and the worth of your trading strategy. 

First the resources needed to be downloaded and installed :

1)  Packages : quantmod, nnet, e1071, tseries, randomforest

2) A file that contains OHLC Data. For this example EUR/USD Data are used. You can download an example file here. I downloaded the EUR/USD historical data here. 

I would also highly recommend getting Data Mining with R : A concise book that goes through an introduction of using R and then presents various case studies one of which is about Using R to predict variations of the S&P Index. The author also provides package "DMwR" that includes all necessary functionality for generating signals, extracting precision / recall metrics of generated models, performing Monte Carlo Estimates and evaluating trading strategies.

After downloading the file from step (2), place the file to a directory of your choice. Now copy and paste the following commands in R :

library(e1071)

library(nnet)
library(randomforest)

library(quantmod)

library(tseries)

Next we import the csv file from the directory that was originally saved  (change "YOUR_PATH" accordingly with the directory path you saved the csv file) :

#get data OHLC from csv file

raw<- read.delim2("/YOUR_PATH/EURUSD.csv",header=TRUE,sep=",")

Now, paste the following to R (change again YOUR_PATH accordingly) :

#convert date

stripday<-strptime(raw$DATE,format="%Y%m%d")

fxdata<-data.frame(stripday,raw)

fxdata$TIME<-NULL

fxdata$TICKER<-NULL

fxdata$DATE<-NULL

colnames(fxdata)<-c("Date","Open","Low","High","Close")

#write data to .csv

write.table(fxdata,"/YOUR_PATH/eurusd.csv",quote=FALSE,sep=",",row.names=FALSE)

##transform to an xts object

EURUSD<-as.xts(read.zoo("YOUR_PATH/eur-usd.csv",sep=",",format="%Y-%m-%d",header=T))

Now we define some technical indicators, the model to work on and a function that generates our trading signals  :

#setup Technical Indicators

myATR <- function(x) ATR(HLC(x))[,'atr']

mySMI <- function(x) SMI(HLC(x))[,'SMI']

myADX <- function(x) ADX(HLC(x))[,'ADX']

myAroon <- function(x) aroon(x[,c('High','Low')])$oscillator

myBB <- function(x) BBands(HLC(x))[,'pctB']

myChaikinVol<-function(x)Delt(chaikinVolatility(x[,c("High","Low")]))[,1]

myCLV <- function(x) EMA(CLV(HLC(x)))[,1]

myMACD <- function(x) MACD(Cl(x))[,2]

mySAR <- function(x) SAR(x[,c('High','Close')]) [,1]

myVolat <- function(x) volatility(OHLC(x),calc="garman")[,1]

myEMA10 <- function(x) EMA(Cl(x),n=10)[,1]

myEMA20 <- function(x) EMA(Cl(x),n=20)[,1]

myEMA30 <- function(x) EMA(Cl(x),n=30)[,1]

myEMA50 <- function(x) EMA(Cl(x),n=50)[,1]

myEMA60 <- function(x) EMA(Cl(x),n=60)[,1]

data.model <- specifyModel(Delt(Cl(EURUSD)) ~

myATR(EURUSD) + mySMI(EURUSD) + myADX(EURUSD) + myAroon(EURUSD) +

myBB(EURUSD) + myChaikinVol(EURUSD) + myCLV(EURUSD) +myEMA10(EURUSD) +myEMA20(EURUSD) +myEMA30(EURUSD) +myEMA50(EURUSD) + myEMA60(EURUSD) +

CMO(Cl(EURUSD)) + EMA(Delt(Cl(EURUSD))) +

myVolat(EURUSD) + myMACD(EURUSD) + RSI(Cl(EURUSD)) +

mySAR(EURUSD) + runMean(Cl(EURUSD)) + runSD(Cl(EURUSD)))

Tdata.train <- as.data.frame(modelData(data.model,

data.window=c('2008-01-01','2010-01-01')))

Tdata.eval <- na.omit(as.data.frame(modelData(data.model,

data.window=c('2010-01-02','2010-11-01'))))

# a very simple signal function

signals<-function(x) {

if(x>=-0.005&&x<=0.005) {result<-"hold"} else

if(x>0.005) {result<-"buy"} else

if(x<-0.005) {result<-"sell"}

result

}

#create class vector that holds TRAINING buy,sell,hold signals

class<-sapply(Tdata.train$Delt.Cl.EURUSD,signals)

#paste both to a new list that holds everything

traindata<-cbind(Tdata.train,class)

#remove Delt.Cl.EURUSD - not needed anymore.
traindata$Delt.Cl.EURUSD<-NULL

#create class vector that  holds TESTING buy,sell,hold signals

class<-sapply(Tdata.eval$Delt.Cl.EURUSD,signals)


#paste to a new list that holds everything
testdata<-cbind(Tdata.eval,class)
testdata$Delt.Cl.EURUSD<-NULL

#get a summary of our traindata
summary(traindata)

The last command prints out some summary statistics of the traindata sample. Notice the 'class' attribute and the distribution of buy, hold and sell signals .

Now we are ready to apply some modeling techniques using traindata and testdata as the datasets to work with. Although i would suggest using also a third sample for validation (remember the "elaborate testing" discussed at the beginning), for this example we will keep things  simple.  More on the next post

Interview with Kaggle CEO Anthony GoldBloom

For those that haven't heard of Kaggle before, Kaggle is a team of people that provide the functionality and support to host Data Mining contests. Here is how it works : Suppose that you are working for a Telco and wish to implement a new Churn prediction model. Rather than running this project in-house, you submit your data to Kaggle. What happens next is that -hopefully- many statisticians globally will each analyze your dataset, produce a model and then submit their prediction model(s) to Kaggle. The best model (and hence its creator) gets the prize which is given by the Telco company. Here is the interview with Kaggle CEO, Anthony GoldBloom :

- What is Kaggle and what new ideas brings to the predictive analytics arena?

Kaggle aims to help companies and researchers make predictions more precise by providing a platform for data prediction competitions. Competitions turn out to be a great way to get the most out of a dataset. This is because there are infinitely many approaches to any data modeling problem. By opening up a data prediction problem to a wide audience, a competition makes it possible to get to the frontier of what is possible given a dataset's inherent noise and richness.

- Can you tell us more about "real-time science" and how it could help Research globally?

Data modeling competitions can facilitate real-time science. Consider the recent announcement about the discovery of genetic markers that correlate with extreme longevity. Work on the study began in 1995, with results published in 2010. Had the study been run as a data modeling competition, the results would have been generated in real time and insights available much sooner (and with a higher level of precision).

Data modeling competitions also benchmark, in real time, new techniques against old. A technique that performs well in competitions can prove its mettle long before any paper can be published, helping the science to progress more quickly.

Competitions also help to avoid situations where valuable techniques are overlooked by the scientific establishment. This aspect of the case for competitions is neatly illustrated by Ruslan Salakhutdinov, now a postdoctoral fellow at the Massachusetts Institute of Technology, who had a new algorithm rejected by the NIPS conference. According to Ruslan, the reviewer ‘basically said “it’s junk and I am very confident it’s junk”’. It later turned out that his algorithm was good enough to make him an early leader in the Netflix Prize (he called his Netflix Prize team NIPS_reject).

- How can companies benefit through Kaggle?

Companies can use Kaggle to gain an advantage over their competitors. Consider a bank that wants to improve the algorithms that vet loan applicants. If a bank can develop a more effective algorithm they will have fewer defaults and can charge lower interest rates than their competitors. Kaggle has proven to be an effective way to improve existing models very quickly.

Competitions are also really useful to companies that want to develop new products and capabilities. Consider a hedge fund that wants to be able to generate long-range weather forecasts in key agriculture regions. They can attempt to hire a weather forecasting expert or they can use Kaggle to throw the problem open to a wide audience. Using Kaggle they can be sure they'll get great results very quickly.

- How is the best model selected?

The competition host will typically split their dataset into two parts - a training dataset and a test dataset. The training dataset includes all explanatory variables as well as the dependent variable (or the answer). The test dataset also includes all the explanatory variables but the dependent variable (or answer) is withheld.

Participants train their models on the training dataset. They then apply their models to generate predictions on the test dataset. Those predictions are then scored on-the-fly against the actual answers (using one of several evaluation methods). Once the competition deadline passes, the team that generates the most accurate predictions gives the winning methodology to the competition host in exchange for the prize money.

Predictive Analytics and Politics - Part 1

One of the most interesting applications of Data/Text Mining and Information Extraction is Politics. I started collecting information from various blogs, websites and forums and applying Information Extraction and Data/Text Mining techniques to extract potentially useful knowledge in this area. By combining different pieces of information one could come up with trends that may tell us what lies ahead of us.

The latest developments in Greece are more or less known to most of people that read International News. The situation is difficult and the voice of citizens in various blogs and forums could give us the sentiment of Greek Web Users. For example :

- Which are the most frequently occurring words?

- Which are the most frequently occurring thoughts?

- What are the things that have to be changed by Greek politicians?

To answer these questions i have started collecting information found on the top 120 Greek blogs, the OpenGov website (a state-run website where Greek citizens express their opinions) and a couple more Greek sites of economic content. For blogs and forums a Java program scans every 20 minutes for new information :

This information is then sent to an annotation engine which analyzes the textual content. Once the text is analyzed we can -for example- produce a keyword vector that we can later use to understand what citizens are saying on the Web. We can then find out answers to many interesting questions such as :

- With which words is Mr George Papandreou (PM of Greece) associated with?

-When there are some very negative words (such as swearing) what other words are found in the same text?

- What does keyword trending tell us? (For example, we identify an increasingly number of swear words in citizen posts)

First let's see some examples regarding the OpenGov website where thousands of citizens have expressed their opinions on the tax policy of the Greek state. The following chart shows us a number of pairwise correlations between written words in these comments :

Under the red rectangle appear two words (dikigoros,iatros) which in Greek mean "Lawyer" and "Medical Doctor" respectively. This essentially tells us that these two professions are used together frequently in citizen discussions. By looking closely at these messages one can reveal that professionals in these two sectors are said to avoid taxes by not issuing receipts.

Next we could use association rule learning to look for some more -potentially interesting - rules :

The highlighted rule although one of low support it could prove interesting : A subset of citizens are requesting that freelancers and the self-employed should be more closely monitored for tax fraud.

Apart from rule learning, it is interesting to identify the proportion of the total dataset for which each rule holds. That also gives us a sense of order with which different ideas and thoughts exist on the mind of citizens.

In the next post : What is the Voice of the Citizen tells us in Blogs and forums?

Building a Knowledge Hub

The web is a huge source of information. It stores facts, thoughts, feelings and intentions of people. It also records what people like and what they don't in an indirect way - something that we are going to be looking at shortly . Some of the examples on harnessing this information were shown previously in this blog, such as :

• Extraction of user opinions, beliefs and values from Twitter

• Prediction of popular stories on Digg

• Prediction of popular Tweets

Consider the following snapshot from a BBC webpage :

The table above shows a representation of the most popular business stories on BBC on the 22nd December 2009. Even though we do not have specific metrics, we intuitively understand that the order with which the stories are listed also tell us the popularity of each post. Notice that the first post on the most read stories talks about the British economy while the last one is a title regarding football.

This is knowledge that we can harness. No doubt it is a very specific kind of knowledge because it tells us only what -mostly- British readers of BBC have found interesting. In other words this is knowledge for a specific population : Most likely in another country -say France- the title about UK being still in recession would not be so interesting but a title about France being in the same situation would. Subject, Time and Location are all important parameters that need to be captured and taken into account.

Let's consider the idea of creating a Knowledge Hub : This could be done by collecting massive amounts of information from Social Media, blogs, comments from forums and news titles (and their popularity). Techniques such as Information Extraction with concept annotation, Data and Text Mining could be used to extract knowledge by combining incidents, opinions, intentions and emotions found from different sources.

I have been monitoring and collecting for the past 3 months news and forum posts generated from/for a specific country. The information collected is then annotated in such a way to extract concepts. This text annotation is matched with keywords of concepts, incidents and intentions. Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness.

It is a very interesting fact that our behavior is recorded -up to a point- by the web. Again, the key is the way that we are able to organize this information into logical chunks and then use this representation to find possible insights.

2009 has been a year of big changes. Best wishes for a Happy and Prosperous New Year for everyone.

Predicting the next Viral Tweet

It is time to use Twitter data for another reason : Can Predictive Analytics be used to identify which tweets have an increased probability to become viral?

First we have to identify the problem and see what information we should consider. Every Tweet has an author, a content and is posted on a specific day and time. More specifically, for every tweet we can collect usage data such as

• Day of Post
• Time of post
• Elapsed minutes since tweet has been posted
• Author of tweet (Twitter username)
  
• Number of followers of the author

and also information such as :

• Subject of post
• Whether the tweet involves a question being asked
• Whether the tweet contains hashtags
• Whether the tweet contains a "Please Re-Tweet" directive (or variants)
  
• Whether a user is mentioned
• The text of the tweet itself.
  

Our goal then is to combine the information mentioned above and come up with a predictive model that when given an author, day, time of post and text of the tweet it will be able to tell us whether this tweet has an increased probability to become viral.

For this Data & Text mining exercise (and keeping in mind that tweets have been sampled from one website and not Twitter itself) let's define what is a viral tweet : After collecting approx. 8000 tweets from dailyrt.com it was found that the median value of Re-tweets is 17. Here we make the assumption that if a tweet exceeds 30 Re-tweets it is considered viral (and actually this specific assumption makes the classification task much easier).

As discussed above, usage data do not tell us anything about the content of a tweet. Usage data tell us about the name of the author, his/her followers, when the tweet has been posted and how many minutes elapsed since its post. Can this information alone predict whether a tweet will become viral? A data mining model predicted (without using the elapsed time as input field) with an overall accuracy of 75.03% whether a tweet can be viral and -perhaps as expected- shown that the most important factor for making a viral tweet is its author. Running a process called Feature Selection tells us just that :



But what we have seen so far only tells us one -the Data Mining- side of the story. With Text Mining we can see the importance of words and authors. To do that, each author is appended at the end of each tweet (so essentially the author becomes a part of each tweet text). Here is what Feature Selection tells us :



A Tweet mentioning Michael Jackson has a great probability of becoming viral but perhaps it should be also posted by a popular author to make a greater impact. Pay attention also to the fact that @mashable and the @theonion are on top of our feature selection list shown above.

The difficult -but also interesting- task is to predict a viral tweet that has an impact not because of its author but because of its content and to do this the methodology of data collection and analysis differs significantly.

On the next post we will see a model predicting viral tweets in action : We will submit several tweets and their author and the model will tell us the probability that each submitted tweet has to become viral.