Personal Data Mining - (Part 2)

On the previous post i described the way that i used to capture a 1-year worth of personal data using my Smartphone with the purpose of identifying trends with my immune system which at times gave me perennial conjunctivitis and also swollen lymph nodes. Now it was time to analyze all of this data in hope that some useful knowledge could be found.

 I had to make a decision for which tools to use. I used WEKA and also decided to give KNIME a try so here is an example of a KNIME workflow :

I first use the File Reader to read in my 1-year worth of life data, then an R Script which is used for several data transformations. I then send the streams of Data first to an R Script which runs the FSelector package with which several Feature Selection algorithms (about 10 of them) are applied to get an understanding of what are the important Features for the problem at hand.

Then another stream sends the Data to an R node which creates dummy Variables and then sends the transformed Features to a Linear Correlation node for further inspection.

A third stream (not shown) sends the data to 3 Machine-Learning algorithms (namely an SVM, Decision Tree and Random Forest) and the Scorer shows how each algorithm performed.

I first executed the FSelector node using 10-fold Cross-Validation because i wanted to get a first feel of the features that are important in identifying some patterns about my perennial conjunctivitis.  7 out of 10 of  FSelector algorithms agreed that :

1) Vitamin D3
2) Garlic
3) Yoghurt

..Appear to have the most predictive power. The problem is that at this point we do not know if any of the features actually help or aggravate my condition. However, the output of FSelector gives an idea on which features should be looked at more closely.

Then the second stream was run, namely the one which sends the data to 3 machine learning algorithms so that i could get a first feel of how the algorithms perform. All three algorithms gave an F-Score of around  59 - 62%.

By looking at the results some patterns appeared to arise (Note the word "appeared")

1) A rather large daily dose  (>5200 IU) of Vitamin D3 appears to be associated with smaller incidences of conjunctivitis
2) Garlic consumption appears to increase my conjunctivitis incidences.
3) Yoghurt consumption appears to increase my conjunctivitis incidences.


For Pattern (1) we need to be aware that Vitamin D3 dosage has a compounding effect so it is rather naive to think that boolean logic applies (see previous post for more).

Next i had to look at patterns (2) and (3). One of the things that i realized when searching the web for the effect of various nutrients in functions of the human body is the fact that you can find for any several entries that some times contradict  each other. My very brief web search has found Garlic and Yoghurt to be "immune boosters". Of course caution should be exercised in drawing any conclusions because of the way the data have been collected and also the problematic origin of the analysis. Moreover, i am not a doctor and i cannot possibly know whether Garlic or Yoghurt can aggravate an immune response in such a way. 

 I began taking Vitamin D3 and eliminating Garlic and Yoghurt from my diet. The result was that over a period of one month i stopped getting bouts of conjunctivitis and incidences of swollen lymph nodes. So has Vitamin D3 acted as an "immune response regulator" and Garlic - Yoghurt as "immune boosters"?

Although my bouts of conjunctivitis have ceased, I am not in any position to make any claims because there are a lot of uncontrolled variables :

- It could be a placebo effect.
-There may be unknown hidden variables that are important
- (My) Genetics
- Environment
- Variations in Dosage and Nutrient Content
- Interactions between nutrients

and lots of others that could not possibly be accounted for under these circumstances.

What i can say (and this is the reason for writing this post) is that analytics may help us to identify several patterns that may then be used to guide a sound knowledge discovery process. If people had the ability to collect data on a daily basis (see Quantified Self) and then analyze them on a massive scale, several unknown patterns that call for closer investigation could emerge.

Personal Data Mining

I believe that i have an overactive immune system. I get recurring bouts of Perennial Conjunctivitis and i also experience at times pain on my neck lymph nodes and my right maxillary sinus. I always believed that all of these symptoms were somehow related. Because of my conjunctivitis i was not able to wear contact lenses. My ophthalmologist confirmed that my eye problems were "allergy-related".

Since September 22nd, 2011 i began a personal experiment. I decided to keep a detailed record of various elements of my everyday life : Whether i had a good night sleep and spent time outdoors, what i ate and how much stress i felt. 

I carry almost always my smartphone with me. So I used a Text-Editing application to record every day as much detailed information as i could. Here is an example of two consecutive dates as these appear in my daily log :

10/11/12, slept/bad, vitamin_c/0, coffee/1, self/ok, stress/low, sausages,cholesterol_food, sugar/5, pasta, tomatoes, mushrooms,  next_sleep/ok 

10/12/12, slept/ok, vitamin_c/500, coffee/2, self/ok, stress/high, bread, honey, milk, sugar/10, meat, garlic, yoghurt, conjunctivitis,  icecream,  next_sleep/ok  

So on the first example date, i did not sleep well the previous night. I had one coffee and roughly 5 teaspoons of sugar the whole day. I was feeling ok with myself, i had sausages and eggs (tagged as cholesterol_food) for breakfast and pasta with mushroom (red) sauce for lunch but no dinner. I managed to sleep well at night. I did not have any signs of an overactive immune system. However, the next day i had conjunctivitis.

I then had to somehow transform the entries to a suitable format - a .csv file- which could then be used by Data Mining Software (such as R and WEKA) for analysis. To do that, a simple Java program was used to transform all log entries to a .csv format using the following rules :

1) Each line represents a day.

2) Each entry is separated by comma (",")

3) If an entry does not contain a forward slash character ("/") then it is treated as a Boolean feature. 

4) If an entry contains a forward slash then it is treated as a Numerical or Categorical feature.

So our two example dates, are transformed like this ( Not all features are shown) :

R was used to perform several pre-processing steps such as coding a function called addfeature which i use to derive new variables from old ones :

data.df<-addfeature("fiber",c("beans","stringbeans","oats","okra","lentils"),data.df)

data.df<-addfeature("cholesterol_food",c("eggs","mayo","octopus","squid"),data.df)

data.df<-addfeature("nuts",c("hazelnuts","walnuts","peanuts","cashews","almonds"),data.df)

data.df<-addfeature("immunity",c("itchyeyes","lymphpain","sinuspain","conjunctivitis"),data.df)   
  

So if on any day i had eggs, mayo(nnaise), octopus, squid or any combination of these foods an entry of cholesterol_food will be used to replace these entries.

 Having the log transformed to the format shown above, i was ready to analyze a 1-year worth of data (in this case IMMUNITY is the target), extract patterns and several hypotheses - for example that "there appears to be a connection between high stress and over-activity of my immune system."

However we must be aware of the dangers that might lead us to incorrect findings. For instance we must take into account the fact that conjunctivitis usually lasts more than one day and also that some features -like Vitamin-C intake- are special in the sense that the representation shown above does not take into account the compounding effect of Vitamin intake. In other words, i might have to take for  n number of days,  an x amount of Vitamin C consecutively to see any effect. Furthermore, this analysis does not take into account the sequence of events. Should we remove foods/ingredients that  normally co-occur or not? How would that affect results? The list of questions and considerations goes on (and then when we finally have some results from the analysis, the first thing to do is to question them).

Using Data Science i was able to almost stop getting sinus and lymph pain and wear again my contact lenses ( i still get symptoms but very-very rarely). Two foods  appeared to be moderately correlated with  signs of an overactive immune system - with one of them being garlic. One would probably argue that i could find that using a simple food diary -  i  doubt about it since things were not so evident. Once this information was found these two foods were eliminated from my diet to see the outcome.

Analysis has also identified a particular Vitamin that was able -in my opinion- to regulate my immune system response so that i could have no food restrictions. Several other patterns (or hypotheses) emerged that could be used for further evaluation by specialized personnel. Whatever i tried, i tried it under the close supervision and consent from Doctor specialists.

It's a logical next step to imagine the potential knowledge and hypotheses extracted by implementing the same experiment on a wider scale (for example by using Kaggle) .

On the next post : More thoughts, results and warnings    

Text Analytics for Telecommunications - Part 1

As discussed in the previous post, performing Text Analytics for a language for which no tools exist is not an easy task. The Case Study which i will present in the European Text Analytics Summit is about analyzing and understanding thousands of Non-English FaceBook posts and Tweets for Telco Brands and their Topics, leading to what is known as Competitive Intelligence.

The Telcos used for the Case Study  are Telenor, MT:S and VIP Mobile which are located in Serbia. The analysis aims to identify  the perception of Customers for each of the  three Companies mentioned and understand the Positive and Negative elements of each Telco as this is captured from the Voice of the Customers - Subscribers.

By analyzing several thousands of Tweets and FaceBook posts and comments we can have a first glimpse of Competitive Intelligence. For example when we wish to identify which words frequently occur with mentions about postpaid packages this is what we find  :

Red boxes show Telco Brands - notice "mts" and "mtsa" which point to the same Telco, namely mt:s.  Blue boxes indicate similar words that should be merged.  From a first look at the results above we see that : 

a) mt:s is found more frequently when users mention PostPaid packages.

b) Telenor and VIP Mobile are not found as frequently as MT:S in PostPaid package conversations.

c) We see several  problems from insufficient pre-processing : Kredit and Kredita (=credit) should merge into one word, the same applies for telefona - telefon, internet - interneta and mts - mtsa.



Notice that we can perform the same High-level analysis for several Telco Topics such as Network, Billing, Customer Care, Promotions, Questions of subscribers and so on. The next task is to identify the reason(s) why MT:S was found to have more mentions about PostPaid packages. Note that at this point we do not know why this is so : It could be the fact that MT:S prices of prepaid packages are high, very cheap or something else is happening that needs to be identified.


The Serbian Language poses extra work because it is a highly inflected language : Even the ending  of  Brand names change according to the usage.  Consider the following :

U mts-u (at mts)
Sa mts-om (With mts)
Bez mts-a (Without mts)


It is evident that a highly inflected language explodes our feature space and for this reason R can come to the rescue with some success. We can use R for changing several synonyms to one word, removing (Serbian) stop words, removing URLs and performing several other pre-processing steps that are necessary prior to an extensive analysis. More on the next post.

Case Study : Competitive Intelligence for Telecommunications

Telcos are a good example of a fast moving business environment and a good candidate for using Competitive Intelligence analysis from Social Media sources. The Case Study involves three major Telcos located in an Eastern European Country and shows the results from the analysis of thousands of Tweets and FaceBook wall posts to understand the following :

- How subscribers perceive each Telco Brand? 

- Which information do subscribers tend to Re-Tweet and "Like" on FaceBook Wall Posts? 

- Which words and Topics are commonly found with Intense feelings / thoughts?

- Which topics are mostly discussed when subscribers compare two or more Telco operators?

- What do subscribers discuss about  Network Quality and Speed, Billing, Promotions, Marketing Events, Customer Care, TV Commercials etc.

- How do they prioritize these topics and which of them are interesting and why?  

- What do subscribers talk about in general (i.e without any Telco Brand being mentioned) regarding Internet speed, Charges and what would they expect to see more?

I will present the Case Study mentioned  above in the forthcoming 9th Annual European Text Analytics Summit in April in London - UK. The Case Study is an example of application of Text Analytics to a language for which currently no tools exist and thus all difficulties and possible solutions will also be discussed. Examples will be also given on analyzing information to different conceptual levels and how this technique provides even more insights in consumer behavior.

The following tools were used for the analysis : 

- GATE to annotate all Topics that occur within Telco conversations (such as "sms", "internet", "dropped call", "network","promotion") and for setting up Conceptual Levels.

- R for pre-processing Text and performing Text Classification, Topic Detection and Cluster Analysis.

- WEKA  for Feature Selection and Text Classification.

- Finally,  Java is used to manage the information that is generated from GATE such as  understanding how subscribers prioritize various Telco Concepts and Topics and also identify important phrases and/or words that frequently occur when these Topics are being discussed.  

Forex Trading with R : Part 2

In the previous post the first steps were given for building the basis for trading forex. Now it is time to build the actual classifiers that  can give us future buy / hold / sell signals.

Assuming that everything is in working order and the instructions given in  the previous post were followed we can start building these classifiers.

First let's train a Neural Network. The following command trains a Neural Network and then applies the trained model on our test data and outputs the predictions for buy/sell/hold signals :

set.seed(134)
nn <- nnet(class~.,traindata, size = 3, rang = 0.1,decay = 0.001, maxit = 3000,trace="F")
table(actual=testdata$class,predicted=predict(nn,newdata=testdata,type="class"))

Note that a seed number was used.  You should either try different seed numbers (so that network weights are re-initialized) or omit the set.seed() directive. You should also experiment with other Neural Net parameters such as the number of iterations (maxit), the learning decay (decay), etc.


The confusion matrix shows us the necessary information for calculating TP, FP, TN,FN rates for each class (ie for each signal type).


Similarly we can train and test a Random Forest :


 rf.model<-randomForest(class~.,data=traindata,nodesize=40,importance=FALSE,mtry=3,ntree=100)
table(actual=testdata$class,predicted=predict(rf.model,newdata=testdata,type="class"))

Now let's train an SVM for our data. We can issue the following command : 

###train SVM

sv<-svm(class~.,traindata,gamma=0.01,cost=5,kernel="radial")

To see how the classifier did on the test set, we enter :

table(actual=testdata$class,predicted=predict(sv,newdata=testdata,type="class"))

Next we can try to optimize parameters of the SVM classifier as follows :


#find optimal values of Gamma and Cost for an RBF- SVM classifier
tuned <- tune(svm, class~., data = traindata,ranges = list(gamma = c(0.0001,0.001,0.05,0.1,0.2,0.3), cost = c(1,5,10,20,50,100,120,130)),tunecontrol = tune.control(sampling = "cross"),cross=10)


tuned


The first command uses 10-fold cross validation to identify the best gamma and cost parameters among some predetermined values. We then issue the command tuned to see which combination of parameters  gives us the lowest classification error. Knowing these parameters we can then use these parameters to train an SVM classifier and see how this model performs (as was shown previously).

Be aware of the following key points :

1. Three sets of data should be used : Training, Test and Validation. The Validation set should not be a part of the optimization (=finding the best algorithm parameters) process.
2. Make sure that you create classifiers for several time periods. Test the performance of any classifier according to the percentage of available data you use for training / testing / validation and the number of periods you use for the sliding window.
3. Make also sure that once you have chosen your model, you use a correct way to test your system by simulating buy / hold / sell signals and taking under consideration all associated trading costs.

Forex Trading with R : Part 1

I recently started learning R - probably something i should have done a long time ago - and since learning by doing is the best way to learn something i decided to  use  R to generate buy/see/hold signals for the EUR/USD Pair. For those that wish to use R for making Trading decisions, this series of posts is a short introduction with which one can pursue the subject further. By no means it is implied that this post's methodology is the one that  you should use  to trade :   different response variables, signal thresholds, technical indicators and classifiers than the ones presented here should be tried. Then, elaborate testing methods should be put to use to assess the  performance of  each classifier and the worth of your trading strategy. 

First the resources needed to be downloaded and installed :

1)  Packages : quantmod, nnet, e1071, tseries, randomforest

2) A file that contains OHLC Data. For this example EUR/USD Data are used. You can download an example file here. I downloaded the EUR/USD historical data here. 

I would also highly recommend getting Data Mining with R : A concise book that goes through an introduction of using R and then presents various case studies one of which is about Using R to predict variations of the S&P Index. The author also provides package "DMwR" that includes all necessary functionality for generating signals, extracting precision / recall metrics of generated models, performing Monte Carlo Estimates and evaluating trading strategies.

After downloading the file from step (2), place the file to a directory of your choice. Now copy and paste the following commands in R :

library(e1071)

library(nnet)
library(randomforest)

library(quantmod)

library(tseries)

Next we import the csv file from the directory that was originally saved  (change "YOUR_PATH" accordingly with the directory path you saved the csv file) :

#get data OHLC from csv file

raw<- read.delim2("/YOUR_PATH/EURUSD.csv",header=TRUE,sep=",")

Now, paste the following to R (change again YOUR_PATH accordingly) :

#convert date

stripday<-strptime(raw$DATE,format="%Y%m%d")

fxdata<-data.frame(stripday,raw)

fxdata$TIME<-NULL

fxdata$TICKER<-NULL

fxdata$DATE<-NULL

colnames(fxdata)<-c("Date","Open","Low","High","Close")

#write data to .csv

write.table(fxdata,"/YOUR_PATH/eurusd.csv",quote=FALSE,sep=",",row.names=FALSE)

##transform to an xts object

EURUSD<-as.xts(read.zoo("YOUR_PATH/eur-usd.csv",sep=",",format="%Y-%m-%d",header=T))

Now we define some technical indicators, the model to work on and a function that generates our trading signals  :

#setup Technical Indicators

myATR <- function(x) ATR(HLC(x))[,'atr']

mySMI <- function(x) SMI(HLC(x))[,'SMI']

myADX <- function(x) ADX(HLC(x))[,'ADX']

myAroon <- function(x) aroon(x[,c('High','Low')])$oscillator

myBB <- function(x) BBands(HLC(x))[,'pctB']

myChaikinVol<-function(x)Delt(chaikinVolatility(x[,c("High","Low")]))[,1]

myCLV <- function(x) EMA(CLV(HLC(x)))[,1]

myMACD <- function(x) MACD(Cl(x))[,2]

mySAR <- function(x) SAR(x[,c('High','Close')]) [,1]

myVolat <- function(x) volatility(OHLC(x),calc="garman")[,1]

myEMA10 <- function(x) EMA(Cl(x),n=10)[,1]

myEMA20 <- function(x) EMA(Cl(x),n=20)[,1]

myEMA30 <- function(x) EMA(Cl(x),n=30)[,1]

myEMA50 <- function(x) EMA(Cl(x),n=50)[,1]

myEMA60 <- function(x) EMA(Cl(x),n=60)[,1]

data.model <- specifyModel(Delt(Cl(EURUSD)) ~

myATR(EURUSD) + mySMI(EURUSD) + myADX(EURUSD) + myAroon(EURUSD) +

myBB(EURUSD) + myChaikinVol(EURUSD) + myCLV(EURUSD) +myEMA10(EURUSD) +myEMA20(EURUSD) +myEMA30(EURUSD) +myEMA50(EURUSD) + myEMA60(EURUSD) +

CMO(Cl(EURUSD)) + EMA(Delt(Cl(EURUSD))) +

myVolat(EURUSD) + myMACD(EURUSD) + RSI(Cl(EURUSD)) +

mySAR(EURUSD) + runMean(Cl(EURUSD)) + runSD(Cl(EURUSD)))

Tdata.train <- as.data.frame(modelData(data.model,

data.window=c('2008-01-01','2010-01-01')))

Tdata.eval <- na.omit(as.data.frame(modelData(data.model,

data.window=c('2010-01-02','2010-11-01'))))

# a very simple signal function

signals<-function(x) {

if(x>=-0.005&&x<=0.005) {result<-"hold"} else

if(x>0.005) {result<-"buy"} else

if(x<-0.005) {result<-"sell"}

result

}

#create class vector that holds TRAINING buy,sell,hold signals

class<-sapply(Tdata.train$Delt.Cl.EURUSD,signals)

#paste both to a new list that holds everything

traindata<-cbind(Tdata.train,class)

#remove Delt.Cl.EURUSD - not needed anymore.
traindata$Delt.Cl.EURUSD<-NULL

#create class vector that  holds TESTING buy,sell,hold signals

class<-sapply(Tdata.eval$Delt.Cl.EURUSD,signals)


#paste to a new list that holds everything
testdata<-cbind(Tdata.eval,class)
testdata$Delt.Cl.EURUSD<-NULL

#get a summary of our traindata
summary(traindata)

The last command prints out some summary statistics of the traindata sample. Notice the 'class' attribute and the distribution of buy, hold and sell signals .

Now we are ready to apply some modeling techniques using traindata and testdata as the datasets to work with. Although i would suggest using also a third sample for validation (remember the "elaborate testing" discussed at the beginning), for this example we will keep things  simple.  More on the next post