Personal Data Mining - (Part 2)

On the previous post i described the way that i used to capture a 1-year worth of personal data using my Smartphone with the purpose of identifying trends with my immune system which at times gave me perennial conjunctivitis and also swollen lymph nodes. Now it was time to analyze all of this data in hope that some useful knowledge could be found.

 I had to make a decision for which tools to use. I used WEKA and also decided to give KNIME a try so here is an example of a KNIME workflow :

I first use the File Reader to read in my 1-year worth of life data, then an R Script which is used for several data transformations. I then send the streams of Data first to an R Script which runs the FSelector package with which several Feature Selection algorithms (about 10 of them) are applied to get an understanding of what are the important Features for the problem at hand.

Then another stream sends the Data to an R node which creates dummy Variables and then sends the transformed Features to a Linear Correlation node for further inspection.

A third stream (not shown) sends the data to 3 Machine-Learning algorithms (namely an SVM, Decision Tree and Random Forest) and the Scorer shows how each algorithm performed.

I first executed the FSelector node using 10-fold Cross-Validation because i wanted to get a first feel of the features that are important in identifying some patterns about my perennial conjunctivitis.  7 out of 10 of  FSelector algorithms agreed that :

1) Vitamin D3
2) Garlic
3) Yoghurt

..Appear to have the most predictive power. The problem is that at this point we do not know if any of the features actually help or aggravate my condition. However, the output of FSelector gives an idea on which features should be looked at more closely.

Then the second stream was run, namely the one which sends the data to 3 machine learning algorithms so that i could get a first feel of how the algorithms perform. All three algorithms gave an F-Score of around  59 - 62%.

By looking at the results some patterns appeared to arise (Note the word "appeared")

1) A rather large daily dose  (>5200 IU) of Vitamin D3 appears to be associated with smaller incidences of conjunctivitis
2) Garlic consumption appears to increase my conjunctivitis incidences.
3) Yoghurt consumption appears to increase my conjunctivitis incidences.


For Pattern (1) we need to be aware that Vitamin D3 dosage has a compounding effect so it is rather naive to think that boolean logic applies (see previous post for more).

Next i had to look at patterns (2) and (3). One of the things that i realized when searching the web for the effect of various nutrients in functions of the human body is the fact that you can find for any several entries that some times contradict  each other. My very brief web search has found Garlic and Yoghurt to be "immune boosters". Of course caution should be exercised in drawing any conclusions because of the way the data have been collected and also the problematic origin of the analysis. Moreover, i am not a doctor and i cannot possibly know whether Garlic or Yoghurt can aggravate an immune response in such a way. 

 I began taking Vitamin D3 and eliminating Garlic and Yoghurt from my diet. The result was that over a period of one month i stopped getting bouts of conjunctivitis and incidences of swollen lymph nodes. So has Vitamin D3 acted as an "immune response regulator" and Garlic - Yoghurt as "immune boosters"?

Although my bouts of conjunctivitis have ceased, I am not in any position to make any claims because there are a lot of uncontrolled variables :

- It could be a placebo effect.
-There may be unknown hidden variables that are important
- (My) Genetics
- Environment
- Variations in Dosage and Nutrient Content
- Interactions between nutrients

and lots of others that could not possibly be accounted for under these circumstances.

What i can say (and this is the reason for writing this post) is that analytics may help us to identify several patterns that may then be used to guide a sound knowledge discovery process. If people had the ability to collect data on a daily basis (see Quantified Self) and then analyze them on a massive scale, several unknown patterns that call for closer investigation could emerge.

Personal Data Mining

I believe that i have an overactive immune system. I get recurring bouts of Perennial Conjunctivitis and i also experience at times pain on my neck lymph nodes and my right maxillary sinus. I always believed that all of these symptoms were somehow related. Because of my conjunctivitis i was not able to wear contact lenses. My ophthalmologist confirmed that my eye problems were "allergy-related".

Since September 22nd, 2011 i began a personal experiment. I decided to keep a detailed record of various elements of my everyday life : Whether i had a good night sleep and spent time outdoors, what i ate and how much stress i felt. 

I carry almost always my smartphone with me. So I used a Text-Editing application to record every day as much detailed information as i could. Here is an example of two consecutive dates as these appear in my daily log :

10/11/12, slept/bad, vitamin_c/0, coffee/1, self/ok, stress/low, sausages,cholesterol_food, sugar/5, pasta, tomatoes, mushrooms,  next_sleep/ok 

10/12/12, slept/ok, vitamin_c/500, coffee/2, self/ok, stress/high, bread, honey, milk, sugar/10, meat, garlic, yoghurt, conjunctivitis,  icecream,  next_sleep/ok  

So on the first example date, i did not sleep well the previous night. I had one coffee and roughly 5 teaspoons of sugar the whole day. I was feeling ok with myself, i had sausages and eggs (tagged as cholesterol_food) for breakfast and pasta with mushroom (red) sauce for lunch but no dinner. I managed to sleep well at night. I did not have any signs of an overactive immune system. However, the next day i had conjunctivitis.

I then had to somehow transform the entries to a suitable format - a .csv file- which could then be used by Data Mining Software (such as R and WEKA) for analysis. To do that, a simple Java program was used to transform all log entries to a .csv format using the following rules :

1) Each line represents a day.

2) Each entry is separated by comma (",")

3) If an entry does not contain a forward slash character ("/") then it is treated as a Boolean feature. 

4) If an entry contains a forward slash then it is treated as a Numerical or Categorical feature.

So our two example dates, are transformed like this ( Not all features are shown) :

R was used to perform several pre-processing steps such as coding a function called addfeature which i use to derive new variables from old ones :

data.df<-addfeature("fiber",c("beans","stringbeans","oats","okra","lentils"),data.df)

data.df<-addfeature("cholesterol_food",c("eggs","mayo","octopus","squid"),data.df)

data.df<-addfeature("nuts",c("hazelnuts","walnuts","peanuts","cashews","almonds"),data.df)

data.df<-addfeature("immunity",c("itchyeyes","lymphpain","sinuspain","conjunctivitis"),data.df)   
  

So if on any day i had eggs, mayo(nnaise), octopus, squid or any combination of these foods an entry of cholesterol_food will be used to replace these entries.

 Having the log transformed to the format shown above, i was ready to analyze a 1-year worth of data (in this case IMMUNITY is the target), extract patterns and several hypotheses - for example that "there appears to be a connection between high stress and over-activity of my immune system."

However we must be aware of the dangers that might lead us to incorrect findings. For instance we must take into account the fact that conjunctivitis usually lasts more than one day and also that some features -like Vitamin-C intake- are special in the sense that the representation shown above does not take into account the compounding effect of Vitamin intake. In other words, i might have to take for  n number of days,  an x amount of Vitamin C consecutively to see any effect. Furthermore, this analysis does not take into account the sequence of events. Should we remove foods/ingredients that  normally co-occur or not? How would that affect results? The list of questions and considerations goes on (and then when we finally have some results from the analysis, the first thing to do is to question them).

Using Data Science i was able to almost stop getting sinus and lymph pain and wear again my contact lenses ( i still get symptoms but very-very rarely). Two foods  appeared to be moderately correlated with  signs of an overactive immune system - with one of them being garlic. One would probably argue that i could find that using a simple food diary -  i  doubt about it since things were not so evident. Once this information was found these two foods were eliminated from my diet to see the outcome.

Analysis has also identified a particular Vitamin that was able -in my opinion- to regulate my immune system response so that i could have no food restrictions. Several other patterns (or hypotheses) emerged that could be used for further evaluation by specialized personnel. Whatever i tried, i tried it under the close supervision and consent from Doctor specialists.

It's a logical next step to imagine the potential knowledge and hypotheses extracted by implementing the same experiment on a wider scale (for example by using Kaggle) .

On the next post : More thoughts, results and warnings