Venture Capitals in an Age of Algorithms (Revisited)

Some time ago i wanted to explore the idea of analyzing several kinds and sources of Information (e.g TechCrunch, TheNextWeb,  News sites and Twitter) to identify promising Investment opportunities in Technology and more specifically Startups. 

Here is a snapshot of a Webpage from TechCrunch :

In many posts in this Blog it was discussed how our Reactions for almost any kind of information are recorded. This was not possible when everyone was reading newspapers in its paper form whereas  now any kind of Text is associated with a number of Views, Re-Tweets, Number of Comments and FaceBook "Likes".

The second important information that is being generated is our Emotions for any Topic as these are expressed within Comments, Twitter and FaceBook posts. The intensity of our emotions is also captured and this information is very important since  whatever we associate with intense emotions really stays within our psyche, fuels our interest and (usually) drives our purchase decisions.

We may then continue with some Exploratory work as follows : We can collect Posts from various Tech sources and their associated Reactions, annotate the text with Sentiment, Events and Topics and analyze this information to understand which Topics and/or Events appear to have an affinity for a high number of Reactions or High Sentiment intensity for Startups or Tech Topics .

As an example, 10K posts from various Tech sources were collected and each one of the posts was marked as generating either HIGH or LOW interest based on the amount of Reactions (Re-Tweets, FB Likes, Comments)  that each post generated. Special filtering is applied for the frequencies of the words that appear in each post :

Then this information is fed to KNIME for further analysis. The implementation which  is shown here is rather naive and simplistic for many reasons :  Only keywords are used as input -as opposed to Topics, Events- and many other parameters that are involved and which will be discussed later but for our example we will keep things simple.

The workflow uses 3 algorithms namely PART (so that some rules are generated), SMO and Random Forests :

 
This -again- is a very naive approach which gave a result of 61.9% (F-Measure) in identifying keywords that commonly appear with posts that generate Interest vs posts that do not. We keep in mind that this knowledge alone is not enough with which a decision can be made but we decide to explore things a little further.

We may find that some words that we expected do appear in posts of High interest (such as Google, Apple, Pinterest). There could be however some words that deserve more of our attention such as Education and Schools which during the analysis appeared to exist more frequently in High Interest posts.

So how can this information be used for a potential investment on a Startup and is there really a way to model new ideas and predict their performance?  Again, it is not suggested here that if you come across a startup aimed in Education you should immediately put your money in but this observation could be one parameter to consider. There are so many other considerations such as whether the idea is novel or not, how many competitors exist, who are the people behind the Startup, whether its founders have created a successful Startup in the past, which people have already invested in the particular Startup, what is the "buzz" that this Startup has generated so far and so on.

Whenever we read about a new startup there are some immediate thoughts going through our minds : Does this sound like a good idea? Is it applicable to me and would it make my life easier? Is this idea truly disruptive or not? What does our "gut feeling" tells us?  

We should always keep in mind that there are limitations to what Predictive Analytics can do but perhaps we can extract some hints that we may then use to make better decisions.

It was also interesting to read this post (hence the use of word "Revisited" in this Post's Title) on Gigaom regarding the same Subject. This is a fascinating area that i started looking at and there will be similar posts in the future on this Subject.

Finding the Right Skillset for Big Data Jobs

Perhaps one of the key skills of a Data Scientist is the ability to be able to collect and access data that are not readily available. 

I was wondering about the trends in Job Postings and more specifically which skills and qualities employers (or agencies) search for on a candidate for a job in "Big Data" so i decided to use R to answer this question.

Of course, one must first find Data (in this case Job Postings) so that they may be analyzed. This is possible by using the library scrapeR of R to scrape content from websites that contain Job Postings. Once this is done, tm package can be used to analyze thousands of Job Advertisements so we may extract useful knowledge.

The analysis which you will see below is based at around one thousand Job Postings that contain the phrase "Big Data". Better pre-processing could help in getting better term co-occurrences but here i aimed in presenting the application.  Once the data are collected we can start by looking at the Frequency distribution of the words found (after removal of stop words) :

Note that the word 'big' is removed from the bar chart. Notice also how the term "experience" (which also includes occurrences of term "experienced") was frequently found in Big Data Job Postings.

Interestingly, the term "skill" (which also counts the term "skilled") is found way below in the frequency diagram.

Next we can use Text Analytics to find which words co-occur with topics of interest. We start by looking at which terms co-occur in Job Postings where Hadoop is mentioned :

Suppose that one wishes to better understand which skills are discussed along with the Java programming language :

 When it comes to skills, it appears that communication skills are those which are important  (as expected):

In the same manner we can  :

-Find the frequencies of skills of interest (e.g Java, Python, Ruby, NoSQL, Oracle DB) and generate trend charts for each of them.

-Run term co-occurrence analysis on the skills which are "good to have" or "preferred".

-Capture early trends on emerging skills (in the "Big Data" case, this could be Pig)

The idea of analyzing Job postings and CVs using Information Extraction (and then using Predictive Analytics once this information becomes structured) is quite interesting. The ability to extract inferred knowledge is also quite challenging : For example could we infer from the text found in CVs  :

-The total number of years of experience in Project Management of an Applicant in case that this has not explicitly been stated in his/her CV?

-Whether an Applicant shows a coherent Career growth through the years ?

-The years needed for an Applicant to move to a Managerial Position?

Personal Data Mining - (Part 2)

On the previous post i described the way that i used to capture a 1-year worth of personal data using my Smartphone with the purpose of identifying trends with my immune system which at times gave me perennial conjunctivitis and also swollen lymph nodes. Now it was time to analyze all of this data in hope that some useful knowledge could be found.

 I had to make a decision for which tools to use. I used WEKA and also decided to give KNIME a try so here is an example of a KNIME workflow :

I first use the File Reader to read in my 1-year worth of life data, then an R Script which is used for several data transformations. I then send the streams of Data first to an R Script which runs the FSelector package with which several Feature Selection algorithms (about 10 of them) are applied to get an understanding of what are the important Features for the problem at hand.

Then another stream sends the Data to an R node which creates dummy Variables and then sends the transformed Features to a Linear Correlation node for further inspection.

A third stream (not shown) sends the data to 3 Machine-Learning algorithms (namely an SVM, Decision Tree and Random Forest) and the Scorer shows how each algorithm performed.

I first executed the FSelector node using 10-fold Cross-Validation because i wanted to get a first feel of the features that are important in identifying some patterns about my perennial conjunctivitis.  7 out of 10 of  FSelector algorithms agreed that :

1) Vitamin D3
2) Garlic
3) Yoghurt

..Appear to have the most predictive power. The problem is that at this point we do not know if any of the features actually help or aggravate my condition. However, the output of FSelector gives an idea on which features should be looked at more closely.

Then the second stream was run, namely the one which sends the data to 3 machine learning algorithms so that i could get a first feel of how the algorithms perform. All three algorithms gave an F-Score of around  59 - 62%.

By looking at the results some patterns appeared to arise (Note the word "appeared")

1) A rather large daily dose  (>5200 IU) of Vitamin D3 appears to be associated with smaller incidences of conjunctivitis
2) Garlic consumption appears to increase my conjunctivitis incidences.
3) Yoghurt consumption appears to increase my conjunctivitis incidences.


For Pattern (1) we need to be aware that Vitamin D3 dosage has a compounding effect so it is rather naive to think that boolean logic applies (see previous post for more).

Next i had to look at patterns (2) and (3). One of the things that i realized when searching the web for the effect of various nutrients in functions of the human body is the fact that you can find for any several entries that some times contradict  each other. My very brief web search has found Garlic and Yoghurt to be "immune boosters". Of course caution should be exercised in drawing any conclusions because of the way the data have been collected and also the problematic origin of the analysis. Moreover, i am not a doctor and i cannot possibly know whether Garlic or Yoghurt can aggravate an immune response in such a way. 

 I began taking Vitamin D3 and eliminating Garlic and Yoghurt from my diet. The result was that over a period of one month i stopped getting bouts of conjunctivitis and incidences of swollen lymph nodes. So has Vitamin D3 acted as an "immune response regulator" and Garlic - Yoghurt as "immune boosters"?

Although my bouts of conjunctivitis have ceased, I am not in any position to make any claims because there are a lot of uncontrolled variables :

- It could be a placebo effect.
-There may be unknown hidden variables that are important
- (My) Genetics
- Environment
- Variations in Dosage and Nutrient Content
- Interactions between nutrients

and lots of others that could not possibly be accounted for under these circumstances.

What i can say (and this is the reason for writing this post) is that analytics may help us to identify several patterns that may then be used to guide a sound knowledge discovery process. If people had the ability to collect data on a daily basis (see Quantified Self) and then analyze them on a massive scale, several unknown patterns that call for closer investigation could emerge.

Personal Data Mining

I believe that i have an overactive immune system. I get recurring bouts of Perennial Conjunctivitis and i also experience at times pain on my neck lymph nodes and my right maxillary sinus. I always believed that all of these symptoms were somehow related. Because of my conjunctivitis i was not able to wear contact lenses. My ophthalmologist confirmed that my eye problems were "allergy-related".

Since September 22nd, 2011 i began a personal experiment. I decided to keep a detailed record of various elements of my everyday life : Whether i had a good night sleep and spent time outdoors, what i ate and how much stress i felt. 

I carry almost always my smartphone with me. So I used a Text-Editing application to record every day as much detailed information as i could. Here is an example of two consecutive dates as these appear in my daily log :

10/11/12, slept/bad, vitamin_c/0, coffee/1, self/ok, stress/low, sausages,cholesterol_food, sugar/5, pasta, tomatoes, mushrooms,  next_sleep/ok 

10/12/12, slept/ok, vitamin_c/500, coffee/2, self/ok, stress/high, bread, honey, milk, sugar/10, meat, garlic, yoghurt, conjunctivitis,  icecream,  next_sleep/ok  

So on the first example date, i did not sleep well the previous night. I had one coffee and roughly 5 teaspoons of sugar the whole day. I was feeling ok with myself, i had sausages and eggs (tagged as cholesterol_food) for breakfast and pasta with mushroom (red) sauce for lunch but no dinner. I managed to sleep well at night. I did not have any signs of an overactive immune system. However, the next day i had conjunctivitis.

I then had to somehow transform the entries to a suitable format - a .csv file- which could then be used by Data Mining Software (such as R and WEKA) for analysis. To do that, a simple Java program was used to transform all log entries to a .csv format using the following rules :

1) Each line represents a day.

2) Each entry is separated by comma (",")

3) If an entry does not contain a forward slash character ("/") then it is treated as a Boolean feature. 

4) If an entry contains a forward slash then it is treated as a Numerical or Categorical feature.

So our two example dates, are transformed like this ( Not all features are shown) :

R was used to perform several pre-processing steps such as coding a function called addfeature which i use to derive new variables from old ones :

data.df<-addfeature("fiber",c("beans","stringbeans","oats","okra","lentils"),data.df)

data.df<-addfeature("cholesterol_food",c("eggs","mayo","octopus","squid"),data.df)

data.df<-addfeature("nuts",c("hazelnuts","walnuts","peanuts","cashews","almonds"),data.df)

data.df<-addfeature("immunity",c("itchyeyes","lymphpain","sinuspain","conjunctivitis"),data.df)   
  

So if on any day i had eggs, mayo(nnaise), octopus, squid or any combination of these foods an entry of cholesterol_food will be used to replace these entries.

 Having the log transformed to the format shown above, i was ready to analyze a 1-year worth of data (in this case IMMUNITY is the target), extract patterns and several hypotheses - for example that "there appears to be a connection between high stress and over-activity of my immune system."

However we must be aware of the dangers that might lead us to incorrect findings. For instance we must take into account the fact that conjunctivitis usually lasts more than one day and also that some features -like Vitamin-C intake- are special in the sense that the representation shown above does not take into account the compounding effect of Vitamin intake. In other words, i might have to take for  n number of days,  an x amount of Vitamin C consecutively to see any effect. Furthermore, this analysis does not take into account the sequence of events. Should we remove foods/ingredients that  normally co-occur or not? How would that affect results? The list of questions and considerations goes on (and then when we finally have some results from the analysis, the first thing to do is to question them).

Using Data Science i was able to almost stop getting sinus and lymph pain and wear again my contact lenses ( i still get symptoms but very-very rarely). Two foods  appeared to be moderately correlated with  signs of an overactive immune system - with one of them being garlic. One would probably argue that i could find that using a simple food diary -  i  doubt about it since things were not so evident. Once this information was found these two foods were eliminated from my diet to see the outcome.

Analysis has also identified a particular Vitamin that was able -in my opinion- to regulate my immune system response so that i could have no food restrictions. Several other patterns (or hypotheses) emerged that could be used for further evaluation by specialized personnel. Whatever i tried, i tried it under the close supervision and consent from Doctor specialists.

It's a logical next step to imagine the potential knowledge and hypotheses extracted by implementing the same experiment on a wider scale (for example by using Kaggle) .

On the next post : More thoughts, results and warnings    

Food Data : The Next Target of Massive Analytics

It has been a very busy period since my last post but also a very interesting one.  At the Social Media Analytics panel of the European Text Analytics Summit there was a question on "What would you suggest to new Entrepreneurs when it comes to Text Analytics". The answer from most of us was "Specialize" :  Build an Exceptionally Good vertical solution.   

Text Analytics has been put to use for several verticals : Finance, Telecommunications, Pharmaceuticals to name a few. Perhaps the next important vertical for Text Analytics is something as Basic -but necessary- as Food. 

Using Analytics for the Food Market is not just about analyzing millions of Tweets to understand and detect Trends on Food consumption, identifying ingredient associations that are liked by Consumers (e.g Olive Oil => Garlic) and the sentiment that a Food experience creates.

Food Sector is a tremendous Market  : Super Markets, Restaurants, Chefs, Books, Magazines, Television Series and Consumers. So Insights from Food Data Analytics could be used by all the "knowledge consumers" mentioned above.

In other words :

- Can we identify emerging trends on the Food Market? And if we can, who are the possible recipients of this knowledge?

- Can we understand and suggest new Food Experiences according to several metrics found whenever Food is discussed in Social Media?

-What other potential sources can be used to collect and then analyze Food Data?

- Can we understand how consumers make choices when it comes to Food?

-Can we Predict Popular Recipes? And how can we monetize from this knowledge?

Text Analytics is  a key technology for transforming all the unstructured information on Food found on the Web. Predictive Analytics can be put to use if we can combine unstructured information with a target variable that we wish to predict.

One of the interesting tasks of a Data Miner is to be able to identify several -actionable and interesting- applications of both Data Mining and Text Mining given some Data. Of importance is also to find and/or to create new Data sources that can help in making better predictions. This is a challenging task but with careful considerations and lots of testing it may well prove to be a worthwhile and rewarding experience.

Coming back to Food Data we could potentially use mentions from Tweets, FB Posts and "Likes", Blog and  Website Posts to capture unstructured information. The hardest part is to be able to somehow incorporate more information about Consumer Behavior as this knowledge -and also to be able to predict Consumer Behavior - would be particularly interesting.

There is a limitation on what Analytics can do especially when we are talking about Predicting Consumer Behavior. As always, proper Data Collection, Pre-processing and thorough Testing is required to reach consistent results.

Text Analytics in Telecommunications - Part 3

It is well known that FaceBook contains a multitude of information that can be potentially analyzed.  A FaceBook page contains several entries (Posts, Photos, Comments, etc) which in turn generate Likes. This data can be analyzed to better understand the behavior of consumers towards a Brand, Product or Service.

Let's look at the analysis of the three FaceBook pages of MT:S, Telenor and VIP Mobile Telcos in Serbia as an example.  The question that this analysis tries to answer is whether we can identify words and phrases that frequently appear in posts that generate any kind of reaction (a "Like", or a Comment) vs words and topics that do not tend to generate reactions . If we are able to differentiate these words then we get an idea on what consumers tend to value more : If a post is of no value to us then we will not tend to Like it and/or comment it.

To perform this analysis we need a list of several thousands of posts (their text) and also the number of Likes and Comments that each post has received. If any post has generated a Like and/or a Comment  then we flag that post as having generated a reaction. The next step is to feed that information to a machine learning algorithm to identify which words have discriminative power (=which words appear more frequently in posts that are liked and/or commented and also which words do not produce any reaction.)

After performing this analysis we essentially come up with a list of words and a metric which tells us the discriminative power of each word. Here is an example of identifying these words :

(Note : Results based on a very limited Data Sample)

Keeping in mind that results shown are extracted from a very limited amount of data, the decision tree depicted above shows us that :

The presence of word Dragi (which means "Dear" in Serbian) means that a post usually does not receive reactions. This makes sense as many posts that reply to subscriber questions start with the word "Dear" and then the first name of the subscriber is added.

novo (="new") is a word that receives a lot of reactions along with hocu (="i want") and dopuna (=recharging credit for prepaid subscriptions). In the same manner we identify more words that are selected to be important in discriminating interesting vs non-interesting posts.  Note that we have to identify the correct context. For example we have to identify what the word novo refers to most of the time : A new cell phone or a new promotion?  From the sample analyzed It appears that :

 1) Subscribers "like" posts that discuss  New devices such as Cell phones and tablets (The Next Step could  be the identification of these devices)

2) Subscribers want new promotions (but we then need to find which types of promotions exactly)

3) Issues with incorrect re-charging are creating a very negative sentiment (but then we need to find  which operator co-occurs with this sentiment and for which cases)

In this way we are able to better understand subscribers, extract the Topics that they are interested in and take all this information into account when creating future initiatives. Note that with this way we can have hints on several potential "hot" topics such as Cell Phone and Tablet Brands, Tariffs, Services, Marketing campaigns, and that this can be performed for each Telco Provider page which means that we can analyze and identify the "hot topics" applicable for each Telco provider.

All the above along with several other uses of Predictive and Text Analytics for Telecommunications i will present in the upcoming European Text Analytics Summit in London, UK.

In the event that a Marketing or PR Agency uses -as in the ways shown above- Social Media Analytics to identify hot topics in News, Sports, TV, Banking and Consumer Goods  a Knowledge Base is created which has many uses : Imagine a scenario where a Telecommunications provider wishes to use a Sport event for a Marketing campaign. We could take into account the hot topics found from a "Sports" analysis and suggest ideas in a much more informed way.  More for this on the next post.

Text Analytics for Telecommunications - Part 2

In the previous post we have seen the problems that a highly inflected language creates and also a very basic example of Competitive Intelligence. The Case Study that i will present in the forthcoming European Text Analytics Summit is about the analysis of Telco Subscriber conversations on FaceBook and Twitter that involve Telenor, MT:S and VIP Mobile located in Serbia.

It is time to see what Topics are found in subscriber conversations. Each Telco has its own FaceBook page which contains posts and comments generated by page curators and subscribers. Each post and comment also generates "Likes" and "Shares". Several types of analysis can be performed to find out :

1) What kind of Topics are discussed in posts and comments of each Telco FaceBook page?
2) What is the sentiment?
3) Which posts (and comments) tend to be liked and shared (=generate Interest and reactions)?

For each FaceBook page post, an identifier is added to the post text which designates the origination page (either MT:S, Telenor or VIP Mobile) of the post. Prior the analysis of  a FaceBook Post which says "We want more promotions" we need to be aware that this text originated -for example- from the MT:S FaceBook Page and not Telenor's.

Identifying the topics discussed in Telco subscribers posts and comments has a number of benefits. We gain a better understanding in the areas that a Telco should focus on. If we find that the topic of INTERNET is on the top  list of discussions repeatedly then this is where a Telco should pay attention. If we find Network mentions  to be associated with a competitor Telco repeatedly (which most likely is not good) we can choose the right time to air commercials implying that we are constantly working for better Network coverage.  We can also identify how much "buzz' was created from a new marketing campaign or  a new phone offer and the sentiment associated with it.

Let's have a look at the first type of analysis, namely Topic Detection. Using Information Extraction we  identify the Topics mentioned in thousands of FaceBook posts and comments for a particular period. Here are the results :

In terms of user engagement on FaceBook, MT:S is the winner since in absolute numbers, its FaceBook page contains more posts and comments than the other two FaceBook Telco Pages. Notice the frequencies of other topics found such as SMILEYs, INTERNET, PROMOCIJA (= Promotion), MREZA (=Network), POSTPAID and ANDROID.

With the chart shown above we become aware of the distribution with which Topics are discussed on all three FaceBook Pages. We do not know what is being discussed for now but we know that subscribers  talked more about Internet, then Promotions (=PROMOCIJA), then Network (=MREZA) and so on.

Let's look at  what Topics exist with PROMOCIJA (=Promotion). In other words, which other Topics are found in FaceBook Posts when Promotions are mentioned? Here are the results :

Most posts collected that discuss about Promotions are actually posts found on the MT:S FaceBook page. Notice also that the presence of Topic HOCU (=I want) which tells us that subscribers simply state that they want new promotions. Here is what the picture looks like for topic INTERNET :

So Telenor is found more frequently in INTERNET mentions. However caution is required since we do not know if all of  these Topic distributions found and their associations with specific Telcos can be attributed to pure chance or not.

It is very important to be confident enough to communicate to any Telco that being associated with Network mentions or any other Topic  is -or is not- simply a random event.