Day of a Data Scientist

Before you start following this tutorial, make sure you have read earlier blog about how to do real time sentiment analysis using spark. Now, if you have followed the blog you will see at the end of code we are pushing data to elastic search. temp.saveJsonToEs("eclipse/test") // Writing to ElasticSearch Prerequisites: Install Elasticsearch ( open source ) Install Kibana ( open source) To push the data to elastic search, you need to include the jar dependency for Elasticsearch. Download Elastic search dependency. Make sure to import following libraries. import org.elasticsearch.spark._ import org.elasticsearch.spark.streaming._ We have two ways to push data to elastic search: In JSON format In Map RDD format We do not have to create index before pushing data. This syntax will create index on the fly. If you want to do manual mappings than you first have to create index with appropriate mappings. To do manual mappings, follow these instructions. Creat...

In this tutorial, I am going to show you how you can use Apache Spark and Twitter Streaming API to get the twitter feed in real time and how you can apply sentiment analysis on the fly. After that we will save that prediction in elastic search, so we can create visualization in Kibana Prerequisites: Understanding of Apache Spark Understanding of some natural language processing This blog is : Not about improving the accuracy of prediction Not about teaching spark  Let's start our tutorial. I will be doing Apache spark coding in scala programming language. We will use some jar dependencies that you have to download from Mavon. Dependencies: twitter4j core spark streaming spark core spark elasticsearch Step 1: Create a Twitter Account Let's create a twitter developer account so we can use your credentials to fetch data from twitter.   https://dev.twitter.com/ Go to My Apps Tab and create a new twitter app or you can use your exist...