Day of a Data Scientist

If you are trying to convert Spark Dataframe to Rdd of labeled point then you might run into a problem while converting feature vector of that dataframe to feature vector of Rdd. When you use VectorAssembler from "org.apache.spark.ml.feature.VectorAssembler"  to create a feature vector you will be generating a dense vector of "ml" package in spark. So, if you try tp convert a rdd of a labelled point to dense vector, it will generate type mismatch error. Rdd of label point expects "vector from mllib package" Solution: row is a Row of a dataframe {row=>LabeledPoint(row.getAs[Int]("label"),org.apache.spark.mllib.linalg.Vectors.fromML((row.getAs("features")))) } Here, row.getAs("features") will give us  org.apache.spark.mllib.linalg.DenseVector and using org.apache.spark.mllib.linalg.Vectors.fromML using we can convert that Dense vector to vectors of "mllib" package.

Before you start following this tutorial, make sure you have read earlier blog about how to do real time sentiment analysis using spark. Now, if you have followed the blog you will see at the end of code we are pushing data to elastic search. temp.saveJsonToEs("eclipse/test") // Writing to ElasticSearch Prerequisites: Install Elasticsearch ( open source ) Install Kibana ( open source) To push the data to elastic search, you need to include the jar dependency for Elasticsearch. Download Elastic search dependency. Make sure to import following libraries. import org.elasticsearch.spark._ import org.elasticsearch.spark.streaming._ We have two ways to push data to elastic search: In JSON format In Map RDD format We do not have to create index before pushing data. This syntax will create index on the fly. If you want to do manual mappings than you first have to create index with appropriate mappings. To do manual mappings, follow these instructions. Creat...

In this tutorial, I am going to show you how you can use Apache Spark and Twitter Streaming API to get the twitter feed in real time and how you can apply sentiment analysis on the fly. After that we will save that prediction in elastic search, so we can create visualization in Kibana Prerequisites: Understanding of Apache Spark Understanding of some natural language processing This blog is : Not about improving the accuracy of prediction Not about teaching spark  Let's start our tutorial. I will be doing Apache spark coding in scala programming language. We will use some jar dependencies that you have to download from Mavon. Dependencies: twitter4j core spark streaming spark core spark elasticsearch Step 1: Create a Twitter Account Let's create a twitter developer account so we can use your credentials to fetch data from twitter.   https://dev.twitter.com/ Go to My Apps Tab and create a new twitter app or you can use your exist...