Speed Up Pandas .to_sql to Insert Data - 100x Faster ( Using SQLalchemy)

-
You need to make following change in pandas package.

Steps:

  1. Go to Pandas Packages.
  2. Go to io folder. Pandas > io
  3. Now Open sql.py file. Pandas > io > sql.py
  4. Look for the method _execute_insert

def _execute_insert(self, conn, keys, data_iter):
    data = [{k: v for k, v in zip(keys, row)} for row in data_iter]
    # conn.execute(self.insert_statement(), data)   
    conn.execute(self.insert_statement().values(data)

Replace your function with above function.

Now, check your speed.

Comments

Post a Comment

Popular posts from this blog

Real Time Twitter Sentiment Analysis Using Spark ,Twitter Streaming API and write to Elastic Serach

-
In this tutorial, I am going to show you how you can use Apache Spark and Twitter Streaming API to get the twitter feed in real time and how you can apply sentiment analysis on the fly. After that we will save that prediction in elastic search, so we can create visualization in Kibana Prerequisites: Understanding of Apache Spark Understanding of some natural language processing This blog is : Not about improving the accuracy of prediction Not about teaching spark  Let's start our tutorial. I will be doing Apache spark coding in scala programming language. We will use some jar dependencies that you have to download from Mavon. Dependencies: twitter4j core spark streaming spark core spark elasticsearch Step 1: Create a Twitter Account Let's create a twitter developer account so we can use your credentials to fetch data from twitter.   https://dev.twitter.com/ Go to My Apps Tab and create a new twitter app or you can use your exist...
Read more

Visualize Real Time Twitter Sentiment Analysis Elastic Search and Kibana

-
Image
Before you start following this tutorial, make sure you have read earlier blog about how to do real time sentiment analysis using spark. Now, if you have followed the blog you will see at the end of code we are pushing data to elastic search. temp.saveJsonToEs("eclipse/test") // Writing to ElasticSearch Prerequisites: Install Elasticsearch ( open source ) Install Kibana ( open source) To push the data to elastic search, you need to include the jar dependency for Elasticsearch. Download Elastic search dependency. Make sure to import following libraries. import org.elasticsearch.spark._ import org.elasticsearch.spark.streaming._ We have two ways to push data to elastic search: In JSON format In Map RDD format We do not have to create index before pushing data. This syntax will create index on the fly. If you want to do manual mappings than you first have to create index with appropriate mappings. To do manual mappings, follow these instructions. Creat...
Read more