Migrating AppEngine Datastore Entities to a SQL Database: Azure Load

In my last post, I went over the process of getting our datastore entities from into a .csv file via BigQuery.  This seemed like a good split point, since getting a .csv or Excel file into an Azure SQL database might be of interest regardless of where the .csv came from.  I'll also quickly go over some Azure gotcha's to look out for.

As a bit of background, initially I was just going to do all the querying from within BigQuery.  But it was a bit clunky, and kind of intimately tied me in to the process of getting the data the supervisors wanted.  My manager pointed out that a more general solution was probably preferable, so I looked into getting the .csv data into a SQL database.  This is the rest of the story...

Migrating AppEngine Datastore Entities to a SQL Database: Extraction

Handwavy, tl;dr version: 

• Back up entities to Cloud Storage
• Import into BigQuery
• Export .csv
• Migrate with SQL Server Import - Export Data tool

Our help ticket system at work (creatively named "HTS") is hosted on Google AppEngine, with data persisted to the HR datastore.  My manager approached me (since I'm the "appengine developer") and asked me if it would be possible to do a "data dump".  Nothing pretty, just get it all out and into a spreadsheet or something so that supervisors could run some rudimentary reporting (the build-in capabilities in HTS weren't cutting it). So I rolled up my sleeves and got to work.  Here's how it all went down...