Enabling U-SQL Advanced Analytics for Local Execution
After we announced the ability for U-SQL to massively distributed Python code in the Azure Data Lake Analytics service, a lot of developers have been asking us when the the Python support will work using U-SQL Local Execution. In this post, we'll describe how to make that happen.
To set expectations, what we will describe below is not officially supported - we do intend to enable Python extensions in Local Execution as part of the product - but until then these instructions are meant for those brave souls who are willing to try something at the cutting edge.
STEP 1: Install the U-SQL Advanced Analytics extensions in your Data Lake Analytics Account
- Launch the Azure Portal
- Navigate to your Data Lake Analytics Account
- Click Sample Scripts
- Click More and select Install U-SQL Extensions
- Wait until the extensions have finished installing - this should take about 5 minutes
STEP 2: Examine the output of the installation
- When Step 1 is finished, it will have created a folder called /usqlext in the default Data Lake Store account
- Go to your Data Lake Analytics Account in the Azure Portal
- Click Data Explorer
- Navigate to your default Data Lake Store account
- Verify that you see a folder called /usqlext
STEP 3: Setting up your ADLTools4VS local "DataRoot"
- Launch Visual Studio
- Select Tools > Options > Azure Data Lake > General
- Under U-SQL Local Run, find and copy the value for DataRoot
- The value will look like this: C:\Users\username\AppData\Local\USQLDataRoot
- Open Explorer and browse to that folder
- Create a "usqlext" folder in that root.
- The full path should look like this: C:\Users\username\AppData\Local\USQLDataRoot\usqlext
- Launch Azure PowerShell and login to your subscription with the Login-AzureRmAccount cmdlet
- Run the cmdlet below to copy the usqlext folder from your Data Lake Analytics account to your local DataRoot
- Export-AdlStoreItem -Account <youradlsaccount> -Path /usqlext -Destination C:\Users\username\AppData\Local\USQLDataRoot\usqlext -Recurse
- This cmdlet will download about 4GB of data from the Data Lake Store and take a few minutes.
STEP 4: Install the U-QL Advanced Analytics assemblies into the Local Execution U-SQL Catalog
- Launch Visual Studio
- Start a new U-SQL project
- Copy the text from C:\Users\username\AppData\Local\USQLDataRoot\usqlext\RegisterAll.usql into your U-SQL script
- Run your U-SQL script in Local Execution mode
- Once the script finishes, you will be able to use the U-SQL Advanced Analytics features -(Python, R, Cognitive) on your own machine.