Azure Data Lake U-SQL April 25 2017 Updates: Introducing Packages, UNPIVOT INCLUDE NULLS, fast file set preview flag, R extension returns dataframes, exporting your cluster database with sample data to your local run and more!

We have concluded the rollout of our April 2017 refresh to all the regions today.

Here are the April 2017 Updates for Azure Data Lake U-SQL and Developer Tooling!

The main items are the release of the package feature that allows you to bundle assembly reference statements, variable declarations into a shareable package and reduce the amount of typing you have to do when including more complex assembly dependencies, allowing UNPIVOT to include rows with nulls, you can return a dataframe from the R extension, and exporting your cluster database with sample data to your local run environment in VisualStudio. We also publicly document the file set scalability improvement opt-in statement. If you use file sets in your query, this is a must try feature!

Thanks to all of you who continue to volunteer to test the new version of the more scalable file set. Now everyone can do it. Please contact us if you have explicit feedback or want to explore the new flexible-schema feature preview for TVF parameters.

Here is the list of topics with links to the detailed release notes:

If you want to use the above preview capabilities, please request access by contacting us.

In order to get access to the new syntactic features and new tool capabilities on your local environment, you will need to refresh your ADL Tools. If you use VisualStudio 2013 or 2015, you can download and install them directly from MSDN or use the new Check for Updates menu item mentioned above. If you are using VisualStudio 2017, you currently have to wait for the next VisualStudio 2017 refresh that should occur about every 6 to 8 weeks. Otherwise you will not be able to use the new features during local run and submission to the cluster will give you syntax errors for the new language features (although you can still submit the script anyway).

You can find more details with examples in the April 2017 release notes (or by clicking on the items above) on our GitHub site, where you also can find our previous release notes.