Azure Data Lake & Azure HDInsight Blog

The official blog for the Azure Data Lake services - Azure Data Lake Analytics, Azure Data Lake Store and Azure HDInsight

Introducing U-SQL database projects – U-SQL database development and deployment made easy (public preview)

Today we are pleased to introduce the U-SQL database project, a new project type in Azure Data Lake...

Author: Yanan Cai - MSFT Date: 07/31/2018

Build hybrid cloud analytics solutions with ADLA Task in SSIS

Today, we are pleased to announce new support for the Azure Data Lake Analytics Task (ADLA Task) in...

Author: Yanan Cai - MSFT Date: 07/23/2018

Easier Azure Data Lake Store management: alerts for folders and files.

The massive scale and capabilities of Azure Data Lake Store are regularly used by companies for big...

Author: José Lara [MSFT] Date: 06/19/2018

Process more files than ever and use Parquet with Azure Data Lake Analytics

In a recent release, Azure Data Lake Analytics (ADLA) takes the capability to process large amounts...

Author: MRys Date: 06/11/2018

Azure Data Lake Analytics and U-SQL Spring 2018 Updates: Parquet support, small files, dynamic output, fast file sets, and much more!

Hello Azure Data Lake and U-SQL fans and followers. It is high time for the release notes for all...

Author: MRys Date: 06/11/2018

Get started with U-SQL: It’s easy!

Azure Data Lake Analytics combines declarative and imperative concepts in the form of a new language...

Author: Ajeta Singhal Date: 05/26/2018

Use AU Analyzer for faster, lower cost Data Lake Analytics

Do you use Data Lake Analytics and wonder how many Analytics Units your jobs should have been...

Author: Matthew Hicks Date: 05/15/2018

Keeping Data Lake Costs Under Control: Creating Alerts for AUs Usage Thresholds.

Have you ever been surprised by a larger-than-expected monthly Azure Data Lake Analytics bill?...

Author: José Lara [MSFT] Date: 04/23/2018

Simple Trick to Stay on top of your Azure Data Lake: Create Alerts using Log Analytics

If you manage one or more Azure Data Lake accounts, do you ever find it hard to stay on top of...

Author: José Lara [MSFT] Date: 04/10/2018

Using the first job run to optimize subsequent runs with Azure Data Lake job AU analyzer

Customers have been telling us it's hard to find the right balance between Analytics Units (AUs) and...

Author: Jie Su Date: 03/30/2018

Delivering Consistency and Accuracy: Improvements for Azure Data Lake Store Reporting

Have you ever uploaded a 70GB file from Windows into your Azure Data Lake Store and noticed that the...

Author: José Lara [MSFT] Date: 03/05/2018

Azure Data Lake Command-line Quickstart using Azure Cloud Shell

Have you been using Azure Data Lake Analytics mainly from the Azure Portal or Visual Studio...

Author: Matthew Hicks Date: 02/13/2018

Struggling to get insights for your Azure Data Lake Store? Azure Log Analytics can help!

Customers love to use Azure Data Lake across their organizations by enabling their Data Lake...

Author: José Lara [MSFT] Date: 02/06/2018

From unstructured data to dashboard with Azure Data Factory and Azure Data Lake

When I joined the Big Data team at Microsoft, sifting through all the technologies and products left...

Author: MattBasile_MSFT Date: 01/25/2018

Debugging Azure Data Lake Job Failures Made Easy (part 2) - Efficiently troubleshoot anomalies in recurring jobs

Azure Data Lake Analytics recently announced advanced job tracking and management features that make...

Author: Yanan Cai - MSFT Date: 01/16/2018

How to Save Money and Control Costs with Azure Data Lake Analytics

Where there is great power there is great responsibility -Winston Churchill Azure Data Lake...

Author: Saveen Reddy Date: 01/08/2018

Get more out of your U-SQL scripts -- Reuse them by adding script parameters with Azure PowerShell

Do you submit U-SQL jobs to Data Lake Analytics using Azure PowerShell? Do you find yourself making...

Author: Matthew Hicks Date: 12/20/2017

Debugging Azure Data Lake Job Failures Made Easy (part 1) - Debug U-SQL job failure of C# custom code

Working with large datasets is hard -- when developers build big data applications, it is impossible...

Author: Yanan Cai - MSFT Date: 12/08/2017

Discover input/output paths of U-SQL jobs using Azure PowerShell

Do you use Azure Data Lake Analytics? Have you ever wanted to programmatically fetch the input and...

Author: Matthew Hicks Date: 12/06/2017

Ad-hoc query support in Azure Data Lake Tools for Visual Studio

Developers using Azure Data Lake Tools for Visual Studio can now create single U-SQL scripts without...

Author: Yanan Cai - MSFT Date: 11/29/2017

Run your PySpark Interactive Query and batch Job in Visual Studio Code

We are excited to introduce the integration of HDInsight PySpark into Visual Studio Code (VSCode),...

Author: JennyJiang Date: 11/22/2017

Getting new insights into your usage of Data Lake Analytics

Users of Azure Data Lake Analytics consistently ask for more insights about their usage for both...

Author: Ajeta Singhal Date: 11/18/2017

Find your U-SQL jobs in Azure Data Lake Analytics with one click.

If your Data Lake Analytics account is heavily used, you might have to scroll through a long list of...

Author: Ajeta Singhal Date: 11/14/2017

Organize your pipeline and recurring jobs easily with Data Lake Analytics (part 2)

Identify performance problems and reduce failed jobs with pipeline and recurring job information In...

Author: Alan Tan (MSFT) Date: 11/01/2017

Continuous integration made easy with MSBuild support for U-SQL (preview)

Azure Data Lake provides enterprise ready big data as a service on Azure. A key requirement of...

Author: Yanan Cai - MSFT Date: 10/24/2017

Simple database copies with the Azure Data Lake Database Export Wizard

Azure Data Lake Tools for Visual Studio now provides the ability to export all or part of a database...

Author: Yanan Cai - MSFT Date: 10/20/2017

Organize your pipeline and recurring jobs easily with Data Lake Analytics (part 1)

Tagging and exploring pipeline and recurring jobs with metadata In a previous blog post, we...

Author: Alan Tan (MSFT) Date: 10/19/2017

Azure Data Lake Tools for Visual Studio Code (VSCode) October Updates

If you are a data scientist looking for a lightweight code editor for U-SQL, try ADL Tools for...

Author: JennyJiang Date: 10/19/2017

Azure Data Lake Analytics and U-SQL Summer 2017 Updates: Introducing GZip on OUTPUT, Catalog Views, Major updates to Cognitive Libraries, Tool support to create your EXTRACT statement and much more!

Hello Azure Data Lake and U-SQL fans and followers. It has been a while since we release the release...

Author: MRys Date: 10/05/2017

Managing Pipeline & Recurring Jobs in Azure Data Lake Analytics Made Easy

Azure Data Lake Analytics (ADLA) is a big data job service that enables you to develop and run data...

Author: Yan Li(Microsoft) Date: 09/19/2017

Directly store streaming data into Azure Data Lake with Azure Event Hubs Capture Provider

Azure Data Lake (ADL) customers use Azure Event Hubs extensively for ingesting streaming data - but...

Author: Sachin C Sheth Date: 08/28/2017

U-SQL Tip: Generating ranges of numbers and dates

Many common scenarios for U-SQL developers require constructing a RowSet made up of a...

Author: Saveen Reddy Date: 08/18/2017

Running XGBoost on Azure HDInsight

XGBoost is a popular open-source distributed gradient boosting library used by many companies in...

Author: Xiaoyong Zhu (MSFT) Date: 08/18/2017

Create U-SQL EXTRACT Script Automatically

In this blog, you will learn how to create U-SQL EXTRACT script automatically using the latest...

Author: Yanan Cai - MSFT Date: 08/08/2017

Integrating Kyligence Analytics Platform with Microsoft Azure HDInsight

This is a guest blog from Shaofeng Shi, Senior Architect from Kyligence Inc. Introducing Kyligence...

Author: Xiaoyong Zhu (MSFT) Date: 07/31/2017

Analyze data in Azure Data Lake Store using familiar-and-powerful Excel 2016

We are excited to announce that as part of the June 2017 updates of Excel 2016, Azure Data Lake...

Author: Sachin C Sheth Date: 07/19/2017

Azure Data Lake Tools for Visual Studio Code (VSCode) July updates

We are pleased to announce the July updates of Azure Data Lake Tools for VSCode. This is a quality...

Author: JennyJiang Date: 07/14/2017

Solving the problem of "Problem with the SSL CA cert (path? access rights?)" for R server on HDInsight

R Server on HDInsight is an ideal platform for performing big data analysis using R interface. If...

Author: Xiaoyong Zhu (MSFT) Date: 07/07/2017

Webinar with Talena: Migrate open source big data applications to HDInsight, add backup & restore to your existing apps running on Apache Hadoop & Spark

“Are you building Big data applications using open source platforms such as Apache Hadoop or Spark,...

Author: rustd Date: 06/28/2017

Run H2O.ai in R on Azure HDInsight

This blog post is authored by Daisy Deng, Abhinav Mithal in Cloud AI group at Microsoft In our...

Author: Xiaoyong Zhu (MSFT) Date: 06/26/2017

Microsoft R Server 9.1 on HDInsight is available!

Today, we are excited to announce that Microsoft R Server 9.1 on Azure HDInsight is generally...

Author: Xiaoyong Zhu (MSFT) Date: 06/19/2017

Managing Your Azure Data Lake Analytics Compute Resources (Job-level Policy)

In Managing Your Azure Data Lake Analytics Compute Resources (Overview) and Account Level Policy, we...

Author: Yan Li(Microsoft) Date: 06/08/2017

Managing Your Azure Data Lake Analytics Compute Resources (Account-level Policy)

In Managing Your Azure Data Lake Analytics Compute Resources (Overview), we introduced why customers...

Author: Yan Li(Microsoft) Date: 06/08/2017

Managing Your Azure Data Lake Analytics Compute Resources (Overview)

Azure Data Lake Analytics (ADLA) is a powerful job service that allows organizations to run small or...

Author: Yan Li(Microsoft) Date: 06/08/2017

Announcing Microsoft Machine Learning Library for Apache Spark

This post is authored by Roope Astala, Senior Program Manager, and Sudarshan Raghunathan, Principal...

Author: Xiaoyong Zhu (MSFT) Date: 06/08/2017

HDInsight tools for IntelliJ May updates

The primary focus for our May updates is to make the Spark development work easier for you in...

Author: JennyJiang Date: 06/01/2017

HDInsight : BUILD Hive Lab is available now

Got some time to learn Big Data Technologies? How about starting with Hive which is considered...

Author: AshishThapliyal Date: 05/31/2017

Azure HDInsight: How to run Presto in one simple step and query across data sources such as Cosmos DB, SQL DB & Hive

I have seen in past few months many inquiries on how to run Presto in HDInsight. In this post we...

Author: AshishThapliyal Date: 05/18/2017

Webinar with Datameer: Modern Data Preparation, Analytics and Insights - Key ingredients for an ML- and AI-ready organization

“Excited about using Big Data to build intelligent applications, but unsure of how to proceed?” Join...

Author: rustd Date: 05/17/2017

Allowing multiple users to access R Server on HDInsight

Recently there are a few customers asking me how to enable multiple users to access R Server on...

Author: Xiaoyong Zhu (MSFT) Date: 05/13/2017

Next>