Discover input/output paths of U-SQL jobs using Azure PowerShell
Do you use Azure Data Lake Analytics? Have you ever wanted to programmatically fetch the input and output paths of one of your U-SQL jobs? Now you can!
This is especially useful for developers who want to quickly debug an input file, fetch the results of an analysis, or check which files a particular job touched. With this new script, we’re helping those users by providing a simple Azure PowerShell script that reduces several steps into a one-line command.
How it works
Each time you submit a U-SQL job, a job folder is created in your Azure Data Lake Store account. This folder contains useful debugging information about the job, including a file called the U-SQL algebra file. This is an XML file containing information about your job graph, the list of input and output files, and other key U-SQL job metadata.
We’ve just published a sample script that reads the U-SQL algebra file for a specified job and returns the input or output files. Give it a try!
Running the sample
First, find the Job ID of the job for which you’d like to get the input or output paths. You can easily find this in the Azure Portal by opening your ADLA account, navigating to the desired job, and copying the Job ID.
Alternatively, you can get the job ID of a job that you just submitted via Azure PowerShell.
PS C:\> $job = Submit-AdlJob -Account contosoadla -ScriptPath myScript.usql PS C:\> $job.JobId Guid ---- c3426e86-85f5-4521-b376-e4b3e8d32d8c
Copy the contents of the sample script to a new local file called Get-AdlJobIoPath.ps1
Open a new PowerShell window. Navigate to the folder containing the sample script.
Log in to Azure.
PS C:\> Login-AzureRmAccount
Run the following command in PowerShell, replacing the bolded parts with your own values. This example gets the input file and table paths of the U-SQL job with Job ID c342... .
PS C:\> .\Get-AdlJobIoPath.ps1 -Account contosoadla -Id c3426e86-85f5-4521-b376-e4b3e8d32d8c -Direction Input Type Path ---- ---- Table master.dbo.TransactionsTbl File adl://contosoadls.azuredatalakestore.net/data/SalesData.csv
Try the sample script today!
The source code for the sample is available in GitHub.
Soon, U-SQL job input and output information will be more readily accessible through our REST API, SDKs, and other interfaces.
Let us know your feedback in the comments. Are you looking for any other samples, features, or improvements? Let us know and vote for them on our UserVoice.