question

RajD-9527 avatar image
0 Votes"
RajD-9527 asked SaurabhSharma-msft commented

Export data from Azure SQL database as json document on to Azure Datalake Gen2 using Powershell

Greetings!!!

I am using the Powershell code below to connect to an Azure SQL Database and would like to export data as json document on to Azure Datalake Gen2. Could you please guide me how to accomplish this.

 $InstanceName = "SQLTEST\DEMO"
 $connectionString = "Server=$InstanceName;Database=sqldb;Integrated Security=True;"
    
 $query = "SELECT * FROM dbo.Inventory"
    
 $connection = New-Object System.Data.SqlClient.SqlConnection
 $connection.ConnectionString = $connectionString
    
 $connection.Open()
 $command = $connection.CreateCommand()
 $command.CommandText = $query
    
 $result = $command.ExecuteReader()
    
 $table = new-object "System.Data.DataTable"
    
 $table.Load($result)
    
 $table | select $table.Columns.ColumnName | ConvertTo-Json
    
 $connection.Close()

Thank you


windows-server-powershellazure-sql-databaseazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @rajd-9527,

Thanks for using Microsoft Q&A !!
I understand correctly, you want to export your SQL query results from an Azure SQL Database to ADLS Gen2 storage as a Json file.
If that's the case, you can achieve this by using your machine's local drive as an interim storage location to generate the file and then upload it to ADLS Gen2 Storage account. Please find below the updated code to upload query results to ADLS Gen2.

 $InstanceName = "{ServerName}.database.windows.net"
 $connectionString = "Server=$InstanceName;Database={DB Name};Authentication=Sql Password;User={UserName};Password={Password}"
        
 $query = "select * from [dbo].[Test]"
        
 $connection = New-Object System.Data.SqlClient.SqlConnection
 $connection.ConnectionString = $connectionString
        
 $connection.Open()
 $command = $connection.CreateCommand()
 $command.CommandText = $query
        
 $result = $command.ExecuteReader()
        
 $table = new-object "System.Data.DataTable"
        
 $table.Load($result)
     
 $inputFilePath = "c:\input_data.json"
 $table | select $table.Columns.ColumnName | ConvertTo-Json | Out-File $inputFilePath
        
 $connection.Close()
    
 $ctx = New-AzStorageContext -StorageAccountName '{Storage Account Name}' -StorageAccountKey '{Storage Account Key}'
    
 $filesystemName = "input"
 $dirname = "/"
 $destPath = $dirname + (Get-Item $inputFilePath).Name
 New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Force


Here I am storing query results as a file on system's local drive using Out-File and later connecting to respective Gen2 Storage account using New-AzStorageContext with storage account key and password. Once connection is established, I am using New-AzDataLakeGen2Item to upload the exported json file to ADLS Gen2 storage account.

Please refer to the documentation to get details on using PowerShell cmdlets on Storage account.
Please let me know if you have any questions.

Thanks
Saurabh


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @SaurabhSharma-msft, Thanks for your response. Is there a way to save the results from a table as json document on to the ADLS Gen2 directory and completely skip having to save the file on a local drive?

Regards.

0 Votes 0 ·

Hi @rajd-9527,
I don't think it is possible but let me try to check internally if there is a way to do it.

Thanks
Saurabh

0 Votes 0 ·
RajD-9527 avatar image RajD-9527 SaurabhSharma-msft ·

Hi @SaurabhSharma-msft, Sure, Thanks very much.

0 Votes 0 ·
Show more comments