question

Gunwant-3141 avatar image
0 Votes"
Gunwant-3141 asked ·

Azure Data Factory - Self Hosted IR

We are using Azure Data Factory to copy SQL data from on premise to Azure Data Lake. For this we have installed Self hosted IR in one of the VM in On-Premise environment.

Is there a way around where we do not need to manage our own VM or run the Self Hosted IR to get the data from on-prem databases to Azure Data Lake using Azure data factory?

azure-data-factory
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered ·

Hi @Gunwant-3141,

Welcome to Microsoft Q&A and thank you for your query.

In order to copy data from local sources to the cloud using Azure Data Factory, it needs to be able to connect to those sources. This can be achieved only by using the self-hosted integration runtime.

The self-hosted integration runtime is a service running in Azure Data Factory, but you can add local compute nodes on local servers in your on-premises network. The installation of a self-hosted integration runtime needs an on-premises machine or a virtual machine inside a private network. A connection is created between the nodes and the integration runtime within your Azure Data Factory (ADF) in Azure. Through this connection, ADF can reach your local data and copy it securely to the cloud. This set-up is very similar to the Power BI on-premises gateway. In fact, the self-hosted integration runtime used to be called the "data management gateway".


An alternative to using the self-hosted integration runtime (IR), is to use an on-premises ETL tool - like SSIS - to push the data to the cloud. However, this means you need a SQL Server license and a server. The self-hosted IR service is free to use and can be run on any local server. If you want to migrate your data infrastructure to the cloud, Azure Data Factory (and other Azure tools like Logic Apps) are a better choice.


Hope this info helps.


Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.




· 5 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you @KranthiPakala-MSFT for reply.

Let's say if I have a VPN gateway or Express Route connectivity between on-premise and Azure. Do we need Azure self hosted IR in that scenario as well?

0 Votes 0 · ·

Hi @Gunwant-3141 ,

Thanks for your response. AFAIK yes, you need a SHIR if you want to move data from On Premise data source to Azure using ADF, except when using SQL Server Integration Services (SSIS) in Azure Data Factory. If your SSIS packages access on-premises data stores, then you can join your Azure-SSIS IR to a virtual network that is connected to the on-premises network. Please refer to below documentation.

This diagram shows the required connections for your Azure-SSIS IR:

9194-azure-ssis-ir.png


Hope this helps.



Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.




0 Votes 0 · ·
azure-ssis-ir.png (74.0 KiB)

Hi @Gunwant-3141,

Just checking to see if the above info was helpful. If this answers your query, please do consider to click “Accept Answer” and "Up-Vote" as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 · ·

Hi @Gunwant-3141,

Following up to see if the above information was helpful.

0 Votes 0 · ·
Gunwant-3141 avatar image Gunwant-3141 KranthiPakala-MSFT ·

Thank you @KranthiPakala-MSFT. It was helpful.


0 Votes 0 · ·