question

monika-4563 avatar image
0 Votes"
monika-4563 asked MartinJaffer-MSFT commented

Best suited Azure solution for Data Integration

have a requirement to do the Data integration from various source systems to a Master Data Management (MDM). Requirement in crux:

Target system is a MDM (Master data management) which needs data in a XML format.
Multiple Source systems to give different type of attributes (call data columns)
Format of source data is varied like CSV, through API, SAP to Azure Data Lake etc..
Some source systems wants to send through API, some through sFTP etc..
My suggested solution is >>> keep all the source files (is source format) in ADL >> Get Data transformation done( in ADF / Synapes / ADL.. I dont know) >> Azure Function to convert transformed code into XML >> Send to Target

I know there are so many gaps in this solution. I dont know about data handling capacity for Logic apps / Functions. Can we use Logic app connectors to get the bulk data and place in Data storage...

Please share your approach best suited solution for such problem statement.

TIA..

azure-data-factoryazure-logic-apps
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@monika-4563 did either answer solve your issue? If so, please mark as accepted answer, otherwise please tell us how we may better assist.

0 Votes 0 ·

@monika-4563 if you found your own solution, could you please share here with the community?

0 Votes 0 ·
JohnAherne-9205 avatar image
0 Votes"
JohnAherne-9205 answered monika-4563 commented

Logic apps cannot handle large amounts of data. Not sure about function apps.

Your best bet would be Azure Data Factory Copy activity which can write out to XML, but I guess it depends on how complex the schema is.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi John,
Target schema is a STEP-XML.
There is issue with Source end also. how to get the huge data from source and place in ADF?

0 Votes 0 ·
MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

Hello @monika-4563 and welcome to Microsoft Q&A.

In my opinion, you should look at Synapse, rather than Data Factory.

Synapse contains most of Data Factory features, but also has notebooks to run custom Pyspark or SQL code. This is useful for the transformations Data Factory is not enough for. Databricks also has these notebooks and is worth looking into. Synapse also is closely integrated with Azure Data Lake Store Gen2.

The Spark pools of Databricks and Synapse are meant for more volume than the Function App / Logic App.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.