Large file options - Synapse Data Factory

Ryan Abbey 1,171 Reputation points
2021-10-19T19:39:19.283+00:00

We have some 1GB+ files that we need to get to Azure, for use within (Synapse) Data Factory, with mimimal inconvenience to the business users needing to provide these files

What we've looked at and why we've stumbled - hopefully someone can point us to something we've missed

  • OneDrive - there does not appear to be a connector for Data Factory which means we need to use Logic Apps - but that has a 50MB file size limit
  • Sharepoint - We haven't actually tested this due to a doubt. We see there's a "Sharepoint Online List" connection in Data Factory, however, our 1GB file is a zip file, does that still come under the "List" definition? Otherwise we are back at Logic Apps and again hit a size limit
  • File share - this is where we are currently at but there's a lot to set up to enable so before going too far down this route, want to find out if there's an easier or better option
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,369 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,624 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-10-20T11:44:00.333+00:00

    Hi @Ryan Abbey ,

    Thank you for posting query in Microsoft Q&A Platform.

    • Since oneDrive connector is not there and you need to go with Logic apps and again Size limit constrain. So definitely we can ignore this option.
    • Sharepoint - Also goes fine. You consider this too.
    • File share - Also goes well.

    But if your data already in on-prem when why to keep that in to oneDrive OR Sharepoint or FileShare. Can't we directly take on-prem data to load in to ADLS gen2?