Multiple CSVs in Blob storage, trying to copy them to a single table in SQL

ntrav 1 Reputation point
2020-06-08T02:37:45.247+00:00

I have approx 30 csv files in blob storage and I am trying to copy them all to a single table in SQL. I have read several forums with similar goals, and I feel like they have pushed me close to the solution, but I have still not been successful.

I have a table in SQL ready for the data, and connected to a pipeline. And I think I have correctly set up the blob dataset to have the container holding the files selected. I then try to use 'ForEach' and activity 'Copy' but this is as far as I get. I am never able to preview this data.

I think if this were to work I would have another pipeline activity to take all the copies and append them to a single table in SQL, but I haven't made it that far. I also haven't found a similar example to this.

I think my blob dataset might not be configured properly. Just generally struggling as I am new to Azure.

Any help would be appreciated

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions

5 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Malleswara Reddy, G 1,631 Reputation points
    2020-06-08T04:23:38.273+00:00

    Hi,

    Are you facing any error? Place your code if it is fine.

    Please go through this video which explains step by step -
    https://www.youtube.com/watch?v=EpDkxTHAhOs&list=PLGjZwEtPN7j8b9dPA0HrtJDptOB69B506

    https://www.mssqltips.com/sqlservertip/6187/azure-data-factory-foreach-activity-example/

    Regards,
    Eshwar

  2. Vaibhav Chaudhari 38,601 Reputation points
    2020-06-08T07:18:07.437+00:00

    Can you share source blob dataset configuration screenshot. It should preview the data if it is set correctly or at least fail if config are not correct.

    Other option would be to merge all files to single file using copy activity and simply another copy activity to load singel file to SQL

    If the response helped, do "Accept Answer" and upvote it - Vaibhav

  3. ntrav 1 Reputation point
    2020-06-08T11:47:54.823+00:00

    9170-blob-config.jpg

    I also get this comment when trying to preview the blob data:

    Please provide actual value of the parameters to preview data
    Parameters for dataset binancedataset

    0 comments
  4. Malleswara Reddy, G 1,631 Reputation points
    2020-06-08T12:24:26.223+00:00

    Please check this video for foreach if the above link does not help you -- https://www.youtube.com/watch?v=P9uk-q-N8a4

    0 comments
  5. Raj Atmuri 1 Reputation point
    2021-06-05T08:04:50.133+00:00

    @ntrav bro did you get that answer

    I also need same solution can you help me

    0 comments