This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Hello team,
I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows couldn't be used.
@Shweta Chavan Thank you for reaching out.
My understanding is that you are looking to incrementally load data from Amazon S3 to Azure SQL Database.
You can use ADF to achieve this. Please follow the instructions in the doc below. The doc destination data source is Azure Blob storage, but you can replace with Azure SQL Database.
Load data into Azure Data Lake Storage Gen2 with Azure Data Factory
If you are also looking to load New and Changed Files by LastModifiedDate:
Incrementally copy new and changed files based on LastModifiedDate by using the Copy Data tool
Please let me know by commenting below how it went.
Regards,
Oury
@Shweta Chavan Could you please check my response above and let us know by commenting below if you have any additional questions or need further clarification.
Regards,
Oury
@Shweta Chavan Could you please check my response above and let us know by commenting below if you have any additional questions or need further clarification.
Regards,
Oury
Hi Oury,
I am trying to use the Filter by last modified functionality of the Copy activity. Ideally only the files whose last modified in S3 is after the Start time (Filter by last modified Azure )should get copied, However all the files are getting copied.
Hi Oury,
I am trying to use Filter by last modified functionality of copy activity. Ideally the files whose LastModified in S3 is after the Start time(Filter by last mofidied) should get copied , however all the files are getting copied.
@Shweta Chavan - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .
Sorry for the delay in response to your question.
In your SQL DB Table, you can create an identity column that auto increments IDENTITY (Property) (Transact-SQL) - SQL Server | Microsoft Learn.
You would leave this column out of the mapping as SQL will be handling the value.
Did you verify the variable is resolving to a UTC time correctly?
Test with a Get Metadata Activity, leave FileName field blank and return child items. Once you get Get Metadata working, you just copy same into Copy Activity.
Regards,
Oury
@shweta chavan Seems like resolving the UTC time resolved the issue.
Please do not forget to mark as accept answer if the reply was helpful in any way.
Regards,
Oury
@Shweta Chavan - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .
Ask: How to copy the data from Amazon S3 to Azure SQL database incrementally?
Solution: The issue is resolved. I am trying to use Filter by last modified functionality of copy activity. Ideally the files whose LastModified in S3 is after the Start time(Filter by last mofidied) should get copied , however all the files are getting copied.
If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.
If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.
Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.