question

SinghKumarDeepesh-9270 avatar image
0 Votes"
SinghKumarDeepesh-9270 asked JogalekarMandar-9214 answered

Data parameterization in the data flow within Azure data factory | Source options - "Fiter by last modified" start time (UTC)

Hi,

We are trying to parameterize the dates ( Source options - "Filter by last modified" start time (UTC)) in the data factory- data flow - but its not allowing us to fetch the expected result.

The date parameterization logic is not getting picked up and is getting trespassed with NO EROORS.88244-capture-df1.png

Input date format: 2021-04-12 04:53:00.000000088293-capturedf2.png

Logic applied - toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss')


azure-data-factory
capture-df1.png (60.4 KiB)
capturedf2.png (6.5 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @singhkumardeepesh-9270,

Thanks for using Microsoft Q&A !!
I am able to reproduce this and checking internally with the products team. I will update you back with my findings.

Thanks
Saurabh

1 Vote 1 ·

@SaurabhSharma-msft Thanks, please let me know if you need anything else to troubleshoot on this, thanks.

1 Vote 1 ·

@SaurabhSharma-msft Any update here on this please?

0 Votes 0 ·
Show more comments
SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @singhkumardeepesh-9270,
If your Input date format: 2021-04-12 04:53:00.0000000 then you need to use 'yyyy-MM-dd HH:mm:ss.SSS' in your toTimeStamp function, so you could try something like this - toTimestamp('2021-04-12 04:53:00.000', 'yyyy-MM-dd HH:mm:ss.SSS') in your code to convert it to required format.

In order to use Time format like toTimestamp($start_date, 'yyyy-mm-dd\'T\'HH:mm:ss') your input date format should be 2021-04-12T04:53:00.
Please refer to the highlighted text -
92364-image.png

Please refer to the documentation. I have also checked this in my environment it works as expected. (See the below gif)
92423-dataflow-datetimeformat.gif

Please let me know if you have any other questions.


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.



image.png (48.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @singhkumardeepesh-9270,

Please let me know if you find above reply useful. If yes, do click on 'Mark as answer' link in above reply. This will help other community members facing similar query to refer to this solution.

Thanks.
Saurabh

0 Votes 0 ·
JogalekarMandar-9214 avatar image
0 Votes"
JogalekarMandar-9214 answered

Were you able to solve this issue. I am trying to apply Filter By Last Modified in Data Flow where source is Azure Data Lake .

It seems the filtering logic is getting passed and all files are being retrieved.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.