question

EvanWellens-2911 avatar image
0 Votes"
EvanWellens-2911 asked EvanWellens-2911 commented

Adding datetime field to json-> mongodb pipeline

We have several pipelines in azure data factories that are mapping json documents in blob storage to documents in mongodb. Now we need some of these to TTL. My understanding is they need an index with a special attribute in order to do so which is of date time. Is there a way to add this field during the pipeline execution ? Thanks in advance for any advice

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered EvanWellens-2911 commented

Hello @EvanWellens-2911 ,
Thanks for the ask and using the Microsoft Q&A platform .
I am assuming that you are using copy activity and if thats the case , please try out adding an addionalcolumn .

https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy

I think you can use the utcnow() as an expression and it should work out .

updated on 9/14

I did tried the below expression and I think it should work .

@formatDateTime(utcnow(), 'yyyy-dd-MM-hh-mm-ss-sss')


132100-image.png

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


utcnow()



image.png (65.9 KiB)
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Many thanks for your suggestion!!

Interesting I was not aware of that being available at the source. I have a data mapping at the sync, I do not see the new column show up or available.

0 Votes 0 ·

Otherwise would there be a way that an existing colunm which is a date time but is being handled as a string be converted ? You are correct I am using copy activity.

0 Votes 0 ·

Ok I figured out how to get it into the output data mapping. But it's a string. I need it to be a real date time.

0 Votes 0 ·

Hello @EvanWellens-2911 ,
II have updated the answer , let me know if that helps .
Thanks
Himanshu

0 Votes 0 ·

Hello @EvanWellens-2911 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·
Show more comments