question

$$ANON_USER$$ avatar image
0 Votes"
$$ANON_USER$$ asked KevalVankudre-2595 commented

Connecting Sharepoint Document Library to ADF via managed identity

Hello there,

I'm trying to implement a solution migrate SharePoint Document library to Azure Blob Storage via Data Factory(only). I have come across multiple solutions which implement logic apps or either postman. I also came across one solution in Microsoft documents here:
https://docs.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list
However, in this scenario, for web activity Body field expects details like client ID, client secret mentioned in plain text. And instead that to be done is there any alternative wherein we do not need these details to be exposed or can use managed identity in the web activity to fetch token.

Any help on this would be appreciated!

@KranthiPakala-MSFT I came across your solution to this and need a little tweak to that.

azure-data-factory
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@KranthiPakala-MSFT can you please help on this

0 Votes 0 ·

Hi anonymous user,

Welcome to Microsoft Q&A forum and thanks for reaching out.

I haven't tried with managed identity but let me double check if it is possible or not and will get back to you soon.

Thank you for your patience.

0 Votes 0 ·

Hello @KranthiPakala-MSFT ,
My main query here is, after following the above mentioned link, it expects the client credentials to be written in body part. I need a workaround to that specific part for my solution to be implemented in my case. So if you can help me with that specific part, would be really appreciated.

0 Votes 0 ·

Hello @KranthiPakala-MSFT, Exploring more into this, I found a way to access filenames via Microsoft Graph. However, these details are in JSON format. So my query here is, how can we fetch the file details from this JSON format and further copy those files to azure. Can you please guide me through this. It would really be helpful. ![97199-json.png][1] [1]: /answers/storage/attachments/97199-json.png

0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KevalVankudre-2595 commented

Hi anonymous user,

Thanks for clarifying the ask. As per the current ADF SharePoint connector limitation, it uses service principal authentication to connect to SharePoint. This is limitation by design at the moment. Hence managed identity not an option

92657-image.png

My main query here is, after following the above mentioned link, it expects the client credentials to be written in body part. I need a workaround to that specific part for my solution to be implemented in my case. So if you can help me with that specific part, would be really appreciated.

In order to avoid exposing client credentials in the body of the web activity while getting the access token, you can store the value of the client credentials in Azure Key Vault. Then have another web activity before GetBearerToken web activity to retrieve the client credentials from Key Vault using Get Secret API - GET {vaultBaseUrl}/secrets/{secret-name}/{secret-version}?api-version=7.1

92733-image.png

92630-image.png

And use output.value of that web activity in body of the GetBearerToken web activity using a dynamic expression as below. This way you can avoid exposing the client credentials in your ADF web activity.

92687-image.png

Hope this info helps. Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



image.png (25.6 KiB)
image.png (10.9 KiB)
image.png (51.6 KiB)
image.png (113.6 KiB)
· 26
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @KranthiPakala-MSFT,

Above solution works perfectly fine.
Further my query is, I want to copy whole folder from SharePoint Document Library as it is to Azure Delta Lake Gen 2. And the SPO library contains mixed file formats of which I will need only .xlsx files. I tried the document link steps as above, but it copies file as BLOB. How do I copy excel files as it is from SharePoint.

Thanks a lot for all the support.

0 Votes 0 ·

Hello @KranthiPakala-MSFT,

Can you please guide me through this.

0 Votes 0 ·

Hi anonymous user

Thanks for getting back and sorry for the delay. As per my understanding you would want to copy the whole folder (but just the xlsx files in it) from SharePoint Document Library to ADLS Gen2 (So you want to keep the folder structure as is with just the xlsx files) , please confirm so that I can double check on it.

Also could you please clarify on this statement "I tried the document link steps as above, but it copies file as BLOB. How do I copy excel files as it is from SharePoint." ?

Thanks

0 Votes 0 ·
Show more comments