Azure data factory AdlsGen1 structured link service fail to read file: Method not found

Jing Ou 56 Microsoft Employee

We create a AdlsGen1 structure stream linked service to read structure stream file. When we created the link service, clicked 'Test connection' and it succeeded, but it failed to preview stream and the error was 'Type=System.MissingMethodException,Message=Method not found: 'System.Threading.Tasks.Task`1<Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationResult> Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(System.String, Microsoft.IdentityModel.Clients.ActiveDirectory.ClientAssertionCertificate)'.,Source=Microsoft.Rest.ClientRuntime.Azure.Authentication,'.

We use service principal for authentication and use cert as the SPI secret. Specifically, we create a cert in azure key vault and bind the cert to the SPI, and input the cert name as secret name in ADF. In other linked services it works well (including ADLS gen2 structure stream linked service). But for Adls gen1, it throws exception as shown before. Could you please have a look?

Harishga 4,250 Reputation points Microsoft Vendor

2024-05-08T07:00:25.7766667+00:00
Hi @Jing Ou
Welcome to Microsoft Q&A platform and thanks for posting your question here.

The error you’re seeing with the ADLS Gen1 structure stream linked service in Azure Data Factory seems to be related to the AcquireTokenAsync method from the Microsoft.IdentityModel.Clients.ActiveDirectory namespace.

Missing Method Exception: The error message indicates that the AcquireTokenAsync method expected by your application is not found. This could be due to a mismatch in the versions of the libraries you are using.

Service Principal Authentication: You mentioned using a service principal with a certificate as the secret. Ensure that the certificate is correctly configured in Azure Key Vault and that the service principal has the necessary permissions.

ADLS Gen1 vs Gen2: It’s noted that while other linked services, including ADLS Gen2, work well, ADLS Gen1 is throwing an exception. This might suggest a specific issue with ADLS Gen1 compatibility or configuration.

To troubleshoot the error encountered with the ADLS Gen1 structure stream linked service in Azure Data Factory (ADF), you can follow these steps:

Check the version of Microsoft.IdentityModel.Clients.ActiveDirectory you're using in your project to ensure compatibility with the rest of your code and other libraries. ADLS Gen1 may require a different version of the library compared to ADLS Gen2.

Consider updating to the latest version of the library if you're using an older version, as the method may have been added in a later version.

Review the configuration of your Azure Data Factory and compare the linked service configuration for ADLS Gen1 with that of ADLS Gen2. There may be differences in how the service principal and certificate are being used.

Verify that the service principal has the correct permissions set up in Azure and that the certificate is properly bound to the service principal.

Ensure that the certificate in Azure Key Vault is correctly configured and that the secret name you're using in ADF matches the name of the certificate.

I hope this information helps you. Let me know if you have any further questions or concerns.
Jing Ou 56 Reputation points Microsoft Employee

2024-05-09T01:52:22.86+00:00

Hi @Harishga Thanks for your response. But there is no way for me to configure the versions of libraries in ADF. Maybe I just don't know how to configure it in ADF, currently what I know is that I only need to fill in some fields such as account name, service principal id and etc. I suppose it's the ADF backend service misconfigure the version of library. Is there anyway for me to workaround it, or do you know anyway to reach out to ADF team? Thanks!
Harishga 4,250 Reputation points Microsoft Vendor

2024-05-09T07:08:33.6133333+00:00

Hi @Jing Ou
To work around this issue, you can try updating the Microsoft.IdentityModel.Clients.ActiveDirectory library version used in your code to match the one used by the ADF backend service. You can find the version used by the ADF backend service in the Azure Data Factory release notes.

If updating the library version does not resolve the issue, you can reach out to the ADF team by creating a support ticket in the Azure portal. To create a support ticket, go to the Azure portal, select your ADF instance, and click on the "Support + troubleshooting" tab. From there, you can create a new support ticket and describe the issue you are facing. The ADF team will then work with you to resolve the issue.

I hope this information helps you. Let me know if you have any further questions or concerns.
Roi Bing 5 Reputation points Microsoft Employee

2024-05-24T16:09:16.21+00:00

@Jing Ou I have the same issue as this, did you finally get a work around?

@Harishga I am setting up the Linked Service within ADF only, so there is no code involve. Looks like the issue is specific to the "Azure Data Lake Storage Gen1 for Cosmos Structured Stream" backend service, that needs the Client.ActiveDirectory library to be upgraded to latest version. Can we get ADF team involve for this? Seeing multiple people has faced the same issue now.

1 answer

Jing Ou 56 Reputation points Microsoft Employee

2024-05-27T01:51:32.56+00:00

@Roi Bing I created a support ticket to ADF team and they respond to me that it's a known issue and still in investigation and no ETA for it.

Regarding to the workaround, you can use managed identity to link to Cosmos Structure Stream. Specially, create a managed identity -> grant the managed identity permission in Azure portal (at least Reader) -> in ADF, use Credential to authenticate, bind the managed identity to the Credential.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Share via

Azure data factory AdlsGen1 structured link service fail to read file: Method not found

1 answer