Synapse Scala | Read Cosmos DB Container data using managed identity

AJITH KUMAR RAI 0

Hi Team,

I tried reading Cosmos data using Managed Identity inside a Synapse Scala notebook, but it's not working as expected.

Already provided access for synapse workspace by using below command

New-AzCosmosDBSqlRoleAssignment -ResourceGroupName <Resourcegroupname> -AccountName <CosmosAccountName> -RoleDefinitionName "Cosmos DB Built-in Data Contributor" -PrincipalId <WorkspaceMangedidentity> -Scope "/"

Please find below steps and error message we are getting.

I created linked service as shown below screen.

User's image

as per this article: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark-3

I created below code block inside my notebook

val linkedServiceName = "MSICosmosDbNoSql1"
val secretlinkedsvc="SecretCosmosDbNoSql1"
val containerName = "Items"

    val df = spark.read.format("cosmos.olap")
      .option("spark.synapse.linkedService", linkedServiceName)
      .option("spark.cosmos.container", containerName)
      .load()

But it is not working with linked service: MSICosmosDbNoSql1

And it is working with SecretCosmosDbNoSql1

Here we want to connect by using Managed identity only.

Just for info: We also Enable Azure Synapse Link at Cosmos side.

Error Message:

Caused by: com.microsoft.azure.synapse.TokenServiceClientResponseStatusException: Token Service returned 'Client Error' (400), with message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=AuthTypeNotSupported, Target=MSICosmosDbNoSql1, Message=Linked Service Auth Type not supported]. TraceId : da03309d-ee99-4b1e-8cfd-f39f73dd3b6f | client-request-id : a363ee69-ab48-487d-abda-6e2e9de40cc8. Error Component : LSR"}

Smaran Thoomu 10,715 Reputation points Microsoft Vendor

2024-05-02T12:01:33.4066667+00:00
Hi @AJITH KUMAR RAI

Thank you for reaching out to the community forum with your query.

Based on your query, it seems that having trouble reading Cosmos DB container data using managed identity inside a Synapse Scala notebook. And you have already provided access for the Synapse workspace using the Azure PowerShell command, but still encountering an error message.

To resolve this issue, you can try the following steps:

Ensure that the managed identity has been granted the necessary permissions to access the Cosmos DB account. You can check this by going to the Cosmos DB account in the Azure portal, selecting "Access Control (IAM)" from the left-hand menu, and verifying that the managed identity has been granted the "Cosmos DB Built-in Data Contributor" role.

Verify that the Synapse workspace has been granted access to the Cosmos DB account. You can check this by going to the Cosmos DB account in the Azure portal, selecting "Firewalls and virtual networks" from the left-hand menu, and adding the Synapse workspace's virtual network to the allowed networks.

Ensure that the linked service in the Synapse workspace has been configured correctly. You can check this by going to the Synapse workspace in the Azure portal, selecting "Linked services" from the left-hand menu, and verifying that the linked service for Cosmos DB has been configured with the correct authentication type (Managed Identity).

Verify that the container name specified in the Scala code block matches the actual container name in the Cosmos DB account.

If the issue still persists, you can try reaching out to Azure support for further assistance.

Hope this helps. Do let us know if you any further queries.
AJITH KUMAR RAI 0 Reputation points

2024-05-03T06:10:43.2566667+00:00

Thank you for the response.

In IAM, the new role is not visible even after the successful execution of the role assignment command. same behavior with other UPNs might be because of these are internal roles.

However, I am able to connect to linked services with MSI, and I can use the linked service with the dataset without any issues and able to view data.

But when I try to read data with a notebook, I am unable to do so.
Smaran Thoomu 10,715 Reputation points Microsoft Vendor

2024-05-03T06:58:33.2133333+00:00
@AJITH KUMAR RAI Thank you for providing additional information. Based on your response, it seems that the role assignment command was executed successfully, but the new role is not visible in the IAM section of the Cosmos DB account. This could be due to replication delays or other issues.

Since you are able to connect to the linked services with MSI and view data without any issues, it's possible that the issue is related to the Scala code block in the notebook. Also When trying to read data with a notebook, the user must have the appropriate access to the notebook file in addition to the Synapse workspace. Here are some steps you can take to troubleshoot the issue:

Verify that the container name specified in the Scala code block matches the actual container name in the Cosmos DB account.

Check the Spark logs for any error messages related to the Scala code block. You can access the Spark logs by going to the Synapse workspace in the Azure portal, selecting "Apache Spark pools" from the left-hand menu, and selecting the Spark pool that the notebook is running on. Then, select "Logs" from the top menu to view the Spark logs.

Try running the Scala code block with a different linked service to see if the issue is specific to the "MSICosmosDbNoSql1" linked service. You can create a new linked service with the same configuration as "MSICosmosDbNoSql1" and test the Scala code block with the new linked service.

If the issue persists, you can try reaching out to Azure support for further assistance.

Hope this helps. Do let us know if you any further queries.
AJITH KUMAR RAI 0 Reputation points

2024-05-04T05:56:44.1366667+00:00
Tried every approach and in Synapse logs it is clearly saying that AuthTypeNotSupported in NoteBook.

Please see below steps I followed.

Step 1: Given permission for Synapse work space over Cosmos DB.

Step 2: Created linked services as mentioned below.

Step 3: Created integration dataset with MSI linked service and successfully able to load container data.

Step 4: Now I am trying same approach in Note Book with both linked services.

val msilinkedServiceName = "MSICosmosDbNoSql1" val keylinkedServiceName="SecretCosmosDbNoSql1" val containerName = "Items" val df = spark.read.format("cosmos.olap") .option("spark.synapse.linkedService", msilinkedServiceName) .option("spark.cosmos.container", containerName) .load()

Here by using keylinkedServiceName it is loading data into df Where as msilinkedServiceName getting AuthTypeNotSupported as mentioned below.

I am running this notebook via a pipeline and with a trigger, not in debug mode.

Caused by: com.microsoft.azure.synapse.TokenServiceClientResponseStatusException: Token Service returned 'Client Error' (400), with message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=AuthTypeNotSupported, Target=MSICosmosDbNoSql1, Message=Linked Service Auth Type not supported]. TraceId : da637472-6768-4bf7-bc90-23f6545aeb97 . Error Component : LSR"}

As per this ticket: Old-Ticket, it seems that this functionality is not supported.

Is there any workaround for this, or is this functionality supported?

1 answer

Smaran Thoomu 10,715 Reputation points Microsoft Vendor

2024-05-06T07:46:28.3866667+00:00
@AJITH KUMAR RAI Thank you for providing additional information. Based on your response, it seems that you have already tried several approaches to resolve the issue, but you are still encountering the "AuthTypeNotSupported" error when trying to read data with the notebook using the MSI linked service.

It's possible that the "AuthTypeNotSupported" error is caused by a limitation in the current implementation of Synapse Analytics. According to the ticket you mentioned, this functionality may not be supported at this time.

However, there are a few workarounds you can try:

Use a different authentication method: Instead of using MSI, you can try using a different authentication method, such as a shared access signature (SAS) token or a connection string. You can create a new linked service with the appropriate authentication method and test the Scala code block with the new linked service.

Use a different notebook: You can try using a different notebook to see if the issue is specific to the current notebook. You can create a new notebook and test the Scala code block with the new notebook.

Reach out to Azure support: If the issue persists, you can try reaching out to Azure support for further assistance. They may be able to provide additional insights or workarounds to help you resolve the issue.

I hope this helps. Do let us know if you any further queries.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Synapse Scala | Read Cosmos DB Container data using managed identity

1 answer