question

PatrickSteiner-3862 avatar image
PatrickSteiner-3862 asked ·

Data Factory Pipeline Copy Activity (Cosmos DB - Mongo API) does not run

A simple data copy activity pipeline in Azure Data Factory to copy collection data from one Cosmos DB Account (Mongo API) to another Cosmos DB Account (Mongo API) does fail with the following error:

 {
     "errorCode": "2200",
     "message": "'Type=Microsoft.Azure.Documents.DocumentClientException,Message=, Request URI: /, RequestStats: , SDK: documentdb-dotnet-sdk/2.5.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,'",
     "failureType": "UserError",
     "target": "dev_copy_mongo_subscriptions",
     "details": []
 }

in Output:

 {
     "dataRead": 44294,
     "dataWritten": 0,
     "sourcePeakConnections": 1,
     "sinkPeakConnections": 1,
     "rowsRead": 12,
     "rowsCopied": 0,
     "copyDuration": 5,
     "throughput": 8.651,
     "errors": [
         {
             "Code": 11000,
             "Message": "'Type=Microsoft.Azure.Documents.DocumentClientException,Message=, Request URI: /, RequestStats: , SDK: documentdb-dotnet-sdk/2.5.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,'",
             "EventType": 0,
             "Category": 5,
             "Data": {},
             "MsgId": null,
             "ExceptionType": null,
             "Source": null,
             "StackTrace": null,
             "InnerEventInfos": []
         }
     ],



azure-data-factoryazure-cosmos-db
2 comments
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Additional information:
The connections to the Cosmos DB Accounts works and delivers information as expected.

0 Votes 0 · ·
Mike-Ubezzi-MSFT avatar image Mike-Ubezzi-MSFT PatrickSteiner-3862 ·

Thank you for opening a support request. I was unable to find anything pertaining to errorCode 2200 and code 11000. There is an issue writing to the configured collection as a connection can be made to the target MongoDB API instance, otherwise you would have seen a connection error. Please update this post with the root cause as identified by support.

1 Vote 1 · ·
HimanshuSinhamfst-5269 avatar image
HimanshuSinhamfst-5269 answered ·

Hello @PatrickSteiner-3862 ,

I am aware that you have raised a request with the MS support , but I did try to copy a very simple document with zipcode and it worked fine for me . The error which you have pasted does not help a lot . The below clip may help you understand what I am doing .


8011-adf-ui.gif



adf-ui.gif (1.6 MiB)
5 comments Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @HimanshuSinhamfst-5269

I tried it again with a new collection - with just 2 simple documents - but I get the same error as already reported.

So even with a simple pipline as you showed, I get the same error.

Microsoft Support did not respond until now

Thank you for your suggestion

Patrick Steiner

0 Votes 0 · ·
Mike-Ubezzi-MSFT avatar image Mike-Ubezzi-MSFT PatrickSteiner-3862 ·

From the log files provided Azure Support, you are using Key Vault to store the access key for Cosmos DB for MongoDB API and that is the difference here. Can you confirm @PatrickSteiner-3862 this is the case? You clearly were able to connect to the source instance but the issue is with the target instance....

0 Votes 0 · ·

Hi @Zagato36
We use Key Vault Secrets to store the connection string for the target MogoDB API. The following I tried: a) Connected a DataSet to the target Mongo DB to a collection, where I manually added documents. Within the dataset I could see those documents in DF. So the connection itself works as expected. I do not have an error, when testing the connection anyway. b) I tried some time ago to access the destination Mongo DB directly using the connection string - did not help, same error. So I am not sure if the problem is within the target! Other differences between Source and Target: Version of Cosmos DB Account, Source Collection limited to 10GB with Throughput Limits on Collection level, in Destination there is no storage limitation with RU/s Limits on Mongo DB Level. So in target collection - there is now a shard key we have to handle. Meanwhile we migrated/copied the documents with almost no problems using the "Studio 3T" tool - so theoretically it should work.

0 Votes 0 · ·
Show more comments
PrithviChawla-4472 avatar image
PrithviChawla-4472 answered ·

I am facing the similar issue while trying to migrate data between two different subscriptions.
I am using Key vault to store the access keys.
The error message I am getting is -

Operation on target Copy data1 failed: 'Type=Microsoft.Azure.Documents.DocumentClientException,Message=, Request URI: /, RequestStats: , SDK: documentdb-dotnet-sdk/2.5.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,'

Is there any resolution yet? @Mike-Ubezzi-MSFT

Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PatrickSteiner-3862 avatar image
PatrickSteiner-3862 answered ·

Hi @PrithviChawla-4472
Yes - there is a solution provided by MS Support:

Symptom:
Pipeline with Copy Activity in context of Cosmos DB Accounts (mongo) does not work
Cause:
ADF Not fully compatible with Mongo API 3.6
Resolution:
Change the host name in connection string from ‘.mongo.cosmos.azure.com’ to ‘.documents.azure.com' and then retry it.

After I adapted the connection strings stored in the Key Vault as shown above, it started to work.

Hope this helps

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It worked! Thanks :)

0 Votes 0 · ·
PrithviChawla-4472 avatar image
PrithviChawla-4472 answered ·

@PatrickSteiner-3862 Thanks, it worked by making this change.

Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.