ADF Copy Data Logs Issue

Casey Choe 0

Hi, I've followed the instructions to obtain the logs for a "Copy data" activity (Quickbooks to Azure Database in Postgresql). There are 2 errors I want logs for:

(1) One was this error, from ADF activity run: Failure happened on 'Sink' side. numeric field overflow

(2) The other was this error, from ADF activity run: API Connection Failed. Bad Request.

Both activities have this setting enabled in this way:

Enable logging: true

Storage connection name: [my storage account name]

Logging level: Info

Logging mode: Best effort

The log files are found in my storage account but the content for all the log files is just: Timestamp,Level,OperationName,OperationItem,Message

Answers from other forum posts have said that means that there are no errors to log, which is obviously untrue.

I understand what the error messages are saying, but I want a more detailed log of the activity/API requests (more than the ADF Input and Output logs) so I can troubleshoot the errors effectively. Any idea what's going on with the log files or if I can obtain the logs I want in a different way?

Vinodh247-1375 11,211 Reputation points

2024-04-20T11:06:24.29+00:00

did you try most of these listed here and nothing worked for you? or do you think that you need more deeper insights that all of these offer?

https://learn.microsoft.com/en-us/azure/data-factory/monitor-data-factory
Harishga 3,430 Reputation points Microsoft Vendor

2024-04-22T12:19:01.65+00:00

@Casey Choe
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Casey Choe 0 Reputation points

2024-04-22T14:27:42.86+00:00

@Vinodh247-1375 Yes I've tried the methods listed there and it wasn't quite the information I was looking for. I have solved error 1, so I am now only interested in error 2 (API Connection Failed. Bad Request.) I would like more insight into the API call itself that the Copy Data activity is performing, so I was hoping the monitor logging setting would help but I encountered the mentioned issue.
Vinodh247-1375 11,211 Reputation points

2024-04-23T04:52:27.3233333+00:00

Casey Choe:

This might help you. Please go through this accepted answer, this has all the points and fix discussed for your error # 2.

https://learn.microsoft.com/en-us/answers/questions/1247709/i-am-getting-a-badrequest-error-while-running-adf
Casey Choe 0 Reputation points

2024-04-23T15:26:54.05+00:00

@Vinodh247-1375 I don't think the link is relevant to my error. My error is a "Copy data" activity and the source is a linked service that uses a token refreshed daily. So that is why I want detailed logs on the API request that the "Copy data" is performing between the linked services.
Casey Choe 0 Reputation points

2024-04-23T15:28:58.3+00:00

@Vinodh247-1375 I don't think that is relevant to my error. I have a "Copy data" activity and the source is a linked service with tokens that refresh daily. So I want detailed logs on the API request / connection to the linked service so I can see the value of the token and understand why it is failing.
Harishga 3,430 Reputation points Microsoft Vendor

2024-04-24T10:39:51.52+00:00
Hi @Casey Choe

Based on the information you provided, it seems like you are looking for more detailed logs for your "Copy data" activity in Azure Data Factory (ADF). The logs you have obtained from your storage account only contain basic information such as timestamp, level, operation name, operation item, and message.

To obtain more detailed logs for your ADF activity, you can try the following steps:

Enable diagnostic settings for your ADF resource in Azure Monitor. This will allow you to collect detailed logs for your ADF activity, including input and output data, activity status, and errors. You can configure the diagnostic settings to send the logs to a storage account.

Use the Azure Data Factory REST API to retrieve detailed logs for your ADF activity. You can use the API to retrieve activity runs, activity run details, and activity run logs. The logs returned by the API include detailed information such as activity input and output data, activity status, and errors.

Regarding the specific errors you mentioned, "Failure happened on 'Sink' side. numeric field overflow" and "API Connection Failed. Bad Request," you can try the following troubleshooting steps:

For the "numeric field overflow" error, check the data types of the columns in your sink database table. Make sure that the data types are compatible with the data being copied from the source. You can also try increasing the precision or scale of the numeric columns in your sink table.

For the "API Connection Failed. Bad Request" error, check the connection settings for your API. Make sure that the API endpoint, authentication credentials, and other settings are correct. You can also try testing the API connection using a tool such as Postman to verify that the API is working correctly.

I hope this information helps you troubleshoot your ADF activity and obtain the detailed logs you need. Let me know if you have any further questions or concerns.