ADF Copy Activity "Enable logging" Setting

Casey Choe 0

Hi, I've followed the instructions to obtain the logs for a "Copy data" activity (Quickbooks to Azure Database in Postgresql). There are 2 errors I want to log:

One was this error, from ADF activity run:
Failure happened on 'Sink' side. 'Type=Npgsql.PostgresException,Message=22003: numeric field overflow,Source=Npgsql,'

But the log file that is returned only has headers and this resource says that means no errors were found?

The other was this error, from ADF activity run:
ErrorCode=UserErrorFailedToConnectOdbcSource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400

ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400

ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400,Source=,'

Both activities have this setting enabled in this way:
User's image

The log files are found in my storage account but the content always only returns:

Timestamp,Level,OperationName,OperationItem,Message

I understand what the error messages are saying, but I want a more detailed log of the activity/API requests (more than the ADF Input and Output logs) so I can troubleshoot the errors effectively. Any idea what's going on with the log files or if I can obtain the logs I want in a different way?

phemanth 5,840 Reputation points Microsoft Vendor

2024-04-22T05:03:37.08+00:00
@Casey Choe

Thanks for reaching out to Microsoft Q&A

You're right, an empty log file with headers only signifies no errors were logged within the copy activity itself. Here's what might be happening and how to get the detailed logs you need:

Understanding the Log File:

The log file captures information about the copy activity execution, not detailed data movement or API requests.

Troubleshooting the Errors:

QuickBooks API Connection Error: (ErrorCode=UserErrorFailedToConnectOdbcSource)

This points to an issue with the connection to QuickBooks. Check the following: QuickBooks Credentials: Ensure your QuickBooks access tokens or credentials used in the connection are valid and haven't expired. Request Format: Review the data you're trying to copy and verify it adheres to the QuickBooks API format requirements. Firewall Restrictions: Make sure there are no firewall rules blocking communication between ADF and QuickBooks.

Postgres Numeric Overflow Error: (Type=Npgsql.PostgresException)

This indicates data being copied exceeds the defined numeric data type in your Azure Database for PostgreSQL table. Modify Schema: If possible, alter the table column data type in your database to accommodate larger values. Data Validation: Consider implementing data validation logic in your ADF pipeline to check for potential overflow issues before copying.

Getting Detailed Logs:

ADF Monitor:

Navigate to the ADF monitor within the Azure Data Factory portal.

Select your pipeline run and drill down to the "Copy Data" activity.

You might find more detailed error messages under the "Inputs", "Outputs", and "Settings" tabs.

While configuring the copy activity in ADF, explore options for enabling verbose logging. This might vary depending on the source and sink types you're using.

Consider utilizing external logging tools like Application Insights or Azure Monitor to capture detailed logs during the data transfer process.

These tools can provide a more granular view of API requests and responses, aiding in troubleshooting.

Refer to the similar post you linked for further insights: https://learn.microsoft.com/en-us/answers/questions/665127/adf-copy-activity-lof-file-is-empty-after-enabled

Hope this helps. Do let us know if you any further queries.
phemanth 5,840 Reputation points Microsoft Vendor

2024-04-23T06:40:16.0833333+00:00

@Casey Choe We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.