What is this pipeline error in Azure Synapse Analytics?

SUNG MIN KIM 1

Run ID : 248db404-306f-4067-bfbc-01743d64212f

It works well until 2021.02.22 07:30 by daily trigger. But I triggered manually on 2021.02.22 18:00, and occurred an error this time.

What is problem with it?
Data type or Blob setting?

Here is Error Detail.

{
"errorCode": "2200",
"message": "ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please make sure SQL DW has access to ADLS Gen2 account,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,Source=.Net SqlClient Data Provider,SqlErrorNumber=107090,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=107090,State=1,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,},],'",
"failureType": "UserError",
"target": "----------------------------------",
"details": []
}

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-23T08:33:48.34+00:00

Hi @SUNG MIN KIM ,

Welcome to Microsoft Q&A and thanks for your query.

Could you please confirm if you are noticing the error consistently when you try to run the pipeline with manual trigger? If so, please confirm if you were able to manual trigger this pipeline without any issue previously or is this the first time you tried manual trigger?

Are you experiencing this issue only with manual trigger or do you also observe the same behavior with daily automated trigger?

Also please do share the complete error message which your are seeing.

We look forward to your response.

Thank you
SUNG MIN KIM 1 Reputation point

2021-02-23T08:57:04.463+00:00

Our customer uses Oracle DB for source. And we import Data with azure.
They only did insert data in the DB between Daily automated trigger and Manual trigger.
A Manual trigger was performed after their INSERT operation.

It occurs both manual and automated trigger.

Sorry, but Do you mean full target table name?
I wrote complete error message and also Run ID.

Can't I know the reason type with that message?
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-02-25T07:09:27.593+00:00

Hi @SUNG MIN KIM ,

Thanks for the details. We are looking into it and will get back to you as soon as we have an update.

Thank you for your patience
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-04T07:48:08.063+00:00

Hi @SUNG MIN KIM ,

Apologizes for the delay in my response. Your issues seems to be exactly the same as this issue : https://learn.microsoft.com/answers/questions/201623/not-enough-columns-in-this-line.html

Could you please confirm what is your source and sink? Also please please confirm if you are using Polybase? Are you noticing this issue consistently or only with a particular set of data?

As discussed in other thread the error message could be misleading but the actual issue might be with the data which was processed during this run failure.

Please let us know the above information so that we can escalate to internal team for further analysis.

Thank you
SUNG MIN KIM 1 Reputation point

2021-03-04T07:54:41.06+00:00

I don't mind! I've already found what is problem.

I think new line character like chr(13) or chr(10) causes problem in this case.

I deleted it with query, but should I have to delete all characters in all column?

Fortunately, In my case, It causes only one column in one table.
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-04T08:03:58.243+00:00

Hi @SUNG MIN KIM ,, Glad to know that you were able to figure out the root cause and thanks for sharing your findings. Yes, incase if you notice the new line characters in other columns, then you might end up with same issue again. It is better to remove those characters to avoid the issue.

Thanks