question

ShaikKhaja-1040 avatar image
0 Votes"
ShaikKhaja-1040 asked HimanshuSinha-MSFT answered

Feature request - Datafactory copy activity default copy option to be polybase at the backend.

Scenario:
1. In azure synapse analytics, built in copy activity uses polybase as default.
2. Pipeline in synapse with source as postgre and sink as synapse dedicated pool works fine with polybase as copy method.
3. However datafactory throws following exception, with same set up of polybase.
4. error:
Operation on target Copy data1 failed: ErrorCode=PolybaseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse. Operation: 'Polybase operation'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,Source=.Net SqlClient Data Provider,SqlErrorNumber=107090,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=107090,State=1,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,},],'

This denotes that, there is something in ADF polybase setting, which is uncommon to Synapse polybase setting.

Feature Request:
1. Can ADF engine use same polybase as in synapse.




azure-data-factoryazure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @ShaikKhaja-1040 ,
Thanks for the feedback and using Microsoft Q&A platform .We will pass on this feedback to the ADF team and share if we get any updates from them .
You can also pass the feedback from the ADF UI
135158-image.png


Thanks
Himanshu


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (26.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.