This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 87%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
I have a ADF extracting 'activity' (activitypointer) from Dynamics 365 Online. The ADF runs for 60+ minutes and fails with the below error for the copy activity. The sink is a gen2 data lake. I have 'AutoResolveIntegrationRuntime' for this data factory. I can extract some other entities like 'Accounts' from CRM without issues. Appreciate the help to fix the issue. I have cross checked some other posts related to this type of ADF errors and this seems to be a different so creating a new thread.
Team, I don't find the related tags to associate this question to Azure Data Factory team and I tried different combination with no luck. Please add the necessary tags if you want the questions to be tagged correctly.
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Direct buffer memory\ntotal entry:19\r\njava.nio.Bits.reserveMemory(Bits.java:658)\r\njava.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)\r\njava.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)\r\norg.apache.parquet.hadoop.codec.SnappyCompressor.setInput(SnappyCompressor.java:97)\r\norg.apache.parquet.hadoop.codec.NonBlockedCompressorStream.write(NonBlockedCompressorStream.java:48)\r\norg.apache.parquet.bytes.CapacityByteArrayOutputStream.writeToOutput(CapacityByteArrayOutputStream.java:219)\r\norg.apache.parquet.bytes.CapacityByteArrayOutputStream.writeTo(CapacityByteArrayOutputStream.java:239)\r\norg.apache.parquet.bytes.BytesInput$CapacityBAOSBytesInput.writeAllTo(BytesInput.java:392)\r\norg.apache.parquet.bytes.BytesInput$SequenceBytesIn.writeAllTo(BytesInput.java:283)\r\norg.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:165)\r\norg.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:98)\r\norg.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:148)\r\norg.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:236)\r\norg.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:122)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:169)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.checkBlockSizeReached(InternalParquetRecordWriter.java:143)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:125)\r\norg.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:292)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchWriter.addRows(ParquetBatchWriter.java:61)\r\n.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Copy Activities",
"details": []
}
I am able to update the tag to 'Azure Data Factory'.
With an update to the Sink side compression from 'Snappy' to 'None', I was able to run the Copy Activity successfully. I still don't understand the relation between the compression and out of memory error the copy activity was throwing. Hope it helps someone else having a similar type of issue.