question

BoneyJacob-7901 avatar image
0 Votes"
BoneyJacob-7901 asked HarrySoehalim-5622 commented

Azure blob file download failed after downloading a few files

I tried to download blob files using BlobClient.downloadToFile(dest), but it failed after downloading a few files.

package com.blob.download;

import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
import com.azure.storage.blob.models.BlobItem;
import com.azure.storage.blob.models.ListBlobsOptions;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Iterator;

public class BlobDownload {
    public static void main(String args[]){
        BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
        //Specify storage uri
        String storageBaseUri = "";
        ///Specify connection string
        String connectionString = "";
        //Specify key
        String key = "";

        builder.connectionString(connectionString);
        BlobServiceClient client = builder.buildClient();

        if (!storageBaseUri.startsWith(client.getAccountUrl())) {
            throw new IllegalArgumentException(
                    "The given credential can not be used for the specified container.");
        }
        String containerName = storageBaseUri.replace(client.getAccountUrl() + "/", "");
        BlobContainerClient blobContainerClient = client.getBlobContainerClient(containerName);

        System.out.println("Downloading " + blobContainerClient.getBlobContainerName() + "/" + key);
        Iterator<BlobItem> keyspaceBlobs =
                blobContainerClient.listBlobs(new ListBlobsOptions().setPrefix(key + "/"), null).iterator();

        while (keyspaceBlobs.hasNext()){
            BlobItem blob = keyspaceBlobs.next();

            Path destFile = Paths.get("/opt/data/", blob.getName());


            try {
                Files.createDirectories(destFile.getParent());
            } catch (IOException e) {
                System.out.println(e);
            }

            blobContainerClient
                    .getBlobClient(blob.getName())
                    .downloadToFile(destFile.toString());
            System.out.println("Download file succeeded : " + destFile.toString());
        }

    }
}


We can simply reproduce this issue in the local environment while trying to download a 3GB file from blob storage.

package com.blob.download;

import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;


public class BlobDownload {
    public static void main(String args[]){
        BlobServiceClientBuilder builder = new BlobServiceClientBuilder();

        // specify storage uri
        String storageBaseUri = "";
        //Specify connection string
        String connectionString = "";
        //Specify 3 GB blob file path inazure
        String blobPath=""
        //Specify destination path
        String dest="/opt/data/md-388-big-Index.db"

        builder.connectionString(connectionString);
        BlobServiceClient client = builder.buildClient();

        if (!storageBaseUri.startsWith(client.getAccountUrl())) {
            throw new IllegalArgumentException(
                    "The given credential can not be used for the specified container.");
        }
        String containerName = storageBaseUri.replace(client.getAccountUrl() + "/", "");
        BlobContainerClient blobContainerClient = client.getBlobContainerClient(containerName);

        blobContainerClient
                .getBlobClient(blobPath)
                .downloadToFile(dest);

    }
}

Dependencies

nettyTcNativeVersion = '2.0.7.Final'

compile group: 'com.azure', name: 'azure-storage-blob', version: '12.4.0'
compile group: 'com.azure', name: 'azure-identity', version: '1.0.6'


I have got the following error message

Exception in thread "main" reactor.core.Exceptions$ReactiveException: java.io.IOException: Connection reset by peer
    at reactor.core.Exceptions.propagate(Exceptions.java:393)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
    at reactor.core.publisher.Mono.block(Mono.java:1678)
    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:99)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:563)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:488)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:457)
    at com.blob.download.BlobDownload.main(BlobDownload.java:57)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        ... 6 more
Caused by: java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1147)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)


Can you please guide me if there is any wrong usage in my implementation?




azure-blob-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Sumarigo-MSFT avatar image
0 Votes"
Sumarigo-MSFT answered HarrySoehalim-5622 commented

@BoneyJacob-7901 Welcome to Microsoft Q&A platform, Thank you for posting your query here!

Apologies for the delay in response here! I see you have posted the similar thread in the SO forum, Refer to the suggestion mentioned over-there and let me know if you have any question!

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.





· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Sorry, Issue still exists for me

0 Votes 0 ·

the only suggestion i can see is "download the file by blocks", which i have no clue as to what that is. If you can point to which class/method name, that will be great.

0 Votes 0 ·
AbhijeetRathod-5937 avatar image
0 Votes"
AbhijeetRathod-5937 answered

I am also facing the same issue any.
I am using Azure SDK.
Maven dependency as below:


 <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.9.0-beta.2</version>
 </dependency>


Please find below the stack trace:

 reactor.core.Exceptions$ReactiveException: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer at 
 reactor.core.Exceptions.propagate(Exceptions.java:393) at 
 reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97) at
  reactor.core.publisher.Mono.block(Mono.java:1678) at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:128) at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:728) at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:699) at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:630) at


If anyone is having any solution, please let me know.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.