Feedback Edit

multithread Module

  • Reference

High performance multi-threaded module to up/download

Calls method in core with thread pool executor to ensure the network is used to its maximum throughput.

Only implements upload and download of (massive) files and directory trees.

Classes

ADLDownloader

Download remote file(s) using chunks and threads

Launches multiple threads for efficient downloading, with chunksize assigned to each. The remote path can be a single file, a directory of files or a glob pattern.
ADLUploader

Upload local file(s) using chunks and threads

Launches multiple threads for efficient uploading, with chunksize assigned to each. The path can be a single file, a directory of files or a glob pattern.

Functions

get_chunk

Download a piece of a remote file and write locally

Internal function used by download.

get_chunk(adlfs, src, dst, offset, size, buffersize, blocksize, shutdown_event=None, retries=10, delay=0.01, backoff=3)

Parameters

adlfs
src
dst
offset
size
buffersize
blocksize
shutdown_event
default value: None
retries
default value: 10
delay
default value: 0.01
backoff
default value: 3

load 

load(filename)

Parameters

filename

merge_chunks 

merge_chunks(adlfs, outfile, files, shutdown_event=None, overwrite=False)

Parameters

adlfs
outfile
files
shutdown_event
default value: None
overwrite
default value: False

put_chunk

Upload a piece of a local file

Internal function used by upload.

put_chunk(adlfs, src, dst, offset, size, buffersize, blocksize, delimiter=None, shutdown_event=None)

Parameters

adlfs
src
dst
offset
size
buffersize
blocksize
delimiter
default value: None
shutdown_event
default value: None

save 

save(instance, filename, keep=True)

Parameters

instance
filename
keep
default value: True