Feedback Edit

DatalakeRESTInterface Class

  • Reference

Call factory for webHDFS endpoints on ADLS

Inheritance
builtins.object
DatalakeRESTInterface

Constructor

DatalakeRESTInterface(store_name=None, token=None, url_suffix='azuredatalakestore.net', api_version='2018-09-01', req_timeout_s=60, **kwargs)

Parameters

store_name
str
default value: None

The name of the Data Lake Store account to execute operations against.

token
dict
default value: None

from auth() or refresh_token() or other ADAL source

url_suffix
str(None)
default value: azuredatalakestore.net

Domain to send REST requests to. The end-point URL is constructed using this and the store_name. If None, use default.

api_version
str(<xref:2018-09-01>)
default value: 2018-09-01

The API version to target with requests. Changing this value will change the behavior of the requests, and can cause unexpected behavior or breaking changes. Changes to this value should be undergone with caution.

req_timeout_s
float(<xref:60>)
default value: 60

This is the timeout for each requests library call.

kwargs
<xref:<xref:optional arguments to auth>>
Required

See auth(). Includes, e.g., username, password, tenant; will pull values from environment variables if not provided.

Methods

call

Execute a REST call
is_successful_response
log_response_and_raise

call

Execute a REST call

call(op, path='', is_extended=False, expected_error_code=None, retry_policy=None, headers={}, **kwargs)

Parameters

op
str
Required

webHDFS operation to perform, one of DatalakeRESTInterface.ends

path
str
Required

filepath on the remote system

is_extended
bool(False)
default value: False

Indicates if the API call comes from the webhdfs extensions path or the basic webhdfs path. By default, all requests target the official webhdfs path. A small subset of custom convenience methods specific to Azure Data Lake Store target the extension path (such as SETEXPIRY).

expected_error_code
int
default value: None

Optionally indicates a specific, expected error code, if any. In the event that this error is returned, the exception will be logged to DEBUG instead of ERROR stream. The exception will still be raised, however, as it is expected that the caller will expect to handle it and do something different if it is raised.

kwargs
dict
default value: None

other parameters, as defined by the webHDFS standard and https://msdn.microsoft.com/en-us/library/mt710547.aspx

headers
default value: {}

is_successful_response 

is_successful_response(response, exception)

Parameters

response
Required
exception
Required

log_response_and_raise 

log_response_and_raise(response, exception, level=40)

Parameters

response
Required
exception
Required
level
default value: 40

Attributes

session

ends 

ends = {'APPEND': ('post', set(), {'syncFlag', 'offset', 'leaseid', 'filesessionid', 'append'}), 'CHECKACCESS': ('get', set(), {'fsaction'}), 'CONCAT': ('post', {'sources'}, {'sources'}), 'CREATE': ('put', set(), {'syncFlag', 'leaseid', 'filesessionid', 'write', 'overwrite'}), 'DELETE': ('delete', set(), {'recursive'}), 'GETCONTENTSUMMARY': ('get', set(), set()), 'GETFILESTATUS': ('get', set(), set()), 'LISTSTATUS': ('get', set(), {'listSize', 'listAfter'}), 'MKDIRS': ('put', set(), set()), 'MODIFYACLENTRIES': ('put', {'aclSpec'}, {'aclSpec'}), 'MSCONCAT': ('post', set(), {'deleteSourceDirectory'}), 'MSGETACLSTATUS': ('get', set(), set()), 'OPEN': ('get', set(), {'filesessionid', 'read', 'length', 'offset'}), 'REMOVEACL': ('put', set(), set()), 'REMOVEACLENTRIES': ('put', {'aclSpec'}, {'aclSpec'}), 'REMOVEDEFAULTACL': ('put', set(), set()), 'RENAME': ('put', {'destination'}, {'destination'}), 'SETACL': ('put', {'aclSpec'}, {'aclSpec'}), 'SETEXPIRY': ('put', {'expiryOption'}, {'expiryOption', 'expireTime'}), 'SETOWNER': ('put', set(), {'group', 'owner'}), 'SETPERMISSION': ('put', set(), {'permission'})}