問題: ファイアウォールが有効になっていると Azure Data Lake Storage (ADLS) Gen1 にアクセスできないProblem: Unable to Access Azure Data Lake Storage (ADLS) Gen1 When Firewall is Enabled

問題点Problem

Azure 仮想ネットワーク (VNet) でファイアウォールが有効になっていて、ADLS Gen1 コネクタを使用して ADLS にアクセスしようとすると、次のエラーで失敗します。When you have a firewall enabled on your Azure virtual network (VNet) and you try to access ADLS using the ADLS Gen1 connector, it fails with the error:

328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError(Py4JJavaError:
An error occurred while calling o196.parquet.: java.lang.RuntimeException:
Could not find ADLS Token at com.databricks.backend.daemon.data.client.adl.AdlCredentialContextTokenProvider$$anonfun$get Token$1.apply(AdlCredentialContextTokenProvider.scala:18)
at com.databricks.backend.daemon.data.client.adl.AdlCredentialContextTokenProvider$$anonfun$get
Token$1.apply(AdlCredentialContextTokenProvider.scala:18)
at scala.Option.getOrElse(Option.scala:121)
at com.databricks.backend.daemon.data.client.adl.AdlCredentialContextTokenProvider.getToken(AdlCredentialContextTokenProvider.scala:18)
at com.microsoft.azure.datalake.store.ADLStoreClient.getAccessToken(ADLStoreClient.java:1036)
at com.microsoft.azure.datalake.store.HttpTransport.makeSingleCall(HttpTransport.java:177)
at com.microsoft.azure.datalake.store.HttpTransport.makeCall(HttpTransport.java:91)
at com.microsoft.azure.datalake.store.Core.getFileStatus(Core.java:655)
at com.microsoft.azure.datalake.store.ADLStoreClient.getDirectoryEntry(ADLStoreClient.java:735)
at com.microsoft.azure.datalake.store.ADLStoreClient.getDirectoryEntry(ADLStoreClient.java:718)
at com.databricks.adl.AdlFileSystem.getFileStatus(AdlFileSystem.java:423)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:94)

原因Cause

これは、ADLS Gen1 コネクタに関する既知の問題です。This is a known issue with the ADLS Gen1 connector.

ソリューションSolution

解決策としては、代わりにADLS Gen2を使用します。The solution is to use ADLS Gen2 instead.