Databricks error: Failed to download bootstrap script from artifact storage

Question

Azure Firewall: Hub & Spoke architecture
Databricks: 7.4 runtime
Azure region: West Us2

I followed this data-exfiltration-protection-with-azure-databricks.html detailed walkthrough guide to setup protect data exfiltration with Azure Firewall and Databricks in our dev environment. After the setup, I was never be able to start my Databricks cluster.

Any help to resolve this error is very much appreciated.

Logs on the cluster

Cluster terminated. Reason: Storage Download Failure  
  
Failed to download bootstrap script from artifact storage. Please check that instances have connectivity to cloud storage.  
Instance bootstrap failed command: DownloadBootstrapScript  
Failure message: (Base64 encoded) VM extension code: ProvisioningState/succeeded  
----REDACTED----  
instanceId: InstanceId(3cf2fd8a96e5407cb868d636fdee9d48)  
workerEnv: workerenv-979924835046728  
Additional details (may be truncated):  
  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: arprodwestus2a1.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: arprodwestus2a3.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: dbartifactsprodwestus2.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: arprodwestusa14.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: arprodwestusa11.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] Command DownloadBootstrapScript finished. Storage Account: dbartifactsprodwestus.blob.core.windows.net [FAILED] [Will try other storage accounts]  
	[Bootstrap Event] {FAILED_COMMAND:DownloadBootstrapScript}  
	[Bootstrap Event] {FAILED_MESSAGE:(Base64 encoded) ----REDACTED----}  
	[Bootstrap Event] SELF BOOTSTRAP FAILED  
	[Bootstrap Event] Can reach arprodwestus2a1.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach arprodwestus2a3.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach dbartifactsprodwestus2.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach arprodwestusa14.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach arprodwestusa11.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach dbartifactsprodwestus.blob.core.windows.net: [SUCCEEDED]  
	[Bootstrap Event] Can reach westus.azuredatabricks.net: [SUCCEEDED]  

Firewall Application Rule Collection

Firewall Network Rule Collection

Answer 1

This issue is resolved now.

This linkudr was very useful in getting the job done.
I mean data exfiltration protection with Azure Databricks and Azure Firewall Premium.

That being said, working with the firewall was never an easy task.

Answer 2

@Porsche Me how did you solve this issue. I'm facing the same error and not sure how to move forward