Not able mount azure fileshare in AKS pod

Question

question

Kailash-9216 asked Apr 29, '22 SaiKishor-MSFT commented May 11, '22

Not able mount azure fileshare in AKS pod

Hello there,
I am trying to mount an Azure fileshare onto AKS pod and seeing this error -

MountVolume.MountDevice failed for volume "somevol" : rpc error: code = Internal desc = volume(#secret#cloudfs#somevol-staging#somevol-staging) mount "//cloudfs.file.core.windows.net/cloudfs" on "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/somevol/globalmount" failed with mount failed: exit status 1 Mounting command: mount Mounting arguments: -t cifs -o dir_mode=0777,actimeo=30,mfsymlinks,file_mode=0777,<masked> //cloudfs.file.core.windows.net/cloudfs /var/lib/kubelet/plugins/kubernetes.io/csi/pv/somevol-staging/globalmount Output: mount error: could not resolve address for cloudfs.file.core.windows.net: Unknown error

I went through the diagnostics procedure @ https://github.com/Azure-Samples/azure-files-samples/tree/master/AzFileDiagnostics/Linux and tried to manually run the mount command on the node (Linux)

mount -t cifs //cloudfs.file.core.windows.net/cloudfs /testmnt -o vers=3.0,username=storageaccountname,password='password',dir_mode=0777,file_mode=0777,sec=ntlmssp

and I get the following error.

Unable to apply new capability set.

Any idea what could be the problem?

Thank you in advance for any help.

Best regards,
Kailash

azure-files

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Kailash-9216 · May 02 at 07:29 AM

2022-05-02T04:26:10.783Z Checking: Try with mounting share using SMB3.0
2022-05-02T04:26:10.786Z mount -t cifs //storageaccount.file.core.windows.net/jenkinsagentcloudfs /mnt/jenkinsagentcloudfs -o vers=3.0,username=stroageaccount,password='password',dir_mode=0777,file_mode=0777,sec=ntlmssp
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

0 ·

Answer 1 · 2022-04-30T00:13:15.89Z

shivapatpi-MSFT answered Apr 30, '22 Kailash-9216 published May 2, '22

Hello @Kailash-9216 ,
As per the mount error , It seems your POD is not able to resolve that fileshare account. If you are using CustomDNS can you kindly check the Custom DNS configuration whether the POD is able to reach out to that storage account file share ?

Also kindly check the address , i think the complete path should be https://cloudfs.file.core.windows.net
One more thing to check:- The secret which you have created which uses Storage Account Key - kindly double check if the Key was copied properly while creating the secret. (Try to recreate the secret)
Additional checks: Validate if your storage account is associated with a private link , if yes Private Link needs to have the association to cluster VNET.

Hope above validations will help out in resolving the issue.

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Kailash-9216 · May 02 at 07:26 AM

Hi Shiva,
I used the diagnostics script @ https://github.com/Azure-Samples/azure-files-samples/tree/master/AzFileDiagnostics/Linux and I am seeing the following error.

mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

I have included below the complete output. I believe the path is correct. I used the commands that Azure portal provides via the "Connect" option and getting the same error. I tried enabling access from all networks as well. Both the storage account and the AKS cluster are in the same region.

root@security-context-demo:/home/azure-files-samples-master/AzFileDiagnostics/Linux# ./AzFileDiagnostics.sh

2022-05-02T04:24:22.851Z Checking: Create a folder MSFileMountDiagLog to save the script output

2022-05-02T04:24:22.863Z Checking: Client running with "Ubuntu" version 18.04, kernel version is 5.4.0-1073-azure
Current Ubuntu distribution version is: 18.04
Current Kernel version is: 5.4.0

2022-05-02T04:24:22.879Z Checking: Check if the dependencies for this script are installed

2022-05-02T04:24:22.886Z Checking: All dependencies for the script are installed

2022-05-02T04:24:22.888Z Checking: Check if cifs-utils is installed
2022-05-02T04:24:22.896Z cifs-utils is already installed on this client

2022-05-02T04:24:22.899Z Checking: Check if keyutils is installed
2022-05-02T04:24:22.901Z Keyutils is already installed!

2022-05-02T04:24:22.903Z Checking: Check if client has at least SMB2.1 support
contd........

Regards,
Kailash

0 ·

Kailash-9216 Kailash-9216 · May 02 at 07:27 AM

2022-05-02T04:24:22.908Z System supports SMB2.1

2022-05-02T04:24:22.910Z Checking: Check if client has SMB 3 Encryption support
2022-05-02T04:24:22.916Z System supports SMB 3 Encryption

2022-05-02T04:24:22.919Z Checking: Check if client has been patched with the recommended kernel update for idle timeout issue
2022-05-02T04:24:22.924Z Kernel has been patched with the fixes that prevent idle timeout issues

2022-05-02T04:24:22.926Z Checking: Check if client has any connectivity issue with storage account
2022-05-02T04:24:22.928Z Type the storage account name, followed by [ENTER]:
storageaccount
2022-05-02T04:25:01.326Z Type the share name, followed by [ENTER]:
jenkinsagentcloudfs
2022-05-02T04:25:14.176Z Choose the Azure Environment:
1) azurecloud 3) azuregermancloud
2) azurechinacloud 4) azureusgovernment
Please enter your choice: 1
2022-05-02T04:25:17.757Z Storage account FQDN is sdlcmjenkinsaksstrgacct.file.core.windows.net
2022-05-02T04:25:17.760Z Getting the Iptables policies

2022-05-02T04:25:17.783Z Test the storage account IP connectivity over TCP port 445
2022-05-02T04:25:18.071Z Port 445 is reachable from this client.

2022-05-02T04:25:18.077Z Checking: Script has validated the client settings and do you want to map drive by script?
1) yes
2) no
Please enter your choice: 1

2022-05-02T04:25:35.289Z Checking: Do you want to turn on diagnostics logs
1) yes
2) no
Please enter your choice: 1

2022-05-02T04:25:38.829Z Checking: type the local mount point, followed by [ENTER]:
/mnt/jenkinsagentcloudfs

0 ·

Answer 2 · 2022-05-05T18:04:07.73Z

SaiKishor-MSFT answered May 5, '22

@Kailash-9216 Can you please refer to this Troubleshooting guide that talks about solutions for the error that you are facing i.e., Mount Error(13)-

Cause 1: Unencrypted communication channel
For security reasons, connections to Azure file shares are blocked if the communication channel isn't encrypted and if the connection attempt isn't made from the same datacenter where the Azure file shares reside. Unencrypted connections within the same datacenter can also be blocked if the Secure transfer required setting is enabled on the storage account. An encrypted communication channel is provided only if the user's client OS supports SMB encryption.

To learn more, see Prerequisites for mounting an Azure file share with Linux and the cifs-utils package.

Solution for cause 1
Connect from a client that supports SMB encryption or connect from a virtual machine in the same datacenter as the Azure storage account that is used for the Azure file share.
Verify the Secure transfer required setting is disabled on the storage account if the client does not support SMB encryption.

Cause 2: Virtual network or firewall rules are enabled on the storage account
If virtual network (VNET) and firewall rules are configured on the storage account, network traffic will be denied access unless the client IP address or virtual network is allowed access.

Solution for cause 2
Verify virtual network and firewall rules are configured properly on the storage account. To test if virtual network or firewall rules is causing the issue, temporarily change the setting on the storage account to Allow access from all networks. To learn more, see Configure Azure Storage firewalls and virtual networks.Please let us know if you have any further questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question