Rotating ADE KEK, adds two new secrets (Wrapped BEKs) for the same VM

Question

question

VasanthaRamanA-7660 asked Mar 8, '22 SumanthMarigowda-MSFT commented Mar 24, '22

Rotating ADE KEK, adds two new secrets (Wrapped BEKs) for the same VM

Hello All,

We enabled ADE for OS+DATA disk for two VMs (RHEL8 - with no data disk attached) with same KEK, using
az vm encryption enable -n vmname -g rsgrp --key-encryption-key kek --disk-encryption-keyvault keyvault-name --volume-type ALL --encrypt-format-all

We tried rotating the KEK for the two VMs by calling the same command above again, now with a newer version of the same KEK. On doing so, we observed that after some time delay, two more secrets of type Wrapped BEKs in addition to existing ones are added for each VM (initially two Wrapped BEK secrets were present and after rotation of KEK it became six). We were expecting just one new secret. Unwrapped values of both the secrets were same. Now, even though the vm encryption status shows only one of those as the ADE secret (obtained by az vm encryption show -n vmname) . If we try to disable/delete the other newly obtained secret, the VM fails to start after a stop, citing "Failed to restart the virtual machine 'vmname'. Error: Error encountered when retrieving secret from the Key Vault with URL: https://kevaultname.vault.azure.net/secrets/secretname/secretversion. Make sure that the secret exists and Key Vault is enabled for volume encryption", even though the secret mentioned is very much present in the vault and is enabled.

Now, Questions are:

Is this a bug or the system is designed this way or maybe, the way we handle key rotation is wrong?
If it is designed this way, will we be charged for the secrets stored in the vault?

azure-disk-encryption

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SumanthMarigowda-MSFT · Mar 09 at 12:17 PM

@VasanthaRamanA-7660 Firstly apologies for the delay response! I am check on this issue internally I will get back to you with the concrete update!

Thanks for your time and co-operation!

0 ·

SumanthMarigowda-MSFT · Mar 24 at 10:13 AM

@VasanthaRamanA-7660 Just checking in to see if the above answer helped. If this answers your query, do click and for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

<<Private message note: This is a private message which only you as Original Poster and Microsoft Moderators can view. Please respond directly to this comment to privately share the requested information. For your privacy, please do not share any Personal Identifiable Information (PII) as a public or private comment. All the private messages on the platform will be deleted on periodic basis>>

,

0 ·

Answer 1 · 2022-03-17T07:29:50.533Z

SumanthMarigowda-MSFT answered Mar 17, '22 SumanthMarigowda-MSFT edited Mar 17, '22

@VasanthaRamanA-7660 Key rotation you should expect to see new secrets generated not only per VM but per volume, in some cases you will see new versions of the secret instead of new secrets, but at the end they are treated on a very similar way, the encryption settings on each drive will be updated to point to the new secrets.

If you so desire to clean up the KeyVault of unused secrets the recommendation is to check all the encryption settings of all the encrypted VMs and correlate with the current content of each specific KeyVault
so you can be sure that you are not deleting a secret or a particular version that is being used, then we strongly encourage you to take a backup of the secret just in case that a restore is needed,
after a secret deletion is a good idea to test out if the VM that was previously associated with it can safely be rebooted,
if not then the secret should be restored and then another case with us can be open to check the reason why the secret on the encryption settings was not updated.

We’re currently looking into the key rotation process and trying to identify any possible causes for your VMs to not start while deleting the secret that was not the one that is on the disk encryption settings for each individual disk.

About the costs, I checked and the information is publicly available here: https://azure.microsoft.com/en-us/pricing/details/key-vault/, the costs associated with secrets is based on secret operations, you can check the cost there by region and currency, if you need more details about that you can reach out to the sales team and they should be able to clarify.

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

screenshot-2021-12-10-121802.png (1.4 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 2 · 2022-03-10T07:18:25.247Z

SumanthMarigowda-MSFT answered Mar 10, '22 VasanthaRamanA-7660 commented Mar 10, '22

@VasanthaRamanA-7660 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

When you perform the Key rotation, a new secret will be added, and what it seems to be happening on a preliminary inspection is that you are getting one secret per volume type after the key rotation, the new secrets are associated with each individual disk and should not be deleted, that will prevent Azure to actually start the VM.

You can check which secret and key are associated with each individual disk by executing the commands posted here: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/how-to-verify-encryption-status#single-pass-1 the newly created secret and Key should always exists and be valid, enabled.

Currently the Key rotation process doesn’t delete the Keys nor secrets on the Key vault and that is to prevent breaking other resources that may be using that Key or Secret, You should be able to delete a secret that is not associated with any VM and that will not brake any VM or prevent it from booting.

Are you trying to delete the newly created secret and that would be reason for VMs are breaking.

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to and ” wherever the information provided helps you, this can be beneficial to other community members.

screenshot-2021-12-10-121802.png (1.4 KiB)

image.png (1.5 KiB)

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VasanthaRamanA-7660 · Mar 10 at 09:02 AM

@Sumarigo-MSFT , Hello and thank you for the response. The VMs in our case, has only OS disk, temporary disk and no additional data disks attached, and on performing the operation in the documentation you shared, we get only one disk listed.

Also, on first time encryption for keyvault, we get only one secret per VM. It is after KEK rotation, that we face more than one secret emerging per VM. Since the output shows only one disk, we are right in expecting one secret per VM right?

0 ·

image.png (56.9 KiB)

question