Migrate to granular role-based access for cluster configurations

We are introducing some important changes to support more fine-grained role-based access to obtain sensitive information. As part of these changes, some action may be required if you are using one of the affected entities/scenarios.

What is changing?

Previously, secrets could be obtained via the HDInsight API by cluster users possessing the Owner, Contributor, or Reader RBAC roles, as they were available to anyone with the */read permission. Going forward, accessing these secrets will require the Microsoft.HDInsight/clusters/configurations/* permission, meaning they can no longer be accessed by users with the Reader role. Secrets are defined as values that could be used to obtain more elevated access than a user's role should allow. These include values such as cluster gateway HTTP credentials, storage account keys, and database credentials.

We are also introducing a new HDInsight Cluster Operator role that will be able to retrieve secrets without being granted the administrative permissions of Contributor or Owner. To summarize:

Role Previously Going Forward
Reader - Read access, including secrets - Read access, excluding secrets
HDInsight Cluster Operator
(New Role)
N/A - Read/write access, including secrets
Contributor - Read/write access, including secrets
- Create and manage all of types of Azure resources.
No change
Owner - Read/write access including secrets
- Full access to all resources
- Delegate access to others
No change

For information on how to add the HDInsight Cluster Operator role assignment to a user to grant them read/write access to cluster secrets, see the below section, Add the HDInsight Cluster Operator role assignment to a user.

Am I affected by these changes?

The following entities and scenarios are affected:

API

The following APIs will be changed or deprecated:

  • GET /configurations/{configurationName} (sensitive information removed)
    • Previously used to obtain individual configuration types (including secrets).
    • This API call will now return individual configuration types with secrets omitted. To obtain all configurations, including secrets, use the new POST /configurations call. To obtain just gateway settings, use the new POST /getGatewaySettings call.
  • GET /configurations (deprecated)
    • Previously used to obtain all configurations (including secrets)
    • This API call will no longer be supported. To obtain all configurations going forward, use the new POST /configurations call. To obtain configurations with sensitive parameters omitted, use the GET /configurations/{configurationName} call.
  • POST /configurations/{configurationName} (deprecated)
    • Previously used to update gateway credentials.
    • This API call will be deprecated and no longer supported. Use the new POST /updateGatewaySettings instead.

The following replacement APIs have been added:

Azure HDInsight Tools for Visual Studio Code

If you are using version 1.1.1 or below, update to the latest version of Azure HDInsight Tools for Visual Studio Code to avoid interruptions.

Azure Toolkit for IntelliJ

If you are using version 3.20.0 or below, update to the latest version of the Azure Toolkit for IntelliJ plugin to avoid interruptions.

Azure Data Lake and Stream Analytics Tools for Visual Studio

Update to version 2.3.9000.1 or later of Azure Data Lake and Stream Analytics Tools for Visual Studio to avoid interruptions. For help with updating, see our documentation, Update Data Lake Tools for Visual Studio.

Azure Toolkit for Eclipse

If you are using version 3.15.0 or below, update to the latest version of the Azure Toolkit for Eclipse to avoid interruptions.

SDK for .NET

Versions 1.x and 2.x

Update to version 2.1.0 of the HDInsight SDK for .NET. Minimal code modifications may be required if you are using a method affected by these changes:

  • ClusterOperationsExtensions.GetClusterConfigurations will no longer return sensitive parameters like storage keys (core-site) or HTTP credentials (gateway).

    • To retrieve all configurations, including sensitive parameters, use ClusterOperationsExtensions.ListConfigurations going forward. Note that users with the 'Reader' role will not be able to use this method. This allows for granular control over which users can access sensitive information for a cluster.
    • To retrieve just HTTP gateway credentials, use ClusterOperationsExtensions.GetGatewaySettings.
  • ClusterOperationsExtensions.GetConnectivitySettings is now deprecated and has been replaced by ClusterOperationsExtensions.GetGatewaySettings.

  • ClusterOperationsExtensions.ConfigureHttpSettings is now deprecated and has been replaced by ClusterOperationsExtensions.UpdateGatewaySettings.

  • ConfigurationsOperationsExtensions.EnableHttp and DisableHttp are now deprecated. HTTP is now always enabled, so these methods are no longer needed.

Versions 3.x and up

Update to version 5.0.0 or later of the HDInsight SDK for .NET. Minimal code modifications may be required if you are using a method affected by these changes:

SDK for Python

Update to version 1.0.0 or later of the HDInsight SDK for Python. Minimal code modifications may be required if you are using a method affected by these changes:

SDK For Java

Update to version 1.0.0 or later of the HDInsight SDK for Java. Minimal code modifications may be required if you are using a method affected by these changes:

SDK For Go

Update to version 27.1.0 or later of the HDInsight SDK for Go. Minimal code modifications may be required if you are using a method affected by these changes:

Az.HDInsight PowerShell

Update to Az PowerShell version 2.0.0 or later to avoid interruptions. Minimal code modifications may be required if you are using a method affected by these changes.

  • Grant-AzHDInsightHttpServicesAccess is now deprecated and has been replaced by the new Set-AzHDInsightGatewayCredential cmdlet.
  • Get-AzHDInsightJobOutput has been updated to support granular role-based access to the storage key.
    • Users with HDInsight Cluster Operator, Contributor, or Owner roles will not be affected.
    • Users with only the Reader role will need to specify the DefaultStorageAccountKey parameter explicitly.
  • Revoke-AzHDInsightHttpServicesAccess is now deprecated. HTTP is now always enabled, so this cmdlet is no longer needed. See the az.HDInsight migration guide for more details.

Add the HDInsight Cluster Operator role assignment to a user

A user with the Contributor or Owner role can assign the HDInsight Cluster Operator role to users that you would want to have read/write access to sensitive HDInsight cluster configuration values (such as cluster gateway credentials and storage account keys).

Using the Azure CLI

The simplest way to add this role assignment is by using the az role assignemnt create command in Azure CLI.

Note

This command must be run by a user with the Contributor or Owner roles, as only they can grant these permissions. The --assignee is the email address of the user to whom you want to assign the HDInsight Cluster Operator role.

Grant role at the resource (cluster) level

az role assignment create --role "HDInsight Cluster Operator" --assignee <user@domain.com> --scope /subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.HDInsight/clusters/<ClusterName>

Grant role at the resource group level

az role assignment create --role "HDInsight Cluster Operator" --assignee user@domain.com -g <ResourceGroupName>

Grant role at the subscription level

az role assignment create --role "HDInsight Cluster Operator" --assignee user@domain.com

Using the Azure portal

You can alternatively use the Azure portal to add the HDInsight Cluster Operator role assignment to a user. See the documentation, Manage access to Azure resources using RBAC and the Azure portal - Add a role assignment.