Azure security baseline for HDInsight

This security baseline applies guidance from the Azure Security Benchmark version 2.0 to HDInsight. The Azure Security Benchmark provides recommendations on how you can secure your cloud solutions on Azure. The content is grouped by the security controls defined by the Azure Security Benchmark and the related guidance applicable to HDInsight.

Note

Controls not applicable to HDInsight, and those for which the global guidance is recommended verbatim, have been excluded. To see how HDInsight completely maps to the Azure Security Benchmark, see the full HDInsight security baseline mapping file.

Network Security

For more information, see the Azure Security Benchmark: Network Security.

NS-1: Implement security for internal traffic

Guidance: Perimeter security in Azure HDInsight is achieved through virtual networks. An enterprise administrator can create a cluster inside a virtual network and use a network security group (NSG) to restrict access to the virtual network. Only the allowed IP addresses in the inbound Network Security Group rules will be able to communicate with the Azure HDInsight cluster. This configuration provides perimeter security. All clusters deployed in a virtual network will also have a private endpoint that resolves to a private IP address inside the Virtual Network for private HTTP access to the cluster gateways.

Based on your applications and enterprise segmentation strategy, restrict or allow traffic between internal resources based on your network security group rules. For specific, well-defined applications (such as a 3-tier app), this can be a highly secure deny by default

Ports required generally across all types of clusters:

  • 22-23 - SSH access to the cluster resources

  • 443 - Ambari, WebHCat REST API, HiveServer ODBC and JDBC

For specific types of clusters and more details review this article.

You can create private HDInsight clusters by configuring specific network properties in an Azure Resource Manager (ARM) template. There are two properties that you use to create private HDInsight clusters:

  • Remove public IP addresses by setting resourceProviderConnection to outbound.

  • Enable Azure Private Link and use Private Endpoints by setting privateLink to enabled.

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

NS-2: Connect private networks together

Guidance: Use Azure ExpressRoute or Azure virtual private network (VPN) to create private connections between Azure datacenters and on-premises infrastructure in a colocation environment. ExpressRoute connections don't go over the public internet, and they offer more reliability, faster speeds, and lower latencies than typical internet connections. For point-to-site VPN and site-to-site VPN, you can connect on-premises devices or networks to a virtual network using any combination of these VPN options and Azure ExpressRoute.

To connect two or more virtual networks in Azure together, use virtual network peering. Network traffic between peered virtual networks is private and is kept on the Azure backbone network.

Responsibility: Customer

Azure Security Center monitoring: None

NS-3: Establish private network access to Azure services

Guidance: Use Azure Private Link to enable private access to HDInsight from your virtual networks without crossing the internet.

Private access is an additional defense in depth measure to the authentication and traffic security offered by Azure services.

Responsibility: Customer

Azure Security Center monitoring: None

NS-4: Protect applications and services from external network attacks

Guidance: Protect your HDInsight resources against attacks from external networks, including distributed denial of service (DDoS) attacks, application-specific attacks, and unsolicited and potentially malicious internet traffic. Use Azure Firewall to protect applications and services against potentially malicious traffic from the internet and other external locations. Protect your assets against DDoS attacks by enabling DDoS standard protection on your Azure virtual networks. Use Azure Security Center to detect misconfiguration risks to your network related resources.

Responsibility: Customer

Azure Security Center monitoring: None

NS-5: Deploy intrusion detection/intrusion prevention systems (IDS/IPS)

Guidance: Use Azure Firewall threat intelligence-based filtering to alert on and/or block traffic to and from known malicious IP addresses and domains. The IP addresses and domains are sourced from the Microsoft Threat Intelligence feed. When payload inspection is required, you can deploy a third-party intrusion detection/intrusion prevention system (IDS/IPS) from Azure Marketplace with payload inspection capabilities. Alternately, you can use host-based IDS/IPS or a host-based endpoint detection and response (EDR) solution in conjunction with or instead of network-based IDS/IPS.

Responsibility: Customer

Azure Security Center monitoring: None

NS-6: Simplify network security rules

Guidance: Use Azure Virtual Network Service Tags to define network access controls on network security groups or Azure Firewall configured for your HDInsight resources. You can use the service tag 'HDInsight' in place of specific IP addresses for the service when creating security rules. By specifying the service tag name in the appropriate source or destination field of a rule, you can allow or deny the traffic for the corresponding service. Microsoft manages the address prefixes encompassed by the service tag and automatically updates the service tag as addresses change.

Responsibility: Customer

Azure Security Center monitoring: None

NS-7: Secure Domain Name Service (DNS)

Guidance: Follow the best practices for DNS security to mitigate against common attacks like dangling DNS, DNS amplifications attacks, DNS poisoning and spoofing, etc.

When Azure DNS is used as your authoritative DNS service, ensure DNS zones and records are protected from accidental or malicious modification using Azure RBAC and resource locks.

Responsibility: Customer

Azure Security Center monitoring: None

Identity Management

For more information, see the Azure Security Benchmark: Identity Management.

IM-1: Standardize Azure Active Directory as the central identity and authentication system

Guidance: HDInsight uses Azure Active Directory (Azure AD) as its central authentication and authorization system. You should standardize Azure AD to govern your organization's identity and access management in:

  • Microsoft Cloud resources, such as the Azure portal, Azure Storage, Azure Virtual Machine (Linux and Windows), Azure Key Vault, PaaS, and SaaS applications.
  • Your organization's resources, such as applications on Azure or your corporate network resources.

Securing Azure AD should be a high priority in your organization's cloud security practice. Azure AD provides an identity secure score to help you assess identity security posture relative to Microsoft's best practice recommendations. Use the score to gauge how closely your configuration matches best practice recommendations, and to make improvements in your security posture.

Configure Azure HDInsight clusters with Enterprise Security Package (ESP), these clusters can be connected to a domain so that users can use their domain credentials to authenticate with the clusters. Three main Azure RBAC built-in roles are available for the resource management of HDInsight:

  • Reader: Read access to HDInsight resources including secrets

  • HDInsight cluster operator: Read and write access to HDInsight resources including secrets

  • Contributor: Read and write access including secrets and ability to execute script actions

For row level security Apache Ranger can be implemented to set access control policies for Hive.

Responsibility: Customer

Azure Security Center monitoring: None

IM-2: Manage application identities securely and automatically

Guidance: HDInsight supports managed identities for its Azure resources. Use managed identities with HDInsight instead of creating service principals to access other resources. HDInsight can natively authenticate to the Azure services/resources that supports Azure AD authentication through a pre-defined access grant rule without using credentials hard coded in source code or configuration files.

Responsibility: Customer

Azure Security Center monitoring: None

IM-3: Use Azure AD single sign-on (SSO) for application access

Guidance: Use Azure HDInsight ID Broker to sign in to Enterprise Security Package (ESP) clusters by using multifactor authentication, without providing any passwords. If you've already signed in to other Azure services, such as the Azure portal, you can sign in to your Azure HDInsight cluster with a single sign-on (SSO) experience.

Responsibility: Customer

Azure Security Center monitoring: None

IM-4: Use strong authentication controls for all Azure Active Directory based access

Guidance: HDInsight uses Azure Active Directory (Azure AD), which supports strong authentication controls through multi-factor authentication (MFA) and strong passwordless methods.

Azure HDInsight clusters with the Enterprise Security Package configured can be connected to a domain so that domain users can use their domain credentials to authenticate with the clusters and run big data jobs. When authenticating with multifactor authentication enabled, users will be challenged to provide a second authentication factor.

Responsibility: Customer

Azure Security Center monitoring: None

IM-5: Monitor and alert on account anomalies

Guidance: HDInsight is integrated with Azure Active Directory, which provides the following data sources:

  • Sign-ins - The sign-ins report provides information about the usage of managed applications and user sign-in activities.
  • Audit logs - Provides traceability through logs for all changes done by various features within Azure AD. Examples of audit logs include changes made to any resource within Azure AD, like adding or removing users, apps, groups, roles, and policies.
  • Risky sign-ins - A risky sign-in is an indicator for a sign-in attempt that might have been performed by someone who is not the legitimate owner of a user account.
  • Users flagged for risk - A risky user is an indicator for a user account that might have been compromised.

These data sources can be integrated with Azure Monitor, Azure Sentinel, or third-party SIEM systems. Azure Security Center can also alert you about certain suspicious activities, such as an excessive number of failed authentication attempts or deprecated accounts in the subscription.

During cluster creation an admin account is created, if Enterprise Security is not enabled and no Azure AD integration is present logs are accessible by logging into the head nodes and reviewing logs from there.

Responsibility: Customer

Azure Security Center monitoring: None

IM-6: Restrict Azure resource access based on conditions

Guidance: HDInsight supports Azure Active Directory (Azure AD) conditional access for a more granular access control based on user-defined conditions, such as user logins from certain IP ranges will need to use multifactor authentication for login. Granular authentication session management policy can also be used for different use cases. These conditional access policies will only apply to user accounts that are authenticating to Azure AD to access and manage the HDInsight service, but they will not apply to service principals, keys, or tokens used to connect to your HDInsight resource.

Responsibility: Customer

Azure Security Center monitoring: None

IM-7: Eliminate unintended credential exposure

Guidance: If using any code related to your Azure HDInsight deployment, you may implement Credential Scanner to identify credentials within code. Credential Scanner will also encourage moving discovered credentials to more secure locations such as Azure Key Vault.

For GitHub, you can use the native secret scanning feature to identify credentials or other forms of secrets within the code.

Responsibility: Customer

Azure Security Center monitoring: None

Privileged Access

For more information, see the Azure Security Benchmark: Privileged Access.

PA-1: Protect and limit highly privileged users

Guidance: The most critical built-in roles for Azure Active Directory (Azure AD) are the Global Administrator and the Privileged Role Administrator, as users assigned to these two roles can delegate administrator roles:

  • Global Administrator / Company Administrator: Users with this role have access to all administrative features in Azure AD, as well as services that use Azure AD identities.
  • Privileged Role Administrator: Users with this role can manage role assignments in Azure AD, as well as within Azure AD Privileged Identity Management (PIM). In addition, this role allows management of all aspects of PIM and administrative units.

Use HDInsight Enterprise Security Package, which has the following privileged roles:

  • Cluster Administrator

  • Cluster Operator

  • Service Administrator

  • Service Operator

  • Cluster User

Create standard operating procedures around the use of dedicated administrative accounts. You should limit the number of highly privileged accounts or roles and protect these accounts at an elevated level. Users with this privilege can directly or indirectly read and modify every resource in your Azure environment.

You can enable just-in-time (JIT) privileged access to Azure resources and Azure AD using Azure AD PIM. JIT grants temporary permissions to perform privileged tasks only when users need it. PIM can also generate security alerts when there is suspicious or unsafe activity in your Azure AD organization.

Responsibility: Customer

Azure Security Center monitoring: None

PA-2: Restrict administrative access to business-critical systems

Guidance: HDInsight uses Azure role-based access control (Azure RBAC) to isolate access to business-critical systems by restricting which accounts are granted privileged access to the subscriptions and management groups they are in.

Ensure that you also restrict access to the management, identity, and security systems that have administrative access to your business-critical access controls, such as Active Directory Domain Controllers (DCs), security tools, and system management tools with agents installed on business-critical systems. Attackers who compromise these management and security systems can immediately weaponize them to compromise business-critical assets.

All types of access controls should be aligned to your enterprise segmentation strategy to ensure consistent access control.

A best practice most enterprises follow is making sure that not every employee has full access to all enterprise resources. Likewise, the admin can define role-based access control policies for the cluster resources. This action is only available in the ESP clusters.

The Hadoop admin can configure role-based access control (RBAC). The configurations secure Apache Hive, HBase, and Kafka with Apache Ranger plugins. Configuring RBAC policies allows you to associate permissions with a role in the organization. This layer of abstraction makes it easier to ensure people have only the permissions needed to do their work responsibilities. Ranger also allows you to audit the data access of employees and any changes done to access control policies.

For example, the admin can configure Apache Ranger to set access control policies for Hive. This functionality ensures row-level and column-level filtering (data masking). And filters the sensitive data from unauthorized users.

Responsibility: Customer

Azure Security Center monitoring: None

PA-3: Review and reconcile user access regularly

Guidance: HDInsight uses Azure Active Directory (Azure AD) accounts to manage its resources, review user accounts, and access assignments regularly to ensure the accounts and their access are valid. You can use Azure AD and access reviews to review group memberships, access to enterprise applications, and role assignments. Azure AD reporting can provide logs to help discover stale accounts. You can also use Azure AD Privileged Identity Management (PIM) to create access review report workflows to facilitate the review process.

In addition, Azure AD PIM can also be configured to alert you when an excessive number of administrator accounts are created, and to identify administrator accounts that are stale or improperly configured.

Responsibility: Customer

Azure Security Center monitoring: None

PA-4: Set up emergency access in Azure AD

Guidance: HDInsight uses Azure Active Directory (Azure AD) as an identity provider. To prevent being accidentally locked out of your Azure AD organization, set up an emergency access account for access when normal administrative accounts cannot be used. Emergency access accounts are usually highly privileged, and they should not be assigned to specific individuals. Emergency access accounts are limited to emergency or 'break glass' scenarios where normal administrative accounts can't be used.

You should ensure that the credentials (such as password, certificate, or smart card) for emergency access accounts are kept secure and known only to individuals who are authorized to use them only in an emergency.

Responsibility: Customer

Azure Security Center monitoring: None

PA-5: Automate entitlement management

Guidance: HDInsight is integrated with Azure Active Directory (Azure AD) to manage its resources. Use Azure AD entitlement management features to automate access request workflows, including access assignments, reviews, and expiration. Dual or multi-stage approval is also supported.

HDInsight uses Apache Ranger and Apache Ambari which can sync with Azure AD.

Responsibility: Customer

Azure Security Center monitoring: None

PA-6: Use privileged access workstations

Guidance: Secured, isolated workstations are critically important for the security of sensitive roles like administrator, developer, and critical service operator. Use highly secured user workstations and/or Azure Bastion for administrative tasks related to managing your HDInsight resources. Use Azure Active Directory (Azure AD), Microsoft Defender Advanced Threat Protection (ATP), and/or Microsoft Intune to deploy a secure and managed user workstation for administrative tasks. The secured workstations can be centrally managed to enforce secured configuration including strong authentication, software and hardware baselines, and restricted logical and network access.

Responsibility: Customer

Azure Security Center monitoring: None

PA-7: Follow just enough administration (least privilege principle)

Guidance: HDInsight is integrated with Azure role-based access control (Azure RBAC) to manage its resources. Azure RBAC allows you to manage Azure resource access through role assignments. You can assign these roles to users, groups service principals, and managed identities. There are pre-defined built-in roles for certain resources, and these roles can be inventoried or queried through tools such as Azure CLI, Azure PowerShell, or the Azure portal. The privileges you assign to resources through the Azure RBAC should be always limited to what is required by the roles. This complements the just-in-time (JIT) approach of Azure AD Privileged Identity Management (PIM) and should be reviewed periodically. Use built-in roles to allocate permissions and only create custom roles when required. HDInsight uses Apache Ranger to allow for more granular control over permissions.

Responsibility: Customer

Azure Security Center monitoring: None

PA-8: Choose approval process for Microsoft support

Guidance: In support scenarios where Microsoft needs to access customer data, HDInsight supports Customer Lockbox to provide an interface for you to review and approve or reject customer data access requests.

Responsibility: Customer

Azure Security Center monitoring: None

Data Protection

For more information, see the Azure Security Benchmark: Data Protection.

DP-1: Discovery, classify and label sensitive data

Guidance: Use tags on resources related to your Azure HDInsight deployments to assist in tracking Azure resources that store or process sensitive information. Classify and identify sensitive data using Azure Purview for any data stored in SQL databases or Azure Storage accounts associated to your HDInsight cluster.

For the underlying platform which is managed by Microsoft, Microsoft treats all customer content as sensitive and goes to great lengths to guard against customer data loss and exposure. To ensure customer data within Azure remains secure, Microsoft has implemented and maintains a suite of robust data protection controls and capabilities.

Responsibility: Shared

Azure Security Center monitoring: None

DP-2: Protect sensitive data

Guidance: Implement separate subscriptions and/or management groups for development, test, and production. Azure HDInsight clusters and any associated storage accounts should be separated by virtual network/subnet, tagged appropriately, and secured within an network security group (NSG) or Azure Firewall. Cluster data should be contained within a secured Azure Storage Account or Azure Data Lake Storage (Gen1 or Gen2).

Responsibility: Customer

Azure Security Center monitoring: None

DP-3: Monitor for unauthorized transfer of sensitive data

Guidance: For Azure HDInsight clusters storing or processing sensitive information, mark the cluster and related resources as sensitive using tags. To reduce the risk of data loss via exfiltration, restrict outbound network traffic for Azure HDInsight clusters using Azure Firewall.

HDInsight does not support automatic monitoring for unauthorized transfer of sensitive data natively.

For the underlying platform which is managed by Microsoft, Microsoft treats all customer content as sensitive and goes to great lengths to guard against customer data loss and exposure. To ensure customer data within Azure remains secure, Microsoft has implemented and maintains a suite of robust data protection controls and capabilities.

Responsibility: Shared

Azure Security Center monitoring: None

DP-4: Encrypt sensitive information in transit

Guidance: HDInsight supports data encryption in transit with TLS v1.2 or greater. Encrypt all sensitive information in transit. Ensure that any clients connecting to your Azure HDInsight cluster or cluster data stores (Azure Storage Accounts or Azure Data Lake Storage Gen1/Gen2) are able to negotiate TLS 1.2 or greater. Microsoft Azure resources will negotiate TLS 1.2 by default.

To complement access controls, data in transit should be protected against 'out of band' attacks (such as traffic capture) using encryption to ensure that attackers cannot easily read or modify the data.

For remote management, use SSH (for Linux) or RDP/TLS (for Windows) instead of an unencrypted protocol. Obsolete SSL, TLS, SSH versions and protocols, and weak ciphers should be disabled.

By default, Azure provides encryption for data in transit between Azure data centers.

Responsibility: Shared

Azure Security Center monitoring: None

DP-5: Encrypt sensitive data at rest

Guidance: If using Azure SQL Database to store Apache Hive and Apache Oozie metadata, ensure SQL data remains encrypted at all times. For Azure Storage Accounts and Data Lake Storage (Gen1 or Gen2), it is recommended to allow Microsoft to manage your encryption keys, however, you have the option to manage your own keys.

HDInsight supports multiple types of encryption in two different layers:

Responsibility: Shared

Azure Security Center monitoring: None

Asset Management

For more information, see the Azure Security Benchmark: Asset Management.

AM-1: Ensure security team has visibility into risks for assets

Guidance: Ensure security teams are granted Security Reader permissions in your Azure tenant and subscriptions so they can monitor for security risks using Azure Security Center.

Depending on how security team responsibilities are structured, monitoring for security risks could be the responsibility of a central security team or a local team. That said, security insights and risks must always be aggregated centrally within an organization.

Security Reader permissions can be applied broadly to an entire tenant (Root Management Group) or scoped to management groups or specific subscriptions.

Responsibility: Customer

Azure Security Center monitoring: None

AM-2: Ensure security team has access to asset inventory and metadata

Guidance: Ensure that security teams have access to a continuously updated inventory of assets on Azure, like HDInsight. Security teams often need this inventory to evaluate their organization's potential exposure to emerging risks, and as an input to continuous security improvements. Create an Azure Active Directory (Azure AD) group to contain your organization's authorized security team and assign them read access to all HDInsight resources, which can be simplified by a single high-level role assignment within your subscription.

Apply tags to your Azure resources, resource groups, and subscriptions to logically organize them into a taxonomy. Each tag consists of a name and a value pair. For example, you can apply the name "Environment" and the value "Production" to all the resources in production.

Responsibility: Customer

Azure Security Center monitoring: None

AM-3: Use only approved Azure services

Guidance: Use Azure Policy to audit and restrict which services users can provision in your environment. Use Azure Resource Graph to query for and discover resources within their subscriptions. You can also use Azure Monitor to create rules to trigger alerts when a non-approved service is detected.

Responsibility: Customer

Azure Security Center monitoring: None

AM-4: Ensure security of asset lifecycle management

Guidance: Establish or update security policies that address asset lifecycle management processes for potentially high impact modifications to HDInsight. These modifications include but are not limited to: permissions (Azure RBAC), data, network configuration, and administrative privilege assignment. Outline any high-impact configurations that the customer should be aware of related to HDInsight.

Remove HDInsight resources when they are no longer needed.

Responsibility: Customer

Azure Security Center monitoring: None

AM-5: Limit users' ability to interact with Azure Resource Manager

Guidance: Use Azure Conditional Access to limit users' ability to interact with Azure Resources Manager by configuring "Block access" for the "Microsoft Azure Management" App.

Responsibility: Customer

Azure Security Center monitoring: None

AM-6: Use only approved applications in compute resources

Guidance: Use Azure Resource Graph to query for and discover all resources (such as compute, storage, network, ports, and protocols etc.), including Azure HDInsight clusters, within your subscriptions. Remove any unapproved Azure resources that you discover. For Azure HDInsight cluster nodes, implement a third-party solution to remove or alert on unapproved software.

Responsibility: Customer

Azure Security Center monitoring: None

Logging and Threat Detection

For more information, see the Azure Security Benchmark: Logging and Threat Detection.

LT-1: Enable threat detection for Azure resources

Guidance: Azure HDInsight does not support defender natively, it uses ClamAV. However when using the Enterprise Security package for HDInsight you can leverage some of the Azure Security Center built-in threat detection capability and enable Azure Defender (formerly Azure Advanced Threat Protection) for your virtual machines associated to HDInsight.

Forward any logs from HDInsight to your SIEM, which can be used to set up custom threat detections. Ensure that you are monitoring different types of Azure assets for potential threats and anomalies. Focus on getting high-quality alerts to reduce false positives for analysts to sort through. Alerts can be sourced from log data, agents, or other data.

Responsibility: Customer

Azure Security Center monitoring: None

LT-2: Enable threat detection for Azure identity and access management

Guidance: Azure Active Directory (Azure AD) provides the following user logs, which can be viewed in Azure AD reporting or integrated with Azure Monitor, Azure Sentinel, or other SIEM/monitoring tools for more sophisticated monitoring and analytics use cases:

  • Sign-ins - The sign-ins report provides information about the usage of managed applications and user sign-in activities.
  • Audit logs - Provides traceability through logs for all changes done by various features within Azure AD. Examples of audit logs include changes made to any resources within Azure AD, like adding or removing users, apps, groups, roles, and policies.
  • Risky sign-ins - A risky sign-in is an indicator for a sign-in attempt that might have been performed by someone who is not the legitimate owner of a user account.
  • Users flagged for risk - A risky user is an indicator for a user account that might have been compromised.

Azure Security Center can also trigger alerts on certain suspicious activities, such as excessive number of failed authentication attempts or deprecated accounts in the subscription. In addition to the basic security hygiene monitoring, Azure Security Center's Threat Protection module can also collect more in-depth security alerts from individual Azure compute resources (virtual machines, containers, app service), data resources (SQL DB and storage), and Azure service layers. This capability allows you to have visibility on account anomalies inside individual resources.

Responsibility: Customer

Azure Security Center monitoring: None

LT-3: Enable logging for Azure network activities

Guidance: Use Azure Security Center and remediate network protection recommendations for the virtual network, subnet, and network security group being used to secure your Azure HDInsight cluster. Enable network security group (NSG) flow logs and send logs into a Azure Storage Account to support traffic audits. You may also send NSG flow logs to a Azure Log Analytics workspace and use Azure Traffic Analytics to provide insights into traffic flow in your Azure cloud. Some advantages of Azure Traffic Analytics are the ability to visualize network activity and identify hot spots, identify security threats, understand traffic flow patterns, and pinpoint network misconfigurations. HDInsight logs all network traffic that it processes for customer access. Enable the network flow capability within your deployed offering resources.

Responsibility: Customer

Azure Security Center monitoring: None

LT-4: Enable logging for Azure resources

Guidance: Activity logs, which are automatically available, contain all write operations (PUT, POST, DELETE) for your HDInsight resources except read operations (GET). Activity logs can be used to find an error when troubleshooting or to monitor how a user in your organization modified a resource.

Enable Azure resource logs for HDInsight. You can use Azure Security Center and Azure Policy to enable resource logs and log data collecting. These logs can be critical for investigating security incidents and performing forensic exercises.

HDInsight also produces security audit logs for the local administer accounts. Enable these local admin audit logs.

Responsibility: Customer

Azure Security Center monitoring: None

LT-5: Centralize security log management and analysis

Guidance: Centralize logging storage for your HDInsight resources for analysis. For each log source, ensure that you have assigned a data owner, access guidance, storage location, what tools are used to process and access the data, and data retention requirements.

Ensure that you are integrating Azure activity logs into your central logging. Ingest logs via Azure Monitor to aggregate security data generated by endpoint devices, network resources, and other security systems. In Azure Monitor, use Log Analytics workspaces to query and perform analytics, and use Azure Storage accounts for long term and archival storage.

In addition, enable and onboard HDInsight log data to Azure Sentinel or a third-party SIEM. Many organizations choose to use Azure Sentinel for 'hot' data that is used frequently and Azure Storage for 'cold' data that is used less frequently.

Responsibility: Customer

Azure Security Center monitoring: None

LT-6: Configure log storage retention

Guidance: Ensure that any storage accounts or Log Analytics workspaces used for storing HDInsight logs have the log retention period set according to your organization's compliance regulations.

Responsibility: Customer

Azure Security Center monitoring: None

LT-7: Use approved time synchronization sources

Guidance: Microsoft maintains time sources for most Azure platform PaaS and SaaS services. For your virtual machines, use a Microsoft default network time protocol (NTP) server for time synchronization unless you have a specific requirement. If you need to stand up your own NTP server, ensure that you secure the UDP service port 123. All logs generated by resources within Azure provide time stamps with the time zone specified by default.

Responsibility: Shared

Azure Security Center monitoring: None

Incident Response

For more information, see the Azure Security Benchmark: Incident Response.

IR-1: Preparation – update incident response process for Azure

Guidance: Ensure your organization has processes to respond to security incidents, has updated these processes for Azure, and is regularly exercising them to ensure readiness.

Responsibility: Customer

Azure Security Center monitoring: None

IR-2: Preparation – setup incident notification

Guidance: Set up security incident contact information in Azure Security Center. This contact information is used by Microsoft to contact you if the Microsoft Security Response Center (MSRC) discovers that your data has been accessed by an unlawful or unauthorized party. You also have options to customize incident alert and notification in different Azure services based on your incident response needs.

Responsibility: Customer

Azure Security Center monitoring: None

IR-3: Detection and analysis – create incidents based on high quality alerts

Guidance: Ensure you have a process to create high-quality alerts and measure the quality of alerts. This allows you to learn lessons from past incidents and prioritize alerts for analysts, so they don't waste time on false positives.

High-quality alerts can be built based on experience from past incidents, validated community sources, and tools designed to generate and clean up alerts by fusing and correlating diverse signal sources.

Azure Security Center provides high-quality alerts across many Azure assets. You can use the ASC data connector to stream the alerts to Azure Sentinel. Azure Sentinel lets you create advanced alert rules to generate incidents automatically for an investigation.

Export your Azure Security Center alerts and recommendations using the export feature to help identify risks to Azure resources. Export alerts and recommendations either manually or in an ongoing, continuous fashion.

Responsibility: Customer

Azure Security Center monitoring: None

IR-4: Detection and analysis – investigate an incident

Guidance: Ensure analysts can query and use diverse data sources as they investigate potential incidents, to build a full view of what happened. Diverse logs should be collected to track the activities of a potential attacker across the kill chain to avoid blind spots. You should also ensure insights and learnings are captured for other analysts and for future historical reference.

The data sources for investigation include the centralized logging sources that are already being collected from the in-scope services and running systems, but can also include:

  • Network data - use network security groups' flow logs, Azure Network Watcher, and Azure Monitor to capture network flow logs and other analytics information.

  • Snapshots of running systems:

    • Use Azure virtual machine's snapshot capability to create a snapshot of the running system's disk.

    • Use the operating system's native memory dump capability to create a snapshot of the running system's memory.

    • Use the snapshot feature of the Azure services or your software's own capability to create snapshots of the running systems.

Azure Sentinel provides extensive data analytics across virtually any log source and a case management portal to manage the full lifecycle of incidents. Intelligence information during an investigation can be associated with an incident for tracking and reporting purposes.

Responsibility: Customer

Azure Security Center monitoring: None

IR-5: Detection and analysis – prioritize incidents

Guidance: Provide context to analysts on which incidents to focus on first based on alert severity and asset sensitivity.

Azure Security Center assigns a severity to each alert to help you prioritize which alerts should be investigated first. The severity is based on how confident Security Center is in the finding or the analytics used to issue the alert, as well as the confidence level that there was malicious intent behind the activity that led to the alert.

Additionally, mark resources using tags and create a naming system to identify and categorize Azure resources, especially those processing sensitive data. It is your responsibility to prioritize the remediation of alerts based on the criticality of the Azure resources and environment where the incident occurred.

Responsibility: Customer

Azure Security Center monitoring: None

IR-6: Containment, eradication and recovery – automate the incident handling

Guidance: Automate manual repetitive tasks to speed up response time and reduce the burden on analysts. Manual tasks take longer to execute, slowing each incident and reducing how many incidents an analyst can handle. Manual tasks also increase analyst fatigue, which increases the risk of human error that causes delays, and degrades the ability of analysts to focus effectively on complex tasks.

Use workflow automation features in Azure Security Center and Azure Sentinel to automatically trigger actions or run a playbook to respond to incoming security alerts. The playbook takes actions, such as sending notifications, disabling accounts, and isolating problematic networks.

Responsibility: Customer

Azure Security Center monitoring: None

Posture and Vulnerability Management

For more information, see the Azure Security Benchmark: Posture and Vulnerability Management.

PV-1: Establish secure configurations for Azure services

Guidance: Use Azure Policy aliases in the "Microsoft.HDInsight" namespace to create custom policies to audit or enforce the network configuration of your HDInsight cluster.

Optionally, if you have a Rapid7, Qualys, or any other vulnerability management platform subscription, you may use script actions to install vulnerability assessment agents on your Azure HDInsight cluster nodes and manage the nodes through the respective portal.

With Azure HDInsight Enterprise Security Package (ESP), you can use Apache Ranger to create and manage fine-grained access control and data obfuscation policies for your data stored in files, folders, databases, tables and rows/columns. The hadoop admin can configure role-based access control (RBAC) to secure Apache Hive, HBase, Kafka and Spark using those plugins in Apache Ranger.

Responsibility: Customer

Azure Security Center monitoring: None

PV-2: Sustain secure configurations for Azure services

Guidance: Use Azure Policy [deny] and [deploy if not exist] to enforce secure settings for your Azure HDInsight clusters and related resources.

Responsibility: Customer

Azure Security Center monitoring: None

PV-3: Establish secure configurations for compute resources

Guidance: Ubuntu images become available for new Azure HDInsight cluster creation within three months of being published. Running clusters aren't auto-patched. Customers must use script actions or other mechanisms to patch a running cluster. As a best practice, you can run these script actions and apply security updates right after the cluster creation

Use Azure Security Center and Azure Policy to establish secure configurations on all compute resources including VMs, containers, and others.

Responsibility: Customer

Azure Security Center monitoring: None

PV-4: Sustain secure configurations for compute resources

Guidance: Azure HDInsight Operating System Images managed and maintained by Microsoft. However, you are responsible for implementing OS-level state configuration for that image.

Use Azure Security Center and Azure Policy to regularly assess and remediate configuration risks on your Azure compute resources such as virtual machines (VMs), containers, and others. In addition, you can use Azure Resource Manager templates, custom operating system images, or Azure Automation State Configuration to maintain the security configuration of the operating system required by your organization. The Microsoft VM templates combined with the Azure Automation State Configuration can assist in meeting and maintaining the security requirements.

Responsibility: Shared

Azure Security Center monitoring: None

PV-5: Securely store custom operating system and container images

Guidance: HDInsight allows customers to manage operating system images or container images. Use Azure role-based access control (Azure RBAC) to ensure that only authorized users can access your custom images. With Azure Shared Image Gallery to share your images with different users, service principals, or Azure Active Directory (AD) groups within your organization. Store container images in Azure Container Registry and use Azure RBAC to ensure that only authorized users have access.

Responsibility: Customer

Azure Security Center monitoring: None

PV-6: Perform software vulnerability assessments

Guidance: HDInsight can use a third-party solution for performing vulnerability assessments on network devices and web applications. When conducting remote scans, do not use a single, perpetual administrative account. Consider implementing JIT provisioning methodology for the scan account. Credentials for the scan account should be protected, monitored, and used only for vulnerability scanning.

As required, export scan results at consistent intervals and compare the results with previous scans to verify that vulnerabilities have been remediated.

Responsibility: Customer

Azure Security Center monitoring: None

PV-7: Rapidly and automatically remediate software vulnerabilities

Guidance: Running HDInsight clusters aren't auto-patched. You must use script actions or other mechanisms to patch a running cluster. As a best practice, you can run these script actions and apply security updates right after the cluster creation.

Responsibility: Customer

Azure Security Center monitoring: None

PV-8: Conduct regular attack simulation

Guidance: As required, conduct penetration testing or red team activities on your Azure resources and ensure remediation of all critical security findings.

Follow the Microsoft Cloud Penetration Testing Rules of Engagement to ensure your penetration tests are not in violation of Microsoft policies. Use Microsoft's strategy and execution of Red Teaming and live site penetration testing against Microsoft-managed cloud infrastructure, services, and applications.

Responsibility: Shared

Azure Security Center monitoring: None

Endpoint Security

For more information, see the Azure Security Benchmark: Endpoint Security.

ES-1: Use Endpoint Detection and Response (EDR)

Guidance: Azure HDInsight does not support defender natively, it uses ClamAV. Forward the ClamAV logs to a centralized SIEM or other detection and alerting system.

Responsibility: Customer

Azure Security Center monitoring: None

ES-2: Use centrally managed modern anti-malware software

Guidance: Azure HDInsight comes with Clamscan pre-installed and enabled for the cluster node images, however you must manage the software and manually aggregate/monitor any logs Clamscan produces.

Responsibility: Customer

Azure Security Center monitoring: None

ES-3: Ensure anti-malware software and signatures are updated

Guidance: Azure HDInsight comes with Clamscan pre-installed and enabled for the cluster node images. Clamscan will perform engine and definition updates automatically and update its anti-malware signatures based on ClamAV’s official virus signature database.

Responsibility: Customer

Azure Security Center monitoring: None

Backup and Recovery

For more information, see the Azure Security Benchmark: Backup and Recovery.

BR-3: Validate all backups including customer-managed keys

Guidance: If Azure Key Vault is being used with your Azure HDInsight deployment, periodically test restoration of backed up customer-managed keys.

Responsibility: Customer

Azure Security Center monitoring: None

BR-4: Mitigate risk of lost keys

Guidance: If Azure Key Vault is being used with your Azure HDInsight deployment, enable soft delete in Key Vault to protect keys against accidental or malicious deletion.

Responsibility: Customer

Azure Security Center monitoring: None

Governance and Strategy

For more information, see the Azure Security Benchmark: Governance and Strategy.

GS-1: Define asset management and data protection strategy

Guidance: Ensure you document and communicate a clear strategy for continuous monitoring and protection of systems and data. Prioritize discovery, assessment, protection, and monitoring of business-critical data and systems.

This strategy should include documented guidance, policy, and standards for the following elements:

  • Data classification standard in accordance with the business risks
  • Security organization visibility into risks and asset inventory
  • Security organization approval of Azure services for use
  • Security of assets through their lifecycle
  • Required access control strategy in accordance with organizational data classification
  • Use of Azure native and third-party data protection capabilities
  • Data encryption requirements for in-transit and at-rest use cases
  • Appropriate cryptographic standards

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

GS-2: Define enterprise segmentation strategy

Guidance: Establish an enterprise-wide strategy to segmenting access to assets using a combination of identity, network, application, subscription, management group, and other controls.

Carefully balance the need for security separation with the need to enable daily operation of the systems that need to communicate with each other and access data.

Ensure that the segmentation strategy is implemented consistently across control types including network security, identity and access models, and application permission/access models, and human process controls.

Responsibility: Customer

Azure Security Center monitoring: None

GS-3: Define security posture management strategy

Guidance: Continuously measure and mitigate risks to your individual assets and the environment they are hosted in. Prioritize high value assets and highly-exposed attack surfaces, such as published applications, network ingress and egress points, user and administrator endpoints, etc.

Responsibility: Customer

Azure Security Center monitoring: None

GS-4: Align organization roles, responsibilities, and accountabilities

Guidance: Ensure that you document and communicate a clear strategy for roles and responsibilities in your security organization. Prioritize providing clear accountability for security decisions, educating everyone on the shared responsibility model, and educate technical teams on technology to secure the cloud.

Responsibility: Customer

Azure Security Center monitoring: None

GS-5: Define network security strategy

Guidance: Establish an Azure network security approach as part of your organization's overall security access control strategy.

This strategy should include documented guidance, policy, and standards for the following elements:

  • Centralized network management and security responsibility
  • Virtual network segmentation model aligned with the enterprise segmentation strategy
  • Remediation strategy in different threat and attack scenarios
  • Internet edge and ingress and egress strategy
  • Hybrid cloud and on-premises interconnectivity strategy
  • Up-to-date network security artifacts (such as network diagrams, reference network architecture)

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

GS-6: Define identity and privileged access strategy

Guidance: Establish an Azure identity and privileged access approaches as part of your organization's overall security access control strategy.

This strategy should include documented guidance, policy, and standards for the following elements:

  • A centralized identity and authentication system and its interconnectivity with other internal and external identity systems
  • Strong authentication methods in different use cases and conditions
  • Protection of highly privileged users
  • Anomaly user activities monitoring and handling
  • User identity and access review and reconciliation process

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

GS-7: Define logging and threat response strategy

Guidance: Establish a logging and threat response strategy to rapidly detect and remediate threats while meeting compliance requirements. Prioritize providing analysts with high-quality alerts and seamless experiences so that they can focus on threats rather than integration and manual steps.

This strategy should include documented guidance, policy, and standards for the following elements:

  • The security operations (SecOps) organization's role and responsibilities
  • A well-defined incident response process aligning with NIST or another industry framework
  • Log capture and retention to support threat detection, incident response, and compliance needs
  • Centralized visibility of and correlation information about threats, using SIEM, native Azure capabilities, and other sources
  • Communication and notification plan with your customers, suppliers, and public parties of interest
  • Use of Azure native and third-party platforms for incident handling, such as logging and threat detection, forensics, and attack remediation and eradication
  • Processes for handling incidents and post-incident activities, such as lessons learned and evidence retention

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

GS-8: Define backup and recovery strategy

Guidance: Establish an Azure backup and recovery strategy for your organization.

This strategy should include documented guidance, policy, and standards for the following elements:

  • Recovery time objective (RTO) and recovery point objective (RPO) definitions in accordance with your business resiliency objectives
  • Redundancy design in your applications and infrastructure setup
  • Protection of backup using access control and data encryption

For more information, see the following references:

Responsibility: Customer

Azure Security Center monitoring: None

Next steps