High Availability of SAP HANA on Azure Virtual Machines (VMs)

On-premises, you can use either HANA System Replication or use shared storage to establish high availability for SAP HANA. On Azure VMs HANA System Replication on Azure is the only supported High Availability function so far. SAP HANA Replication consists of one primary node and at least one secondary node. Changes to the data on the primary node are replicated to the secondary node synchronously or asynchronously.

This article describes how to deploy the virtual machines, configure the virtual machines, install the cluster framework, install and configure SAP HANA System Replication. In the example configurations, installation commands etc. instance number 03 and HANA System ID HN1 is used.

Read the following SAP Notes and papers first

Overview

To achieve high availability, SAP HANA is installed on two virtual machines. The data is replicated using HANA System Replication.

SAP HANA High Availability overview

The SAP HANA SR setup uses a dedicated virtual hostname and virtual IP addresses. On Azure, a load balancer is required to use a virtual IP address. The following list shows the configuration of the load balancer.

  • Frontend configuration
    • IP address 10.0.0.13 for hn1-db
  • Backend configuration
    • Connected to primary network interfaces of all virtual machines that should be part of HANA System Replication
  • Probe Port
    • Port 62503
  • Loadbalancing rules
    • 30313 TCP
    • 30315 TCP
    • 30317 TCP

Deploying Linux

The resource agent for SAP HANA is included in SUSE Linux Enterprise Server for SAP Applications. The Azure Marketplace contains an image for SUSE Linux Enterprise Server for SAP Applications 12 that you can use to deploy new virtual machines.

Deploy with template

You can use one of the quickstart templates on github to deploy all required resources. The template deploys the virtual machines, the load balancer, availability set etc. To deploy the template, follow these steps:

  1. Open the database template or the converged template on the Azure portal. The database template only creates the load-balancing rules for a database whereas the converged template also creates the load-balancing rules for an ASCS/SCS and ERS (Linux only) instance. If you plan to install an SAP NetWeaver based system and you also want to install the ASCS/SCS instance on the same machines, use the converged template.
  2. Enter the following parameters
    1. Sap System ID
      Enter the SAP system ID of the SAP system you want to install. The ID is going to be used as a prefix for the resources that are deployed.
    2. Stack Type (only applicable if you use the converged template)
      Select the SAP NetWeaver stack type
    3. Os Type
      Select one of the Linux distributions. For this example, select SLES 12
    4. Db Type
      Select HANA
    5. Sap System Size
      The amount of SAPS the new system is going to provide. If you are not sure how many SAPS the system requires, ask your SAP Technology Partner or System Integrator
    6. System Availability
      Select HA
    7. Admin Username and Admin Password
      A new user is created that can be used to log on to the machine.
    8. New Or Existing Subnet
      Determines whether a new virtual network and subnet should be created or an existing subnet should be used. If you already have a virtual network that is connected to your on-premises network, select existing.
    9. Subnet ID
      The ID of the subnet to which the virtual machines should be connected to. To connect the virtual machine to your on-premises network, select the subnet of your VPN or Express Route virtual network. The ID usually looks like /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.Network/virtualNetworks/<virtual network name>/subnets/<subnet name>

Manual Deployment

  1. Create a Resource Group
  2. Create a Virtual Network
  3. Create an Availability Set
    Set max update domain
  4. Create a Load Balancer (internal)
    Select VNET created in the second
  5. Create Virtual Machine 1
    Use at least SLES4SAP 12 SP1, in this example we will use the SLES4SAP 12 SP2 image https://ms.portal.azure.com/#create/SUSE.SUSELinuxEnterpriseServerforSAPApplications12SP2PremiumImage-ARM
    SLES for SAP 12 SP2 (Premium)
    Select Availability Set created earlier
  6. Create Virtual Machine 2
    Use at least SLES4SAP 12 SP1, in this example we will use the SLES4SAP 12 SP1 BYOS image https://ms.portal.azure.com/#create/SUSE.SUSELinuxEnterpriseServerforSAPApplications12SP2PremiumImage-ARM
    SLES for SAP 12 SP2 (Premium)
    Select Availability Set created earlier
  7. Add Data Disks
  8. Configure the load balancer
    1. Create a frontend IP pool
      1. Open the load balancer, select frontend IP pool, and click Add
      2. Enter the name of the new frontend IP pool (for example hana-frontend)
      3. Set the Assignment to Static and enter the IP address (for example 10.0.0.13)
      4. Click OK
      5. After the new frontend IP pool is created, write down its IP address
    2. Create a backend pool
      1. Open the load balancer, select backend pools, and click Add
      2. Enter the name of the new backend pool (for example hana-backend)
      3. Click Add a virtual machine
      4. Select the Availability Set you created earlier
      5. Select the virtual machines of the SAP HANA cluster
      6. Click OK
    3. Create a health probe
      1. Open the load balancer, select health probes, and click Add
      2. Enter the name of the new health probe (for example hana-hp)
      3. Select TCP as protocol, port 62503, keep Interval 5 and Unhealthy threshold 2
      4. Click OK
    4. SAP HANA 1.0: Create load balancing rules
      1. Open the load balancer, select load balancing rules and click Add
      2. Enter the name of the new load balancer rule (for example hana-lb-30315)
      3. Select the frontend IP address, backend pool, and health probe you created earlier (for example hana-frontend)
      4. Keep protocol TCP, enter port 30315
      5. Increase idle timeout to 30 minutes
      6. Make sure to enable Floating IP
      7. Click OK
      8. Repeat the steps above for port 30317
    5. SAP HANA 2.0: Create load balancing rules for system database
      1. Open the load balancer, select load balancing rules and click Add
      2. Enter the name of the new load balancer rule (for example hana-lb-30313)
      3. Select the frontend IP address, backend pool, and health probe you created earlier (for example hana-frontend)
      4. Keep protocol TCP, enter port 30313
      5. Increase idle timeout to 30 minutes
      6. Make sure to enable Floating IP
      7. Click OK
      8. Repeat the steps above for port 30314
    6. SAP HANA 2.0: Create load balancing rules for first the tenant database
      1. Open the load balancer, select load balancing rules and click Add
      2. Enter the name of the new load balancer rule (for example hana-lb-30340)
      3. Select the frontend IP address, backend pool, and health probe you created earlier (for example hana-frontend)
      4. Keep protocol TCP, enter port 30340
      5. Increase idle timeout to 30 minutes
      6. Make sure to enable Floating IP
      7. Click OK
      8. Repeat the steps above for port 30341 and 30342

For more information about the required ports for SAP HANA, read chapter Connections to Tenant Databases of the SAP HANA Tenant Databases guide or SAP Note 2388694.

Create Pacemaker cluster

Follow the steps in Setting up Pacemaker on SUSE Linux Enterprise Server in Azure to create a basic Pacemaker cluster for this HANA server. You can also use the same Pacemaker cluster for SAP HANA and SAP NetWeaver (A)SCS.

Installing SAP HANA

The following items are prefixed with either [A] - applicable to all nodes, [1] - only applicable to node 1 or [2] - only applicable to node 2 of the Pacemaker cluster.

  1. [A] Setup disk layout

    1. LVM

      We generally recommend using LVM for volumes that store data and log files. The example following assumes that the virtual machines have four data disks attached that should be used to create two volumes.

      List all available disks

      
      ls /dev/disk/azure/scsi1/lun*
      

      Example output

      /dev/disk/azure/scsi1/lun0  /dev/disk/azure/scsi1/lun1  /dev/disk/azure/scsi1/lun2  /dev/disk/azure/scsi1/lun3
      

      Create physical volumes for all disks that you want to use.

      
      sudo pvcreate /dev/disk/azure/scsi1/lun0
      sudo pvcreate /dev/disk/azure/scsi1/lun1
      sudo pvcreate /dev/disk/azure/scsi1/lun2
      sudo pvcreate /dev/disk/azure/scsi1/lun3
      

      Create a volume group for the data files, one volume group for the log files and one for the shared directory of SAP HANA

      
      sudo vgcreate vg_hana_data_HN1 /dev/disk/azure/scsi1/lun0 /dev/disk/azure/scsi1/lun1
      sudo vgcreate vg_hana_log_HN1 /dev/disk/azure/scsi1/lun2
      sudo vgcreate vg_hana_shared_HN1 /dev/disk/azure/scsi1/lun3
      

      Create the logical volumes

      
      sudo lvcreate -l 100%FREE -n hana_data vg_hana_data_HN1
      sudo lvcreate -l 100%FREE -n hana_log vg_hana_log_HN1
      sudo lvcreate -l 100%FREE -n hana_shared vg_hana_shared_HN1
      sudo mkfs.xfs /dev/vg_hana_data_HN1/hana_data
      sudo mkfs.xfs /dev/vg_hana_log_HN1/hana_log
      sudo mkfs.xfs /dev/vg_hana_shared_HN1/hana_shared
      

      Create the mount directories and copy the UUID of all logical volumes

      
      sudo mkdir -p /hana/data/HN1
      sudo mkdir -p /hana/log/HN1
      sudo mkdir -p /hana/shared/HN1
      # write down the ID of /dev/vg_hana_data_HN1/hana_data, /dev/vg_hana_log_HN1/hana_log and /dev/vg_hana_shared_HN1/hana_shared
      sudo blkid
      

      Create fstab entries for the three logical volumes

      
      sudo vi /etc/fstab
      

      Insert this line to /etc/fstab

      
      /dev/disk/by-uuid/<UUID of /dev/mapper/vg_hana_data_HN1-hana_data> /hana/data/HN1 xfs  defaults,nofail  0  2
      /dev/disk/by-uuid/<UUID of /dev/mapper/vg_hana_log_HN1-hana_log> /hana/log/HN1 xfs  defaults,nofail  0  2
      /dev/disk/by-uuid/<UUID of /dev/mapper/vg_hana_shared_HN1-hana_shared> /hana/shared/HN1 xfs  defaults,nofail  0  2
      

      Mount the new volumes

      
      sudo mount -a
      
    2. Plain Disks
      For demo systems, you can place your HANA data and log files on one disk. The following commands create a partition on /dev/disk/azure/scsi1/lun0 and format it with xfs.

      
      sudo sh -c 'echo -e "n\n\n\n\n\nw\n" | fdisk /dev/disk/azure/scsi1/lun0'
      sudo mkfs.xfs /dev/disk/azure/scsi1/lun0-part1
      
      # write down the ID of /dev/disk/azure/scsi1/lun0-part1
      sudo /sbin/blkid
      sudo vi /etc/fstab
      

      Insert this line to /etc/fstab

      
      /dev/disk/by-uuid/<UUID> /hana xfs  defaults,nofail  0  2
      

      Create the target directory and mount the disk.

      
      sudo mkdir /hana
      sudo mount -a
      
  2. [A] Setup host name resolution for all hosts
    You can either use a DNS server or modify the /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the hostname in the following commands

    sudo vi /etc/hosts
    

    Insert the following lines to /etc/hosts. Change the IP address and hostname to match your environment

    
    10.0.0.5 hn1-db-0
    10.0.0.6 hn1-db-1
    
  3. [A] Install HANA HA packages

    sudo zypper install SAPHanaSR
    

To install SAP HANA System Replication, follow chapter 4 of the SAP HANA SR Performance Optimized Scenario guide at https://www.suse.com/products/sles-for-sap/resource-library/sap-best-practices/

  1. [A] Run hdblcm from the HANA DVD

    • Choose installation -> 1
    • Select additional components for installation -> 1
    • Enter Installation Path [/hana/shared]: -> ENTER
    • Enter Local Host Name [..]: -> ENTER
    • Do you want to add additional hosts to the system? (y/n) [n]: -> ENTER
    • Enter SAP HANA System ID: <SID of HANA e.g. HN1>
    • Enter Instance Number [00]:
      HANA Instance number. Use 03 if you used the Azure Template or followed the manual deployment
    • Select Database Mode / Enter Index [1]: -> ENTER
    • Select System Usage / Enter Index [4]:
      Select the system Usage
    • Enter Location of Data Volumes [/hana/data/HN1]: -> ENTER
    • Enter Location of Log Volumes [/hana/log/HN1]: -> ENTER
    • Restrict maximum memory allocation? [n]: -> ENTER
    • Enter Certificate Host Name For Host '...' [...]: -> ENTER
    • Enter SAP Host Agent User (sapadm) Password:
    • Confirm SAP Host Agent User (sapadm) Password:
    • Enter System Administrator (hdbadm) Password:
    • Confirm System Administrator (hdbadm) Password:
    • Enter System Administrator Home Directory [/usr/sap/HN1/home]: -> ENTER
    • Enter System Administrator Login Shell [/bin/sh]: -> ENTER
    • Enter System Administrator User ID [1001]: -> ENTER
    • Enter ID of User Group (sapsys) [79]: -> ENTER
    • Enter Database User (SYSTEM) Password:
    • Confirm Database User (SYSTEM) Password:
    • Restart system after machine reboot? [n]: -> ENTER
    • Do you want to continue? (y/n):
      Validate the summary and enter y to continue
  2. [A] Upgrade SAP Host Agent
    Download the latest SAP Host Agent archive from the SAP Softwarecenter and run the following command to upgrade the agent. Replace the path to the archive to point to the file you downloaded.

    sudo /usr/sap/hostctrl/exe/saphostexec -upgrade -archive <path to SAP Host Agent SAR>
    

Configure SAP HANA 2.0 System Replication

The following items are prefixed with either [A] - applicable to all nodes, [1] - only applicable to node 1 or [2] - only applicable to node 2 of the Pacemaker cluster.

  1. [1] Create Tenant Database

    If you are using SAP HANA 2.0 or MDC, create a tenant database for your SAP NetWeaver system. Replace NW1 with the SID of your SAP system.

    Log in as <hanasid>adm and execute the following command

    
    hdbsql -u SYSTEM -p "passwd" -i 03 -d SYSTEMDB 'CREATE DATABASE NW1 SYSTEM USER PASSWORD "passwd"'
    
  2. [1] Configure System Replication on First Node

    Log in as <hanasid>adm and backup the databases

    
    hdbsql -d SYSTEMDB -u SYSTEM -p "passwd" -i 03 "BACKUP DATA USING FILE ('initialbackupSYS')"
    hdbsql -d HN1 -u SYSTEM -p "passwd" -i 03 "BACKUP DATA USING FILE ('initialbackupHN1')"
    hdbsql -d NW1 -u SYSTEM -p "passwd" -i 03 "BACKUP DATA USING FILE ('initialbackupNW1')"
    

    Copy the system PKI files to secondary

    
    scp /usr/sap/HN1/SYS/global/security/rsecssfs/data/SSFS_HN1.DAT hn1-db-1:/usr/sap/HN1/SYS/global/security/rsecssfs/data/
    scp /usr/sap/HN1/SYS/global/security/rsecssfs/key/SSFS_HN1.KEY hn1-db-1:/usr/sap/HN1/SYS/global/security/rsecssfs/key/
    

    Create the primary site.

    
    hdbnsutil -sr_enable –-name=SITE1
    
  3. [2] Configure System Replication on Second Node

    Register the second node to start the system replication. Log in as <hanasid>adm and run the following command

    
     sapcontrol -nr 03 -function StopWait 600 10
     hdbnsutil -sr_register --remoteHost=hn1-db-0 --remoteInstance=03 --replicationMode=sync --name=SITE2 
     

Configure SAP HANA 1.0 System Replication

  1. [1] Create the Required Users

    Log in as root and run the following command. Make sure to replace bold strings (HANA System ID HN1 and instance number 03) with the values of your SAP HANA installation.

    
     PATH="$PATH:/usr/sap/HN1/HDB03/exe"
     hdbsql -u system -i 03 'CREATE USER hdbhasync PASSWORD "passwd"' 
     hdbsql -u system -i 03 'GRANT DATA ADMIN TO hdbhasync' 
     hdbsql -u system -i 03 'ALTER USER hdbhasync DISABLE PASSWORD LIFETIME' 
     
  2. [A] Create keystore entry

    Log in as root and run the following command to create a new keystore entry.

    
     PATH="$PATH:/usr/sap/HN1/HDB03/exe"
     hdbuserstore SET hdbhaloc localhost:30315 hdbhasync passwd
     
  3. [1] Backup database

    Log in as root and backup the databases

    
    PATH="$PATH:/usr/sap/HN1/HDB03/exe"
    hdbsql -d SYSTEMDB -u system -i 03 "BACKUP DATA USING FILE ('initialbackup')"
    

    If you use a multi-tenant installation, also backup the tenant database

       
    hdbsql -d HN1 -u system -i 03 "BACKUP DATA USING FILE ('initialbackup')"
    
  4. [1] Configure System Replication on First Node

    Log in as <hanasid>adm and create the primary site.

    
     su - hdbadm
     hdbnsutil -sr_enable –-name=SITE1
     
  5. [2] Configure System Replication on Secondary Node.

    Log in as <hanasid>adm and register the secondary site.

    
     sapcontrol -nr 03 -function StopWait 600 10
     hdbnsutil -sr_register --remoteHost=hn1-db-0 --remoteInstance=03 --replicationMode=sync --name=SITE2 
     

Create SAP HANA cluster resources

First, create the HANA topology. Run the following commands on one of the Pacemaker cluster nodes.


   sudo crm configure property maintenance-mode=true

   # replace the bold string with your instance number and HANA system ID

   sudo crm configure primitive rsc_SAPHanaTopology_HN1_HDB03 ocf:suse:SAPHanaTopology \
     operations \$id="rsc_sap2_HN1_HDB03-operations" \
     op monitor interval="10" timeout="600" \
     op start interval="0" timeout="600" \
     op stop interval="0" timeout="300" \
     params SID="HN1" InstanceNumber="03"

   sudo crm configure clone cln_SAPHanaTopology_HN1_HDB03 rsc_SAPHanaTopology_HN1_HDB03 \
     meta is-managed="true" clone-node-max="1" target-role="Started" interleave="true"
   

Next, create the HANA resources.


   # replace the bold string with your instance number, HANA system ID and the frontend IP address of the Azure load balancer. 

   sudo crm configure primitive rsc_SAPHana_HN1_HDB03 ocf:suse:SAPHana \
     operations \$id="rsc_sap_HN1_HDB03-operations" \
     op start interval="0" timeout="3600" \
     op stop interval="0" timeout="3600" \
     op promote interval="0" timeout="3600" \
     op monitor interval="60" role="Master" timeout="700" \
     op monitor interval="61" role="Slave" timeout="700" \
     params SID="HN1" InstanceNumber="03" PREFER_SITE_TAKEOVER="true" \
     DUPLICATE_PRIMARY_TIMEOUT="7200" AUTOMATED_REGISTER="false"

   sudo crm configure ms msl_SAPHana_HN1_HDB03 rsc_SAPHana_HN1_HDB03 \
     meta is-managed="true" notify="true" clone-max="2" clone-node-max="1" \
     target-role="Started" interleave="true"

   sudo crm configure primitive rsc_ip_HN1_HDB03 ocf:heartbeat:IPaddr2 \
     meta target-role="Started" is-managed="true" \
     operations \$id="rsc_ip_HN1_HDB03-operations" \
     op monitor interval="10s" timeout="20s" \
     params ip="10.0.0.13"

   sudo crm configure primitive rsc_nc_HN1_HDB03 anything \
     params binfile="/usr/bin/nc" cmdline_options="-l -k 62503" \
     op monitor timeout=20s interval=10 depth=0

   sudo crm configure group g_ip_HN1_HDB03 rsc_ip_HN1_HDB03 rsc_nc_HN1_HDB03

   sudo crm configure colocation col_saphana_ip_HN1_HDB03 2000: g_ip_HN1_HDB03:Started \
     msl_SAPHana_HN1_HDB03:Master  

   sudo crm configure order ord_SAPHana_HN1_HDB03 2000: cln_SAPHanaTopology_HN1_HDB03 \
     msl_SAPHana_HN1_HDB03

   # Cleanup the HANA resources. The HANA resources might have failed because of a known issue.
   sudo crm resource cleanup rsc_SAPHana_HN1_HDB03

   sudo crm configure property maintenance-mode=false
   

Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.


   sudo crm_mon -r

   # Online: [ hn1-db-0 hn1-db-1 ]
   #
   # Full list of resources:
   #
   # stonith-sbd     (stonith:external/sbd): Started hn1-db-0
   # rsc_st_azure    (stonith:fence_azure_arm):      Started hn1-db-1
   # Clone Set: cln_SAPHanaTopology_HN1_HDB03 [rsc_SAPHanaTopology_HN1_HDB03]
   #     Started: [ hn1-db-0 hn1-db-1 ]
   # Master/Slave Set: msl_SAPHana_HN1_HDB03 [rsc_SAPHana_HN1_HDB03]
   #     Masters: [ hn1-db-0 ]
   #     Slaves: [ hn1-db-1 ]
   # Resource Group: g_ip_HN1_HDB03
   #     rsc_ip_HN1_HDB03   (ocf::heartbeat:IPaddr2):       Started hn1-db-0
   #     rsc_nc_HN1_HDB03   (ocf::heartbeat:anything):      Started hn1-db-0
   

Test cluster setup

This chapter describes how you can test your setup. Every test assumes that you are root and the SAP HANA master is running on the virtual machine hn1-db-0.

Fencing Test

You can test the setup of the fencing agent by disabling the network interface on node hn1-db-0.


sudo ifdown eth0

The virtual machine should now get restarted or stopped depending on your cluster configuration. If you set the stonith-action to off, the virtual machine is going to be stopped and the resources are migrated to the running virtual machine.

Once you start the virtual machine again, the SAP HANA resource fails to start as secondary if you set AUTOMATED_REGISTER="false". In this case, configure the HANA instance as secondary by executing this command:


su - hn1adm

# Stop the HANA instance just in case it is running
sapcontrol -nr 03 -function StopWait 600 10
hdbnsutil -sr_register --remoteHost=hn1-db-1 --remoteInstance=03 --replicationMode=sync --name=SITE1

# switch back to root and cleanup the failed state
exit
crm resource cleanup msl_SAPHana_HN1_HDB03 hn1-db-0

Testing a manual failover

You can test a manual failover by stopping the pacemaker service on node hn1-db-0.


service pacemaker stop

After the failover, you can start the service again. If you set AUTOMATED_REGISTER="false", the SAP HANA resource on hn1-db-0 fails to start as secondary. In this case, configure the HANA instance as secondary by executing this command:


service pacemaker start
su - hn1adm

# Stop the HANA instance just in case it is running
sapcontrol -nr 03 -function StopWait 600 10
hdbnsutil -sr_register --remoteHost=hn1-db-1 --remoteInstance=03 --replicationMode=sync --name=SITE1 


# Switch back to root and cleanup the failed state
exit
crm resource cleanup msl_SAPHana_HN1_HDB03 hn1-db-0

Testing a migration

You can migrate the SAP HANA master node by executing the following command


crm resource migrate msl_SAPHana_HN1_HDB03 hn1-db-1
crm resource migrate g_ip_HN1_HDB03 hn1-db-1

if you set AUTOMATED_REGISTER="false", this sequence of commands should migrate the SAP HANA master node and the group that contains the virtual IP address to hn1-db-1. The SAP HANA resource on hn1-db-0 fails to start as secondary. In this case, configure the HANA instance as secondary by executing this command:


su - hn1adm

# Stop the HANA instance just in case it is running
sapcontrol -nr 03 -function StopWait 600 10
hdbnsutil -sr_register --remoteHost=hn1-db-1 --remoteInstance=03 --replicationMode=sync --name=SITE1 

The migration creates location constraints that need to be deleted again.


crm configure edited

# Delete location constraints that are named like the following contraint. You should have two constraints, one for the SAP HANA resource and one for the IP address group.
location cli-prefer-g_ip_HN1_HDB03 g_ip_HN1_HDB03 role=Started inf: hn1-db-1

You also need to clean up the state of the secondary node resource


# Switch back to root and cleanup the failed state
exit
crm resource cleanup msl_SAPHana_HN1_HDB03 hn1-db-0

Next steps