Azure Virtual Machines high availability for SAP NetWeaver on Red Hat Enterprise Linux

This article describes how to deploy the virtual machines, configure the virtual machines, install the cluster framework, and install a highly available SAP NetWeaver 7.50 system. In the example configurations, installation commands etc. ASCS instance number 00, ERS instance number 02, and SAP System ID NW1 is used. The names of the resources (for example virtual machines, virtual networks) in the example assume that you have used the ASCS/SCS template with Resource Prefix NW1 to create the resources.

Read the following SAP Notes and papers first

Overview

To achieve high availability, SAP NetWeaver requires shared storage. GlusterFS is configured in a separate cluster and can be used by multiple SAP systems.

SAP NetWeaver High Availability overview

SAP NetWeaver ASCS, SAP NetWeaver SCS, SAP NetWeaver ERS, and the SAP HANA database use virtual hostname and virtual IP addresses. On Azure, a load balancer is required to use a virtual IP address. The following list shows the configuration of the (A)SCS and ERS load balancer.

Important

Multi-SID clustering of SAP ASCS/ERS with Red Hat Linux as guest operating system in Azure VMs is NOT supported. Multi-SID clustering describes the installation of multiple SAP ASCS/ERS instances with different SIDs in one Pacemaker cluster.

(A)SCS

  • Frontend configuration
    • IP address 10.0.0.7
  • Backend configuration
    • Connected to primary network interfaces of all virtual machines that should be part of the (A)SCS/ERS cluster
  • Probe Port
    • Port 620<nr>
  • Load-balancing rules
    • 32<nr> TCP
    • 36<nr> TCP
    • 39<nr> TCP
    • 81<nr> TCP
    • 5<nr>13 TCP
    • 5<nr>14 TCP
    • 5<nr>16 TCP

ERS

  • Frontend configuration
    • IP address 10.0.0.8
  • Backend configuration
    • Connected to primary network interfaces of all virtual machines that should be part of the (A)SCS/ERS cluster
  • Probe Port
    • Port 621<nr>
  • Load-balancing rules
    • 32<nr> TCP
    • 33<nr> TCP
    • 5<nr>13 TCP
    • 5<nr>14 TCP
    • 5<nr>16 TCP

Setting up GlusterFS

SAP NetWeaver requires shared storage for the transport and profile directory. Read GlusterFS on Azure VMs on Red Hat Enterprise Linux for SAP NetWeaver on how to set up GlusterFS for SAP NetWeaver.

Setting up (A)SCS

You can either use an Azure Template from GitHub to deploy all required Azure resources, including the virtual machines, availability set and load balancer or you can deploy the resources manually.

Deploy Linux via Azure Template

The Azure Marketplace contains an image for Red Hat Enterprise Linux that you can use to deploy new virtual machines. You can use one of the quickstart templates on GitHub to deploy all required resources. The template deploys the virtual machines, the load balancer, availability set etc. Follow these steps to deploy the template:

  1. Open the ASCS/SCS template on the Azure portal
  2. Enter the following parameters
    1. Resource Prefix
      Enter the prefix you want to use. The value is used as a prefix for the resources that are deployed.
    2. Stack Type
      Select the SAP NetWeaver stack type
    3. Os Type
      Select one of the Linux distributions. For this example, select RHEL 7
    4. Db Type
      Select HANA
    5. Sap System Count
      The number of SAP system that run in this cluster. Select 1.
    6. System Availability
      Select HA
    7. Admin Username, Admin Password or SSH key
      A new user is created that can be used to sign in to the machine.
    8. Subnet ID
      If you want to deploy the VM into an existing VNet where you have a subnet defined the VM should be assigned to, name the ID of that specific subnet. The ID usually looks like /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.Network/virtualNetworks/<virtual network name>/subnets/<subnet name>

Deploy Linux manually via Azure portal

You first need to create the virtual machines for this cluster. Afterwards, you create a load balancer and use the virtual machines in the backend pools.

  1. Create a Resource Group
  2. Create a Virtual Network
  3. Create an Availability Set
    Set max update domain
  4. Create Virtual Machine 1
    Use at least RHEL 7, in this example the Red Hat Enterprise Linux 7.4 image https://portal.azure.com/#create/RedHat.RedHatEnterpriseLinux74-ARM
    Select Availability Set created earlier
  5. Create Virtual Machine 2
    Use at least RHEL 7, in this example the Red Hat Enterprise Linux 7.4 image https://portal.azure.com/#create/RedHat.RedHatEnterpriseLinux74-ARM
    Select Availability Set created earlier
  6. Add at least one data disk to both virtual machines
    The data disks are used for the /usr/sap/<SAPSID> directory
  7. Create a Load Balancer (internal)
    1. Create the frontend IP addresses
      1. IP address 10.0.0.7 for the ASCS
        1. Open the load balancer, select frontend IP pool, and click Add
        2. Enter the name of the new frontend IP pool (for example nw1-ascs-frontend)
        3. Set the Assignment to Static and enter the IP address (for example 10.0.0.7)
        4. Click OK
      2. IP address 10.0.0.8 for the ASCS ERS
        • Repeat the steps above to create an IP address for the ERS (for example 10.0.0.8 and nw1-aers-backend)
    2. Create the backend pools
      1. Create a backend pool for the ASCS
        1. Open the load balancer, select backend pools, and click Add
        2. Enter the name of the new backend pool (for example nw1-ascs-backend)
        3. Click Add a virtual machine.
        4. Select the Availability Set you created earlier
        5. Select the virtual machines of the (A)SCS cluster
        6. Click OK
      2. Create a backend pool for the ASCS ERS
        • Repeat the steps above to create a backend pool for the ERS (for example nw1-aers-backend)
    3. Create the health probes
      1. Port 62000 for ASCS
        1. Open the load balancer, select health probes, and click Add
        2. Enter the name of the new health probe (for example nw1-ascs-hp)
        3. Select TCP as protocol, port 62000, keep Interval 5 and Unhealthy threshold 2
        4. Click OK
      2. Port 62102 for ASCS ERS
        • Repeat the steps above to create a health probe for the ERS (for example 62102 and nw1-aers-hp)
    4. Load-balancing rules
      1. 3200 TCP for ASCS
        1. Open the load balancer, select load-balancing rules and click Add
        2. Enter the name of the new load balancer rule (for example nw1-lb-3200)
        3. Select the frontend IP address, backend pool, and health probe you created earlier (for example nw1-ascs-frontend)
        4. Keep protocol TCP, enter port 3200
        5. Increase idle timeout to 30 minutes
        6. Make sure to enable Floating IP
        7. Click OK
      2. Additional ports for the ASCS
        • Repeat the steps above for ports 3600, 3900, 8100, 50013, 50014, 50016 and TCP for the ASCS
      3. Additional ports for the ASCS ERS
        • Repeat the steps above for ports 3302, 50213, 50214, 50216 and TCP for the ASCS ERS

Important

Do not enable TCP timestamps on Azure VMs placed behind Azure Load Balancer. Enabling TCP timestamps will cause the health probes to fail. Set parameter net.ipv4.tcp_timestamps to 0. For details see Load Balancer health probes.

Create Pacemaker cluster

Follow the steps in Setting up Pacemaker on Red Hat Enterprise Linux in Azure to create a basic Pacemaker cluster for this (A)SCS server.

Prepare for SAP NetWeaver installation

The following items are prefixed with either [A] - applicable to all nodes, [1] - only applicable to node 1 or [2] - only applicable to node 2.

  1. [A] Setup host name resolution

    You can either use a DNS server or modify the /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the hostname in the following commands

    sudo vi /etc/hosts
    

    Insert the following lines to /etc/hosts. Change the IP address and hostname to match your environment

    # IP addresses of the GlusterFS nodes
    10.0.0.40 glust-0
    10.0.0.41 glust-1
    10.0.0.42 glust-2
    # IP address of the load balancer frontend configuration for SAP NetWeaver ASCS
    10.0.0.7 nw1-ascs
    # IP address of the load balancer frontend configuration for SAP NetWeaver ASCS ERS
    10.0.0.8 nw1-aers
    
  2. [A] Create the shared directories

    sudo mkdir -p /sapmnt/NW1
    sudo mkdir -p /usr/sap/trans
    sudo mkdir -p /usr/sap/NW1/SYS
    sudo mkdir -p /usr/sap/NW1/ASCS00
    sudo mkdir -p /usr/sap/NW1/ERS02
    
    sudo chattr +i /sapmnt/NW1
    sudo chattr +i /usr/sap/trans
    sudo chattr +i /usr/sap/NW1/SYS
    sudo chattr +i /usr/sap/NW1/ASCS00
    sudo chattr +i /usr/sap/NW1/ERS02
    
  3. [A] Install GlusterFS client and other requirements

    sudo yum -y install glusterfs-fuse resource-agents resource-agents-sap
    
  4. [A] Check version of resource-agents-sap

    Make sure that the version of the installed resource-agents-sap package is at least 3.9.5-124.el7

    sudo yum info resource-agents-sap
    
    # Loaded plugins: langpacks, product-id, search-disabled-repos
    # Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
    # Installed Packages
    # Name        : resource-agents-sap
    # Arch        : x86_64
    # Version     : 3.9.5
    # Release     : 124.el7
    # Size        : 100 k
    # Repo        : installed
    # From repo   : rhel-sap-for-rhel-7-server-rpms
    # Summary     : SAP cluster resource agents and connector script
    # URL         : https://github.com/ClusterLabs/resource-agents
    # License     : GPLv2+
    # Description : The SAP resource agents and connector script interface with
    #          : Pacemaker to allow SAP instances to be managed in a cluster
    #          : environment.
    
  5. [A] Add mount entries

    sudo vi /etc/fstab
    
    # Add the following lines to fstab, save and exit
    glust-0:/NW1-sapmnt /sapmnt/NW1 glusterfs backup-volfile-servers=glust-1:glust-2 0 0
    glust-0:/NW1-trans /usr/sap/trans glusterfs backup-volfile-servers=glust-1:glust-2 0 0
    glust-0:/NW1-sys /usr/sap/NW1/SYS glusterfs backup-volfile-servers=glust-1:glust-2 0 0
    

    Mount the new shares

    sudo mount -a
    
  6. [A] Configure SWAP file

    sudo vi /etc/waagent.conf
    
    # Set the property ResourceDisk.EnableSwap to y
    # Create and use swapfile on resource disk.
    ResourceDisk.EnableSwap=y
    
    # Set the size of the SWAP file with property ResourceDisk.SwapSizeMB
    # The free space of resource disk varies by virtual machine size. Make sure that you do not set a value that is too big. You can check the SWAP space with command swapon
    # Size of the swapfile.
    ResourceDisk.SwapSizeMB=2000
    

    Restart the Agent to activate the change

    sudo service waagent restart
    
  7. [A] RHEL configuration

    Configure RHEL as described in SAP Note 2002167

Installing SAP NetWeaver ASCS/ERS

  1. [1] Create a virtual IP resource and health-probe for the ASCS instance

    sudo pcs node standby nw1-cl-1
    
    sudo pcs resource create fs_NW1_ASCS Filesystem device='glust-0:/NW1-ascs' \
      directory='/usr/sap/NW1/ASCS00' fstype='glusterfs' \
      options='backup-volfile-servers=glust-1:glust-2' \
      --group g-NW1_ASCS
    
    sudo pcs resource create vip_NW1_ASCS IPaddr2 \
      ip=10.0.0.7 cidr_netmask=24 \
      --group g-NW1_ASCS
    
    sudo pcs resource create nc_NW1_ASCS azure-lb port=62000 \
      --group g-NW1_ASCS
    

    Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

    sudo pcs status
    
    # Node nw1-cl-1: standby
    # Online: [ nw1-cl-0 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
    
  2. [1] Install SAP NetWeaver ASCS

    Install SAP NetWeaver ASCS as root on the first node using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the ASCS, for example nw1-ascs, 10.0.0.7 and the instance number that you used for the probe of the load balancer, for example 00.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
    sudo firewall-cmd --zone=public  --add-port=4237/tcp
    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin
    

    If the installation fails to create a subfolder in /usr/sap/NW1/ASCS00, try setting the owner and group of the ASCS00 folder and retry.

    sudo chown nw1adm /usr/sap/NW1/ASCS00
    sudo chgrp sapsys /usr/sap/NW1/ASCS00
    
  3. [1] Create a virtual IP resource and health-probe for the ERS instance

    sudo pcs node unstandby nw1-cl-1
    sudo pcs node standby nw1-cl-0
    
    sudo pcs resource create fs_NW1_AERS Filesystem device='glust-0:/NW1-aers' \
      directory='/usr/sap/NW1/ERS02' fstype='glusterfs' \
      options='backup-volfile-servers=glust-1:glust-2' \
     --group g-NW1_AERS
    
    sudo pcs resource create vip_NW1_AERS IPaddr2 \
      ip=10.0.0.8 cidr_netmask=24 \
     --group g-NW1_AERS
    
    sudo pcs resource create nc_NW1_AERS azure-lb port=62102 \
     --group g-NW1_AERS
    

    Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

    sudo pcs status
    
    # Node nw1-cl-0: standby
    # Online: [ nw1-cl-1 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-1
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
    
  4. [2] Install SAP NetWeaver ERS

    Install SAP NetWeaver ERS as root on the second node using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the ERS, for example nw1-aers, 10.0.0.8 and the instance number that you used for the probe of the load balancer, for example 02.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
    sudo firewall-cmd --zone=public  --add-port=4237/tcp
    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin
    

    If the installation fails to create a subfolder in /usr/sap/NW1/ERS02, try setting the owner and group of the ERS02 folder and retry.

    sudo chown nw1adm /usr/sap/NW1/ERS02
    sudo chgrp sapsys /usr/sap/NW1/ERS02
    
  5. [1] Adapt the ASCS/SCS and ERS instance profiles

    • ASCS/SCS profile
    sudo vi /sapmnt/NW1/profile/NW1_ASCS00_nw1-ascs
    
    # Change the restart command to a start command
    #Restart_Program_01 = local $(_EN) pf=$(_PF)
    Start_Program_01 = local $(_EN) pf=$(_PF)
    
    # Add the keep alive parameter
    enque/encni/set_so_keepalive = true
    
    • ERS profile
    sudo vi /sapmnt/NW1/profile/NW1_ERS02_nw1-aers
    
    # Change the restart command to a start command
    #Restart_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)
    Start_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)
    
    # remove Autostart from ERS profile
    # Autostart = 1
    
  6. [A] Configure Keep Alive

    The communication between the SAP NetWeaver application server and the ASCS/SCS is routed through a software load balancer. The load balancer disconnects inactive connections after a configurable timeout. To prevent this, you need to set a parameter in the SAP NetWeaver ASCS/SCS profile and change the Linux system settings. Read SAP Note 1410736 for more information.

    The ASCS/SCS profile parameter enque/encni/set_so_keepalive was already added in the last step.

    # Change the Linux system configuration
    sudo sysctl net.ipv4.tcp_keepalive_time=120
    
  7. [A] Update the /usr/sap/sapservices file

    To prevent the start of the instances by the sapinit startup script, all instances managed by Pacemaker must be commented out from /usr/sap/sapservices file. Do not comment out the SAP HANA instance if it will be used with HANA SR.

    
    sudo vi /usr/sap/sapservices
    
    # On the node where you installed the ASCS, comment out the following line
    # LD_LIBRARY_PATH=/usr/sap/NW1/ASCS00/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/NW1/ASCS00/exe/sapstartsrv pf=/usr/sap/NW1/SYS/profile/NW1_ASCS00_nw1-ascs -D -u nw1adm
    
    # On the node where you installed the ERS, comment out the following line
    # LD_LIBRARY_PATH=/usr/sap/NW1/ERS02/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/NW1/ERS02/exe/sapstartsrv pf=/usr/sap/NW1/ERS02/profile/NW1_ERS02_nw1-aers -D -u nw1adm
    
  8. [1] Create the SAP cluster resources

If using enqueue server 1 architecture (ENSA1), define the resources as follows:

sudo pcs property set maintenance-mode=true
   
   sudo pcs resource create rsc_sap_NW1_ASCS00 SAPInstance \
    InstanceName=NW1_ASCS00_nw1-ascs START_PROFILE="/sapmnt/NW1/profile/NW1_ASCS00_nw1-ascs" \
    AUTOMATIC_RECOVER=false \
    meta resource-stickiness=5000 migration-threshold=1 \
    --group g-NW1_ASCS
   
   sudo pcs resource create rsc_sap_NW1_ERS02 SAPInstance \
    InstanceName=NW1_ERS02_nw1-aers START_PROFILE="/sapmnt/NW1/profile/NW1_ERS02_nw1-aers" \
    AUTOMATIC_RECOVER=false IS_ERS=true \
    --group g-NW1_AERS
      
   sudo pcs constraint colocation add g-NW1_AERS with g-NW1_ASCS -5000
   sudo pcs constraint location rsc_sap_NW1_ASCS00 rule score=2000 runs_ers_NW1 eq 1
   sudo pcs constraint order g-NW1_ASCS then g-NW1_AERS kind=Optional symmetrical=false
   
   sudo pcs node unstandby nw1-cl-0
   sudo pcs property set maintenance-mode=false
   

SAP introduced support for enqueue server 2, including replication, as of SAP NW 7.52. Starting with ABAP Platform 1809, enqueue server 2 is installed by default. See SAP note 2630416 for enqueue server 2 support. If using enqueue server 2 architecture (ENSA2), install resource agent resource-agents-sap-4.1.1-12.el7.x86_64 or newer and define the resources as follows:

sudo pcs property set maintenance-mode=true
   
   sudo pcs resource create rsc_sap_NW1_ASCS00 SAPInstance \
    InstanceName=NW1_ASCS00_nw1-ascs START_PROFILE="/sapmnt/NW1/profile/NW1_ASCS00_nw1-ascs" \
    AUTOMATIC_RECOVER=false \
    meta resource-stickiness=5000 \
    --group g-NW1_ASCS
   
   sudo pcs resource create rsc_sap_NW1_ERS02 SAPInstance \
    InstanceName=NW1_ERS02_nw1-aers START_PROFILE="/sapmnt/NW1/profile/NW1_ERS02_nw1-aers" \
    AUTOMATIC_RECOVER=false IS_ERS=true \
    --group g-NW1_AERS
      
   sudo pcs constraint colocation add g-NW1_AERS with g-NW1_ASCS -5000
   sudo pcs constraint order g-NW1_ASCS then g-NW1_AERS kind=Optional symmetrical=false
   
   sudo pcs node unstandby nw1-cl-0
   sudo pcs property set maintenance-mode=false
   

If you are upgrading from an older version and switching to enqueue server 2, see SAP note 2641322.

Make sure that the cluster status is ok and that all resources are started. It is not important on which node the resources are running.

sudo pcs status
   
   # Online: [ nw1-cl-0 nw1-cl-1 ]
   #
   # Full list of resources:
   #
   # rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
   #  Resource Group: g-NW1_ASCS
   #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
   #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
   #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
   #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
   #  Resource Group: g-NW1_AERS
   #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
   #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
   #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
   #      rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
   
  1. [A] Add firewall rules for ASCS and ERS on both nodes

    # Probe Port of ASCS
    sudo firewall-cmd --zone=public --add-port=62000/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=62000/tcp
    sudo firewall-cmd --zone=public --add-port=3200/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=3200/tcp
    sudo firewall-cmd --zone=public --add-port=3600/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=3600/tcp
    sudo firewall-cmd --zone=public --add-port=3900/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=3900/tcp
    sudo firewall-cmd --zone=public --add-port=8100/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=8100/tcp
    sudo firewall-cmd --zone=public --add-port=50013/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50013/tcp
    sudo firewall-cmd --zone=public --add-port=50014/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50014/tcp
    sudo firewall-cmd --zone=public --add-port=50016/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50016/tcp
    # Probe Port of ERS
    sudo firewall-cmd --zone=public --add-port=62102/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=62102/tcp
    sudo firewall-cmd --zone=public --add-port=3302/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=3302/tcp
    sudo firewall-cmd --zone=public --add-port=50213/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50213/tcp
    sudo firewall-cmd --zone=public --add-port=50214/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50214/tcp
    sudo firewall-cmd --zone=public --add-port=50216/tcp --permanent
    sudo firewall-cmd --zone=public --add-port=50216/tcp
    

SAP NetWeaver application server preparation

Some databases require that the database instance installation is executed on an application server. Prepare the application server virtual machines to be able to use them in these cases.

The steps bellow assume that you install the application server on a server different from the ASCS/SCS and HANA servers. Otherwise some of the steps below (like configuring host name resolution) are not needed.

  1. Setup host name resolution

    You can either use a DNS server or modify the /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the hostname in the following commands

    sudo vi /etc/hosts
    

    Insert the following lines to /etc/hosts. Change the IP address and hostname to match your environment

    # IP addresses of the GlusterFS nodes
    10.0.0.40 glust-0
    10.0.0.41 glust-1
    10.0.0.42 glust-2
    # IP address of the load balancer frontend configuration for SAP NetWeaver ASCS
    10.0.0.7 nw1-ascs
    # IP address of the load balancer frontend configuration for SAP NetWeaver ASCS ERS
    10.0.0.8 nw1-aers
    # IP address of the load balancer frontend configuration for database
    10.0.0.13 nw1-db
    
  2. Create the sapmnt directory

    sudo mkdir -p /sapmnt/NW1
    sudo mkdir -p /usr/sap/trans
    
    sudo chattr +i /sapmnt/NW1
    sudo chattr +i /usr/sap/trans
    
  3. Install GlusterFS client and other requirements

    sudo yum -y install glusterfs-fuse uuidd
    
  4. Add mount entries

    sudo vi /etc/fstab
    
    # Add the following lines to fstab, save and exit
    glust-0:/NW1-sapmnt /sapmnt/NW1 glusterfs backup-volfile-servers=glust-1:glust-2 0 0
    glust-0:/NW1-trans /usr/sap/trans glusterfs backup-volfile-servers=glust-1:glust-2 0 0
    

    Mount the new shares

    sudo mount -a
    
  5. Configure SWAP file

    
    sudo vi /etc/waagent.conf
    
    # Set the property ResourceDisk.EnableSwap to y
    # Create and use swapfile on resource disk.
    ResourceDisk.EnableSwap=y
    
    # Set the size of the SWAP file with property ResourceDisk.SwapSizeMB
    # The free space of resource disk varies by virtual machine size. Make sure that you do not set a value that is too big. You can check the SWAP space with command swapon
    # Size of the swapfile.
    ResourceDisk.SwapSizeMB=2000
    

    Restart the Agent to activate the change

    
    sudo service waagent restart
    

Install database

In this example, SAP NetWeaver is installed on SAP HANA. You can use every supported database for this installation. For more information on how to install SAP HANA in Azure, see High availability of SAP HANA on Azure VMs on Red Hat Enterprise Linux. For a list of supported databases, see SAP Note 1928533.

  1. Run the SAP database instance installation

    Install the SAP NetWeaver database instance as root using a virtual hostname that maps to the IP address of the load balancer frontend configuration for the database for example nw1-db and 10.0.0.13.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin
    

SAP NetWeaver application server installation

Follow these steps to install an SAP application server.

  1. Prepare application server

    Follow the steps in the chapter SAP NetWeaver application server preparation above to prepare the application server.

  2. Install SAP NetWeaver application server

    Install a primary or additional SAP NetWeaver applications server.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a non-root user to connect to sapinst.

    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin
    
  3. Update SAP HANA secure store

    Update the SAP HANA secure store to point to the virtual name of the SAP HANA System Replication setup.

    Run the following command to list the entries as <sapsid>adm

    hdbuserstore List
    

    This should list all entries and should look similar to

    
    DATA FILE       : /home/nw1adm/.hdb/nw1-di-0/SSFS_HDB.DAT
    KEY FILE        : /home/nw1adm/.hdb/nw1-di-0/SSFS_HDB.KEY
    
    KEY DEFAULT
      ENV : 10.0.0.14:30313
      USER: SAPABAP1
      DATABASE: NW1
    

    The output shows that the IP address of the default entry is pointing to the virtual machine and not to the load balancer's IP address. This entry needs to be changed to point to the virtual hostname of the load balancer. Make sure to use the same port (30313 in the output above) and database name (HN1 in the output above)!

    su - nw1adm
    hdbuserstore SET DEFAULT nw1-db:30313@NW1 SAPABAP1 <password of ABAP schema>
    

Test the cluster setup

  1. Manually migrate the ASCS instance

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    

    Run the following commands as root to migrate the ASCS instance.

    [root@nw1-cl-0 ~]# pcs resource move rsc_sap_NW1_ASCS00
    
    [root@nw1-cl-0 ~]# pcs resource clear rsc_sap_NW1_ASCS00
    
    # Remove failed actions for the ERS that occurred as part of the migration
    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ERS02
    

    Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
    
  2. Simulate node crash

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
    

    Run the following command as root on the node where the ASCS instance is running

    [root@nw1-cl-1 ~]# echo b > /proc/sysrq-trigger
    

    The status after the node is started again should look like this.

    Online: [ nw1-cl-0 nw1-cl-1 ]
    
    Full list of resources:
    
    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    
    Failed Actions:
    * rsc_sap_NW1_ERS02_monitor_11000 on nw1-cl-0 'not running' (7): call=45, status=complete, exitreason='',
        last-rc-change='Tue Aug 21 13:52:39 2018', queued=0ms, exec=0ms
    

    Use the following command to clean the failed resources.

    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ERS02
    

    Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    
  3. Kill message server process

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    

    Run the following commands as root to identify the process of the message server and kill it.

    [root@nw1-cl-0 ~]# pgrep ms.sapNW1 | xargs kill -9
    

    If you only kill the message server once, it will be restarted by sapstart. If you kill it often enough, Pacemaker will eventually move the ASCS instance to the other node. Run the following commands as root to clean up the resource state of the ASCS and ERS instance after the test.

    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ASCS00
    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ERS02
    

    Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
    
  4. Kill enqueue server process

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
    

    Run the following commands as root on the node where the ASCS instance is running to kill the enqueue server.

    [root@nw1-cl-1 ~]# pgrep en.sapNW1 | xargs kill -9
    

    The ASCS instance should immediately fail over to the other node. The ERS instance should also fail over after the ASCS instance is started. Run the following commands as root to clean up the resource state of the ASCS and ERS instance after the test.

    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ASCS00
    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ERS02
    

    Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    
  5. Kill enqueue replication server process

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    

    Run the following command as root on the node where the ERS instance is running to kill the enqueue replication server process.

    [root@nw1-cl-1 ~]# pgrep er.sapNW1 | xargs kill -9
    

    If you only run the command once, sapstart will restart the process. If you run it often enough, sapstart will not restart the process and the resource will be in a stopped state. Run the following commands as root to clean up the resource state of the ERS instance after the test.

    [root@nw1-cl-0 ~]# pcs resource cleanup rsc_sap_NW1_ERS02
    

    Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    
  6. Kill enqueue sapstartsrv process

    Resource state before starting the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    

    Run the following commands as root on the node where the ASCS is running.

    [root@nw1-cl-0 ~]# pgrep -fl ASCS00.*sapstartsrv
    # 59545 sapstartsrv
    
    [root@nw1-cl-0 ~]# kill -9 59545
    

    The sapstartsrv process should always be restarted by the Pacemaker resource agent as part of the monitoring. Resource state after the test:

    rsc_st_azure    (stonith:fence_azure_arm):      Started nw1-cl-0
     Resource Group: g-NW1_ASCS
         fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started nw1-cl-0
         nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started nw1-cl-0
         vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-0
         rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started nw1-cl-0
     Resource Group: g-NW1_AERS
         fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started nw1-cl-1
         nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started nw1-cl-1
         vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started nw1-cl-1
         rsc_sap_NW1_ERS02  (ocf::heartbeat:SAPInstance):   Started nw1-cl-1
    

Next steps