Troubleshoot backup of SAP HANA databases on Azure

This article provides troubleshooting information for backing up SAP HANA databases on Azure virtual machines. For more information on the SAP HANA backup scenarios we currently support, see Scenario support.

Prerequisites and Permissions

Refer to the prerequisites and What the pre-registration script does sections before configuring backups.

Common user errors

UserErrorHANAInternalRoleNotPresent

Error message Azure Backup does not have required role privileges to carry out Backup and Restore operations
Possible causes All operations will fail with this error when the Backup user (AZUREWLBACKUPHANAUSER) doesn't have the SAP_INTERNAL_HANA_SUPPORT role assigned or the role may have been overwritten.
Recommended action Download and run the pre-registration script on the SAP HANA instance, or manually assign the SAP_INTERNAL_HANA_SUPPORT role to the Backup user (AZUREWLBACKUPHANAUSER).

Note

If you are using HANA 2.0 SPS04 Rev 46 and later, this error doesn't occur as the use of the SAP_INTERNAL_HANA_SUPPORT role is deprecated in these HANA versions.

UserErrorInOpeningHanaOdbcConnection

Error message Failed to connect to HANA system
Possible causes
  • Connection to HANA instance failed
  • System DB is offline
  • Tenant DB is offline
  • Backup user (AZUREWLBACKUPHANAUSER) doesn't have enough permissions/privileges.
Recommended action Check if the system is up and running. If the database(s) is up and running, ensure that the required permissions are set by downloading and running the pre-registration script on the SAP HANA instance.

UserErrorHanaInstanceNameInvalid

Error message The specified SAP HANA instance is either invalid or can't be found
Possible causes
  • The specified SAP HANA instance is either invalid or can't be found.
  • Multiple SAP HANA instances on a single Azure VM can't be backed up.
Recommended action
  • Ensure that only one HANA instance is running on the Azure VM.
  • Run the script from the Discover DB pane (you can also find this here) with the correct SAP HANA instance to resolve the issue.

UserErrorHANALSNValidationFailure

Error message Backup log chain is broken
Possible causes HANA LSN Log chain break can be triggered for various reasons, including:
  • Azure Storage call failure to commit backup.
  • The Tenant DB is offline.
  • Extension upgrade has terminated an in-progress Backup job.
  • Unable to connect to Azure Storage during backup.
  • SAP HANA has rolled back a transaction in the backup process.
  • A backup is complete, but catalog is not yet updated with success in HANA system.
  • Backup failed from Azure Backup perspective, but success from HANA's perspective - the log backup/catalog destination may have been updated from backint to file system, or the backint executable may have been changed.
Recommended action To resolve this issue, Azure Backup triggers an auto-heal Full backup. While this auto-heal backup is in progress, all log backups are triggered by HANA fail with OperationCancelledBecauseConflictingAutohealOperationRunningUserError. Once the auto-heal Full backup is complete, logs and all other backups will start working as expected.
If you do not see an auto-heal full backup triggered or any successful backup (Full/Differential/ Incremental) in 24 hours, contact Microsoft support.

UserErrorSDCtoMDCUpgradeDetected

Error message SDC to MDC upgrade detected.
Possible causes When an SDC system is upgraded to MDC, backups fail with this error.
Recommended action To troubleshoot and resolve the issue, see SDC to MDC upgrade.

UserErrorInvalidBackintConfiguration

Error message Backups will fail with this error when the Backint Configuration is incorrectly updated.
Possible causes The backint configuration updated during the Configure Protection flow by Azure Backup is either altered/updated by the customer.
Recommended action Check if the following (backint) parameters are set:
  • [catalog_backup_using_backint:true]
  • [enable_accumulated_catalog_backup:false]
  • [parallel_data_backup_backint_channels:1]
  • [log_backup_timeout_s:900)]
  • [backint_response_timeout:7200]
If backint-based parameters are present at the HOST level, remove them. However, if parameters aren't present at the HOST level but have been manually modified at a database level, ensure that the database level values are set above. Or, run stop protection with retain backup data from the Azure portal, and then select Resume backup.

UserErrorIncompatibleSrcTargetSystemsForRestore

Error message The source and target systems for restore are incompatible.
Possible causes The restore flow fails with this error when the source and target HANA databases, and systems are incompatible.
Recommended action Ensure that your restore scenario isn't in the following list of possible incompatible restores:
Case 1: SYSTEMDB cannot be renamed during restore.
Case 2: Source - SDC and target - MDC: The source database cannot be restored as SYSTEMDB or tenant DB on the target.
Case 3: Source - MDC and target - SDC: The source database (SYSTEMDB or tenant DB) cannot be restored to the target.
To learn more, see the note 1642148 in the SAP support launchpad.

UserErrorHANAPODoesNotExist

Error message Database configured for backup does not exist.
Possible causes If a database that has been configured for backup is deleted, then all scheduled and ad-hoc backups on this database will fail.
Recommended action Verify if the database is deleted. Re-create the database or stop protection (with or without retain data) for the database.

UserErrorInsufficientPrivilegeOfDatabaseUser

Error message Azure Backup does not have enough privileges to carry out Backup and Restore operations.
Possible causes Backup user (AZUREWLBACKUPHANAUSER) created by the pre-registration script doesn't have one or more of the following roles assigned:
  • For MDC, DATABASE ADMIN and BACKUP ADMIN (for HANA 2.0 SPS05 and later) to create new databases during restore.
  • For SDC, BACKUP ADMIN to create new databases during restore.
  • CATALOG READ to read the backup catalog.
  • SAP_INTERNAL_HANA_SUPPORT to access a few private tables. Only required for SDC and MDC versions prior to HANA 2.0 SPS04 Rev 46. This is not required for HANA 2.0 SPS04 Rev 46 and later as we are getting the required information from public tables now with the fix from HANA team.
Recommended action To resolve the issue, add the required roles and permissions manually to the Backup user (AZUREWLBACKUPHANAUSER), or download and run the pre-registration script on the SAP HANA instance.

UserErrorDatabaseUserPasswordExpired

Error message Database/Backup user's password expired.
Possible causes The Database/Backup user created by the pre-registration script doesn't set expiry for the password. However, if it was altered, you may see this error.
Recommended action Download and run the pre-registration script on the SAP HANA instance to resolve the issue.

UserErrorInconsistentSSFS

Error message SAP HANA error
Possible causes Inconsistent Secure Storage File System (SSFS) error received from SAP HANA Engine.
Recommended action Work with the SAP HANA team to fix this issue. To learn more, see the SAP note 0002097613.

UserErrorCannotConnectToAzureActiveDirectoryService

Error message Unable to connect to the AAD service from the HANA system.
Possible causes Firewall or proxy settings as Backup extension's plugin service account is not allowing the outbound connection to AAD.
Recommended action Fix the firewall or proxy settings for the outbound connection to AAD to succeed.

UserErrorMisConfiguredSslCaStore

Error message Misconfigured CA store
Possible causes Backup extension's plugin host process is unable to access the root CA store (in /var/lib/ca-certificates/ca-bundle.pem for SLES).
Recommended action Fix the CA store issue by using chmod o+r to restore the original permission. Then restart the plugin host service for Backups and Restores to succeed.

UserErrorBackupFailedAsRemedialBackupInProgress

Error message Remedial Backup in progress.
Possible causes Azure Backup triggers a remedial full backup to handle LSN log chain break. While this remedial full is in progress, backups (Full/ Differential/Incremental) triggered through the portal/CLI fails with this error.
Recommended action Wait for the remedial full backup to complete successfully before triggering another backup.

OperationCancelledBecauseConflictingOperationRunningUserError

Error message Conflicting operation in progress.
Possible causes A Full/Differential/Incremental backup triggered through portal/CLI/native HANA clients, while another Full/Differential/Incremental backup is already in progress.
Recommended action Wait for the active backup job to complete before triggering a new Full/delta backup.

OperationCancelledBecauseConflictingAutohealOperationRunning UserError

Error message Auto-heal Full backup in progress.
Possible causes Azure Backup triggers an auto-heal Full backup to resolve UserErrorHANALSNValidationFailure. While this auto-heal backup is in progress, all the log backups triggered by HANA fail with OperationCancelledBecauseConflictingAutohealOperationRunningUserError.
Once the auto-heal Full backup is complete, logs and all other backups will start working as expected.
Recommended action Wait for the auto-heal Full backup to complete before triggering a new Full/delta backup.

UserErrorHanaPreScriptNotRun

Error message Pre-registration script not run.
Possible causes The SAP HANA pre-registration script for setting up the environment has not been run.
Recommended action Download and run the pre-registration script on the SAP HANA instance.

UserErrorTargetPOExistsOverwriteNotSpecified

Error message Target database cannot be overwritten for Restore.
Possible causes Target database exists, but can't be overwritten. Force overwrite isn't set in the Restore flow on portal/CLI.
Recommended action Restore database with the force overwrite option selected, or restore to a different target database.

UserErrorRecoverySysScriptFailedToTriggerRestore

Error message RecoverySys.py could not be run successfully to restore System DB.
Possible causes Possible causes for System DB restore to fail are:
  • Azure Backup is unable to find Recoverysys.py on the HANA machine. This happens when the HANA environment isn't set up properly.
  • Recoverysys.py is present, but triggering this script has failed to invoke HANA to perform the restore.
  • Recoverysys.py has successfully invoked HANA to perform the restore, but HANA fails to restore.
Recommended action
  • For issue 1, work with the SAP HANA team to fix the issue.
  • For 2 and 3, see the log trace by running the HDSetting.sh command in sid-adm prompt. For example, /usr/sap/SID/HDB00/HDBSetting.sh.
Share these findings with the SAP HANA team to get the issue fixed.

UserErrorDBNameNotInCorrectFormat

Error message Restored database name not in correct format.
Possible causes The Restored database name that you have provided is not in the acceptable/expected format.
Recommended action Ensure that the restored database name starts with a letter and shouldn't contain any symbol, other than digits or an underscore.
It can contain a maximum of 127 characters only and must not begin with "_SYS_".

UserErrorDefaultSidAdmDirectoryChanged

Error message Default sid-adm directory changed.
Possible causes The default sid-adm directory was changed, and HDBSetting.sh is not available in this default directory.
Recommended action If HXE is the SID, ensure that environment variable HOME is set to /usr/sap/HXE/home as sid-adm user.

UserErrorHDBsettingsScriptNotFound

Error message HDBSetting.sh file cannot be found.
Possible causes System databases restore failed as the <sid>adm user environment couldn't find the HDBsettings.sh file to trigger restore.
Recommended action Work with the SAP HANA team to fix this issue.

If HXE is the SID, ensure that environment variable HOME is set to /usr/sap/HXE/home as sid-adm user.

Restore checks

Single Container Database (SDC) restore

Take care of inputs while restoring a single container database (SDC) for HANA to another SDC machine. The database name should be given with lowercase and with "sdc" appended in brackets. The HANA instance will be displayed in capitals.

Assume an SDC HANA instance "H21" is backed up. The backup items page will show the backup item name as "h21(sdc)". If you attempt to restore this database to another target SDC, say H11, then following inputs need to be provided.

Restored SDC database name

Note the following points:

  • By default, the restored db name will be populated with the backup item name. In this case, h21(sdc).
  • Selecting the target as H11 won't change the restored db name automatically. It should be edited to h11(sdc). Regarding SDC, the restored db name will be the target instance ID with lowercase letters and 'sdc' appended in brackets.
  • Since SDC can have only single database, you also need to select the checkbox to allow override of the existing database data with the recovery point data.
  • Linux is case-sensitive. So be careful to preserve the case.

Multiple Container Database (MDC) restore

In multiple container databases for HANA, the standard configuration is SYSTEMDB + 1 or more Tenant DBs. Restoring an entire SAP HANA instance means to restore both SYSTEMDB and Tenant DBs. One restores SYSTEMDB first and then proceeds for Tenant DB. System DB essentially means to override the system information on the selected target. This restore also overrides the BackInt related information in the target instance. So after the system DB is restored to a target instance, run the pre-registration script again. Only then the subsequent tenant DB restores will succeed.

Back up a replicated VM

Scenario 1

The original VM was replicated using Azure Site Recovery or Azure VM backup. The new VM was built to simulate the old VM. That is, the settings are exactly the same. (This is because the original VM was deleted and the restore was done from VM backup or Azure Site Recovery).

This scenario could include two possible cases. Learn how to back up the replicated VM in both of these cases:

  1. The new VM created has the same name, and is in the same resource group and subscription as the deleted VM.

    • The extension is already present on the VM, but isn't visible to any of the services
    • Run the pre-registration script
    • Re-register the extension for the same machine in the Azure portal (Backup -> View details -> Select the relevant Azure VM -> Re-register)
    • The already existing backed up databases (from the deleted VM) should then start successfully being backed up
  2. The new VM created has either:

    • a different name than the deleted VM
    • the same name as the deleted VM but is in a different resource group or subscription (as compared to the deleted VM)

    If this is the case, then do the following steps:

    • The extension is already present on the VM, but isn't visible to any of the services
    • Run the pre-registration script
    • If you discover and protect the new databases, you'll start seeing duplicate active databases in the portal. To avoid this, Stop protection with retain data for the old databases. Then continue with the remaining steps.
    • Discover the databases to enable backup
    • Enable backups on these databases
    • The already existing backed up databases (from the deleted VM) will continue to be stored in the vault (with their backups being retained according to the policy)

Scenario 2

The original VM was replicated using Azure Site Recovery or Azure VM backup. The new VM was built out of the content – to be used as a template. This is a new VM with a new SID.

Follow these steps to enable backups on the new VM:

  • The extension is already present on the VM, but not visible to any of the services
  • Run the pre-registration script. Based on the SID of the new VM, two scenarios can arise:
    • The original VM and the new VM have the same SID. The pre-registration script will run successfully.
    • The original VM and the new VM have different SIDs. The pre-registration script will fail. Contact support to get help in this scenario.
  • Discover the databases that you want to back up
  • Enable backups on these databases

SDC version upgrade or MDC version upgrade on the same VM

Upgrades to the OS, SDC version change, or MDC version change that don't cause a SID change can be handled as follows:

  • Ensure that the new OS version, SDC, or MDC version are currently supported by Azure Backup
  • Stop protection with retain data for the database
  • Perform the upgrade or update
  • Rerun the pre-registration script. Often, the upgrade process may remove the necessary roles. Running the pre-registration script will help verify all the required roles.
  • Resume protection for the database again

SDC to MDC upgrade with no change in SID

Upgrades from SDC to MDC that don't cause a SID change can be handled as follows:

  • Ensure that the new MDC version is currently supported by Azure Backup
  • Stop protection with retain data for the old SDC database
  • Perform the upgrade. After completion, the HANA system is now MDC with a system DB and tenant DBs
  • Rerun the pre-registration script
  • Re-register the extension for the same machine in the Azure portal (Backup -> View details -> Select the relevant Azure VM -> Re-register)
  • Select Rediscover DBs for the same VM. This action should show the new DBs in step 3 as SYSTEMDB and Tenant DB, not SDC
  • The older SDC database will continue to exist in the vault and have the old backed-up data retained according to the policy
  • Configure backup for these databases

SDC to MDC upgrade with a change in SID

Upgrades from SDC to MDC that cause a SID change can be handled as follows:

  • Ensure that the new MDC version is currently supported by Azure Backup
  • Stop protection with retain data for the old SDC database
  • Perform the upgrade. After completion, the HANA system is now MDC with a system DB and tenant DBs
  • Rerun the pre-registration script with correct details (new SID and MDC). Due to a change in SID, you may face issues with successfully running the script. Contact Azure Backup support if you face issues.
  • Re-register the extension for the same machine in the Azure portal (Backup -> View details -> Select the relevant Azure VM -> Re-register)
  • Select Rediscover DBs for the same VM. This action should show the new DBs in step 3 as SYSTEMDB and Tenant DB, not SDC
  • The older SDC database will continue to exist in the vault and have old backed up data retained according to the policy
  • Configure backup for these databases

Re-registration failures

Check for one or more of the following symptoms before you trigger the re-register operation:

  • All operations (such as backup, restore, and configure backup) are failing on the VM with one of the following error codes: WorkloadExtensionNotReachable, UserErrorWorkloadExtensionNotInstalled, WorkloadExtensionNotPresent, WorkloadExtensionDidntDequeueMsg.

  • If the Backup Status area for the backup item is showing Not reachable, rule out all the other causes that might result in the same status:

    • Lack of permission to perform backup-related operations on the VM
    • The VM is shut down, so backups can't take place
    • Network issues

These symptoms may arise for one or more of the following reasons:

  • An extension was deleted or uninstalled from the portal.
  • The VM was restored back in time through in-place disk restore.
  • The VM was shut down for an extended period, so the extension configuration on it expired.
  • The VM was deleted, and another VM was created with the same name and in the same resource group as the deleted VM.

In the preceding scenarios, we recommend that you trigger a re-register operation on the VM.

Next steps