Starting with version 0.180 the SAPHanaSR-ScaleOut package supports SAP HANA scale-out multi-target system replication. That means you can connect a third HANA site by system replication to either of the two HANA sites which are managed by the SUSE HA cluster. If you are already running a SAP HANA scale-out database with SUSE HA, you can perform a SAP HANA Scale-Out Multi-Target Upgrade.

Picture: On the way to Multi-Target Replication

In this blog article you will learn how to upgrade the SUSE HA cluster and the HANA HADR provider from old-style to the multi-target aware setup.

Our blog SAPHanaSR-ScaleOut for large ERP systems describes the old-style. The blog article SAPHanaSR-ScaleOut for Multi-Target explains the new multi-target aware setup. So now let us learn about the upgrade procedure by discussing seven questions:

• When do I need to upgrade my cluster configuration?
• Which cluster attributes and configuration items will change?
• How does the overall upgrade procedure look like?
• Which prerequisites are needed for upgrading to multi-target srHook attributes?
• What exactly are the tasks I need to do?
• Where can I find further information?
• What to take with?

When do I need to upgrade my cluster configuration?

Two situations make it advised upgrading the SUSE HA cluster from supporting HANA scale-out old-style system replication to multi-target system replication:

• HANA multi-target system replication is a business need.
• A regularly scheduled software update includes SAPHanaSR-ScaleOut 0.180 and it is not absolutely clear that HANA multi-target system replication will never become a topic in the future.

If no multi-target support is needed, but the SAPHanaSR-ScaleOut package is updated, than you do not need changing configuration or other additional action. Just remember to reload the HANA HA/DR provider hook script SAPHanaSR.py on both sites after the package upgrade.

Which cluster attributes and configuration items will change?

Inside the SUSE HA CIB the major attribute srHook for the HA/DR provider system replication status will change. Up to three site-specific attributes will replace the former single global one. In addition, a set of six newly introduced auxiliary attributes simplyfies this and future updates. For example, new node attributes gra and gsh show the version of the resource agent and the HA/DR provider hook script. Manual page SAPHanaSR-manageAttr(8) contains more details.

Global cib-time                 prim sec srHook sync_state
-----------------------------------------------------------
C11    Mon Jun 15 17:40:59 2020 S2   S1  SOK   SOK

Site lpt        lss mns    srHook srr
--------------------------------------
S1   30         4   suse11 SOK    S
S2   1592235659 4   suse21 PRIM   P
S3                         SOK

Hosts  clone_state node_state roles                         score  site
------------------------------------------------------------------------
suse11 DEMOTED     online     master1:master:worker:master  100    S1
suse12 DEMOTED     online     slave:slave:worker:slave      -12200 S1
suse21 PROMOTED    online     master1:master:worker:master  150    S2
suse22 DEMOTED     online     slave:slave:worker:slave      -10000 S2
suse00             online

Example: SAPHanaSR-showAttr with both attributes

In the “Global” section at the top you see the old-style srHook, coloured orange. The new multi-target site-specific srHook attributes appear in the “Site” section in the middle, coloured green. The above CIB attributes are recorded in lab for illustrating the changes. In real life you must not mix old-style and multi-target attributes.

The old-style HANA HA/DR provider hook script SAPHanaSR.py will be replaced by the multi-target aware SAPHanaSrMultiTarget.py. To accomplish that, HANA’s global.ini configuration needs to be changed in memory and on disk. Finally on OS level outside the SUSE HA cluster the sudoers permission needs to be adapted to the new HA/DR provider hook script. Manual page SAPHanaSrMultiTarget.py(7) and the sections below are showing details.

How does the overall upgrade procedure look like?

A certain procedure leads from defined entry state with old-style HANA HA/DR provider hook and SUSE HA cluster into defined target state with multi-target enabled hook and cluster. Blog article SAPHanaSR-ScaleOut for large ERP systems describes the entry state. You can find the target state described in detail in a separate blog article SAPHanaSR-ScaleOut for Multi-Target . Now let us have an overview of the needed steps. You will find details later.

At a glance the upgrade procedure looks like this:

1. Initially check if everything looks fine.
2. Set SUSE HA cluster resources SAPHanaController and SAPHanaTopology into maintenance.
3. Install multi-target aware SAPHanaSR-ScaleOut package on all nodes.
4. Adapt sudoers permission on all nodes.
5. Replace HANA HA/DR provider configuration on both sites.
6. Check SUSE HA and HANA HA/DR provider for matching defined upgrade entry state.
7. Upgrade srHook attribute from old-style to multi-target.
8. Check SUSE HA cluster for matching defined upgrade target state.
9. Set SUSE HA cluster resources SAPHanaController and SAPHanaTopology from maintenance to managed.
10. Optionally connect third HANA site via system replication outside of the SUSE HA cluster.
11. Finally check if everything looks fine.

As final result of this procedure, the RAs and hook script are upgraded from old-style to multi-target. Further the SUSE HA cluster’s old-style global srHook attribute hana_<sid>_glob_srHook is replaced by site-aware attributes hana_<sid>_site_srHook_<SITE>. The new attributes might stay empty until HANA raises srConnectionChanged() events and triggers the new hook script for the first time. Further, the HANA global configuration and Linux sudoers permissions are adapted. New auxiliary SUSE HA cluster attributes are introduced. The sections below and manual page SAPHanaSR-manageAttr(8) give more details.

Which prerequisites are needed for upgrading to multi-target srHook attributes?

For successful and smooth upgrade, you need the following prerequisites:

• SAP HANA supports multi-target system replication and HA/DR provider.
• All cluster nodes are online in the cluster and there are no current errors in the cluster or HANA.
• Package SAPHanaSR-ScaleOut of identical new version installed on all nodes, including majority maker.
• Resource agents SAPHanaController and SAPHanaTopology new and identical on all nodes, including majority maker.
• HADR provider hook script SAPHanaSrMultiTarget.py new and identical on all nodes, including majority maker.
• Sufficient sudoers permission on all nodes, including majority maker.
• Correct and identical entries for HA/DR provider in global.ini at both sites
• During upgrade the resources SAPHanaController and SAPHanaTopology need to be set into maintenance. HANA needs to reload its HA/DR provider configuration and hook scripts.
• The procedure has been successfully tested on a test system before applying it to the production system.

The upgrade will remove the global srHook attribute. Instead it will introduce site-specific attributes. Values for srHook attributes are written only on HANA srConnectionChanged() event. So the new attribute for HANA secondary site might stay empty in case HANA is not reloaded after the upgrade. In that case the polling attribute sync_state represents HANA’s system replication status as fallback.

SAPHanaSR-manageAttr will always check prerequisites before changing the CIB attribute from one common hana_<sid>_glob_srHook to site-sepcific attributes hana_<sid>_site_srHook_<SITE>. By calling “SAPHanaSR-manageAttr –check …” you can run that built-in check before trying an upgrade. Doing so is a good idea. Manual pages SAPHanaSR-manageAttr(8) and SAPHanaSrMultiTarget.py(7) contain more details on checking upgrade prerequisites.

Note: SAPHanaSR-manageAttr might report an error in case the sudoers permission is more generic than needed. This might be, for example “hana_<sid>_*” instead of “hana_<sid>_site_srHook *”. That error message is irritating. However, it should not affect the effective upgrade. So the message might change in later versions.

What exactly are the tasks I need to do?

Obviously you need to implement all the steps and checks outlined earlier. The needed exact commands and arguments are depending on the specific environment. Further it is a good idea to document the initial and final status of the system. Also writing down the complete procedure details as run book is good idea. This run book should include all commands with its exact parameters and expected results. Even better is preparing executable shell scripts. If no test system is available, please set it up first. Rehearsing the upgrade procedure on a test cluster before applying on production is mandatory.

We can not write down a complete upgrade procedure that works for all customer environments. Instead we give detailed examples for some important tasks:

• Check and document status of SUSE HA cluster and HANA system replication (step 1)
• Upgrade SUSE HA cluster srHook attribute from old-style to multi-target (step 7)
• Set resources SAPHanaController and SAPHanaTopology back from maintenance into managed mode (step 9)

You will find the selected examples in detail in our blog SAP HANA Scale-Out Upgrade Details . Also the manual pages SAPHanaSR-manageAttr(8), SAPHanaSrMultiTarget.py(7), SAPHanaSR-showAttr(8) and SAPHanaSR_maintenance_examples(7) are showing examples.

Where can I find further information?

Please have a look at the reference part of this blog series (link will follow soon).

– Related blog articles

https://www.suse.com/c/tag/towardszerodowntime

– Product documentation

https://documentation.suse.com/

https://www.suse.com/releasenotes/

– Manual pages
SAPHanaSR-manageAttr(8), SAPHanaSR-ScaleOut(7), ocf_suse_SAPHanaController(7), SAPHanaSrMultiTarget.py(7), SAPHanaSR-ScaleOut_basic_cluster(7), SAPHanaSR_maintenance_examples(7), SAPHanaSR-showAttr(8), ha_related_suse_tids(7), crm(8), crm_attribute(8), crm_mon(8), sudo(8), cs_wait_for_idle(8),
cs_clusterstate(8)

What to take with?

• You can upgrade existing SUSE HA clusters for HANA scale-out to support multi-target instead of old-style system replication.
• The upgrade will change cluster resource agents and internal attributes as well as the HANA HA/DR provider script.
• You may use an regular software maintenance window to prepare the cluster for multi-target business needs.
• Rehearsing the upgrade procedure on a test cluster before applying on production is mandatory.