Clustering for HANA and SCS

High Availability and Disaster Recovery: Clustering for SAP HANA and SCS

Introduction: Why High Availability Matters

In the modern enterprise landscape, SAP systems serve as the digital backbone for critical operations, ranging from financial accounting and supply chain management to customer relationship management. When an SAP system experiences downtime, the business impact is often immediate and severe, resulting in lost productivity, missed sales opportunities, and potential damage to the company's reputation. High Availability (HA) is the architectural practice of ensuring that these systems remain operational and accessible even in the event of hardware, software, or network failures.

At the core of an SAP environment, two components are particularly critical: the SAP HANA database and the SAP Central Services (SCS). The SAP HANA database stores the entirety of your organization's transactional data, while the SCS manages the locking mechanisms and message queues that facilitate communication between different SAP application servers. If either of these components fails, the entire SAP landscape effectively grinds to a halt.

Clustering is the primary technical solution used to achieve high availability for these components. By grouping multiple servers into a logical unit, we create a failover mechanism where a secondary node can automatically take over the responsibilities of a primary node if the latter becomes unresponsive. This lesson explores the intricate technical details of clustering for HANA and SCS, providing you with the knowledge to design, implement, and maintain these critical systems.

Understanding the Architecture of SAP HA

To build a reliable cluster, one must first understand the distinction between the database layer and the application layer. The SAP HANA database is a memory-intensive, column-oriented database that requires specific synchronization techniques to ensure that data remains consistent across nodes. The SAP Central Services (SCS) layer, on the other hand, is a lighter process that relies heavily on shared file systems and virtual IP addresses to manage client connections.

The Role of Pacemaker and Corosync

In the Linux-based SAP ecosystem, the industry standard for clustering is the combination of Pacemaker and Corosync. Corosync acts as the messaging layer, providing a reliable communication channel between nodes in the cluster. It constantly monitors the "heartbeat" of each node to detect failures quickly. Pacemaker serves as the cluster resource manager; it makes decisions about where resources (like IP addresses, file systems, or database processes) should run and handles the orchestration of failover events.

Callout: The Distinction Between Pacemaker and Corosync Corosync is primarily responsible for the group membership and messaging service. It ensures that every node in the cluster knows which other nodes are active. Pacemaker, however, is the brains of the operation. It interprets the information provided by Corosync and executes the recovery scripts necessary to move services from a failed node to a healthy one.

Shared Storage vs. Replicated Storage

When setting up a cluster, you must decide how data is handled during a failure. For the SCS, a shared file system (such as NFS or a clustered file system like GFS2) is typically used to ensure that the message queues and enqueue tables are accessible to whichever node is currently active. For HANA, however, the primary method is System Replication. HANA System Replication (HSR) asynchronously or synchronously ships transaction logs from the primary database to the secondary database, ensuring that the secondary is always prepared to take over.

Implementing High Availability for SAP HANA

SAP HANA clustering requires a deep integration between the database software and the cluster management software. The SAPHanaSR resource agent is the standard tool used to bridge this gap, allowing Pacemaker to query the status of the HANA database and control its lifecycle.

Steps to Configure HANA System Replication (HSR)

1. Prerequisite Verification: Ensure that both the primary and secondary nodes have the same SAP SID and instance numbers. Verify that the underlying storage and network configurations are identical across both nodes.
2. HANA Replication Setup: On the primary node, initialize the replication process. You will need to use the hdbnsutil command to configure the secondary site as a replica of the primary.
3. Cluster Integration: Install the resource-agents-sap-hana package on your Linux distribution. This package contains the necessary scripts for Pacemaker to manage HANA instances.
4. Creating the Resource: Define the HANA resource within Pacemaker. This involves setting parameters such as the SID, the instance number, and the virtual IP address that clients will use to connect to the database.

Code Example: Defining a HANA Resource in Pacemaker

The following command provides a conceptual view of how you might add a HANA resource to the cluster configuration. Note that in production environments, you would typically use a configuration file or the crmsh tool to handle these definitions.

# Example: Adding a HANA resource to Pacemaker
crm configure primitive rsc_HANA_HDB_00 ocf:suse:SAPHana \
  params SID="HDB" InstanceNumber="00" PREFER_SITE_TAKEOVER="true" \
  op monitor interval="60" role="Master" timeout="700" \
  op monitor interval="61" role="Slave" timeout="700" \
  op promote interval="0" timeout="3600" \
  op demote interval="0" timeout="3600"

In this snippet, we define a resource named rsc_HANA_HDB_00. We set the SID and InstanceNumber to match the SAP system. The monitor operations are crucial; they tell the cluster to check the status of the HANA database every 60 seconds. The promote and demote operations define how the cluster handles the transition of the database from a standby (Slave) role to an active (Master) role during a failover event.

Note: Always verify your cluster configuration using the crm_verify -L command before committing changes. A syntax error in your configuration can lead to "split-brain" scenarios where both nodes believe they are the primary, potentially corrupting your database.

Implementing High Availability for SAP SCS

The SAP Central Services (SCS) consists of two main components: the Message Server and the Enqueue Server. Because these components maintain the state of user locks, they must be highly available to prevent data inconsistency.

The Enqueue Replication Server (ERS)

To protect the Enqueue Server, we deploy an Enqueue Replication Server (ERS) on the secondary node. The ERS stores a copy of the lock table in its memory. If the primary SCS fails, the ERS is promoted to become the new primary, and it uses the replicated lock table to resume operations without losing information about active user transactions.

Steps to Configure SCS Clustering

1. Shared File System: Set up a shared directory (usually /sapmnt/<SID>) that is accessible by both nodes. This ensures that configuration files and logs are consistent.
2. Virtual IP (VIP): Configure a floating IP address that moves between the SCS node and the ERS node. Clients connect to this VIP, so they remain unaware of which physical node is currently hosting the SCS.
3. Resource Grouping: Group the SCS resource and the VIP together in Pacemaker. This ensures that whenever the SCS moves, the IP address moves with it, maintaining a seamless experience for application servers.

Warning: Never attempt to run the SCS and the ERS on the same physical node during normal operation. Doing so defeats the purpose of high availability, as a single hardware failure would take down both the primary and the backup. Use "colocation constraints" in Pacemaker to ensure they stay on different nodes.

Best Practices and Industry Standards

Achieving true high availability is as much about process as it is about technology. Even with a perfectly configured cluster, poor operational habits can lead to unnecessary downtime.

Regular Failover Testing

The most common mistake administrators make is assuming the cluster works without ever testing it. You should perform scheduled failover tests in a non-production environment, and eventually in production during maintenance windows. This ensures that your automation scripts, network configurations, and storage mounts are actually capable of surviving a real outage.

Monitoring and Alerting

A cluster that fails silently is dangerous. Implement robust monitoring that tracks not just the health of the HANA database, but also the health of the cluster itself. If the cluster enters a "failed" state, your monitoring system should immediately notify the on-call engineer.

Fencing and STONITH

STONITH (Shoot The Other Node In The Head) is a critical concept in clustering. If a node becomes unresponsive, the cluster must ensure that it is completely shut down before the other node takes over its resources. This prevents data corruption. Always use a hardware-based fencing mechanism (like an intelligent power distribution unit or a hypervisor API) to guarantee that the failed node cannot perform any further I/O operations.

Configuration Management

Keep your cluster configuration files under version control. If an emergency occurs, you want to be able to revert to a "known good" configuration quickly. Document every parameter change, especially those related to timeouts and monitoring intervals, as these are often the first things to cause issues during high-load scenarios.

Common Pitfalls and How to Avoid Them

Even experienced architects encounter common issues when setting up HANA and SCS clusters. Understanding these pitfalls allows you to proactively mitigate risks.

1. Network Latency Issues

HANA System Replication is highly sensitive to network latency. If you are running in a synchronous replication mode, every commit on the primary must wait for an acknowledgement from the secondary. If the network between your nodes is slow, your application performance will suffer significantly.

• Solution: Always ensure a dedicated, low-latency network connection (preferably 10Gbps or higher) between cluster nodes for replication traffic.

2. Improper Resource Timeouts

If your monitor interval is too short or your timeout value is too aggressive, the cluster might trigger a failover unnecessarily during a temporary spike in system load. This is known as "flapping."

• Solution: Tune your timeout values based on your specific workload. Start with the vendor-recommended defaults and increase them only if you observe false-positive failovers.

3. Ignoring OS-Level Settings

Clustering software relies heavily on the underlying operating system. Issues with systemd services, kernel parameters, or file system mount options can cause the cluster resource agent to fail.

• Solution: Use the official SAP-certified operating system images and follow the SAP-specific hardening guides. Do not deviate from these configurations unless you have a specific, documented requirement.

4. Split-Brain Scenarios

A split-brain occurs when nodes lose communication with each other and both attempt to claim the primary role. This is catastrophic for database consistency.

• Solution: Implement a robust quorum mechanism. In a two-node cluster, you must use a "tie-breaker" node or a cloud-based disk witness to ensure that only one node can ever achieve a majority vote.

Detailed Step-by-Step: Testing a Failover

Performing a manual failover is the best way to understand how your cluster behaves. Follow these steps to conduct a safe test.

1. Check System Health: Before starting, verify that both nodes are healthy and that the HANA database is in the SAPHana-SR state of primary on Node A and secondary on Node B.

   # Check HANA replication status
   hdbnsutil -sr_state
   

2. Initiate Failover: Use the Pacemaker command to move the resource to the other node.

   # Move the resource to Node B
   crm resource move rsc_HANA_HDB_00 nodeB
   

3. Monitor the Transition: Watch the logs on both nodes to see the demotion of the primary and the promotion of the secondary.

   # Watch the cluster status
   crm_mon -r
   

4. Verify Application Connectivity: Ensure that your SAP application servers can still connect to the database via the virtual IP address.
5. Revert and Clean Up: Once the test is successful, move the resources back to the original node and clear any constraints created by the move command.

   # Clear the move constraint
   crm resource unmove rsc_HANA_HDB_00
   

Advanced Clustering Considerations

While basic clustering handles hardware failures, advanced HA designs address more complex scenarios.

Multi-Target Replication

For larger landscapes, you might need more than just one secondary site. HANA supports multi-target replication, where a primary node replicates to two or more secondary nodes. This is useful for scenarios where you need one secondary for local high availability and another for disaster recovery in a different geographic region.

Cloud-Specific HA

In public cloud environments (such as AWS, Azure, or GCP), clustering works differently. You often cannot use traditional IP-based clustering because the cloud network layer manages IP addresses differently. Instead, you must use cloud-native APIs to update routing tables or load balancer configurations during a failover event. Always consult the specific SAP HA guide provided by your cloud service provider.

The Role of Kernel Parameters

The sysctl settings on your Linux hosts can significantly impact cluster stability. Parameters like net.core.rmem_max and net.core.wmem_max should be tuned to handle the high throughput of HANA replication traffic. Additionally, ensuring that the tuned profile for SAP HANA is active on both nodes is essential for maintaining the performance characteristics required by the database.

Frequently Asked Questions (FAQ)

Q: Can I use the same cluster for both HANA and SCS? A: While technically possible, it is generally considered a bad practice. The resource requirements and failure characteristics of HANA and SCS are different. It is better to have separate clusters to avoid complex dependency chains and to make troubleshooting easier.

Q: What happens if the cluster quorum is lost? A: If a cluster loses quorum (e.g., in a three-node cluster, two nodes go down), the remaining node will usually stop all managed resources to prevent data corruption. This is a safety feature. You must restore quorum before the services can be brought back online.

Q: How do I handle maintenance on one of the nodes? A: Put the node into "maintenance mode" using the cluster management tools. This tells Pacemaker to ignore the node for health checks and prevents it from attempting to move resources to it during your maintenance window.

Q: Is synchronous replication always better than asynchronous? A: Synchronous replication guarantees zero data loss (RPO=0), but it comes at the cost of performance. Asynchronous replication is faster but carries the risk of losing the most recent transactions if the primary fails. Choose based on your RPO requirements and network capabilities.

Key Takeaways

1. Clustering is Essential: High Availability is not an optional feature for SAP systems; it is a fundamental requirement to ensure business continuity and protect data integrity.
2. Layered Strategy: You must treat the database (HANA) and the application services (SCS) as distinct entities, each requiring its own HA strategy and replication mechanism.
3. The Importance of Fencing: Never underestimate the importance of STONITH. Preventing "split-brain" scenarios is the single most important aspect of cluster design to avoid catastrophic database corruption.
4. Testing is Mandatory: A cluster that has not been tested is a cluster that will fail when you need it most. Incorporate regular failover drills into your operational lifecycle.
5. Network Matters: The performance of HANA replication is directly tied to the quality of your network infrastructure. Invest in dedicated, high-speed, low-latency links between nodes.
6. Simplicity Wins: Keep your cluster configurations as simple as possible. Avoid unnecessary complexity, as it increases the surface area for errors and makes recovery more difficult during an incident.
7. Document Everything: Maintain detailed documentation of your cluster architecture, including timeout settings, resource dependencies, and failover procedures, so that any team member can respond effectively during an outage.

By following these principles and maintaining a disciplined approach to configuration and testing, you can build a resilient SAP environment that supports your business objectives without interruption. Remember that technology is only one part of the equation; the human element—your team's readiness and ability to manage these systems—is equally vital to your success in maintaining high availability.