Pacemaker and STONITH

Mastering High Availability: Pacemaker and STONITH

Introduction: The Imperative of Uninterrupted Service

In the modern digital landscape, the expectation for services to remain online around the clock is absolute. Whether you are running a database for a financial application, a web server for an e-commerce platform, or a backend service for a mobile application, downtime translates directly to lost revenue, diminished user trust, and operational chaos. Achieving this reliability requires more than just high-quality hardware; it requires a sophisticated software architecture capable of detecting failures and recovering from them automatically without human intervention.

This is where Pacemaker enters the picture. Pacemaker is a cluster resource manager that orchestrates the behavior of services across a cluster of nodes. It ensures that your applications remain available even when individual servers fail. However, high availability (HA) is not merely about moving a service from a broken node to a healthy one. It is about maintaining the integrity of your data and preventing "split-brain" scenarios where two nodes believe they are the primary owner of a resource, leading to data corruption.

To solve the most dangerous problems in cluster management, Pacemaker utilizes a mechanism known as STONITH—an acronym for "Shoot The Other Node In The Head." While the name sounds aggressive, the concept is fundamental to the stability of distributed systems. This lesson provides an in-depth exploration of how Pacemaker works, how to implement STONITH, and the best practices required to build a resilient, production-grade cluster.

Understanding Pacemaker: The Orchestrator

Pacemaker operates by managing a collection of nodes and the services (or "resources") they host. It does not exist in a vacuum; it relies on a cluster stack to provide communication and membership information. In most Linux environments, Pacemaker sits on top of Corosync, which handles the low-level messaging and quorum (the "voting" system that determines if a cluster is healthy enough to operate).

The Architecture of Pacemaker

Pacemaker is designed to be highly flexible. It can manage virtually any service that can be controlled via a script. It treats resources as entities that can be started, stopped, monitored, and migrated. When a node fails, Pacemaker detects the loss of communication, evaluates which resources were running on that node, and automatically moves them to a surviving node.

Core Concepts

To work effectively with Pacemaker, you must understand three fundamental concepts:

• Resources: These are the services you want to keep running, such as a virtual IP address, a web server process (like Apache or Nginx), or a filesystem mount.
• Constraints: These are the rules that dictate how resources behave. For example, you might have a "colocation constraint" that insists a database must run on the same node as its storage, or a "location constraint" that prefers the database to run on your most powerful server.
• Cluster Membership: Every node in the cluster must be aware of every other node. If a node stops responding, the remaining nodes must decide whether they are still capable of providing service.

Callout: Pacemaker vs. Traditional Load Balancing It is common to confuse high availability clusters with load balancers. A load balancer distributes traffic across multiple healthy nodes simultaneously to improve performance and capacity. Pacemaker, by contrast, is a failover mechanism. It is designed to ensure that a specific resource is active on exactly one node at a time. If the primary node fails, Pacemaker promotes the secondary node to active. They are complementary technologies, not substitutes for one another.

The Necessity of STONITH

In a two-node cluster, the biggest danger is the "split-brain" scenario. Imagine Node A and Node B are connected via a network cable. If that cable is cut, Node A cannot see Node B, and Node B cannot see Node A. Both nodes might assume the other has crashed, and both might attempt to "take over" the primary role, such as mounting the same shared filesystem. This leads to catastrophic data corruption.

How STONITH Prevents Corruption

STONITH ensures that if a node is suspected of being unresponsive or acting erratically, it is physically powered off or fenced from the network before any other node attempts to take over its resources. By using a power management device (such as a remote power switch, an Intelligent Platform Management Interface (IPMI) controller, or a cloud-based API), the cluster ensures that the "dead" node is truly dead before the "living" node proceeds.

Types of STONITH Mechanisms

• Power-based fencing: This is the gold standard. The cluster sends a command to a PDU (Power Distribution Unit) or IPMI interface to physically cut power to the node.
• Network-based fencing: Using managed network switches, the cluster can disable the network ports associated with the target node, effectively isolating it from the cluster and the storage.
• Storage-based fencing: Some storage arrays allow a node to be "fenced" at the SCSI level, preventing it from writing to shared disks.
• Hypervisor-based fencing: In virtualized environments, the cluster can interact with the hypervisor (like VMware or KVM) to shut down or pause the virtual machine.

Warning: The Dangers of Disabling STONITH Many beginners are tempted to disable STONITH during testing because it is complex to configure. Never do this in a production environment. Without STONITH, you have no guarantee of data integrity. If your cluster encounters a network partition, the absence of STONITH is a direct invitation to data corruption.

Configuring Pacemaker and STONITH: A Step-by-Step Guide

For this guide, we assume a two-node cluster running a common Linux distribution like RHEL, AlmaLinux, or Ubuntu. We will use pcs (Pacemaker Configuration System) as the command-line interface, as it is the industry standard for managing these clusters.

Step 1: Installing the Stack

First, ensure you have the necessary packages installed on all nodes.

# Example for RHEL/AlmaLinux
sudo yum install pcs pacemaker corosync fence-agents-all
# Enable and start the services
sudo systemctl enable --now pcsd

Step 2: Authenticating the Nodes

The nodes need to communicate securely. You must authenticate the nodes to each other using the pcs cluster user.

# Run this on one node
sudo pcs host auth node1 node2 -u hacluster -p your_secure_password

Step 3: Creating the Cluster

Define the cluster and start it across all members.

sudo pcs cluster setup my_cluster node1 node2
sudo pcs cluster start --all
sudo pcs cluster enable --all

Step 4: Configuring STONITH

This is the most critical step. If you are using physical servers, you likely have an IPMI interface. You will need to configure a fencing device for each node.

# Example: Adding an IPMI fence device for node1
sudo pcs stonith create fence_node1 fence_ipmilan pcmk_host_list=node1 ipaddr=192.168.1.50 login=admin passwd=password
# Example: Adding an IPMI fence device for node2
sudo pcs stonith create fence_node2 fence_ipmilan pcmk_host_list=node2 ipaddr=192.168.1.51 login=admin passwd=password

Step 5: Verifying the Configuration

Check the status of the cluster to ensure everything is running correctly.

sudo pcs status

The output should show that both nodes are online and that the STONITH resources are active. If you see errors regarding "quorum" or "fencing," do not proceed until they are resolved.

Deep Dive: Managing Resources with Pacemaker

Once the cluster is up and STONITH is configured, you can begin adding resources. Pacemaker uses "Resource Agents" to manage services. These are standardized scripts that follow the Open Cluster Framework (OCF) specification.

Adding a Virtual IP Address (VIP)

A common requirement is to have a single IP address that clients connect to, which moves between nodes.

sudo pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.1.100 cidr_netmask=24 op monitor interval=30s

This command creates a resource named VirtualIP using the IPaddr2 agent. The monitor operation ensures that Pacemaker periodically checks if the IP is still active on the node.

Adding a Service (e.g., Apache)

If you want to manage a web server, you add it as a resource.

sudo pcs resource create WebServer ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf op monitor interval=1min

Constraints: Defining Relationships

Resources often have dependencies. If you have a WebServer, it must have the VirtualIP to serve traffic. You can enforce this with a constraint:

# Create a group to ensure they start together
sudo pcs resource group add WebGroup VirtualIP WebServer

By grouping these resources, Pacemaker ensures that they are always started on the same node in the specified order.

Best Practices for Production Clusters

Building a reliable cluster requires adherence to several industry-standard practices. These guidelines help prevent common operational failures.

1. Use Dedicated Interconnects: Always use a dedicated, physically separate network for cluster heartbeat traffic. Do not share this network with public traffic or storage traffic. A network congestion event on your public network should never cause your cluster to lose quorum.
2. Test Fencing Regularly: Perform "chaos testing." Manually trigger a fence event by shutting down a node or pulling the network cable to ensure that Pacemaker correctly identifies the failure, fences the node, and migrates the resources.
3. Keep Time Synchronized: Use NTP (Network Time Protocol) or Chrony on all nodes. Cluster logs and state machines rely on accurate timestamps. Time drift can cause erratic behavior in cluster membership.
4. Monitor Cluster Logs: Pacemaker is extremely verbose. Use a log aggregator to monitor /var/log/messages or journalctl. Alerts should be configured to notify administrators the moment a resource migrates or a node is fenced.
5. Avoid Over-Complexity: Keep your resource tree as simple as possible. Deep nesting of constraints and complex dependencies make debugging significantly harder when things go wrong.

Callout: The "Quorum" Concept In a cluster, quorum is the minimum number of nodes required to make decisions. If you have a two-node cluster and one node fails, you technically lose quorum. Pacemaker handles this by allowing you to set no-quorum-policy=ignore for two-node setups. Always be aware of your quorum settings, as they dictate how the cluster behaves when it loses half its members.

Common Pitfalls and Troubleshooting

Even with a perfect setup, clusters can encounter issues. Here are the most frequent mistakes and how to address them.

1. The "Flapping" Resource

A flapping resource is one that keeps failing and restarting. This often happens if the monitor interval is too short or if the resource depends on a service that isn't fully ready when the node starts.

• Fix: Increase the monitor interval and add a start-delay parameter to the resource configuration to give the underlying service time to initialize.

2. Misconfigured Fencing

If your fencing device is unreachable, the cluster will enter a "blocked" state. It will refuse to migrate resources because it cannot guarantee the safety of the failing node.

• Fix: Ensure the management IP addresses for your PDU or IPMI are accessible from the cluster nodes at all times. Test the stonith_admin tool to manually trigger a fence during a maintenance window.

3. Network Partitioning (Split-Brain)

If your network becomes unstable, the nodes may lose communication. If STONITH is not functioning, both nodes will attempt to become primary.

• Fix: Ensure your STONITH configuration is redundant. In some environments, it is beneficial to configure two fencing mechanisms (e.g., IPMI and a network switch) so that if one method fails, the second serves as a backup.

4. Ignoring Resource Logs

When a resource fails to start, the pcs status command will show it as "failed." However, it won't tell you why.

• Fix: Always look at the resource agent logs. For OCF agents, the logs are typically found in the system journal. Use journalctl -u pacemaker to see the exact exit code of the script that failed.

Comparison of Fencing Methods

Summary and Key Takeaways

High Availability is the backbone of reliable infrastructure. By utilizing Pacemaker for orchestration and STONITH for data integrity, you create a system that is not only automated but also safe. Remember that high availability is not a "set it and forget it" configuration; it is a discipline that requires ongoing testing, monitoring, and refinement.

Key Takeaways for Your Cluster Strategy:

1. Data Integrity First: Never prioritize uptime over data integrity. Always implement STONITH, even if it adds complexity to your deployment.
2. Automation is Essential: Pacemaker is designed to handle failures without human intervention. Ensure your resource agents are configured to be idempotent (safe to run multiple times) and self-healing.
3. Communication is Key: Use a dedicated, isolated network for cluster heartbeat traffic to prevent "false positives" where healthy nodes are fenced due to network congestion.
4. Test for Failure: A cluster that has never been tested for failure is a cluster that will fail when you least expect it. Regularly simulate node failures and verify that the cluster recovers as expected.
5. Keep it Lean: Complexity is the enemy of stability. Use the simplest possible configuration to achieve your availability goals.
6. Monitor and Alert: You cannot fix what you do not know is broken. Configure active monitoring for your cluster state and ensure that your operations team receives immediate alerts on any failover events.
7. Document the Topology: Maintain clear documentation of your resource constraints and interdependencies. When a failure occurs at 3:00 AM, you need to understand the relationship between your services instantly.

By mastering these principles, you move beyond simple server management and into the realm of distributed systems engineering. Pacemaker and STONITH provide the tools to ensure that your services remain available, consistent, and resilient, regardless of the challenges your underlying hardware may face.

Frequently Asked Questions (FAQ)

Q: Can I run Pacemaker with only one node?

A: Technically, yes, but it defeats the purpose of high availability. Pacemaker is designed for clusters. A single-node "cluster" provides no failover capability and will not protect you against hardware failure.

Q: What happens if the STONITH device itself fails?

A: This is a "fencing loop." If the cluster cannot fence a node, it will generally freeze all operations to prevent data corruption. This is why it is best practice to have redundant fencing methods or a highly reliable, out-of-band management network.

Q: Is Pacemaker only for Linux servers?

A: Pacemaker is primarily designed for Linux environments and integrates deeply with the Linux kernel and systemd. While other operating systems have their own clustering solutions, Pacemaker is the standard for the Linux ecosystem.

Q: Does Pacemaker support cloud environments?

A: Yes. Modern cloud providers offer APIs that allow Pacemaker to fence instances. You can use specialized fencing agents for AWS, Azure, and GCP that interact with the cloud provider's management plane to stop or reboot instances.

Q: How do I perform maintenance on a node?

A: Never just shut down a node that is part of a cluster. Always use the pcs node standby <nodename> command. This tells Pacemaker to gracefully move all resources off that node to other members of the cluster, allowing you to perform your maintenance safely. When you are finished, use pcs node unstandby <nodename> to bring it back into service.