Azure Fence Agent and SBD

High Availability and Disaster Recovery: Mastering Azure Fence Agents and SBD

Introduction: The Critical Need for Fencing in High Availability

In the world of enterprise computing, High Availability (HA) is the bedrock of service reliability. When we deploy applications across multiple nodes in a cluster, our primary goal is to ensure that if one node fails, the service transitions to another node without human intervention. However, the most dangerous scenario in a cluster is not a total failure; it is the "split-brain" scenario, where two nodes both believe they are the primary owner of a resource, such as a shared disk or a virtual IP address. This can lead to catastrophic data corruption and total system instability.

To prevent this, we use a mechanism called "fencing." Fencing is the process of isolating a node that is suspected of being unresponsive, ensuring that it cannot access shared resources or communicate with clients. In the context of the cloud—specifically Microsoft Azure—managing this fencing process requires specialized tools that understand the cloud environment's API. This is where the Azure Fence Agent and SBD (STONITH Block Device) come into play.

Understanding how to implement these technologies is critical for any systems architect or administrator working in Azure. Without proper fencing, your cluster is essentially a ticking time bomb. In this lesson, we will explore the architecture of cluster fencing, the mechanics of the Azure Fence Agent, the role of SBD, and how to configure these components to build a truly resilient infrastructure.

The Concept of STONITH: Why "Shoot The Other Node In The Head" Matters

At the heart of fencing is the concept of STONITH, an acronym that stands for "Shoot The Other Node In The Head." While the name sounds aggressive, the logic is sound and necessary. When a cluster node stops responding to heartbeat signals, the other nodes in the cluster must decide whether the unresponsive node is actually dead or just experiencing a temporary network delay. If the cluster assumes the node is dead and starts the service elsewhere, but the original node is still alive and writing to the disk, data corruption is inevitable.

Fencing provides a definitive way to stop that node. By cutting off its power or its access to the storage, we ensure that the "zombie" node can no longer cause harm. In a physical data center, this was often done using Power Distribution Units (PDUs) or Integrated Lights-Out (iLO/DRAC) cards. In the cloud, we don't have physical access to the power supply, so we use API-based calls to the cloud provider's management plane to achieve the same result.

Callout: Fencing vs. Heartbeating It is vital to distinguish between heartbeating and fencing. Heartbeating is the communication mechanism used by cluster nodes to monitor each other's health. Fencing is the corrective action taken once a heartbeat failure is detected. You cannot have a reliable cluster with heartbeating alone; you must have a fencing mechanism to resolve the ambiguity of an unresponsive node.

Azure Fence Agent: Architecture and Operation

The Azure Fence Agent is a software component, typically part of the fence-agents package in Linux distributions like RHEL, SUSE, or Ubuntu. It acts as an interface between the cluster management software (such as Pacemaker) and the Azure Resource Manager (ARM). When the cluster determines that a node needs to be fenced, it invokes the Azure Fence Agent with the necessary credentials.

The agent then authenticates with Azure using a Service Principal or a Managed Identity. Once authenticated, it sends a command to the Azure API to "power off" or "reboot" the virtual machine. Because this request is handled by the Azure fabric, it is highly reliable. Even if the guest operating system on the node is completely frozen or kernel-panicked, the Azure fabric can still force a shutdown of the VM.

Prerequisites for Azure Fence Agent

Before you can configure the agent, you must ensure the following requirements are met:

• Service Principal or Managed Identity: The cluster nodes need permission to modify their own power state. A Service Principal with the "Contributor" role on the resource group containing the VMs is the standard approach.
• Networking: The nodes must be able to reach the Azure management endpoints (login.microsoftonline.com and management.azure.com). If you are in a locked-down virtual network, you may need to configure an Azure Firewall or a Service Endpoint.
• Resource Naming: You must know the exact resource names, resource group names, and subscription IDs for the nodes you intend to fence.

Configuring the Azure Fence Agent

To configure the agent within a Pacemaker cluster, you typically use the crmsh or pcs command-line tools. Below is an example of how to define a fencing resource for an Azure node:

# Example command for creating a fence device in Pacemaker
pcs stonith create azure_fence_device fence_azure_arm \
  login="your-app-id" \
  passwd="your-password" \
  tenant="your-tenant-id" \
  subscriptionId="your-subscription-id" \
  resourceGroup="your-resource-group" \
  pcmk_host_map="node1:vm-node-1;node2:vm-node-2" \
  op monitor interval=60s

In this code snippet, we define a resource named azure_fence_device using the fence_azure_arm agent. The pcmk_host_map parameter is critical; it maps the cluster node name (e.g., node1) to the actual name of the virtual machine in Azure (e.g., vm-node-1).

SBD: STONITH Block Device Explained

While the Azure Fence Agent relies on the cloud provider's API, SBD offers a different approach to fencing. SBD uses a shared block device—a small disk accessible by all nodes in the cluster—to communicate status. Instead of an API call, a node that needs to be fenced is "marked" by writing a specific message to the shared disk.

How SBD Works

Every node in the cluster constantly monitors the SBD partition. If Node A determines that Node B is unresponsive, Node A writes a "reset" or "fence" command to the SBD device. Node B, if it is still alive but perhaps just experiencing network issues, will see this command on the shared disk and immediately self-terminate or reboot.

This is a very fast and reliable method because it does not depend on the external cloud API, which can sometimes experience latency or rate-limiting. However, it requires a storage solution that supports multiple simultaneous writers and provides strict consistency, such as Azure Shared Disks or an iSCSI target.

When to Use SBD vs. Azure Fence Agent

Choosing between these two depends on your specific architecture.

Callout: The Hybrid Approach Many high-stakes environments use a hybrid approach. They configure the Azure Fence Agent as the primary fencing mechanism and use SBD as a "watchdog" or secondary mechanism. This provides redundancy: if the Azure API is unreachable, the SBD can still initiate a fence, and if the storage is corrupted, the API can still perform a hard power-off.

Step-by-Step Implementation: Configuring Azure Fence Agent

Let’s walk through the process of setting up a robust fencing environment in an Azure-based cluster. We will assume you are using Pacemaker on a Linux distribution.

Step 1: Create a Service Principal

You need an identity that the cluster nodes can use to perform actions in your subscription. Run the following command in your Azure CLI:

az ad sp create-for-rbac --name "ClusterFenceSP" --role "Contributor" --scopes /subscriptions/<sub-id>/resourceGroups/<rg-name>

Keep the output of this command secure, as it contains the credentials the fence agent will use.

Step 2: Install the Necessary Packages

Ensure the fencing agents are installed on all nodes in your cluster:

# On RHEL/CentOS/AlmaLinux
sudo yum install fence-agents-azure-arm

# On Ubuntu
sudo apt-get install fence-agents

Step 3: Define the Fencing Resource

As shown in the previous section, use pcs to define the resource. It is important to set the pcmk_host_map correctly. If your cluster node names differ from your Azure VM names, the fencing will fail.

Step 4: Testing the Fence

Never deploy a cluster to production without testing the fencing mechanism. To test it, you can manually trigger a fence from the command line:

# Manually test the fence agent
stonith_admin -F node2

Watch the Azure portal while running this command. You should see the status of vm-node-2 transition to "Stopping" and then "Stopped" within a few seconds. If this happens, your configuration is successful.

Best Practices for High Availability in Azure

When designing for HA, it is easy to overlook the "small" details that cause system-wide failures. Follow these industry-standard best practices to keep your cluster stable.

1. Use Multiple Fencing Levels

Don't rely on a single fence device. Configure both the Azure Fence Agent and an SBD device if your workload is critical. This creates a "defense in depth" strategy where the failure of one mechanism does not compromise the entire cluster.

2. Monitor Fencing Events

Fencing should not be a silent event. Configure alerts in Azure Monitor to notify your team whenever a STONITH action occurs. If a node is being fenced, it indicates an underlying problem (like a kernel panic, memory exhaustion, or network instability) that requires investigation.

3. Keep Credentials Secure

The Service Principal credentials used by your fence agent have "Contributor" access to your resource group. Store these in Azure Key Vault and use Managed Identities whenever possible to avoid hardcoding secrets in your cluster configuration files.

4. Test During Maintenance Windows

Fencing causes downtime for the target node. Never perform a test on a production node during business hours. Always schedule a maintenance window and verify that the cluster resources fail over to the surviving nodes as expected during the fencing process.

5. Avoid "Over-Fencing"

If your heartbeat timeouts are too aggressive, the cluster might fence a node that is merely experiencing a minor network blip. Tune your heartbeat settings (e.g., deadtime in corosync.conf) to be sensitive enough to detect real failures but lenient enough to ignore transient network noise.

Common Pitfalls and How to Avoid Them

Even experienced administrators run into issues with cluster fencing. Here are the most frequent mistakes and how to steer clear of them.

Mistake 1: Incorrect Host Mapping

The most common error is a mismatch between the cluster node name and the Azure VM resource name. If the agent tries to fence "node1" but the Azure API only knows "prod-web-01," the fence will fail.

• Fix: Always use the az vm list command to verify the exact names of your VMs before configuring the pcmk_host_map.

Mistake 2: Missing Permissions

If the Service Principal assigned to the cluster does not have the correct permissions, the agent will return an "Unauthorized" or "Forbidden" error.

• Fix: Verify the Service Principal's role assignment in the Azure portal. Ensure it has at least the "Virtual Machine Contributor" role on the specific resource group where the nodes reside.

Mistake 3: API Throttling

Azure has API rate limits. If your cluster is constantly flapping and sending too many fence requests in a short period, Azure may throttle the requests, leading to failed fencing.

• Fix: Ensure your cluster heartbeat settings are stable. If you are experiencing frequent flapping, investigate the network connectivity between nodes rather than relying on the fence agent to "fix" the problem.

Mistake 4: Ignoring the "Watchdog"

If you are using SBD, you must have a watchdog timer enabled on the Linux kernel. The watchdog acts as a final safety net; if the node stops updating the watchdog, the hardware will force a reboot.

• Fix: Ensure the softdog module is loaded and the sbd service is configured to interact with it.

Troubleshooting Fencing Failures

When fencing fails, the cluster enters an "unclean" state, and resources may not start. Troubleshooting this requires a systematic approach.

1. Check Cluster Logs: Look at /var/log/messages or use journalctl -u pacemaker to see the exact error returned by the fence agent.
2. Verify API Connectivity: From one of the nodes, try to reach the Azure management API using curl. If you cannot reach it, check your Network Security Group (NSG) rules.
3. Validate Credentials: Use the Azure CLI to log in as the Service Principal from the command line of a cluster node. If you can't log in, the credentials are invalid or expired.
4. Inspect the Resource Group: Ensure the nodes have not been moved to a different resource group or renamed, which would invalidate the configuration.

Warning: The "Unclean" State If a node fails to fence, the cluster will often refuse to start services on other nodes to protect data integrity. This is the intended behavior. Do not try to manually force resources to start until you have resolved the fencing failure. Manually forcing resources can lead to the very split-brain scenario you were trying to prevent.

Comparison: Azure Fence Agent vs. Traditional Hardware Fencing

It is helpful to understand how cloud-based fencing differs from the traditional physical fencing we used in the past.

In the physical world, we relied on hardware-level control. In the cloud, we rely on the provider's software-defined infrastructure. While the mechanism changes, the goal remains identical: ensuring that a non-responsive node is rendered incapable of impacting the shared data state.

Advanced Configuration: Using Managed Identities

Modern Azure deployments prefer Managed Identities over Service Principals because they eliminate the need to manage secret rotation. You can assign a Managed Identity to your cluster VMs and grant that identity the "Contributor" role.

To use a Managed Identity with the Azure Fence Agent, you typically omit the login, passwd, and tenant parameters in the configuration, as the agent will automatically detect the identity assigned to the VM. This is a significant security improvement, as it removes long-lived credentials from your configuration files.

# Example of a simplified fence creation using Managed Identity
pcs stonith create azure_fence_device fence_azure_arm \
  subscriptionId="your-subscription-id" \
  resourceGroup="your-resource-group" \
  pcmk_host_map="node1:vm-node-1;node2:vm-node-2" \
  managed_identity=true \
  op monitor interval=60s

This approach is highly recommended for all new deployments. It reduces the surface area for credential leaks and simplifies the management lifecycle of your cluster.

Conclusion: Key Takeaways for High Availability

Implementing high availability in Azure is not just about spinning up multiple VMs; it is about managing the state of those VMs when things go wrong. Through the use of the Azure Fence Agent and SBD, you can build systems that are resilient to both software crashes and network isolation.

Here are the essential takeaways from this lesson:

• Fencing is Mandatory: Never run a cluster without a fencing mechanism. The risk of data corruption due to split-brain is too high to ignore.
• Understand the Tooling: The Azure Fence Agent leverages the cloud management plane to power off nodes, while SBD uses shared storage to coordinate state. Both are valid, but they serve different architectural needs.
• Prioritize Security: Use Managed Identities whenever possible to authenticate your fencing agents. This removes the risk associated with managing and rotating Service Principal secrets.
• Test Regularly: A fencing mechanism that hasn't been tested is a mechanism that will likely fail when you need it most. Include fencing tests in your regular disaster recovery drills.
• Monitor and Alert: Treat fencing events as high-priority incidents. They are early warnings of deeper infrastructure issues that require immediate attention.
• Defense in Depth: Consider a hybrid approach using both API-based fencing and SBD to ensure that your cluster remains protected even if one of your management paths becomes unavailable.
• Configuration Accuracy: Always double-check your host mappings and permissions. Most fencing failures are caused by simple configuration errors rather than bugs in the software itself.

By mastering these concepts, you transition from simply "hosting" applications to "engineering" resilient services that can survive the inherent unpredictability of cloud environments. As you move forward, continue to refine your cluster configurations and always prioritize the integrity of your data above all else.