ExpressRoute for Hybrid Connectivity

ExpressRoute for Hybrid Connectivity in SAP Infrastructure

Introduction: Bridging the Gap Between On-Premises and Cloud

When organizations migrate their mission-critical SAP workloads to the cloud, the most significant challenge often lies not in the application logic itself, but in the connectivity between the existing on-premises data center and the cloud environment. SAP environments are notoriously sensitive to latency, bandwidth fluctuations, and security risks. A standard internet connection, while convenient, lacks the predictability and reliability required for a high-performance SAP landscape, especially when dealing with large-scale database replication, real-time user traffic, and complex integrations with legacy systems.

This is where Azure ExpressRoute becomes the cornerstone of a well-architected SAP infrastructure. ExpressRoute provides a private, dedicated connection between your on-premises network and Microsoft Azure, effectively extending your data center into the cloud. Unlike a Virtual Private Network (VPN) that traverses the public internet, ExpressRoute uses a private connection provided by a connectivity partner. This ensures that your SAP traffic remains isolated, consistent, and secure, providing the deterministic performance characteristics necessary for SAP S/4HANA or SAP BW/4HANA implementations. Understanding how to design, implement, and manage ExpressRoute is not merely a technical task; it is a fundamental requirement for maintaining the uptime and reliability of your entire enterprise resource planning system.

Understanding the Architecture of ExpressRoute

At its core, ExpressRoute is a Layer 3 connection that bypasses the public internet entirely. It connects your edge router at your on-premises location to the Microsoft edge routers at an Azure peering location. This architecture is built on the Border Gateway Protocol (BGP), which is the standard routing protocol used to exchange routing information between autonomous systems on the internet. In the context of ExpressRoute, BGP allows your on-premises network and your Azure Virtual Network (VNet) to "talk" to each other, advertising IP address ranges so that traffic knows exactly how to reach its destination.

The Components of ExpressRoute

To successfully implement ExpressRoute for your SAP landscape, you need to be familiar with its primary components:

• ExpressRoute Circuit: This is the logical connection between your on-premises network and Microsoft through a connectivity provider. It is the physical or virtual pipe that carries your traffic.
• ExpressRoute Gateway: This is a specific type of virtual network gateway that you deploy in your Azure VNet. It is responsible for routing traffic between the VNet and the ExpressRoute circuit.
• Private Peering: This is the primary configuration used for SAP workloads. It allows you to connect to your VNets using private IP addresses.
• Microsoft Peering: This is used to connect to Azure public services (like Azure Storage or SQL Database) over the ExpressRoute circuit, though it is less common for standard SAP application traffic.
• Connectivity Provider: These are the telecommunications companies, network service providers, or system integrators that manage the physical cabling and connection between your data center and the Azure peering point.

Callout: ExpressRoute vs. Site-to-Site VPN While a Site-to-Site VPN is an excellent choice for development environments or smaller, non-production SAP workloads, it is generally insufficient for production landscapes. VPNs run over the public internet, meaning they are subject to jitter, packet loss, and unpredictable latency. ExpressRoute, by contrast, offers a private path with guaranteed bandwidth and lower, more consistent latency, which is essential for the synchronous communication required by SAP database clusters and application servers.

Designing for SAP Performance and Reliability

When designing an ExpressRoute strategy for SAP, you must account for the specific networking requirements of the SAP stack. SAP relies heavily on the connection between the application tier and the database tier, as well as the connection between users and the application servers. If your database is on-premises and your application servers are in the cloud (or vice-versa), the latency introduced by your network will directly impact the user experience and the speed of transaction processing.

Latency Considerations

SAP typically requires a round-trip time (RTT) of less than 2 milliseconds between application servers and database servers for high-performance operations. When you introduce a network link, every millisecond counts. You must ensure that your ExpressRoute circuit is connected to the Azure region that is geographically closest to your on-premises data center to minimize the physical distance, and therefore the latency, of the signal.

Bandwidth Planning

You must calculate the bandwidth requirements not just for daily operations, but for peak loads and maintenance windows. Consider the following scenarios:

• SAP Database Replication: If you are performing high-availability (HA) replication between on-premises and Azure, you need substantial, sustained throughput.
• Data Migration: During the initial migration of your SAP database (which could be several terabytes in size), you need enough bandwidth to complete the transfer within your allotted maintenance window.
• Batch Processing: Large batch jobs that transfer data between SAP and external systems can saturate a link if not properly prioritized.

Note: Always provision for 20-30% more bandwidth than your peak calculated usage to account for unexpected traffic spikes or potential failover scenarios where all traffic might be routed through a single connection.

Implementation Steps: A Practical Guide

Implementing ExpressRoute involves coordination between your internal network team, your connectivity provider, and the Azure cloud team. The process is broken down into three main phases: provisioning the circuit, configuring the routing, and connecting the gateway.

Phase 1: Provisioning the Circuit

1. Select a Provider: Choose a connectivity provider that has a presence in your preferred Azure peering location.
2. Order the Circuit: Create an ExpressRoute circuit in the Azure portal. You will receive a Service Key.
3. Provide the Service Key: Give this key to your connectivity provider. They will use it to link their network to your Azure circuit.
4. Circuit Status: Once the provider completes their work, the circuit status in the Azure portal will change from "Provider Status: Not Provisioned" to "Provider Status: Provisioned."

Phase 2: Configuring Private Peering

Private peering is the standard configuration for SAP. You need to define the BGP settings:

• Peer Subnet: You need a /30 subnet for the primary link and a /30 subnet for the secondary link (for redundancy). These must be public or private IP addresses that you own.
• VLAN ID: The VLAN ID used for tagging the traffic on the connection.
• ASN (Autonomous System Number): You will need your own ASN (or a private one if you don't have a public one) to establish the BGP session with Microsoft.

Phase 3: Connecting the VNet

Once the peering is established, you need to connect your VNet to the circuit:

1. Create a Gateway Subnet: Within your VNet, create a specific subnet named GatewaySubnet. This is a requirement for any virtual network gateway.
2. Deploy ExpressRoute Gateway: Create an ExpressRoute-type gateway in your VNet.
3. Create the Connection: In the ExpressRoute circuit settings, navigate to "Connections" and add a new connection, selecting the ExpressRoute gateway you just created.

Code Example: Automating ExpressRoute Configuration with Azure CLI

While the Azure Portal is useful for visualization, automating your infrastructure deployment using the Azure CLI or PowerShell ensures consistency and repeatability. Below is an example of how to create an ExpressRoute connection using the Azure CLI.

# 1. Create the ExpressRoute Circuit
az network express-route create \
  --resource-group SAP-Prod-RG \
  --name SAP-ExpressRoute-Circuit \
  --location eastus \
  --bandwidth 1000 \
  --provider "Equinix" \
  --peering-location "Washington DC" \
  --sku-family MeteredData \
  --sku-tier Standard

# 2. Retrieve the Service Key to give to the provider
az network express-route show \
  --resource-group SAP-Prod-RG \
  --name SAP-ExpressRoute-Circuit \
  --query serviceKey

# 3. Create the Gateway Subnet (Assuming VNet exists)
az network vnet subnet create \
  --resource-group SAP-Prod-RG \
  --vnet-name SAP-VNet \
  --name GatewaySubnet \
  --address-prefixes 10.0.0.0/27

# 4. Create the ExpressRoute Gateway
az network vnet-gateway create \
  --resource-group SAP-Prod-RG \
  --name SAP-ER-Gateway \
  --vnet-name SAP-VNet \
  --gateway-type ExpressRoute \
  --sku Standard \
  --public-ip-address "" # Not needed for ExpressRoute

Explanation of the code:

• The az network express-route create command initializes the logical circuit. We specify the bandwidth (in Mbps) and the provider.
• The serviceKey is the unique identifier that links your Azure resource to the physical hardware installed by your provider.
• The GatewaySubnet is a mandatory requirement for Azure routing components.
• The az network vnet-gateway create command deploys the virtual gateway into the VNet, which acts as the BGP speaker that talks to your on-premises routers.

Best Practices for SAP Hybrid Connectivity

To ensure your SAP environment remains stable and performant, you should adhere to these industry-standard practices.

Redundancy and High Availability

Never rely on a single ExpressRoute circuit for production SAP workloads. If the physical fiber is cut or the provider has an outage, your entire SAP landscape goes offline.

• Dual Circuits: Always deploy two ExpressRoute circuits from two different peering locations or two different providers.
• Active-Active or Active-Passive: Configure your BGP routing so that if the primary circuit fails, traffic automatically reroutes to the secondary circuit.
• VPN Failover: For mission-critical environments, consider a Site-to-Site VPN as a tertiary failover mechanism, although it should not be the primary connection due to performance limitations.

BGP Route Filtering

Control the traffic being advertised between your data center and Azure. Use route maps or prefix lists to ensure that you are only advertising the necessary subnets. If you accidentally advertise your entire corporate internal network to Azure, or vice-versa, you may create routing loops or security vulnerabilities.

Monitoring and Diagnostics

Use Azure Network Watcher and ExpressRoute metrics to monitor the health of your connection. Pay close attention to:

• Bits In/Out: To identify potential bandwidth bottlenecks.
• ARP/Route Table: To troubleshoot connectivity issues where packets are dropped or misrouted.
• BGP State: Ensure your BGP sessions are consistently in the "Established" state.

Warning: Do not attempt to use ExpressRoute for public-facing traffic. ExpressRoute is designed for private, internal communication. All public-facing traffic for your SAP web dispatchers or Fiori apps should still be routed through standard internet gateways or Azure Front Door, protected by Web Application Firewalls (WAF).

Common Pitfalls and How to Avoid Them

Even with careful planning, several common mistakes can derail an ExpressRoute implementation.

1. Asymmetric Routing

This occurs when traffic from your on-premises network to Azure takes one path, but the return traffic from Azure to your on-premises network takes a different path. This often happens if you have multiple connections (e.g., ExpressRoute and a VPN) to the same location.

• How to avoid: Ensure that your BGP path selection metrics (like AS Path Prepending) are correctly configured so that return traffic follows the intended path.

2. MTU Mismatch

SAP systems often generate large packets, especially during database backups or system copies. If the Maximum Transmission Unit (MTU) size on your on-premises router does not match the MTU size on the Azure virtual gateway, packets will be dropped or fragmented, leading to severe performance degradation.

• How to avoid: Standardize your MTU at 1500 bytes across the entire path. If you need Jumbo Frames (up to 9000 bytes), ensure that every single device in the path—from your switch to the provider and the Azure gateway—supports them.

3. Over-subscription

Some connectivity providers offer "shared" ports where you share bandwidth with other customers. If the provider does not effectively manage this, you might experience "noisy neighbor" issues where your bandwidth is throttled during peak times.

• How to avoid: Request a dedicated port or a clear Service Level Agreement (SLA) regarding guaranteed bandwidth and throughput.

Quick Reference: ExpressRoute Connectivity Options

Frequently Asked Questions (FAQ)

Q: Can I use ExpressRoute to connect to multiple Azure VNets? A: Yes. You can connect multiple VNets to a single ExpressRoute circuit, provided they are in the same geopolitical region. You can also use ExpressRoute Global Reach to connect circuits across different regions.

Q: What happens if my ExpressRoute circuit reaches its bandwidth limit? A: Unlike the public internet, which might just slow down, an ExpressRoute circuit will start dropping packets once the provisioned bandwidth limit is exceeded. This can cause severe SAP application errors or database synchronization failures. Monitor your bandwidth usage closely.

Q: Do I need to encrypt my traffic over ExpressRoute? A: ExpressRoute is a private connection, but it is not inherently encrypted. If your organization's security policy requires data-in-transit encryption, you should implement MACsec (if supported by your provider) or use an application-level encryption method like TLS/SSL for your SAP communications.

Q: Can I change my bandwidth limit after the circuit is created? A: Yes, you can scale your ExpressRoute bandwidth up or down without needing to delete and recreate the circuit. This is a significant advantage for organizations that have seasonal fluctuations in data traffic.

Conclusion: The Strategic Value of ExpressRoute

Implementing ExpressRoute for your SAP infrastructure is a strategic investment in the stability and performance of your business processes. By moving away from the unpredictability of the public internet and establishing a dedicated, private path to the cloud, you ensure that your SAP application servers, databases, and users can communicate with the speed and reliability that modern business demands.

The process of designing and implementing this connectivity requires a deep understanding of networking fundamentals, BGP routing, and the specific needs of the SAP stack. By following the best practices outlined in this lesson—such as ensuring redundant paths, properly monitoring BGP health, and avoiding common pitfalls like asymmetric routing—you create a foundation that supports not only your current SAP landscape but also your future growth in the cloud.

Key Takeaways for Success

1. Prioritize Latency: For SAP, network latency is the enemy. Always choose the peering location closest to your on-premises data center to minimize the physical distance of your traffic.
2. Design for Redundancy: Never deploy a single ExpressRoute circuit for production. Always use at least two circuits with diverse paths to ensure your SAP system remains operational during provider outages.
3. Monitor Your Traffic: Use Azure Network Watcher and native ExpressRoute metrics to keep a close eye on bandwidth, packet loss, and BGP health. Proactive monitoring prevents small issues from becoming major outages.
4. Standardize MTU Settings: Ensure your MTU is consistent across your entire network path to prevent packet fragmentation and performance drops, which are common killers of high-throughput SAP database traffic.
5. Plan for Capacity: Calculate your bandwidth requirements based on peak loads, including database replication and migration tasks, and provision extra capacity to handle unexpected bursts.
6. Automate for Consistency: Use Infrastructure-as-Code (IaC) templates or scripts to deploy your network components. This ensures that your configuration is repeatable and reduces the risk of human error in your production environment.
7. Security Matters: Remember that ExpressRoute is private, but not encrypted. Always evaluate your data-in-transit requirements and consider implementing encryption layers like MACsec or TLS where necessary to meet your compliance standards.

By treating the network as a first-class citizen in your SAP migration project, you ensure that the move to the cloud delivers the performance and reliability your users expect, rather than becoming a bottleneck that slows down your business operations.