Direct Connect Troubleshooting

Lesson: Direct Connect Troubleshooting

Introduction: Why Connectivity Matters

In the modern landscape of cloud computing and hybrid infrastructure, the link between your on-premises data center and your cloud environment is arguably the most critical component of your network. AWS Direct Connect (DX) provides a dedicated physical connection from your internal network to the cloud, bypassing the public internet to provide consistent, predictable performance. However, because this connection bridges two entirely different networking environments, it introduces a unique set of complexities. When a connection fails or degrades, the impact is often immediate, affecting everything from database replication to user-facing application latency.

Troubleshooting Direct Connect is not just about checking if the "link is up." It involves a deep understanding of the OSI model, specifically focusing on the Physical, Data Link, and Network layers. Whether you are dealing with BGP (Border Gateway Protocol) session flaps, packet loss due to MTU mismatches, or routing policy errors, the ability to methodically isolate the issue is a core skill for any network engineer. This lesson will guide you through the lifecycle of a Direct Connect issue, from initial identification to resolution, using industry-standard diagnostic tools and best practices.

1. Understanding the Direct Connect Architecture

Before diving into troubleshooting, you must understand the path your traffic takes. A Direct Connect connection consists of several logical and physical layers that must be functioning in harmony.

• Physical Layer: The physical fiber optic cable connecting your router (Customer Gateway) to the AWS Direct Connect location. This involves light levels, transceiver compatibility, and physical port health.
• Data Link Layer (Layer 2): This layer handles the 802.1Q VLAN tagging. Each Virtual Interface (VIF) is assigned a specific VLAN ID. If the tagging is mismatched between your router and the AWS device, the connection will fail to establish.
• Network Layer (Layer 3): This is where the BGP peering resides. You assign a private or public ASN (Autonomous System Number) to your router, and AWS provides the corresponding peer IP addresses. BGP is the control plane that handles routing prefixes.

Callout: The "Black Box" Reality A common misconception is that Direct Connect is a single "pipe." In reality, it is a series of interconnected services. When you troubleshoot, always treat the physical connection, the VIF, and the BGP session as distinct entities. An issue with the BGP state does not necessarily mean your physical fiber is broken, just as a physical link failure will always result in a BGP drop.

2. The Troubleshooting Methodology: A Step-by-Step Approach

When a user reports that "the cloud is slow" or "we cannot reach our database," do not jump straight to the cloud console. Follow a structured path to rule out the most common points of failure.

Step 1: Verify Physical Connectivity

Check your local router interface status. If the interface is "down/down," you have a physical layer problem. This often points to a bad SFP module, a dirty fiber patch cable, or a cross-connect issue within the data center.

• Check Light Levels: If your router supports DOM (Digital Optical Monitoring), check the RX/TX light levels. If the light levels are outside the specified range (usually between -3dBm and -10dBm), the interface will flap or fail entirely.
• Verify Port Configuration: Ensure the port speed and duplex settings match. Direct Connect ports are typically configured for 1Gbps or 10Gbps, with full-duplex enabled.

Step 2: Validate the Data Link Layer (VLANs)

If the physical port is up but you cannot ping the BGP neighbor IP, the problem is likely at the VLAN level. Each VIF is tied to a specific VLAN tag.

1. Log into your router and check the sub-interface configuration.
2. Ensure the VLAN tag matches the ID provided in the AWS Management Console.
3. Use a packet capture tool like tcpdump or Wireshark to see if you are receiving any traffic on that sub-interface. If you see ARP requests but no ARP replies, the VLAN tagging is likely incorrect on one end.

Step 3: Troubleshoot the BGP Session

This is where the majority of Direct Connect issues occur. BGP is a chatty protocol that requires precise configuration.

• Check the State: Is the BGP session in Active, Idle, or Established?
• Idle: The router is not trying to connect. Check your configuration for typos in the neighbor IP or ASN.
• Active: The router is trying to connect but failing. This usually points to a firewall blocking TCP port 179 or an incorrect MD5 password.
• Established: The session is up. If you are still not seeing routes, check your route maps or prefix filters.

Tip: BGP Authentication Many organizations use MD5 authentication for BGP sessions. If you are seeing "Authentication Failed" logs in your router, ensure the key matches exactly on both sides. Remember that even a trailing space in the key string will cause a mismatch.

3. Practical Troubleshooting: Code and Command Examples

Let's look at how to perform these checks using standard Cisco IOS-XE syntax, which is common in many enterprise environments.

Checking Interface Status

To verify the physical connection and VLAN status, use the following commands:

# Verify physical interface status
show interfaces GigabitEthernet0/0/0 status

# Verify sub-interface and VLAN tagging
show interfaces GigabitEthernet0/0/0.100

Explanation: The show interfaces command tells you if the line protocol is up. If the line protocol is down, the interface is physically connected, but the switch or device on the other side is not responding to keep-alives or is configured for a different encapsulation.

Inspecting BGP Neighbors

Use these commands to diagnose why a session might be dropping:

# Show BGP neighbor details
show ip bgp neighbors 169.254.0.1

# Check for BGP state changes
show ip bgp summary

Explanation: The show ip bgp summary command is your best friend. It shows the State/PfxRcd column. If this column shows a number, you are receiving routes. If it shows "Idle" or "Active," the session is down. If it shows "Connect," the router is waiting for the TCP handshake on port 179 to complete.

Debugging BGP Packets

If the session stays in "Active," you need to see what is happening during the handshake:

# Enable BGP debugging (use with caution!)
debug ip bgp events
debug ip bgp keepalives

Warning: Debugging in Production Never run debug commands on a production router during peak hours unless absolutely necessary. High-volume debug logs can consume CPU cycles, potentially causing the router to become unresponsive or dropping legitimate traffic. Always clear the debugs immediately after capturing the necessary data using undebug all.

4. Common Pitfalls and How to Avoid Them

Even experienced engineers fall victim to simple mistakes. Understanding these common traps will save you hours of downtime.

MTU Mismatches

The Maximum Transmission Unit (MTU) defines the largest packet size that can be transmitted. By default, most interfaces are set to 1500 bytes. However, if your cloud environment uses Jumbo Frames (9001 bytes) and your local router is set to 1500, you will experience "black-holing." Small packets (like pings) will pass through, but large packets (like database queries or file transfers) will be dropped, causing applications to hang.

• How to avoid: Always perform an MTU test using the ping command with the "do not fragment" bit set.
  • Linux: ping -M do -s 1472 <destination_ip>
  • Cisco: ping <destination_ip> size 1472 df-bit
  • Note: 1472 bytes + 28 bytes of overhead = 1500 bytes. If this passes, your path is clear for standard frames.

Asymmetric Routing

Asymmetric routing occurs when traffic leaves your network via Direct Connect but returns via a VPN or the public internet. This can confuse stateful firewalls, which see an incoming packet without having seen the corresponding outgoing request. This results in the firewall dropping the return traffic.

• How to avoid: Use BGP Local Preference and AS-Path Prepending to influence traffic flow. Ensure that your route advertisements are consistent across all paths. If you have a backup VPN, ensure its metrics are higher than the Direct Connect path.

Route Filtering and Redistribution

Sometimes the session is up, and routes are being exchanged, but you still cannot reach a specific subnet. This is usually due to route maps or prefix lists that are improperly configured.

• How to avoid: Always verify your route maps. Use show ip bgp neighbors <ip> advertised-routes to see what you are sending to AWS, and show ip bgp neighbors <ip> routes to see what you are receiving.

5. Advanced Diagnostics: The AWS Side

If your local router looks perfect, you must investigate the AWS side of the house. The AWS Management Console provides several tools for Direct Connect health.

Using the Direct Connect Health Check

AWS provides a "Connection Health" dashboard. This will show you if there are any physical layer errors on the AWS side, such as CRC errors or input errors. If you see high CRC errors on the AWS side, it almost always points to a faulty fiber cable or a bad SFP transceiver.

Virtual Interface (VIF) Troubleshooting

A VIF acts as the bridge between your Direct Connect connection and your Virtual Private Cloud (VPC) or Direct Connect Gateway. If the VIF state is "Down," you cannot route traffic.

• Check the VIF State: In the AWS Console, navigate to the Direct Connect section and look at your VIFs. If the state is "Down," check if the associated Direct Connect Gateway or Virtual Private Gateway is properly configured.
• VIF Type: Remember that Public VIFs are for accessing AWS public services (like S3), while Private VIFs are for accessing your VPCs. You cannot reach a private IP address over a Public VIF.

6. Industry Best Practices for Reliability

Troubleshooting is reactive; architecting for reliability is proactive. Follow these guidelines to minimize the need for troubleshooting.

• Redundancy is Mandatory: Never rely on a single Direct Connect connection for production traffic. Always provision at least two physical connections, preferably in different Direct Connect locations, to protect against localized outages.
• Monitor Everything: Use SNMP or CloudWatch to monitor interface traffic, CPU load, and BGP state changes. Set up alerts for when a BGP session drops so you are notified before the users report a problem.
• Document Your Topology: Keep an updated diagram of your network, including VLAN IDs, BGP ASNs, peer IP addresses, and MD5 keys. When an outage occurs, having this information at your fingertips is invaluable.
• Perform Regular Failover Tests: Once a quarter, manually shut down your primary Direct Connect interface to ensure your traffic correctly shifts to the secondary path without human intervention. This is the only way to prove your redundancy actually works.

Callout: The "Human Factor" in Troubleshooting Most network outages are caused by configuration changes, not hardware failure. Before you start deep-packet inspection, ask the team: "Did anyone change a route map, update an ACL, or modify a firewall rule in the last 24 hours?" Nine times out of ten, the culprit is a recent change.

7. Troubleshooting Checklist: A Quick Reference

When the network is down, use this checklist to maintain focus:

8. Common Questions (FAQ)

Q: Why does my BGP session flap every few minutes? A: Flapping is usually caused by either a physical layer issue (like a bad fiber cable causing intermittent link drops) or a "keep-alive" mismatch. Check your router logs for "Interface down/up" messages. If the interface is stable, check if there is a BGP hold-time mismatch between your router and the AWS configuration.

Q: I can ping the AWS peer IP, but I cannot reach my EC2 instances. What is wrong? A: This is a routing or security group issue. Check your route table in the AWS VPC. Ensure you have a route pointing the destination traffic back to the Virtual Private Gateway (VGW) or Direct Connect Gateway. Also, verify that your EC2 Security Groups allow traffic from your on-premises IP ranges.

Q: Should I use a Public or Private VIF? A: Use a Private VIF if you need to access resources inside a VPC (like EC2, RDS, or Lambda). Use a Public VIF if you need to access AWS services that have public endpoints (like S3, DynamoDB, or AWS APIs) and you want that traffic to stay off the public internet.

Q: How can I verify that my traffic is actually going over the Direct Connect? A: Use a traceroute. If the path takes you through your ISP's gateway or multiple public internet hops, your traffic is not traversing the Direct Connect. The traceroute should show a direct jump from your router to the AWS network edge.

9. Key Takeaways for Network Engineers

1. Methodology is Key: Always start at the physical layer and work your way up to the application layer. Do not assume the BGP session is the problem until you have confirmed the physical link and VLAN are stable.
2. BGP is the Brain: Most Direct Connect issues are BGP-related. Mastering the show ip bgp commands and understanding how to read BGP states is essential for rapid incident response.
3. MTU Matters: Do not ignore packet size. MTU mismatches are a silent killer of network performance, often manifesting as "partial connectivity" where small packets work but large ones fail.
4. Change Management: Always correlate network issues with recent changes. A vast majority of outages are self-inflicted through incorrect route maps or ACL updates.
5. Redundancy is Non-Negotiable: A single point of failure is a ticking time bomb. Design your architecture with secondary paths and perform failover testing to ensure they function as expected.
6. Monitoring and Alerting: You cannot fix what you cannot see. Ensure you have real-time visibility into your interface health and BGP status so you can respond proactively.
7. Documentation Saves Time: In the heat of an outage, you do not want to be searching for VLAN IDs or MD5 keys. Keep a centralized, secure repository of your network configuration details.

By following these principles and maintaining a disciplined approach to troubleshooting, you will be well-equipped to manage even the most complex Direct Connect environments. Remember that network troubleshooting is as much about patience and logic as it is about technical knowledge. Keep your tools sharp, your logs clean, and your documentation up to date.