IPsec Configuration

Module: Secure Windows Server Infrastructure

Section: Secure Network

Lesson Title: IPsec Configuration

When we think about network security, our minds often jump straight to firewalls at the perimeter or antivirus software on workstations. However, in a modern infrastructure, the internal network can no longer be considered a "trusted" zone. Attackers who gain a foothold on a single machine often move laterally across the network, sniffing traffic and capturing sensitive credentials or data in transit. This is where Internet Protocol Security, or IPsec, becomes one of the most powerful tools in your Windows Server security arsenal. IPsec isn't just a single protocol; it is a suite of protocols designed to ensure that communication between computers is authenticated, private, and untampered with.

In this lesson, we will dive deep into how IPsec works within the Windows ecosystem. We will move beyond the basic theory and look at how to actually implement it to protect your servers. Whether you are looking to enforce domain isolation so that only authorized domain-joined machines can talk to your file servers, or you need to encrypt traffic between a web server and a backend database, understanding IPsec configuration is essential. By the end of this guide, you will have the knowledge to move from a network where data flows in the clear to a "Zero Trust" environment where every packet is verified and protected.

---

Understanding the Core Components of IPsec

Before we click any buttons in the management console, we need to understand what IPsec is actually doing under the hood. IPsec operates at the Network Layer (Layer 3) of the OSI model. This is important because it means it can protect almost any application traffic—whether it is SQL, SMB, or custom internal applications—without the application needing to be "IPsec-aware."

IPsec provides three main pillars of protection:

1. Authentication: It confirms that the sender is who they claim to be. In a Windows environment, this is usually handled by Kerberos (for domain members) or digital certificates.
2. Integrity: It uses cryptographic hashing to ensure that the data has not been modified while traveling between the two points.
3. Confidentiality: It encrypts the data so that even if an attacker captures the packets using a tool like Wireshark, they cannot read the contents.

AH vs. ESP: The Two Main Protocols

IPsec primarily uses two different protocols to handle these tasks: the Authentication Header (AH) and the Encapsulating Security Payload (ESP).

• Authentication Header (AH): AH provides authentication and integrity for the entire packet, including the IP header. However, it does not provide encryption. Because it signs the IP header, it often breaks when passing through Network Address Translation (NAT) devices, as NAT changes the IP addresses in the header, causing the integrity check to fail.
• Encapsulating Security Payload (ESP): ESP is the workhorse of modern IPsec. It can provide authentication, integrity, and confidentiality (encryption). While it traditionally didn't sign the outer IP header, it is much more compatible with NAT and is the standard choice for almost all modern Windows Server implementations.

Callout: Transport Mode vs. Tunnel Mode

It is vital to distinguish between these two modes of operation.

Transport Mode is used for end-to-end communication between two specific hosts (e.g., a workstation and a server). Only the payload of the IP packet is protected, and the original IP headers remain visible. This is what you will use for most internal Windows Server security tasks.

Tunnel Mode is used when you are creating a gateway-to-gateway connection, such as a Site-to-Site VPN. The entire original IP packet is wrapped inside a new IP packet. This is typically handled by routers or dedicated VPN gateways, though Windows Server can act as one.

---

The Negotiation Process: IKE and SAs

Computers don't just start sending encrypted data immediately. They first have to agree on how they are going to talk. This "agreement" is called a Security Association (SA). The process of reaching this agreement is handled by the Internet Key Exchange (IKE) protocol.

In Windows, this happens in two distinct phases:

Phase 1: Main Mode

In this phase, the two computers establish a secure, authenticated channel. They decide which encryption algorithm (like AES-256) and which integrity algorithm (like SHA-256) they will use to protect their own negotiation. They also perform a Diffie-Hellman exchange to create a shared secret key without ever sending that key over the network. Once Main Mode is finished, the two computers have a "secure tunnel" just for talking about security.

Phase 2: Quick Mode

Now that they have a secure channel from Phase 1, they use it to negotiate the "Quick Mode" SAs. These are the specific rules for the actual data you want to protect (e.g., "all traffic on port 445 should be encrypted with AES-128"). Because Phase 1 is already established, Quick Mode is very fast and can happen multiple times to refresh keys or set up different protections for different types of traffic.

---

Authentication Methods in Windows Server

One of the most common reasons IPsec deployments fail is a misunderstanding of authentication. If two servers can't prove who they are to each other, the IPsec negotiation will fail, and traffic will likely be dropped.

Windows supports several authentication methods:

1. Kerberos V5: This is the default and easiest method for domain-joined machines. Since the servers already trust the Domain Controller, they can use Kerberos to prove their identity to one another.
2. Digital Certificates: This is the most secure and flexible method. It is required if you are communicating with non-domain members, machines in different forests, or devices across the internet. It requires a Public Key Infrastructure (PKI) like Active Directory Certificate Services.
3. Pre-Shared Key (PSK): This involves typing a secret password into both machines. While easy to set up, it is considered insecure for production because the key is stored in plain text in the configuration and is difficult to manage at scale.

Feature	Kerberos V5	Digital Certificates	Pre-Shared Key (PSK)
Ease of Setup	Very High	Moderate (Requires PKI)	High
Security Level	High	Very High	Low
Scalability	Excellent (Domain-wide)	Excellent (Cross-org)	Poor
Best Use Case	Internal Domain Traffic	Remote Access / Non-Domain	Labs / Testing only

---

Practical Implementation: Creating Connection Security Rules

In Windows Server, you configure IPsec through the Windows Defender Firewall with Advanced Security (WFAS) console. Unlike standard firewall rules that simply "Allow" or "Block" traffic, IPsec is configured using Connection Security Rules.

Step-by-Step: Securing Traffic Between Two Specific Servers

Let's say you have a Web Server (Web01) and a Database Server (DB01). You want to ensure that all communication between them is encrypted.

1. Open the Management Console

On either server, click Start, type wf.msc, and press Enter. Navigate to Connection Security Rules in the left-hand pane.

2. Start the New Rule Wizard

Right-click Connection Security Rules and select New Rule. You will see several rule types:

• Isolation: Used to restrict communication based on domain membership or health status.
• Authentication Exemption: Used for machines that cannot do IPsec (like older printers).
• Server-to-Server: Used for specific connections between identified computers.
• Tunnel: Used for VPN-like configurations.

Select Server-to-Server and click Next.

3. Define the Endpoints

You must specify the IP addresses of the two "endpoints."

• Endpoint 1: Enter the IP address of Web01.
• Endpoint 2: Enter the IP address of DB01. By being specific here, you ensure that the IPsec overhead only applies to this specific traffic path.

4. Choose Requirements

This is a critical step. You have three choices:

• Request authentication for inbound and outbound connections: The server will try to use IPsec. If the other side doesn't support it, the traffic will still go through in the clear. This is great for testing.
• Require authentication for inbound and outbound connections: If the IPsec negotiation fails, the traffic is dropped. This is the goal for a secure environment.
• Require authentication for inbound and request for outbound: Often used when a server must accept secure connections but still needs to talk to unmanaged external resources.

For our scenario, select Require authentication for inbound and outbound connections.

5. Select Authentication Method

Since both servers are in the same domain, select Computer (Kerberos V5). This simplifies things as you don't have to manage certificates.

6. Apply and Name

Choose the profiles (Domain, Private, Public) where this rule applies. Usually, for internal servers, you just need "Domain." Give the rule a descriptive name like "Encrypt Web-to-DB Traffic."

Note: You must perform these steps (or apply a similar Group Policy) to both servers. If Web01 "Requires" IPsec but DB01 hasn't been told to use it, Web01 will drop all incoming traffic from DB01, and the application will break.

---

Using PowerShell for IPsec Configuration

While the GUI is great for learning, administrators often prefer PowerShell for speed and repeatability. Windows provides the NetSecurity module to manage these rules.

To create the same Server-to-Server rule we just discussed using PowerShell, you would use the New-NetIPsecRule cmdlet.

# Create a rule requiring encryption between two specific IPs
New-NetIPsecRule -DisplayName "Secure Web to DB" `
    -RemoteAddress 192.168.1.50 `
    -Endpoint1 192.168.1.10 `
    -Endpoint2 192.168.1.50 `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -AuthenticationMethod Kerberos `
    -Enabled True

Explaining the Parameters:

• -RemoteAddress: Tells the firewall which traffic triggers this rule.
• -InboundSecurity / -OutboundSecurity: Sets the requirement level (Request vs. Require).
• -AuthenticationMethod: Defines how the machines prove their identity.

If you want to go a step further and ensure the traffic is not just authenticated but also encrypted, you need to modify the Quick Mode settings. By default, Windows might only perform authentication and integrity.

# Define a custom cryptographic proposal for encryption
$CryptoProposal = New-NetIPsecQuickModeCryptoProposal -Encryption AES256 -Hash SHA256

# Create a Quick Mode settings object
$QMSetting = New-NetIPsecQuickModeSetting -DisplayName "AES-256-Encryption" -CryptoProposal $CryptoProposal

# Apply this to a rule
New-NetIPsecRule -DisplayName "Encrypted SQL Traffic" `
    -Protocol TCP `
    -LocalPort 1433 `
    -QuickModeSetting $QMSetting.Name `
    -InboundSecurity Require `
    -OutboundSecurity Require

In this example, we specifically targeted TCP port 1433 (SQL Server). This is a much more granular approach, ensuring that only the sensitive database traffic incurs the CPU overhead of encryption, while other traffic (like ICMP pings) remains fast and unencrypted.

---

Domain Isolation: A Zero Trust Foundation

One of the most powerful applications of IPsec in a Windows environment is Domain Isolation. The goal of domain isolation is to ensure that your managed assets (servers and workstations) only accept connections from other managed assets that have successfully authenticated.

Imagine a scenario where a contractor plugs a rogue laptop into an Ethernet jack in your office. Without domain isolation, that laptop might be able to scan your network, find an open SMB share, or attempt to brute-force a server. With domain isolation, your servers will simply ignore any packets coming from that laptop because the laptop cannot perform a Kerberos authentication.

Implementing Domain Isolation via Group Policy

You don't want to configure this manually on 500 servers. Instead, use Group Policy Objects (GPOs).

1. Create a GPO: Name it "IPsec - Domain Isolation."
2. Navigate to: Computer Configuration > Policies > Windows Settings > Security Settings > Windows Defender Firewall with Advanced Security.
3. Create a Connection Security Rule: Use the "Isolation" type.
4. Requirement: Set it to "Request authentication for inbound and outbound."

   Warning: Never start with "Require." If you apply a "Require" policy to your entire domain and something goes wrong (like a time sync issue), you will effectively lock every computer out of the network. Always start with "Request," verify that your "Main Mode" SAs are forming correctly in the monitoring tool, and then switch to "Require."

5. Authentication: Use Kerberos V5.

Once applied, every machine in your domain will attempt to "knock" on the door using IPsec before establishing a connection.

---

Best Practices for IPsec Configuration

Implementing IPsec can be complex, and mistakes can lead to network-wide outages. Following these industry best practices will help ensure a smooth rollout.

1. Use Strong Cryptography

The default settings in older versions of Windows Server might include outdated algorithms like DES or MD5. Modern standards require:

• Encryption: AES-128 or AES-256.
• Integrity: SHA-256 or higher.
• Key Exchange: Diffie-Hellman Group 14 or higher (2048-bit). Avoid using "Null" encryption unless you specifically only care about authentication and integrity (which is rare in high-security environments).

2. The "Request-to-Require" Migration Strategy

As mentioned earlier, always deploy IPsec rules in "Request" mode first. This allows you to monitor the Main Mode and Quick Mode statistics in the Windows Defender Firewall monitor. If you see that 100% of your traffic is successfully negotiating SAs, you can safely flip the switch to "Require."

3. Account for Network Services

Certain services do not play well with IPsec or simply cannot support it.

• DHCP and DNS: If a computer can't get an IP address because it hasn't authenticated via IPsec yet, but it can't authenticate because it doesn't have an IP, you have a "chicken and egg" problem. Always create Authentication Exemptions for your core infrastructure services like DHCP, DNS, and Domain Controllers (specifically for their initial authentication traffic).
• ICMP: Many administrators exempt ICMP (Ping) from IPsec rules to make troubleshooting easier.

4. Monitor CPU Overhead

While modern processors have AES-NI (Advanced Encryption Standard New Instructions) that make hardware-accelerated encryption very fast, IPsec still uses some CPU. On high-traffic database servers or file servers, monitor the CPU usage after enabling encryption to ensure it doesn't impact application performance.

Callout: IPsec and NAT

IPsec can be tricky when traffic passes through a NAT (Network Address Translation) device. When ESP is used, Windows uses a feature called NAT Traversal (NAT-T). This wraps the IPsec ESP packet inside a UDP 4500 packet. Most modern firewalls handle this automatically, but if you are having trouble connecting over the internet or through a complex internal firewall, ensure that UDP port 500 (IKE) and UDP port 4500 (NAT-T) are open.

---

Troubleshooting IPsec Connections

When IPsec fails, it usually fails silently—traffic just stops flowing. Here is how to figure out what went wrong.

1. The WFAS Monitor

The best place to start is the Monitoring section of the wf.msc console.

• Main Mode SAs: If this list is empty, the two computers aren't even finishing the first handshake. Check for mismatched authentication methods (e.g., one side wants a certificate, the other wants Kerberos) or blocked UDP 500 ports.
• Quick Mode SAs: If Main Mode is successful but Quick Mode is empty, the computers agree on who they are, but they can't agree on how to protect the data. Check your cryptographic suites (AES vs 3DES) and ensure they match on both ends.

2. Event Viewer

Windows logs IPsec events in the Security log. Look for:

• Event ID 4652: Failed Main Mode negotiation.
• Event ID 4653: Failed Quick Mode negotiation. The "Failure Reason" in these logs is often surprisingly helpful, pointing to things like "No certificate found" or "Invalid proposal."

3. Check Time Synchronization

Kerberos authentication relies heavily on time. If the clock on your server is more than five minutes off from the clock on the other endpoint or the Domain Controller, IPsec negotiation via Kerberos will fail. Ensure your NTP (Network Time Protocol) configuration is healthy across the environment.

---

Common Pitfalls and How to Avoid Them

Even experienced administrators run into these common IPsec traps:

• The "All-or-Nothing" Mistake: Attempting to encrypt all traffic on the entire network at once. Solution: Start with a single, non-critical application path (like a test web server to its database) to refine your GPO settings.
• Forgetting the Firewall Rules: IPsec handles the "how" of the connection, but the standard firewall rules still handle the "what." If your IPsec rule is perfect, but your inbound firewall rule blocks port 445, your file share will still be inaccessible. Remember that you can create firewall rules that specifically say "Allow this connection only if it is secure."
• Certificate Expiration: If you use certificates for IPsec, the day those certificates expire is the day your network stops working. Solution: Use Auto-enrollment via Active Directory Certificate Services to ensure certificates are renewed automatically before they expire.
• MTU Issues: Encryption adds extra bytes (overhead) to every packet. This can sometimes cause the packet to exceed the Maximum Transmission Unit (MTU) of the network, leading to fragmentation and dropped packets. Solution: If you notice weird performance issues or "hanging" connections, you may need to adjust the MTU on your network adapters or ensure that PMTU (Path MTU) Discovery is working.

---

Quick Reference: IPsec Ports and Protocols

If you are configuring hardware firewalls between your Windows Servers, you must permit the following traffic for IPsec to function:

Protocol / Port	Name	Purpose
UDP 500	IKE	Initial security negotiation (Main Mode).
UDP 4500	NAT-T	IPsec traffic passing through a NAT device.
IP Protocol 50	ESP	The actual encrypted data packets.
IP Protocol 51	AH	Authenticated (but not encrypted) data packets.

---

Summary and Key Takeaways

IPsec is a deep and sometimes intimidating topic, but it is one of the most effective ways to secure a Windows Server infrastructure. By moving security down to the packet level, you create a network that is resilient against snooping and unauthorized access.

Here are the key takeaways from this lesson:

• Defense in Depth: IPsec provides a layer of security that works even if an attacker is already inside your physical or virtual network.
• Authentication is Key: Most IPsec issues stem from authentication failures. Kerberos is your best friend for domain-joined servers, while certificates are necessary for everything else.
• ESP is the Standard: Use ESP instead of AH for modern deployments to ensure compatibility with NAT and to provide full encryption (confidentiality).
• Granularity Matters: You don't have to encrypt everything. Use Server-to-Server rules to protect sensitive data paths (like SQL or SMB) while leaving less sensitive traffic unburdened.
• Transition Carefully: Always use the "Request" setting when first deploying IPsec rules. Only move to "Require" after you have verified successful negotiations in the monitoring console.
• Automation is Essential: Use Group Policy and PowerShell to maintain a consistent IPsec policy across your server estate. Manual configuration is prone to errors that can cause outages.
• Monitor and Maintain: Keep an eye on the Security Event logs and your PKI health (if using certificates) to prevent silent failures from becoming major incidents.

By mastering IPsec, you are moving away from the old "castle and moat" security model and toward a modern, identity-based security architecture where every connection is verified and every bit of data is protected.