How to Troubleshoot FortiGate IPsec VPN Phase 1 and Phase 2 Problems

Site-to-site IPsec VPNs form the backbone of modern enterprise WAN architectures. When an IPsec tunnel fails or drops packets, business-critical applications halt, requiring rapid diagnosis. Understanding how to perform systematic FortiGate IPsec VPN troubleshooting allows network security engineers to isolate failures quickly. This guide provides a proven, step-by-step methodology to diagnose Phase 1 negotiation, Phase 2 security associations, routing behavior, and firewall policy enforcement on FortiOS.

Real-Life Scenario

An enterprise organization maintains a central Headquarter (HQ) datacenter protected by a FortiGate firewall and a remote Branch office using a second FortiGate. The business relies on an IPsec tunnel to carry internal application traffic between the Branch network (10.0.2.0/24) and HQ servers (10.0.1.0/24).

Following a scheduled maintenance window, users at the Branch office report complete loss of connectivity to internal HQ applications. The tunnel status in the FortiOS Web-based Manager displays as down, and automated health checks are failing. As the lead security engineer, you must systematically isolate whether the failure stems from IKE Phase 1 authentication, Phase 2 selector mismatches, routing table omissions, or firewall policy blocks.

Lab Topology

 [ Branch Host ] 
 (10.0.2.10)
      |
      | internal (port2)
 [ Branch-FortiGate ] 
 (WAN: port1 - 198.51.100.2)
      |
      | Public Internet / WAN Simulation
      | (UDP 500 / UDP 4500 / ESP)
      |
 (WAN: port1 - 192.0.2.2)
 [ HQ-FortiGate ]
      | internal (port2)
      |
 [ HQ Application Server ]
 (10.0.1.10)

Example Addressing and Objects

The following table outlines the IP addresses, interface designations, and firewall objects used throughout this tutorial. All public and private IP addresses are LAB/EXAMPLE values and must be adapted to match your production environment.

Prerequisites

• Two FortiGate units running FortiOS (GUI paths and CLI commands in this tutorial reflect FortiOS 7.0 and higher; minor variations exist in earlier releases).
• Administrative access via SSH and HTTPS to both FortiGate devices.
• Basic understanding of IKEv1/IKEv2 protocols, proposal negotiation, and routing principles.

GUI Configuration Overview

To inspect or create an IPsec tunnel using the FortiOS Web-based Manager, navigate to the VPN management menu. Note that exact GUI menu names may vary slightly across different FortiOS releases and platform models.

1. Go to VPN > IPsec Tunnels and select Create New > IPsec Tunnel.
2. Enter a descriptive tunnel name (e.g., HQ-to-Branch) and choose Custom template for complete control over Phase 1 and Phase 2 parameters.
3. In the Network section, specify the public IP address of the remote peer (e.g., 198.51.100.2) and set the outgoing WAN interface.
4. In the Authentication section, select Pre-shared Key and enter the matching secret key on both firewalls.
5. In the Phase 1 Proposal section, configure matching encryption (e.g., AES256), authentication (e.g., SHA256), and Diffie-Hellman (DH) groups (e.g., Group 14 or 20). Select IKE version 2 for optimal performance and stability.
6. In the Phase 2 Selectors section, configure the local and remote subnets. For route-based VPNs, standard practice uses named address objects or 0.0.0.0/0 selectors.
7. Navigate to Network > Static Routes and add a route directing traffic for the remote subnet to the newly created IPsec virtual interface.
8. Navigate to Policy & Objects > Firewall Policy and create bidirectional security policies allowing traffic between internal LAN interfaces and the IPsec virtual interface.

CLI Configuration

Below is the complete FortiOS CLI configuration for the primary HQ FortiGate firewall. Adapt interface names and public IP addresses for your specific deployment.

config vpn ipsec phase1-interface
    edit "HQ-to-Branch"
        set interface "port1"
        set ike-version 2
        set keylife 86400
        set peertype any
        set net-device disable
        set proposal aes256-sha256
        set dpd on-idle
        set remote-gw 198.51.100.2
        set psksecret EXAMPLE_PRESHARED_KEY_123
        set dhgrp 14 20
    next
end

config vpn ipsec phase2-interface
    edit "HQ-to-Branch-P2"
        set phase1name "HQ-to-Branch"
        set proposal aes256-sha256
        set dhgrp 14 20
        set keylifeseconds 43200
        set src-subnet 10.0.1.0 255.255.255.0
        set dst-subnet 10.0.2.0 255.255.255.0
    next
end

config router static
    edit 0
        set dst 10.0.2.0 255.255.255.0
        set device "HQ-to-Branch"
    next
end

config firewall policy
    edit 0
        set name "LAN-to-Branch-VPN"
        set srcintf "port2"
        set dstintf "HQ-to-Branch"
        set srcaddr "HQ-LAN-Subnet"
        set dstaddr "Branch-LAN-Subnet"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 0
        set name "Branch-VPN-to-LAN"
        set srcintf "HQ-to-Branch"
        set dstintf "port2"
        set srcaddr "Branch-LAN-Subnet"
        set dstaddr "HQ-LAN-Subnet"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end

How the Traffic Flows

Understanding packet processing within FortiOS is critical for effective troubleshooting. When an end-user host on the Branch network initiates a connection to an internal HQ server, traffic traverses FortiOS in the following sequence:

1. Routing Lookup (Ingress): The ingress FortiGate receives an unencrypted packet on its internal interface (port2) and evaluates the routing table. It finds a static route matching the destination IP (10.0.1.10) pointing to the IPsec tunnel interface.
2. Firewall Policy Check: The firewall verifies whether an active firewall policy permits traffic from the ingress LAN interface to the IPsec egress virtual interface. If approved, processing moves to encryption.
3. Phase 1 SA Check: FortiOS checks if an active Phase 1 Security Association (SA) exists with the remote peer IP address. If no Phase 1 SA is present, the IKE daemon initiates an IKE negotiation exchange over UDP port 500 (or UDP port 4500 if NAT is detected).
4. Phase 2 SA Negotiation: Once Phase 1 authenticates, the peers negotiate Phase 2 SAs using Quick Mode (IKEv1) or CREATE_CHILD_SA (IKEv2), validating local and remote subnet selectors.
5. Encapsulation and Egress: FortiOS encrypts the payload using ESP (IP Protocol 50) or UDP port 4500 (NAT-Traversal). It prepends the external public WAN headers and transmits the packet via the physical WAN interface.
6. Decapsulation (Remote Peer): The receiving FortiGate captures the inbound ESP packets, decrypts the traffic using the active Phase 2 SA, evaluates its local firewall policy (VPN to LAN), and forwards the cleartext packet to the destination server.

Verification

Before launching invasive debug routines, execute basic status commands to determine the high-level state of Phase 1 and Phase 2 negotiations.

Check active Phase 1 Security Associations:

diagnose vpn ike gateway list name HQ-to-Branch

This command displays peer IP addresses, assigned roles (initiator or responder), active IKE versions, used proposals, and current lifetime timers. Look for state indicator established.

Next, examine Phase 2 Security Associations and packet counters:

diagnose vpn tunnel list name HQ-to-Branch-P2

Verify that both incoming and outgoing SPIs (Security Parameter Indexes) are present. Check that packet counters for decapsulate and encapsulate increase when sending test traffic.

To inspect active routing entries for the remote network:

get router info routing-table details 10.0.2.0

A Structured Workflow for FortiGate IPsec VPN Troubleshooting

When an IPsec tunnel fails to initiate or drops packets, follow this repeatable troubleshooting workflow to systematically isolate the fault layer.

Step 1: Physical Link and UDP Transport Verification

Before analyzing IPsec protocol negotiations, confirm underlying network reachability between the WAN interfaces. Intermediate firewalls or ISP path blocks often prevent UDP port 500 or 4500 packets from reaching the FortiGate.

Run a targeted packet sniffer on the outgoing WAN interface:

diagnose sniffer packet port1 "host 198.51.100.2 and (port 500 or port 4500 or proto 50)" 4 10 l

While the sniffer runs, trigger tunnel negotiation from the CLI:

execute vpn ike gateway initiate HQ-to-Branch

Analysis: If you see outgoing UDP 500 packets but receive no incoming reply, check intermediate routers, upstream ISP filters, or local-in policies on the remote FortiGate.

Step 2: Phase 1 Diagnostics (IKE Debugging)

If transport connectivity is confirmed but Phase 1 fails to reach an established state, enable the IKE debug daemon. This provides real-time visibility into authentication and proposal negotiation errors.

Enable Phase 1 debug logs:

diagnose debug reset
diagnose debug console timestamp enable
diagnose debug application ike -1
diagnose debug enable

Attempt to initiate the tunnel again using execute vpn ike gateway initiate HQ-to-Branch. Observe the output for specific diagnostic signatures:

• PSK mismatch / Authentication failed: Indicates the pre-shared key does not match on both ends. Re-enter the PSK on both firewalls.
• no proposal chosen: Indicates mismatched encryption algorithms, authentication hashes, or DH groups in Phase 1 configuration.
• peer set constraint failed: ID mismatch: Indicates the Local ID or Remote ID configured under Phase 1 settings does not match the peer’s expected value.

Cleanup Step: Always disable debug tracing immediately after capturing relevant logs:

diagnose debug disable
diagnose debug reset

Step 3: Phase 2 Diagnostics (Tunnel Selectors & Proposals)

If Phase 1 establishes successfully but Phase 2 fails, the issue involves Phase 2 proposals, Perfect Forward Secrecy (PFS), or mismatched subnet selectors (Proxy IDs).

Examine the detailed tunnel state:

diagnose vpn tunnel list name HQ-to-Branch-P2

If Phase 2 fails to bring up SAs, use IKE debugging filtered specifically to Phase 2 operations:

diagnose debug reset
diagnose debug console timestamp enable
diagnose debug application ike 255
diagnose debug enable

Look for signature error messages in the terminal output:

• msg="failed to find phase2": The subnet selectors (Local Address and Remote Address) defined on Peer A do not exactly match the inverse definitions on Peer B. For example, if HQ defines 10.0.1.0/24 -> 10.0.2.0/24, Branch must define 10.0.2.0/24 -> 10.0.1.0/24.
• no SA proposal chosen: Check for mismatched Phase 2 encryption, hash algorithms, or DH group settings (PFS).

Disable debugging once complete:

diagnose debug disable
diagnose debug reset

Step 4: End-to-End Traffic Verification using Debug Flow

If both Phase 1 and Phase 2 status checks show established SAs, but traffic fails to pass across the tunnel, the issue resides in firewall policies, Network Address Translation (NAT), or internal server routing.

Use the FortiOS trace debug engine to follow a test packet through the firewall processing path:

diagnose debug reset
diagnose debug console timestamp enable
diagnose debug flow filter saddr 10.0.1.10
diagnose debug flow filter daddr 10.0.2.10
diagnose debug flow show console enable
diagnose debug flow trace start 20
diagnose debug enable

Initiate a ping or TCP connection from host 10.0.1.10 toward 10.0.2.10. Analyze the trace output for common execution outputs:

• Denied by forward policy check: Indicates a missing or misconfigured firewall policy allowing traffic from the source LAN interface to the IPsec virtual tunnel interface.
• No matching route exists: Indicates a missing static or dynamic route for the target destination subnet.
• Match policy-0 ... SNAT 10.0.1.10 -> x.x.x.x: Indicates that source NAT (NAT mode) is active on the firewall policy. Applying NAT to site-to-site IPsec traffic changes the inner source IP, breaking Phase 2 selector matches or remote return routing. Ensure NAT is disabled on IPsec policies.

Disable the debug flow engine after diagnostic completion:

diagnose debug flow trace stop
diagnose debug disable
diagnose debug reset

Common Mistakes

• Mismatched Pre-Shared Keys: Typographic errors in pre-shared keys prevent Phase 1 completion. Always verify keys or re-enter them on both peers when encountering authentication failures.
• Proxy ID / Subnet Selector Mismatches: When building IPsec tunnels to non-FortiGate appliances (or using strict Phase 2 configurations), ensure proxy IDs match symmetrically. Route-based FortiGate to FortiGate tunnels typically work best with selectors set to 0.0.0.0/0 on both ends.
• Accidental Policy NAT: Enabling NAT on site-to-site firewall policies alters original packet header addresses, preventing hosts on remote sites from communicating properly across the tunnel.
• Missing Return Static Routes: Creating a route to send traffic out the VPN tunnel is necessary, but the remote firewall must also have an active route pointing back to the initiating subnet.
• Blackhole Routes Missing during Flapping: When an IPsec tunnel interface goes down, static routes associated with that interface are withdrawn. If secondary routes exist, traffic might unexpectedly leak out a public WAN interface as cleartext unless prevented by blackhole routes or strict policies.
• TCP MSS and Path MTU Issues: Encapsulation overhead adds extra bytes to IPsec packets. If host packets set the Don’t Fragment (DF) bit, large frames may drop silently. Ensure TCP MSS clamping is configured on the firewall policies or interface.

Production Considerations

Deploying site-to-site IPsec VPNs in high-availability enterprise environments requires careful planning beyond standard connectivity checks:

• IKE Debug Impact: Never leave diagnose debug application ike -1 active continuously in production. High-volume IKE tracing consumes CPU resources and fills system logging buffers.
• Dead Peer Detection (DPD) Tuning: Configure DPD to rapidly detect missing peer heartbeats. Use on-idle or on-demand settings based on whether traffic flow across the tunnel is continuous or bursty.
• MSS Clamping Configuration: To prevent packet fragmentation problems across IPsec tunnels, enforce TCP MSS reduction on your VPN policies using CLI settings:

  config firewall policy
      edit <policy_id>
          set tcp-mss-sender 1360
          set tcp-mss-receiver 1360
      next
  end
  

• Redundant WAN and SD-WAN Integration: In multi-WAN environments, incorporate IPsec virtual interfaces directly into FortiOS SD-WAN zones. This allows dynamic failover, performance-based steering, and automated tunnel monitoring across secondary ISP paths.

Related FortiGate Guides

• FortiGate site-to-site IPsec VPN
• FortiGate remote-access IPsec VPN
• FortiGate session troubleshooting
• FortiGate SD-WAN configuration

Summary

Troubleshooting FortiGate IPsec VPN issues requires a logical, layered diagnostic approach. For a complete deployment example, see FortiGate site-to-site IPsec VPN configuration and FortiGate SSL VPN configuration. Begin by validating basic network transport over UDP port 500 and UDP port 4500. Use real-time IKE application debugs to resolve Phase 1 authentication and Phase 2 selector mismatches. Once security associations establish, use FortiOS packet sniffers and the debug flow engine to verify static routing and confirm that firewall policies allow bidirectional traffic without unintended source NAT. By adhering to this structured workflow, security engineers can quickly resolve IPsec failures and maintain reliable enterprise connectivity.