FortiGate Packet Sniffer and Debug Flow for Real Troubleshooting

Network security engineers frequently face intermittent connection failures, silent packet drops, and routing misconfigurations. Standard ping tests and session table checks often fail to reveal why traffic stops inside a firewall. On Fortinet FortiGate appliances, mastering the FortiGate packet sniffer debug flow workflow gives you deep visibility into kernel-level packet processing.

This comprehensive technical guide demonstrates how to combine FortiGate’s built-in packet sniffer and debug flow engine to troubleshoot application connection issues. You will learn how to verify packet arrival, track stateful routing lookups, evaluate policy matches, inspect NAT translations, and analyze hardware offloading behavior.

Real-Life Scenario

In this scenario, enterprise users on an internal corporate network cannot access a hosted financial database application. The application operates on non-standard port TCP 8443 at an off-site data center.

• Symptom: Client browsers and database connectors report a “Connection Timed Out” error when attempting to reach the remote application server.
• Objective: Use FortiGate diagnostic tools to isolate whether the issue is caused by physical ingress failure, routing table drops, firewall policy rejection, improper Source NAT (SNAT), or asymmetric return routing.
• Scope: Execute a structured CLI-based diagnostic workflow on FortiGate without causing CPU performance degradation or system instability.

Lab Topology

The following diagram illustrates the traffic flow between the internal client, the FortiGate security gateway, and the target database server across the WAN connection.

+---------------------+           +-----------------------------------+           +-----------------------+
|  Internal Client    |           |    FortiGate Firewall (Lab)       |           |   App Database Server |
|  192.0.2.10/24      |-----------| Ingress: port2 (192.0.2.1/24)    |-----------|   203.0.113.50:8443       |
|  (User Subnet)      |           | Egress:  port1 (198.51.100.1/24)  |           |   (Data Center Subnet)|
+---------------------+           +-----------------------------------+           +-----------------------+

Example Addressing and Objects

All network addresses, interface names, and policy identifiers used in this tutorial are example values configured for a controlled laboratory environment. Adapt these parameters to match your specific production architecture.

Prerequisites

• Administrative CLI access (SSH or direct Console) with full super_admin read/write privileges.
• Basic understanding of stateful firewall session handling, 3-way TCP handshakes, and NAT logic.
• An active traffic source generating connection attempts during the diagnostic capture window.
• Verification of system CPU and memory load before initiating packet captures or debug flows.

GUI Packet Capture Configuration

FortiOS provides a graphical tool to capture network traffic directly into standard PCAP format. While the GUI sniffer cannot explain internal FortiGate kernel decisions, it offers an accessible method to confirm physical packet arrival and inspect payload details.

Note: GUI navigation paths can vary slightly across different FortiOS releases and feature settings. Ensure feature visibility for Packet Capture is enabled under System > Feature Visibility if required.

1. Log into the FortiGate GUI administration console.
2. Navigate to Network > Packet Capture.
3. Click Create New in the top toolbar.
4. Select the target Interface (for example, port2).
5. Set Max Packets to Capture (for example, 1000) to prevent memory exhaustion.
6. Enable Filters and specify the parameters:
   • Host: 192.0.2.10
   • Port: 8443
7. Select Save and Start to activate packet acquisition.
8. Trigger test traffic from the internal host.
9. Select Stop once packets are collected, then click Download PCAP File to analyze the capture in Wireshark.

While GUI packet capture helps confirm whether frames reach an interface, it cannot explain why a packet was dropped by a firewall policy, dropped by RPF checks, or misrouted. To inspect kernel-level policy and routing logic, you must use the CLI.

CLI Diagnostic Workflow

The FortiGate Command Line Interface provides two essential diagnostic utilities: diagnose sniffer packet and diagnose debug flow. A structured diagnostic approach uses the sniffer first to establish wire-level presence, followed by debug flow to trace kernel logic.

Step 1: Running the Built-in CLI Packet Sniffer

The built-in sniffer displays real-time packet headers or payloads passing through network interfaces. The basic CLI syntax for the packet sniffer is:

diagnose sniffer packet <interface> '<filter>' <verbose_level> <count> <timestamp_format>

Understanding verbose levels is crucial when selecting diagnostic parameters:

• 1: Print header of packets.
• 2: Print header and IP payload data in hex/ASCII.
• 3: Print header and Ethernet packet payload in hex/ASCII.
• 4: Print header of packets with interface name.
• 5: Print header and IP payload data with interface name.
• 6: Print header, Ethernet payload, and interface name with absolute timestamp.

Execute the following command to capture TCP port 8443 traffic across all interfaces simultaneously using verbose level 4:

diagnose sniffer packet any 'host 192.0.2.10 and port 8443' 4 20 a

Analyzing typical output from a successful initial packet capture:

2026-03-30 10:15:01.123456 port2 in 192.0.2.10.51234 -> 203.0.113.50.8443: syn 3452130491
2026-03-30 10:15:01.123510 port1 out 198.51.100.1.51234 -> 203.0.113.50.8443: syn 3452130491

This output proves two crucial steps: the initial TCP SYN frame entered port2, and the FortiGate successfully processed SNAT and routed the packet out of port1. However, if the packet enters port2 but no line shows it leaving port1, you must invoke the debug flow engine to determine where it was dropped.

Step 2: Executing Debug Flow Analysis

The debug flow utility traces the FortiGate kernel path for individual packets. It details interface reception, route table lookups, stateful session creation, firewall policy matching, NAT translations, and drop conditions.

To safely run a debug flow, execute the exact sequence below in your CLI session:

diagnose debug disable
diagnose debug reset
diagnose debug flow filter clear
diagnose debug flow filter saddr 192.0.2.10
diagnose debug flow filter daddr 203.0.113.50
diagnose debug flow filter port 8443
diagnose debug flow show function-name enable
diagnose debug console timestamp enable
diagnose debug flow trace start 100
diagnose debug enable

Generate a connection attempt from client 192.0.2.10 to server 203.0.113.50:8443 while monitoring the console output.

Step 3: Safe Diagnostic Cleanup Command Sequence

When you complete the diagnostic trace, run this explicit cleanup sequence to restore normal CPU processing and disable active debugging processes:

diagnose debug disable
diagnose debug flow trace stop
diagnose debug flow filter clear
diagnose debug reset

How Traffic Flows in the FortiGate Kernel

To correctly interpret diagnostic traces, you must understand how FortiOS processes incoming IP frames. When a packet reaches a physical interface, the FortiGate stateful packet processing engine handles it through a defined pipeline:

1. Ingress Header Validation: The firewall inspects IP headers, verifies checksums, and drops corrupt packets.
2. Stateful Session Lookup: FortiOS checks its internal session table. If a session already exists for the TCP 5-tuple (source IP, source port, destination IP, destination port, protocol), subsequent security checks are streamlined.
3. Reverse Path Forwarding (RPF) Check: The engine checks if the packet arrived on the interface that routing tables would use to return traffic to the source IP. If RPF fails, the packet drops silently.
4. Routing Lookup (For First Packet / SYN): The firewall queries the routing table to identify the egress interface and next-hop gateway address.
5. Firewall Policy Matching: FortiOS evaluates policies sequentially top-to-bottom based on source/destination interfaces, source/destination address objects, services, and schedules.
6. Stateful Session Creation: Upon hitting an “ACCEPT” policy, the kernel constructs a session entry. If Security Profiles (IPS, Antivirus, Web Filtering) apply, traffic routes to flow-based or proxy-based inspection engines.
7. Source/Destination NAT Execution: The kernel modifies source IP/ports (SNAT) or destination IP/ports (DNAT) according to firewall policy or Central NAT rules.
8. Egress Processing and Offloading: The packet exits the egress interface. If traffic qualifies for hardware offloading (NP6/NP7 SPU ASICs), subsequent packets bypass the main CPU kernel state engine entirely.

Verification and Debug Trace Output Analysis

Below is an annotated analysis of a trace captured during an application connection test. This step-by-step breakdown explains what each log entry proves.

Example 1: Successful Packet Traversal and SNAT

1: 2026-03-30 10:20:05.102341 id=20013 trace_id=1 msg="vd-root:0:received a packet(proto=6, 192.0.2.10:52134->203.0.113.50:8443) from port2. flag [S], seq 1000, ack 0, win 64240"
2: 2026-03-30 10:20:05.102355 id=20013 trace_id=1 msg="allocate a new session-0004abcd, npu shares=00000000/00000000"
3: 2026-03-30 10:20:05.102368 id=20013 trace_id=1 msg="find a route: flag=00000001 gw-198.51.100.254 via port1"
4: 2026-03-30 10:20:05.102380 id=20013 trace_id=1 msg="Allowed by Forward Policy-5:"
5: 2026-03-30 10:20:05.102392 id=20013 trace_id=1 msg="SNAT 192.0.2.10->198.51.100.1:52134"
6: 2026-03-30 10:20:05.102410 id=20013 trace_id=1 msg="msg_id=0: output[port1] egress intf port1"

Detailed Line Analysis:

• Line 1: Confirms physical reception of a TCP SYN flag (flag [S]) on ingress interface port2. Proves ingress connectivity works.
• Line 2: Indicates no existing session was found; the kernel allocates session structure 0004abcd.
• Line 3: Confirms successful FIB lookup. Next hop is 198.51.100.254 via physical interface port1. Proves valid egress routing.
• Line 4: Matches incoming traffic against Firewall Policy ID 5. Proves policy criteria (source, destination, service) match.
• Line 5: Evaluates Source NAT. Translates client private IP 192.0.2.10 to WAN interface address 198.51.100.1.
• Line 6: Hands off packet to egress physical driver for port1. Proves successful firewall output execution.

Example 2: Common Failure Output – Policy Drop

If the debug trace displays the following log, the connection failure occurs inside the policy evaluation phase:

1: 2026-03-30 10:25:12.334112 id=20013 trace_id=2 msg="vd-root:0:received a packet(proto=6, 192.0.2.10:52135->203.0.113.50:8443) from port2. flag [S], seq 2000, ack 0, win 64240"
2: 2026-03-30 10:25:12.334125 id=20013 trace_id=2 msg="allocate a new session-0004abce, npu shares=00000000/00000000"
3: 2026-03-30 10:25:12.334138 id=20013 trace_id=2 msg="find a route: flag=00000001 gw-198.51.100.254 via port1"
4: 2026-03-30 10:25:12.334150 id=20013 trace_id=2 msg="Denied by forward policy 0"

Analysis: Match failure on forward policy. Policy 0 represents the implicit firewall deny rule. This proves that either no policy permits traffic from port2 to port1, or existing policies do not include TCP port 8443 or host object 203.0.113.50.

Example 3: Common Failure Output – Reverse Path Forwarding Drop

1: 2026-03-30 10:30:15.551201 id=20013 trace_id=3 msg="vd-root:0:received a packet(proto=6, 192.0.2.10:52136->203.0.113.50:8443) from port2. flag [S], seq 3000, ack 0, win 64240"
2: 2026-03-30 10:30:15.551220 id=20013 trace_id=3 msg="Reverse path check fail: val=1, intf=port2, src=192.0.2.10"
3: 2026-03-30 10:30:15.551231 id=20013 trace_id=3 msg="pdrop code=103(ip_src_check_failed), drop"

Analysis: FortiOS rejected the packet due to strict or loose RPF verification. The routing table indicates that packets destined for source address 192.0.2.10 should be routed via a different interface, not port2. This typically points to asymmetric routing or missing internal static routes.

Troubleshooting Decision Matrix

Use this decision matrix to isolate and resolve drop causes based on CLI diagnostic results:

Common Mistakes

• Unfiltered Debugging: Running diagnose debug flow trace start without defining exact host or port filters. On high-throughput appliances, this can lock up management SSH sessions and cause high CPU utilization.
• Forgetting Session Offloading Behavior: Expecting every frame of an active TCP stream to appear in debug flow output. Network Processors (NP6/NP7) bypass CPU debug hooks once sessions establish.
• Misinterpreting Packet Direction: Using diagnose sniffer packet any without checking incoming versus outgoing interface identifiers (in vs out).
• Leaving Debug On: Failing to run diagnose debug disable after finishing troubleshooting sessions, causing unnecessary logging overhead.
• Ignoring Central NAT Rules: Searching for policy-based NAT configuration issues when the firewall operates in Central NAT mode.

Production Considerations

When executing troubleshooting commands in critical enterprise environments, implement these operational safeguard procedures:

Handling Hardware ASIC Offloading During Debugging

By default, the FortiGate kernel offloads established stateful sessions to hardware SPU/NPU processors (such as NP6 or NP7). When offloading occurs, subsequent packets bypass the main CPU state engine, meaning debug flow traces will show session creation and then stop logging data for that flow.

If you must capture full, ongoing packet traces for an existing stream, temporarily disable auto ASIC offloading on the specific testing firewall policy:

config firewall policy
    edit <policy_id>
        set auto-asic-offload disable
    next
end

Note: Remember to re-enable auto-asic-offload immediately after testing to prevent sustained CPU load on production systems.

Managing Session Limits and Buffer Sizing

To avoid console line buffer overflow during high-traffic captures, limit trace lines explicitly using diagnose debug flow trace start <count>. Setting a low number (such as 20 or 50) ensures that the output remains manageable and stops automatically after capturing the necessary packets.

Related FortiGate Guides

• FortiGate session troubleshooting with flow debug, sessions and logs
• FortiGate IPsec VPN troubleshooting
• FortiGate Syslog and SIEM integration
• FortiGate security profiles

Summary

Using the FortiGate packet sniffer debug flow methodology transforms complex connection troubleshooting into a structured, step-by-step diagnostic workflow. By first establishing packet presence with diagnose sniffer packet and then tracing kernel processing logic with diagnose debug flow, network engineers can quickly isolate whether issues stem from cabling, routing tables, policy rules, NAT settings, or external destination networks.

Consistently using specific diagnostic filters and executing cleanup steps after testing ensures that troubleshooting operations remain safe, non-disruptive, and effective across all enterprise deployments.