SDWAN:Chapter-3: SDWAN Overlays & ADVPNs

Fortinet Secure SD-WAN: Overlay VPNs

As we know now, FortiGate delivers fast, scalable, and flexible Secure SD-WAN on-premises and in the cloud. Fortinet Secure SD-WAN supports cloud-first, security-sensitive, and global enterprises, as well as the hybrid workforce. Our Secure Networking approach uses one operating system and consolidates SD-WAN, next-generation firewall (NGFW), advanced routing, and ZTNA application gateway functions.

Today, we will look at the first and most important component of the Fortinet SD-WAN solution which is Overlay VPNS. Overlay VPNS are Dial-up IPSEC VPNs used to create path between Hub and spoke over any underlay transport medium. The underlay medium can be anything like

·        MPLS

·        ILL

·        Broadband

·        4G/5G

·        P2P (But it is recommended not to use SD-WAN using P2P links)

Overlay VPNs are of two types which are used in SD-WAN

·        Dial-up VPNs (between Hub and Spoke)

·        ADVPN (Auto Discovery VPNs)

The diagram below depicts the dialup VPNs between DC/DR and Spoke location with different underlay like MPLS and ILL. If carefully seen, the strong and uninterrupted underlay connection will only facilitate the stable overlay connection between two sites.

The tunnels are created based on respective underlays created available at site and used to pass the traffic using dynamic protocols (BGP) which we will see another blog.

 Dial up VPNs:

·        Overlay VPNs basic and very important building block of the Fortinet SD-WAN technology.

·        The overlay tunnel used here are nothing but IPSEC Tunnels between Hub and spoke with difference is these are dialup VPNs with configuration different at Hub and spoke side.

·        Overlay VPNs uses Range of IPs in the configuration instead of the static IP at the Spoke end firewall.

·        Spoke location firewall connects with Hub based on need and IPs are assigned to spoke side from the range given in IPSEC configuration.

·        The rest of the IPSEC configuration is like the normal site-to-site configuration.

·        Hub and spoke configuration will vary slightly where Hub will have range and spoke will not have any range.

·        In case of the Internet underlay links, cross Tunnels can be created for redundancy in case of cross ILL Links (Primary at Hub and secondary at spoke) goes down.

·        For MPLS, there is no cross overlays possible as MPLS is a private cloud.

·        Dial UP tunnels going from spoke towards Hub side can be load balanced (maximum Bandwidth sla) using SDWAN rules.

ADVPN (Auto Discovery VPNs):

·        ADVPN (Auto Discovery VPN) is an IPsec technology that allows a traditional hub-and-spoke VPN’s spokes to establish dynamic, on-demand, direct tunnels between each other to avoid routing through the topology's hub device.

·        The primary advantage is that it provides full meshing capabilities to a standard hub-and-spoke topology without having permeant tunnels between spokes.

·        This greatly reduces the provisioning effort for full spoke-to-spoke low delay reachability and addresses the scalability issues associated with very large fully meshed VPN networks.

·        SD-WAN load-balance mode rules (or services) do not support ADVPN members. Other modes' rules, such as SLA and priority, support ADVPN members.

·        To enable the ADVPN shortcut tunnels, few command like auto-discovery-sender, auto-discovery-receiver, etc need to be added in IPSEC tunnels at Hub and spoke side.

·        In ADVPN, the first control packet will traverse through Hub and then for all subsequent data packets, a direct shortcut tunnel will be created between tow spoke location.

·        ADVPN Tunnels are Identified in format <Parent IPSEC Name>_0..1..2….n

o   Eg. H1T1S1 is parent tunnel, then first ADVPN Tunnel is H1T1S1_0

 

LAB Setup:

We will make a Use of below lab network to understand the basics of the Dialup VPNs and ADVPNs. I have taken one MPLS and one ILL link to demonstrate the overlay connectivity between DC and spoke1 & spoke-2. The lab consists of

·        DC-1

·        DC-2 (AWS DC)

·        Spoke-1

·        Spoke-2

·        Spoke-3

·        Spoke-4

·        Respective LAN Location

·        FMG: Central Management and SDWAN deployment

·        FAZ: Log collection

·        Dummy MPLS for underlay

·        3 Dummy ILLs for Underlay connectivity.

·        Net (Management network) for Firewall GUI connectivity

Hub Side VPN Configuration:

Spoke Side Configuration:

Dial UP Tunnel Demonstration

Hub Side

HUB01 # get vpn ipsec tunnel summary

'ETB-COCO_2' 40.10.1.10:0  selectors(total,up): 1/0  rx(pkt,err): 0/0  tx(pkt,err): 0/2

'H1T1S1_0' 10.10.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 8782/0  tx(pkt,err): 8781/0 ß Dialup Tunnel with Spoke-1

'H1T1S1_1' 10.20.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 999/0  tx(pkt,err): 998/0 ß Dialup Tunnel with Spoke-2

'H1-MPLS_0' 40.10.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 8875/0  tx(pkt,err): 8874/0ß Dial up Tunnel with Soke-1

 

Spoke-1 Side

Spoke-1-kol # get vpn ipsec tunnel summary

'H1-MPLS' 40.1.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 9099/0  tx(pkt,err): 9100/2

'H1T1S1' 10.1.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 9008/0  tx(pkt,err): 9009/2

Spoke-1-kol # diagnose sys sdwan health-check

Health Check(DC1_SLA):

Seq(2 H1-MPLS): state(alive), packet-loss(0.000%) latency(1.829), jitter(0.553), mos(4.403), bandwidth-up(65534999), bandwidth-dw(65534999), bandwidth-bi(131069998) sla_map=0x1

Seq(3 H1T1S1): state(alive), packet-loss(0.000%) latency(1.733), jitter(0.758), mos(4.403), bandwidth-up(65534999), bandwidth-dw(65534999), bandwidth-bi(131069998) sla_map=0x1

Spoke-2 Side

Spoke-2-Jai # get vpn ipsec tunnel summary

'H1T1S1' 10.1.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 1336/0  tx(pkt,err): 1337/5

Spoke-2-Jai # diagnose sys sdwan health-check

Health Check(DC1_SLA):

Seq(3 H1T1S1): state(alive), packet-loss(0.000%) latency(1.652), jitter(0.572), mos(4.403), bandwidth-up(65534999), bandwidth-dw(65534999), bandwidth-bi(131069998) sla_map=0x1

ADVPN Tunnel Demonstration

Spoke-1 LAN IP: 10.112.20.250

Spoke-2 LAN IP: 10.112.20.250

We will ping spoke-2 LAN IP from Spoke-1 LAN IP and vice versa. 

On Hub Side: The first control Packet will be traversed via Hub

 

HUB01 # id=65308 trace_id=3 func=print_pkt_detail line=5842 msg="vd-root:0 received a packet(proto=1, 10.112.20.250:2->10.112.50.250:2048) tun_id=192.168.242.2 from H1T1S1. type=8, code=0, id=2, seq=1."

id=65308 trace_id=3 func=init_ip_session_common line=6028 msg="allocate a new session-0000f725, tun_id=192.168.242.2"

id=65308 trace_id=3 func=rpdb_srv_match_input line=1040 msg="Match policy routing id=1: to 10.112.50.250 via ifindex-21"

id=65308 trace_id=3 func=__vf_ip_route_input_rcu line=2012 msg="find a route: flag=00000000 gw-192.168.242.3 via H1T1S1"

id=65308 trace_id=3 func=__iprope_tree_check line=535 msg="gnum-100004, use addr/intf hash, len=3"

id=65308 trace_id=3 func=fw_forward_handler line=1000 msg="Allowed by Policy-102:"

id=65308 trace_id=3 func=ip_session_confirm_final line=3087 msg="npu_state=0x1100, hook=4"

id=65308 trace_id=3 func=ids_receive line=430 msg="send to ips"

id=65308 trace_id=3 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface H1T1S1, tun_id=0.0.0.0"

id=65308 trace_id=3 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel H1T1S1_1 vrf 0"

id=65308 trace_id=3 func=esp_output4 line=920 msg="IPsec encrypt/auth"

id=65308 trace_id=3 func=nipsec_set_ipsec_sa_enc line=920 msg="Trying to offload IPsec encrypt SA (p1/p2/spi={H1T1S1_1/H1T1S1/0x13f157fa}), npudev=-1, skb-dev=port2"

id=65308 trace_id=3 func=nipsec_set_ipsec_sa_enc line=965 msg="IPSec encrypt SA (p1/p2/spi={H1T1S1_1/H1T1S1/0x13f157fa}) offloading-check failed, reason_code=2."

id=65308 trace_id=3 func=ipsec_output_finish line=641 msg="send to 10.1.1.2 via intf-port2"

id=65308 trace_id=4 func=print_pkt_detail line=5842 msg="vd-root:0 received a packet(proto=1, 10.112.50.250:2->10.112.20.250:0) tun_id=192.168.242.3 from H1T1S1. type=0, code=0, id=2, seq=1."

id=65308 trace_id=4 func=resolve_ip_tuple_fast line=5930 msg="Find an existing session, id-0000f725, reply direction"

id=65308 trace_id=4 func=rpdb_srv_match_input line=1040 msg="Match policy routing id=1: to 10.112.20.250 via ifindex-21"

id=65308 trace_id=4 func=__vf_ip_route_input_rcu line=2012 msg="find a route: flag=00000000 gw-192.168.242.2 via H1T1S1"

id=65308 trace_id=4 func=npu_handle_session44 line=1322 msg="Trying to offloading session from H1T1S1 to H1T1S1, skb.npu_flag=00000000 ses.state=00012204 ses.npu_state=0x00001108"

id=65308 trace_id=4 func=fw_forward_dirty_handler line=438 msg="state=00012204, state2=00004001, npu_state=00001108"

id=65308 trace_id=4 func=ids_receive line=430 msg="send to ips"

 

On spoke-1 Side: A shortcut Tunnel will be created with Spoke-2

 

Spoke-1-kol # get vpn ipsec tunnel summary | grep H1T1S1

'H1T1S1_0' 10.20.1.10:0  selectors(total,up): 2/2  rx(pkt,err): 4/0  tx(pkt,err):ß ADVPN with spoke-2

'H1T1S1' 10.1.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 20540/0  tx(pkt,err): 20543/2

 

Spoke-1-kol # diagnose sys sdwan health-check | grep H1T1S1

Seq(3 H1T1S1): state(alive), packet-loss(0.000%) latency(1.423), jitter(0.335), mos(4.403), bandwidth-up(65534999), bandwidth-dw(65534999), bandwidth-bi(131069998) sla_map=0x1

Seq(3 H1T1S1_0): state(alive), packet-loss(0.000%) latency(1.330), jitter(0.182), mos(4.404), bandwidth-up(65535000), bandwidth-dw(65535000), bandwidth-bi(131070000) sla_map=0x1 ß ADVPN Tunnel with Spoke 2

 

On spoke-2 Side: A shortcut Tunnel will be created with Spoke-1

Spoke-2-Jai # get vpn ipsec tunnel summary | grep H1T1S1

'H1T1S1_0' 10.10.1.10:0  selectors(total,up): 2/2  rx(pkt,err): 3/0  tx(pkt,err): 4/2

'H1T1S1' 10.1.1.10:0  selectors(total,up): 1/1  rx(pkt,err): 12832/0  tx(pkt,err): 12833/5

Spoke-2-Jai # diagnose sys sdwan health-check | grep H1T1S1

Seq(3 H1T1S1): state(alive), packet-loss(0.000%) latency(1.957), jitter(0.899), mos(4.403), bandwidth-up(65534999), bandwidth-dw(65534999), bandwidth-bi(131069998) sla_map=0x1

Seq(3 H1T1S1_0): state(alive), packet-loss(0.000%) latency(1.568), jitter(0.854), mos(4.403), bandwidth-up(65535000), bandwidth-dw(65535000), bandwidth-bi(131070000) sla_map=0x1

 

In Next Blog, we will look at the BGP routing used per overlay and how the LAN routes per locations are exchanged. We will also look, what are key differences in BGP for the Hub and spoke personas in more details. Till Then, kindly go through the blog and comment in case of any query.