Learning VPP: IPsec GRE over VxLAN

logo_fdio-300x184

Overview

The goal is to create a layer-2 encrypted tunnel and hide inner network IP addresses.

To achieve this goal, the traffic will be encapsulated in GRE, protected with IPsec and encapsulated into VxLAN.

GRE is a tunneling protocol developed by Cisco. The GRE frame looks as follows.

GRE frame

VXLAN tunnel is an L2 overlay on top of an L3 network underlay. It uses the UDP protocol to traverse the network. The VXLAN frame looks as follows.

VXLAN frame

IPsec supports tunnel and transport modes. As far as our tunnel is based on GRE, the transport mode will be used. In this mode, only a payload of the IP packet is encrypted and/or authenticated and the IP header is not touched. The resulting frame looks as follows.

IPSEC frame in transport mode

Setup

Two Ubuntu VMs with VPP ver. 19.01 and two Ubuntu VMs representing hosts.

VXLAN setup (1)

VPP configuration

In terms of VPP we need to create two loopbacks. One loopback will be bridged with GRE-IPsec tunnel, while another will be bridged with VxLAN tunnel. And using routing we will direct traffic into the first loopback where it will be encapsulated into GRE header and encrypted with IPsec. Then the traffic will be routed into a second loopback where it will receive VxLAN header.

Router1

ipsec sa add 10 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec sa add 20 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58

loopback create mac 1a:ab:3c:4d:5e:7f
set interface ip address loop0 10.100.0.7/31
set int mtu 1360 loop0
set int l2 learn loop0 disable
create ipsec gre tunnel src 10.101.0.7 dst 10.101.0.6 local-sa 10 remote-sa 20
set int state ipsec-gre0 up
create bridge-domain 12 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1
set bridge-domain arp entry 12 10.100.0.7 1a:ab:3c:4d:5e:7f
set int l2 bridge loop0 12 bvi
set int l2 bridge ipsec-gre0 12 1

loopback create mac 1a:2b:3c:4d:5e:7f
set interface ip address loop1 10.101.0.7/31
create vxlan tunnel src 192.168.31.76 dst 192.168.31.47 vni 13
create bridge-domain 13 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1
set bridge-domain arp entry 13 10.101.0.7 1a:2b:3c:4d:5e:7f
set interface l2 bridge vxlan_tunnel0 13 1
set interface l2 bridge loop1 13 bvi

ip route add 10.10.10.0/24 via 10.100.0.6

Router2

ipsec sa add 10 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec sa add 20 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58

loopback create mac 1a:ab:3c:4d:5e:6f
set interface ip address loop0 10.100.0.6/31
set int mtu 1360 loop0
set int l2 learn loop0 disable
create ipsec gre tunnel src 10.101.0.6 dst 10.101.0.7 local-sa 10 remote-sa 20
set int state ipsec-gre0 up
create bridge-domain 12 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1
set bridge-domain arp entry 12 10.100.0.6 1a:ab:3c:4d:5e:6f
set int l2 bridge loop0 12 bvi
set int l2 bridge ipsec-gre0 12 1

loopback create mac 1a:2b:3c:4d:5e:6f
set interface ip address loop1 10.101.0.6/31
create vxlan tunnel src 192.168.31.47 dst 192.168.31.76 vni 13
create bridge-domain 13 learn 0 forward 1 uu-flood 1 flood 1 arp-term 1
set bridge-domain arp entry 13 10.101.0.6 1a:2b:3c:4d:5e:6f
set interface l2 bridge vxlan_tunnel0 13 1
set interface l2 bridge loop1 13 bvi

ip route add 20.20.20.0/24 via 10.100.0.7

Results

Encap trace

00:04:26:418264: dpdk-input
  GigabitEthernet0/8/0 rx queue 0
  buffer 0xddb4: current data 0, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
                 ext-hdr-valid 
                 l4-cksum-computed l4-cksum-correct 
  PKT MBUF: port 1, nb_segs 1, pkt_len 98
    buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr 0x8e376d80
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
  ICMP: 20.20.20.2 -> 10.10.10.2
    tos 0x00, ttl 64, length 84, checksum 0x66a0
    fragment id 0x97e7, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x2ea5
00:04:26:418338: ethernet-input
  frame: flags 0x3, hw-if-index 2, sw-if-index 2
  IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
00:04:26:418355: ip4-input-no-checksum
  ICMP: 20.20.20.2 -> 10.10.10.2
    tos 0x00, ttl 64, length 84, checksum 0x66a0
    fragment id 0x97e7, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x2ea5
00:04:26:418369: ip4-lookup
  fib 0 dpo-idx 14 flow hash: 0x00000000
  ICMP: 20.20.20.2 -> 10.10.10.2
    tos 0x00, ttl 64, length 84, checksum 0x66a0
    fragment id 0x97e7, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x2ea5
00:04:26:418385: ip4-rewrite
  tx_sw_if_index 3 dpo-idx 14 : ipv4 via 10.100.0.6 loop0: mtu:1360 1aab3c4d5e6f1aab3c4d5e7f0800 flow hash: 0x00000000
  00000000: 1aab3c4d5e6f1aab3c4d5e7f08004500005497e740003f0167a0141414020a0a
  00000020: 0a0208002ea555ec004d8f3afb5d0000000025b60400000000001011
00:04:26:418393: loop0-output
  loop0
  IP4: 1a:ab:3c:4d:5e:7f -> 1a:ab:3c:4d:5e:6f
  ICMP: 20.20.20.2 -> 10.10.10.2
    tos 0x00, ttl 63, length 84, checksum 0x67a0
    fragment id 0x97e7, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x2ea5
00:04:26:418417: l2-input
  l2-input: sw_if_index 3 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f
00:04:26:418423: l2-fwd
  l2-fwd:   sw_if_index 3 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f bd_index 1 result [0x1020000000004, 4] none
00:04:26:418428: l2-output
  l2-output: sw_if_index 4 dst 1a:ab:3c:4d:5e:6f src 1a:ab:3c:4d:5e:7f data 08 00 45 00 00 54 97 e7 40 00 3f 01
00:04:26:418434: ipsec-gre0-output
  ipsec-gre0
  00000000: 1aab3c4d5e6f1aab3c4d5e7f08004500005497e740003f0167a0141414020a0a
  00000020: 0a0208002ea555ec004d8f3afb5d0000000025b6040000000000101112131415
  00000040: 161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435
  00000060: 36370000000000000000000000000000000000000000000000000000
00:04:26:418436: ipsec-gre0-tx
  GRE: tunnel 0 len 122 src 10.101.0.7 dst 10.101.0.6 sa-id 10
00:04:26:418440: esp4-encrypt
  esp: spi 1001 seq 134 crypto aes-cbc-128 integrity sha1-96
00:04:26:418468: ip4-lookup
  fib 0 dpo-idx 19 flow hash: 0x00000000
  IPSEC_ESP: 10.101.0.7 -> 10.101.0.6
    tos 0x00, ttl 254, length 168, checksum 0xa74d
    fragment id 0x0000
00:04:26:418471: ip4-rewrite
  tx_sw_if_index 5 dpo-idx 19 : ipv4 via 10.101.0.6 loop1: mtu:9000 1a2b3c4d5e6f1a2b3c4d5e7f0800 flow hash: 0x00000000
  00000000: 1a2b3c4d5e6f1a2b3c4d5e7f0800450000a800000000fd32a84d0a6500070a65
  00000020: 0006000003e9000000870573ff85554266537fd108913fe1aba4e3fc
00:04:26:418485: loop1-output
  loop1
  IP4: 1a:2b:3c:4d:5e:7f -> 1a:2b:3c:4d:5e:6f
  IPSEC_ESP: 10.101.0.7 -> 10.101.0.6
    tos 0x00, ttl 253, length 168, checksum 0xa84d
    fragment id 0x0000
00:04:26:418488: l2-input
  l2-input: sw_if_index 5 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f
00:04:26:418489: l2-fwd
  l2-fwd:   sw_if_index 5 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f bd_index 2 result [0x1020000000006, 6] none
00:04:26:418491: l2-output
  l2-output: sw_if_index 6 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f data 08 00 45 00 00 a8 00 00 00 00 fd 32
00:04:26:418493: vxlan4-encap
  VXLAN encap to vxlan_tunnel0 vni 13
00:04:26:418499: ip4-rewrite
  tx_sw_if_index 1 dpo-idx 18 : ipv4 via 192.168.31.47 GigabitEthernet0/3/0: mtu:9000 08002768d11e0800275a18a50800 flow hash: 0x00000001
  00000000: 08002768d11e0800275a18a50800450000da00000000fd11fd46c0a81f4cc0a8
  00000020: 1f2f12b512b500c600000800000000000d001a2b3c4d5e6f1a2b3c4d
00:04:26:418500: GigabitEthernet0/3/0-output
  GigabitEthernet0/3/0
  IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
  UDP: 192.168.31.76 -> 192.168.31.47
    tos 0x00, ttl 253, length 218, checksum 0xfd46
    fragment id 0x0000
  UDP: 4789 -> 4789
    length 198, checksum 0x0000
00:04:26:418502: GigabitEthernet0/3/0-tx
  GigabitEthernet0/3/0 tx queue 0
  buffer 0x1c332: current data -50, length 232, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
  PKT MBUF: port 65535, nb_segs 1, pkt_len 232
    buf_len 2176, data_len 232, ol_flags 0x0, data_off 78, phys_addr 0x8e70cd00
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
  UDP: 192.168.31.76 -> 192.168.31.47
    tos 0x00, ttl 253, length 218, checksum 0xfd46
    fragment id 0x0000
  UDP: 4789 -> 4789
    length 198, checksum 0x0000

Decap trace

00:04:26:419224: dpdk-input
  GigabitEthernet0/3/0 rx queue 0
  buffer 0x1afa: current data 0, length 232, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
                 ext-hdr-valid 
                 l4-cksum-computed l4-cksum-correct 
  PKT MBUF: port 0, nb_segs 1, pkt_len 232
    buf_len 2176, data_len 232, ol_flags 0x0, data_off 128, phys_addr 0x8dc6bf00
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
  UDP: 192.168.31.47 -> 192.168.31.76
    tos 0x00, ttl 253, length 218, checksum 0xfd46
    fragment id 0x0000
  UDP: 4789 -> 4789
    length 198, checksum 0x0000
00:04:26:419299: ethernet-input
  frame: flags 0x3, hw-if-index 1, sw-if-index 1
  IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
00:04:26:419313: ip4-input-no-checksum
  UDP: 192.168.31.47 -> 192.168.31.76
    tos 0x00, ttl 253, length 218, checksum 0xfd46
    fragment id 0x0000
  UDP: 4789 -> 4789
    length 198, checksum 0x0000
00:04:26:419320: ip4-lookup
  fib 0 dpo-idx 5 flow hash: 0x00000000
  UDP: 192.168.31.47 -> 192.168.31.76
    tos 0x00, ttl 253, length 218, checksum 0xfd46
    fragment id 0x0000
  UDP: 4789 -> 4789
    length 198, checksum 0x0000
00:04:26:419328: ip4-local
    UDP: 192.168.31.47 -> 192.168.31.76
      tos 0x00, ttl 253, length 218, checksum 0xfd46
      fragment id 0x0000
    UDP: 4789 -> 4789
      length 198, checksum 0x0000
00:04:26:419334: ip4-udp-lookup
  UDP: src-port 4789 dst-port 4789
00:04:26:419345: vxlan4-input
  VXLAN decap from vxlan_tunnel0 vni 13 next 1 error 0
00:04:26:419358: l2-input
  l2-input: sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f
00:04:26:419365: l2-learn
  l2-learn: sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 2
00:04:26:419375: l2-fwd
  l2-fwd:   sw_if_index 6 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 2 result [0x700000005, 5] static age-not bvi 
00:04:26:419381: ip4-input
  IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
    tos 0x00, ttl 253, length 168, checksum 0xa84d
    fragment id 0x0000
00:04:26:419385: ip4-lookup
  fib 0 dpo-idx 8 flow hash: 0x00000000
  IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
    tos 0x00, ttl 253, length 168, checksum 0xa84d
    fragment id 0x0000
00:04:26:419387: ip4-local
    IPSEC_ESP: 10.101.0.6 -> 10.101.0.7
      tos 0x00, ttl 253, length 168, checksum 0xa84d
      fragment id 0x0000
00:04:26:419390: ipsec-if-input
  IPSec: spi 1000 seq 93
00:04:26:419399: esp4-decrypt
  esp: crypto aes-cbc-128 integrity sha1-96
00:04:26:419421: ipsec-gre-input
  GRE: tunnel -1 len 122 src 10.101.0.6 dst 10.101.0.7
00:04:26:419427: l2-input
  l2-input: sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f
00:04:26:419429: l2-learn
  l2-learn: sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f bd_index 1
00:04:26:419435: l2-fwd
  l2-fwd:   sw_if_index 4 dst 1a:ab:3c:4d:5e:7f src 1a:ab:3c:4d:5e:6f bd_index 1 result [0x700000003, 3] static age-not bvi 
00:04:26:419438: ip4-input
  ICMP: 10.10.10.2 -> 20.20.20.2
    tos 0x00, ttl 63, length 84, checksum 0x82f4
    fragment id 0xbc93
  ICMP echo_reply checksum 0x36a5
00:04:26:419441: ip4-lookup
  fib 0 dpo-idx 26 flow hash: 0x00000000
  ICMP: 10.10.10.2 -> 20.20.20.2
    tos 0x00, ttl 63, length 84, checksum 0x82f4
    fragment id 0xbc93
  ICMP echo_reply checksum 0x36a5
00:04:26:419445: ip4-rewrite
  tx_sw_if_index 2 dpo-idx 26 : ipv4 via 20.20.20.2 GigabitEthernet0/8/0: mtu:9000 0800275467a20800278833fd0800 flow hash: 0x00000000
  00000000: 0800275467a20800278833fd080045000054bc9300003e0183f40a0a0a021414
  00000020: 1402000036a555ec004d8f3afb5d0000000025b60400000000001011
00:04:26:419449: GigabitEthernet0/8/0-output
  GigabitEthernet0/8/0
  IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
  ICMP: 10.10.10.2 -> 20.20.20.2
    tos 0x00, ttl 62, length 84, checksum 0x83f4
    fragment id 0xbc93
  ICMP echo_reply checksum 0x36a5
00:04:26:419455: GigabitEthernet0/8/0-tx
  GigabitEthernet0/8/0 tx queue 0
  buffer 0x1c359: current data 38, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
  PKT MBUF: port 65535, nb_segs 1, pkt_len 98
    buf_len 2176, data_len 98, ol_flags 0x0, data_off 166, phys_addr 0x8e70d6c0
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
  IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
  ICMP: 10.10.10.2 -> 20.20.20.2
    tos 0x00, ttl 62, length 84, checksum 0x83f4
    fragment id 0xbc93
  ICMP echo_reply checksum 0x36a5

References