Learning VPP: VxLAN over IPsec

logo_fdio-300x184

Overview

The goal is to create a layer-2 encrypted tunnel. The traffic will be encapsulated in VxLAN and protected with IPsec.

VXLAN tunnel is an L2 overlay on top of an L3 network underlay. It uses the UDP protocol to traverse the network. The VXLAN frame looks as follows.

VXLAN frame

IPsec supports tunnel and transport modes. As far as our tunnel is based on VxLAN, the transport mode will be used. In this mode, only a payload of the IP packet is encrypted and/or authenticated and the IP header is not touched. The resulting frame looks as follows.

IPSEC frame in transport mode

Setup

Two Ubuntu VMs with VPP ver. 19.01 and two Ubuntu VMs representing hosts.

VXLAN setup (1)

VPP configuration

Router1

loopback create mac 1a:2b:3c:4d:5e:8f
create bridge-domain 13 learn 1 forward 1 uu-flood 1 flood 1 arp-term 0
create vxlan tunnel src 192.168.31.47 dst 192.168.31.76 vni 13
set interface l2 bridge vxlan_tunnel0 13 1
set interface l2 bridge loop0 13 bvi
set interface ip table loop0 0
set interface ip address loop0 10.100.0.6/31
ip route table 0 20.20.20.0/24 via loop0
ipsec sa add 10 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec sa add 20 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec spd add 1
set interface ipsec spd loop0 1
ipsec policy add spd 1 priority 100 inbound action bypass protocol 50
ipsec policy add spd 1 priority 100 outbound action bypass protocol 50
ipsec policy add spd 1 priority 10 outbound action protect sa 10 local-ip-range 10.10.10.0 - 10.10.10.255 remote-ip-range 20.20.20.0 - 20.20.20.255
ipsec policy add spd 1 priority 10 inbound action protect sa 20 local-ip-range 10.10.10.0 - 10.10.10.255 remote-ip-range 20.20.20.0 - 20.20.20.255

Router2

loopback create mac 1a:2b:3c:4d:5e:7f
create bridge-domain 13 learn 1 forward 1 uu-flood 1 flood 1 arp-term 0
create vxlan tunnel src 192.168.31.76 dst 192.168.31.47 vni 13
set interface l2 bridge vxlan_tunnel0 13 1
set interface l2 bridge loop0 13 bvi
set interface ip table loop0 0
set interface ip address loop0 10.100.0.7/31
ip route table 0 10.10.10.0/24 via loop0
ipsec sa add 10 spi 1001 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec sa add 20 spi 1000 esp crypto-alg aes-cbc-128 crypto-key 4a506a794f574265564551694d653768 integ-alg sha1-96 integ-key 4339314b55523947594d6d3547666b45764e6a58
ipsec spd add 1
set interface ipsec spd loop0 1
ipsec policy add spd 1 priority 100 inbound action bypass protocol 50
ipsec policy add spd 1 priority 100 outbound action bypass protocol 50
ipsec policy add spd 1 priority 10 outbound action protect sa 10 local-ip-range 20.20.20.0 - 20.20.20.255 remote-ip-range 10.10.10.0 - 10.10.10.255
ipsec policy add spd 1 priority 10 inbound action protect sa 20 local-ip-range 20.20.20.0 - 20.20.20.255 remote-ip-range 10.10.10.0 - 10.10.10.255

Results

Encap trace

00:01:37:265053: dpdk-input
GigabitEthernet0/8/0 rx queue 0
buffer 0xe663: current data 0, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 1, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr 0x91b99940
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x5e94
fragment id 0x9ff3, flags DONT_FRAGMENT
ICMP echo_request checksum 0x5d06
00:01:37:265092: ethernet-input
frame: flags 0x3, hw-if-index 2, sw-if-index 2
IP4: 08:00:27:54:67:a2 -> 08:00:27:88:33:fd
00:01:37:265103: ip4-input-no-checksum
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x5e94
fragment id 0x9ff3, flags DONT_FRAGMENT
ICMP echo_request checksum 0x5d06
00:01:37:265110: ip4-lookup
fib 0 dpo-idx 21 flow hash: 0x00000000
ICMP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 64, length 84, checksum 0x5e94
fragment id 0x9ff3, flags DONT_FRAGMENT
ICMP echo_request checksum 0x5d06
00:01:37:265118: ip4-rewrite
tx_sw_if_index 3 dpo-idx 21 : ipv4 via 10.100.0.6 loop0: mtu:9000 1a2b3c4d5e6f1a2b3c4d5e7f0800 flow hash: 0x00000000
00000000: 1a2b3c4d5e6f1a2b3c4d5e7f0800450000549ff340003f015f94141414020a0a
00000020: 0a0208005d0605030016ba1c935d0000000088930100000000001011
00:01:37:265123: ipsec4-output
spd 1
00:01:37:265131: esp4-encrypt
esp: spi 1001 seq 19 crypto aes-cbc-128 integrity sha1-96
00:01:37:265168: loop0-output
loop0
IP4: 1a:2b:3c:4d:5e:7f -> 1a:2b:3c:4d:5e:6f
IPSEC_ESP: 20.20.20.2 -> 10.10.10.2
tos 0x00, ttl 254, length 136, checksum 0x8022
fragment id 0x0000
00:01:37:265175: l2-input
l2-input: sw_if_index 3 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f
00:01:37:265179: l2-fwd
l2-fwd:   sw_if_index 3 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f bd_index 1 result [0x1010000000004, 4] none
00:01:37:265183: l2-output
l2-output: sw_if_index 4 dst 1a:2b:3c:4d:5e:6f src 1a:2b:3c:4d:5e:7f data 08 00 45 00 00 88 00 00 00 00 fe 32
00:01:37:265188: vxlan4-encap
VXLAN encap to vxlan_tunnel0 vni 13
00:01:37:265192: ip4-rewrite
tx_sw_if_index 1 dpo-idx 15 : ipv4 via 192.168.31.47 GigabitEthernet0/3/0: mtu:9000 08002768d11e0800275a18a50800 flow hash: 0x00000001
00000000: 08002768d11e0800275a18a50800450000ba00000000fd11fd66c0a81f4cc0a8
00000020: 1f2f3b6112b500a600000800000000000d001a2b3c4d5e6f1a2b3c4d
00:01:37:265194: GigabitEthernet0/3/0-output
GigabitEthernet0/3/0
IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
UDP: 192.168.31.76 -> 192.168.31.47
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 15201 -> 4789
length 166, checksum 0x0000
00:01:37:265196: GigabitEthernet0/3/0-tx
GigabitEthernet0/3/0 tx queue 0
buffer 0x1aca6: current data -50, length 200, free-list 0, clone-count 0, totlen-nifb 0, trace 0x1
PKT MBUF: port 65535, nb_segs 1, pkt_len 200
buf_len 2176, data_len 200, ol_flags 0x0, data_off 78, phys_addr 0x916b2a00
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:5a:18:a5 -> 08:00:27:68:d1:1e
UDP: 192.168.31.76 -> 192.168.31.47
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 15201 -> 4789
length 166, checksum 0x0000

Decap trace

00:01:37:265912: dpdk-input
GigabitEthernet0/3/0 rx queue 0
buffer 0x357c: current data 0, length 200, free-list 0, clone-count 0, totlen-nifb 0, trace 0x2
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 200
buf_len 2176, data_len 200, ol_flags 0x0, data_off 128, phys_addr 0x918d5f80
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 62150 -> 4789
length 166, checksum 0x0000
00:01:37:265941: ethernet-input
frame: flags 0x3, hw-if-index 1, sw-if-index 1
IP4: 08:00:27:68:d1:1e -> 08:00:27:5a:18:a5
00:01:37:265945: ip4-input-no-checksum
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 62150 -> 4789
length 166, checksum 0x0000
00:01:37:265947: ip4-lookup
fib 0 dpo-idx 5 flow hash: 0x00000000
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 62150 -> 4789
length 166, checksum 0x0000
00:01:37:265951: ip4-local
UDP: 192.168.31.47 -> 192.168.31.76
tos 0x00, ttl 253, length 186, checksum 0xfd66
fragment id 0x0000
UDP: 62150 -> 4789
length 166, checksum 0x0000
00:01:37:265954: ip4-udp-lookup
UDP: src-port 62150 dst-port 4789
00:01:37:265959: vxlan4-input
VXLAN decap from vxlan_tunnel0 vni 13 next 1 error 0
00:01:37:265964: l2-input
l2-input: sw_if_index 4 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f
00:01:37:265967: l2-learn
l2-learn: sw_if_index 4 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 1
00:01:37:265971: l2-fwd
l2-fwd:   sw_if_index 4 dst 1a:2b:3c:4d:5e:7f src 1a:2b:3c:4d:5e:6f bd_index 1 result [0x700000003, 3] static age-not bvi
00:01:37:265974: ip4-input
IPSEC_ESP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 254, length 136, checksum 0x8022
fragment id 0x0000
00:01:37:265976: ipsec4-input
esp: sa_id 20 spd 1 spi 1000 seq 19
00:01:37:265980: esp4-decrypt
esp: crypto aes-cbc-128 integrity sha1-96
00:01:37:266019: ip4-input
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 254, length 84, checksum 0x8087
fragment id 0x0000
ICMP echo_reply checksum 0x6506
00:01:37:266021: ip4-lookup
fib 0 dpo-idx 23 flow hash: 0x00000000
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 254, length 84, checksum 0x8087
fragment id 0x0000
ICMP echo_reply checksum 0x6506
00:01:37:266024: ip4-rewrite
tx_sw_if_index 2 dpo-idx 23 : ipv4 via 20.20.20.2 GigabitEthernet0/8/0: mtu:9000 0800275467a20800278833fd0800 flow hash: 0x00000000
00000000: 0800275467a20800278833fd08004500005400000000fd0181870a0a0a021414
00000020: 14020000650605030016ba1c935d0000000088930100000000001011
00:01:37:266025: GigabitEthernet0/8/0-output
GigabitEthernet0/8/0
IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 253, length 84, checksum 0x8187
fragment id 0x0000
ICMP echo_reply checksum 0x6506
00:01:37:266029: GigabitEthernet0/8/0-tx
GigabitEthernet0/8/0 tx queue 0
buffer 0x1accd: current data 0, length 98, free-list 0, clone-count 0, totlen-nifb 0, trace 0x2
ip4
PKT MBUF: port 65535, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr 0x916b33c0
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 08:00:27:88:33:fd -> 08:00:27:54:67:a2
ICMP: 10.10.10.2 -> 20.20.20.2
tos 0x00, ttl 253, length 84, checksum 0x8187
fragment id 0x0000
ICMP echo_reply checksum 0x6506

References

1 thought on “Learning VPP: VxLAN over IPsec

  1. Hello…Thank you for posting this tutorial… I have followed the exact steps mentioned above…but i am seeing some strange behavior… After IPsec decryption, I see the source and dst IP of the packet is changed to vxlan src IP and dst IP

    19:44:56:819532: dpdk-input
    GigabitEthernet0/8/0 rx queue 0
    buffer 0x45bcf3: current data 0, length 200, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x2
    ext-hdr-valid
    l4-cksum-computed l4-cksum-correct
    PKT MBUF: port 0, nb_segs 1, pkt_len 200
    buf_len 2176, data_len 200, ol_flags 0x0, data_off 128, phys_addr 0xc04f3d40
    packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
    rss 0x0 fdir.hi 0x0 fdir.lo 0x0
    IP4: 08:00:27:66:6f:7b -> 08:00:27:c0:3f:f0
    UDP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 186, checksum 0x1bdd
    fragment id 0x0000
    UDP: 57332 -> 4789
    length 166, checksum 0x0000
    19:44:56:819588: ethernet-input
    frame: flags 0x3, hw-if-index 1, sw-if-index 1
    IP4: 08:00:27:66:6f:7b -> 08:00:27:c0:3f:f0
    19:44:56:819595: ip4-input-no-checksum
    UDP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 186, checksum 0x1bdd
    fragment id 0x0000
    UDP: 57332 -> 4789
    length 166, checksum 0x0000
    19:44:56:819597: nat44-ed-out2in
    NAT44_OUT2IN_ED_FAST_PATH: sw_if_index 1, next index 4, session -1
    19:44:56:819600: nat44-ed-out2in-slowpath
    NAT44_OUT2IN_ED_SLOW_PATH: sw_if_index 1, next index 1, session -1
    19:44:56:819604: ip4-lookup
    fib 0 dpo-idx 5 flow hash: 0x00000000
    UDP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 186, checksum 0x1bdd
    fragment id 0x0000
    UDP: 57332 -> 4789
    length 166, checksum 0x0000
    19:44:56:819607: ip4-local
    UDP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 186, checksum 0x1bdd
    fragment id 0x0000
    UDP: 57332 -> 4789
    length 166, checksum 0x0000
    19:44:56:819608: ip4-udp-lookup
    UDP: src-port 57332 dst-port 4789
    19:44:56:819610: vxlan4-input
    VXLAN decap from vxlan_tunnel1 vni 10 next 1 error 0
    19:44:56:819612: l2-input
    l2-input: sw_if_index 5 dst aa:aa:aa:aa:aa:aa src bb:bb:bb:bb:bb:bb
    19:44:56:819614: l2-learn
    l2-learn: sw_if_index 5 dst aa:aa:aa:aa:aa:aa src bb:bb:bb:bb:bb:bb bd_index 2
    19:44:56:819616: l2-fwd
    l2-fwd: sw_if_index 5 dst aa:aa:aa:aa:aa:aa src bb:bb:bb:bb:bb:bb bd_index 2 result [0x700000013, 19] static age-not bvi
    19:44:56:819618: ip4-input
    IPSEC_ESP: 3.3.3.3 -> 1.1.1.1
    tos 0x00, ttl 63, length 136, checksum 0x634a
    fragment id 0xcff2, flags DONT_FRAGMENT
    19:44:56:819620: ipsec4-input-feature
    IPSEC_ESP: sa_id 20 spd 1 policy 25 spi 1001 (0x000003e9) seq 21
    19:44:56:819622: esp4-decrypt
    esp: crypto aes-cbc-128 integrity sha1-96 pkt-seq 21 sa-seq 0 sa-seq-hi 0
    19:44:56:819631: ip4-input-no-checksum
    ICMP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 134, checksum 0x1c21
    fragment id 0x0000
    ICMP unknown 0xdf checksum 0x12b5
    19:44:56:819632: ip4-lookup
    fib 0 dpo-idx 5 flow hash: 0x00000000
    ICMP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 134, checksum 0x1c21
    fragment id 0x0000
    ICMP unknown 0xdf checksum 0x12b5
    19:44:56:819632: ip4-local
    ICMP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 134, checksum 0x1c21
    fragment id 0x0000
    ICMP unknown 0xdf checksum 0x12b5
    19:44:56:819633: ip4-icmp-input
    ICMP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 134, checksum 0x1c21
    fragment id 0x0000
    ICMP unknown 0xdf checksum 0x12b5
    19:44:56:819634: ip4-punt
    ICMP: 192.168.16.2 -> 192.168.16.3
    tos 0x00, ttl 253, length 134, checksum 0x1c21
    fragment id 0x0000
    ICMP unknown 0xdf checksum 0x12b5
    19:44:56:819634: error-punt
    rx:loop2
    19:44:56:819635: punt
    ip4-icmp-input: unknown type

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s