MACsec is an interesting alternative to existing tunneling solutions, that protects Layer 2 by performing integrity, origin authentication and, optionally, encryption. Normal use-case is to use MACsec between hosts and access switches, between two hosts or between two switches. This article is a leftover from MACsec on Linux that I first tested in 2016 when support for MACsec was just included in the kernel. I will describe how MACsec is used together with a Layer 2 GRE tunnel to protect the traffic between two remote sites, over WAN or Internet, like a site-to-site VPN at Layer 2.

Scenario Overview

Today, I am not going to present MACsec - you can read my previous post about MACsec implementation on Linux. Instead, this post presents a Layer 2 site-to-site VPN scenario using two Linux machines that perform MACsec inside a GRETAP tunnel - GRETAP is a Layer 2 tunnel. Here is a diagram with some high level notes:

MACsec over WAN Overview

  • there are two remote sites, Site 1 and Site 2, connected over a private WAN or over the Internet
  • both sites use same IP address space
  • a Layer 2 Tunnel is connecting the sites - in our demo, we are using a GRETAP tunnel, but any other L2 tunneling protocols could also be used

For demo purposes, I am using network namespaces (netns) and virtual ethernet interfaces (veth) to simulate the entire scenario on a single Linux machine. Let's jump right into it (assuming your machine is a Linux box), clone my github repository and run the demo-setup-MACsecOverWAN.sh bash script, as indicated below:

# preparation
mkdir macsec-demo && cd macsec-demo

# clone my demo repository from github
git clone https://github.com/costiser/macsec-over-wan.git

# run the setup script with "sudo"
cd macsec-over-wan
sudo ./demo-setup-MACsecOverWAN.sh

MACsec over WAN Implementation

Demo Overview

All commands in this demo must be executed with sudo because creating network namespaces and veth interfaces require root privileges!

Running the sudo setup-MACsecOverWAN-layer2.sh bash script will do the following:

  • creates two network namespaces, host1 and host2, representing two host devices
  • creates two network namespaces, nsra and nsrb, representing Linux Routers that are the tunnel endpoints
  • creates a network namespace, wan, for the WAN or Internet cloud

MACsec over WAN

Test the connectivity between the two hosts with ICMP pings - use ip netns exec <namespace> <command> to execute commands inside the network namespaces:

sudo ip netns exec host1 ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.204 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.129 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=0.123 ms
64 bytes from 192.168.1.2: icmp_seq=4 ttl=64 time=0.120 ms
...

While ping is running, capture the traffic in the wan namespace in order to confirm the MACsec and GRETAP overlay:

sudo ip netns exec wan tcpdump -neli wan1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wan1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:21:55.814765 00:aa:aa:aa:aa:aa > 00:aa:aa:1f:1f:1f, ethertype IPv4 (0x0800), length 168: 1.1.1.1 > 2.2.2.2: GREv0, proto TEB (0x6558), length 134: 00:00:00:00:00:01 > 00:00:00:00:00:02, ethertype Unknown (0x88e5), length 130:
    0x0000:  2c00 0000 0019 0000 0011 1111 0001 aafd  ,...............
    0x0010:  7858 40ec c447 2ab3 1463 6205 272b 85ef  [email protected]*..cb.'+..
    0x0020:  6a7b 7419 c3ec 4300 d0ab 9922 797a cf72  j{t...C...."yz.r
    0x0030:  33ec 07fe 6b6b 3095 0e63 5743 06b7 813b  3...kk0..cWC...;
    0x0040:  5501 7c3c 529f 72ec 668d 24d4 c443 e9f9  U.|..e.}.
    0x0070:  0bab fae3                                ....
21:21:55.814832 00:aa:aa:1f:1f:1f > 00:aa:aa:aa:aa:aa, ethertype IPv4 (0x0800), length 168: 2.2.2.2 > 1.1.1.1: GREv0, proto TEB (0x6558), length 134: 00:00:00:00:00:02 > 00:00:00:00:00:01, ethertype Unknown (0x88e5), length 130:
    0x0000:  2c00 0000 001a 0000 0022 2222 0001 1bd5  ,........"""....
    0x0010:  10f4 a646 7057 7acb 690b 1841 b35d 606a  ...FpWz.i..A.]`j
    0x0020:  93f9 f78c 95fb 424f e92e 9828 7953 d99b  ......BO...(yS..
    0x0030:  e710 1a52 3218 4ed2 0854 e8aa 339b 4129  ...R2.N..T..3.A)
    0x0040:  07d0 cce5 ff29 b398 ded1 8549 b83c d094  .....).....I.<..
    0x0050:  1480 434f ca8b 17e4 e5bd 9fcc 5d33 617a  ..CO........]3az
    0x0060:  9651 3e68 1a0b 4db1 dcc8 3444 8d2e 0579  .Q>h..M...4D...y
    0x0070:  2392 8953                                #..S
The first EtherType in the output of the tcpdump is 0x6558 that indicates a Transparent Ethernet bridging, meaning the GRETAP.
The Unknown (0x88e5) is actually the EtherType for MAC security (IEEE 802.1AE).




Solution Explained

In order to explain how all pieces come together, let's look at the nsra and nsrb Linux routers that provide IP routing connectivity between the sites. Since MACsec operates at Layer 2, we need to build a Layer 2 tunnel between the sites. For that, we use GRETAP but any other Layer 2 Tunneling protocol can be used instead.

Note that I am setting some easy-to-read MAC addresses on most of the interfaces.

# on site1's Linux router (nsra)
ip netns exec nsra ip link add gretap1 type gretap local 1.1.1.1 remote 2.2.2.2
ip netns exec nsra ip link set gretap1 address 00:00:00:11:11:11
ip netns exec nsra ip link set gretap1 up

# on site2's Linux router (nsrb)
ip netns exec nsrb ip link add gretap1 type gretap local 2.2.2.2 remote 1.1.1.1
ip netns exec nsrb ip link set gretap1 address 00:00:00:22:22:22
ip netns exec nsrb ip link set gretap1 up

Now, we build the MACsec tunnel inside the GRETAP interface. The MACsec endpoint addresses are the ones of the parent gretap1 interface.

# on site1's Linux router (nsra)
ip netns exec nsra ip link add link gretap1 macsec1 type macsec encrypt on
ip netns exec nsra ip macsec add macsec1 tx sa 0 pn 1 on key 01 11111111111111111111111111111111
ip netns exec nsra ip macsec add macsec1 rx address 00:00:00:22:22:22 port 1
ip netns exec nsra ip macsec add macsec1 rx address 00:00:00:22:22:22 port 1 sa 0 pn 1 on key 02 22222222222222222222222222222222
ip netns exec nsra ip link set macsec1 up

# on site2's Linux router (nsrb)
ip netns exec nsrb ip link add link gretap1 macsec1 type macsec encrypt on
ip netns exec nsrb ip macsec add macsec1 tx sa 0 pn 1 on key 02 22222222222222222222222222222222
ip netns exec nsrb ip macsec add macsec1 rx address 00:00:00:11:11:11 port 1
ip netns exec nsrb ip macsec add macsec1 rx address 00:00:00:11:11:11 port 1 sa 0 pn 1 on key 01 11111111111111111111111111111111
ip netns exec nsrb ip link set macsec1 up

Last piece of the puzzle is to "force" traffic into the macsec1 interface. For that, I'm using a simple Linux bridge, br0, on each of the Linux routers that "bridges" the macsec1 interface with the internal LAN interface of the site. Note that you could achieve the same also with an Open vSwitch (OVS) switch.

# on site1's Linux router (nsra)
ip netns exec nsra ip link add br0 type bridge
ip netns exec nsra ip link set veth11 master br0
ip netns exec nsra ip link set macsec1 master br0
ip netns exec nsra ip link set br0 up

# on site2's Linux router (nsrb)
ip netns exec nsrb ip link add br0 type bridge
ip netns exec nsrb ip link set veth22 master br0
ip netns exec nsrb ip link set macsec1 master br0
ip netns exec nsrb ip link set br0 up

Packet Analysis

Let's have a quick look the traffic protected by MACsec. Perform a tcpdump packet capture in the wan network namespace, as if someone in the Internet is sniffing the traffic. Save the capture in a file named macsec.pcap:

sudo ip netns exec wan tcpdump -i wan1 -w macsec.pcap

Generate some traffic from host1. Note that I am also clearing the ARP cache since I want to capture also the broadcast ARP Requests:

# clear ARP cache for host2
sudo ip netns exec host1 arp -d 192.168.1.2

# generate some traffic
sudo ip netns exec host1 ping 192.168.1.2

PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.300 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.173 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=0.136 ms
^C
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 31ms
rtt min/avg/max/mdev = 0.136/0.203/0.300/0.070 ms

# check the ARP cache
sudo ip netns exec host1 arp -an
? (192.168.1.2) at 00:00:00:00:00:02 [ether] on veth1

Opening the macsec.pcap with Wireshark would show multiple MACsec packets carrying both ARP and ICMP data. MACsec protects all Layer 2 traffic, including broadcasts. Some of the important things to note are included in the diagram below:

MACsec Over WAN Packet Capture

Taken the ARP Request broadcast frame as an example, we can see that MACsec inserts a Security Tag (16 Bytes) between the original Ethernet header and the Data, then appends an Integrity Check Value / ICV (16 Bytes) at the end:

MACsec Frame Overview

MTU

WARNING

Everytime tunneling is involved, you must consider MTU!

Everytime tunneling is involved, you must consider MTU. In this scenario, we are using several layers of extra encapsulations, so we must decrease the MTU on the hosts. Let's do the math of what we add extra:

  • Original ETH: 14 bytes (since this is now transparently bridged over WAN)
  • MACsec: 32 bytes
  • GRETAP: 4 bytes
  • IP: 20 bytes
  • TOTAL: 70 bytes overhead
  • MTU: 1500 - 70 = 1430

We can easily confirm the Path MTU by sending ICMP with DF-bit set from the testing host1 namespace. If the calculation is correct, then the maximum ICMP packet that we can send is: 1430 - 20 (IP header) - 8 (ICMP header) = 1402. Anything bigger than 1402 would be dropped:

# WORKING: Max size of ICMP packets: 1430(MTU)-8(ICMP)-20(IP)=1402
sudo ip netns exec host1 ping -M do -c2 192.168.1.2 -s 1402
PING 192.168.1.2 (192.168.1.2) 1402(1430) bytes of data.
1410 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.144 ms
1410 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.149 ms

--- 192.168.1.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 19ms
rtt min/avg/max/mdev = 0.144/0.146/0.149/0.012 ms

# NOT WORKING: size 1403
sudo ip netns exec host1 ping -M do -c2 192.168.1.2 -s 1403
PING 192.168.1.2 (192.168.1.2) 1403(1431) bytes of data.

--- 192.168.1.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 2ms
WARNING

As you notice in the above output, when size is too big, there is nobody sending back Message too long because everything is tunnelled at Layer-2, there is no Layer 3 in this tunnel! Oversized frames are silently dropped!

Of course, instead of decreasing the MTU on the clients side, another option would be to increase it on the WAN path. This would be possible if the WAN is a network under your control, but it would be impossible if you use the Internet.

Performance

It is well known that encryption drastically decreases the network throughput. MACsec is no exception to that. Vendors providing MACsec capabilities claim line-rate performance for it. In Linux, there are efforts to offload MACsec onto NICs, but as far as I know, such work is still in progress.

In my demo, I wanted to see how much does the network performance decreases when uses MACsec. For that, I first wanted to get a baseline when the two hosts are connected via WAN with plain GRE without any MACsec involved. I have another bash script that creates the plain GRE setup and then I'm running iperf, with default options, between the host clients.

# create the demo with plain GRE
sudo ./demo-setup-GREplain.sh

# start iperf server on host2
sudo ip netns exec host2 iperf -s

# start iperf client on host1
# subsequent commands with "P" option tells iperf to use multiple parallel client threads
sudo ip netns exec host1 iperf -c 192.168.1.2
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 3
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 5
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 7

The reported baseline performance in my case was close to 27 Gbps for a single thread and up to 100 Gbps for -P 7 (seven client threads):

# output snippets from the iperf server
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  32.1 GBytes  27.6 Gbits/sec
...
[SUM]  0.0-10.0 sec  81.2 GBytes  69.7 Gbits/sec
...
[SUM]  0.0-10.0 sec   108 GBytes  92.5 Gbits/sec
...
[SUM]  0.0-10.0 sec   117 GBytes   100 Gbits/sec

Now cleanup this plain GRE demo with sudo ./demo-cleanup-GREplain.sh.

Then we start again the MACsec setup demo, we manually configure the proper MTU on clients (that is not part of the demo script), then perform the same iperf tests:

# create the demo with MACsec over GRETAP
sudo ./demo-setup-MACsecOverWAN.sh

# configure proper MTU on clients
sudo ip netns exec host1 ip link set veth1 mtu 1430
sudo ip netns exec host2 ip link set veth2 mtu 1430

# start iperf server on host2
sudo ip netns exec host2 iperf -s

# start iperf client on host1
# subsequent commands with "P" option tells iperf to use multiple parallel client threads
sudo ip netns exec host1 iperf -c 192.168.1.2
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 3
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 5
sudo ip netns exec host1 iperf -c 192.168.1.2 -P 7

The results, in my case, were 2 Gbps for a single client thread and up to 3.5 Gbps for seven threads (-P 7). Below is a snippet from the iperf -s output:

# output snippets from the iperf server with MACsec over GRETAP
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.41 GBytes  2.07 Gbits/sec
...
[SUM]  0.0-10.0 sec  3.01 GBytes  2.58 Gbits/sec
...
[SUM]  0.0-10.0 sec  3.79 GBytes  3.25 Gbits/sec
...
[SUM]  0.0-10.0 sec  4.17 GBytes  3.57 Gbits/sec
Since everything runs in a single machine, simulated with network namespaces and veth interfaces, the iperf performance depends on how busy and how powerful your Linux machine is.

You could try to tweak the Linux network stack as described in another post, but I did not spend any time on that.

To clear the entire setup, use sudo ./demo-cleanup-MACsecOverWAN.sh script. This concludes an article that I'm happy to have done with a delay of three years from my original MACsec on Linux post.


As always, thank you for your interest and looking forward to your comments!