SR-IOV Trusted VF and VLAN trunking

Use case:
SR-IOV VF port binding to VNF provisions the higher level of throughput and latency
performance which is a generic demand for the mission critical infrastructure. Such ports can
work in “access”,flat”, or “promiscuous mode” in relation to the VNF interface and carry VLAN
tags accordingly.
VNFs with large number of ingress/egress ports (i.e. vFirewalls) can use these features to carry
VLAN tags as per customer design needs.

Red Hat OpenStack Platform 14 (Community release “Rocky”) implements a “Trusted VF”
feature which supports such uses cases. Using this feature customer can have:-

  • VLAN tunneling (QinQ)
  • Extend VLANs inside of VNF to Upstream ToR
  • Port binding inside VNF using multiple VFs.
Topology

NIC & Drivers:

i40e: Intel(R) Ethernet Connection XL710 Network Driver – version 2.3.2-k

Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

Details:

Let’s take the first use case of VLAN tunneling (QinQ).

In this case, VLANs configured inside the VNF are tunneled by the VLAN configured on the  SR-IOV VF of the host node. Mac address of the VNF interfaces of VNF are learned by the upstream switch with VLAN tag of the host node VF and hides VLANs of the VNFs.

By default, “trust” mode is off on host node for VF. This can be modified through Red Hat OpenStack Platform Director (OpenStack Triple-O) using Heat templates. The following is the corresponding Heat template parameter.

NovaPCIPassthrough:

    – devname: “p7p3”

      trusted: “true”

      physical_network: “sriov-1”

The above parameter configures port “p7p3” as a “trusted” port.

Once OpenStack stack is deployed, we need to create Neutron network, subnet, and port for the VFs.

openstack network create –internal –provider-network-type vlan –provider-physical-network sriov-1 –provider-segment 30 –disable-port-security VLAN_Tun
openstack subnet create –network VLAN_Tun –subnet-range 6.6.6.0/24 –no-dhcp VLAN_Tun_Sub

Create SR-IOV VF port for two compute nodes:

openstack port create –network VLAN_Tun –vnic-type direct VNF1VF –binding-profile trusted=true
openstack port create –network VLAN_Tun –vnic-type direct VNF2VF –binding-profile trusted=true

Let’s create instances with OpenStack ports.

openstack server create –flavor F1 –image Image1 –nic net-id=<Management_Network_UUID> –nic port-id=<VNF1VF_PortID> –key-name test –availability-zone nova:compute-0.localdomain VNF1
openstack server create –flavor F1 –image Image1 –nic net-id=<Management_Network_UUID> –nic port-id=<VNF2VF_PortID –key-name test –availability-zone nova:compute-1.localdomain VNF2

Once both VNFs are up and running, we need to create subinterfaces and assign mac addresses to them. By default, VLAN subinterfaces will pick VF’s mac address.

Inside VNF1:

sudo ip link add link eth1 name eth1.201 type vlan id 201sudo ip link set dev eth1.201 address fa:16:3e:c5:c7:e9 sudo ip link set eth1.201 upsudo ip addr add 11.11.11.5/24 dev eth1.201
sudo ip link add link eth1 name eth1.202 type vlan id 202sudo ip link set dev eth1.202 address fa:16:3e:46:e8:07sudo ip link set eth1.202 upsudo ip addr add 12.12.12.6/24 dev eth1.202

Inside VNF2:

sudo ip link add link eth1 name eth1.201 type vlan id 201sudo ip link set dev eth1.201 address fa:16:3e:3d:b6:5dsudo ip link set eth1.201 upsudo ip addr add 11.11.11.9/24 dev eth1.201
sudo ip link add link eth1 name eth1.202 type vlan id 202sudo ip link set dev eth1.202 address fa:16:3e:27:b3:25sudo ip link set eth1.202 upsudo ip addr add 12.12.12.4/24 dev eth1.202

When “trust” is off, only broadcast frames from VLAN tagged VNF interfaces are permitted to be forwarded. Enabling “trust” on the VF port puts it into promiscuous mode. In this mode, Unicast and Multicast frames with VLAN tags other than VF’s are allowed to be forwarded.

Validation:

Output from host node with “trust” enabled:

8: p7p3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000

   link/ether f8:f2:1e:03:c8:c4 brd ff:ff:ff:ff:ff:ff

   vf 4 MAC fa:16:3e:24:73:1c, vlan 30, spoof checking off, link-state enable, trust on

Now, lets ping from VNF1 to VNF 2 with 2 different VLANs:

[cloud-user@VNF1 ~]$ ping -c 1 11.11.11.9

PING 11.11.11.9 (11.11.11.9) 56(84) bytes of data.

64 bytes from 11.11.11.9: icmp_seq=1 ttl=64 time=0.071 ms

— 11.11.11.9 ping statistics —

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.071/0.071/0.071/0.000 ms

[cloud-user@VNF1 ~]$ ping -c 1 12.12.12.4

PING 12.12.12.4 (12.12.12.4) 56(84) bytes of data.

64 bytes from 12.12.12.4: icmp_seq=1 ttl=64 time=0.077 ms

— 12.12.12.4 ping statistics —

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.077/0.077/0.077/0.000 ms

Let’s capture ICMP packets on VNF2.

07:29:07.894983 fa:16:3e:c5:c7:e9 > fa:16:3e:3d:b6:5d, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 57952, offset 0, flags [DF], proto ICMP (1), length 84)

   11.11.11.5 > 11.11.11.9: ICMP echo request, id 26622, seq 1, length 64

07:29:07.895005 fa:16:3e:3d:b6:5d > fa:16:3e:c5:c7:e9, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 22048, offset 0, flags [none], proto ICMP (1), length 84)

   11.11.11.9 > 11.11.11.5: ICMP echo reply, id 26622, seq 1, length 64

07:31:24.757144 fa:16:3e:46:e8:07 > fa:16:3e:27:b3:25, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 64206, offset 0, flags [DF], proto ICMP (1), length 84)

   12.12.12.6 > 12.12.12.4: ICMP echo request, id 26629, seq 1, length 64

07:31:24.757165 fa:16:3e:27:b3:25 > fa:16:3e:46:e8:07, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 21260, offset 0, flags [none], proto ICMP (1), length 84)

   12.12.12.4 > 12.12.12.6: ICMP echo reply, id 26629, seq 1, length 64

Let’s see what the upstream ToR mac forwarding table looks like:

root> show ethernet-switching table vlan-id 30

MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static

          SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)

Ethernet switching table : 6 entries, 6 learned

Routing instance : default-switch

   Vlan                MAC MAC         Age Logical

   name                address flags              interface

   vlan30              fa:16:3e:05:ab:9a D             – xe-0/0/24.0

   vlan30              fa:16:3e:24:73:1c D             – xe-0/0/10.0

   vlan30              fa:16:3e:27:b3:25 D             – xe-0/0/24.0

   vlan30              fa:16:3e:3d:b6:5d D             – xe-0/0/24.0

   vlan30              fa:16:3e:46:e8:07 D             – xe-0/0/10.0

   vlan30              fa:16:3e:c5:c7:e9 D             – xe-0/0/10.0

As we can see,  all the mac addresses are learned on VLAN 30 which is VLAN tag of the SR-IOV VF on the host node.

Frames that egress from Compute1 are encapsulated with another VLAN header with VLAN tag 30 and forwarded to upstream port. Upstream switch processes outer VLAN header and sends it to Compute2. Outer VLAN header gets decapsulated on Compute2, leaving it with the inner VLAN tag (VLAN 201 or 202) and sent to the VNF interfaces where it is further decapsulated and consumed by the interface.

Upstreams ports are configured in trunk mode in order to preserve outer VLAN header.

Now, let’s consider second use case of extending VLANs defined inside the VNF to the upstream ToR.

In order to do that, host node VF should be configured in a “flat” mode.

TripleO Heat template can help defining provider network as a flat network. This may require redeploy of the stack.

Once VF is in “flat” mode and “trust” is enabled, we may need one more configuration inside both of the VNFs.

sudo ethtool -K eth1 rxvlan off

Disable rxvlan on VNF VF interface. This will disable VLAN stripping at VF and let tagged VLAN pass as it is. We need this as we want to extend VNF tagged VLAN to upstream ToR.

Let’s perform ping test again as was done earlier.

[cloud-user@testpmd ~]$ ping -c 1 11.11.11.9

PING 11.11.11.9 (11.11.11.9) 56(84) bytes of data.

64 bytes from 11.11.11.9: icmp_seq=1 ttl=64 time=0.090 ms

— 11.11.11.9 ping statistics —

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.090/0.090/0.090/0.000 ms

[cloud-user@testpmd ~]$ ping -c 1 12.12.12.4

PING 12.12.12.4 (12.12.12.4) 56(84) bytes of data.

64 bytes from 12.12.12.4: icmp_seq=1 ttl=64 time=0.095 ms

— 12.12.12.4 ping statistics —

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.095/0.095/0.095/0.000 ms

Let’s capture ICMP packets at VNF2.

12:51:56.888792 fa:16:3e:c5:c7:e9 > fa:16:3e:3d:b6:5d, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25714, offset 0, flags [DF], proto ICMP (1), length 84)

   11.11.11.5 > 11.11.11.9: ICMP echo request, id 22138, seq 1, length 64

12:51:56.888818 fa:16:3e:3d:b6:5d > fa:16:3e:c5:c7:e9, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 2531, offset 0, flags [none], proto ICMP (1), length 84)

   11.11.11.9 > 11.11.11.5: ICMP echo reply, id 22138, seq 1, length 64

12:51:59.425069 d0:07:ca:34:e9:1b > 01:80:c2:00:00:0e, ethertype LLDP (0x88cc), length 369: LLDP, length 355

12:53:01.155959 fa:16:3e:46:e8:07 > fa:16:3e:27:b3:25, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 28825, offset 0, flags [DF], proto ICMP (1), length 84)

   12.12.12.6 > 12.12.12.4: ICMP echo request, id 22139, seq 1, length 64

12:53:01.155983 fa:16:3e:27:b3:25 > fa:16:3e:46:e8:07, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 49609, offset 0, flags [none], proto ICMP (1), length 84)

   12.12.12.4 > 12.12.12.6: ICMP echo reply, id 22139, seq 1, length 64

Let’s see upstream mac learning for VLANs.

root> show ethernet-switching table vlan-id 30       

{master:0}

root> show ethernet-switching table vlan-id 201   

MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static

          SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)

Ethernet switching table : 2 entries, 2 learned

Routing instance : default-switch

   Vlan                MAC MAC         Age Logical

   name                address flags              interface

   vlan201             fa:16:3e:3d:b6:5d D             – xe-0/0/24.0

   vlan201             fa:16:3e:c5:c7:e9 D             – xe-0/0/10.0

{master:0}

root> show ethernet-switching table vlan-id 202    

MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static

          SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)

Ethernet switching table : 2 entries, 2 learned

Routing instance : default-switch

   Vlan                MAC MAC         Age Logical

   name                address flags              interface

   vlan202             fa:16:3e:27:b3:25 D             – xe-0/0/24.0

   vlan202             fa:16:3e:46:e8:07 D             – xe-0/0/10.0

{master:0}

root>

As we can see, VNF macs are learned on VLANs defined inside VNF.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s