Use case:
SR-IOV VF port binding to VNF provisions the higher level of throughput and latency
performance which is a generic demand for the mission critical infrastructure. Such ports can
work in “access”,flat”, or “promiscuous mode” in relation to the VNF interface and carry VLAN
tags accordingly.
VNFs with large number of ingress/egress ports (i.e. vFirewalls) can use these features to carry
VLAN tags as per customer design needs.
Red Hat OpenStack Platform 14 (Community release “Rocky”) implements a “Trusted VF”
feature which supports such uses cases. Using this feature customer can have:-
- VLAN tunneling (QinQ)
- Extend VLANs inside of VNF to Upstream ToR
- Port binding inside VNF using multiple VFs.
NIC & Drivers:
i40e: Intel(R) Ethernet Connection XL710 Network Driver – version 2.3.2-k
Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
Details:
Let’s take the first use case of VLAN tunneling (QinQ).
In this case, VLANs configured inside the VNF are tunneled by the VLAN configured on the SR-IOV VF of the host node. Mac address of the VNF interfaces of VNF are learned by the upstream switch with VLAN tag of the host node VF and hides VLANs of the VNFs.
By default, “trust” mode is off on host node for VF. This can be modified through Red Hat OpenStack Platform Director (OpenStack Triple-O) using Heat templates. The following is the corresponding Heat template parameter.
NovaPCIPassthrough:
– devname: “p7p3”
trusted: “true”
physical_network: “sriov-1”
The above parameter configures port “p7p3” as a “trusted” port.
Once OpenStack stack is deployed, we need to create Neutron network, subnet, and port for the VFs.
| openstack network create –internal –provider-network-type vlan –provider-physical-network sriov-1 –provider-segment 30 –disable-port-security VLAN_Tun |
| openstack subnet create –network VLAN_Tun –subnet-range 6.6.6.0/24 –no-dhcp VLAN_Tun_Sub |
Create SR-IOV VF port for two compute nodes:
| openstack port create –network VLAN_Tun –vnic-type direct VNF1VF –binding-profile trusted=true openstack port create –network VLAN_Tun –vnic-type direct VNF2VF –binding-profile trusted=true |
Let’s create instances with OpenStack ports.
| openstack server create –flavor F1 –image Image1 –nic net-id=<Management_Network_UUID> –nic port-id=<VNF1VF_PortID> –key-name test –availability-zone nova:compute-0.localdomain VNF1 openstack server create –flavor F1 –image Image1 –nic net-id=<Management_Network_UUID> –nic port-id=<VNF2VF_PortID –key-name test –availability-zone nova:compute-1.localdomain VNF2 |
Once both VNFs are up and running, we need to create subinterfaces and assign mac addresses to them. By default, VLAN subinterfaces will pick VF’s mac address.
Inside VNF1:
| sudo ip link add link eth1 name eth1.201 type vlan id 201sudo ip link set dev eth1.201 address fa:16:3e:c5:c7:e9 sudo ip link set eth1.201 upsudo ip addr add 11.11.11.5/24 dev eth1.201 sudo ip link add link eth1 name eth1.202 type vlan id 202sudo ip link set dev eth1.202 address fa:16:3e:46:e8:07sudo ip link set eth1.202 upsudo ip addr add 12.12.12.6/24 dev eth1.202 |
Inside VNF2:
| sudo ip link add link eth1 name eth1.201 type vlan id 201sudo ip link set dev eth1.201 address fa:16:3e:3d:b6:5dsudo ip link set eth1.201 upsudo ip addr add 11.11.11.9/24 dev eth1.201 sudo ip link add link eth1 name eth1.202 type vlan id 202sudo ip link set dev eth1.202 address fa:16:3e:27:b3:25sudo ip link set eth1.202 upsudo ip addr add 12.12.12.4/24 dev eth1.202 |
When “trust” is off, only broadcast frames from VLAN tagged VNF interfaces are permitted to be forwarded. Enabling “trust” on the VF port puts it into promiscuous mode. In this mode, Unicast and Multicast frames with VLAN tags other than VF’s are allowed to be forwarded.
Validation:
Output from host node with “trust” enabled:
8: p7p3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether f8:f2:1e:03:c8:c4 brd ff:ff:ff:ff:ff:ff
vf 4 MAC fa:16:3e:24:73:1c, vlan 30, spoof checking off, link-state enable, trust on
Now, lets ping from VNF1 to VNF 2 with 2 different VLANs:
[cloud-user@VNF1 ~]$ ping -c 1 11.11.11.9
PING 11.11.11.9 (11.11.11.9) 56(84) bytes of data.
64 bytes from 11.11.11.9: icmp_seq=1 ttl=64 time=0.071 ms
— 11.11.11.9 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.071/0.071/0.071/0.000 ms
[cloud-user@VNF1 ~]$ ping -c 1 12.12.12.4
PING 12.12.12.4 (12.12.12.4) 56(84) bytes of data.
64 bytes from 12.12.12.4: icmp_seq=1 ttl=64 time=0.077 ms
— 12.12.12.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.077/0.077/0.077/0.000 ms
Let’s capture ICMP packets on VNF2.
07:29:07.894983 fa:16:3e:c5:c7:e9 > fa:16:3e:3d:b6:5d, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 57952, offset 0, flags [DF], proto ICMP (1), length 84)
11.11.11.5 > 11.11.11.9: ICMP echo request, id 26622, seq 1, length 64
07:29:07.895005 fa:16:3e:3d:b6:5d > fa:16:3e:c5:c7:e9, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 22048, offset 0, flags [none], proto ICMP (1), length 84)
11.11.11.9 > 11.11.11.5: ICMP echo reply, id 26622, seq 1, length 64
07:31:24.757144 fa:16:3e:46:e8:07 > fa:16:3e:27:b3:25, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 64206, offset 0, flags [DF], proto ICMP (1), length 84)
12.12.12.6 > 12.12.12.4: ICMP echo request, id 26629, seq 1, length 64
07:31:24.757165 fa:16:3e:27:b3:25 > fa:16:3e:46:e8:07, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 21260, offset 0, flags [none], proto ICMP (1), length 84)
12.12.12.4 > 12.12.12.6: ICMP echo reply, id 26629, seq 1, length 64
Let’s see what the upstream ToR mac forwarding table looks like:
root> show ethernet-switching table vlan-id 30
MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static
SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
vlan30 fa:16:3e:05:ab:9a D – xe-0/0/24.0
vlan30 fa:16:3e:24:73:1c D – xe-0/0/10.0
vlan30 fa:16:3e:27:b3:25 D – xe-0/0/24.0
vlan30 fa:16:3e:3d:b6:5d D – xe-0/0/24.0
vlan30 fa:16:3e:46:e8:07 D – xe-0/0/10.0
vlan30 fa:16:3e:c5:c7:e9 D – xe-0/0/10.0
As we can see, all the mac addresses are learned on VLAN 30 which is VLAN tag of the SR-IOV VF on the host node.
Frames that egress from Compute1 are encapsulated with another VLAN header with VLAN tag 30 and forwarded to upstream port. Upstream switch processes outer VLAN header and sends it to Compute2. Outer VLAN header gets decapsulated on Compute2, leaving it with the inner VLAN tag (VLAN 201 or 202) and sent to the VNF interfaces where it is further decapsulated and consumed by the interface.
Upstreams ports are configured in trunk mode in order to preserve outer VLAN header.
Now, let’s consider second use case of extending VLANs defined inside the VNF to the upstream ToR.
In order to do that, host node VF should be configured in a “flat” mode.
TripleO Heat template can help defining provider network as a flat network. This may require redeploy of the stack.
Once VF is in “flat” mode and “trust” is enabled, we may need one more configuration inside both of the VNFs.
sudo ethtool -K eth1 rxvlan off
Disable rxvlan on VNF VF interface. This will disable VLAN stripping at VF and let tagged VLAN pass as it is. We need this as we want to extend VNF tagged VLAN to upstream ToR.
Let’s perform ping test again as was done earlier.
[cloud-user@testpmd ~]$ ping -c 1 11.11.11.9
PING 11.11.11.9 (11.11.11.9) 56(84) bytes of data.
64 bytes from 11.11.11.9: icmp_seq=1 ttl=64 time=0.090 ms
— 11.11.11.9 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.090/0.090/0.090/0.000 ms
[cloud-user@testpmd ~]$ ping -c 1 12.12.12.4
PING 12.12.12.4 (12.12.12.4) 56(84) bytes of data.
64 bytes from 12.12.12.4: icmp_seq=1 ttl=64 time=0.095 ms
— 12.12.12.4 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.095/0.095/0.095/0.000 ms
Let’s capture ICMP packets at VNF2.
12:51:56.888792 fa:16:3e:c5:c7:e9 > fa:16:3e:3d:b6:5d, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25714, offset 0, flags [DF], proto ICMP (1), length 84)
11.11.11.5 > 11.11.11.9: ICMP echo request, id 22138, seq 1, length 64
12:51:56.888818 fa:16:3e:3d:b6:5d > fa:16:3e:c5:c7:e9, ethertype 802.1Q (0x8100), length 102: vlan 201, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 2531, offset 0, flags [none], proto ICMP (1), length 84)
11.11.11.9 > 11.11.11.5: ICMP echo reply, id 22138, seq 1, length 64
12:51:59.425069 d0:07:ca:34:e9:1b > 01:80:c2:00:00:0e, ethertype LLDP (0x88cc), length 369: LLDP, length 355
12:53:01.155959 fa:16:3e:46:e8:07 > fa:16:3e:27:b3:25, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 28825, offset 0, flags [DF], proto ICMP (1), length 84)
12.12.12.6 > 12.12.12.4: ICMP echo request, id 22139, seq 1, length 64
12:53:01.155983 fa:16:3e:27:b3:25 > fa:16:3e:46:e8:07, ethertype 802.1Q (0x8100), length 102: vlan 202, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 49609, offset 0, flags [none], proto ICMP (1), length 84)
12.12.12.4 > 12.12.12.6: ICMP echo reply, id 22139, seq 1, length 64
Let’s see upstream mac learning for VLANs.
root> show ethernet-switching table vlan-id 30
{master:0}
root> show ethernet-switching table vlan-id 201
MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static
SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
vlan201 fa:16:3e:3d:b6:5d D – xe-0/0/24.0
vlan201 fa:16:3e:c5:c7:e9 D – xe-0/0/10.0
{master:0}
root> show ethernet-switching table vlan-id 202
MAC flags (S – static MAC, D – dynamic MAC, L – locally learned, P – Persistent static
SE – statistics enabled, NM – non configured MAC, R – remote PE MAC, O – ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical
name address flags interface
vlan202 fa:16:3e:27:b3:25 D – xe-0/0/24.0
vlan202 fa:16:3e:46:e8:07 D – xe-0/0/10.0
{master:0}
root>
As we can see, VNF macs are learned on VLANs defined inside VNF.
Welcome to Haresh's blogs!! 