Lets understand OpenVPN !

VPN is not a new concept and almost everyone working with IT or ITES sectors are aware of it. However, Today’s pandemic situation have taken VPN concepts beyond IT folks. For example, People working in Financial institutions, Bank employees who are working from home, use it today to connect to their company’s IT resources over secure VPN channel.

OpenVPN is one such opensource VPN solution which i am going to discuss today. OpenVPN has many use cases/deployments though I would focus on its VPN client part and how does it work in one’s laptop.I won’t speak on installation part as there are already many articles present on internet. The Linux distro I used here is rhel 7.9 and OpenVPN version is 2.4.11.

I will divide this OpenVPN blog in 2 parts. 1st is about establishing VPN connection and 2nd is about Packet forwarding over VPN networks.

Part 1 – Establishing the VPN connection

Basically, to connect to server, we need few specified parameters from the client side which would be used to establish the connection. OpenVPN package provides a simple client config file. Most of them are self explanatory and I am highlighting few of them here.

# Specify that we are a client and that we will be pulling certain config file directives from the server.
client
# Use the same setting as you are using on the server. On most systems, the VPN will not function unless you partially or fully disable the firewall for the TUN/TAP interface.
;dev tap
dev tun
# Are we connecting to a TCP or UDP server?  Use the same setting as on the server.
;proto tcp
proto udp
# The hostname/IP and port of the server. You can have multiple remote entries to load balance between the servers.
remote my-server-1 1194
;remote my-server-2 1194
# Downgrade privileges after initialization (non-Windows only)
;user nobody
;group nobody
# SSL/TLS parms. See the server config file for more description.  It's best to use a separate .crt/.key file pair for each client.  A single ca file can be used for all clients.
ca ca.crt
cert client.crt
key client.key
# Select a cryptographic cipher. If the cipher option is used on the server then you must also specify it here. Note that v2.4 client/server will automatically negotiate AES-256-GCM in TLS mode. See also the data-ciphers option in the manpage
cipher AES-256-CBC

Provide appropriate values in this config file as per your server configuration and supply to Openvpn daemon on the system to start connection process. I will explain few parameters with details later in the blog.

There is another approach using D-bus. D-Bus is a Message-oriented middleware mechanism that allows communication between multiple processes running concurrently on the same machine. An API through D-Bus which allows applications to query and control network configuration and state. In this way, applications can check or configure networking through D-BUS.

Below is the capture from the machine which has OpenVPN session established.

root 25779 0.0 0.0 333388 4344 ? Sl 14:07 0:00 /usr/libexec/nm-openvpn-service --bus-name org.freedesktop.NetworkManager.openvpn.Connection_82

nm-open+ 25850 0.1 0.0 79480 4776 ? S 14:07 0:46 /usr/sbin/openvpn --remote ovpn-haresh.haresh.com 443 udp --nobind --dev tun --cipher AES-256-CBC --auth-nocache --verify-x509-name ovpn.haresh.com name --reneg-sec 0 --verb 1 --syslog nm-openvpn --tun-mtu 1360 --script-security 2 --up /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 25779 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_82 --tun -- --up-restart --persist-key --persist-tun --management /var/run/NetworkManager/nm-openvpn-f281b867-85e1-4979-8adf-ad9fac216a7c unix --management-client-user root --management-client-group root --management-query-passwords --auth-retry interact --route-noexec --ifconfig-noexec --client --auth-user-pass --ca /etc/pki/tls/certs/2015-RH-IT-Root-CA.pem --user nm-openvpn --group nm-openvpn

Here, we have a bus which is used by nm-openvpn and openvpn processes. NetworkManager-openvpn-gnome is GNOME version of nm(NetworkManager) for openvpn. Once installed, You could see dialogue box as shown below. This basically invokes OpenVPN daemon with supplied parameters and values.

Once, VPN server is added and connection initiated(depend on security mechanism, user would need to authenticate self e.g. RSA token based), you would see D-bus used by nm-openvpn to supply these values to openvpn daemon to negotiate and establish the connection. Monitoring D-bus system is possible but out of scope for this blog.

You can import the config file (Provided by Server admin) as well instead going through dialogue box.

You have probably observed the users running nm-openvpn and openvpn processes are different. user “nm-openvpn” and “openvpn” are created while installing these packages. Lets look at them.

openvpn:x:994:992:OpenVPN:/etc/openvpn:/sbin/nologin
nm-openvpn:x:992:987:Default user for running openvpn spawned by NetworkManager:/:/sbin/nologin

Let me explain few parameters. “–nobind” specifies that the connection needed dhcp ips rather static one. Parameter “–route-noexec” specifies that dont install any routes explicitly rather pass to system via script. “–ifconfig-noexec” tells that dont run ifconfig commands but pass scripts to the system. Other parameters like Key,Certificates are straightforward but important to establish the connection.

One specific parameter here is interesting of all. That’s it “tun”. If you have observed in sample config file, we specified that we need TUN type device. This interface is Hero of OpenVPN client and need to understand in details.

First lets get some details about “tun”.

[haresh@localhost network-scripts]$ sudo ethtool tun0
Settings for tun0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
MDI-X: Unknown
Current message level: 0xffffffa1 (-95)
drv ifup tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000
Link detected: yes

[haresh@localhost network-scripts]$ sudo ethtool -i tun0
driver: tun
version: 1.6
firmware-version:
expansion-rom-version:
bus-info: tun
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

[haresh@localhost network-scripts]$ ifconfig tun0
tun0: flags=4305 mtu 1360
inet 10.10.112.228 netmask 255.255.240.0 destination 10.10.112.228
inet6 fe80::68d3:fcd0:2db3:edd prefixlen 64 scopeid 0x20
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 100 (UNSPEC)
RX packets 7068 bytes 2212579 (2.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7518 bytes 715893 (699.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[haresh@localhost network-scripts]$ ip link show tun0
26: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 100
    link/none 

This “tun” driver is kernel driver module and available in all linux distro. You might have noticed by now that, Why tun0 interface has speed on 10 Mbps, No Mac address, What is POINTTOPOINT and NOARP? Moreover, Who created tun0?

Lets look at packet captured on tun0 interface when we try to ping server sitting other side of VPN.

Why Ethernet header is missing? And how would this packet be traversed over internet? BTW, ping was successful. To understand this, Again we need to go establishment process. See the log messages.

nm-openvpn[13696]: [ovpn.haresh.com] Peer Connection Initiated with [AF_INET]66.187.232.64:443
nm-openvpn[13696]: TUN/TAP device tun0 opened
nm-openvpn[13696]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 13674 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_8 --tun -- tun0 1360 1412 10.10.112.22 255.255.240.0 init
NetworkManager[1030]:   [1614535444.1003] manager: (tun0): new Tun device (/org/freedesktop/NetworkManager/Devices/9)

As we can see, tun0 interface created at time of establishing the connection by openvpn. tun0 receives the ip address, netmask, routes, gateway information etc. from the server. MTU (1360) was also a property of tun0.

Still our questions on unanswered in regard to tun0. Lets focus again. We now know that tun0 is has some role to play in sending/receiving traffic. But how?

So, tun0 is a char device. Amid unanswered questions, one more, why char device?

“Character devices can be created by kernel modules (“tun” in our case). When a device is created, the creator provides pointers to functions that implement handle standard calls like open, read, etc. They come in two flavors: Character Devices and Block Devices.Block Devices tend to be storage devices, capable of buffering output and storing data for later retrieval. Character Devices are things like audio or graphics cards, or input devices like keyboard and mouse.”

When tun module loads, it will register itself for char device under /net. But wont create tun0 (A char device) till we initiate a connection.

[haresh@localhost ~]$ ls -la /dev/char/ | grep tun
lrwxrwxrwx. 1 root root 10 Apr 16 13:02 10:200 -> ../net/tun

So, tun0 is a char device file from which openvpn (user space application) reads the packets or writes the packets using IOCTL calls. tun driver represent tun0 to kernel as virtual NIC.

Alright, So, answer to some questions, Tun0 interface doesn’t need Mac address because, its data read and write limited to single system. And consider “Raw” with just IPs (I will explain in 2nd part for IPs). Same being the reason Why we see NOARP on tun0 interface. Tun0 IP never need to broadcast itself on the network. All communication between machines of same VPN network has to traverse through VPN server (Work as proxy to all). Same being the reason for tun0 being POINTTOPOINT. Regarding speed, Take a look at these code lines from tun driver.

https://github.com/torvalds/linux/blob/master/drivers/net/tun.c#L3456

static void tun_default_link_ksettings(struct net_device *dev,
				       struct ethtool_link_ksettings *cmd)
{
	ethtool_link_ksettings_zero_link_mode(cmd, supported);
	ethtool_link_ksettings_zero_link_mode(cmd, advertising);
	cmd->base.speed		= SPEED_10;
	cmd->base.duplex	= DUPLEX_FULL;
	cmd->base.port		= PORT_TP;
	cmd->base.phy_address	= 0;
	cmd->base.autoneg	= AUTONEG_DISABLE;
}

One question, All this is fine but still inside the machine, how about its routing over internet? Ultimately, Packets need to traverse internet. Answer of this question is in our Part 2 – Packet forwarding over VPN networks.

Part 2 – Packet forwarding over VPN networks

Lets take a logical view of packet forwarding. So, When we ping to the IP of the machine resides other side of VPN, something like this happens,

• Ping generates the packet, at IP layer, Source ip – tun0 ip, Destination ip – remote machine ip
• networking stack give it to tun0, a tun interface
• it is read by openvpn (User space application)
• Packet gets encrypted with security keys (Based on TLS negotiations)
• openvpn registers over udp port in the stack, at Layer 4
• Reformulate the packet with destination ip of VPN server (ovpn.haresh.com’s ip)and sent back to networking stack, Back to IP layer
• Service provider’s ip gives on host machine would be used as source ip
• packet sent to internet
• Same process in reverse order happens when packet from internet received by host machine.

Lets see this with some evidence from the host machine.

We again need to go back to establishing process and find something.

NetworkManager[1030]:   [1614535444.1191] vpn-connection[0x55929810a530,f281b867-85e1-4979-8adf-ad9fac216a7c,"Haresh (DC2)",8:(tun0)]: Data:   Internal DNS: 10.11.5.19
NetworkManager[1030]:   [1614535444.1191] vpn-connection[0x55929810a530,f281b867-85e1-4979-8adf-ad9fac216a7c,"Haresh (DC2)",8:(tun0)]: Data:   Internal DNS: 10.5.30.160
NetworkManager[1030]:   [1614535444.1191] vpn-connection[0x55929810a530,f281b867-85e1-4979-8adf-ad9fac216a7c,"Haresh (DC2)",8:(tun0)]: Data:   DNS Domain: 'haresh.com'

dnsmasq[2254]: reading /etc/resolv.conf
dnsmasq[2254]: using nameserver 10.11.5.19#53
dnsmasq[2254]: using nameserver 10.5.30.160#53
dnsmasq[2254]: using nameserver 192.168.1.254#53

As we can see, while establishing the connection, tun0 received few DNS server IPs as well. What were they for?

[haresh@localhost ~]$ nslookup help
Server: 10.11.5.19
Address: 10.11.5.19#53

This is intended to identify VPN vs non VPN traffic to control which traffic would pass via VPN network (e.g tun0) and service provider network (IPs given by service provider network when connecting to internet). Remember, in VPN config dialogue box at the beginning, we have a small check box stating “Use this connection only for resource on its networks”. If you ticked that, then this traffic segregation takes place.

So, now when i ping server1.haresh.com or http://www.google.com, a DNS query packet would be sent over tun0 to DNS server resides behind VPN server and DNS server would answer back to DNS query with IP details of domain name. Lets look at routing table now.

[haresh@localhost network-scripts]$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.254 0.0.0.0 UG 0 0 0 wlp58s0
10.0.0.0 10.10.112.1 255.0.0.0 UG 0 0 0 tun0 <<<<<<<<This is the reason
10.10.112.0 0.0.0.0 255.255.240.0 U 0 0 0 tun0
66.187.232.64 192.168.1.254 255.255.255.255 UGH 0 0 0 wlp58s0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 wlp58s0
192.168.1.254 0.0.0.0 255.255.255.255 UH 0 0 0 wlp58s0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

As you can see DNS server IP (10.11.5.19) would be routed via tun0. as all 10.X networks are reachable via tun0 as per routing table. So, now if DNS server resolves the domain name with IP other than 10.X like http://www.google.com, it would be traversed via wlp58s0 (My laptop’s wireless interface) and if domain name has IP with 10.X, they all would be sent using tun0.

Now, lets looks at the packet when it goes out via wlp58s0 to internet with destination of VPN server.

You can see UDP destination port 443. And IPs are host machine’s service provider provided as src ip and VPN server’s ip as destination. Entire Ping packet now is part of Data (Encrypted as well).

UDP src port is the port which openvpn application registered with networking stack.

[hakhande@hakhande ~]$ sudo netstat -tunlp | grep  vpn
[sudo] password for hakhande: 
udp        0      0 0.0.0.0:48424           0.0.0.0:*                           21053/openvpn       

So, in reverse order, when packet hits back laptop’s interface from VPN server, packets handed over to openvpn application, which in turn de-crypt the packet and write to tun0 interface using IOCTLs. Kernel networking stack resolves tun0 ip and hand it over the packet back to the user space application which used tun0 (Ping in our case).

OpenVPN has its own protocol header. So, when OpenVPN sends/receives the packet, it does apply its header. OpenVPN uses TLS mechanism to securely transmit data. Lets look at packet at NIC.

Now, whatever configuration in networking stack happened due to VPN connection, would be removed back when we disconnect the VPN connection. Lets look at disconnection logs.

NetworkManager[1030]:   [1614535344.7644] audit: op="connection-deactivate" uuid="f281b867-85e1-4979-8adf-ad9fac216a7c" name="Haresh (DC2)" pid=3216 uid=112847 result="success"
dnsmasq[2254]: reading /etc/resolv.conf
dnsmasq[2254]: using nameserver 192.168.1.254#53  <<<<RESTORED
NetworkManager[1030]:   [1614535344.7955] device (tun0): state change: activated -&gt; unmanaged (reason 'connection-assumed', sys-iface-state: 'external')
dbus[946]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
nm-openvpn[5540]: SIGTERM received, sending exit notification to peer
systemd: Starting Network Manager Script Dispatcher Service...
NetworkManager[1030]:   [1614535344.8168] vpn-connection[0x55929810a110,f281b867-85e1-4979-8adf-ad9fac216a7c,"Haresh (DC2)",0]: VPN plugin: state changed: stopping (5)
NetworkManager[1030]:   [1614535344.8169] vpn-connection[0x55929810a110,f281b867-85e1-4979-8adf-ad9fac216a7c,"Haresh (DC2)",0]: VPN plugin: state changed: stopped (6)
dbus[946]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
systemd: Started Network Manager Script Dispatcher Service.
nm-dispatcher: req:1 'vpn-down' [tun0]: new request (5 scripts)
nm-dispatcher: req:1 'vpn-down' [tun0]: start running ordered scripts...
nm-dispatcher: req:2 'down' [tun0]: new request (5 scripts)
nm-dispatcher: req:2 'down' [tun0]: start running ordered scripts...
journal: /org/gnome/OnlineAccounts/Accounts/account_1614530467_0: Setting AttentionNeeded to FALSE because EnsureCredentials() succeded
ntpd[947]: Deleting interface #8 tun0, fe80::68d3:fcd0:2db3:edd#123, interface stats: received=0, sent=0, dropped=0, active_time=4669 secs
ntpd[947]: Deleting interface #7 tun0, 10.10.112.228#123, interface stats: received=0, sent=0, dropped=0, active_time=4669 secs
avahi-daemon[925]: Withdrawing workstation service for tun0.
journal: Removing a network device that was not added

I hope, You enjoyed reading this blog and got basics of OpenVPN.