FreeBSD hype

July 6, 2021

I was not an active user of BSD family of operating systems. So, when I decided to choose FreeBSD for my next VPS it was mainly based on a hype that the company X uses FreeBSD to take full advantage of networking. Of course, there were other reasons, given that the software I was going to run on it was developed on this system. But the hype was the final drop.

In a FreeBSD world it's probably too pretentious to say its a hype. It's a very specific system, not for masses. But it looks as always: it attracts people who actually have very little experience and very loud voice.

Anyway, never take hype into account. Never.

Let every successful networking company uses FreeBSD.
Let every end user has sex with Windows.</li>
Let every developer uses Git.
Let every keen beginner stay with Rust.

You should find your own way.

Of course, my bad experience is an experience too, and it's very valuable experience. If I didn't try it I would be unaware of such a problem. What I'm disappointed with, is simply that final drop.

You, my lonely visitor may ask: so many words so far, but what the problem is?

Sorry, for boring you, here is the problem. In most cases I setup OpenVPN to have another exit point just in case. OpenVPN requires NAT to route the traffic outside. This is the source of the problem.

So, I have set OpenVPN up on FreeBSD VPS. But when I tried to use this VPN I noticed it's speed is Kbit/s range. Actually, the VPS has 200Mbit/s connection. I tried iperf3 from different locations in both directions. The results were quite similar and fell in Kbit/s range. For example, Europe-US transfer (FreeBSD IP address is YYY.YYY.YYY.YYY):

Accepted connection from XXX.XXX.XXX.XXX, port 41244
[  5] local YYY.YYY.YYY.YYY port 5201 connected to XXX.XXX.XXX.XXX port 41246
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.02   sec  16.8 KBytes   136 Kbits/sec
[  5]   1.02-2.01   sec  16.8 KBytes   138 Kbits/sec
[  5]   2.01-3.01   sec  2.80 KBytes  23.0 Kbits/sec
[  5]   3.01-4.01   sec  8.41 KBytes  68.9 Kbits/sec
[  5]   4.01-5.01   sec  7.01 KBytes  58.0 Kbits/sec
[  5]   5.01-6.01   sec  4.21 KBytes  34.3 Kbits/sec
[  5]   6.01-7.01   sec  7.01 KBytes  57.6 Kbits/sec
[  5]   7.01-8.01   sec  14.0 KBytes   115 Kbits/sec
[  5]   8.01-9.02   sec  14.0 KBytes   114 Kbits/sec
[  5]   9.02-10.00  sec  14.0 KBytes   117 Kbits/sec
[  5]  10.00-10.16  sec  2.80 KBytes   143 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.16  sec   108 KBytes  87.1 Kbits/sec                  receiver

You may argue that US-Europe is too long connection, well, here is another EU-US example, Linux hosts:

Accepted connection from XXX.XXX.XXX.XXX, port 1032
[  5] local ZZZ.ZZZ.ZZZ.ZZZ port 5201 connected to XXX.XXX.XXX.XXX port 54688
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  16.4 KBytes   134 Kbits/sec
[  5]   1.00-2.00   sec  82.0 KBytes   672 Kbits/sec
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec
[  5]   3.00-4.00   sec  10.4 MBytes  86.9 Mbits/sec
[  5]   4.00-5.00   sec  11.2 MBytes  93.9 Mbits/sec
[  5]   5.00-6.00   sec  11.2 MBytes  94.0 Mbits/sec
[  5]   6.00-7.00   sec  11.2 MBytes  93.7 Mbits/sec
[  5]   7.00-8.01   sec  11.2 MBytes  93.3 Mbits/sec
[  5]   8.01-9.00   sec  11.1 MBytes  93.5 Mbits/sec
[  5]   9.00-10.00  sec  11.2 MBytes  93.7 Mbits/sec
[  5]  10.00-10.21  sec  2.31 MBytes  93.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.21  sec  82.2 MBytes  67.5 Mbits/sec                  receiver

And, FreeBSD-Linux both in Europe:

Accepted connection from ZZZ.ZZZ.ZZZ.ZZZ, port 58869
[  5] local YYY.YYY.YYY.YYY port 5201 connected to ZZZ.ZZZ.ZZZ.ZZZ port 55689
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  41.0 KBytes   336 Kbits/sec
[  5]   1.00-2.00   sec  73.8 KBytes   605 Kbits/sec
[  5]   2.00-3.00   sec   131 KBytes  1.08 Mbits/sec
[  5]   3.00-4.00   sec  95.7 KBytes   784 Kbits/sec
[  5]   4.00-5.00   sec  94.3 KBytes   773 Kbits/sec
[  5]   5.00-6.00   sec  71.1 KBytes   581 Kbits/sec
[  5]   6.00-7.00   sec  91.6 KBytes   752 Kbits/sec
[  5]   7.00-8.00   sec   101 KBytes   829 Kbits/sec
[  5]   8.00-9.00   sec  93.0 KBytes   761 Kbits/sec
[  5]   9.00-10.00  sec   120 KBytes   987 Kbits/sec
[  5]  10.00-10.09  sec  8.20 KBytes   783 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.09  sec   921 KBytes   748 Kbits/sec                  receiver

I wrote a letter to the hosting provider with such examples, they told me the server my VPS is hosted on looks too heavily loaded, they would try to move my VPS to other server. This did not help.

Meantime I started to think if I could miss something. I never had any problems with Linux network stack, so I was confident if I set anything up by official manual, it should have to work like a charm.

The official FreeBSD Handbook mentions two implementations of NAT: userland and in-kernel. "Oh, dear, you're a numskull, you haven't tried in-kernel NAT before writing to the support" - that was my thought when I re-read this manual. I was probably too busy with OpenVPN issues to keep all networking stuff in mind and copy-pasted rc.conf lines from OpenVPN installation manual without due thinking.

Okay, I have tried in-kernel NAT. Was it helpful? - actually, no. Only inbound speed has increased to 200Mbit/s. Outbound speed stayed in Kbit/s range.

What's the conclusion?

I have already written the conclusion in the beginning.