FreeBSD hype
I was not an active user of BSD family of operating systems. So, when I decided to choose FreeBSD for my next VPS it was mainly based on a hype that the company X uses FreeBSD to take full advantage of networking. Of course, there were other reasons, given that the software I was going to run on it was developed on this system. But the hype was the final drop.
In a FreeBSD world it's probably too pretentious to say its a hype. It's a very specific system, not for masses. But it looks as always: it attracts people who actually have very little experience and very loud voice.
Anyway, never take hype into account. Never.
- Let every successful networking company uses FreeBSD.
- Let every end user has sex with Windows.</li>
- Let every developer uses Git.
- Let every keen beginner stay with Rust.
You should find your own way.
Of course, my bad experience is an experience too, and it's very valuable experience. If I didn't try it I would be unaware of such a problem. What I'm disappointed with, is simply that final drop.
You, my lonely visitor may ask: so many words so far, but what the problem is?
Sorry, for boring you, here is the problem. In most cases I setup OpenVPN to have another exit point just in case. OpenVPN requires NAT to route the traffic outside. This is the source of the problem.
So, I have set OpenVPN up on FreeBSD VPS. But when I tried to use this VPN I noticed it's speed is Kbit/s range. Actually, the VPS has 200Mbit/s connection. I tried iperf3 from different locations in both directions. The results were quite similar and fell in Kbit/s range. For example, Europe-US transfer (FreeBSD IP address is YYY.YYY.YYY.YYY):
Accepted connection from XXX.XXX.XXX.XXX, port 41244 [ 5] local YYY.YYY.YYY.YYY port 5201 connected to XXX.XXX.XXX.XXX port 41246 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.02 sec 16.8 KBytes 136 Kbits/sec [ 5] 1.02-2.01 sec 16.8 KBytes 138 Kbits/sec [ 5] 2.01-3.01 sec 2.80 KBytes 23.0 Kbits/sec [ 5] 3.01-4.01 sec 8.41 KBytes 68.9 Kbits/sec [ 5] 4.01-5.01 sec 7.01 KBytes 58.0 Kbits/sec [ 5] 5.01-6.01 sec 4.21 KBytes 34.3 Kbits/sec [ 5] 6.01-7.01 sec 7.01 KBytes 57.6 Kbits/sec [ 5] 7.01-8.01 sec 14.0 KBytes 115 Kbits/sec [ 5] 8.01-9.02 sec 14.0 KBytes 114 Kbits/sec [ 5] 9.02-10.00 sec 14.0 KBytes 117 Kbits/sec [ 5] 10.00-10.16 sec 2.80 KBytes 143 Kbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.16 sec 108 KBytes 87.1 Kbits/sec receiver
You may argue that US-Europe is too long connection, well, here is another EU-US example, Linux hosts:
Accepted connection from XXX.XXX.XXX.XXX, port 1032 [ 5] local ZZZ.ZZZ.ZZZ.ZZZ port 5201 connected to XXX.XXX.XXX.XXX port 54688 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 16.4 KBytes 134 Kbits/sec [ 5] 1.00-2.00 sec 82.0 KBytes 672 Kbits/sec [ 5] 2.00-3.00 sec 2.38 MBytes 20.0 Mbits/sec [ 5] 3.00-4.00 sec 10.4 MBytes 86.9 Mbits/sec [ 5] 4.00-5.00 sec 11.2 MBytes 93.9 Mbits/sec [ 5] 5.00-6.00 sec 11.2 MBytes 94.0 Mbits/sec [ 5] 6.00-7.00 sec 11.2 MBytes 93.7 Mbits/sec [ 5] 7.00-8.01 sec 11.2 MBytes 93.3 Mbits/sec [ 5] 8.01-9.00 sec 11.1 MBytes 93.5 Mbits/sec [ 5] 9.00-10.00 sec 11.2 MBytes 93.7 Mbits/sec [ 5] 10.00-10.21 sec 2.31 MBytes 93.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.21 sec 82.2 MBytes 67.5 Mbits/sec receiver
And, FreeBSD-Linux both in Europe:
Accepted connection from ZZZ.ZZZ.ZZZ.ZZZ, port 58869 [ 5] local YYY.YYY.YYY.YYY port 5201 connected to ZZZ.ZZZ.ZZZ.ZZZ port 55689 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 41.0 KBytes 336 Kbits/sec [ 5] 1.00-2.00 sec 73.8 KBytes 605 Kbits/sec [ 5] 2.00-3.00 sec 131 KBytes 1.08 Mbits/sec [ 5] 3.00-4.00 sec 95.7 KBytes 784 Kbits/sec [ 5] 4.00-5.00 sec 94.3 KBytes 773 Kbits/sec [ 5] 5.00-6.00 sec 71.1 KBytes 581 Kbits/sec [ 5] 6.00-7.00 sec 91.6 KBytes 752 Kbits/sec [ 5] 7.00-8.00 sec 101 KBytes 829 Kbits/sec [ 5] 8.00-9.00 sec 93.0 KBytes 761 Kbits/sec [ 5] 9.00-10.00 sec 120 KBytes 987 Kbits/sec [ 5] 10.00-10.09 sec 8.20 KBytes 783 Kbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.09 sec 921 KBytes 748 Kbits/sec receiver
I wrote a letter to the hosting provider with such examples, they told me the server my VPS is hosted on looks too heavily loaded, they would try to move my VPS to other server. This did not help.
Meantime I started to think if I could miss something. I never had any problems with Linux network stack, so I was confident if I set anything up by official manual, it should have to work like a charm.
The official FreeBSD Handbook mentions two implementations of NAT: userland and in-kernel. "Oh, dear, you're a numskull, you haven't tried in-kernel NAT before writing to the support" - that was my thought when I re-read this manual. I was probably too busy with OpenVPN issues to keep all networking stuff in mind and copy-pasted rc.conf lines from OpenVPN installation manual without due thinking.
Okay, I have tried in-kernel NAT. Was it helpful? - actually, no. Only inbound speed has increased to 200Mbit/s. Outbound speed stayed in Kbit/s range.
What's the conclusion?
I have already written the conclusion in the beginning.