Using LXC

While the mainstream tendency was migrating to clouds, at my past work we continued using bare metal servers. So I dug into virtualization quite lazily, occasionally using VirtualBox and giving Qubes OS a try (yuck!). However, during the recent overhaul of my home network I finally moved to Linux containers. This blog post summarizes my experience, focusing on Armbian.

References

• https://wiki.debian.org/LXC
• https://stgraber.org/2013/12/20/lxc-1-0-blog-post-series/
• https://blog.benoitblanchon.fr/lxc-unprivileged-container/

For setting up Armbian refer to my blog post From Sid down to Bullseye: installing Armbian on NanoPi M4 v2.

Installing LXC

apt install lxc lxc-templates libvirt0 libpam-cgfs bridge-utils uidmap debootstrap distro-info

I don't use default bridge, I prefer to configure it manually in /etc/network/interfaces. So I don't need lxc-net service:

systemctl stop lxc-net
systemctl disable lxc-net

Make change to /etc/default/lxc-net:

USE_LXC_BRIDGE="false"

If you prefer to setup a bridge interface manually, run the following commands:

brctl addbr br0
brctl addif br0 eth0
ip link set br0 up

Given that the bridge interface is br0, /etc/lxc/default.conf should contain

lxc.net.0.link = br0

Change apparmor profile, the reason is running unprivileged containers:

lxc.apparmor.profile = unconfined

For privileged containers I used `unconfigured`. Actually, I don't know what's the difference.

lxc.apparmor.profile = unconfigured

Creating LXC container

For the first time I played with privileged containers but soon started using unprivileged ones. Unprivileged containers have some limitations, for example, it's impossible to run nfs-kernel-server or mount a block device, even if you give access to that device and even if you are able to read the data from it. Mounting file systems is allowed for root only.

Let's create container `test`:

lxc-create -n test -t debian -- -r bullseye

Before starting the container a few tweaks are necessary.

If the container is unprivileged, add the following to the configuration file /var/lib/lxc/test/config:

# Map user and group ids
lxc.include = /usr/share/lxc/config/debian.userns.conf
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536

In addition to the above change, make sure both /etc/subuid and /etc/subgid on the host system contain

root:100000:65536

Then, configure networking in /var/lib/lxc/test/rootfs/etc/network/interfaces.

Also, I configure SSH server. First, make the following changes to /var/lib/lxc/test/rootfs/etc/ssh/sshd_config:

PermitRootLogin yes
PubkeyAuthentication yes
PasswordAuthentication no

And then,

mkdir /var/lib/lxc/test/rootfs/root/.ssh
chmod 700 /var/lib/lxc/test/rootfs/root/.ssh
cp -a .ssh/authorized_keys /var/lib/lxc/test/rootfs/root/.ssh/

i.e. I use the same public keys from the host system. You can use different ones.

For unprivileged containers change the owner of rootfs:

chown -R 100000:100000 /var/lib/lxc/test/rootfs

By default dmesg output is available from unprivileged container. Here's a security tweak to /etc/sysctl.conf:

kernel.dmesg_restrict = 1

Running LXC container

If the new container is succesfully started with

lxc-start test

you can either login via SSH or attach from the host system: lxc-attach test.

If something went wrong, you can start the container in foreground:

lxc-start -F -n test

however, in all my cases the output was useless and I had to play with configuration file.

So, if everything went okay and you're in the container, a minimal usable system is necessary:

apt install less nano psutils rsync apt-utils iputils-tracepath inetutils-ping \
            dnsutils nftables curl gnupg2 ca-certificates lsb-release debian-archive-keyring
apt install --no-install-recommends rsyslog logrotate cron

The second command needs --no-install-recommends because all those packages want Exim.

Using block devices and mounting file systems

Actiually, I did not try that from a privileged container. This section contains instructions for unprivileged containers only.

To use a block device in a container, change group of the block device to 100000 The owner may remain root. To automate this, create /etc/udev/rules.d/90-sda-permissions.rules with the following line (assuming the device is sda):

KERNEL=="sda", ACTION=="add", GROUP="100000"

Next, allow use of block device in the container. Add the following lines to /var/lib/lxc/test/config:

lxc.cgroup.devices.allow = b 8:0 rwm
lxc.mount.entry = /dev/sda dev/sda none bind,create=file

Open question is how to make this by UUID? Namely, sda name and 8:0 which are major:minor numbers may change, UUID is stable.

As I already told, you can read-write block device from a container, but you can't mount a file system. For filesystems, they should be mounted on the host system and then you can use bind mounts in the configuration file:

lxc.mount.entry = /mnt/filestore mnt/filestore none bind 0 0

VPN

I tried Wireguard as a client in unprivileged containers. It works without any tweaks. Installation wants too much and needs a command line option:

apt install --no-install-recommends wireguard wireguard-tools

I tried OpenVPN client from a privileged container. It needs the following lines to the configuration file:

lxc.hook.autodev = /var/lib/lxc/test/autodev
lxc.cgroup2.devices.allow = c 10:200 rwm

where autodev hook contains the following:

#!/bin/bash

pushd ${LXC_ROOTFS_MOUNT}/dev

mkdir net
mknod net/tun c 10 200
chmod 666 net/tun

popd

That's all. Bye for now.