sourcediver.org

about software and freediving

Using Legacy LXC Containers With NixOS as Host

I moved my server from a hacky Debian install to a hopefully maintain- and scalable NixOS installation. However I have most of my services in LXC containers to create at least some seperation and smart network management.

You might ask why I don’t use Docker, here’s why: Docker is a rather nice solution but they accelerated way too fast. Too many people jumped on the band wagon (Microsoft, lol) and the flawed architecture (RUN apt-get install) became the new standard on how to deploy machines to production. Without the huge funding from the dark side of ycombinator, Docker would never have been as “successful” as it is today. It would have been an alternative to LXC with less than 1000 forks on GitHub and something far better could have emerged more quickly. If something groundbreaking surfaces now, the whole world needs to uninstall Docker and convert all machines to the new solution. Software should convince with features not with marketing. </rant>

NixOS seems like a good idea to overcome most of the issues I have with LXC, Docker and Debian. This article will tell you how to start your LXC containers on NixOS.

While the lxc package is already present, containers do not run out of the box since a few folders are missing and need to be created by hand. This article will soon be out of date once I overcome my everlasting procastination phase and I will be able to create a PR that reflects everything I write in this article. Wait? Why am I writing this article again instead of creating the PR right away…?

Since I also use veth adapters, I will show you some parts of my configuration.nix that are responsible for creating the bridge, iptables entries which looks something like this.

configuration.nix (configuration.nix) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
{ config, pkgs, ... }:

{
  # YOUR BOOT STUFF HERE
  networking.hostName = "nixos-lxc-demo";

  networking.bridges = {
    lxc0 = {
      interfaces = [];
    };
  };

  networking.interfaces = {
    enp0s3 = {
     ipAddress = "192.168.23.23";
     prefixLength = 24;
    };
    lxc0 = {
     ipAddress = "10.0.42.1";
     prefixLength = 24;
    };
  };

  networking.nameservers = ["8.8.8.8"];
  networking.defaultGateway = "192.168.23.1";

  # time and locale
  time.timeZone = "Europe/Berlin";
   i18n = {
     consoleFont = "Lat2-Terminus16";
     consoleKeyMap = "us";
     defaultLocale = "en_US.UTF-8";
   };

  # Firewall
  networking.firewall.enable = true;
  networking.firewall.allowPing = true;
  networking.nat = {
    enable = true;
    externalInterface = "enp0s3";
    externalIP = "192.168.23.1";
    internalIPs = ["10.0.42.0/24"];
    internalInterfaces = ["lxc0"];
    forwardPorts = [
      # proxy
      { sourcePort = 80; destination = "10.0.42.42"; }
    ];
  };

  virtualisation.lxc = {
    enable = true;
    defaultConfig = "lxc.network.type = empty";
  };
}

You will need to create a few directories after running nixos-rebuild:

1
2
sudo mkdir -p /var/cache/lxc
sudo mkdir -p /var/lib/lxc/rootfs

Once this is done you can migrate or create your containers. If you have legacy containers from e.g. a debian installation you need to alter your LXC config a bit. Most of these LXC containers include a config like flavor.common.conf which cannot be resolved since it lives somewhere in the nix-store.

So you should create a symlink to the file in the nix-store or copy the file to /var/lib/lxc/flavor.common.conf Then you want to create a file that you can name something like /var/lib/lxc/nixos_flavour.custom.conf. In this case it is /var/lib/lxc/nixos_debian.custom.conf.

1
2
lxc.include = /var/lib/lxc/debian.common.conf
lxc.aa_profile = unconfined

In your container config you can simply use this line to include the flavor.common.conf indirectly while adding the AppArmor option that is currently necessary when using rather old LXC containers. So for the example above you want to add this line:

1
lxc.include = /var/lib/lxc/nixos_debian.custom.conf

This might not be the ideal solution but currently works for me. In the end you want to put every option into one single config which will not change when the hash of lxc changes in the nix-store.

Now to systemd, our beloved overlord. Starting with version 230, it behaves like a good init daemon should behave:

It kills all processes created within a user session that are not being killed by the user when the session is destroyed. So if you start your containers using lxc-start, systemd will kill them once you log off or disconnect from your server. meh. (this also breaks tmux, screen, most vnc servers)

While I see the point of this, I only need the LXC containers for some weeks until every service is converted to a .nix expression - so writing a systemd unit for it is not really worth it.

Therefore I put this into my configuration.nix:

1
2
3
  services.logind.extraConfig =
    "KillUserProcesses=no"
  ;

2016/12/07: Updated configuration.nix, networking.nat

Comments