Container Configuration¶

Container configuration is a YAML file which is usually put into /config/<service_name>.yaml into container image itself.

Note

Curently container configuration may be put into any folder inside the image, but we may fix this folder later. The arbitrary path for container configuration may be a security vulnerability.

The somewhat minimal configuration is looks like following:

kind: Daemon
user-id: 1
volumes:
  /tmp: !Tmpfs { size: 100m }
executable: /bin/sleep
arguments: [60]

See overview for guidelines.

Variables¶

Container can declare some things, that can be changed in specific instantiation of the service, for example:

variables:
  tcp_port: !TcpPort
kind: Daemon
user-id: 1
volumes:
  /tmp: !Tmpfs { size: 100m }
executable: /bin/some_program
arguments:
- "--listen=localhost:@{tcp_port}"

The variables key declares variable names and types. Value for these variables can be provided in variables in Process Config.

There are the following types of variables:

TcpPort

Allows a number between 1-65535 and ensures that the number matches port range allowed in sandbox (see allow-tcp-ports)

Changed in version 0.17.4: Added activation parameter as a shortcut to support systemd activation protocol. I.e. the following (showing two ports for more comprehensive example):

variables:
  port1: !TcpPort { activation: systemd }
  port2: !TcpPort { activation: systemd }

Means to add something like this:

  variables:
    port1: !TcpPort
    port2: !TcpPort
  tcp-ports:
    "@{port1}":
      fd: 3
    "@{port2}":
      fd: 4
  environ:
    LISTEN_FDS: 1
    LISTEN_FDNAMES: "port1:port2"
    LISTEN_PID: "@{lithos:pid}"

This works for any number of sockets. And it requires that
``LISTEN_FDS`, ``LISTEN_FDNAMES``, ``LISTEN_PID`` were absent in the
``environ`` as written in the file. Also it doesn't allow fine-grained
control over parameters of the socket and file descriptor numbers.
Use full form if you need specific options.

Choice

Allows a value from a fixed set of choices (example: !Choice ["high-priority", "low-priority"])

Name

Allows a value that matches regex ^[0-9a-zA-Z_-]+$. Useful for passing names of things into a script without having a chance to keep value unescaped when passing somewhere within a script or using it as a filename.

New in version 0.10.3.

DottedName

Allows arbitrary DNS-like name. It’s defined as dot-separated name with only alphanumeric and underscores, where no component could start or end with a dash and no consequent dots allowed.

New in version 0.17.4.

All entries of @{variable_name} are substituted in the following fields:

arguments
The values of environ (not in the keys yet)
The key in the tcp-ports (i.e. port number)

The expansion in any other place does not work yet, but may be implemented in the future. Only declared variables can be substituted. Trying to substitute undeclared variables or non-existing built-in variable results into configuration syntax error.

There are the number of builtin variables that start with lithos::

lithos:name: Name of the process, same as inserted in LITHOS_NAME environment variable
lithos:config_filename: Full path of this configuration file as visible from within container
lithos:pid: Pid of the process as visible inside of the container. Note: this variable can only be in environment and can only be full value of the variable. I.e. PID: “@{lithos:pid}” is fine, but PID: “pid is @{lithos:pid}” is not allowed. (In most cases this variable is exaclty 2, this is expected but might not be always true in some cases).

More built-in variables may be added in the future. Built-in variables doesn’t have to be declared.

Reference¶

kind¶

One of Daemon (default), Command or CommandOrDaemon.

The Daemon is long-running process that is monitored by supervisor.

The Command things are just one-off tasks, for example to initialize local file system data, or to check health of daemon process. The Command things are run by lithos_cmd utility

The CommandOrDaemon may be used in both ways, based on how it was declared in Process Config. In the command itself you can distinguish how it is run by /cmd. in LITHOS_NAME or cgroup name or better you can pass variable to a specific command and/or daemon.

New in version 0.10.3: ContainerOrDaemon mode

user-id¶: The numeric user indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of 0 is not allowed.

group-id¶: The numeric group indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of 0 is not allowed.

memory-limit¶

The memory limit for process and it’s children. This is enforced by cgroups, so this needs memory cgroup to be enabled (otherwise its no-op). See cgroup-controllers for more info. Default: nolimit.

You can use ki, Mi and Gi units for memory accounting. See integer-units.

Changed in version 0.14.0: Previously it only set memory.limit_in_bytes but now it also sets memory.memsw.limit_in_bytes if the latter exists (otherwise skipping silently). This helps to kill processes earlier instead of swapping out to disk.

cpu-shares¶

The number of CPU shares for the process. Default is 1024 which means all processes get equal share. You may split them to different values like 768 for one process and 256 for another one.

This is enforced by cgroups, so this needs cpu cgroup to be enabled (otherwise its no-op). See cgroup-controllers for more info.

fileno-limit¶: The limit on file descriptors for process. Default 1024.

restart-timeout¶: The minimum time to wait between subsequent restarts of failed processes in seconds. This is to ensure that it doesn’t boggles down CPU. Default is 1 second. It’s enough so that lithos itself do not hang. But it should be bigger for heavy-weight processes. Note: this is time between restarts, i.e. if process were running more than this number of seconds it will be restarted immediately.

kill-timeout¶

(default 5 seconds) The time to wait for application to die. If it is not dead by this number of seconds we kill it with KILL.

You should not rely on this timeout to be precise for multiple reasons:

Unidentified children are killed with a default timeout (5 sec). This includes children which are being killed when their configuration is removed.
When lithos is restarted (i.e. to reload a configuration) during the timeout, the timeout is reset. I.e. the process may hang more than this time.

executable¶: The path to executable to run. Only absolute paths are allowed.

arguments¶: The list of arguments for the command. Except argument zero.

environ¶: The mapping of values that are set for process. You must set all needed environment variables here. The only variable that is propagated by default is TERM. Also few special LITHOS_ variables may be set. This means you must set all the basic LANG, HOME and so on explicitly. This is to ensure that your environment is always the same regardless of where you run process.

secret-environ¶

Similarlty to environ but contains encrypted environment variables. For example:

secret-environ:
  DB_PASSWORD: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc

Note: if environment variable is both in environ and secret-environ which one overrides is not specified for now.

You can encrypt variables using lithos_crypt:

lithos_crypt encrypt -k key.pub -d "secret" -n "some.namespace"

You only need public key for encryption. So the idea is that public key is published somewhere and anyone, even users having to access to server/private key can add a secret.

The -n / --namespace parameter must match one of the secrets-namespaces defined for project’s sandbox.

Usually there is only one private key for every deployment (cluster), and a single namespace per project. But in some cases you might need single lithos config for multiple destinations or just want to rotate private key smoothly. So you can put secret(s) encoded for multiple keys and/or namespaces:

secret-environ:
  DB_PASSWORD:
  - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE
  - v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q==
  - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE

Note: technically you can encrypt different secrets here, we can’t enforce that, but it’s very discouraged.

The underlying encyrption is curve25519xsalsa20poly1305 which is compatible with libnacl and libsodium.

See Encrypted Variables for more info.

This option conflicts with secret-environ-file.

secret-environ-file¶

Path to the file where to read secret environ from. Instead of including secret-environ in the container config itself you can use a separate file where data is contained. This is useful to keep single set of secrets shared between multiple containers.

The target file is also yaml, but it containers just mapping of names of the secrets to their values (or lists). For example:

PASSWD1: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc
PASSWD2:
- v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE
- v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q==

Absolute paths here interpreted relative to the container root and relative paths are interpreted relative to the container config itself. Note: we currently support reading file from container’s filesystem only, whether reading from a volume works or not is unspecified at the moment.

This option conflicts with secret-environ.

workdir¶: The working directory for target process. Default is /. Working directory must be absolute.

resolv-conf¶

Parameters of the /etc/resolv.conf file to generate. Default configuration is:
resolv-conf:
    mount: nil  # which basically means "auto"
    copy-from-host: true
Which means resolv.conf from host where lithos is running is copied to the “state” directory of the container. Then if /etc/resolv.conf in container is a file (and not a symlink) resolv conf is mounted over the /etc/resolv.conf.

More options are expected to be added later.

Changed in version 0.15.0: mount option added. Previously to make use of resolv.conf you should symlink ln -s /state/resolv.conf /etc/resolv.conf in the container’s image.

Another change is that copy-from-host copies file that is specified in sandbox’s resolv.conf which default to /etc/resolv.conf but may be different.

Parameters:

copy-from-host

(default true) Copy resolv.conf file from host machine.

Note: even if copy-from-host is true, additional-hosts from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places.

Changed in version v0.11.0: The parameter used to be false by default, because we were thinking about better (perceived) isolation.

mount

(default nil, which means “auto”) Mount copied resolv.conf file over /etc/resolv.conf.

nil enables mounting if /etc/resolv.conf is present in the container and is a file (not a symlink) and also copy-from-host is true

New in version 0.15.0.

hosts-file¶

Parameters of the /etc/hosts file to generate. Default configuration is:
hosts-file:
    mount: nil  # which basically means "auto"
    localhost: true
    public-hostname: true
    copy-from-host: false
Changed in version 0.15.0: mount option added. Previously to make use of resolv.conf you should symlink ln -s /state/resolv.conf /etc/resolv.conf in the container’s image.

Another change is that copy-from-host copies file that is specified in sandbox’s resolv.conf which default to /etc/resolv.conf but may be different.

Parameters:

copy-from-host

(default true) Copy hosts file from host machine.

Note: even if copy-from-host is true, additional-hosts from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places.

Changed in version v0.11.0: The parameter used to be false by default, because we were thinking about better (perceived) isolation. And also because hostname in Ubuntu doesn’t resolve to real IP of the host. But we find those occassions where it matters to be quite rare in practice and using hosts-file as well as resolv.conf from the host system as the most expected and intuitive behavior.

mount

(default nil, which means “auto”) Mount produced hosts file over /etc/hosts.

nil enables mounting if /etc/hosts is present in the container and is a file (not a symlink).

Value of true fails if /etc/hosts is not a file. Value of false leaves /etc/hosts intact.

New in version 0.15.0.

localhost

(default is true when copy-from-host is false) A boolean which defines whether to add 127.0.0.1 localhost record to hosts

public-hostname

(default is true when copy-from-host is false) Add to hosts file the result of gethostname system call along with the ip address that name resolves into.

uid-map, gid-map¶

The list of mapping for uids(gids) in the user namespace of the container. If they are not specified the user namespace is not used. This setting allows to run processes with uid zero without the risk of being the root on host system.

Here is a example of maps:

uid-map:
- {inside: 0, outside: 1000, count: 1}
- {inside: 1, outside: 1, count: 1}
gid-map:
- {inside: 0, outside: 100, count: 1}

Note

Currently you may have uid-map either in a sandbox or in a container config, not both.

stdout-stderr-file¶

This redirects both stdout and stderr to a file. The path is opened inside the container. So must reside on one of the mounted writeable Volumes. Probably you want Persistent volume. While it can be on Tmpfs or Statedir the applicability of such thing is very limited.

Usually log is put into the directory specified by stdio-log-dir.

interactive¶: (default false) Useful only for containers of kind Command. If true lithos_cmd doesn’t clobber stdin and doesn’t redirect stdout and stderr to a log file, effectively allowing command to be used for interactive commands or as a part of pipeline.

Note

for certain use cases, like pipelines it might be better to use fifo’s (see man mkfifo) and a Daemon instead of this one because daemons may be restarted on death or for software upgrade, while Command is not supervised by lithos.

New in version 0.6.3.

Changed in version ≥0.5: Commands were always interactive

restart-process-only¶

(default false) If true when restarting process (i.e. in case process died or was killed), lithos restarts just the failed process. This means container will not be recreated, volumes will not be remounted, tmpfs will not be cleaned and some daemon processes may leave running.

By default lithos_knot which is pid 1 in the container exits when process dies. Which means all other processes will die on KILL signal, and container will be removed and created again. It’s a little bit slower but safer default. This leaves no hanging daemons, orphan files in state dir and tmpfs garbage.

volumes¶: The mapping of mountpoint to volume definition. See Volumes for more info

tcp-ports¶

Binds address and provides file descriptor to the child process. All the children receive dup of the same file descriptor, so may all do accept() simultaneously. The configuration looks like:

tcp-ports:
  7777:
    fd: 3
    host: 0.0.0.0
    listen-backlog: 128
    reuse-addr: true
    reuse-port: false

All the fields except fd are optional.

Programs may require to pass listening file descriptor number by some means (usually environment). For example to run nginx with port bound (so you don’t need to start it as root) you need:

tcp-ports:
  80:
    fd: 3
    set-non-block: true
environ:
  NGINX: "3;"

To run gunicorn you may want:

tcp-ports:
  80:
    fd: 3
environ:
  GUNICORN_FD: "3"

More examples are in Handing TCP Ports

Parameters:

key

TCP port number.

Warning

The paramters (except fd) do not change after socket is bound even if configuration change
You can’t bind same port with different hostnames in a single process (previously there was a global limit for the single port for whole lithos master, currently this is limited just because tcp-ports is a mapping)

Port parameter should be unique amongst all containers. But sharing port works because it is useful if you are doing smooth software upgrade (i.e. you have few old processes running and few new processes running both sharing same port/file-descriptor). Running them on single port is not the best practices for smooth software upgrade but that topic if out of scope of this documentation.

fd

Required. File descriptor number

host

(default is 0.0.0.0 meaning all addresses) Host to bind to. It must be IP address, hostname is not supported.

listen-backlog

(default 128) the value to pass to the listen() system call. The value is capped by net.core.somaxconn

reuse-addr

(default true) Sets SO_REUSEADDR socket option

reuse-port

(default false) If set to true this changes behavior of the lithos with respect of the socket. In default case lithos binds socket as quick as possible and passes to each child on start. When this set to true, lithos creates a separate socket and calls bind for each process start. This has two consequences:

Socket is not bound when no processes started (i.e. they are failing)
Each process gets separate in-kernel queue of connections to accept

This should be set to true only on very high performant servers that experience assymetric workload in default case.

set-non-block

(default false) Sets socket into non-blocking mode. This is usually done by an application itself but some of them (especially ones, that don’t expect socket to be created by an external utility, e.g. nginx) don’t do it themselves.

external

(default false) If set to true listen on the port in the external network (host network of the system not bridged network). This is only effective if bridged-network is enabled for container.

Changed in version 0.17.0: Previously we only allowed external ports to be declared in lithos config. It was expected that container in bridged network can listen port itself. But it turned out file descriptors are still convenient for some use-cases even inside a bridge.

metadata¶: (optional) Allows to add arbitrary metadata to lithos configuration file. Lithos does not use and does not validate this data in any way (except that it must be a valid YAML). The metadata can be used by other tools that inspect lithos configs and extract data from it. In particular, we use metadata for our deployment tools (to keep configuration files more consolidated instead of keeping then in small fragments).

normal-exit-codes¶

(optional) A list of exit codes which are considered normal for process death. This currently only improves failures metric. See Determining Failure.

Note: by default even 0 exit code is considered an error for daemons, and for commands (lithos_cmd) 0 is considered successful.

This setting is intended for daemons which may voluntarily exit for some reason (soft memory limit, version upgrade, configuration reload).

It’s not recommended to add 0 or 1 into the list, as some commands threat them pretty arbitrarily. For example 0 is exit code of most utilities running –help so this mistake will not be detected. And 1 is used for arbitrary crashes in scripting languages. So the good idea is to define some specific code in range of 8..120 to define successful exit.