Container Configuration¶
Container configuration is a YAML file which is usually put into
/config/<service_name>.yaml
into container image itself.
Note
Curently container configuration may be put into any folder inside the image, but we may fix this folder later. The arbitrary path for container configuration may be a security vulnerability.
The somewhat minimal configuration is looks like following:
kind: Daemon
user-id: 1
volumes:
/tmp: !Tmpfs { size: 100m }
executable: /bin/sleep
arguments: [60]
See overview for guidelines.
Variables¶
Container can declare some things, that can be changed in specific instantiation of the service, for example:
variables:
tcp_port: !TcpPort
kind: Daemon
user-id: 1
volumes:
/tmp: !Tmpfs { size: 100m }
executable: /bin/some_program
arguments:
- "--listen=localhost:@{tcp_port}"
The variables
key declares variable names and types. Value for these
variables can be provided in variables
in Process Config.
There are the following types of variables:
- TcpPort
Allows a number between 1-65535 and ensures that the number matches port range allowed in sandbox (see
allow-tcp-ports
)Changed in version 0.17.4: Added
activation
parameter as a shortcut to support systemd activation protocol. I.e. the following (showing two ports for more comprehensive example):variables: port1: !TcpPort { activation: systemd } port2: !TcpPort { activation: systemd }
Means to add something like this:
variables: port1: !TcpPort port2: !TcpPort tcp-ports: "@{port1}": fd: 3 "@{port2}": fd: 4 environ: LISTEN_FDS: 1 LISTEN_FDNAMES: "port1:port2" LISTEN_PID: "@{lithos:pid}" This works for any number of sockets. And it requires that ``LISTEN_FDS`, ``LISTEN_FDNAMES``, ``LISTEN_PID`` were absent in the ``environ`` as written in the file. Also it doesn't allow fine-grained control over parameters of the socket and file descriptor numbers. Use full form if you need specific options.
- Choice
- Allows a value from a fixed set of choices
(example:
!Choice ["high-priority", "low-priority"]
) - Name
Allows a value that matches regex
^[0-9a-zA-Z_-]+$
. Useful for passing names of things into a script without having a chance to keep value unescaped when passing somewhere within a script or using it as a filename.New in version 0.10.3.
- DottedName
Allows arbitrary DNS-like name. It’s defined as dot-separated name with only alphanumeric and underscores, where no component could start or end with a dash and no consequent dots allowed.
New in version 0.17.4.
All entries of @{variable_name}
are substituted in the following fields:
The expansion in any other place does not work yet, but may be implemented in the future. Only declared variables can be substituted. Trying to substitute undeclared variables or non-existing built-in variable results into configuration syntax error.
There are the number of builtin variables that start with lithos:
:
- lithos:name
- Name of the process, same as inserted in
LITHOS_NAME
environment variable - lithos:config_filename
- Full path of this configuration file as visible from within container
- lithos:pid
- Pid of the process as visible inside of the container. Note: this variable
can only be in environment and can only be full value of the variable.
I.e. PID: “@{lithos:pid}” is fine,
but PID: “pid is @{lithos:pid}” is not allowed. (In most cases
this variable is exaclty
2
, this is expected but might not be always true in some cases).
More built-in variables may be added in the future. Built-in variables doesn’t have to be declared.
Reference¶
-
kind
¶ One of
Daemon
(default),Command
orCommandOrDaemon
.The
Daemon
is long-running process that is monitored by supervisor.The
Command
things are just one-off tasks, for example to initialize local file system data, or to check health of daemon process. TheCommand
things are run bylithos_cmd
utilityThe
CommandOrDaemon
may be used in both ways, based on how it was declared in Process Config. In the command itself you can distinguish how it is run by/cmd.
inLITHOS_NAME
or cgroup name or better you can pass variable to a specific command and/or daemon.New in version 0.10.3:
ContainerOrDaemon
mode
-
user-id
¶ The numeric user indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of
0
is not allowed.
-
group-id
¶ The numeric group indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of
0
is not allowed.
-
memory-limit
¶ The memory limit for process and it’s children. This is enforced by cgroups, so this needs memory cgroup to be enabled (otherwise its no-op). See
cgroup-controllers
for more info. Default: nolimit.You can use
ki
,Mi
andGi
units for memory accounting. See integer-units.Changed in version 0.14.0: Previously it only set
memory.limit_in_bytes
but now it also setsmemory.memsw.limit_in_bytes
if the latter exists (otherwise skipping silently). This helps to kill processes earlier instead of swapping out to disk.
The number of CPU shares for the process. Default is
1024
which means all processes get equal share. You may split them to different values like768
for one process and256
for another one.This is enforced by cgroups, so this needs cpu cgroup to be enabled (otherwise its no-op). See
cgroup-controllers
for more info.
-
fileno-limit
¶ The limit on file descriptors for process. Default
1024
.
-
restart-timeout
¶ The minimum time to wait between subsequent restarts of failed processes in seconds. This is to ensure that it doesn’t boggles down CPU. Default is
1
second. It’s enough so that lithos itself do not hang. But it should be bigger for heavy-weight processes. Note: this is time between restarts, i.e. if process were running more than this number of seconds it will be restarted immediately.
-
kill-timeout
¶ (default
5
seconds) The time to wait for application to die. If it is not dead by this number of seconds we kill it withKILL
.You should not rely on this timeout to be precise for multiple reasons:
- Unidentified children are killed with a default timeout (5 sec). This includes children which are being killed when their configuration is removed.
- When lithos is restarted (i.e. to reload a configuration) during the timeout, the timeout is reset. I.e. the process may hang more than this time.
-
executable
¶ The path to executable to run. Only absolute paths are allowed.
-
arguments
¶ The list of arguments for the command. Except argument zero.
-
environ
¶ The mapping of values that are set for process. You must set all needed environment variables here. The only variable that is propagated by default is
TERM
. Also few specialLITHOS_
variables may be set. This means you must set all the basicLANG
,HOME
and so on explicitly. This is to ensure that your environment is always the same regardless of where you run process.
-
secret-environ
¶ Similarlty to
environ
but contains encrypted environment variables. For example:secret-environ: DB_PASSWORD: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc
Note: if environment variable is both in
environ
andsecret-environ
which one overrides is not specified for now.You can encrypt variables using
lithos_crypt
:lithos_crypt encrypt -k key.pub -d "secret" -n "some.namespace"
You only need public key for encryption. So the idea is that public key is published somewhere and anyone, even users having to access to server/private key can add a secret.
The
-n
/--namespace
parameter must match one of thesecrets-namespaces
defined for project’s sandbox.Usually there is only one private key for every deployment (cluster), and a single namespace per project. But in some cases you might need single lithos config for multiple destinations or just want to rotate private key smoothly. So you can put secret(s) encoded for multiple keys and/or namespaces:
secret-environ: DB_PASSWORD: - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE - v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q== - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE
Note: technically you can encrypt different secrets here, we can’t enforce that, but it’s very discouraged.
The underlying encyrption is curve25519xsalsa20poly1305 which is compatible with libnacl and libsodium.
See Encrypted Variables for more info.
This option conflicts with
secret-environ-file
.
-
secret-environ-file
¶ Path to the file where to read secret environ from. Instead of including
secret-environ
in the container config itself you can use a separate file where data is contained. This is useful to keep single set of secrets shared between multiple containers.The target file is also yaml, but it containers just mapping of names of the secrets to their values (or lists). For example:
PASSWD1: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc PASSWD2: - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE - v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q==
Absolute paths here interpreted relative to the container root and relative paths are interpreted relative to the container config itself. Note: we currently support reading file from container’s filesystem only, whether reading from a volume works or not is unspecified at the moment.
This option conflicts with
secret-environ
.
-
workdir
¶ The working directory for target process. Default is
/
. Working directory must be absolute.
-
resolv-conf
¶ Parameters of the
/etc/resolv.conf
file to generate. Default configuration is:resolv-conf: mount: nil # which basically means "auto" copy-from-host: true
Which means
resolv.conf
from host where lithos is running is copied to the “state” directory of the container. Then if/etc/resolv.conf
in container is a file (and not a symlink) resolv conf is mounted over the/etc/resolv.conf
.More options are expected to be added later.
Changed in version 0.15.0:
mount
option added. Previously to make use ofresolv.conf
you should symlinkln -s /state/resolv.conf /etc/resolv.conf
in the container’s image.Another change is that
copy-from-host
copies file that is specified in sandbox’sresolv.conf
which default to/etc/resolv.conf
but may be different.Parameters:
- copy-from-host
(default
true
) Copyresolv.conf
file from host machine.Note: even if
copy-from-host
istrue
,additional-hosts
from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places.Changed in version v0.11.0: The parameter used to be
false
by default, because we were thinking about better (perceived) isolation.- mount
(default
nil
, which means “auto”) Mount copiedresolv.conf
file over/etc/resolv.conf
.nil enables mounting if
/etc/resolv.conf
is present in the container and is a file (not a symlink) and alsocopy-from-host
is trueNew in version 0.15.0.
-
hosts-file
¶ Parameters of the
/etc/hosts
file to generate. Default configuration is:hosts-file: mount: nil # which basically means "auto" localhost: true public-hostname: true copy-from-host: false
Changed in version 0.15.0:
mount
option added. Previously to make use ofresolv.conf
you should symlinkln -s /state/resolv.conf /etc/resolv.conf
in the container’s image.Another change is that
copy-from-host
copies file that is specified in sandbox’sresolv.conf
which default to/etc/resolv.conf
but may be different.Parameters:
- copy-from-host
(default
true
) Copy hosts file from host machine.Note: even if
copy-from-host
istrue
,additional-hosts
from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places.Changed in version v0.11.0: The parameter used to be
false
by default, because we were thinking about better (perceived) isolation. And also because hostname in Ubuntu doesn’t resolve to real IP of the host. But we find those occassions where it matters to be quite rare in practice and usinghosts-file
as well asresolv.conf
from the host system as the most expected and intuitive behavior.- mount
(default
nil
, which means “auto”) Mount producedhosts
file over/etc/hosts
.nil enables mounting if
/etc/hosts
is present in the container and is a file (not a symlink).Value of
true
fails if/etc/hosts
is not a file. Value offalse
leaves/etc/hosts
intact.New in version 0.15.0.
- localhost
- (default is true when
copy-from-host
is false) A boolean which defines whether to add127.0.0.1 localhost
record tohosts
- public-hostname
- (default is true when
copy-from-host
is false) Add tohosts
file the result ofgethostname
system call along with the ip address that name resolves into.
-
uid-map, gid-map
¶ The list of mapping for uids(gids) in the user namespace of the container. If they are not specified the user namespace is not used. This setting allows to run processes with
uid
zero without the risk of being theroot
on host system.Here is a example of maps:
uid-map: - {inside: 0, outside: 1000, count: 1} - {inside: 1, outside: 1, count: 1} gid-map: - {inside: 0, outside: 100, count: 1}
Note
Currently you may have uid-map either in a sandbox or in a container config, not both.
-
stdout-stderr-file
¶ This redirects both stdout and stderr to a file. The path is opened inside the container. So must reside on one of the mounted writeable Volumes. Probably you want
Persistent
volume. While it can be onTmpfs
orStatedir
the applicability of such thing is very limited.Usually log is put into the directory specified by
stdio-log-dir
.
-
interactive
¶ (default
false
) Useful only for containers of kindCommand
. Iftrue
lithos_cmd doesn’t clobber stdin and doesn’t redirect stdout and stderr to a log file, effectively allowing command to be used for interactive commands or as a part of pipeline.Note
for certain use cases, like pipelines it might be better to use fifo’s (see
man mkfifo
) and aDaemon
instead of this one because daemons may be restarted on death or for software upgrade, whileCommand
is not supervised by lithos.New in version 0.6.3.
Changed in version ≥0.5: Commands were always interactive
-
restart-process-only
¶ (default
false
) If true when restarting process (i.e. in case process died or was killed), lithos restarts just the failed process. This means container will not be recreated, volumes will not be remounted, tmpfs will not be cleaned and some daemon processes may leave running.By default
lithos_knot
which is pid 1 in the container exits when process dies. Which means all other processes will die onKILL
signal, and container will be removed and created again. It’s a little bit slower but safer default. This leaves no hanging daemons, orphan files in state dir and tmpfs garbage.
-
tcp-ports
¶ Binds address and provides file descriptor to the child process. All the children receive dup of the same file descriptor, so may all do
accept()
simultaneously. The configuration looks like:tcp-ports: 7777: fd: 3 host: 0.0.0.0 listen-backlog: 128 reuse-addr: true reuse-port: false
All the fields except
fd
are optional.Programs may require to pass listening file descriptor number by some means (usually environment). For example to run nginx with port bound (so you don’t need to start it as root) you need:
tcp-ports: 80: fd: 3 set-non-block: true environ: NGINX: "3;"
To run gunicorn you may want:
tcp-ports: 80: fd: 3 environ: GUNICORN_FD: "3"
More examples are in Handing TCP Ports
Parameters:
- key
TCP port number.
Warning
- The paramters (except
fd
) do not change after socket is bound even if configuration change - You can’t bind same port with different hostnames in a
single process (previously there was a global limit for the
single port for whole lithos master, currently this is limited
just because
tcp-ports
is a mapping)
Port parameter should be unique amongst all containers. But sharing port works because it is useful if you are doing smooth software upgrade (i.e. you have few old processes running and few new processes running both sharing same port/file-descriptor). Running them on single port is not the best practices for smooth software upgrade but that topic if out of scope of this documentation.
- The paramters (except
- fd
- Required. File descriptor number
- host
- (default is
0.0.0.0
meaning all addresses) Host to bind to. It must be IP address, hostname is not supported. - listen-backlog
- (default
128
) the value to pass to the listen() system call. The value is capped bynet.core.somaxconn
- reuse-addr
- (default
true
) SetsSO_REUSEADDR
socket option - reuse-port
(default
false
) If set totrue
this changes behavior of the lithos with respect of the socket. In default case lithos binds socket as quick as possible and passes to each child on start. When this set totrue
, lithos creates a separate socket and calls bind for each process start. This has two consequences:- Socket is not bound when no processes started (i.e. they are failing)
- Each process gets separate in-kernel queue of connections to accept
This should be set to
true
only on very high performant servers that experience assymetric workload in default case.- set-non-block
- (default
false
) Sets socket into non-blocking mode. This is usually done by an application itself but some of them (especially ones, that don’t expect socket to be created by an external utility, e.g. nginx) don’t do it themselves. - external
(default
false
) If set totrue
listen on the port in the external network (host network of the system not bridged network). This is only effective ifbridged-network
is enabled for container.Changed in version 0.17.0: Previously we only allowed external ports to be declared in lithos config. It was expected that container in bridged network can listen port itself. But it turned out file descriptors are still convenient for some use-cases even inside a bridge.
-
metadata
¶ (optional) Allows to add arbitrary metadata to lithos configuration file. Lithos does not use and does not validate this data in any way (except that it must be a valid YAML). The metadata can be used by other tools that inspect lithos configs and extract data from it. In particular, we use metadata for our deployment tools (to keep configuration files more consolidated instead of keeping then in small fragments).
-
normal-exit-codes
¶ (optional) A list of exit codes which are considered normal for process death. This currently only improves
failures
metric. See Determining Failure.Note: by default even
0
exit code is considered an error for daemons, and for commands (lithos_cmd
)0
is considered successful.This setting is intended for daemons which may voluntarily exit for some reason (soft memory limit, version upgrade, configuration reload).
It’s not recommended to add 0 or 1 into the list, as some commands threat them pretty arbitrarily. For example 0 is exit code of most utilities running –help so this mistake will not be detected. And 1 is used for arbitrary crashes in scripting languages. So the good idea is to define some specific code in range of 8..120 to define successful exit.