--- title: "Contractor: a local Ick worker" author: Lars Wirzenius date: work in progress bindings: worker.yaml ... Introduction ============================================================================= A CI system executes a sequence of build steps to build a project, usually on a server. A common problem is that to debug a build failure, one needs to push changes to the CI system, to add logging or other debugging help. This is tedious, and often slow. For Ick, we want a way to perform all the build steps locally, via the command line, and without setting up a server or other infrastructure. The aim is to give developers a way to debug, inspect, and tweak project build specifications easily, quickly, and without bothering other users of CI. For this to work well, the local build tool must behave as closely as possible to the CI server. We call the command line tool to emulate a build on an Ick worker a **contractor**. Design alternatives ----------------------------------------------------------------------------- A realtively obvious way of doing this is to perform all builds in a Docker container. Docker is common, if not quite ubiquitous, and relatively easy to use. However, Docker is not great for security isolation, and the public images are not always benign. However, Docker being very popular means it should not be disregarded lightly. Other container implementations exist and could be used instead of Docker: systemd-nspawn, Kubernetes, Podman, and more. Ick has, so far, been using systemd-nspawn. They also do not necessarily have a great story for security isolation, and some may require too much infrastructure. However, the [Bubblewrap][] (bwrap) tool is specifically meant for security isolation. It seems like a good basis for Ick, going forward. [Bubblewrap]: https://github.com/containers/bubblewrap All the container solutions are Linux specific. This may be a problem in the long term. Ick may want to support other operating systems, later. However, the server's worker hosts and the contractor may need to be implemented differently for different operating systems. For now, we only care about Linux. Architecture ============================================================================= The contractor architecture is as follows. ~~~dot digraph "arch" { buildyaml [shape=tab, label="Build specification", style=filled]; artifacts [shape=tab, style=filled]; contractor [shape=ellipse, label="Ick contractor", style=filled]; bwrap [shape=ellipse, label="Security container"]; workspace [shape=box, label="Temp workspace"]; systree [shape=box, label="Temp system tree"]; network [shape=octagon, label="Network"]; buildyaml -> contractor; contractor -> systree; contractor -> workspace; contractor -> network; contractor -> bwrap; bwrap -> workspace; bwrap -> systree; artifacts -> contractor; contractor -> artifacts; } ~~~ There are several components: * A **build specification** provided by the user. Lists all the projects to build, and how to build them. * An **artifact store**, a local directory, maintained by the contractor. Persistent across invocations of the contractor. * A **workspace**, a local temporary directory, created by the controller for the duration a project build. Populated by the build steps of a project. * A **system tree**, a local temporary directory, created by the controller for the duration of a project build. Populated by contractor build actions to install an operating system and build dependencies. * A **security container** implemented using Bubblewrap, combining the workspace and the system tree into a process using Linux kernel namespaces. All untrusted build steps are executed in the security container, without network access. The contractor reads a build specfication from a file, instead of retrieving it from the Ick controller (see [indepenence](#req-independent)), and executes all the build steps defined therein (see [Build specification](#buildspec). From an architectural point of view, a build specification defines where to get the source code from, per-project parameters, and a sequence of build steps to build the project. Each build step is one the following: * An action implemented directly in the contractor. * An action implemented by executing a well-known, vetted external program, invoked in a secure way. * An action implemented by executing a program installed into a container. * An arbitrary snippet of shell code from the build specification. Actions implemented by the contractor or by well-known tools are considered safe and secure. They will be able to access the network, the host file system. Arbitrary shell snippets, however, are considered unsafe and insecure, and will be security isolated using [Bubblewrap][] and executed without network access. Further, shell snippets will only be run in a specially constructed container, not software installed on the host system. The container's **system tree** (all the software installed in the container) is constructed from a build artifact, possibly using the debootstrap program. However, the build may install any software it wants to the container. This is necessary so that the build can have exactly the build environment it wants, with all build tools and build dependencies it needs. The contractor executes also the trusted, external programs using bwrap, but with network access, and read-only access to the host's operating system. The contractor implmements a persisten local artifact store, probably as a plain directory, and provides build steps for controlled access to it. Simple threat modelling ----------------------------------------------------------------------------- * The main threat comes from executing unvetted code that may have been written by an attacker: the shell code involved in executing build steps, and the code being built. * Bubblewrap isolates any such code so that it can't attack the build host (information leak, privilege elevation), except by a denial of service by using too much resources (CPU, RAM, disk) on the build host. Such code will not have network access. * The build step actions must be constrained in ways that prevent executing untrusted code outside a security container. * The contractor is to provide controlled, secure build step primitives for setting up a security container and workspace for building a project. This will include installing any build dependencies. Build step actions ----------------------------------------------------------------------------- The contractor implements the following actions, which the build specification can use freely. [debootstrap]: https://wiki.debian.org/Debootstrap empty-workspace : Create a new, empty workspace. The workspace will be automatically deleted after the build has finished, unless the user who invoked the contractor indicates otherwise. create-artifact : Create an artifact from some or all files in the workspace. unpack-artifact : Retrieve an artifact and unpack it into the workspace. create-systree : Create a new, empty directory to act as the system tree, and retrieve an artifact, and unpack it to the new directory. debootstrap : Run the [debootstrap][] program to install Debian into the workspace. Also install any other packages into the debootstrap'd tree. git : Clone a git repository into the workspace. shell : Execute the given shell snippet. Table: The actions are implemented as follows action where? network? ------- ------- --------- empty-workspace contractor no create-artifact contractor no unpack-artifact contractor no create-systree contractor no debootstrap bwrap on host yes git bwrap on host yes shell bwrap on systree no It is intended that any actions implemented by the contractor itself, or by invoking well-known, vetted programs on the host, are safe and secure, and cannot cause a information leaks or privilege escalation. Build specifications {#buildspec} ============================================================================= This is different from what Ick currently does. That's intentional: we're keeping things simple, and are also exploring the solution space. A build specification is a YAML file, defining one or more projects to build. Each project has a name (unique to the build spec), and a sequence of build steps. Each build step defines an action, and possibly step parameters. ~~~yaml projects: - project: build-unstable-systree build-steps: - action: empty-workspace - action: debootstrap suite: unstable packages: - build-essential - action: create-artifact artifact-name: unstable-systree - project: hello actions: - action: create-systree artifact-name: unstable-systree container-name: default - action: empty-workspace - action: shell container-name: default shell: | echo hello, world, from security container ~~~ Requirements ============================================================================= This chapter lists requirements for the Ick contractor. These requirements are not meant to be automatically verifiable. For specific, automatically testable acceptance criteria, see the later [chapter with acceptance tests for Contractor](#acceptance). Each requirement here is given a unique mnemnoic id for easier reference in discussions. ## Secure {#req-secure} Builds run by the contractor should be secure. The code run in the build steps should not be able to attack or negatively affect the host computer. ## IdenticalBuilds {#req-builds} The contractor should execute a build identically to the Ick server: unless the code being built intentionally makes a distinction, the results should be the same, or at least any differences should not be due to the contractor being different from the server. ## IdenticalInputs {#req-inputs} Further, the contractor should read exactly the same build specfications as the Ick server. This may mean that the contractor defines a new, better language for build specifications, which Ick itself will implement later. ## Easy {#req-easy} The user should be able to use the contractor as just another command line utility. It must not require setting up daemons, server processes, servers, or other infrastructure. ## Independent {#req-independent} The controller should not require, or automatically use, any of the Ick server components: the controller, the IDP, or the artifact stores. Any such interactions should be invoked explicitly by the user (e.g., "fetch this artifact from the artifact store"). Acceptance criteria for Ick contractor {#acceptance} ============================================================================= Debian stable systree ----------------------------------------------------------------------------- Scenario to build a base Debian stable systree artifact, and run something in it and check the output. Install non-base packages ----------------------------------------------------------------------------- Scenario using debootstrap action that installs additional packages. Create artifact ----------------------------------------------------------------------------- Scenario that creates an artifact from parts of the workspace, and restores it in a different project. Network isolation ----------------------------------------------------------------------------- Scenario that checks the security container prevents shell snippets from accessing the network. Filesystem isolation ----------------------------------------------------------------------------- Scenario that checks the security container prevents shell snippts from seeing or modifying the host's filesystem: /etc, /home, /tmp, /var/tmp at least. Also, checks that the user in the container can't modfify the system tree. User isolation ----------------------------------------------------------------------------- Scenario that checks the security container has a separate user/group db from the host. Hostname isolation ----------------------------------------------------------------------------- Scenario that checks the security container has a specific hostname. Build environment setup ----------------------------------------------------------------------------- Scenario that checks the security container has /workspace as cwd, and a specific uid and gid. Get sources using git ----------------------------------------------------------------------------- Scenario that clone the project's source code using git from a git server. Known problems ============================================================================= * If debootstrap is the only way to construct a system tree, how can a build install packages from non-Debian package repositories? For example, from a repository of the user's own packages? * Projects need to be able to depend on each other, so that when one builds successfully, that can trigger another project to build. For example, when a systree artifact changes, anything that uses it should probably be rebuilt.