path: root/worker.md

---
title: "Contractor: a local Ick worker"
author: Lars Wirzenius
date: work in progress
bindings: worker.yaml
...


Introduction
=============================================================================

A CI system executes a sequence of build steps to build a project,
usually on a server. A common problem is that to debug a build
failure, one needs to push changes to the CI system, to add logging or
other debugging help. This is tedious, and often slow.

For Ick, we want a way to perform all the build steps locally, via the
command line, and without setting up a server or other infrastructure.
The aim is to give developers a way to debug, inspect, and tweak
project build specifications easily, quickly, and without bothering
other users of CI.

For this to work well, the local build tool must behave as closely as
possible to the CI server.

We call the command line tool to emulate a build on an Ick worker a
**contractor**.


Design alternatives
-----------------------------------------------------------------------------

A realtively obvious way of doing this is to perform all builds in a
Docker container. Docker is common, if not quite ubiquitous, and
relatively easy to use. However, Docker is not great for security
isolation, and the public images are not always benign. However,
Docker being very popular means it should not be disregarded lightly.

Other container implementations exist and could be used instead of
Docker: systemd-nspawn, Kubernetes, Podman, and more. Ick has, so far,
been using systemd-nspawn. They also do not necessarily have a great
story for security isolation, and some may require too much
infrastructure.

However, the [Bubblewrap][] (bwrap) tool is specifically meant for
security isolation. It seems like a good basis for Ick, going forward.

[Bubblewrap]: https://github.com/containers/bubblewrap

All the container solutions are Linux specific. This may be a problem
in the long term. Ick may want to support other operating systems,
later. However, the server's worker hosts and the contractor may need
to be implemented differently for different operating systems. For
now, we only care about Linux.


Architecture
-----------------------------------------------------------------------------

The contractor architecture is as follows.

~~~dot
digraph "arch" {
  buildyaml [shape=box, label="Build specification"];
  artifacts [shape=box];
  workspace [shape=box, label="Temp workspace"];
  systree   [shape=box, label="System tree"];

  buildyaml -> contractor;
  contractor -> bwrap;
  bwrap -> workspace;
  bwrap -> systree;
  artifacts -> contractor;
  contractor -> artifacts;
}
~~~

The contractor reads a build specfication from a file, instead of
retrieving it from the Ick controller (see
[indepenence](#req-independent)), and executes all the build steps
defined therein (see [Build specification](#buildspec). From an
architectural point of view, a build specification defines where to
get the source code from, per-project parameters, and a sequence of
build steps to build the project.

Each build step is one the following:

* An action implemented directly in the contractor.

* An action implemented by executing a well-known, vetted external
  program, invoked in a secure way.

* An action implemented by executing a program installed into a
  container.

* An arbitrary snippet of shell code from the build specification.

Actions implemented by the contractor or by well-known tools are
considered safe and secure. They will be able to access the network,
the host file system. Arbitrary shell snippets, however, are
considered unsafe and insecure, and will be security isolated using
[Bubblewrap][] and executed without network access. Further, shell
snippets will only be run in a specially constructed container, not
software installed on the host system. The container's **system tree**
(all the software installed in the container) is constructed from a
build artifact, possibly using the debootstrap program.

However, the build may install any software it wants to the container.
This is necessary so that the build can have exactly the build
environment it wants, with all build tools and build dependencies it
needs.

The contractor executes also the trusted, external programs using
bwrap, but with network access, and read-only access to the host's
operating system.

The contractor implmements a local artifact store, probably as a plain
directory, and provides build steps for controlled access to it.


Simple threat modelling
-----------------------------------------------------------------------------

* The main threat comes from executing unvetted code that may have
  been written by an attacker: the shell code involved in executing
  build steps, and the code being built.

* Bubblewrap isolates any such code so that it can't attack the build
  host (information leak, privilege elevation), except by a denial of
  service by using too much resources (CPU, RAM, disk) on the build
  host. Such code will not have network access.

* The build step actions must be constrained in ways that prevent
  executing untrusted code outside a security container.

* The contractor is to provide controlled, secure build step
  primitives for setting up a security container and workspace for
  building a project. This will include installing any build
  dependencies.


Build step actions
-----------------------------------------------------------------------------

The contractor implements the following actions, which the build
specification can use freely.

[debootstrap]: https://wiki.debian.org/Debootstrap

empty-workspace

:   Create a new, empty workspace. The workspace will be automatically
    deleted after the build has finished, unless the user who invoked
    the contractor indicates otherwise.

create-artifact

:   Create an artifact from some or all files in the workspace.

unpack-artifact

:   Retrieve an artifact and unpack it into the workspace.

create-systree

:   Create a new, empty directory to act as the system tree, and
    retrieve an artifact, and unpack it to the new directory.

debootstrap

:   Run the [debootstrap][] program to install Debian into the
    workspace. Also install any other packages into the debootstrap'd
    tree.

shell

:   Execute the given shell snippet.

Table: The actions are implemented as follows

action              where?              network?
-------             -------             ---------
empty-workspace     contractor          no
create-artifact     contractor          no
unpack-artifact     contractor          no
create-systree      contractor          no
debootstrap         bwrap on host       yes
shell               bwrap on systree    no

It is intended that any actions implemented by the contractor itself,
or by invoking well-known, vetted programs on the host, are safe and
secure, and cannot cause a information leaks or privilege escalation.

Build specifications {#buildspec}
=============================================================================

This is different from what Ick currently does. That's intentional:
we're keeping things simple, and are also exploring the solution
space.

A build specification is a YAML file, defining one or more projects to
build. Each project has a name (unique to the build spec), and a
sequence of build steps. Each build step defines an action, and
possibly step parameters.

~~~yaml
projects:
- project: build-unstable-systree
  build-steps:
  - action: empty-workspace
  - action: debootstrap
    suite: unstable
    packages:
    - build-essential
  - action: create-artifact
    artifact-name: unstable-systree

- project: hello
  actions:
  - action: create-systree
    artifact-name: unstable-systree
    container-name: default
  - action: empty-workspace
  - action: shell
    container-name: default
    shell: |
      echo hello, world, from security container
~~~





Requirements
=============================================================================

This chapter lists requirements for the Ick contractor. These
requirements are not meant to be automatically verifiable. For
specific, automatically testable acceptance criteria, see the later
[chapter with acceptance tests for Contractor](#acceptance).

Each requirement here is given a unique mnemnoic id for easier
reference in discussions.

##  Secure {#req-secure}

Builds run by the contractor should be secure. The code run in the
build steps should not be able to attack or negatively affect the
host computer.


## IdenticalBuilds {#req-builds}

The contractor should execute a build identically to the Ick
server: unless the code being built intentionally makes a
distinction, the results should be the same, or at least any
differences should not be due to the contractor being different
from the server.


## IdenticalInputs {#req-inputs}

Further, the contractor should read exactly the same build
specfications as the Ick server. This may mean that the
contractor defines a new, better language for build
specifications, which Ick itself will implement later.


## Easy {#req-easy}

The user should be able to use the contractor as just another
command line utility. It must not require setting up daemons,
server processes, servers, or other infrastructure.


## Independent {#req-independent}

The controller should not require, or automatically use, any of
the Ick server components: the controller, the IDP, or the
artifact stores. Any such interactions should be invoked
explicitly by the user (e.g., "fetch this artifact from the
artifact store").



Acceptance criteria for Ick contractor {#acceptance}
=============================================================================

Debian stable systree
-----------------------------------------------------------------------------

Scenario to build a base Debian stable systree artifact, and run
something in it and check the output.

Install non-base packages
-----------------------------------------------------------------------------

Scenario using debootstrap action that installs additional packages.

Create artifact
-----------------------------------------------------------------------------

Scenario that creates an artifact from parts of the workspace, and
restores it in a different project.

Network isolation
-----------------------------------------------------------------------------

Scenario that checks the security container prevents shell snippets
from accessing the network.

Filesystem isolation
-----------------------------------------------------------------------------

Scenario that checks the security container prevents shell snippts
from seeing or modifying the host's filesystem: /etc, /home, /tmp,
/var/tmp at least.

Also, checks that the user in the container can't modfify the system
tree.

User isolation
-----------------------------------------------------------------------------

Scenario that checks the security container has a separate user/group
db from the host.

Hostname isolation
-----------------------------------------------------------------------------

Scenario that checks the security container has a specific hostname.

Build environment setup
-----------------------------------------------------------------------------

Scenario that checks the security container has /workspace as cwd, and
a specific uid and gid.


Known problems
=============================================================================

* If debootstrap is the only way to construct a system tree, how can a
  build install packages from non-Debian package repositories? For
  example, from a repository of the user's own packages?

* Projects need to be able to depend on each other, so that when one
  builds successfully, that can trigger another project to build. For
  example, when a systree artifact changes, anything that uses it
  should probably be rebuilt.