path: root/worker.md

---
title: "Contractor: a local Ick worker"
author: Lars Wirzenius
date: work in progress
bindings: worker.yaml
...


Introduction
=============================================================================

A CI system executes a sequence of build steps to build a project,
usually on a server. A common problem is that to debug a build
failure, one needs to push changes to the CI system, to add logging or
other debugging help. This is tedious, and often slow.

For Ick, we want a way to perform all the build steps locally, via the
command line, and without setting up a server or other infrastructure.
The aim is to give developers a way to debug, inspect, and tweak
project build specifications easily, quickly, and without bothering
other users of CI.

For this to work well, the local build tool must behave as closely as
possible to the CI server.

We call the command line tool to emulate a build on an Ick worker a
**contractor**.


Design alternatives
-----------------------------------------------------------------------------

A realtively obvious way of doing this is to perform all builds in a
Docker container. Docker is common, if not quite ubiquitous, and
relatively easy to use. However, Docker is not great for security
isolation, and the public images are not always benign. However,
Docker being very popular means it should not be disregarded lightly.

Other container implementations exist and could be used instead of
Docker: systemd-nspawn, Kubernetes, Podman, and more. Ick has, so far,
been using systemd-nspawn. They also do not necessarily have a great
story for security isolation, and some may require too much
infrastructure.

However, the [Bubblewrap][] (bwrap) tool is specifically meant for
security isolation. It seems like a good basis for Ick, going forward.

[Bubblewrap]: https://github.com/containers/bubblewrap

All the container solutions are Linux specific. This may be a problem
in the long term. Ick may want to support other operating systems,
later. However, the server's worker hosts and the contractor may need
to be implemented differently for different operating systems. For
now, we only care about Linux.


Architecture
-----------------------------------------------------------------------------

The contractor architecture is as follows.

~~~dot
digraph "arch" {
  buildyaml [shape=box, label="Build specification"];
  artifacts [shape=box];
  workspace [shape=box, label="Temp workspace"];
  systree   [shape=box, label="System tree"];

  buildyaml -> contractor;
  contractor -> bwrap;
  bwrap -> workspace;
  bwrap -> systree;
  artifacts -> contractor;
  contractor -> artifacts;
}
~~~

The contractor reads a build specfication from a file, instead of
retrieving it from the Ick controller (see
[indepenence](#req-independent)), and executes all the build steps
defined therein (see [Build specification](#buildspec). From an
architectural point of view, a build specification defines where to
get the source code from, per-project parameters, and a sequence of
build steps to build the project.

Each build step is one the following:

* An action implemented directly in the contractor.

* An action implemented by executing a well-known, vetted external
  program.

* An action implemented by executing a program installed into a
  container.

* An arbitrary snippet of shell code from the build specification.

Actions implemented by the contractor or by well-known tools are
considered safe and secure. They will be able to access the network,
the host file system. Arbitrary shell snippets, however, are
considered unsafe and insecure, and will be security isolated using
[Bubblewrap][] and executed without network access. Further, shell
snippets will only be run in a specially constructed container, not
software installed on the host system. The container's **system tree**
(all the software installed in the container) is constructed from a
build artifact, possibly using the debootstrap program.

However, the build may install any software it wants to the container.
This is necessary so that the build can have exactly the build
environment it wants, with all build tools and build dependencies it
needs.

The controller executes also the trusted, external programs using
bwrap, but with network access, and read-only access to the host's
operating system.


Build step actions
-----------------------------------------------------------------------------

The contractor implements the following actions, which the build
specification can use freely.

[debootstrap]: https://wiki.debian.org/Debootstrap

empty-workspace

:   Create a new, empty workspace. The workspace will be automatically
    deleted after the build has finished, unless the user who invoked
    the contractor indicates otherwise.

create-artifact

:   Create an artifact from some or all files in the workspace.

unpack-artifact

:   Retrieve an artifact and unpack it into the workspace.

create-systree

:   Create a new, empty directory to act as the system tree, and
    retrieve an artifact, and unpack it to the new directory.

debootstrap

:   Run the [debootstrap][] program to install Debian into the
    workspace.

apt

:   Install packages using the apt tool into the container's system
    tree.

shell

:   Execute the given shell snippet.

Table: The actions are implemented as follows

action              where?              network?
-------             -------             ---------
empty-workspace     contractor          no
create-artifact     contractor          no
unpack-artifact     contractor          no
create-systree      contractor          no
debootstrap         bwrap on host       yes
apt                 bwrap on systree    yes
shell               bwrap on systree    no

It is intended that any actions implemented by the contractor itself,
or by invoking well-known, vetted programs on the host, are safe and
secure, and cannot cause a information leaks or privilege escalation.

Build specifications {#buildspec}
=============================================================================

FIXME.





Requirements
=============================================================================

This chapter lists requirements for the Ick contractor. These
requirements are not meant to be automatically verifiable. For
specific, automatically testable acceptance criteria, see the later
[chapter with acceptance tests for Contractor](#acceptance).

Each requirement here is given a unique mnemnoic id for easier
reference in discussions.

##  Secure {#req-secure}

Builds run by the contractor should be secure. The code run in the
build steps should not be able to attack or negatively affect the
host computer.


## IdenticalBuilds {#req-builds}

The contractor should execute a build identically to the Ick
server: unless the code being built intentionally makes a
distinction, the results should be the same, or at least any
differences should not be due to the contractor being different
from the server.


## IdenticalInputs {#req-inputs}

Further, the contractor should read exactly the same build
specfications as the Ick server. This may mean that the
contractor defines a new, better language for build
specifications, which Ick itself will implement later.


## Easy {#req-easy}

The user should be able to use the contractor as just another
command line utility. It must not require setting up daemons,
server processes, servers, or other infrastructure.


## Independent {#req-independent}

The controller should not require, or automatically use, any of
the Ick server components: the controller, the IDP, or the
artifact stores. Any such interactions should be invoked
explicitly by the user (e.g., "fetch this artifact from the
artifact store").



Acceptance criteria for Ick contractor {#acceptance}
=============================================================================

FIXME.