From 5ef257e55c9ecbf10cdb098c8d5954611fa1ddd8 Mon Sep 17 00:00:00 2001
From: Lars Wirzenius <lwirzenius@wikimedia.org>
Date: Fri, 7 Feb 2020 15:40:59 +0200
Subject: Add: start of a contractor Subplot

---
 worker.md   | 239 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 worker.yaml |   1 +
 2 files changed, 240 insertions(+)
 create mode 100644 worker.md
 create mode 100644 worker.yaml

diff --git a/worker.md b/worker.md
new file mode 100644
index 0000000..2c61105
--- /dev/null
+++ b/worker.md
@@ -0,0 +1,239 @@
+---
+title: "Contractor: a local Ick worker"
+author: Lars Wirzenius
+date: work in progress
+bindings: worker.yaml
+...
+
+
+Introduction
+=============================================================================
+
+A CI system executes a sequence of build steps to build a project,
+usually on a server. A common problem is that to debug a build
+failure, one needs to push changes to the CI system, to add logging or
+other debugging help. This is tedious, and often slow.
+
+For Ick, we want a way to perform all the build steps locally, via the
+command line, and without setting up a server or other infrastructure.
+The aim is to give developers a way to debug, inspect, and tweak
+project build specifications easily, quickly, and without bothering
+other users of CI.
+
+For this to work well, the local build tool must behave as closely as
+possible to the CI server.
+
+We call the command line tool to emulate a build on an Ick worker a
+**contractor**.
+
+
+Design alternatives
+-----------------------------------------------------------------------------
+
+A realtively obvious way of doing this is to perform all builds in a
+Docker container. Docker is common, if not quite ubiquitous, and
+relatively easy to use. However, Docker is not great for security
+isolation, and the public images are not always benign. However,
+Docker being very popular means it should not be disregarded lightly.
+
+Other container implementations exist and could be used instead of
+Docker: systemd-nspawn, Kubernetes, Podman, and more. Ick has, so far,
+been using systemd-nspawn. They also do not necessarily have a great
+story for security isolation, and some may require too much
+infrastructure.
+
+However, the [Bubblewrap][] (bwrap) tool is specifically meant for
+security isolation. It seems like a good basis for Ick, going forward.
+
+[Bubblewrap]: https://github.com/containers/bubblewrap
+
+All the container solutions are Linux specific. This may be a problem
+in the long term. Ick may want to support other operating systems,
+later. However, the server's worker hosts and the contractor may need
+to be implemented differently for different operating systems. For
+now, we only care about Linux.
+
+
+Architecture
+-----------------------------------------------------------------------------
+
+The contractor architecture is as follows.
+
+~~~dot
+digraph "arch" {
+  buildyaml [shape=box, label="Build specification"];
+  artifacts [shape=box];
+  workspace [shape=box, label="Temp workspace"];
+  systree   [shape=box, label="System tree"];
+
+  buildyaml -> contractor;
+  contractor -> bwrap;
+  bwrap -> workspace;
+  bwrap -> systree;
+  artifacts -> contractor;
+  contractor -> artifacts;
+}
+~~~
+
+The contractor reads a build specfication from a file, instead of
+retrieving it from the Ick controller (see
+[indepenence](#req-independent)), and executes all the build steps
+defined therein (see [Build specification](#buildspec). From an
+architectural point of view, a build specification defines where to
+get the source code from, per-project parameters, and a sequence of
+build steps to build the project.
+
+Each build step is one the following:
+
+* An action implemented directly in the contractor.
+
+* An action implemented by executing a well-known, vetted external
+  program.
+
+* An action implemented by executing a program installed into a
+  container.
+
+* An arbitrary snippet of shell code from the build specification.
+
+Actions implemented by the contractor or by well-known tools are
+considered safe and secure. They will be able to access the network,
+the host file system. Arbitrary shell snippets, however, are
+considered unsafe and insecure, and will be security isolated using
+[Bubblewrap][] and executed without network access. Further, shell
+snippets will only be run in a specially constructed container, not
+software installed on the host system. The container's **system tree**
+(all the software installed in the container) is constructed from a
+build artifact, possibly using the debootstrap program.
+
+However, the build may install any software it wants to the container.
+This is necessary so that the build can have exactly the build
+environment it wants, with all build tools and build dependencies it
+needs.
+
+The controller executes also the trusted, external programs using
+bwrap, but with network access, and read-only access to the host's
+operating system.
+
+
+Build step actions
+-----------------------------------------------------------------------------
+
+The contractor implements the following actions, which the build
+specification can use freely.
+
+[debootstrap]: https://wiki.debian.org/Debootstrap
+
+empty-workspace
+
+:   Create a new, empty workspace. The workspace will be automatically
+    deleted after the build has finished, unless the user who invoked
+    the contractor indicates otherwise.
+
+create-artifact
+
+:   Create an artifact from some or all files in the workspace.
+
+unpack-artifact
+
+:   Retrieve an artifact and unpack it into the workspace.
+
+create-systree
+
+:   Create a new, empty directory to act as the system tree, and
+    retrieve an artifact, and unpack it to the new directory.
+
+debootstrap
+
+:   Run the [debootstrap][] program to install Debian into the
+    workspace.
+
+apt
+
+:   Install packages using the apt tool into the container's system
+    tree.
+
+shell
+
+:   Execute the given shell snippet.
+
+Table: The actions are implemented as follows
+
+action              where?              network?
+-------             -------             ---------
+empty-workspace     contractor          no
+create-artifact     contractor          no
+unpack-artifact     contractor          no
+create-systree      contractor          no
+debootstrap         bwrap on host       yes
+apt                 bwrap on systree    yes
+shell               bwrap on systree    no
+
+It is intended that any actions implemented by the contractor itself,
+or by invoking well-known, vetted programs on the host, are safe and
+secure, and cannot cause a information leaks or privilege escalation.
+
+Build specifications {#buildspec}
+=============================================================================
+
+FIXME.
+
+
+
+
+
+Requirements
+=============================================================================
+
+This chapter lists requirements for the Ick contractor. These
+requirements are not meant to be automatically verifiable. For
+specific, automatically testable acceptance criteria, see the later
+[chapter with acceptance tests for Contractor](#acceptance).
+
+Each requirement here is given a unique mnemnoic id for easier
+reference in discussions.
+
+##  Secure {#req-secure}
+
+Builds run by the contractor should be secure. The code run in the
+build steps should not be able to attack or negatively affect the
+host computer.
+
+
+## IdenticalBuilds {#req-builds}
+
+The contractor should execute a build identically to the Ick
+server: unless the code being built intentionally makes a
+distinction, the results should be the same, or at least any
+differences should not be due to the contractor being different
+from the server.
+
+
+## IdenticalInputs {#req-inputs}
+
+Further, the contractor should read exactly the same build
+specfications as the Ick server. This may mean that the
+contractor defines a new, better language for build
+specifications, which Ick itself will implement later.
+
+
+## Easy {#req-easy}
+
+The user should be able to use the contractor as just another
+command line utility. It must not require setting up daemons,
+server processes, servers, or other infrastructure.
+
+
+## Independent {#req-independent}
+
+The controller should not require, or automatically use, any of
+the Ick server components: the controller, the IDP, or the
+artifact stores. Any such interactions should be invoked
+explicitly by the user (e.g., "fetch this artifact from the
+artifact store").
+
+
+
+Acceptance criteria for Ick contractor {#acceptance}
+=============================================================================
+
+FIXME.
diff --git a/worker.yaml b/worker.yaml
new file mode 100644
index 0000000..fe51488
--- /dev/null
+++ b/worker.yaml
@@ -0,0 +1 @@
+[]
-- 
cgit v1.2.1