Publish log entry

1 files changed, 154 insertions, 0 deletions

diff --git a/blog/2017/12/28/design_for_building_systrees_and_using_them_for_containers.mdwn b/blog/2017/12/28/design_for_building_systrees_and_using_them_for_containers.mdwn
new file mode 100644
index 0000000..6e27bc8
--- /dev/null
+++ b/blog/2017/12/28/design_for_building_systrees_and_using_them_for_containers.mdwn
@@ -0,0 +1,154 @@
+[[!meta title="Design for building systrees and using them for containers"]]
+[[!tag software-design]]
+[[!meta date="2017-12-28 21:46"]]
+
+I've written before that I want Ick to do builds in an environment
+isolated both from the host and, optionally, from the network.
+
+* Isolation from host makes it easier to install any build
+  dependencies needed to build a project. They can be installed inside
+  a container. Installing things directly on the build host (worker)
+  is risky, and may result in a non-functional build host. Also, if
+  each project can install whatever it wants on a build host,
+  different projects may keep altering the build environment in ways
+  that make other projects fail to build. They will certainly not
+  built reproducibly, if there's no guarantee about what's in the
+  build environment.
+
+* Isolation from the network means that builds can't depend on things
+  outside the build environment. This is good for the safety and
+  security of the build. (Some builds inherently need to access the
+  network, of course, so that needs to be possible.)
+
+I've decided to use containers, as provided by **systemd-nspawn**, for
+the isolation, at least for now. This is not set in stone, and can be
+re-evaluated after ALPHA-1 is out. nspawn acts more or less like the
+chroot command, except it provides better isolation. It's a very easy
+interface to running thing in isolation.
+
+I will add the concept of **systree** (system tree) to Ick. A systree
+is a directory tree in which all the software (operating system,
+applications, libraries, etc) that are needed for a build are
+installed. This directory will form the root directory inside the
+container.
+
+In addition, the container will bind-mount a **workspace** directory
+from the host system, as `/workspace` inside the container. The
+workspace starts out empty, but the pipeline may fill it with stuff.
+Typically it will contain the source code of a project, but it can be
+anything. In this case, it is where the systree gets built.
+
+Ick will allow a pipeline action to be executed in one of three ways:
+on the build host directly (`where: host`), in a chroot using the
+workspace (`where: chroot`), in a container (`where: container`). In
+the latter case Ick will construct the systree, create an empty
+directory as the workspace, and execute the action inside a container
+using the systree as the root directory, and the workspace as
+`/workspace`.
+
+The systree for a container is constructed by running `debootstrap` on
+the host, to fill the workspace, and then **archiving** the workspace.
+The archive will be stored somewhere else. I've added the **blob
+service** component for this. When it's needed, Ick (worker-manager)
+will get it from the blob service, and unpack it into the systree
+directory. Voila! 
+
+The archive can, of course, be re-used any number of times, on any
+number of worker hosts.
+
+I will further make Ick allow the user to specify how the systrees get
+constructed. The user will specify a project to build a systree:
+
+    project: systree_stretch
+    parameters:
+      debian_codename: stretch
+      packages:
+        - git
+        - ikiwiki
+        - rsync
+        - python3
+      systree_name: stretch-ikiwiki
+    pipelines:
+      - build_systree
+
+In the above example, the project defines three parameters
+(`debian_codename`, `packages`, and `systree_name`), and a pipeline.
+The pipeline looks like this:
+
+    name: build_systree
+    parameters:
+      - debian_codename
+      - packages
+      - systree_name
+    actions:
+      - debootstrap: auto
+        where: host
+      - python:
+          import subprocess
+          packages = params['packages']
+          subprocess.check_call(['apt', 'install', '-y'] + packages)
+        where: chroot
+      - archive: /workspace
+        where: host
+
+This pipeline takes the same three parameters, and runs three actions.
+The `debootstrap` action is run on the host, and uses the
+`debootstrap` program to install Debian into a directory. The action
+will instruct the program to install into the workspace directory. The
+`auto` parameter means the action will get the name of the Debian
+release from the `debian_codename` parameter.
+
+The `python` action then installs additional packages into the
+workspace. It runs in a chroot, not container, as there isn't a
+systree yet in which to run. Running in a chroot means apt will
+install packages in the chroot, not the build host.
+
+The `archive` action creates a tarball of the workspace, and uploads
+it to the blob service. The action takes the name of the blob from the
+`systree_name` parameter.
+
+After this, we have a systree blob in the blob service. We next use it
+to run a build.
+
+    project: ick.liw.fi
+    parameters:
+      git_url: git://git.liw.fi/ick.liw.fi.git
+      git_ref: master
+      rsync_target: www-data@ick.liw.fi:/srv/http/ick.liw.fi
+    pipelines:
+      - ikiwiki_website
+
+This example again defines some parameters, and then specifies a
+pipeline. There's nothing about systrees or containers here yet. That
+goes into the pipeline.
+
+    name: ikiwiki_website
+    parameters:
+      - git_url
+      - git_ref
+      - rsync_target
+    systree: stretch-ikiwiki
+    actions:
+      - shell: |
+          url="$(params | jq -r .git_url)"
+          ref="$(params | jq -r .git_ref)"
+          tgt="$(params | jq -r .rsync_target)"
+          git clone "$url" -b "$ref" src
+          mkdir html
+          cd src
+          ikiwiki --setup ikiwiki.setup --verbose
+          cd ../html
+          rsync -av --delete . "$tgt/."
+        where: container
+
+This specifies that the `stretch-ikiwiki` blob is to be retrieved from
+the blob service, and unpacked as a systree. Then the shell code is to
+be run inside a container using the unpacked systree.
+
+Voila! We have a way to construct systrees and use them to run other
+things in containers.
+
+Compared to my current CI system, this means I don't need to keep
+installing build dependecies of my various personal projects to the CI
+build hosts, which is a small relief.
+