Add chapter on Debian package building

2 files changed, 339 insertions, 0 deletions

diff --git a/doc/050-debian.yarn b/doc/050-debian.yarn
new file mode 100644
index 0000000..878c650
--- /dev/null
+++ b/doc/050-debian.yarn
@@ -0,0 +1,256 @@
+# Building Debian packages
+
+Ick projects may be marked as type `debian`, which means that instead
+of running arbitrary shell commands, Ick runs a built-in sequence of
+commands to build Debian source and binary packages. This sequence, or
+pipeline, is meant to be similar to what any free software project
+would do if they build Debian packages themselves. However, given the
+many variations in how projects are run, this pipeline is not
+guaranteed to work for more than the simple projects the author
+develops.
+
+## Assumptions
+
+The Debian pipeline in Ick only supports non-native packages using the
+"3.0 (quilt)" source package format.
+
+## Pipeline overview
+
+The build process starts from a git commit with all the source files
+are to be built.
+
+1. A release tar archive is built by exporting the files from git.
+2. A Debian source package is built (`.dsc`, `.orig.tar.xz`,
+   `.debian.tar.xz`).
+3. Debian binary packages (`.deb`) are built for each target (CPU
+   architecture, Debian release).
+4. The source and binary packages are uploaded to an APT repository
+   specific for the ick
+
+The rest of this chapter covers things in more detail.
+
+Note that this pipeline does not cover running unit tests, or other
+build-time tests. The Debian packaging for the software being built
+may or may not run such tests (it probably should), but this pipeline
+does not run such tests outside of the Debian package build.
+
+## Differences from Debian
+
+Debian builds each version of software once, and copies the built
+packages to different stages of their releases: uploads to go
+"unstable", and are copied first to "testing" and then to "stable".
+This means that the same exact build gets used at each stage. This
+works fine for Debian, but it's not appropriate for outsiders.
+Outsiders usually want to release each version of their software for
+several Debian releases, and that requires building it separately for
+each.
+
+As an example, the Ick author also develops Obnam, and when making a
+release, wants users of the current Debian stable release to be able
+to use that version of Obnam. In addition, he wants to support the
+previous Debian release ("oldstable") and the unstable version.
+Further, he wants to do this not just for releases, but for continuous
+integration, so he learns as soon as possible if he make changes that
+might, for example, not work in the Debian oldstable release.
+
+Apart from that, the build tools and policy compliance should match
+those of Debian.
+
+## CI versus release builds
+
+The build process for CI builds versus release builds should be
+identical, except for the choice of version numbers. A CI build should
+happen in exactly the same way as a release build. Otherwise there's a
+chance that the release build won't work. There's also no need for
+them to be different.
+
+However, it's important that CI and release builds to not mix. It's
+important that users don't use a CI build by mistake, or that a
+release build has a CI build in its build environment.
+
+## Version numbers
+
+The pipeline needs to know the version number of the software. This is
+what Debian calls the upstream version. In addition, the pipeline
+needs to know the Debian packaging version, and it will need to amend
+the full Debian version automatically builds.
+
+A CI build should probably not assume that the upstream version or the
+Debian packaging version gets maintained manually in the git
+repository. It is just a fiddly detail that is easy to forget. Release
+builds must, of course, be able to rely on that version.
+
+For example, the master branch (used for CI builds) might have the
+upstream version as that of the previous release, assuming that
+releases are prepared in the master branch and tagged there.
+Alternatively, the release might be prepared in a different branch,
+and master has no version number specified.
+
+FIXME: What's the best way? I only need one for my own projects, and
+everything else can be handled later.
+
+## The build environments
+
+The target environments must be clean. We'll use the **pbuilder** tool
+to do the actual builds, which provides clean environments, assuming
+the separation of CI and release builds. However, maintaining pbuilder
+"basetgz" tar archives is outside the scope of Ick.
+
+## The git repository
+
+The source code is extracted from a git repository. The build pipeline
+starts from a commit, which may be indicated with a commit id, a tag.
+or as the tip of a branch.
+
+For release builds, the commit should be indicated using a signed,
+annotated tag. The tag is created by the developer or release manager,
+not by the build software. Indeed, creating a suitably named tag
+should be what triggers a release build.
+
+For CI builds, the tip of a branch is the way to go.
+
+## Git branches and Debian packaging files
+
+FIXME: It is unclear to me whether it's best for me and in general to
+keep the Debian packaging in one branch (master) or separate branches.
+If the latter, things may need to be merged (manually or
+automatically), which is riskier and/or more work. But it may be ugly
+to litter master with packating for one distro (but it's what I do
+now). Needs thinking.
+
+## Release tar archive
+
+The first step of the build pipeline is to create a release tar
+archive. It is the opinion of the author that despite current fashion
+release tar archives are still necessary. For many reasons it is
+important to be able to archive the exact source code that is used for
+a build, and a tar archive is a good way to do that. A signed git tag
+in a git repository is much more complicated to manage, and requires
+more special tooling to trust. A single tar archive per release is
+very easy to keep track of and distribute.
+
+The tar archive is done for CI builds as well as for releases.
+
+FIXME: Debian version is not -1 or other reason to not re-gen
+orig.tar.xz. Get orig.tar.xz from APT repo instead of generating it?
+
+## Debian source package
+
+The Debian source package is created separately from building the
+binary packages. The same source package is used for all building of
+binaries of the same version.
+
+A Debian source package consists of several files. When using the "3.0
+(quilt)" source package format, the files are:
+
+* `orig.tar.xz` --- the release tar archive ("the upstream tarball").
+* `debian.tar.xz` --- the Debian packaging changes to be added to the
+  release files (usually the `debian/*` files, but may include changes
+  to upstream files).
+* `.dsc` --- a description of the source files, in particular naming
+  the other files and showing their checksums.
+
+The source package is built on the first target specified in the ick
+file.
+
+## Debian binary packages
+
+Given a Debian source package, the binary packages are built
+separately for each target. This requires copying the source package
+to the target, and running the build there.
+
+Binary packages need to fiddle with version numbers, or they will all
+build files with the same name. Given a simple source package
+`foo_1.2-1.dsc`, all the binary packages for amd64 would be called
+`foo_1.2-1_amd64.deb`, even if they're build for different Debian
+releases. To avoid such collisions, the binary builds append the
+Debian release version to the version: `foo_1.2-1.debian7_amd64.deb`
+for the "wheezy" release, versus `foo_1.2-1.debian8_amd64.deb` for
+jessie. (FIXME: where do we get the "debian7" from?)
+
+The builds for the Debian "unstable" release do not append anything to
+the version number. This is so that the author can upload those
+versions to Debian, if he chooses.
+
+Debian binary packages can be architecture independent. For any given
+Debian release, such packages should be built only once. That one
+build produces a package that can be used everywhere. Building it
+elsewhere would be wasteful, and also result in package filename
+collisions. When setting up this build pipeline, we choose a target
+for each Debian release for building binary independent packages.
+
+The build of a Debian binary package is thus:
+
+1. Unpack the source package.
+2. If not building for Debian unstable, add a `debian/changelog` entry
+   with a new version number with the Debian release appended. The
+   changelog entry should say something like "Build for $target".
+3. If on the chosen target for architecture independent packages for a
+   given Debian release, build all binary packages. Otherwise, build
+   only architecture specific binary packages.
+
+## Uploading to an APT repository
+
+Ick sets up an APT repository for each state directory. Once the
+source package and all binary packages are built, they are uploaded to
+the APT repository, but not anywhere else. Ick manages the repository
+itself, automatically.
+
+FIXME: The repository has separate pockets for CI and release builds:
+`unstable` is for releases, `unstable-ci` is for CI builds. CI builds
+will use both pockets, but release builds only the release pocket.
+
+FIXME: Further builds should use that APT repository.
+
+FIXME: signing packages?
+
+
+## The pipeline implementation
+
+The Debian package build pipeline contains a number of steps. To allow
+the steps to be easily combined and reordered, the are not integrated
+deeply into the Ick program, but provided by a separate helper program
+instead, called `cleanly`. This de-couples the pipeline and Ick, and
+allows the pipeline to be used with other CI systems as well.
+
+### Extract project metadata
+
+The first thing we need to do is determine, preferably automatically,
+what the name of the project is and what its version is.
+
+    SCENARIO Debian package build pipeline
+    GIVEN a source code repository for project foo version 1.2-1
+
+    WHEN user runs, in foo, cleanly get-name
+    THEN the output is "foo\n"
+
+    WHEN user runs, in foo, cleanly get-upstream-version
+    THEN the output is "1.2\n"
+
+### Create upstream release tar archive
+
+We create a tar archive by extracing the source code from git. The
+operation needs to know the upstream name and version number.
+
+    WHEN user runs, in foo, cleanly tarball
+    THEN file foo-1.2.tar.xz exists
+
+### Create Debian source package
+
+Next up is the Debian source package. We assume the upstream project
+provides the Debian packaging files in the same branch and that the
+Debian version part is just 1. (FIXME: This will need to be made more
+flexible later.)
+
+    WHEN user runs, in foo, cleanly dsc
+    THEN file foo_1.2.orig.tar.xz exists
+    AND file foo_1.2-1.debian.tar.xz exists
+    AND file foo_1.2-1.dsc exists
+
+### Build a Debian binary package
+
+This is where things get interesting. We build a Debian binary
+package, given a source package.
+
+    WHEN user runs, in foo, cleanly deb
+    THEN file foo_1.2-1_all.deb exists
diff --git a/doc/900-implements.yarn b/doc/900-implements.yarn
index c613860..3fc92c7 100644
--- a/doc/900-implements.yarn
+++ b/doc/900-implements.yarn
@@ -72,3 +72,86 @@ Inspect the captured output of the latest ick file.
 
     IMPLEMENTS THEN ick output doesn't contain "(.+)"
     ! grep "$MATCH_1" "$DATADIR/output"
+
+## Creating a git repository
+
+We need a git repository with some source code. It should have a
+master branch, which includes at least one file. We'll create a README
+file for that. We also add basic Debian packaging, so packages can be
+built. Following the author's habit, the Debian packaging goes into
+the master branch as well.
+
+    IMPLEMENTS GIVEN a source code repository for project (.+) version (.+)
+    gitdir="$DATADIR/$MATCH_1"
+    git init "$gitdir"
+    cd "$gitdir"
+    echo "This is $MATCH_1" > README
+    git add README
+    git commit -m "Add README"
+
+    mkdir debian
+    mkdir debian/source
+    echo '3.0 (quilt)' > debian/source/format
+    echo Public domain > debian/copyright
+    echo 9 > debian/compat
+    dch --create --package "$MATCH_1" --newversion "$MATCH_2" \
+        "Initial version."
+
+    cat << EOF > debian/control
+    Source: $MATCH_1
+    Maintainer: John Doe <john@example.com>
+    Section: python
+    Priority: optional
+    Standards-Version: 3.9.6
+    Build-Depends: debhelper (>= 7.3.8)
+
+    Package: $MATCH_1
+    Architecture: all
+    Depends: \${misc:Depends}
+    Description: this is a test package
+     Test package is this.
+
+    EOF
+
+    printf '#!/usr/bin/make -f\n%%:\n\tdh $@\n' > debian/rules
+    chmod +x debian/rules
+
+    git add debian
+    git commit -m "Add Debian packaging"
+
+
+## Running "cleanly"
+
+We need to run the `cleanly` command in various ways. We invoke it
+from the Ick source tree, and save the standard output and error, for
+later investigation.
+
+    IMPLEMENTS WHEN user runs, in (.+), cleanly (.+)
+    cd "$DATADIR/$MATCH_1"
+    # Note that we do NOT quote the second match. We let the shell
+    # parse it as-is. This is necessary to allow multiple arguments.
+    if ! "$SRCDIR/cleanly" --no-default-config \
+        --test-mode \
+        --log "$DATADIR/cleanly.log" \
+        --results "$DATADIR" $MATCH_2 \
+        > "$DATADIR/cleanly.stdout" \
+        2> "$DATADIR/cleanly.stderr"
+    then
+        cat "$DATADIR/cleanly.stdout"
+        cat "$DATADIR/cleanly.stderr" 1>&2
+        exit 1
+    fi
+
+## Checking "cleanly" output
+
+Check what the output of the latest invocation of `cleanly` was.
+
+    IMPLEMENTS THEN the output is "(.*)"
+    printf "$MATCH_1" | diff - "$DATADIR/cleanly.stdout"
+
+## Checking if a file exists
+
+Does a file exist?
+
+    IMPLEMENTS THEN file (.+) exists
+    test -e "$DATADIR/$MATCH_1"