path: root/doc/050-debian.yarn

# Building Debian packages: a discussion

Ick projects may be marked as type `debian`, which means that instead
of running arbitrary shell commands, Ick runs a built-in sequence of
commands to build Debian source and binary packages. This sequence, or
pipeline, is meant to be similar to what any free software project
would do if they build Debian packages themselves. However, given the
many variations in how projects are run, this pipeline is not
guaranteed to work for more than the simple projects the author
develops.

## Assumptions

The Debian pipeline in Ick only supports non-native packages using the
"3.0 (quilt)" source package format.

## The assumed workflow

Ick is being written for its author, and to support its author's
development workflow. This section outlines that workflow. The author
may change the workflow in the future, but to do that, a working CI
system is a necessity, so it won't happen until Ick works for him.

* The author is both upstream and Debian packager of several pieces
  of software. Some of the packages get uploaded to Debian. All of
  them get uploaded to the author's own APT repository.

* The upstream code and the Debian packaging code are kept in the same
  branch. They are maintained more or less in sync. The author is
  currently happy with having the upstream tarball contain the
  `debian` directory in the source tree.

* The author writes programs that are simple to build and package, and
  do not need a `make dist` type of operation for release tarball
  generation. The files to go into the release tarball are kept in git
  as is, and none of them are generated files.

## Pipeline overview

The build process starts from a git commit with all the source files
are to be built.

1. A release tar archive is built by exporting the files from git.
2. A Debian source package is built (`.dsc`, `.orig.tar.xz`,
   `.debian.tar.xz`).
3. Debian binary packages (`.deb`) are built for each target (CPU
   architecture, Debian release).
4. The source and binary packages are uploaded to an APT repository
   specific for the ick

The rest of this chapter covers things in more detail.

Note that this pipeline does not cover running unit tests, or other
build-time tests. The Debian packaging for the software being built
may or may not run such tests (it probably should), but this pipeline
does not run such tests outside of the Debian package build.

## Differences from Debian

Debian builds each version of software once, and copies the built
packages to different stages of their releases: uploads to go
"unstable", and are copied first to "testing" and then to "stable".
This means that the same exact build gets used at each stage. This
works fine for Debian, but it's not appropriate for outsiders.
Outsiders usually want to release each version of their software for
several Debian releases, and that requires building it separately for
each.

As an example, the Ick author also develops Obnam, and when making a
release, wants users of the current Debian stable release to be able
to use that version of Obnam. In addition, he wants to support the
previous Debian release ("oldstable") and the unstable version.
Further, he wants to do this not just for releases, but for continuous
integration, so he learns as soon as possible if he make changes that
might, for example, not work in the Debian oldstable release.

Apart from that, the build tools and policy compliance should match
those of Debian.

## CI versus release builds

The build process for CI builds versus release builds should be
identical, except for the choice of version numbers. A CI build should
happen in exactly the same way as a release build. Otherwise there's a
chance that the release build won't work. There's also no need for
them to be different.

However, it's important that CI and release builds to not mix. It's
important that users don't use a CI build by mistake, or that a
release build has a CI build in its build environment.

## Version numbers

The pipeline needs to know the version number of the software. This is
what Debian calls the upstream version. In addition, the pipeline
needs to know the Debian packaging version, and it will need to amend
the full Debian version automatically builds.

A CI build should probably not assume that the upstream version or the
Debian packaging version gets maintained manually in the git
repository. It is just a fiddly detail that is easy to forget. Release
builds must, of course, be able to rely on that version.

For example, the master branch (used for CI builds) might have the
upstream version as that of the previous release, assuming that
releases are prepared in the master branch and tagged there.
Alternatively, the release might be prepared in a different branch,
and master has no version number specified.

FIXME: What's the best way? I only need one for my own projects, and
everything else can be handled later.

## The build environments

The target environments must be clean. We'll use the **pbuilder** tool
to do the actual builds, which provides clean environments, assuming
the separation of CI and release builds. However, maintaining pbuilder
"basetgz" tar archives is outside the scope of Ick.

## The git repository

The source code is extracted from a git repository. The build pipeline
starts from a commit, which may be indicated with a commit id, a tag.
or as the tip of a branch.

For release builds, the commit should be indicated using a signed,
annotated tag. The tag is created by the developer or release manager,
not by the build software. Indeed, creating a suitably named tag
should be what triggers a release build.

For CI builds, the tip of a branch is the way to go.

## Git branches and Debian packaging files

FIXME: It is unclear to me whether it's best for me and in general to
keep the Debian packaging in one branch (master) or separate branches.
If the latter, things may need to be merged (manually or
automatically), which is riskier and/or more work. But it may be ugly
to litter master with packating for one distro (but it's what I do
now). Needs thinking.

## Release tar archive

The first step of the build pipeline is to create a release tar
archive. It is the opinion of the author that despite current fashion
release tar archives are still necessary. For many reasons it is
important to be able to archive the exact source code that is used for
a build, and a tar archive is a good way to do that. A signed git tag
in a git repository is much more complicated to manage, and requires
more special tooling to trust. A single tar archive per release is
very easy to keep track of and distribute.

The tar archive is done for CI builds as well as for releases.

FIXME: Debian version is not -1 or other reason to not re-gen
orig.tar.xz. Get orig.tar.xz from APT repo instead of generating it?

## Debian source package

The Debian source package is created separately from building the
binary packages. The same source package is used for all building of
binaries of the same version.

A Debian source package consists of several files. When using the "3.0
(quilt)" source package format, the files are:

* `orig.tar.xz` --- the release tar archive ("the upstream tarball").
* `debian.tar.xz` --- the Debian packaging changes to be added to the
  release files (usually the `debian/*` files, but may include changes
  to upstream files).
* `.dsc` --- a description of the source files, in particular naming
  the other files and showing their checksums.

The source package is built on the first target specified in the ick
file.

## Debian binary packages

Given a Debian source package, the binary packages are built
separately for each target. This requires copying the source package
to the target, and running the build there.

Binary packages need to fiddle with version numbers, or they will all
build files with the same name. Given a simple source package
`foo_1.2-1.dsc`, all the binary packages for amd64 would be called
`foo_1.2-1_amd64.deb`, even if they're build for different Debian
releases. To avoid such collisions, the binary builds append the
Debian release version to the version: `foo_1.2-1.debian7_amd64.deb`
for the "wheezy" release, versus `foo_1.2-1.debian8_amd64.deb` for
jessie. (FIXME: where do we get the "debian7" from?)

The builds for the Debian "unstable" release do not append anything to
the version number. This is so that the author can upload those
versions to Debian, if he chooses.

Debian binary packages can be architecture independent. For any given
Debian release, such packages should be built only once. That one
build produces a package that can be used everywhere. Building it
elsewhere would be wasteful, and also result in package filename
collisions. When setting up this build pipeline, we choose a target
for each Debian release for building binary independent packages.

The build of a Debian binary package is thus:

1. Unpack the source package.
2. If not building for Debian unstable, add a `debian/changelog` entry
   with a new version number with the Debian release appended. The
   changelog entry should say something like "Build for $target".
3. If on the chosen target for architecture independent packages for a
   given Debian release, build all binary packages. Otherwise, build
   only architecture specific binary packages.

## Uploading to an APT repository

Ick sets up an APT repository for each state directory. Once the
source package and all binary packages are built, they are uploaded to
the APT repository, but not anywhere else. Ick manages the repository
itself, automatically.

FIXME: The repository has separate pockets for CI and release builds:
`unstable` is for releases, `unstable-ci` is for CI builds. CI builds
will use both pockets, but release builds only the release pocket.

FIXME: Further builds should use that APT repository.

FIXME: signing packages?