# Building Debian packages: a discussion Ick projects may be marked as type `debian`, which means that instead of running arbitrary shell commands, Ick runs a built-in sequence of commands to build Debian source and binary packages. This sequence, or pipeline, is meant to be similar to what any free software project would do if they build Debian packages themselves. However, given the many variations in how projects are run, this pipeline is not guaranteed to work for more than the simple projects the author develops. ## Assumptions The Debian pipeline in Ick only supports non-native packages using the "3.0 (quilt)" source package format. ## The assumed workflow Ick is being written for its author, and to support its author's development workflow. This section outlines that workflow. The author may change the workflow in the future, but to do that, a working CI system is a necessity, so it won't happen until Ick works for him. * The author is both upstream and Debian packager of several pieces of software. Some of the packages get uploaded to Debian. All of them get uploaded to the author's own APT repository. * The upstream code and the Debian packaging code are kept in the same branch. They are maintained more or less in sync. The author is currently happy with having the upstream tarball contain the `debian` directory in the source tree. * The author writes programs that are simple to build and package, and do not need a `make dist` type of operation for release tarball generation. The files to go into the release tarball are kept in git as is, and none of them are generated files. ## Pipeline overview The build process starts from a git commit with all the source files are to be built. 1. A release tar archive is built by exporting the files from git. 2. A Debian source package is built (`.dsc`, `.orig.tar.xz`, `.debian.tar.xz`). 3. Debian binary packages (`.deb`) are built for each target (CPU architecture, Debian release). 4. The source and binary packages are uploaded to an APT repository specific for the ick The rest of this chapter covers things in more detail. Note that this pipeline does not cover running unit tests, or other build-time tests. The Debian packaging for the software being built may or may not run such tests (it probably should), but this pipeline does not run such tests outside of the Debian package build. ## Differences from Debian Debian builds each version of software once, and copies the built packages to different stages of their releases: uploads to go "unstable", and are copied first to "testing" and then to "stable". This means that the same exact build gets used at each stage. This works fine for Debian, but it's not appropriate for outsiders. Outsiders usually want to release each version of their software for several Debian releases, and that requires building it separately for each. As an example, the Ick author also develops Obnam, and when making a release, wants users of the current Debian stable release to be able to use that version of Obnam. In addition, he wants to support the previous Debian release ("oldstable") and the unstable version. Further, he wants to do this not just for releases, but for continuous integration, so he learns as soon as possible if he make changes that might, for example, not work in the Debian oldstable release. Apart from that, the build tools and policy compliance should match those of Debian. ## CI versus release builds The build process for CI builds versus release builds should be identical, except for the choice of version numbers. A CI build should happen in exactly the same way as a release build. Otherwise there's a chance that the release build won't work. There's also no need for them to be different. However, it's important that CI and release builds to not mix. It's important that users don't use a CI build by mistake, or that a release build has a CI build in its build environment. ## Version numbers The pipeline needs to know the version number of the software. This is what Debian calls the upstream version. In addition, the pipeline needs to know the Debian packaging version, and it will need to amend the full Debian version automatically builds. A CI build should probably not assume that the upstream version or the Debian packaging version gets maintained manually in the git repository. It is just a fiddly detail that is easy to forget. Release builds must, of course, be able to rely on that version. For example, the master branch (used for CI builds) might have the upstream version as that of the previous release, assuming that releases are prepared in the master branch and tagged there. Alternatively, the release might be prepared in a different branch, and master has no version number specified. FIXME: What's the best way? I only need one for my own projects, and everything else can be handled later. ## The build environments The target environments must be clean. We'll use the **pbuilder** tool to do the actual builds, which provides clean environments, assuming the separation of CI and release builds. However, maintaining pbuilder "basetgz" tar archives is outside the scope of Ick. ## The git repository The source code is extracted from a git repository. The build pipeline starts from a commit, which may be indicated with a commit id, a tag. or as the tip of a branch. For release builds, the commit should be indicated using a signed, annotated tag. The tag is created by the developer or release manager, not by the build software. Indeed, creating a suitably named tag should be what triggers a release build. For CI builds, the tip of a branch is the way to go. ## Git branches and Debian packaging files FIXME: It is unclear to me whether it's best for me and in general to keep the Debian packaging in one branch (master) or separate branches. If the latter, things may need to be merged (manually or automatically), which is riskier and/or more work. But it may be ugly to litter master with packating for one distro (but it's what I do now). Needs thinking. ## Release tar archive The first step of the build pipeline is to create a release tar archive. It is the opinion of the author that despite current fashion release tar archives are still necessary. For many reasons it is important to be able to archive the exact source code that is used for a build, and a tar archive is a good way to do that. A signed git tag in a git repository is much more complicated to manage, and requires more special tooling to trust. A single tar archive per release is very easy to keep track of and distribute. The tar archive is done for CI builds as well as for releases. FIXME: Debian version is not -1 or other reason to not re-gen orig.tar.xz. Get orig.tar.xz from APT repo instead of generating it? ## Debian source package The Debian source package is created separately from building the binary packages. The same source package is used for all building of binaries of the same version. A Debian source package consists of several files. When using the "3.0 (quilt)" source package format, the files are: * `orig.tar.xz` --- the release tar archive ("the upstream tarball"). * `debian.tar.xz` --- the Debian packaging changes to be added to the release files (usually the `debian/*` files, but may include changes to upstream files). * `.dsc` --- a description of the source files, in particular naming the other files and showing their checksums. The source package is built on the first target specified in the ick file. ## Debian binary packages Given a Debian source package, the binary packages are built separately for each target. This requires copying the source package to the target, and running the build there. Binary packages need to fiddle with version numbers, or they will all build files with the same name. Given a simple source package `foo_1.2-1.dsc`, all the binary packages for amd64 would be called `foo_1.2-1_amd64.deb`, even if they're build for different Debian releases. To avoid such collisions, the binary builds append the Debian release version to the version: `foo_1.2-1.debian7_amd64.deb` for the "wheezy" release, versus `foo_1.2-1.debian8_amd64.deb` for jessie. (FIXME: where do we get the "debian7" from?) The builds for the Debian "unstable" release do not append anything to the version number. This is so that the author can upload those versions to Debian, if he chooses. Debian binary packages can be architecture independent. For any given Debian release, such packages should be built only once. That one build produces a package that can be used everywhere. Building it elsewhere would be wasteful, and also result in package filename collisions. When setting up this build pipeline, we choose a target for each Debian release for building binary independent packages. The build of a Debian binary package is thus: 1. Unpack the source package. 2. If not building for Debian unstable, add a `debian/changelog` entry with a new version number with the Debian release appended. The changelog entry should say something like "Build for $target". 3. If on the chosen target for architecture independent packages for a given Debian release, build all binary packages. Otherwise, build only architecture specific binary packages. ## Uploading to an APT repository Ick sets up an APT repository for each state directory. Once the source package and all binary packages are built, they are uploaded to the APT repository, but not anywhere else. Ick manages the repository itself, automatically. FIXME: The repository has separate pockets for CI and release builds: `unstable` is for releases, `unstable-ci` is for CI builds. CI builds will use both pockets, but release builds only the release pocket. FIXME: Further builds should use that APT repository. FIXME: signing packages?