[[!meta title=".ick files"]] Ick files ----------------------------------------------------------------------------- An `.ick` file is the input to the `icktool make-it-so` command. It uses YAML syntax, and has two top-level entries: `projects` and `pipelines`. Each of those is a list, one of projecs to build, and one of pipelines. A project has a name, optionally defines some parameters, and lists one or more pipelines. A pipeline is a re-useable sequence of build actions to achieve a goal, such as checking out code from version control, or building binaries from source code. You can roughly think of a pipeline as a subroutine that the project can call. Each pipeline gets all the project parameters, and can do things based on the parameter values. In the example above, there is one project, called `say_hello`, which defines one parameters, `target`, with the value `world`. It uses one pipeline, `greet`. There is one pipeline, `greet`, which accepts the paremeter `target`, and consists of one action, which consists of running a snippet of shell code on the build host. The snippet extracts the value of the parameter. The snippet uses a pre-defined shell function `params`, which outputs the values of all parameters, and extracts the value of the `target` parameter with the `jq` tool. Actions are implemented and executed by the worker manager, which is an agent running on each build host. The worker manager queries the controller for an action to execute, executes it, and reports any output and exit code back to the controller. The controller keeps track of all projects, pipelines, builds, build output, and build results. Note that `icktool make-it-so` does not touch in any way project or pipeline resources not mentioned in the .ick file. If the controller knows of a project foo, but it is not in the .ick file, the foo project is not modified, or deleted. It remains exactly as it was. Also, projects may reference any pipelines the controller knows about, even if not mentioned in the .ick file. Pipelines: "where" ----------------------------------------------------------------------------- Each action in a pipeline MUST specify where it is run. There are three possibilities: * `host` – directly on the host * `chroot` – in a chroot of the workspace * `container` – in a container using a defined system tree (systree) and workspace bind-mounted at /workspace You should strive to run all actions in containers, since that makes it hardest to mess up the worker host. Actions are run as root in the container, and in a chroot. On the host, they can use `sudo` to gain root access. Pipelines: actions ----------------------------------------------------------------------------- The worker manager defines (at least) the following actions: * `shell: snippet` Run `snippet` using the shell. The cwd is the workspace. The shell function `params` outputs all parameters as JSON. The `jq` tool can be used to extract the value of a specific parameter. * `python: snippet` Run `snippet` using Python 3. The cwd is the workspace. The global variable `params` is a dict with all parameters. * `debootstrap: auto` Run the debootstrap command in the workspace. If the value given to the `debootstrap` key is is `auto`, the Debian dist installed into the workspace is taken from the parameter `debian_codename`. The dist can also be named explicitly as the value. The mirror defaults to `http://deb.debian.org/debian`, but can be given explicitly, if needed. * `archive: workspace` Take the current content of the workspace, put it in a compressed tar archive, and upload it to the artifact store. Get the name of the artifact from the parameter named in the `name_from` field, if given, and the`artifact_name` parameter if not. Optionally archive only parts of the workspace. User can give a list of globs in the `globs` field, and anything that matches any of the shell filename patterns will be included. Default is to archive everything. Example: - archive: workspace name_from: debian_packages globs: - "*_*" This would archive everything at the root of the workspace with an underscore in the name. The artifact will be named using the value of the `debian_packages` project parameter. * `archive: systree` This is identical to `archive: workspace`, except it archives the system tree instead of the workspace. * `action: populate_systree` Get an artifact, from the artifact store, and unpack it as the systree. The artifact is assumed to be a compressed tar archive. The name of the artifact comes from the `systree_name` parameter, or the parameter named in the `name_from` field. If the artifact does not exist, end the build with failure. * `action: populate_workspace` Identical to `action: populate_systree`, except it unpacks to the workspace. The default parameter name is `workspace_name`. If the artifact does not exist, do nothing. * `action: git` Clone the git repository given in `git_url` into the directory named in `git_dir`, checking out the branch/tag/commit named in `git_ref`. If the directory already exists, do a `git remote update` inside it instead. This is intended to do all the networking parts of getting source code from a git server. It should be run with `where: host`, so it has network access and can use the worker's ssh key (for non-public repositories). Further actions, inside a container, can do other operations. * `action: git_mirror` This will be replacing the `git` action. It has not yet been implemented. You can use the `git` action as before, for now. Eventually the `git` action will be removed, after a suitable transition period. Mirror one or more git repositories specified in the `sources` project variable. The parameter is expected to have as its value a list of dicts, each dict containing the following fields: * `name` — name of the repository * `repo` — URL of the repository * `location` — ignored by `git_mirror`, but can be used to specify where the clone'd repository is to be checked out * `ref` — ignored by `git_mirror`, but can be used to specify which ref (branch or tag or commit) should be checked out Additionally, `git_mirror` will use the `git_repo_base` project parameter: if the `repo` field is a relative URL, it will be joined with the `git_repo_base` value to form the full URL. If `repo` is a full URL, it is used as is. Note that `git_mirror` does NOT do the checking out, only the initial mirroring (as if by `git clone --mirror $repo .mirror/$name`) or updating of the mirror (as if by `cd .mirror/$name && git remote update --prune`). Ick provides a pipeline that uses the `git_mirror` action and additional `shell` or `python` actions to do the checkout. * `action: rsync` Copy the content of a directory in the workspace (named in `rsync_src`) to a remote server (named in `rsync_target`. This should be run on the host, to have network access and use of the worker's ssh key. * `action: dput` Upload all Debian .changes files to the APT repository that is part of an ick cluster. No parameters needed. This should be run on the host, to have network access and use of the worker's ssh key. * `action: notify` Use the notification service to send notifications of the current state of the build. You usually don't need to use these actions manually, the controller adds a notification action automatically when a build ends. Notifications ----------------------------------------------------------------------------- Ick can send an email when a build ends. This is done automatically. Set the project parameter `notify` to a list of email addresses that should be notified. Standard pipelines ----------------------------------------------------------------------------- Ick comes with a set of "standard" pipelines, named `ick/get_sources` and the like. They are stored and documented in the ick source code: The standard pipelines are not installed automatically. The ick instance admin needs to install and update them manually. Example .ick files ----------------------------------------------------------------------------- The repository has some .ick files, used by ick's author for his personal instances. General advice ----------------------------------------------------------------------------- On each ick instance, set up one or more projects that build systrees. Build one for each target Debian release: `jessie`, `stretch`, `unstable`, etc. Build each of these projects to populate the artifact store with systree artifacts. Each actual project should use one of the artifacts to do the build in a container. You can use the `ick/build_debian_systree` pipeline to build systrees with Debian. When building in a container, you can install more things inside the container. If you build repeatedly, it may be worth having more than just the minimal container. Instead have an artifact with all the build dependencies installed, so each build does not have to start with installing them, saving time. Ick can build several things at once. As many things as there are workers. However, each build (currently) runs on one worker. There's currently no way to stop a triggered build. This is a missing feature.