From 41b4db598d78706582192655b7ea65a9046f54ab Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Sun, 10 May 2020 07:51:14 +0300 Subject: Change: combine vmdb2.{md,mdwn}, simplify, clarify, fix --- vmdb2.md | 311 ++++++++++++++++++++++++++++++++++++++++++++----------------- vmdb2.mdwn | 197 --------------------------------------- vmdb2.yaml | 8 +- 3 files changed, 230 insertions(+), 286 deletions(-) delete mode 100644 vmdb2.mdwn diff --git a/vmdb2.md b/vmdb2.md index 14bf1f4..484dc5a 100644 --- a/vmdb2.md +++ b/vmdb2.md @@ -1,129 +1,256 @@ ---- -title: "vmdb2: create Debian disk images" -author: Lars Wirzenius -date: work in progress -bindings: vmdb2.yaml -functions: vmdb2.py -... +[vmdb2]: http://vmdb2.liw.fi/ +[debootstrap]: https://packages.debian.org/unstable/debootstrap +[HTML]: https://vmdb2-manual.liw.fi/ +[PDF]: https://vmdb2-manual.liw.fi/vmdb2.pdf -Introduction -============================================================================= +# Introduction -vmdb2 is a program for producing disk images with Debian installed. -This document is a manual of sorts, and an automated test suite for it. +[vmdb2][] builds disk images with Debian installed. The images can be +used for virtual machines, or can be written to USB flash memory +devices, and hardware computers can be booted off them. -vmdb2 installs Debian onto a disk, or disk image. It is -like the [debootstrap][] tool, except the end result is a disk or disk -image, not a directory tree. vmdb2 takes care of creating -partitions, and filesystems, and allows some more customization than -vmdebootstrap does. +This manual is published as [HTML][] and as [PDF][]. -[vmdb2]: http://vmdb2.liw.fi/ -[debootstrap]: https://packages.debian.org/unstable/debootstrap +## Installation -Specification files -============================================================================= +You can get vmdb2 by getting the source code from git, either +[author's server][] or [gitlab.com][]. -A vmdb2 specification file is a YAML file that looks like this: +[author's server]: http://git.liw.fi/vmdb2/ +[gitlab.com]: https://gitlab.com/larswirzenius/vmdb2 -~~~yaml +You can then run it from the source tree: + +~~~sh +$ sudo /path/to/vmdb2/vmdb2 ... +~~~ + +In Debian 10 ("buster") and its derivatives, you can also install the +vmdb2 package: + +~~~sh +$ apt install vmdb2 +~~~ + +For any other systems, we have no instructions. If you figure it out, +please tell us how. + + +## Why vmdb2 given vmdebootstrap already existed + +vmdb2 is a successor of the vmdebootstrap program, written by the same +author, to fix a number of architectural problems and limitations with +the old program. The new program is not compatible with the old one; +that would've required keeping the problems, as well. + +`vmdebootstrap` was the first attempt by it author to write a tool to +build system images. It turned out to not be well designed. +Specifically, it was not easily extensible to be as flexible as a tool +of this sort should be. + + +## Why vmdb2 given other tools already exist + +The author likes to write tools for himself and had some free time. He +sometimes prefers to write his own tools rather than spend time and +energy evaluating and improving existing tools. He admits this is a +character flaw. + +Also, he felt ashamed of how messy `vmdebootstrap` turned out to be. + +If nobody else likes `vmdb2`, that just means the author had some fun +on his own. + + +# Getting started + +You need to make a *specification file* (in YAML) that tells vmdb2 +what kind of image to build, and how. An example: + +~~~{.yaml .numberLines} steps: - mkimg: "{{ output }}" size: 4G - - - mklabel: gpt + - mklabel: msdos device: "{{ output }}" - - mkpart: primary device: "{{ output }}" start: 0% - end: 1G - tag: efi - - - mkpart: primary - device: "{{ output }}" - start: 1G end: 100% tag: / - - kpartx: "{{ output }}" - - - mkfs: vfat - partition: efi - - mkfs: ext4 partition: / - - mount: / - - - unpack-rootfs: / - - debootstrap: buster mirror: http://deb.debian.org/debian target: / - unless: rootfs_unpacked - - apt: install packages: - - linux-image-amd64 - fs-tag: / - unless: rootfs_unpacked - - - cache-rootfs: / - unless: rootfs_unpacked - + - linux-image-amd64 + tag: / - fstab: / - - - grub: uefi + - grub: bios tag: / - efi: efi ~~~ -The list of steps produces the kind of image that the user wants (or -else an unholy mess). The specification file can easily be shared, and -put under version control. +The source repository of vmdb2 has more examples, which are also +automatically tested, unlike the above one. + +The list of steps builds the kind of image that the user wants. The +specification file can easily be shared, and put under version +control. Every action in a step is provided by a plugin to vmdb2. Each action -(technically, "step runner") is a well-defined task, which may be -parameterised by some of the key/value pairs in the step. For example, -`mkimg` would create a disk image file. In the above example it is a -raw disk image file, as opposed to some other format. The image is 4 -gigabytes in size. `mkfs` creates an ext4 filesystem in the image -file; in thie example there are no partitions. And so on. +is a well-defined task, which may be parameterised by some of the +key/value pairs in the step. For example, `mkimg` would create a raw +disk image file. The image is 4 gigabytes in size. `mkpart` creates a +partition, and `mkfs` an ext4 filesystem in the partition. And so on. Steps may need to clean up after themselves. For example, a step that mounts a filesystem will need to unmount it at the end of the image creation. Also, if a later step fails, then the unmount needs to -happen as well. This is called a "teardown". Some steps are provided -by a plugin that handles the teardown automatically, others may need -to provide instructions for the teardown in the specification file. +happen as well. This is called a "teardown". By providing well-defined steps that the user may combine as they wish, vmdb2 gives great flexibility without much complexity, but at the cost of forcing the user to write a longer specification file than -a simple command line invocation. vmdb2 is based on an earlier -program, vmdebootstrap, that took the other approach, and it proved to -be untenable. +a simple command line invocation. + +To use this, save the specification into `test.vmdb`, and run the +following command: +~~~sh +$ sudo vmdb2 test.vmdb --output test.img --verbose +~~~ -A happy path -============================================================================= +Alternatively the specification can be passed in via stdin by setting the +file name to `-`, like so: + +~~~sh +$ cat test.vmdb | sudo vmdb2 - --output test.img --verbose +~~~ + +This will take a long time, mostly at the `debootstrap` step. See +below for speeding that up by caching the result. + +Due to the kinds of things vmdb2 does (such as mounting, creating +device nodes, etc), it needs to be run using root privileges. For the +same reason, it probably can't be run in an unprivileged container. + + +## All images must be partitioned + +At this time, vmdb2 does not support building partitioned images +without partition, or images without a partition table. Such support +may be added later. If this would be useful, do tell the authors. + + +## Tags + +Instead of device filenames, which vary from run to run, vmdb2 steps +refer to block devices inside the image, and their mount points, by +symbolic names called tags. Tags are any names that the user likes, +and vmdb2 does not assign meaning to them. They're just strings. + + +## Jinja2 expansion + +To refer to the filename specified with the `--output` or `--image` +command line options, you can use [Jinja2](http://jinja.pocoo.org/) +templating. The variables `output` and `image` can be used. + +~~~yaml +- mkimg: "{{ output }}" +- mklabel: "{{ image }}" +~~~ + +The difference is that `--output` creates a new file, or truncates an +existing file, whereas `--images` requires the file to already exist. +The former is better for image file, the latter for real block +devices. + + +## Speed up image creation by caching + +Building an image can take several minutes, and that's with fast +access to a Debian mirror and an SSD. The slowest part is typically +running debootstrap, and that always results in the same output, for a +given Debian release. This means its easy to cache. + +vmdb2 has the two actions `cache-roots` and `unpack-rootfs` and the +command line option `--rootfs-tarball` to allow user to cache. The +user uses the option to name a file. `cache-rootfs` takes the root +filesystem and stores it into the file as a compress tar archive +("tarball"). `unpack-rootfs` unpacks the tarball. This allows vmdb2 to +skip running debootstrap needlessly. + +The specify which steps should be skipped, the `unless` field can be +used: `unpack-rootfs` sets the `rootfs_unpacked` flag if it actually +unpacks a tarball, and `unless` allows checking for that flag. If the +tarball doesn't exist, the flag is not set. + +~~~yaml +- unpack-rootfs: root + +- debootstrap: buster + target: root + unless: rootfs_unpacked + +- cache-rootfs: root + unless: rootfs_unpacked +~~~ + +If the tarball exists, it is unpacked, and the `debootstrap` and +`cache-rootfs` steps are skipped. If the tarball doesn't exist, the +unpack step is silently skipped, and the debootstrap and caching steps +are performed instead. + +It's possible to have any number of steps between the unpack and the +cache steps. However, note that if you change anything within those +steps, or time passes and you want to include the new packages that +have made it into Debian, you need to delete the tarball so it is run +again. + + +# Acceptance criteria + +[Subplot]: https://subplot.liw.fi/ + +This chapter documents the user-level acceptance criteria for vmdb2, +and how they are to be verified. It's meant to be processed with the +[Subplot][] tool, but understood by all users of and contributors to +the vmdb2 software. The criteria and their verification are expressed +as *scenarios*. + +For reasons of speed, security, and reliability, these scenarios test +only the core functionality of vmdb2. All the useful steps for +actually building images are left out.. Those are tested by actually +building images. However, those useful steps are not useful, if the +core that invokes them is rotten. + + +## A happy path The first case we look at is one for the happy path: a specification with two echo steps, and nothing else. It's very simple, and nothing goes wrong when executing it. In addition to the actual thing to do, -each step may also define a "teardown" thing to do. For example, if -the step mounts a filesystem, the teardown would unmount it. +each step also defines a "teardown" thing to do. We check that all the +steps and teardown steps are performed, in the right order. + +Note that the "echo" step is provided by vmdb2 explicitly for this +kind of testing, and that the teardown field in the step is +implemented by the echo step. It's not a generic feature. ~~~scenario given a specification file called happy.vmdb when user runs vmdb2 -v happy.vmdb --output=happy.img then exit code is 0 -then stdout contains "foo" followed by "bar" -then stdout contains "bar" followed by "bar_teardown" -then stdout contains "bar_teardown" followed by "foo_teardown" +and stdout contains "foo" followed by "bar" +and stdout contains "bar" followed by "bar_teardown" +and stdout contains "bar_teardown" followed by "foo_teardown" ~~~ ~~~{.file #happy.vmdb .yaml .numberLines} @@ -135,8 +262,7 @@ steps: ~~~ -Jinja2 templating in specification file values -============================================================================= +## Jinja2 templating in specification file values vmdb2 allows values in specification files to be processed by the Jinja2 templating engine. This allows users to do thing such as write @@ -149,7 +275,7 @@ specification file would instead do thing like create the file. given a specification file called j2.vmdb when user runs vmdb2 -v j2.vmdb --output=foo.img then exit code is 0 -then stdout contains "image is foo.img" followed by "bar" +and stdout contains "image is foo.img" followed by "bar" ~~~ ~~~{.file #j2.vmdb .yaml .numberLines} @@ -159,21 +285,21 @@ steps: ~~~ -Error handling -============================================================================= +## Error handling -Sometimes things do not quite go as they should. What does vmdb2 do -then? +Sometimes things do not quite go as they should. Does vmdb2 do things +in the right order then? This scenario uses the "error" step provided +for testing this kind of thing. ~~~scenario given a specification file called unhappy.vmdb when user runs vmdb2 -v unhappy.vmdb --output=unhappy.img then exit code is 1 -then stdout contains "foo" followed by "yikes" -then stdout contains "yikes" followed by "WAT?!" -then stdout contains "WAT?!" followed by "foo_teardown" -then stdout does NOT contain "bar_step" -then stdout does NOT contain "bar_teardown" +and stdout contains "foo" followed by "yikes" +and stdout contains "yikes" followed by "WAT?!" +and stdout contains "WAT?!" followed by "foo_teardown" +and stdout does NOT contain "bar_step" +and stdout does NOT contain "bar_teardown" ~~~ ~~~{.file #unhappy.vmdb .yaml .numberLines} @@ -185,3 +311,16 @@ steps: - echo: bar teardown: bar_teardown ~~~ + + +# Step reference manual + + +--- +title: Building Debian system images with vmdb2 +author: Lars Wirzenius +bindings: vmdb2.yaml +functions: vmdb2.py +documentclass: report +... + diff --git a/vmdb2.mdwn b/vmdb2.mdwn deleted file mode 100644 index 4573605..0000000 --- a/vmdb2.mdwn +++ /dev/null @@ -1,197 +0,0 @@ ---- -title: Building Debian system images with vmdb2 -author: Lars Wirzenius -date: work-in-progress -... - -Introduction -============================================================================= - -vmdb2 builds disk images with Debian installed. The images can be used -for virtual machines, or can be written to USB flash memory devices, -and hardware computers can be booted off them. It is a successor of -the vmdebootstrap program, written by the same author, to fix a number -of architectural problems with the old program. The new program is not -compatible with the old one; that would've required keeping the -problems, as well. - -This manual is published as HTML at - and as a PDF at -. - -Why vmdb2 given vmdebootstrap already existed ------------------------------------------------------------------------------ - -`vmdebootstrap` was the first attempt by Lars Wirzenius to write a -tool to build system images. It turned out to not be well designed. -Specifically, it was not easily extensible to be as flexible as a tool -of this sort should be. - -Why vmdb2 given other tools already exist ------------------------------------------------------------------------------ - -Lars likes to write tools for himself and had some free time. He -sometimes prefers to write his own tools rather than spend time and -energy evaluating and improving existing tools. He admits this is a -character flaw. - -Also, he felt ashamed of how messy `vmdebootstrap` turned out to be. - -If nobody else likes `vmdb2`, that just means Lars had some fun on his -own. - - -Installation -============================================================================= - -You can get vmdb2 by getting the source code from git: - - git clone git://git.liw.fi/vmdb2 - -You can then run it from the source tree: - - sudo /path/to/vmdb2/vmdb2 ... - -In Debian 10 ("buster") and its derivatives, you can also install the -vmdb2 package: - - apt install vmdb2 - -For any other systems, we have no instructions. If you figure it out, -please tell us how. - - -Getting started -============================================================================= - -vmdb2 works by reading specification file with instructions for how an -image should be built, using YAML syntax, and following those -instructions. A minimal specification file example: - - steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: gpt - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - tag: root - - - kpartx: "{{ output }}" - - - mkfs: ext4 - partition: root - - - mount: root - - - debootstrap: buster - mirror: http://deb.debian.org/debian - target: root - - - apt: install - packages: - - linux-image-amd64 - tag: root - - - grub: bios - tag: root - console: serial - -The above creates a four gigabyte file, creates a GPT partition table, -a single partition, with an ext4 filesystem, and installs Debian release -buster onto it. It also installs a kernel, and a boot loader. - -To use this, save the specification into `test.vmdb`, and run the -following command: - - sudo vmdb2 --output test.img --verbose test.vmdb - -Alternatively the specification can be passed in via stdin by setting the -file name to `-`, lke so: - - cat test.vmdb | sudo vmdb2 --output test.img --verbose - - -This will take a long time, mostly at the `debootstrap` step. - - -Unpartitioned images ------------------------------------------------------------------------------ - -At this time, vmdb2 does not support building unpartitioned images, or -images without a partition table. Such support may be added later. If -this would be useful, do tell the authors. - - -Tags ------------------------------------------------------------------------------ - -Instead of device filenames, vmdb2 steps refer to block devices inside -the image, and their mount points, by symbolic names called tags. Tags -are any names that the user likes, and vmdb2 does not assign meaning -to them. They're just strings. - - -Jinja2 expansion ------------------------------------------------------------------------------ - -To refer to the filename specified with the `--output` or `--image` -command line options, you can use [Jinja2](http://jinja.pocoo.org/) -templating. The variables `output` and `image` can be used. - - - mkimg: "{{ output }}" - - - mklabel: "{{ image }}" - -The difference is that `--output` creates a new file, or truncates an -existing file, whereas `--images` requires the file to already exist. -The former is better for image file, the latter for real block -devices. - - -Speed up image creation by caching the root filesystem ------------------------------------------------------------------------------ - -Building an image can take several minutes, and that's with fast -access to a Debian mirror and an SSD. The slowest part is typically -running debootstrap, and that always results in the same output, for a -given Debian release. This means its easy to cache. - -vmdb2 has the two actions `cache-roots` and `unpack-rootfs` and the -command line option `--rootfs-tarball` to allow user to cache. The -user uses the option to name a file. `cache-rootfs` takes the root -filesystem and stores it into the file as a compress tar archive -("tarball"). `unpack-rootfs` unpacks the tarball. This allows vmdb2 to -skip running debootstrap needlessly. - -The specify which steps should be skipped, the `unless` field can be -used: `unpack-rootfs` sets the `rootfs_unpacked` flag if it actually -unpacks a tarball, and `unless` allows checking for that flag. If the -tarball doesn't exist, the flag is not set. - - - unpack-rootfs: root - - - debootstrap: buster - target: root - unless: rootfs_unpacked - - - cache-rootfs: root - unless: rootfs_unpacked - -If the tarball exists, it is unpacked, and the `debootstrap` and -`cache-rootfs` steps are skipped. - -It's possible to have any number of steps between the unpack and the -cache steps. However, note that if you change anything within those -steps, or time passes and you want to include the new packages that -have made it into Debian, you need to delete the tarball so it is run -again. - -TODO: unless, caching, tags, jinja2 - -Step reference manual -============================================================================= - diff --git a/vmdb2.yaml b/vmdb2.yaml index 1a66759..0d0849d 100644 --- a/vmdb2.yaml +++ b/vmdb2.yaml @@ -1,14 +1,16 @@ -- given: a specification file called (?P\S+) +- given: a specification file called {filename} function: given_file -- when: user runs vmdb2 -v (?P\S+) --output=(?P\S+) +- when: user runs vmdb2 -v {filename} --output={output} function: run_vmdb2 -- then: exit code is (?P\d+) +- then: exit code is {exit_code} function: exit_code_is - then: stdout contains "(?P.+)" followed by "(?P.+)" + regex: true function: stdout_contains - then: stdout does NOT contain "(?P.+)" + regex: true function: stdout_does_not_contain -- cgit v1.2.1