Add: some documentation

16 files changed, 614 insertions, 0 deletions

diff --git a/format.sh b/format.sh
new file mode 100755
index 0000000..e199934
--- /dev/null
+++ b/format.sh
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+set -eu
+
+pandoc \
+    --self-contained \
+    --standalone \
+    --css vmdb2.css \
+    --toc \
+    --number-sections \
+    -o vmdb2.html \
+    vmdb2.mdwn vmdb/plugins/*.mdwn
+
+pandoc \
+    --pdf-engine=xelatex \
+    --toc \
+    --number-sections \
+    -Vdocumentclass=report \
+    -Vgeometry:a4paper \
+    -Vfontsize:12pt \
+    -Vmainfont:FreeSerif \
+    -Vsansfont:FreeSans \
+    -Vmonofont:FreeMonoBold \
+    '-Vgeometry:top=2cm, bottom=2.5cm, left=2cm, right=1cm' \
+    -o vmdb2.pdf \
+    vmdb2.mdwn vmdb/plugins/*.mdwn
diff --git a/vmdb/plugins/ansible.mdwn b/vmdb/plugins/ansible.mdwn
new file mode 100644
index 0000000..8929c57
--- /dev/null
+++ b/vmdb/plugins/ansible.mdwn
@@ -0,0 +1,47 @@
+Step: `ansible`
+-----------------------------------------------------------------------------
+
+Run Ansible using a provided playbook, to configure the image. vmdb2
+sets up Ansible so that it treats the image as the host being
+configured (via the `chroot` connecion). The image MUST have Python
+installed (version 2 or 3 depending on Ansible version).
+
+Step keys:
+
+* `ansible` — REQUIRED; value is the tag of the root filesystem.
+
+* `playbook` — REQUIRED; value is the filename of the Ansible
+  playbook, relative to the .vmdb file.
+
+Example (in the .vmdb file):
+
+    - apt: install
+      tag: root
+      packages: [python]
+
+    - ansible: root
+      playbook: foo.yml
+
+Example (`foo.yml`):
+
+    - hosts: image
+      tasks:
+
+        - name: "set /etc/hostname"
+          shell: |
+            echo "{{ hostname }}" > /etc/hostname
+
+        - name: "unset root password"
+          shell: |
+            sed -i '/^root:[^:]*:/s//root::/' /etc/passwd
+
+        - name: "configure networking"
+          copy:
+            content: |
+              auto eth0
+              iface eth0 inet dhcp
+              iface eth0 inet6 auto
+            dest: /etc/network/interfaces.d/wired
+
+      vars:
+        hostname: discworld
diff --git a/vmdb/plugins/apt.mdwn b/vmdb/plugins/apt.mdwn
new file mode 100644
index 0000000..dd83f18
--- /dev/null
+++ b/vmdb/plugins/apt.mdwn
@@ -0,0 +1,21 @@
+Step: `apt`
+-----------------------------------------------------------------------------
+
+Install packages using apt, which needs to already have been
+installed.
+
+Step keys:
+
+* `apt` — REQUIRED; value MUST be `install`.
+
+* `tag` — REQUIRED; value is the tag for the root filesystem.
+
+* `packages` — REQUIRED; value is a list of packages to install.
+
+Example (in the .vmdb file):
+
+    - apt: install
+      tag: root
+      packages:
+      - python
+      - linux-image-amd64
diff --git a/vmdb/plugins/chroot.mdwn b/vmdb/plugins/chroot.mdwn
new file mode 100644
index 0000000..87fd5ec
--- /dev/null
+++ b/vmdb/plugins/chroot.mdwn
@@ -0,0 +1,35 @@
+Step: `chroot`
+-----------------------------------------------------------------------------
+
+Run a shell snippet in a chroot inside the image.
+
+Step keys:
+
+* `chroot` — REQUIRED; value is the tag for the root filesystem.
+
+* `shell` — REQUIRED; the shell snippet to run
+
+Example (in the .vmdb file):
+
+    - chroot: root
+      shell: |
+          echo I am in chroot
+
+
+Step: `shell`
+-----------------------------------------------------------------------------
+
+Run a shell snippet on the host. This is not run in a chroot, and can
+access the host system.
+
+Step keys:
+
+* `root-fs` — REQUIRED; value is the tag for the root filesystem.
+
+* `shell` — REQUIRED; the shell snippet to run
+
+Example (in the .vmdb file):
+
+    - root-fs: root
+      shell: |
+          echo I am in NOT in chroot.
diff --git a/vmdb/plugins/debootstrap.mdwn b/vmdb/plugins/debootstrap.mdwn
new file mode 100644
index 0000000..23a56c9
--- /dev/null
+++ b/vmdb/plugins/debootstrap.mdwn
@@ -0,0 +1,20 @@
+Step: `debootstrap`
+-----------------------------------------------------------------------------
+
+Install packages using apt, which needs to already have been
+installed.
+
+Step keys:
+
+* `debootstrap` — REQUIRED; value is the codename of the Debian
+  release to install: `stretch`, `buster`, etc.
+
+* `target` — REQUIRED; value is the tag for the root filesystem.
+
+* `mirror` — OPTIONAL; which Debian mirror to use
+
+Example (in the .vmdb file):
+
+    - debootstrap: stretch
+      target: root
+      mirror: http://mirror.example.com/debian
diff --git a/vmdb/plugins/grub.mdwn b/vmdb/plugins/grub.mdwn
new file mode 100644
index 0000000..1d69460
--- /dev/null
+++ b/vmdb/plugins/grub.mdwn
@@ -0,0 +1,44 @@
+Step: `grub`
+-----------------------------------------------------------------------------
+
+Install the GRUB bootloader to the image. Works on a PC, for
+traditional BIOS booting or modern UEFI booting. Does not (yet?)
+support Secure Boot.
+
+Warning: This is the least robust part of vmdb2.
+
+Step keys:
+
+* `grub` — REQUIRED; value MUST be one of `uefi` and `bios`, for
+  a UEFI or a BIOS boot, respectively. (FIXME: these are valid for a
+  PC; not sure what other archs require, if grub even works there.)
+
+* `tag` — REQUIRED; value is the tag for the root filesystem.
+
+* `efi` — REQUIRED for UEFI; value is the tag for the EFI
+  filesystem.
+
+* `console` — OPTIONAL; set to `serial` to configure the image
+  to use a serial console.
+
+* `device` — OPTIONAL; which device to install GRUB onto; this
+  is needed when installing to a real hard drive, instead of an image.
+
+Example (in the .vmdb file):
+
+    - grub: bios
+      tag: root
+
+Same, but for UEFI:
+
+    - grub: uefi
+      tag: root
+      efi: efi
+      console: serial
+
+Install to a real hard disk (named with the `--image` option):
+
+    - grub: uefi
+      tag: root
+      efi: efi
+      image-dev: "{{ image }}"
diff --git a/vmdb/plugins/luks.mdwn b/vmdb/plugins/luks.mdwn
new file mode 100644
index 0000000..4b80f63
--- /dev/null
+++ b/vmdb/plugins/luks.mdwn
@@ -0,0 +1,33 @@
+Step: `luks`
+-----------------------------------------------------------------------------
+
+Set up disk encryption using LUKS with the `cryptsetup` utility. The
+encryption passphrase is read from a file or from the output of a
+command. The encrypted disk gets opened and can be mounted using a
+separate tag for the cleartext view.
+
+Step keys:
+
+* `cryptsetup` — REQUIRED; value is the tag for the encrypted
+  block device. This is not directly useable by users, or mountable.
+
+* `tag` — REQUIRED; the tag for the de-crypted block device.
+  This is what gets mounted and visible to users.
+
+* `key-file` — OPTIONAL; file from where passphrase is read.
+
+* `key-cmd` — OPTIONAL; command to run, passphrase is the first
+  line of its standard output.
+
+Example (in the .vmdb file):
+
+    - cryptsetup: root
+      tag: root_crypt
+      key-file: disk.pass
+
+Same, except run a command to get passphrase (in this case
+[pass](https://www.passwordstore.org/)):
+
+    - cryptsetup: root
+      tag: root_crypt
+      key-cmd: pass show disk-encryption
diff --git a/vmdb/plugins/lvm2.mdwn b/vmdb/plugins/lvm2.mdwn
new file mode 100644
index 0000000..76f5892
--- /dev/null
+++ b/vmdb/plugins/lvm2.mdwn
@@ -0,0 +1,42 @@
+Step: `vgcreate`
+-----------------------------------------------------------------------------
+
+Create an LVM2 volume group (VG), and also initialise the physical
+volumes for it.
+
+
+Step keys:
+
+* `vgcreate` — REQUIRED; value is the tag for the volume group.
+  This gets initialised with `vgcreate`.
+
+* `physical` — REQUIRED; list of tags for block devices
+  (partitions) to use as physical volumes. These get initialised with
+  `pvcreate`.
+
+Example (in the .vmdb file):
+
+    - vgcreate: rootvg
+      physical:
+      - my_partition
+      - other_partition
+
+
+Step: `lvcreate`
+-----------------------------------------------------------------------------
+
+Create an LVM2 logical volume (LV) in an existing volume group.
+
+Step keys:
+
+* `lvcreate` — REQUIRED; value is the tag for the volume group.
+
+* `name` — REQUIRED; tag for the new LV block device.
+
+* `size` — REQUIRED; size of the new LV.
+
+Example (in the .vmdb file):
+
+    - lvcreate: rootvg
+      name: rootfs
+      size: 1G
diff --git a/vmdb/plugins/mkfs.mdwn b/vmdb/plugins/mkfs.mdwn
new file mode 100644
index 0000000..87243cd
--- /dev/null
+++ b/vmdb/plugins/mkfs.mdwn
@@ -0,0 +1,15 @@
+Step: `mkfs`
+-----------------------------------------------------------------------------
+
+Create a filesystem.
+
+Step keys:
+
+* `mkfs` — REQUIRED; filesystem type, such as `mkfs` or `vfat`.
+
+* `partition` — REQUIRED; tag for the block device to use.
+
+Example (in the .vmdb file):
+
+    - mkfs: ext4
+      partition: root
diff --git a/vmdb/plugins/mkimg.mdwn b/vmdb/plugins/mkimg.mdwn
new file mode 100644
index 0000000..e2998ac
--- /dev/null
+++ b/vmdb/plugins/mkimg.mdwn
@@ -0,0 +1,15 @@
+Step: `mkimg`
+-----------------------------------------------------------------------------
+
+Create a new image file of a desired size.
+
+Step keys:
+
+* `mkimage` — REQUIRED; name of file to create.
+
+* `size` — REQUIRED; size of the image.
+
+Example (in the .vmdb file):
+
+    - mkimg: "{{ output }}"
+      size: 4G
diff --git a/vmdb/plugins/mount.mdwn b/vmdb/plugins/mount.mdwn
new file mode 100644
index 0000000..cf447f1
--- /dev/null
+++ b/vmdb/plugins/mount.mdwn
@@ -0,0 +1,17 @@
+Step: `mount`
+-----------------------------------------------------------------------------
+
+Mount a filesystem.
+
+Step keys:
+
+* `mount` — REQUIRED; tag of filesystem to mount.
+
+* `dirname` — OPTIONAL; the mount point.
+
+* `mount-on` — OPTIONAL; tag of already mounted filesystem in
+  image. (FIXME: this may be wrong?)
+
+Example (in the .vmdb file):
+
+    - mount: root
diff --git a/vmdb/plugins/partition.mdwn b/vmdb/plugins/partition.mdwn
new file mode 100644
index 0000000..cc325d4
--- /dev/null
+++ b/vmdb/plugins/partition.mdwn
@@ -0,0 +1,57 @@
+Step: `mklabel`
+-----------------------------------------------------------------------------
+
+Create a partition table on a block device.
+
+Step keys:
+
+* `mklabel` — REQUIRED; type of partition table, MUST be one of
+  `msdos` and `gpt`.
+
+* `device` — REQUIRED; tag for the block device.
+
+Example (in the .vmdb file):
+
+    - mklabel: "{{ output }}"
+      size: 4G
+
+Step: `mkpart`
+-----------------------------------------------------------------------------
+
+Create a partition.
+
+Step keys:
+
+* `mkpart` — REQUIRED; type of partition to create: use
+  `primary` (but any value acceped by `parted` is OK).
+
+* `device` — REQUIRED; filename of block device where to create
+  partition.
+
+* `start` — REQUIRED; where does the partition start?
+
+* `end` — REQUIRED; where does the partition end?
+
+* `tag` — REQUIRED; tag for the new partition.
+
+Example (in the .vmdb file):
+
+    - mkpart: primary
+      device: "{{ output }}"
+      start: 0%
+      end: 100%
+      tag: root
+
+Step: `kpartx`
+-----------------------------------------------------------------------------
+
+Create loop devices for partitions in an image file. Not needed when
+installing to a real block device, instead of an image file.
+
+Step keys:
+
+* `kpartx` — REQUIRED; filename of block device with partitions.
+
+Example (in the .vmdb file):
+
+    - kpartx: "{{ output }}"
diff --git a/vmdb/plugins/qemudebootstrap.mdwn b/vmdb/plugins/qemudebootstrap.mdwn
new file mode 100644
index 0000000..5d0234e
--- /dev/null
+++ b/vmdb/plugins/qemudebootstrap.mdwn
@@ -0,0 +1,28 @@
+Step: `qemu-debootstrap`
+-----------------------------------------------------------------------------
+
+Install packages using apt, which needs to already have been
+installed, for a different architecture than the host where vmdb2 is
+being run. For example, for building an image for a Raspberry Pi on an
+Intel PC.
+
+Step keys:
+
+* `qemu-debootstrap` — REQUIRED; value is the codename of the Debian
+  release to install: `stretch`, `buster`, etc.
+
+* `target` — REQUIRED; value is the tag for the root filesystem.
+
+* `mirror` — OPTIONAL; which Debian mirror to use.
+
+* `arch` — REQUIRED; the foreign architecture touse.
+
+* `variant` — OPTIONAL; the variant for debootstrap.
+
+Example (in the .vmdb file):
+
+    - qemu-debootstrap: stretch
+      target: root
+      mirror: http://mirror.example.com/debian
+      arch: arm64
+      variant: buildd
diff --git a/vmdb/plugins/rootfs_cache.mdwn b/vmdb/plugins/rootfs_cache.mdwn
new file mode 100644
index 0000000..3f4ee0b
--- /dev/null
+++ b/vmdb/plugins/rootfs_cache.mdwn
@@ -0,0 +1,29 @@
+Step: `cache-rootfs`
+-----------------------------------------------------------------------------
+
+Create a tarball of the root filesystem in the image.
+
+Step keys:
+
+* `cache-rootfs` — REQUIRED; tag of root filesystem on image.
+
+Example (in the .vmdb file):
+
+    - cache-rootfs: root
+      unless: rootfs_unpacked
+
+
+Step: `unpack-rootfs`
+-----------------------------------------------------------------------------
+
+Unpack a tarball of the root filesystem to the image, and set the
+`rootfs_unpacked` condition to true. If the tarball doesn't exist, do
+nothing and leave the `rootfs_unpacked` condition to false.
+
+Step keys:
+
+* `unpack-rootfs` — REQUIRED; tag for the root filesystem.
+
+Example (in the .vmdb file):
+
+    - unpack-rootfs: root
diff --git a/vmdb2.css b/vmdb2.css
new file mode 100644
index 0000000..7c5c4dd
--- /dev/null
+++ b/vmdb2.css
@@ -0,0 +1,33 @@
+html {
+    font-family: "FreeSerif", serif;
+}
+
+h1, h2, h3, h4, h5, h6 {
+    font-family: "FreeSans", sans;
+}
+
+header h1.title {
+    display: block;
+    text-align: center;
+    font-family: "FreeSans", sans;
+    font-size: 200%;
+}
+
+header h2.author {
+    display: block;
+    text-align: center;
+    font-family: "FreeSans", sans;
+    font-size: 150%;
+}
+
+
+header h3.date {
+    display: block;
+    text-align: center;
+    font-family: "FreeSans", sans;
+    font-size: 150%;
+}
+
+pre {
+    margin-left: 4em;
+}
diff --git a/vmdb2.mdwn b/vmdb2.mdwn
new file mode 100644
index 0000000..11d640e
--- /dev/null
+++ b/vmdb2.mdwn
@@ -0,0 +1,152 @@
+---
+title: vmdb2 manual
+...
+
+Introduction
+=============================================================================
+
+vmdb2 builds disk images with Debian installed. The images can be used
+for virtual machines, or can be written to USB flash memory devices,
+and hardware computers can be booted off them. It is a successor of
+the vmdebootstrap program, written by the same author, to fix a number
+of architectural problems with the old program. The new program is not
+compatible with the old one; that would've required keeping the
+problems, as well.
+
+
+Installation
+-----------------------------------------------------------------------------
+
+You can get vmdb2 by getting the source code from git:
+
+    git clone git://git.liw.fi/vmdb2
+
+You can then run it from the source tree:
+
+    sudo /path/to/vmdb2/vmdb2 ...
+
+In Debian 10 ("buster") and its derivatives, you can also install the
+vmdb2 package:
+
+    apt install vmdb2
+
+For any other systems, we have no instructions. If you figure it out,
+please tell us how.
+
+
+Getting started
+=============================================================================
+
+vmdb2 works by reading specification file with instructions for how an
+image should be built, using YAML syntax, and following those
+instructions. A minimal specification file example:
+
+    steps:
+      - mkimg: "{{ output }}"
+        size: 4G
+
+      - mklabel: gpt
+        device: "{{ output }}"
+
+      - mkpart: primary
+        device: "{{ output }}"
+        start: 0%
+        end: 100%
+        tag: root
+
+      - mkfs: ext4
+        partition: root
+
+      - mount: root
+
+      - debootstrap: stretch
+        mirror: http://deb.debian.org/debian
+        target: root
+
+      - apt: install
+        packages:
+          - linux-image-amd64
+        tag: root-fs
+
+      - grub: bios
+        tag: root
+
+The above creates a four gigabyte file, creates a GPT partition table,
+a single partition, with a filesystem, and installs Debian release
+stretch onto it. It also installs a kernel, and a boot loader.
+
+To use this, save the specification into `test.vmdb`, and run the
+following command:
+
+    sudo vmdb2 test.vmdb --output test.img --verbose
+
+This will take a long time, mostly at the `debootstrap` step.
+
+
+Tags
+-----------------------------------------------------------------------------
+
+Instead of device filenames, vmdb2 steps refer to block devices inside
+the image, and their mount points, by symbolic names called tags. Tags
+are any names that the user likes, and vmdb2 does not assign meaning
+to them. They're just strings.
+
+
+Jinja2 expansion
+-----------------------------------------------------------------------------
+
+To refer to the filename specified with the `--output` or `--image`
+command line options, you can use [Jinja2](http://jinja.pocoo.org/)
+templating. The variables `output` and `image` can be used.
+
+    - mkimg: "{{ output }}"
+
+    - mklabel: "{{ image }}"
+
+The difference is that `--output` creates a new file, or truncates an
+existing file, whereas `--images` requires the file to already exist.
+The former is better for image file, the latter for real block
+devices.
+
+
+Speed up image creasing by caching the root filesystem
+-----------------------------------------------------------------------------
+
+Building an image can take several minutes, and that's with fast
+access to a Debian mirror and an SSD. The slowest part is typically
+running debootstrap, and that always results in the same output, for a
+given Debian release. This means its easy to cache.
+
+vmdb2 has the two actions `cache-roots` and `unpack-rootfs` and the
+command line option `--rootfs-tarball` to allow user to cache. Thhe
+user uses the option to name a file. `cache-rootfs` takes the root
+filesystem and stores it into the file as a compress tar archive
+("tarball"). `unpack-rootfs` unpacks the tarball. This allows vmdb2 to
+skip running debootstrap needlessly.
+
+The specify which steps should be skipped, the `unless` field can be
+used: `unpack-rootfs` sets the `rootfs-unpacked` flag if it actually
+unpacks a tarball, and `unless` allows checking for that flag. If the
+tarball doesn't exist, the flag is not set.
+
+    - unpack-rootfs: root
+
+    - debootstrap: stretch
+      target: root
+      unless: rootfs-unpacked
+
+    - cache-rootfs: root
+      unless: rootfs-unpacked
+
+If the tarball exists, it's unpacked, and the `debootstrap` and
+`cache-rootfs` steps are skipped.
+
+It's possible to have any number of steps between the unpack and the
+cache steps. However, note that if you change the steps, you need to
+delete the tarball to run them.
+
+TODO: unless, caching, tags, jinja2
+
+Step reference manual
+=============================================================================
+