NEWS for vmdb2, the Debian disk image builder ============================================================================= Project is now in "selfish maintenance mode". I make changes when I need them for myself, but I'm not developing new features otherwise. I will review and merge patches, though. Version 0.40, released 2024-03-23 ----------------------------------------------------------------------------- Version number is bumped a lot to avoid some incorrectly tagged versions that Lars is very sorry for. Other than that, this version is mainly a few bug fixes, mostly from other people. * Documentation fix: `README.md` points to `base.vmdb` as an example, rather than `pc.vmdb`, which has long since been deleted. * `ppc64el.vmdb` has been fixed so that it at least builds. It's unknown whether it builds an image that boots. * From Matej Kovac, fixes to `resolv.conf` configuration: don't copy the host's configuration. * From Peter Palfrader, a fix the `cryptsetup` plugin to import the `logging` module. * From Victor Westerhuis, a fix to using the right `grub-ieee1275` depending on the target system. Version 0.28, released 2023-08-02 ----------------------------------------------------------------------------- There's a lot of bug fixes, some new features, and a better test suite. Bug fixes: * Fix problem with updating `/etc/default/grub` with a shorter version. The `grub` plugin now truncates the excess length of the file. Thank you to Tim Small for the fix. * The `grub` and `virtualfs` plugins now mount virtual filesystems only once. The `grub` plugin also mounts `/proc` sufficiently early for to handle a root file system being in a logical volume. The `virtualfs` plugins bind mounts the host's `/dev` for correct handling of LVM images. Thank you to rtkapiper for the change. * The `unpack_rootfs` plugin now handles `/etc/resolv.conf` being a symlink better, by copying that file pointed at by the symlink as well. Thank you to rtkapiper for the change. * The `kpartx` plugin now creates a loop device if needed. Thank you to rtkapiper for the change. * The `mount` plugin now correctly normalizes mount point paths. Thank you to rtkapiper for the change. * The `run_even_if_skipped` feature has returned, because it's actually needed for corner cases of `debootstrap` use. * The `debootstrap` plugin only obeys the `arch` field if it's actually set. Previously, the plugin treated a missing `arch` field as set. * The generated `/etc/fstab` no longer uses options for file systems that are specific for the ext4 file system. * The generated `/etc/fstab` file is now sorted by mount point. This makes `mount -a` work more reliably. Thank you to rtkapiper for the change. * The `--version` option now works. Breaking changes: * The `luks` and `cryptsetup` plugins have been combined. There can be only the `cryptsetup` one. Any vmdb file that uses `luks` is broken with this release. New features: * The `cache_rootfs` plugin now allows forcing overwriting a rootfs tarball. Thank you to rtkapiper for the change. * The `debootstrap` plugin can now optionally install TLS CA certificates to allow repositories behind `https` URLs. Thank you to rtkapiper for the change. * vmdb2 now supports installing Debian onto images with existing partitions and logical volumes. Among other changes, there's a new `lvscan` plugin. Thank you to rtkapiper for the change. * The `ansible` plugin now sets any Ansible variables on the command line invocation of `ansible-playbook`, to make them more visible and debugging easier. Thank you to rtkapiper for the change. * With the `--verbose` option, the stdout and stderr output of commands run by vmdb2 are shown and logged in real time. Thank you to rtkapiper for the change. Other changes: * The argon2id key derivation function is now used for LUKS full disk encryption. * The zerofree utility no longer spams the log file with percentages of how much of the work it has done. * Improved test suite. This is invisible to anyone by vmdb2 developers, but helps vmdb2 be tested for more cases. Version 0.27, released 2023-04-07 ----------------------------------------------------------------------------- * When `vmdb2` creates an ext4 file system, it avoids using the `large_dir` and `metadata_csum_seed` file system features, to avoid incompatibility with versions of `grub2` older than what will be in the Debian 12 (bookworm) release. * The kernel parameters can now be set in the spec file, for the `grub` plugin. Thanks, Tim Small. * The `qemu-debootstrap` plugin has been removed. The plain `debootstrap` plugin can be used instead, even for building an image for a foreign architecture. Thank you to Antonio Terceiro. * The `run_even_if_skipped` feature of the plugin architecture has been dropped, as nothing was using it anymore. * When running command, the `LC_ALL` environment variable is now set to `C.UTF8`, because Ansible insists on the locale it is being used in having UTF8 support. * When `vmdb2` modifies the `/etc/default/grub` file, it now modifies values in place instead of just appending new lines to the end of the file. Thanks, Tim Small. Version 0.26, released 2022-04-20 ----------------------------------------------------------------------------- * Bug fix: the Ansible plugin now correctly deletes the temporary files it creates. Version 0.25, released 2022-02-19 ----------------------------------------------------------------------------- * Florian La Roche fixed the grub plugin to not set systemd.show_status multiple times, and other issues with how vmdb2 generates the kernel command line. * Zygmunt Krynicki changed vmdb2 to detect if it should use the --no-extra-removable option. Debian and Ubuntu versions of grub differ on whether it's needed. * Andy Piper fixed cleaning up mapped partitions in Docker on macOS. * Andy Piper added a "fill" option to the lvcreate plugin. * Andy Piper added support for filling ext2/3/4 file systems with zeroes after they're unmounted. This happens by default, but can be disabled in the mount step. * Andy Piper added the copy_dir plugin to copy directory trees into the build chroot. * Andy Piper added the set_part plugin for setting flags on partition. * Andy Piper fixed the grub plugin to run "apt update" before grub is installed to handle the case where no packages get installed previously in a vmdb2 run. * Andy Piper fixed the grub plugin to work if the root file system is on a logical volume. * Andy Pipe added the --variable option to vmdb2 for setting variables for jinja2 template expansion on .vmdb files. * Andy Piper extended the ansible plugin to make it more configurable. * Andy Piper fixed the debootstrap plugin to not run apt-get update unnecessarily, and to add the include and install_keyring fields, to allow additional packages to be installed. * Lars Wirzenius added the cryptsetup setup. * Sebastian Bachmann fixed bugs related partition naming, when device names end in a number, and redundant removal of loop devices. * Lars Wirzenius made the debootstrap plugin require the target directory to not exist, or be an empty directory, with an override for this check. * Lars Wirzenius changed the test suite so there is one script that can run it entirely. It's called check-all. The old check-all is now check-images. * Lars Wirzenius changed the luks plugin to use luks2 by default. Version 0.24, released 2021-08-22 ----------------------------------------------------------------------------- * This release disables the non-Intel architectures in the `./check-all` scripts. The builds work when done manually, but now when done by the script. I'm tired of debugging this and will be rewriting the test suite to be more sane. * Aaron Dewes merged `qemu-debootstrap` with plain `debootstrap`. As of Debian 10 (buster) plain `debootstrap` can do the same foreign bootstrap thing. * Andres Salomon fixed the logic for the `apt` plugin's `recommends` option. The option has reversed the meaning to make it easier for the user to get right. * Lars Wirzenius disabled using `realpath` to resolve the output device to not include symbolic links, to fix LVM2 support. This probably breaks something else. * Lars Wirzenius changed the `lvcreate` step to run the `lvcreate` command in a way the prevents it from asking questions. Version 0.23, released 2021-04-01 ----------------------------------------------------------------------------- * Matthias Klein added the optional `options` field to the `mkfs` step to set aditional options for mkfs. * Gunnar Wolf added support for not installing recommended packages in the `apt` step in a .vmdb file. Version 0.22, released 2021-01-16 ----------------------------------------------------------------------------- * This release has a lot of changes from Ryutaroh Matsumoto and Christian Kastner to fix and clean up things, and to add support for building images for I385 (UEFI and BIOS), armhf+UEFI, and ppc64el. Version 0.21, released 2020-12-18 ----------------------------------------------------------------------------- * The grub plugin is able to install the UEFI version of grub on arm64 targets, patch from David Edmondson.[ * The documentation tile for the `resize_rootfs` step is now fixed. Version 0.20, released 2020-10-18 ----------------------------------------------------------------------------- * All use of the Python `cliapp` library has been dropped from vmdb2 now. * New step `resize-rootfs` adds a systemd service to resize the root file system to fit the actual drive on boot. Version 0.19, released 2020-09-16 ----------------------------------------------------------------------------- Change in this release are entirely internal code changes that should not affect users of vmdb2. * Python Code has been formatted with the Black code formatting tool. * Changes have been made towards dropping the dependency on the `cliapp` library: the Python standard library module `subprocess` is used instead of `cliapp.runcmd`, and the `cliapp` plugin system has been replaced with a custom one. Version 0.18, released 2020-08-13 ----------------------------------------------------------------------------- * The way the manual was formatted could sometimes lead to different plugins' documentation being merged into the same section. This has now been fixed. Version 0.17, released 2020-07-13 ----------------------------------------------------------------------------- * The luks plugin now runs the `cryptsetup` utility with the `-q` (alias for `--batch-mode`) option for batch mode, to prevent the utility from asking any questions of the user. Fix by Birger Schacht. Version 0.16, released 2020-06-06 ----------------------------------------------------------------------------- * Progress output is now flushed after every write. This matters, when the output goes to a pipe, because then Python would buffer the output, meaning that progress output gets delayed until the buffer fills up, and that's useless. * The documentation is now formatted using Subplot. Previously, the manual and the acceptance test suite were split into separate, but confusingly named files (vmdb2.md vs vmdb2.mdwn); now they are combined. The manual has been cleaned up a bit. * Add a `quiet` field to the **grub step** to configure the kernel boot to be quiet or not. Default is now not quiet, which is a change from before. * Add a `timeout` field to the **grub step** to configure grub menu timeout. Default is now zero seconds, which is a change from the Debian default of five seconds. * Symbolic links can now be used in the **mkpart step** for device files, thanks to Tim Small. Version 0.15, released 2020-05-05 ----------------------------------------------------------------------------- ### New and changed features * Document and rename the `virtual-filesystems` plugin. This allows installation of Debian packages that require `/proc` to be mounted to install, such as `ca-certificates-java`. Undo previous changes to have `/proc` automatically mounted in the chroot, when running commands in the chroot. That wasn't enough (e.g., it didn't cover Ansible), and `virtual-filesystems` seems generally like the more general and better solution. The `grub` step handles the filesystems it needs itself, but doesn't mind if they're already mounted. This change should NOT require changes to existing, working .vmdb files. If it does, please report a bug. * The .vmdb file can now be passed in via the standard input, by specifying `-` as the filename. Patch from voney. * The manual now documents that un-partitioned images are not supported, at least not yet. * The "All went fine" message is now written at the very end of the vmdb2 run, if at all. Previously all the messages from teardown parts of steps were printed after it, hiding it effectively. * The standard output and error output of the commands vmdb2 runs are no longer logged in real time, only after the command has finished. This is a side effect of the `cliapp.runcmd` helper function no longer being used. The Python standard library subprocess module is used directly instead. This change is a step to drop the dependency on the cliapp library. ### Bug fixes * The `grub` step documentation now uses the correct key for specifying the image device (`image-dev`). Thanks: Sebastian Bachmann. * The `mkpart` step now waits for the new device node to be created. Previously it sometimes took enough time to be created that the next step failed. This only affects when the output disk is a real block device, not a disk image file. ### Changes mainly affecting vmdb2 and plugin developers * The `./check` script only runs the yarn integration testing tool if it's new enough to have the yarnutils Python library included. Yarn is being replaced with Subplot. * Step runners must now use a new way of specifying expected and allowed keys for steps. The new way specifies types for mandatory keys, and default values for optional keys. The old `get_required_keys` method is gone. This change makes mistakes in step runner code a little less likely, and opens up the possibility of generating documentation about step keys automatically. * Failures when un-mounting filesystems are now ignored. Such failures can happen for various reasons, but by far the most common reason is because the mount had already been unmounted and it's simpler to just ignore failures than keep meticulous track of what's been unmounted already. Version 0.14.1 ----------------------------------------------------------------------------- This is a bug fix release to fix bugs in the 0.14 release that slipped through due to insufficient testing. * The shell plugin source file was misnamed. Fixed now. * Fixes to problems found while building real images. The new `check-external` script builds my own images, and Debian's Raspberry Pi images. Version 0.14, released 2020-03-21 ----------------------------------------------------------------------------- It has been two years since the previous release. These release notes are a little sketchy. * All plugins now provide exactly one type of step. This is a little simpler than having multiple steps provided by the same plugin. Simplicity is good. * There is now some better automated testing. * vmdb2 now checks at start time that all required keys are set for all steps and that keys have values of the correct type. * Source code for vmdb2 is now hosted on gitlab.com and git.liw.fi, moved there from GitHub. * Fixed smoke test timeout for starting a VM to 5 min. The previous timeout (30 s) was short enough that tests often failed because of it. * The `debootstrap` action now finishes with `apt-get update`. For some reason thing had changed so that this became necessary. * Christian Schlüter added the optional `label` field to the `mkfs` step to set the filesystem label upon creation. * Changed the `grub` step to take an optional `image-dev` field to override the heuristics for finding the device where GRUB is to be installed. This was necessary to allow vmdb2 to install Debian onto real hardware. * The smoke test now tests a UEFI image as well. * The rootfs unpacking action now copies `/etc/resolv.conf` into the chroot, overwriting what the tarball has. This allows vmdb2 to work with a tarball generated in a different network location. * Add plugins for doing full-disk encryption via cryptsetup, and LVM2. * There is now some documentation. * Christian Schlüter added support for f2fs labels in the `mkfs` step. * Step runners may now implement the `run_even_if_skipped` method to have code that is run whether the step is skipped or not. The `debootstrap` step now uses this to run `apt-get update` always. This allows a rootfs tarball that is old to be used, without the Packages files being too old to be usable. * The `cache_rootfs` step now caches all the explicitly mounted filesystems, not just the root filesystem. Version 0.13.2, released 2018-05-06 ------------------------------------ * New build, after CI screwed up. Version 0.13.1, released 2018-04-30 ---------------------------------- * Fix build-dependency on pandoc. Version 0.13.1, released 2018-04-30 ---------------------------------- Version 0.12, released 2018-02-24 ---------------------------------- * Minor Debian packaging fixes. Version 0.11, released 2018-02-10 ---------------------------------- * Build fixes so that .deb doesn't contain Python 2.7 stuff, which isn't wanted or used. Version 0.10, released 2018-02-10 --------------------------------- * Build tweaks to make builds work on Debian unstable. Version 0.9, released 2017-10-11 --------------------------------- * Stuart Prescott added a `variant` parameter to the `vmdebootstrap` plugin. * Stuart and Lars documented some of the dependencies in README. If they're insufficient, report a bug please. * Stuart added a `qemu-debootstrap` plugin to build images for a foreign architecture. * Lars Wirzenius added the function `vmdb.runcm_chroot` for executing programs inside a chroot. It's a short wrapper around `vmdb.runcmd`, but is a little easier to get right than invoking **chroot**(8) via `runcmd`. Michael Stapelberg fixed the function. * Lars Wirzenius changed it so that `.vmdb` files can have structured YAML values and their constituent strings are used as Jinja2 templates. Previously vmdb2 would just crash. This opens a possibility to have things like "list of packages to install" for the `apt` step as a YAML list, instead of having the step runner parse a space delimited list. * Michael Stapelberg added the `fs-type` parameter to the `mkpart` step so that Raspberry Pi images can be created. * Lars Wirzenius changed the `apt` step. Previously the `apt` field values was the name of one package to install. Now the value MUST be `install` and a separate field `packages` is a YAML list of package names to install. These will all be installed at once. * Michael Stapelberg and Lars Wirzenius added support for a `components` list to the `qemu-debootstrap` step, to allow giving a list of components to the debootstrap run by the step. Version 0.8, released 2017-07-29 --------------------------------- * The `mount` plugin can now specify a mount point on an already mounted filesystem, such as /boot inside the root filesystem already mounted. This allows constructing a system with a separate /boot partition. Thanks to Stuart Prescott for reporting the lack of this feature. Version 0.7, released 2017-06-18 --------------------------------- * New plugin `virtuals` provides step `mount-virtual-filesystems` for mounting virtual filesystems such as `/proc` and `/dev` as well. * The `apt` step installs `eatmydata` and runs `apt` under it, to speed up package installs. Version 0.6, released 2017-06-11 --------------------------------- * setup.py now installs the plugins, making the .deb package actually be usable. Version 0.5, released 2017-06-04 --------------------------------- * The "unless:" part of vmdb spec files now actually works. Previously it was entrely unimplmented (there was code to implement the check, but it was never actually called). The apt and debootstrap plugins have been fixed to not do "has rootfs tarball been unpacked" checks themselves. Version 0.4, released 2017-06-03 --------------------------------- * Add a rudimenteary smoke test yarn for vmdb2-built images. * Add a plugin to run Ansible to configure an image at build time. Doing this via Ansible saves me from having to write equivalent functionality directly into vmdb2, which would be quite a lot of work. (If someone wants puppet, a plugin for that is probably easily doable, assuming it can run against a chroot. I don't now, I've never used puppet.) Version 0.3, released 2017-05-21 --------------------------------- * Simplify progress reporting to go to stdout, plus stop logging progress reports ar as errors. * Add a BIOS flavor for installing GRUB. Version 0.2, released 2017-05-14 --------------------------------- * Add plugin to provide steps to cache the rootfs, and to unpack the rootfs from the cache, instead of running debootstrap or installing packages. This speeds up iteration time from about 9 minutes to 40 seconds on my laptop. See small.vmdb for an example. * Add a generic "unless this condition is true" functionality to steps. If a step has a field "unless: foo" it is skipped the variable foo exists and is true. The variables are set by steps, and currently only the rootfs unpacking step sets a variable (the `rootfs_unpacked` variable). This allows debootstrap to be skipped if the rootfs has already been created by unpacking a cached tarball. Version 0.1, released 2017-05-13 ----------------------------------------------------------------------------- This is the first release. It can build a UEFI image for the amd64 architecture. It's not meant to really be useful for other people.