diff options
| author | Lars Wirzenius <liw@liw.fi> | 2019-11-02 09:25:12 +0200 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2019-11-02 09:25:12 +0200 |
| commit | f0588d57f82d610e2a93732ab0a3cd1e0b36f7bc (patch) | |
| tree | e2a57aa218000feb0316c4982d69074eb847edfb | |
| parent | 949ae63daa42bcff415832ecbfe81d9487076125 (diff) | |
| download | vmdb2-f0588d57f82d610e2a93732ab0a3cd1e0b36f7bc.tar.gz | |
Change: move Debian packaging files to root of branch
80 files changed, 0 insertions, 4355 deletions
@@ -1,185 +0,0 @@ -NEWS for vmdb2, the Debian disk image builder -============================================================================= - -Version 0.13.2+git, not yet released ------------------------------------- - -* Source code for vmdb2 is now hosted on git.liw.fi, moved there from - GitHub. - -* Fixed smoke test timeout for starting a VM to 5 min. The previous - timeout (30 s) was short enough that tests often failed because of - it. - -* The `debootstrap` action now finishes with `apt-get update`. For - some reason thing had changed so that this became necessary. - -* Christian Schlüter added the optional `label` field to the `mkfs` - step to set the filesystem label upon creation. - -* Changed the `grub` step to take an optional `image-dev` field to - override the heuristics for finding the device where GRUB is to be - installed. This was necessary to allow vmdb2 to install Debian onto - real hardware. - -* The smoke test now tests a UEFI image as well. - -* The rootfs unpacking action now copies `/etc/resolv.conf` into the - chroot, overwriting what the tarball has. This allows vmdb2 to work - with a tarball generated in a different network location. - -* Add plugins for doing full-disk encryption via cryptsetup, and LVM2. - -* There is now some documentation. - -* Christian Schlüter added support for f2fs labels in the `mkfs` step. - -* Step runners may now implement the `run_even_if_skipped` method to - have code that is run whether the step is skipped or not. The - `debootstrap` step now uses this to run `apt-get update` always. - This allows a rootfs tarball that is old to be used, without the - Packages files being too old to be usable. - -* The `cache_rootfs` step now caches all the explicitly mounted - filesystems, not just the root filesystem. - -Version 0.13.2, released 2018-05-06 ------------------------------------- - -* New build, after CI screwed up. - -Version 0.13.1, released 2018-04-30 ----------------------------------- - -* Fix build-dependency on pandoc. - -Version 0.13.1, released 2018-04-30 ----------------------------------- - - -Version 0.12, released 2018-02-24 ----------------------------------- - -* Minor Debian packaging fixes. - -Version 0.11, released 2018-02-10 ----------------------------------- - -* Build fixes so that .deb doesn't contain Python 2.7 stuff, which - isn't wanted or used. - -Version 0.10, released 2018-02-10 ---------------------------------- - -* Build tweaks to make builds work on Debian unstable. - -Version 0.9, released 2017-10-11 ---------------------------------- - -* Stuart Prescott added a `variant` parameter to the `vmdebootstrap` - plugin. - -* Stuart and Lars documented some of the dependencies in README. If - they're insufficient, report a bug please. - -* Stuart added a `qemu-debootstrap` plugin to build images for a - foreign architecture. - -* Lars Wirzenius added the function `vmdb.runcm_chroot` for executing - programs inside a chroot. It's a short wrapper around `vmdb.runcmd`, - but is a little easier to get right than invoking **chroot**(8) via - `runcmd`. Michael Stapelberg fixed the function. - -* Lars Wirzenius changed it so that `.vmdb` files can have structured - YAML values and their constituent strings are used as Jinja2 - templates. Previously vmdb2 would just crash. This opens a - possibility to have things like "list of packages to install" for - the `apt` step as a YAML list, instead of having the step runner - parse a space delimited list. - -* Michael Stapelberg added the `fs-type` parameter to the `mkpart` - step so that Raspberry Pi images can be created. - -* Lars Wirzenius changed the `apt` step. Previously the `apt` field - values was the name of one package to install. Now the value MUST be - `install` and a separate field `packages` is a YAML list of package - names to install. These will all be installed at once. - -* Michael Stapelberg and Lars Wirzenius added support for a - `components` list to the `qemu-debootstrap` step, to allow giving a - list of components to the debootstrap run by the step. - -Version 0.8, released 2017-07-29 ---------------------------------- - -* The `mount` plugin can now specify a mount point on an already - mounted filesystem, such as /boot inside the root filesystem already - mounted. This allows constructing a system with a separate /boot - partition. Thanks to Stuart Prescott for reporting the lack of this - feature. - -Version 0.7, released 2017-06-18 ---------------------------------- - -* New plugin `virtuals` provides step `mount-virtual-filesystems` for - mounting virtual filesystems such as `/proc` and `/dev` as well. - -* The `apt` step installs `eatmydata` and runs `apt` under it, to - speed up package installs. - -Version 0.6, released 2017-06-11 ---------------------------------- - -* setup.py now installs the plugins, making the .deb package actually - be usable. - -Version 0.5, released 2017-06-04 ---------------------------------- - -* The "unless:" part of vmdb spec files now actually works. Previously - it was entrely unimplmented (there was code to implement the check, - but it was never actually called). The apt and debootstrap plugins - have been fixed to not do "has rootfs tarball been unpacked" checks - themselves. - -Version 0.4, released 2017-06-03 ---------------------------------- - -* Add a rudimenteary smoke test yarn for vmdb2-built images. - -* Add a plugin to run Ansible to configure an image at build time. - Doing this via Ansible saves me from having to write equivalent - functionality directly into vmdb2, which would be quite a lot of - work. (If someone wants puppet, a plugin for that is probably easily - doable, assuming it can run against a chroot. I don't now, I've - never used puppet.) - -Version 0.3, released 2017-05-21 ---------------------------------- - -* Simplify progress reporting to go to stdout, plus stop logging - progress reports ar as errors. - -* Add a BIOS flavor for installing GRUB. - -Version 0.2, released 2017-05-14 ---------------------------------- - -* Add plugin to provide steps to cache the rootfs, and to unpack the - rootfs from the cache, instead of running debootstrap or installing - packages. This speeds up iteration time from about 9 minutes to 40 - seconds on my laptop. See small.vmdb for an example. - -* Add a generic "unless this condition is true" functionality to - steps. If a step has a field "unless: foo" it is skipped the - variable foo exists and is true. The variables are set by steps, and - currently only the rootfs unpacking step sets a variable (the - `rootfs_unpacked` variable). This allows debootstrap to be skipped - if the rootfs has already been created by unpacking a cached - tarball. - -Version 0.1, released 2017-05-13 ------------------------------------------------------------------------------ - -This is the first release. It can build a UEFI image for the amd64 -architecture. It's not meant to really be useful for other people. @@ -1,197 +0,0 @@ -README for vmdb2 or vmdebootstrap 2nd generation ; -*- mode: markdown;-*- -============================================================================= - -[vmdb2][] is a program for producing a disk image with Debian -installed. - -[vmdb2]: https://vmdb2.liw.fi/ - -Introduction ------------------------------------------------------------------------------ - -[vmdebootstrap][] installs Debian onto a disk image. It is like the -[debootstrap][] tool, except the end result is a bootable disk image, -not a directory. vmdebootstrap takes care of creating partitions, and -filesystems, and allows some more customization than the older -vmdebootstrap does. - -vmdebootstrap is also a messy pile of kludge, and rather inflexible. -vmdb2 is a re-implementation from scratch, without a need for -backwards compatibility. It aims to provide more flexibility than -vmdebootstrap, without becoming anywhere near as complicated. Think of -vmdb2 as "vmdebootstrap the second generation". The name has changed -to allow the two tools to installable in parallel, which is important -for a transition period. - -The main user-visible difference between vmdebootstrap and vmdb2 is -that the older program provides extensibility via a legion of command -line options and the newer program by providing a domain specific -language to express what kind of Debian system is to be created. - -(Lars Wirzenius wrote both vmdebootstrap and vmdb2 and is entitled to -sneer at his younger self. It's his way of dealing with the mountain -of guilt of making something as awful as vmdebootstrap.) - -[vmdebootstrap]: http://liw.fi/vmdebootstrap/ -[debootstrap]: https://packages.debian.org/unstable/debootstrap - - -Getting vmdb2 ------------------------------------------------------------------------------ - -vmdb2 source code is available via git: - -* <http://git.liw.fi/vmdb2> - -It used to be on GitHub as well, but was withdrawn from there due to -GitHub being a proprietary service and a dislike of its development -workflow. - -Requirements: - -The following tools are used by vmdb2 (Debian package names in brackets). - -* `kpartx` [kpartx, mkpart command] -* `parted` [`parted`, mklabel command] -* `qemu-img` [`qemu-utils`, mkimg command] -* `qemu-user-static` [`qemu-user-static`, qemu-debootstrap command] - -The following Python modules are used by vmdb2 (Debian package names in brackets). - -* cliapp [`python3-cliapp`] -* jinja2 [`python3-jinja2`] -* yaml [`python3-yaml`] - - -Dependencies for smoke.sh ------------------------------------------------------------------------------ - -You probably need the following installed to run the smoke test: - -- git -- python3-coverage-test-runner -- python3-cliapp -- python3-jinja2 -- pylint3 -- cmdtest 0.31 or later -- qemu-utils -- parted -- kpartx -- debootstrap -- expect -- qemu-system - - -Tutorial ------------------------------------------------------------------------------ - -To use vmdb2, git clone the source and at the root of the source tree -run the following command: - - sudo ./vmdb2 --output simple.img simple.vmdb --log simple.log - -`--output simple.img` specifies that the output image is called -`simple.img`, the specification is `simple.vmdb` and the log file goes -to `simple.log`. - - -Plugins and steps ------------------------------------------------------------------------------ - -The `vmdb2` architecture consists of a main program that reads the -input file, finds a matching "step runner" for each step used in the -input file, and then runs the steps in order. If there's a problem, it -runs corresponding "teardown" steps in reverse order of the steps. - -A step might be "mount this filesystem", and the corresponding -teardown is "unmount". - -Steps (and teardowns) are provided by plugins; see the `vmdb/plugins` -directory in the source tree. Steps are intended to be very cohesive -and lowly coupled. They may share some state (such as mounted -filesystems) via the `State` object, but not in any other way. A -plugin may provide multiple steps. - -See the plugin directory for which steps currently exist. A list of -steps that will become incomplete as soon as development continues: - -* chroot (run shell snippet in chroot) -* shell (run shell snippet without chroot) -* debootstrap (run deboostrap) -* apt (install packagers in chroot with apt) -* mkimg (create disk image) -* mklabel (create partition table on a disk image) -* mkpart (create partition) -* mkfs (create filesystem in a partition) -* mount (mount filesystem, teardown unmounts it automatically) - -See `simple.vmdb` for examples. Note how the file uses Jinja2 -templating for value fields to get value of `--output` in the right -places. Also note how creating a partition or mounting a filesystem -assigns a "tag" that can be referenced in steps where the -partition/filesystem is needed, without having to know the actual path -to the device node or mount point. - - -Writing plugins ------------------------------------------------------------------------------ - -More step runners would be good, and will be added based on -actual reported needs by users ("I need to have this to..."), not -speculatively ("This seems like a good idea"). - -To write a plugin, see the existing ones for examples, and put it in -`vmdb/plugins/foo_plugin.py` for some value of `foo`. - -Plugins are meant to be very easy to write. If not, there's probably -something wrong with `vmdb2`. Please raise the issue. - - -Hacking ------------------------------------------------------------------------------ - -To run automated tests: - - ./check - -This only runs the unit tests and build tests. To run a smoke test -that actually builds and boots images: - - sudo ./smoke.sh cache.tar.gz - -where `cache.tar.gz` caches the debootstrap output for a future run. - -You'll need the yarn program (part of the [cmdtest][] package), and -also [CoverageTestRunner][] for running the unit tests. - -[cmdtest]: http://liw.fi/cmdtest/ -[CoverageTestRunner]: http://liw.fi/coverage-test-runner/ - - -Contact ------------------------------------------------------------------------------ - -To contact Lars, email is best: `liw@liw.fi`. - -There is an IRC channel for vmdb2: irc.oftc.net network, `#vmdb2`. - - -Legalese ------------------------------------------------------------------------------ - -Copyright 2017-2019 Lars Wirzenius - -This program is free software: you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. - -You should have received a copy of the GNU General Public License -along with this program. If not, see <http://www.gnu.org/licenses/>. - -=*= License: GPL-3+ =*= diff --git a/ansible.vmdb b/ansible.vmdb deleted file mode 100644 index 3319eb5..0000000 --- a/ansible.vmdb +++ /dev/null @@ -1,46 +0,0 @@ -# This is a sample VMDB2 input file to specify a simple -# system that boots on a PC with BIOS. - -steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: msdos - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - part-tag: root-part - - - mkfs: ext4 - partition: root-part - - - mount: root-part - fs-tag: root-fs - - - unpack-rootfs: root-fs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: root-fs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - - python - fs-tag: root-fs - unless: rootfs_unpacked - - - cache-rootfs: root-fs - unless: rootfs_unpacked - - - ansible: root-fs - playbook: ansible.yml - - - grub: bios - root-fs: root-fs - root-part: root-part - device: "{{ output }}" diff --git a/ansible.yml b/ansible.yml deleted file mode 100644 index bb23ca1..0000000 --- a/ansible.yml +++ /dev/null @@ -1,7 +0,0 @@ -- hosts: image - roles: - - set_hostname - - unset_root_password - vars: - hostname: pc - diff --git a/debian/changelog b/changelog index f70dad6..f70dad6 100644 --- a/debian/changelog +++ b/changelog @@ -1,12 +0,0 @@ -#!/bin/sh - -set -eu - -python3 -m CoverageTestRunner --ignore-missing-from=without-tests yarns vmdb -yarn \ - --shell=python2 \ - --shell-arg '' \ - --shell-library yarns/lib.py \ - --env "PYTHONPATH=$(pwd)/yarns" \ - --cd-datadir \ - yarns/*.yarn "$@" diff --git a/code-of-conduct.md b/code-of-conduct.md deleted file mode 100644 index 6019502..0000000 --- a/code-of-conduct.md +++ /dev/null @@ -1,73 +0,0 @@ -# Contributor Covenant Code of Conduct - -## Our Pledge - -In the interest of fostering an open and welcoming environment, we as -contributors and maintainers pledge to making participation in our project and -our community a harassment-free experience for everyone, regardless of age, body -size, disability, ethnicity, gender identity and expression, level of experience, -education, socio-economic status, nationality, personal appearance, race, -religion, or sexual identity and orientation. - -## Our Standards - -Examples of behavior that contributes to creating a positive environment -include: - -* Using welcoming and inclusive language -* Being respectful of differing viewpoints and experiences -* Gracefully accepting constructive criticism -* Focusing on what is best for the community -* Showing empathy towards other community members - -Examples of unacceptable behavior by participants include: - -* The use of sexualized language or imagery and unwelcome sexual attention or - advances -* Trolling, insulting/derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or electronic - address, without explicit permission -* Other conduct which could reasonably be considered inappropriate in a - professional setting - -## Our Responsibilities - -Project maintainers are responsible for clarifying the standards of acceptable -behavior and are expected to take appropriate and fair corrective action in -response to any instances of unacceptable behavior. - -Project maintainers have the right and responsibility to remove, edit, or -reject comments, commits, code, wiki edits, issues, and other contributions -that are not aligned to this Code of Conduct, or to ban temporarily or -permanently any contributor for other behaviors that they deem inappropriate, -threatening, offensive, or harmful. - -## Scope - -This Code of Conduct applies both within project spaces and in public spaces -when an individual is representing the project or its community. Examples of -representing a project or community include using an official project e-mail -address, posting via an official social media account, or acting as an appointed -representative at an online or offline event. Representation of a project may be -further defined and clarified by project maintainers. - -## Enforcement - -Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported by contacting the project team at liw@liw.fi. All -complaints will be reviewed and investigated and will result in a response that -is deemed necessary and appropriate to the circumstances. The project team is -obligated to maintain confidentiality with regard to the reporter of an incident. -Further details of specific enforcement policies may be posted separately. - -Project maintainers who do not follow or enforce the Code of Conduct in good -faith may face temporary or permanent repercussions as determined by other -members of the project's leadership. - -## Attribution - -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, -available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html - -[homepage]: https://www.contributor-covenant.org diff --git a/debian/copyright b/copyright index effd826..effd826 100644 --- a/debian/copyright +++ b/copyright diff --git a/format.sh b/format.sh deleted file mode 100755 index 0cb04fb..0000000 --- a/format.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/sh - -set -eu - -cleanup() -{ - rm -rf "$tmp" -} - -tmp="$(mktemp -d)" -trap cleanup EXIT - -version="$(git describe)" -sed "s/^date: .*/date: $version/" vmdb2.mdwn > "$tmp/prelude.mdwn" - -pandoc \ - --self-contained \ - --standalone \ - --css vmdb2.css \ - --toc \ - --number-sections \ - -o vmdb2.html \ - "$tmp/prelude.mdwn" vmdb/plugins/*.mdwn - -pandoc \ - --toc \ - --number-sections \ - -Vdocumentclass=report \ - -Vgeometry:a4paper \ - -Vfontsize:12pt \ - -Vmainfont:FreeSerif \ - -Vsansfont:FreeSans \ - -Vmonofont:FreeMonoBold \ - '-Vgeometry:top=2cm, bottom=2.5cm, left=2cm, right=1cm' \ - -o vmdb2.pdf \ - "$tmp/prelude.mdwn" vmdb/plugins/*.mdwn diff --git a/lvm2.lukskey b/lvm2.lukskey deleted file mode 100644 index 9f592eb..0000000 --- a/lvm2.lukskey +++ /dev/null @@ -1 +0,0 @@ -hunter2 diff --git a/lvm2.vmdb b/lvm2.vmdb deleted file mode 100644 index 169f501..0000000 --- a/lvm2.vmdb +++ /dev/null @@ -1,71 +0,0 @@ -# This is a sample VMDB2 input file to specify a simple -# system that boots on a PC with BIOS. Uses LVM2. - -steps: - - mkimg: "{{ image }}" - size: 4G - - - mklabel: msdos - device: "{{ image }}" - - - mkpart: primary - device: "{{ image }}" - start: 1M - end: 100M - tag: /boot - - - mkpart: primary - device: "{{ image }}" - start: 100M - end: 2G - tag: /rootpv - - - cryptsetup: /rootpv - tag: rootpv_crypt - key-cmd: echo lvm2.lukskey - - - vgcreate: rootvg - physical: - - rootpv_crypt - - - lvcreate: rootvg - name: rootfs - size: 1G - - - mkfs: ext2 - partition: boot - - - mkfs: ext4 - partition: rootfs - - - mount: /boot - - - mount: /boot - - - mount: rootfs - - - unpack-rootfs: rootfs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: rootfs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - tag: rootfs - unless: rootfs_unpacked - - - cache-rootfs: rootfs - unless: rootfs_unpacked - - - chroot: rootfs - shell: | - sed -i '/^root:[^:]*:/s//root::/' /etc/passwd - echo pc-vmdb2 > /etc/hostname - - - grub: bios - tag: rootfs - image-dev: "{{ image }}" - console: serial diff --git a/pc.vmdb b/pc.vmdb deleted file mode 100644 index 46ae16e..0000000 --- a/pc.vmdb +++ /dev/null @@ -1,51 +0,0 @@ -# This is a sample VMDB2 input file to specify a simple -# system that boots on a PC with BIOS. - -steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: msdos - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - part-tag: root-part - - - mkfs: ext4 - partition: root-part - - - mount: root-part - fs-tag: root-fs - - - unpack-rootfs: root-fs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: root-fs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - fs-tag: root-fs - unless: rootfs_unpacked - - - cache-rootfs: root-fs - unless: rootfs_unpacked - - - chroot: root-fs - shell: | - apt -y install python - - - shell: | - printf '[pc]\n%s hostname=pc\n' "$ROOT" > pc.inventory - ansible-playbook -i pc.inventory -c chroot pc.yml - root-fs: root-fs - - - grub: bios - root-fs: root-fs - root-part: root-part - device: "{{ output }}" diff --git a/pylint.conf b/pylint.conf deleted file mode 100644 index c07e542..0000000 --- a/pylint.conf +++ /dev/null @@ -1,42 +0,0 @@ -[MASTER] -persistent=no - -[MESSAGES CONTROL] -disable= - abstract-class-little-used, - abstract-class-not-used, - arguments-differ, - attribute-defined-outside-init, - cyclic-import, - fixme, - global-statement, - inconsistent-return-statements, - interface-not-implemented, - invalid-name, - locally-disabled, - missing-docstring, - no-else-return, - no-member, - no-self-use, - protected-access, - redefined-builtin, - redefined-variable-type, - star-args, - too-few-public-methods, - too-many-arguments, - too-many-branches, - too-many-instance-attributes, - too-many-lines, - too-many-locals, - too-many-public-methods, - too-many-statements, - unidiomatic-typecheck, - unsubscriptable-object, - unsupported-membership-test, - unused-argument - -[REPORTS] -reports=no - -[SIMILARITIES] -min-similarity-lines=999 diff --git a/roles/set_hostname/tasks/main.yml b/roles/set_hostname/tasks/main.yml deleted file mode 100644 index f1b9649..0000000 --- a/roles/set_hostname/tasks/main.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: set /etc/hostname - shell: | - echo "{{ hostname }}" > /etc/hostname diff --git a/roles/unset_root_password/tasks/main.yml b/roles/unset_root_password/tasks/main.yml deleted file mode 100644 index 67134fd..0000000 --- a/roles/unset_root_password/tasks/main.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: unset root password - shell: | - sed -i '/^root:[^:]*:/s//root::/' /etc/passwd diff --git a/setup.py b/setup.py deleted file mode 100755 index 3b89429..0000000 --- a/setup.py +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env python3 -# Copyright (C) 2017 Lars Wirzenius <liw@liw.fi> -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -from distutils.core import setup, Extension -from distutils.cmd import Command -from distutils.command.build import build -from distutils.command.clean import clean -import os -import glob - -import cliapp - -import vmdb - - -class Build(build): - - def run(self): - build.run(self) - self.build_manpage('vmdb2', '') - self.format_yarns() - - def build_manpage(self, program, lang): - return - # building manpage fails on by unstable CI worker for mysterious - # reasons, will re-enable later - print('building manpage for %s (lang=%s)' % (program, lang)) - self.generate_troff(program, lang) - self.format_man_as_txt(program) - - def generate_troff(self, program, lang): - with open('%s.1%s' % (program, lang), 'w') as f: - cliapp.runcmd( - ['python', program, - '--generate-manpage=%s.1%s.in' % (program, lang), - '--output=%s.1' % program], - stdout=f) - - def format_man_as_txt(self, program): - env = dict(os.environ) - env['MANWIDTH'] = '80' - with open('%s.1.txt' % program, 'w') as f: - cliapp.runcmd( - ['man', '-l', '%s.1' % program], - ['col', '-b'], - stdout=f, - env=env) - - def format_yarns(self): - print('building yarns') - cliapp.runcmd(['make', '-C', 'yarns']) - - -setup( - name='vmdb2', - version=vmdb.__version__, - description='create disk image with Debian installed', - author='Lars Wirzenius', - author_email='liw@liw.fi', - url='http://liw.fi/vmdebootstrap/', - scripts=['vmdb2'], - packages=['vmdb', 'vmdb.plugins'], - data_files=[('share/man/man1', glob.glob('*.1'))], - cmdclass={ - 'build': Build, - }, -) diff --git a/simple.vmdb b/simple.vmdb deleted file mode 100644 index a3446ea..0000000 --- a/simple.vmdb +++ /dev/null @@ -1,29 +0,0 @@ -steps: - - mklabel: gpt - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - part-tag: root-part - - - mkfs: ext4 - partition: root-part - - - mount: root-part - fs-tag: root-fs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: root-fs - - - apt: install - packages: - - linux-image-amd64 - fs-tag: root-fs - - - shell: | - echo Disk usage of this installation: - du -sh "$ROOT" - root-fs: root-fs diff --git a/smoke-pc.vmdb b/smoke-pc.vmdb deleted file mode 100644 index 50b4afc..0000000 --- a/smoke-pc.vmdb +++ /dev/null @@ -1,51 +0,0 @@ -steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: msdos - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 1M - end: 10M - tag: unused - - - mkpart: primary - device: "{{ output }}" - start: 10M - end: 100% - tag: rootfs - - - kpartx: "{{ output }}" - - - mkfs: ext4 - partition: rootfs - label: smoke - - - mount: rootfs - - - unpack-rootfs: rootfs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: rootfs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - tag: rootfs - unless: rootfs_unpacked - - - cache-rootfs: rootfs - unless: rootfs_unpacked - - - chroot: rootfs - shell: | - sed -i '/^root:[^:]*:/s//root::/' /etc/passwd - echo pc-vmdb2 > /etc/hostname - - - grub: bios - tag: rootfs - console: serial diff --git a/smoke-uefi.vmdb b/smoke-uefi.vmdb deleted file mode 100644 index 18c72cf..0000000 --- a/smoke-uefi.vmdb +++ /dev/null @@ -1,55 +0,0 @@ -steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: gpt - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 1G - tag: efifs - - - mkpart: primary - device: "{{ output }}" - start: 1G - end: 100% - tag: rootfs - - - kpartx: "{{ output }}" - - - mkfs: vfat - partition: efifs - - - mkfs: ext4 - partition: rootfs - label: smoke - - - mount: rootfs - - - unpack-rootfs: rootfs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: rootfs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - tag: rootfs - unless: rootfs_unpacked - - - cache-rootfs: rootfs - unless: rootfs_unpacked - - - chroot: rootfs - shell: | - sed -i '/^root:[^:]*:/s//root::/' /etc/passwd - echo pc-vmdb2 > /etc/hostname - - - grub: uefi - tag: rootfs - efi: efifs - console: serial diff --git a/smoke.sh b/smoke.sh deleted file mode 100755 index fa7ff0c..0000000 --- a/smoke.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/sh - -set -eu - -tarball="$1" -shift - -yarn smoke.yarn --env ROOTFS_TARBALL="$tarball" "$@" diff --git a/smoke.yarn b/smoke.yarn deleted file mode 100644 index c0a7499..0000000 --- a/smoke.yarn +++ /dev/null @@ -1,98 +0,0 @@ -# Smoke test vmdb2-built images - -This yarn file builds a basic image, and runs it under Qemu. The image -is configured to have a serial console enabled, and the scenario below -will log in as root (no password), and command the virtual machine to -power off. If this works, the image is at least minimally functional, -so the "smoke test" passes. - -More elaborate testing can be added, but smoke testing is enough for -now. - -To run this yarn file, you need to run it as root (since it needs to -build an image as root), and you need to run on an amd64 system (and -thus build an amd64 image). Additionally, you need to add the folloing -option to yarn: - - EXAMPLE - --env ROOTFS_TARBALL=/path/to/rootfs/tarball - -This means vmdb2 will use the given tarball when creating an image, -and if the tarball doesn't exist yet, it will create it. This makes -testing multiple time much faster. - - SCENARIO smoke test image - WHEN user runs vmdb smoke-pc.vmdb --output smoke-pc.img - THEN user can BIOS boot smoke-pc.img and power it off from root shell - - WHEN user runs vmdb smoke-uefi.vmdb --output smoke-uefi.img - THEN user can UEFI boot smoke-uefi.img and power it off from root shell - - IMPLEMENTS WHEN user runs vmdb (\S+) --output (\S+) - "$SRCDIR/vmdb2" --no-default-config "$SRCDIR/$MATCH_1" \ - --output "$DATADIR/$MATCH_2" \ - --log "$DATADIR/vmdb.log" \ - --verbose \ - --rootfs-tarball "$ROOTFS_TARBALL" - - IMPLEMENTS THEN user can BIOS boot (\S+) and power it off from root shell - cd "$DATADIR" - img="$MATCH_1" - cat << EOF > run.sh - qemu-system-x86_64 -drive file="$img",format=raw -m 1024 -nographic - EOF - chmod a+rx run.sh - cat << EOF > expect.txt - set timeout 300 - proc abort {} { - puts "ERROR ERROR\n" - exit 1 - } - spawn ./run.sh - expect "login: " - send "root\n" - expect "# " - send "poweroff\r" - set timeout 5 - expect { - "reboot: Power down" {puts poweroffing\n} - eof abort - timeout abort - } - expect eof - EOF - expect -d expect.txt > expect.out - - IMPLEMENTS THEN user can UEFI boot (\S+) and power it off from root shell - cd "$DATADIR" - img="$MATCH_1" - cat << EOF > run.sh - cp /usr/share/OVMF/OVMF_VARS.fd . - qemu-system-x86_64 \ - -m 1024 \ - -drive if=pflash,format=raw,unit=0,file=/usr/share/ovmf/OVMF.fd,readonly=on \ - -drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \ - -drive format=raw,file="$img" \ - -nographic - EOF - chmod a+rx run.sh - cat << EOF > expect.txt - set timeout 300 - proc abort {} { - puts "ERROR ERROR\n" - exit 1 - } - spawn ./run.sh - expect "login: " - send "root\n" - expect "# " - send "poweroff\r" - set timeout 5 - expect { - "reboot: Power down" {puts poweroffing\n} - eof abort - timeout abort - } - expect eof - EOF - expect -d expect.txt > expect.out diff --git a/debian/source/format b/source/format index 163aaf8..163aaf8 100644 --- a/debian/source/format +++ b/source/format diff --git a/uefi.vmdb b/uefi.vmdb deleted file mode 100644 index 4d72766..0000000 --- a/uefi.vmdb +++ /dev/null @@ -1,50 +0,0 @@ -# This is a sample VMDB2 input file that specifies a simple system for -# a PC that boots with UEFI. - -steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: gpt - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 1G - part-tag: efi-part - - - mkpart: primary - device: "{{ output }}" - start: 1G - end: 100% - part-tag: root-part - - - mkfs: vfat - partition: efi-part - - - mkfs: ext4 - partition: root-part - - - mount: root-part - fs-tag: root-fs - - - unpack-rootfs: root-fs - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: root-fs - unless: rootfs_unpacked - - - apt: install - packages: - - linux-image-amd64 - fs-tag: root-fs - unless: rootfs_unpacked - - - cache-rootfs: root-fs - unless: rootfs_unpacked - - - grub: uefi - rootfs: root-fs - efi-part: efi-part diff --git a/vmdb/__init__.py b/vmdb/__init__.py deleted file mode 100644 index 91aa015..0000000 --- a/vmdb/__init__.py +++ /dev/null @@ -1,40 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -from .version import __version__, __version_info__ -from .state import State -from .step_list import ( - StepRunnerList, - StepRunnerInterface, - NoMatchingRunner, - StepError, -) -from .runcmd import ( - runcmd, - runcmd_chroot, - set_verbose_progress, - progress, - error, -) -from .tags import Tags, UnknownTag, TagInUse, AlreadyHasDev, AlreadyMounted -from .unmount import unmount, NotMounted -from .spec import ( - Spec, - expand_templates, -) -from .app import Vmdb2 diff --git a/vmdb/app.py b/vmdb/app.py deleted file mode 100644 index 5073a05..0000000 --- a/vmdb/app.py +++ /dev/null @@ -1,121 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import logging -import sys - -import cliapp - -import vmdb - - -class Vmdb2(cliapp.Application): - - def add_settings(self): - self.settings.string( - ['image'], - 'create image file FILE', - metavar='FILE') - - self.settings.boolean( - ['verbose', 'v'], - 'verbose output') - - def setup(self): - self.step_runners = vmdb.StepRunnerList() - - def process_args(self, args): - if len(args) != 1: - sys.exit("No image specification was given on the command line.") - - vmdb.set_verbose_progress(self.settings['verbose']) - - spec = self.load_spec_file(args[0]) - state = vmdb.State() - state.tags = vmdb.Tags() - params = self.create_template_vars(state) - steps = spec.get_steps(params) - - # Check that we have step runners for each step - for step in steps: - self.step_runners.find(step) - - steps_taken, core_meltdown = self.run_steps(steps, state) - if core_meltdown: - vmdb.progress('Something went wrong, cleaning up!') - else: - vmdb.progress('All went fine, cleaning up.') - self.run_teardowns(steps_taken, state) - - if core_meltdown: - logging.error('An error occurred, exiting with non-zero exit code') - sys.exit(1) - - def load_spec_file(self, filename): - vmdb.progress('Load spec file {}'.format(filename)) - spec = vmdb.Spec() - spec.load_file(filename) - return spec - - def run_steps(self, steps, state): - return self.run_steps_helper( - steps, state, 'Running step: %r', 'run', False) - - def run_teardowns(self, steps, state): - return self.run_steps_helper( - list(reversed(steps)), state, 'Running teardown: %r', 'teardown', True) - - def run_steps_helper(self, steps, state, msg, method_name, keep_going): - core_meltdown = False - steps_taken = [] - - even_if_skipped = method_name + '_even_if_skipped' - for step in steps: - try: - logging.info(msg, step) - steps_taken.append(step) - runner = self.step_runners.find(step) - if runner.skip(step, self.settings, state): - logging.info('Skipping as requested by unless') - method_names = [even_if_skipped] - else: - method_names = [method_name, even_if_skipped] - - methods = [ - getattr(runner, name) - for name in method_names - if hasattr(runner, name) - ] - - for method in methods: - logging.info('Calling %s', method) - method(step, self.settings, state) - except BaseException as e: - vmdb.error(str(e)) - core_meltdown = True - if not keep_going: - break - - return steps_taken, core_meltdown - - def create_template_vars(self, state): - vars = dict() - for key in self.settings: - vars[key] = self.settings[key] - vars.update(state.as_dict()) - return vars diff --git a/vmdb/plugins/__init__.py b/vmdb/plugins/__init__.py deleted file mode 100644 index 5cb230b..0000000 --- a/vmdb/plugins/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -# This file exists to make vmdb.plugins be a Python package, so it can -# be installed by setup.py. diff --git a/vmdb/plugins/ansible.mdwn b/vmdb/plugins/ansible.mdwn deleted file mode 100644 index 8929c57..0000000 --- a/vmdb/plugins/ansible.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -Step: `ansible` ------------------------------------------------------------------------------ - -Run Ansible using a provided playbook, to configure the image. vmdb2 -sets up Ansible so that it treats the image as the host being -configured (via the `chroot` connecion). The image MUST have Python -installed (version 2 or 3 depending on Ansible version). - -Step keys: - -* `ansible` — REQUIRED; value is the tag of the root filesystem. - -* `playbook` — REQUIRED; value is the filename of the Ansible - playbook, relative to the .vmdb file. - -Example (in the .vmdb file): - - - apt: install - tag: root - packages: [python] - - - ansible: root - playbook: foo.yml - -Example (`foo.yml`): - - - hosts: image - tasks: - - - name: "set /etc/hostname" - shell: | - echo "{{ hostname }}" > /etc/hostname - - - name: "unset root password" - shell: | - sed -i '/^root:[^:]*:/s//root::/' /etc/passwd - - - name: "configure networking" - copy: - content: | - auto eth0 - iface eth0 inet dhcp - iface eth0 inet6 auto - dest: /etc/network/interfaces.d/wired - - vars: - hostname: discworld diff --git a/vmdb/plugins/ansible_plugin.py b/vmdb/plugins/ansible_plugin.py deleted file mode 100644 index 9620535..0000000 --- a/vmdb/plugins/ansible_plugin.py +++ /dev/null @@ -1,64 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os -import tempfile - -import cliapp - -import vmdb - - -class AnsiblePlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(AnsibleStepRunner()) - - -class AnsibleStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['ansible', 'playbook'] - - def run(self, step, settings, state): - tag = step['ansible'] - playbook = step['playbook'] - mount_point = state.tags.get_mount_point(tag) - - state.ansible_inventory = self.create_inventory(mount_point) - vmdb.progress( - 'Created {} for Ansible inventory'.format(state.ansible_inventory)) - - env = dict(os.environ) - env['ANSIBLE_NOCOWS'] = '1' - vmdb.runcmd( - ['ansible-playbook', '-c', 'chroot', - '-i', state.ansible_inventory, playbook], - env=env) - - def teardown(self, step, settings, state): - if hasattr(state, 'ansible_inventory'): - vmdb.progress('Removing {}'.format(state.ansible_inventory)) - os.remove(state.ansible_inventory) - - def create_inventory(self, chroot): - fd, filename = tempfile.mkstemp() - os.write(fd, '[image]\n{}\n'.format(chroot).encode()) - os.close(fd) - return filename diff --git a/vmdb/plugins/apt.mdwn b/vmdb/plugins/apt.mdwn deleted file mode 100644 index dd83f18..0000000 --- a/vmdb/plugins/apt.mdwn +++ /dev/null @@ -1,21 +0,0 @@ -Step: `apt` ------------------------------------------------------------------------------ - -Install packages using apt, which needs to already have been -installed. - -Step keys: - -* `apt` — REQUIRED; value MUST be `install`. - -* `tag` — REQUIRED; value is the tag for the root filesystem. - -* `packages` — REQUIRED; value is a list of packages to install. - -Example (in the .vmdb file): - - - apt: install - tag: root - packages: - - python - - linux-image-amd64 diff --git a/vmdb/plugins/apt_plugin.py b/vmdb/plugins/apt_plugin.py deleted file mode 100644 index 22177dc..0000000 --- a/vmdb/plugins/apt_plugin.py +++ /dev/null @@ -1,83 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os - -import cliapp - -import vmdb - - -class AptPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(AptStepRunner()) - - -class AptStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['apt', 'packages'] - - def run(self, step, settings, state): - operation = step['apt'] - if operation != 'install': - raise Exception('"apt" must always have value "install"') - - packages = step['packages'] - tag = step.get('tag') - if tag is None: - tag = step['fs-tag'] - mount_point = state.tags.get_mount_point(tag) - - if not self.got_eatmydata(state): - self.install_packages(mount_point, [], ['eatmydata']) - state.got_eatmydata = True - self.install_packages(mount_point, ['eatmydata'], packages) - - if step.get('clean', True): - self.clean_cache(mount_point) - - def got_eatmydata(self, state): - return hasattr(state, 'got_eatmydata') and getattr(state, 'got_eatmydata') - - def install_packages(self, mount_point, argv_prefix, packages): - env = os.environ.copy() - env['DEBIAN_FRONTEND'] = 'noninteractive' - - vmdb.runcmd_chroot( - mount_point, - argv_prefix + - ['apt-get', 'update'], - env=env) - - vmdb.runcmd_chroot( - mount_point, - argv_prefix + - ['apt-get', '-y', '--no-show-progress', 'install'] + packages, - env=env) - - def clean_cache(self, mount_point): - env = os.environ.copy() - env['DEBIAN_FRONTEND'] = 'noninteractive' - - vmdb.runcmd_chroot( - mount_point, - ['apt-get', 'clean'], - env=env) diff --git a/vmdb/plugins/chroot.mdwn b/vmdb/plugins/chroot.mdwn deleted file mode 100644 index 87fd5ec..0000000 --- a/vmdb/plugins/chroot.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -Step: `chroot` ------------------------------------------------------------------------------ - -Run a shell snippet in a chroot inside the image. - -Step keys: - -* `chroot` — REQUIRED; value is the tag for the root filesystem. - -* `shell` — REQUIRED; the shell snippet to run - -Example (in the .vmdb file): - - - chroot: root - shell: | - echo I am in chroot - - -Step: `shell` ------------------------------------------------------------------------------ - -Run a shell snippet on the host. This is not run in a chroot, and can -access the host system. - -Step keys: - -* `root-fs` — REQUIRED; value is the tag for the root filesystem. - -* `shell` — REQUIRED; the shell snippet to run - -Example (in the .vmdb file): - - - root-fs: root - shell: | - echo I am in NOT in chroot. diff --git a/vmdb/plugins/chroot_plugin.py b/vmdb/plugins/chroot_plugin.py deleted file mode 100644 index 9e4fee0..0000000 --- a/vmdb/plugins/chroot_plugin.py +++ /dev/null @@ -1,58 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os - -import cliapp - -import vmdb - - -class ChrootPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(ChrootStepRunner()) - self.app.step_runners.add(ShellStepRunner()) - - -class ChrootStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['chroot', 'shell'] - - def run(self, step, settings, state): - fs_tag = step['chroot'] - shell = step['shell'] - - mount_point = state.tags.get_mount_point(fs_tag) - vmdb.runcmd_chroot(mount_point, ['sh', '-c', shell]) - - -class ShellStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['shell', 'root-fs'] - - def run(self, step, settings, state): - shell = step['shell'] - fs_tag = step['root-fs'] - - env = dict(os.environ) - env['ROOT'] = state.tags.get_mount_point(fs_tag) - vmdb.runcmd(['sh', '-c', shell], env=env) diff --git a/vmdb/plugins/debootstrap.mdwn b/vmdb/plugins/debootstrap.mdwn deleted file mode 100644 index 23a56c9..0000000 --- a/vmdb/plugins/debootstrap.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -Step: `debootstrap` ------------------------------------------------------------------------------ - -Install packages using apt, which needs to already have been -installed. - -Step keys: - -* `debootstrap` — REQUIRED; value is the codename of the Debian - release to install: `stretch`, `buster`, etc. - -* `target` — REQUIRED; value is the tag for the root filesystem. - -* `mirror` — OPTIONAL; which Debian mirror to use - -Example (in the .vmdb file): - - - debootstrap: stretch - target: root - mirror: http://mirror.example.com/debian diff --git a/vmdb/plugins/debootstrap_plugin.py b/vmdb/plugins/debootstrap_plugin.py deleted file mode 100644 index f52f718..0000000 --- a/vmdb/plugins/debootstrap_plugin.py +++ /dev/null @@ -1,49 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import cliapp - -import vmdb - - -class DebootstrapPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(DebootstrapStepRunner()) - - -class DebootstrapStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['debootstrap', 'target', 'mirror'] - - def run(self, step, settings, state): - suite = step['debootstrap'] - tag = step['target'] - target = state.tags.get_mount_point(tag) - mirror = step['mirror'] - variant = step.get('variant', '-') - if not (suite and tag and target and mirror): - raise Exception('missing arg for debootstrap step') - vmdb.runcmd(['debootstrap', '--variant', variant, suite, target, mirror]) - - def run_even_if_skipped(self, step, settings, state): - tag = step['target'] - target = state.tags.get_mount_point(tag) - vmdb.runcmd_chroot(target, ['apt-get', 'update']) diff --git a/vmdb/plugins/echo_plugin.py b/vmdb/plugins/echo_plugin.py deleted file mode 100644 index c09a387..0000000 --- a/vmdb/plugins/echo_plugin.py +++ /dev/null @@ -1,46 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import logging - -import cliapp - -import vmdb - - -class EchoPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(EchoStepRunner()) - - -class EchoStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['echo'] - - def run(self, step, settings, state): - text = step['echo'] - vmdb.progress('{}'.format(text)) - - def teardown(self, step, settings, state): - if 'teardown' in step: - text = step['teardown'] - vmdb.progress('{}'.format(text)) - logging.info('%s', text) diff --git a/vmdb/plugins/error_plugin.py b/vmdb/plugins/error_plugin.py deleted file mode 100644 index e40e1ec..0000000 --- a/vmdb/plugins/error_plugin.py +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import cliapp - -import vmdb - - -class ErrorPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(ErrorStepRunner()) - - -class ErrorStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['error', 'teardown'] - - def run(self, step, settings, state): - # We use vmdb.progress here to get output to go to stdout, - # instead of stderr. We want that for tests. - vmdb.progress('{}'.format(step['error'])) - raise vmdb.StepError('an error occurred') - - def teardown(self, step, settings, state): - # We use vmdb.progress here to get output to go to stdout, - # instead of stderr. We want that for tests. - vmdb.progress('{}'.format(step['teardown'])) diff --git a/vmdb/plugins/grub.mdwn b/vmdb/plugins/grub.mdwn deleted file mode 100644 index 1d69460..0000000 --- a/vmdb/plugins/grub.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -Step: `grub` ------------------------------------------------------------------------------ - -Install the GRUB bootloader to the image. Works on a PC, for -traditional BIOS booting or modern UEFI booting. Does not (yet?) -support Secure Boot. - -Warning: This is the least robust part of vmdb2. - -Step keys: - -* `grub` — REQUIRED; value MUST be one of `uefi` and `bios`, for - a UEFI or a BIOS boot, respectively. (FIXME: these are valid for a - PC; not sure what other archs require, if grub even works there.) - -* `tag` — REQUIRED; value is the tag for the root filesystem. - -* `efi` — REQUIRED for UEFI; value is the tag for the EFI - filesystem. - -* `console` — OPTIONAL; set to `serial` to configure the image - to use a serial console. - -* `device` — OPTIONAL; which device to install GRUB onto; this - is needed when installing to a real hard drive, instead of an image. - -Example (in the .vmdb file): - - - grub: bios - tag: root - -Same, but for UEFI: - - - grub: uefi - tag: root - efi: efi - console: serial - -Install to a real hard disk (named with the `--image` option): - - - grub: uefi - tag: root - efi: efi - image-dev: "{{ image }}" diff --git a/vmdb/plugins/grub_plugin.py b/vmdb/plugins/grub_plugin.py deleted file mode 100644 index 64522ec..0000000 --- a/vmdb/plugins/grub_plugin.py +++ /dev/null @@ -1,263 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -# Installing GRUB onto a disk image is a bit of a black art. I haven't -# found any good documentation for it. This plugin is written based on -# de-ciphering the build_openstack_image script. Here is an explanation -# of what I _THINK_ is happening. -# -# The crucial command is grub-install. It needs a ton of options to -# work correctly: see below in the code for the list, and the manpage -# for an explanation of what each of them means. We will be running -# grub-install in a chroot so that we use the version in the Debian -# version we're installing, rather than the host system, which might -# be any Debian version. -# -# To run grub-install in a chroot, we need to set up the chroot in -# various ways. Firstly, we need to tell grub-install which device -# file the image has. We can't just give it the image file itself, -# since it isn't inside the chroot, so instead we arrange to have a -# loop block device that covers the whole image file, and we bind -# mount /dev into the chroot so the device is available. -# -# grub-install seems to also require /proc and /sys so we bind mount -# those into the chroot as well. -# -# We install the UEFI version of GRUB, and for that we additionally -# bind mount the EFI partition in the image. Oh yeah, you MUST have -# one. -# -# We also make sure the right GRUB package is installed in the chroot, -# before we run grub-install. -# -# Further, there's some configuration tweaking we need to do. See the -# code. Don't ask me why they're necessary. -# -# For cleanliness, we also undo any bind mounts into the chroot. Don't -# want to leave them in case they cause trouble. -# -# Note that this is currently rather strongly assuming that UEFI and -# the amd64 (a.k.a. x86_64) architecture are being used. These should -# probably not be hardcoded. Patch welcome. - -# To use this plugin: write steps to create a root filesystem, and an -# VFAT filesystem to be mounted as /boot/efi. Install Debian onto the -# root filesystem. Then install grub with a step like this: -# -# - grub: uefi -# tag: root-part -# efi: efi-part -# -# Here: "tag" is the tag for the root filesystem (and corresponding -# partition), and efi is tag for the EFI partition. -# -# The grub step will take of the rest. - - -import logging -import os -import re - -import cliapp - -import vmdb - - -class GrubPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(GrubStepRunner()) - - -class GrubStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['grub', 'tag'] - - def run(self, step, settings, state): - state.grub_mounts = [] - flavor = step['grub'] - if flavor == 'uefi': - self.install_uefi(step, settings, state) - elif flavor == 'bios': - self.install_bios(step, settings, state) - else: - raise Exception('Unknown GRUB flavor {}'.format(flavor)) - - def install_uefi(self, step, settings, state): - if not 'efi' in step and 'efi-part' not in step: - raise Exception('"efi" or "efi-part" required in UEFI GRUB installtion') - - vmdb.progress('Installing GRUB for UEFI') - grub_package = 'grub-efi-amd64' - grub_target = 'x86_64-efi' - self.install_grub(step, settings, state, grub_package, grub_target) - - def install_bios(self, step, settings, state): - vmdb.progress('Installing GRUB for BIOS') - grub_package = 'grub-pc' - grub_target = 'i386-pc' - self.install_grub(step, settings, state, grub_package, grub_target) - - def install_grub(self, step, settings, state, grub_package, grub_target): - console = step.get('console', None) - - tag = step.get('tag') - if tag is None: - tag = step['root-fs'] - root_dev = state.tags.get_dev(tag) - chroot = state.tags.get_mount_point(tag) - - image_dev = step.get('image-dev') - if image_dev is None: - image_dev = self.get_image_loop_device(root_dev) - - if 'efi' in step: - efi = step['efi'] - efi_dev = state.tags.get_dev(efi) - elif 'efi-part' in step: - efi = step['efi-part'] - efi_dev = state.tags.get_dev(efi) - else: - efi_dev = None - - self.bind_mount_many(chroot, ['/dev', '/proc', '/sys'], state) - if efi_dev: - self.mount(chroot, efi_dev, '/boot/efi', state) - self.install_package(chroot, grub_package) - - kernel_params = [ - 'biosdevname=0', - 'net.ifnames=0', - 'consoleblank=0', - 'systemd.show_status=true', - 'rw', - 'quiet', - 'systemd.show_status=false', - ] - if console == 'serial': - kernel_params.extend([ - 'quiet', - 'loglevel=3', - 'rd.systemd.show_status=false', - 'systemd.show_status=false', - 'console=tty0', - 'console=ttyS0,115200n8', - ]) - - self.set_grub_cmdline_config(chroot, kernel_params) - self.add_grub_crypto_disk(chroot) - if console == 'serial': - self.add_grub_serial_console(chroot) - - vmdb.runcmd_chroot(chroot, ['grub-mkconfig', '-o', '/boot/grub/grub.cfg']) - vmdb.runcmd_chroot( - chroot, [ - 'grub-install', - '--target=' + grub_target, - '--no-nvram', - '--force-extra-removable', - '--no-floppy', - '--modules=part_msdos part_gpt', - '--grub-mkdevicemap=/boot/grub/device.map', - image_dev, - ] - ) - -# self.unmount(state) - - def teardown(self, step, settings, state): - self.unmount(state) - - def unmount(self, state): - mounts = getattr(state, 'grub_mounts', []) - mounts.reverse() - while mounts: - mount_point = mounts.pop() - try: - vmdb.unmount(mount_point) - except vmdb.NotMounted as e: - logging.warning(str(e)) - - def get_image_loop_device(self, partition_device): - # We get /dev/mappers/loopXpY and return /dev/loopX - # assert partition_device.startswith('/dev/mapper/loop') - - m = re.match(r'^/dev/mapper/(?P<loop>.*)p\d+$', partition_device) - if m is None: - raise Exception('Do not understand partition device name {}'.format( - partition_device)) - assert m is not None - - loop = m.group('loop') - return '/dev/{}'.format(loop) - - def bind_mount_many(self, chroot, paths, state): - for path in paths: - self.mount(chroot, path, path, state, mount_opts=['--bind']) - - def mount(self, chroot, path, mount_point, state, mount_opts=None): - chroot_path = self.chroot_path(chroot, mount_point) - if not os.path.exists(chroot_path): - os.makedirs(chroot_path) - - if mount_opts is None: - mount_opts = [] - - vmdb.runcmd(['mount'] + mount_opts + [path, chroot_path]) - state.grub_mounts.append(chroot_path) - - def chroot_path(self, chroot, path): - return os.path.normpath(os.path.join(chroot, '.' + path)) - - def install_package(self, chroot, package): - env = os.environ.copy() - env['DEBIAN_FRONTEND'] = 'noninteractive' - vmdb.runcmd_chroot( - chroot, - ['apt-get', '-y', '--no-show-progress', 'install', package], - env=env) - - def set_grub_cmdline_config(self, chroot, kernel_params): - param_string = ' '.join(kernel_params) - - filename = self.chroot_path(chroot, '/etc/default/grub') - - with open(filename) as f: - text = f.read() - - lines = text.splitlines() - lines = [line for line in lines - if not line.startswith('GRUB_CMDLINE_LINUX_DEFAULT')] - lines.append('GRUB_CMDLINE_LINUX_DEFAULT="{}"'.format(param_string)) - - with open(filename, 'w') as f: - f.write('\n'.join(lines) + '\n') - - def add_grub_serial_console(self, chroot): - filename = self.chroot_path(chroot, '/etc/default/grub') - - with open(filename, 'a') as f: - f.write('GRUB_TERMINAL=serial\n') - f.write('GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 ' - '--word=8 --parity=no --stop=1"\n') - - def add_grub_crypto_disk(self, chroot): - filename = self.chroot_path(chroot, '/etc/default/grub') - with open(filename, 'a') as f: - f.write('GRUB_ENABLE_CRYPTODISK=y\n') diff --git a/vmdb/plugins/luks.mdwn b/vmdb/plugins/luks.mdwn deleted file mode 100644 index 4b80f63..0000000 --- a/vmdb/plugins/luks.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -Step: `luks` ------------------------------------------------------------------------------ - -Set up disk encryption using LUKS with the `cryptsetup` utility. The -encryption passphrase is read from a file or from the output of a -command. The encrypted disk gets opened and can be mounted using a -separate tag for the cleartext view. - -Step keys: - -* `cryptsetup` — REQUIRED; value is the tag for the encrypted - block device. This is not directly useable by users, or mountable. - -* `tag` — REQUIRED; the tag for the de-crypted block device. - This is what gets mounted and visible to users. - -* `key-file` — OPTIONAL; file from where passphrase is read. - -* `key-cmd` — OPTIONAL; command to run, passphrase is the first - line of its standard output. - -Example (in the .vmdb file): - - - cryptsetup: root - tag: root_crypt - key-file: disk.pass - -Same, except run a command to get passphrase (in this case -[pass](https://www.passwordstore.org/)): - - - cryptsetup: root - tag: root_crypt - key-cmd: pass show disk-encryption diff --git a/vmdb/plugins/luks_plugin.py b/vmdb/plugins/luks_plugin.py deleted file mode 100644 index 9d48d99..0000000 --- a/vmdb/plugins/luks_plugin.py +++ /dev/null @@ -1,86 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import logging -import os -import tempfile - -import cliapp - -import vmdb - - -class LuksPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(CryptsetupStepRunner()) - - -class CryptsetupStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['cryptsetup'] - - def run(self, step, settings, state): - underlying = step['cryptsetup'] - crypt_name = step['tag'] - - state.tmp_key_file = None - key_file = step.get('key-file') - key_cmd = step.get('key-cmd') - if key_file is None and key_cmd is None: - raise Exception( - 'cryptsetup step MUST define one of key-file or key-cmd') - - if key_file is None: - output = vmdb.runcmd(['sh', '-c', key_cmd]) - output = output.decode('UTF-8') - key = output.splitlines()[0] - fd, key_file = tempfile.mkstemp() - state.tmp_key_file = key_file - os.close(fd) - open(key_file, 'w').write(key) - - dev = state.tags.get_dev(underlying) - if dev is None: - for t in state.tags.get_tags(): - logging.debug( - 'tag %r dev %r mp %r', t, state.tags.get_dev(t), - state.tags.get_mount_point(t)) - assert 0 - - vmdb.runcmd(['cryptsetup', 'luksFormat', dev, key_file]) - vmdb.runcmd( - ['cryptsetup', 'open', '--type', 'luks', '--key-file', key_file, dev, - crypt_name]) - - crypt_dev = '/dev/mapper/{}'.format(crypt_name) - assert os.path.exists(crypt_dev) - state.tags.append(crypt_name) - state.tags.set_dev(crypt_name, crypt_dev) - - def teardown(self, step, settings, state): - x = state.tmp_key_file - if x is not None and os.path.exists(x): - os.remove(x) - - crypt_name = step['tag'] - - crypt_dev = '/dev/mapper/{}'.format(crypt_name) - vmdb.runcmd(['cryptsetup', 'close', crypt_dev]) diff --git a/vmdb/plugins/lvm2.mdwn b/vmdb/plugins/lvm2.mdwn deleted file mode 100644 index 76f5892..0000000 --- a/vmdb/plugins/lvm2.mdwn +++ /dev/null @@ -1,42 +0,0 @@ -Step: `vgcreate` ------------------------------------------------------------------------------ - -Create an LVM2 volume group (VG), and also initialise the physical -volumes for it. - - -Step keys: - -* `vgcreate` — REQUIRED; value is the tag for the volume group. - This gets initialised with `vgcreate`. - -* `physical` — REQUIRED; list of tags for block devices - (partitions) to use as physical volumes. These get initialised with - `pvcreate`. - -Example (in the .vmdb file): - - - vgcreate: rootvg - physical: - - my_partition - - other_partition - - -Step: `lvcreate` ------------------------------------------------------------------------------ - -Create an LVM2 logical volume (LV) in an existing volume group. - -Step keys: - -* `lvcreate` — REQUIRED; value is the tag for the volume group. - -* `name` — REQUIRED; tag for the new LV block device. - -* `size` — REQUIRED; size of the new LV. - -Example (in the .vmdb file): - - - lvcreate: rootvg - name: rootfs - size: 1G diff --git a/vmdb/plugins/lvm2_plugin.py b/vmdb/plugins/lvm2_plugin.py deleted file mode 100644 index 03a98c0..0000000 --- a/vmdb/plugins/lvm2_plugin.py +++ /dev/null @@ -1,76 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os - -import cliapp - -import vmdb - - -class Lvm2Plugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(VgcreateStepRunner()) - self.app.step_runners.add(LvcreateStepRunner()) - - -class VgcreateStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['vgcreate', 'physical'] - - def run(self, step, settings, state): - vgname = self.get_vg(step) - physical = self.get_pv(step, state) - - for phys in physical: - vmdb.runcmd(['pvcreate', '-ff', '--yes', phys]) - vmdb.runcmd(['vgcreate', vgname] + physical) - - def teardown(self, step, settings, state): - vgname = self.get_vg(step) - vmdb.runcmd(['vgchange', '-an', vgname]) - - def get_vg(self, step): - return step['vgcreate'] - - def get_pv(self, step, state): - return [ - state.tags.get_dev(tag) - for tag in step['physical'] - ] - - -class LvcreateStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['lvcreate'] - - def run(self, step, settings, state): - vgname = step['lvcreate'] - lvname = step['name'] - size = step['size'] - - vmdb.runcmd(['lvcreate', '--name', lvname, '--size', size, vgname]) - - lvdev = '/dev/{}/{}'.format(vgname, lvname) - assert os.path.exists(lvdev) - state.tags.append(lvname) - state.tags.set_dev(lvname, lvdev) diff --git a/vmdb/plugins/mkfs.mdwn b/vmdb/plugins/mkfs.mdwn deleted file mode 100644 index 87243cd..0000000 --- a/vmdb/plugins/mkfs.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Step: `mkfs` ------------------------------------------------------------------------------ - -Create a filesystem. - -Step keys: - -* `mkfs` — REQUIRED; filesystem type, such as `mkfs` or `vfat`. - -* `partition` — REQUIRED; tag for the block device to use. - -Example (in the .vmdb file): - - - mkfs: ext4 - partition: root diff --git a/vmdb/plugins/mkfs_plugin.py b/vmdb/plugins/mkfs_plugin.py deleted file mode 100644 index da24130..0000000 --- a/vmdb/plugins/mkfs_plugin.py +++ /dev/null @@ -1,51 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import cliapp - -import vmdb - - -class MkfsPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(MkfsStepRunner()) - - -class MkfsStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['mkfs', 'partition'] - - def run(self, step, settings, state): - fstype = step['mkfs'] - tag = step['partition'] - device = state.tags.get_dev(tag) - - cmd = ['/sbin/mkfs', '-t', fstype] - if 'label' in step: - if fstype == 'vfat': - cmd.append('-n') - elif fstype == 'f2fs': - cmd.append('-l') - else: - cmd.append('-L') - cmd.append(step['label']) - cmd.append(device) - vmdb.runcmd(cmd) diff --git a/vmdb/plugins/mkimg.mdwn b/vmdb/plugins/mkimg.mdwn deleted file mode 100644 index e2998ac..0000000 --- a/vmdb/plugins/mkimg.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Step: `mkimg` ------------------------------------------------------------------------------ - -Create a new image file of a desired size. - -Step keys: - -* `mkimage` — REQUIRED; name of file to create. - -* `size` — REQUIRED; size of the image. - -Example (in the .vmdb file): - - - mkimg: "{{ output }}" - size: 4G diff --git a/vmdb/plugins/mkimg_plugin.py b/vmdb/plugins/mkimg_plugin.py deleted file mode 100644 index 44fc57d..0000000 --- a/vmdb/plugins/mkimg_plugin.py +++ /dev/null @@ -1,43 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import cliapp - -import vmdb - - -class MkimgPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(MkimgStepRunner()) - self.app.settings.bytesize( - ['size'], - 'size of output image', - default='1GiB') - - -class MkimgStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['mkimg'] - - def run(self, step, settings, state): - filename = step['mkimg'] - size = step['size'] - vmdb.runcmd(['qemu-img', 'create', '-f', 'raw', filename, size]) diff --git a/vmdb/plugins/mount.mdwn b/vmdb/plugins/mount.mdwn deleted file mode 100644 index cf447f1..0000000 --- a/vmdb/plugins/mount.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -Step: `mount` ------------------------------------------------------------------------------ - -Mount a filesystem. - -Step keys: - -* `mount` — REQUIRED; tag of filesystem to mount. - -* `dirname` — OPTIONAL; the mount point. - -* `mount-on` — OPTIONAL; tag of already mounted filesystem in - image. (FIXME: this may be wrong?) - -Example (in the .vmdb file): - - - mount: root diff --git a/vmdb/plugins/mount_plugin.py b/vmdb/plugins/mount_plugin.py deleted file mode 100644 index 2ef3ef1..0000000 --- a/vmdb/plugins/mount_plugin.py +++ /dev/null @@ -1,85 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import logging -import os -import tempfile - -import cliapp - -import vmdb - - -class MountPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(MountStepRunner()) - - -class MountStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['mount'] - - def run(self, step, settings, state): - self.mount_rootfs(step, settings, state) - - def teardown(self, step, settings, state): - self.unmount_rootfs(step, settings, state) - - def mount_rootfs(self, step, settings, state): - tag = step['mount'] - dirname = step.get('dirname') - mount_on = step.get('mount-on') - - device = state.tags.get_dev(tag) - - if dirname: - if not mount_on: - raise Exception('no mount-on tag given') - - if not state.tags.has_tag(mount_on): - raise Exception('cannot find tag {}'.format(mount_on)) - - mount_point = os.path.join( - state.tags.get_mount_point(mount_on), './' + dirname) - - if not os.path.exists(mount_point): - os.makedirs(mount_point) - else: - mount_point = tempfile.mkdtemp() - - vmdb.runcmd(['mount', device, mount_point]) - state.tags.set_mount_point(tag, mount_point, cached=True) - - return mount_point - - def unmount_rootfs(self, step, settings, state): - tag = step['mount'] - mount_point = state.tags.get_mount_point(tag) - if mount_point is None: - return - - try: - vmdb.unmount(mount_point) - except vmdb.NotMounted as e: - logging.warning(str(e)) - - if not step.get('mount-on'): - os.rmdir(mount_point) diff --git a/vmdb/plugins/partition.mdwn b/vmdb/plugins/partition.mdwn deleted file mode 100644 index cc325d4..0000000 --- a/vmdb/plugins/partition.mdwn +++ /dev/null @@ -1,57 +0,0 @@ -Step: `mklabel` ------------------------------------------------------------------------------ - -Create a partition table on a block device. - -Step keys: - -* `mklabel` — REQUIRED; type of partition table, MUST be one of - `msdos` and `gpt`. - -* `device` — REQUIRED; tag for the block device. - -Example (in the .vmdb file): - - - mklabel: "{{ output }}" - size: 4G - -Step: `mkpart` ------------------------------------------------------------------------------ - -Create a partition. - -Step keys: - -* `mkpart` — REQUIRED; type of partition to create: use - `primary` (but any value acceped by `parted` is OK). - -* `device` — REQUIRED; filename of block device where to create - partition. - -* `start` — REQUIRED; where does the partition start? - -* `end` — REQUIRED; where does the partition end? - -* `tag` — REQUIRED; tag for the new partition. - -Example (in the .vmdb file): - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - tag: root - -Step: `kpartx` ------------------------------------------------------------------------------ - -Create loop devices for partitions in an image file. Not needed when -installing to a real block device, instead of an image file. - -Step keys: - -* `kpartx` — REQUIRED; filename of block device with partitions. - -Example (in the .vmdb file): - - - kpartx: "{{ output }}" diff --git a/vmdb/plugins/partition_plugin.py b/vmdb/plugins/partition_plugin.py deleted file mode 100644 index 60d4225..0000000 --- a/vmdb/plugins/partition_plugin.py +++ /dev/null @@ -1,121 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os -import stat - -import cliapp - -import vmdb - - -class PartitionPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(MklabelStepRunner()) - self.app.step_runners.add(MkpartStepRunner()) - self.app.step_runners.add(KpartxStepRunner()) - - -class MklabelStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['mklabel', 'device'] - - def run(self, step, settings, state): - label_type = step['mklabel'] - device = step['device'] - vmdb.runcmd(['parted', '-s', device, 'mklabel', label_type]) - - -class MkpartStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['mkpart', 'device', 'start', 'end'] - - def run(self, step, settings, state): - part_type = step['mkpart'] - device = step['device'] - start = step['start'] - end = step['end'] - tag = step.get('tag') or step.get('part-tag') - if tag is None: - tag = step['part-tag'] - fs_type = step.get('fs-type', 'ext2') - - orig = self.list_partitions(device) - vmdb.runcmd(['parted', '-s', device, 'mkpart', part_type, fs_type, start, end]) - - state.tags.append(tag) - if self.is_block_dev(device): - new = self.list_partitions(device) - diff = self.diff_partitions(orig, new) - assert len(diff) == 1 - vmdb.progress('remembering partition {} as {}'.format(diff[0], tag)) - state.tags.set_dev(tag, diff[0]) - - def is_block_dev(self, filename): - st = os.lstat(filename) - return stat.S_ISBLK(st.st_mode) - - def list_partitions(self, device): - output = vmdb.runcmd(['parted', '-m', device, 'print']) - output = output.decode('UTF-8') - partitions = [ - line.split(':')[0] - for line in output.splitlines() - if ':' in line - ] - return [ - word if word.startswith('/') else '{}{}'.format(device, word) - for word in partitions - ] - - def diff_partitions(self, old, new): - return [ - line - for line in new - if line not in old - ] - - -class KpartxStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['kpartx'] - - def run(self, step, settings, state): - device = step['kpartx'] - tags = state.tags.get_tags() - devs = self.kpartx(device) - for tag, dev in zip(tags, devs): - vmdb.progress('remembering {} as {}'.format(dev, tag)) - state.tags.set_dev(tag, dev) - - def kpartx(self, device): - output = vmdb.runcmd(['kpartx', '-asv', device]).decode('UTF-8') - for line in output.splitlines(): - words = line.split() - if words[0] == 'add': - name = words[2] - yield '/dev/mapper/{}'.format(name) - - def teardown(self, step, settings, state): - device = step['kpartx'] - vmdb.runcmd(['kpartx', '-dsv', device]) diff --git a/vmdb/plugins/qemudebootstrap.mdwn b/vmdb/plugins/qemudebootstrap.mdwn deleted file mode 100644 index 5d0234e..0000000 --- a/vmdb/plugins/qemudebootstrap.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -Step: `qemu-debootstrap` ------------------------------------------------------------------------------ - -Install packages using apt, which needs to already have been -installed, for a different architecture than the host where vmdb2 is -being run. For example, for building an image for a Raspberry Pi on an -Intel PC. - -Step keys: - -* `qemu-debootstrap` — REQUIRED; value is the codename of the Debian - release to install: `stretch`, `buster`, etc. - -* `target` — REQUIRED; value is the tag for the root filesystem. - -* `mirror` — OPTIONAL; which Debian mirror to use. - -* `arch` — REQUIRED; the foreign architecture touse. - -* `variant` — OPTIONAL; the variant for debootstrap. - -Example (in the .vmdb file): - - - qemu-debootstrap: stretch - target: root - mirror: http://mirror.example.com/debian - arch: arm64 - variant: buildd diff --git a/vmdb/plugins/qemudebootstrap_plugin.py b/vmdb/plugins/qemudebootstrap_plugin.py deleted file mode 100644 index 9a6ed97..0000000 --- a/vmdb/plugins/qemudebootstrap_plugin.py +++ /dev/null @@ -1,58 +0,0 @@ -# Copyright 2017 Lars Wirzenius and Stuart Prescott -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import cliapp - -import vmdb - - -class QemuDebootstrapPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(QemuDebootstrapStepRunner()) - - -class QemuDebootstrapStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['qemu-debootstrap', 'target', 'mirror', 'arch'] - - def run(self, step, settings, state): - suite = step['qemu-debootstrap'] - tag = step['target'] - target = state.tags.get_mount_point(tag) - mirror = step['mirror'] - variant = step.get('variant', '-') - arch = step['arch'] - components = step.get('components', ['main']) - if not (suite and tag and target and mirror and arch): - raise Exception('missing arg for qemu-debootstrap step') - vmdb.runcmd( - ['qemu-debootstrap', - '--arch', arch, - '--variant', variant, - '--components', ','.join(components), suite, - target, - mirror]) - vmdb.runcmd_chroot(target, ['apt-get', 'update']) - - def run_even_if_skipped(self, step, settings, state): - tag = step['target'] - target = state.tags.get_mount_point(tag) - vmdb.runcmd_chroot(target, ['apt-get', 'update']) diff --git a/vmdb/plugins/rootfs_cache.mdwn b/vmdb/plugins/rootfs_cache.mdwn deleted file mode 100644 index 3f4ee0b..0000000 --- a/vmdb/plugins/rootfs_cache.mdwn +++ /dev/null @@ -1,29 +0,0 @@ -Step: `cache-rootfs` ------------------------------------------------------------------------------ - -Create a tarball of the root filesystem in the image. - -Step keys: - -* `cache-rootfs` — REQUIRED; tag of root filesystem on image. - -Example (in the .vmdb file): - - - cache-rootfs: root - unless: rootfs_unpacked - - -Step: `unpack-rootfs` ------------------------------------------------------------------------------ - -Unpack a tarball of the root filesystem to the image, and set the -`rootfs_unpacked` condition to true. If the tarball doesn't exist, do -nothing and leave the `rootfs_unpacked` condition to false. - -Step keys: - -* `unpack-rootfs` — REQUIRED; tag for the root filesystem. - -Example (in the .vmdb file): - - - unpack-rootfs: root diff --git a/vmdb/plugins/rootfs_cache_plugin.py b/vmdb/plugins/rootfs_cache_plugin.py deleted file mode 100644 index edfb3d4..0000000 --- a/vmdb/plugins/rootfs_cache_plugin.py +++ /dev/null @@ -1,98 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import os - -import cliapp - -import vmdb - - -class RootFSCachePlugin(cliapp.Plugin): - - def enable(self): - self.app.settings.string( - ['rootfs-tarball'], - 'store rootfs cache tar archives in FILE', - metavar='FILE') - - self.app.step_runners.add(MakeCacheStepRunner()) - self.app.step_runners.add(UnpackCacheStepRunner()) - - -class MakeCacheStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['cache-rootfs'] - - def run(self, step, settings, state): - fs_tag = step['cache-rootfs'] - rootdir = state.tags.get_mount_point(fs_tag) - tar_path = settings['rootfs-tarball'] - opts = step.get('options', '--one-file-system').split() - if not tar_path: - raise Exception('--rootfs-tarball MUST be set') - dirs = self._find_cacheable_mount_points(state.tags, rootdir) - - tags = state.tags - for tag in tags.get_tags(): - vmdb.progress( - 'tag {} mounted {} cached {}'.format( - tag, tags.get_mount_point(tag), tags.is_cached(tag))) - - vmdb.progress('caching rootdir {}'.format(rootdir)) - vmdb.progress('caching relative {}'.format(dirs)) - if not os.path.exists(tar_path): - vmdb.runcmd( - ['tar'] + opts + ['-C', rootdir, '-caf', tar_path] + dirs) - - def _find_cacheable_mount_points(self, tags, rootdir): - return list(sorted( - self._make_relative(rootdir, tags.get_mount_point(tag)) - for tag in tags.get_tags() - if tags.is_cached(tag) - )) - - def _make_relative(self, rootdir, dirname): - assert dirname == rootdir or dirname.startswith(rootdir + '/') - if dirname == rootdir: - return '.' - return dirname[len(rootdir) + 1:] - - -class UnpackCacheStepRunner(vmdb.StepRunnerInterface): - - def get_required_keys(self): - return ['unpack-rootfs'] - - def run(self, step, settings, state): - fs_tag = step['unpack-rootfs'] - rootdir = state.tags.get_mount_point(fs_tag) - tar_path = settings['rootfs-tarball'] - if not tar_path: - raise Exception('--rootfs-tarball MUST be set') - if os.path.exists(tar_path): - vmdb.runcmd( - ['tar', '-C', rootdir, '-xf', tar_path, '--numeric-owner']) - self.copy_resolv_conf(rootdir) - state.rootfs_unpacked = True - - def copy_resolv_conf(self, rootdir): - filename = os.path.join(rootdir, 'etc', 'resolv.conf') - vmdb.runcmd(['cp', '/etc/resolv.conf', filename]) diff --git a/vmdb/plugins/virtuals_plugin.py b/vmdb/plugins/virtuals_plugin.py deleted file mode 100644 index 092c505..0000000 --- a/vmdb/plugins/virtuals_plugin.py +++ /dev/null @@ -1,77 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - - -import logging -import os - -import cliapp - -import vmdb - - -class VirtualFilesystemMountPlugin(cliapp.Plugin): - - def enable(self): - self.app.step_runners.add(VirtualFilesystemMountStepRunner()) - - -class VirtualFilesystemMountStepRunner(vmdb.StepRunnerInterface): - - virtuals = [ - ['none', '/proc', 'proc'], - ['none', '/dev', 'devtmpfs'], - ['none', '/dev/pts', 'devpts'], - ['none', '/dev/shm', 'tmpfs'], - ['none', '/run', 'tmpfs'], - ['none', '/run/lock', 'tmpfs'], - ['none', '/sys', 'sysfs'], - ] - - def get_required_keys(self): - return ['mount-virtual-filesystems'] - - def run(self, step, settings, state): - fstag = step['mount-virtual-filesystems'] - mount_point = state.tags.get_mount_point(fstag) - self.mount_virtuals(mount_point, state) - - def teardown(self, step, settings, state): - self.unmount_virtuals(state) - - def mount_virtuals(self, rootfs, state): - if not hasattr(state, 'virtuals'): - state.virtuals = [] - - for device, mount_point, fstype in self.virtuals: - path = os.path.join(rootfs, './' + mount_point) - if not os.path.exists(path): - os.mkdir(path) - vmdb.runcmd(['mount', '-t', fstype, device, path]) - state.virtuals.append(path) - logging.debug('mounted virtuals: %r', state.virtuals) - - def unmount_virtuals(self, state): - logging.debug('unmounting virtuals: %r', state.virtuals) - for mount_point in reversed(state.virtuals): - try: - vmdb.unmount(mount_point) - except vmdb.NotMounted as e: - logging.warning(str(e)) - except cliapp.AppException: - vmdb.warning('Something went wrong while unmounting. Ignoring.') diff --git a/vmdb/runcmd.py b/vmdb/runcmd.py deleted file mode 100644 index e1d88a5..0000000 --- a/vmdb/runcmd.py +++ /dev/null @@ -1,68 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import logging -import os -import sys - -import cliapp - - -_verbose = False - - -def set_verbose_progress(verbose): - global _verbose - _verbose = verbose - - -def error(msg): - logging.error(msg, exc_info=True) - if _verbose: - sys.stderr.write('ERROR: {}\n'.format(msg)) - - -def progress(msg): - logging.info(msg) - if _verbose: - sys.stdout.write('{}\n'.format(msg)) - - -def runcmd(argv, *argvs, **kwargs): - progress('Exec: %r' % (argv,)) - kwargs['stdout_callback'] = _log_stdout - kwargs['stderr_callback'] = _log_stderr - env = kwargs.get('env', os.environ.copy()) - env['LC_ALL'] = 'C' - kwargs['env'] = env - return cliapp.runcmd(argv, *argvs, **kwargs) - - -def runcmd_chroot(chroot, argv, *argvs, **kwargs): - full_argv = ['chroot', chroot] + argv - return runcmd(full_argv, *argvs, **kwargs) - - -def _log_stdout(data): - logging.debug('STDOUT: %r', data) - return data - - -def _log_stderr(data): - logging.debug('STDERR: %r', data) - return data diff --git a/vmdb/spec.py b/vmdb/spec.py deleted file mode 100644 index 1f3fac9..0000000 --- a/vmdb/spec.py +++ /dev/null @@ -1,51 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import jinja2 -import yaml - - -class Spec: - - def __init__(self): - self._dict = None - - def load_file(self, filename): - with open(filename) as f: - self._dict = yaml.safe_load(f) - - def as_dict(self): - return dict(self._dict) - - def get_steps(self, params): - return expand_templates(self._dict['steps'], params) - - -def expand_templates(value, params): - if isinstance(value, str): - template = jinja2.Template(value) - return template.render(**params) - elif isinstance(value, list): - return [expand_templates(x, params) for x in value] - elif isinstance(value, dict): - return { - key: expand_templates(value[key], params) - for key in value - } - else: - assert 0, 'Unknown value type: {!r}'.format(value) diff --git a/vmdb/spec_tests.py b/vmdb/spec_tests.py deleted file mode 100644 index 635079a..0000000 --- a/vmdb/spec_tests.py +++ /dev/null @@ -1,109 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import io -import os -import tempfile -import unittest - -import yaml - -import vmdb - - -class SpecTests(unittest.TestCase): - - spec_yaml = ''' - steps: - - step: foo - arg: "{{ var1 }}" - - step: bar - ''' - - def setUp(self): - self.filename = write_temp_file(bytes(self.spec_yaml, 'ascii')) - self.spec = vmdb.Spec() - - def tearDown(self): - if os.path.exists(self.filename): - os.remove(self.filename) - - def test_loads_spec(self): - self.spec.load_file(self.filename) - self.assertEqual(self.spec.as_dict(), as_dict(self.spec_yaml)) - - def test_expands_templates(self): - self.spec.load_file(self.filename) - params = { - 'var1': 'value1', - } - steps = self.spec.get_steps(params) - self.assertEqual( - steps, - [ - { - 'step': 'foo', - 'arg': 'value1', - }, - { - 'step': 'bar', - }, - ] - ) - -class ExpandTemplatesTests(unittest.TestCase): - - def test_raises_assert_if_given_incomprehensible_value(self): - with self.assertRaises(AssertionError): - vmdb.expand_templates(None, {}) - - def test_returns_same_given_string_without_template(self): - self.assertEqual(vmdb.expand_templates('foo', {}), 'foo') - - def test_expands_simple_string_template(self): - params = { - 'foo': 'bar', - } - self.assertEqual(vmdb.expand_templates('{{ foo }}', params), 'bar') - - def test_expands_list_of_templates(self): - params = { - 'foo': 'bar', - } - self.assertEqual(vmdb.expand_templates(['{{ foo }}'], params), ['bar']) - - def test_expands_dict_of_templates(self): - params = { - 'foo': 'bar', - } - self.assertEqual( - vmdb.expand_templates({'key': '{{ foo }}'}, params), - {'key': 'bar'} - ) - - -def write_temp_file(data): - fd, filename = tempfile.mkstemp() - os.write(fd, data) - os.close(fd) - return filename - - -def as_dict(yaml_text): - with io.StringIO(yaml_text) as f: - return yaml.safe_load(f) diff --git a/vmdb/state.py b/vmdb/state.py deleted file mode 100644 index 88fde39..0000000 --- a/vmdb/state.py +++ /dev/null @@ -1,30 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -class State: - - def __init__(self): - self._attrs = {} # make sure this attribute exists - self._attrs = self.as_dict() - - def as_dict(self): - return { - key: getattr(self, key) - for key in dir(self) - if not key in self._attrs - } diff --git a/vmdb/step_list.py b/vmdb/step_list.py deleted file mode 100644 index f69b4bc..0000000 --- a/vmdb/step_list.py +++ /dev/null @@ -1,86 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import cliapp - - -class StepRunnerInterface: # pragma: no cover - - def get_required_keys(self): - raise NotImplementedError() - - def run(self, step_spec, settings, state): - raise NotImplementedError() - - def run_even_if_skipped(self, step_spec, settings, state): - pass - - def teardown(self, step_spec, settings, state): - # Default implementation does nop, so that sub-classes don't - # need to have a nop teardown. - pass - - def skip(self, step_spec, settings, state): - # Return true if step should be skipped and not run. Does not - # apply to teardowns. - - # Skipping is indicated by the step having a field 'unless', - # which is either the name of a variable (field in state), or - # a list of such names. If all variables have a value that - # evaluates as truth, the step is skipped. - - value = step_spec.get('unless', None) - if value is None: - return False - - if isinstance(value, list): - return all(getattr(state, field, False) for field in value) - return getattr(state, value, False) - - -class StepRunnerList: - - def __init__(self): - self._runners = [] - - def __len__(self): - return len(self._runners) - - def add(self, runner): - self._runners.append(runner) - - def find(self, step_spec): - actual = set(step_spec.keys()) - for runner in self._runners: - required = set(runner.get_required_keys()) - if actual.intersection(required) == required: - return runner - raise NoMatchingRunner(actual) - - -class StepError(cliapp.AppException): - - pass - - -class NoMatchingRunner(cliapp.AppException): - - def __init__(self, keys): - super(NoMatchingRunner, self).__init__( - 'No runner implements step with keys {}'.format( - ', '.join(keys))) diff --git a/vmdb/step_list_tests.py b/vmdb/step_list_tests.py deleted file mode 100644 index 4d052d7..0000000 --- a/vmdb/step_list_tests.py +++ /dev/null @@ -1,57 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import unittest - -import vmdb - - -class StepRunnerListTests(unittest.TestCase): - - def test_is_empty_initially(self): - steps = vmdb.StepRunnerList() - self.assertEqual(len(steps), 0) - - def test_adds_a_runner(self): - steps = vmdb.StepRunnerList() - runner = DummyStepRunner() - steps.add(runner) - self.assertEqual(len(steps), 1) - - def test_finds_correct_runner(self): - steps = vmdb.StepRunnerList() - runner = DummyStepRunner() - steps.add(runner) - found = steps.find({'foo': None, 'bar': None}) - self.assertEqual(runner, found) - - def test_raises_error_if_runner_not_found(self): - steps = vmdb.StepRunnerList() - runner = DummyStepRunner() - steps.add(runner) - with self.assertRaises(vmdb.NoMatchingRunner): - steps.find({'foo': None}) - - -class DummyStepRunner(vmdb.StepRunnerInterface): - - def run(self, *args): - pass - - def get_required_keys(self): - return ['foo', 'bar'] diff --git a/vmdb/tags.py b/vmdb/tags.py deleted file mode 100644 index e9d73b3..0000000 --- a/vmdb/tags.py +++ /dev/null @@ -1,99 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -# Unmount a directory, including any mount points under that -# directory. If /mnt/foo is given, and /mnt/foo/bar is also mounted, -# unmount /mnt/foo/bar first, and /mnt/foo then. Look for sub-mounts -# in /proc/mounts. - - -class Tags: - - def __init__(self): - self._tags = {} - self._tagnames = [] - - def get_tags(self): - return list(self._tags.keys()) - - def has_tag(self, tag): - return tag in self._tags - - def get_dev(self, tag): - item = self._get(tag) - return item['dev'] - - def get_mount_point(self, tag): - item = self._get(tag) - return item['mount_point'] - - def is_cached(self, tag): - item = self._get(tag) - return item.get('cached', False) - - def append(self, tag): - if tag in self._tags: - raise TagInUse(tag) - self._tagnames.append(tag) - self._tags[tag] = { - 'dev': None, - 'mount_point': None, - } - - def set_dev(self, tag, dev): - item = self._get(tag) - if item['dev'] is not None: - raise AlreadyHasDev(tag) - item['dev'] = dev - - def set_mount_point(self, tag, mount_point, cached=False): - item = self._get(tag) - if item['mount_point'] is not None: - raise AlreadyMounted(tag) - item['mount_point'] = mount_point - item['cached'] = cached - - def _get(self, tag): - item = self._tags.get(tag) - if item is None: - raise UnknownTag(tag) - return item - - -class TagInUse(Exception): - - def __init__(self, tag): - super().__init__('Tag already used: {}'.format(tag)) - - -class UnknownTag(Exception): - - def __init__(self, tag): - super().__init__('Unknown tag: {}'.format(tag)) - - -class AlreadyHasDev(Exception): - - def __init__(self, tag): - super().__init__('Already has device: {}'.format(tag)) - - -class AlreadyMounted(Exception): - - def __init__(self, tag): - super().__init__('Already mounted tag: {}'.format(tag)) diff --git a/vmdb/tags_tests.py b/vmdb/tags_tests.py deleted file mode 100644 index 928b875..0000000 --- a/vmdb/tags_tests.py +++ /dev/null @@ -1,110 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import unittest - - -import vmdb - - -class TagsTests(unittest.TestCase): - - def test_lists_not_tags_initally(self): - tags = vmdb.Tags() - self.assertEqual(tags.get_tags(), []) - - def test_tells_if_tag_exists(self): - tags = vmdb.Tags() - self.assertFalse(tags.has_tag('foo')) - tags.append('foo') - self.assertTrue(tags.has_tag('foo')) - self.assertEqual(tags.get_tags(), ['foo']) - - def test_remembers_order(self): - tags = vmdb.Tags() - tags.append('foo') - tags.append('bar') - self.assertTrue(tags.get_tags(), ['foo', 'bar']) - - def test_get_dev_raises_error_for_unknown_tag(self): - tags = vmdb.Tags() - with self.assertRaises(vmdb.UnknownTag): - tags.get_dev('does-not-exist') - - def test_get_mount_point_raises_error_for_unknown_tag(self): - tags = vmdb.Tags() - with self.assertRaises(vmdb.UnknownTag): - tags.get_mount_point('does-not-exist') - - def test_raises_error_for_reused_tag(self): - tags = vmdb.Tags() - tags.append('tag') - with self.assertRaises(vmdb.TagInUse): - tags.append('tag') - - def test_sets_dev(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_dev('first', '/dev/foo') - self.assertEqual(tags.get_tags(), ['first']) - self.assertEqual(tags.get_dev('first'), '/dev/foo') - self.assertEqual(tags.get_mount_point('first'), None) - - def test_adds_mount_point(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_mount_point('first', '/mnt/foo') - self.assertEqual(tags.get_tags(), ['first']) - self.assertEqual(tags.get_dev('first'), None) - self.assertEqual(tags.get_mount_point('first'), '/mnt/foo') - - def test_mount_point_is_uncached_by_default(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_mount_point('first', '/mnt/foo') - self.assertFalse(tags.is_cached('first')) - - def test_mount_point_can_be_made_cached(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_mount_point('first', '/mnt/foo', cached=True) - self.assertTrue(tags.is_cached('first')) - - def test_set_dev_raises_error_for_unknown_tag(self): - tags = vmdb.Tags() - with self.assertRaises(vmdb.UnknownTag): - tags.set_dev('first', '/mnt/foo') - - def test_set_mount_point_raises_error_for_unknown_tag(self): - tags = vmdb.Tags() - with self.assertRaises(vmdb.UnknownTag): - tags.set_mount_point('first', '/mnt/foo') - - def test_set_mount_point_raises_error_for_double_mount(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_mount_point('first', '/mnt/foo') - with self.assertRaises(vmdb.AlreadyMounted): - tags.set_mount_point('first', '/mnt/foo') - - def test_set_dev_raises_error_for_double_dev(self): - tags = vmdb.Tags() - tags.append('first') - tags.set_dev('first', '/dev/foo') - with self.assertRaises(vmdb.AlreadyHasDev): - tags.set_dev('first', '/dev/foo') diff --git a/vmdb/unmount.py b/vmdb/unmount.py deleted file mode 100644 index 1d9aaa5..0000000 --- a/vmdb/unmount.py +++ /dev/null @@ -1,77 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -# Unmount a directory, including any mount points under that -# directory. If /mnt/foo is given, and /mnt/foo/bar is also mounted, -# unmount /mnt/foo/bar first, and /mnt/foo then. Look for sub-mounts -# in /proc/mounts. - - -import vmdb - - -def unmount(what, mounts=None, real_unmount=None): - if mounts is None: # pragma: no cover - mounts = _read_proc_mounts() - if real_unmount is None: # pragma: no cover - real_unmount = _real_unmount - - mounts = _parse_proc_mounts(mounts) - dirnames = _find_what_to_unmount(mounts, what) - for dirname in dirnames: - real_unmount(dirname) - - -def _read_proc_mounts(): # pragma: no cover - with open('/proc/mounts') as f: - return f.read() - - -def _real_unmount(what): # pragma: no cover - vmdb.runcmd(['umount', what]) - - -def _parse_proc_mounts(text): - return [ - line.split()[:2] - for line in text.splitlines() - ] - - -def _find_what_to_unmount(mounts, what): - dirname = _find_mount_point(mounts, what) - dirnameslash = dirname + '/' - to_unmount = [ - point - for dev, point in mounts - if point == dirname or point.startswith(dirnameslash) - ] - return list(reversed(sorted(to_unmount))) - - -def _find_mount_point(mounts, what): - for dev, point in mounts: - if what in (dev, point): - return point - raise NotMounted(what) - - -class NotMounted(Exception): - - def __init__(self, what): - super().__init__('Not mounted: {}'.format(what)) diff --git a/vmdb/unmount_tests.py b/vmdb/unmount_tests.py deleted file mode 100644 index 1ea3662..0000000 --- a/vmdb/unmount_tests.py +++ /dev/null @@ -1,78 +0,0 @@ -# Copyright 2018 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -import unittest - - -import vmdb - - -class UnmountTests(unittest.TestCase): - - def setUp(self): - self.mounts = ProcMounts() - - def unmount(self, what): - vmdb.unmount( - what, - mounts=str(self.mounts), - real_unmount=self.mounts.unmount) - - def test_raises_error_if_not_mounted(self): - with self.assertRaises(vmdb.NotMounted): - self.unmount('/foo') - - def test_unmounts_mounted_dir(self): - self.mounts.mount('/dev/foo', '/foo') - self.unmount('/foo') - self.assertFalse(self.mounts.is_mounted('/foo')) - - def test_unmounts_mounted_dir_with_submounts(self): - self.mounts.mount('/dev/foo', '/foo') - self.mounts.mount('/dev/bar', '/foo/bar') - self.unmount('/foo') - self.assertFalse(self.mounts.is_mounted('/foo')) - self.assertFalse(self.mounts.is_mounted('/foo/bar')) - - -class ProcMounts: - - def __init__(self): - self.mounts = [] - - def is_mounted(self, what): - return any(what in mount for mount in self.mounts) - - def mount(self, device, point): - self.mounts.append((device, point)) - - def unmount(self, what): - self.mounts = [ - mount - for mount in self.mounts - if what not in mount - ] - - def __str__(self): - return ''.join( - '{}\n'.format(self.mount_line(mount)) - for mount in self.mounts - ) - - def mount_line(self, mount): - return '{} {} fstype options 0 0'.format(mount[0], mount[1]) diff --git a/vmdb/version.py b/vmdb/version.py deleted file mode 100644 index b5bdb0d..0000000 --- a/vmdb/version.py +++ /dev/null @@ -1,2 +0,0 @@ -__version__ = "0.13.2+git" -__version_info__ = (0, 13, 2, '+git') @@ -1,5 +0,0 @@ -#!/usr/bin/python3 - -import vmdb - -vmdb.Vmdb2(version=vmdb.__version__).run() diff --git a/vmdb2.1.in b/vmdb2.1.in deleted file mode 100644 index d4b2eed..0000000 --- a/vmdb2.1.in +++ /dev/null @@ -1,50 +0,0 @@ -.\" Copyright 2017 Lars Wirzenius <liw@liw.fi> -.\" -.\" This program is free software: you can redistribute it and/or modify -.\" it under the terms of the GNU General Public License as published by -.\" the Free Software Foundation, either version 3 of the License, or -.\" (at your option) any later version. -.\" -.\" This program is distributed in the hope that it will be useful, -.\" but WITHOUT ANY WARRANTY; without even the implied warranty of -.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -.\" GNU General Public License for more details. -.\" -.\" You should have received a copy of the GNU General Public License -.\" along with this program. If not, see <http://www.gnu.org/licenses/>. -.\" -.TH VMDB2 1 -.SH NAME -vmdb2 \- create a disk image with Debian installed -.SH SYNOPSIS -.B vmdb -.RI [ options ] -.IR SPECFILE ... -.SH DESCRIPTION -.B vmdb2 -reads a -.I "specification file" -that describes a disk image, -and builds the corresponding image. -The specification file uses YAML syntax, -and contains a list of steps. -Each step tells vmdb2 what to do next. -.SH OPTIONS -.SH BUGS -This is unfinished at this time. -.SH EXAMPLE -To build a disk image: -.PP -.nf -.RS -vmdb2 foo.vmdb -.RE -.fi -.SH "SEE ALSO" -.BR vmdebootstrap (1). -.PP -See the -.B vmdb2 -manual and test suite for a full explanation of specification files. -This may be installed as -.IR /usr/share/doc/vmdb2/manual.html .
\ No newline at end of file diff --git a/vmdb2.css b/vmdb2.css deleted file mode 100644 index 7c5c4dd..0000000 --- a/vmdb2.css +++ /dev/null @@ -1,33 +0,0 @@ -html { - font-family: "FreeSerif", serif; -} - -h1, h2, h3, h4, h5, h6 { - font-family: "FreeSans", sans; -} - -header h1.title { - display: block; - text-align: center; - font-family: "FreeSans", sans; - font-size: 200%; -} - -header h2.author { - display: block; - text-align: center; - font-family: "FreeSans", sans; - font-size: 150%; -} - - -header h3.date { - display: block; - text-align: center; - font-family: "FreeSans", sans; - font-size: 150%; -} - -pre { - margin-left: 4em; -} diff --git a/vmdb2.mdwn b/vmdb2.mdwn deleted file mode 100644 index a3a2a5d..0000000 --- a/vmdb2.mdwn +++ /dev/null @@ -1,182 +0,0 @@ ---- -title: Building Debian system images with vmdb2 -author: Lars Wirzenius -date: work-in-progress -... - -Introduction -============================================================================= - -vmdb2 builds disk images with Debian installed. The images can be used -for virtual machines, or can be written to USB flash memory devices, -and hardware computers can be booted off them. It is a successor of -the vmdebootstrap program, written by the same author, to fix a number -of architectural problems with the old program. The new program is not -compatible with the old one; that would've required keeping the -problems, as well. - -This manual is published as HTML at -<https://vmdb2-manual.liw.fi/> and as a PDF at -<https://vmdb2-manual.liw.fi/vmdb2.pdf>. - -Why vmdb2 given vmdebootstrap already existed ------------------------------------------------------------------------------ - -`vmdebootstrap` was the first attempt by Lars Wirzenius to write a -tool to build system images. It turned out to not be well designed. -Specifically, it was not easily extensible to be as flexible as a tool -of this sort should be. - -Why vmdb2 given other tools already exist ------------------------------------------------------------------------------ - -Lars likes to write tools for himself and had some free time. He -sometimes prefers to write his own tools rather than spend time and -energy evaluating and improving existing tools. He admits this is a -character flaw. - -Also, he felt ashamed of how messy `vmdebootstrap` turned out to be. - -If nobody else likes `vmdb2`, that just means Lars had some fun on his -own. - - -Installation -============================================================================= - -You can get vmdb2 by getting the source code from git: - - git clone git://git.liw.fi/vmdb2 - -You can then run it from the source tree: - - sudo /path/to/vmdb2/vmdb2 ... - -In Debian 10 ("buster") and its derivatives, you can also install the -vmdb2 package: - - apt install vmdb2 - -For any other systems, we have no instructions. If you figure it out, -please tell us how. - - -Getting started -============================================================================= - -vmdb2 works by reading specification file with instructions for how an -image should be built, using YAML syntax, and following those -instructions. A minimal specification file example: - - steps: - - mkimg: "{{ output }}" - size: 4G - - - mklabel: gpt - device: "{{ output }}" - - - mkpart: primary - device: "{{ output }}" - start: 0% - end: 100% - tag: root - - - kpartx: "{{ output }}" - - - mkfs: ext4 - partition: root - - - mount: root - - - debootstrap: stretch - mirror: http://deb.debian.org/debian - target: root - - - apt: install - packages: - - linux-image-amd64 - tag: root - - - grub: bios - tag: root - console: serial - -The above creates a four gigabyte file, creates a GPT partition table, -a single partition, with a filesystem, and installs Debian release -stretch onto it. It also installs a kernel, and a boot loader. - -To use this, save the specification into `test.vmdb`, and run the -following command: - - sudo vmdb2 test.vmdb --output test.img --verbose - -This will take a long time, mostly at the `debootstrap` step. - - -Tags ------------------------------------------------------------------------------ - -Instead of device filenames, vmdb2 steps refer to block devices inside -the image, and their mount points, by symbolic names called tags. Tags -are any names that the user likes, and vmdb2 does not assign meaning -to them. They're just strings. - - -Jinja2 expansion ------------------------------------------------------------------------------ - -To refer to the filename specified with the `--output` or `--image` -command line options, you can use [Jinja2](http://jinja.pocoo.org/) -templating. The variables `output` and `image` can be used. - - - mkimg: "{{ output }}" - - - mklabel: "{{ image }}" - -The difference is that `--output` creates a new file, or truncates an -existing file, whereas `--images` requires the file to already exist. -The former is better for image file, the latter for real block -devices. - - -Speed up image creasing by caching the root filesystem ------------------------------------------------------------------------------ - -Building an image can take several minutes, and that's with fast -access to a Debian mirror and an SSD. The slowest part is typically -running debootstrap, and that always results in the same output, for a -given Debian release. This means its easy to cache. - -vmdb2 has the two actions `cache-roots` and `unpack-rootfs` and the -command line option `--rootfs-tarball` to allow user to cache. Thhe -user uses the option to name a file. `cache-rootfs` takes the root -filesystem and stores it into the file as a compress tar archive -("tarball"). `unpack-rootfs` unpacks the tarball. This allows vmdb2 to -skip running debootstrap needlessly. - -The specify which steps should be skipped, the `unless` field can be -used: `unpack-rootfs` sets the `rootfs_unpacked` flag if it actually -unpacks a tarball, and `unless` allows checking for that flag. If the -tarball doesn't exist, the flag is not set. - - - unpack-rootfs: root - - - debootstrap: stretch - target: root - unless: rootfs_unpacked - - - cache-rootfs: root - unless: rootfs_unpacked - -If the tarball exists, it's unpacked, and the `debootstrap` and -`cache-rootfs` steps are skipped. - -It's possible to have any number of steps between the unpack and the -cache steps. However, note that if you change the steps, you need to -delete the tarball to run them. - -TODO: unless, caching, tags, jinja2 - -Step reference manual -============================================================================= - diff --git a/without-tests b/without-tests deleted file mode 100644 index 5745e27..0000000 --- a/without-tests +++ /dev/null @@ -1,24 +0,0 @@ -yarns/lib.py -vmdb/__init__.py -vmdb/app.py -vmdb/runcmd.py -vmdb/state.py -vmdb/version.py -vmdb/plugins/__init__.py -vmdb/plugins/ansible_plugin.py -vmdb/plugins/apt_plugin.py -vmdb/plugins/chroot_plugin.py -vmdb/plugins/debootstrap_plugin.py -vmdb/plugins/echo_plugin.py -vmdb/plugins/error_plugin.py -vmdb/plugins/grub_plugin.py -vmdb/plugins/kernel_plugin.py -vmdb/plugins/lvm2_plugin.py -vmdb/plugins/luks_plugin.py -vmdb/plugins/mkfs_plugin.py -vmdb/plugins/mkimg_plugin.py -vmdb/plugins/mount_plugin.py -vmdb/plugins/qemudebootstrap_plugin.py -vmdb/plugins/partition_plugin.py -vmdb/plugins/rootfs_cache_plugin.py -vmdb/plugins/virtuals_plugin.py diff --git a/yarns/100-mvp.yarn b/yarns/100-mvp.yarn deleted file mode 100644 index b66c794..0000000 --- a/yarns/100-mvp.yarn +++ /dev/null @@ -1,168 +0,0 @@ ---- -title: vmdb2 MVP with echo and error -author: Lars Wirzenius -date: work in progress -... - - -Introduction -============================================================================= - -vmdb2 is a program for producing a disk image with Debian installed. -This document is a manual of sorts, and an automated test suite for it. - -[vmdebootstrap][] installs Debian onto a disk, or disk image. It is -like the [debootstrap][] tool, except the end result is a disk or disk -image, not a directory tree. vmdebootstrap takes care of creating -partitions, and filesystems, and allows some more customization than -vmdebootstrap does. - -vmdebootstrap is also a messy pile of kludge, and not flexible enough. -vmdb2 is a re-implementation from scratch, without a need for -backwards compatibility. It aims to provide more flexibility than -vmdeboostrap, without becoming anywhere near as complicated. Think of -vmdb2 as "vmdebootstrap the second generation". The name has changed -to allow the two tools to installable in paralllel. - -The main user-visible difference between vmdebootstrap and vmdb2 is -that the older program provides extensibitlity via a legion of command -line options and the newer program by having the user user a domain -specific language to express what kind of Debian system they want to -create. - -(Lars Wirzenius wrote both vmdebootstrap and vmdb2 and is entitled to -sneer at his younger self.) - -[vmdebootstrap]: http://liw.fi/vmdebootstrap/ -[debootstrap]: https://packages.debian.org/unstable/debootstrap - - -Contact ------------------------------------------------------------------------------ - -To make contact with the vmdb2 project you can email Lars directly -(`liw@liw.fi?`) or use the `#vmdb2` IRC channel on the irc.oftc.net -network. - - - -Specification files -============================================================================= - -A vmdb2 specification file is a YAML file that looks like this (this -is an imaginary example that doens't actually work right now): - - EXAMPLE - steps: - - mkimg: raw - size: 4G - - mkfs: ext4 - label: vmdb2rootfs - - debootstrap: jessie - - shell: | - rm -rf /usr/share/man - - adduser: jimbo - gecos: James Bond - shell: /bin/zsh - - sudo: jimbo - - ifupdown: eth0 - auto: yes - dhcp: yes - - apt: openssh-server - - convert: qcow2 - - compress: xz - -The list of steps produces the kind of image that the user wants (or -else an unholy mess). The specification file can easily be shared, and -put under version control. - -Every action in a step is provided by a plugin to vmdb2. Each action -(technically, "step runner") is a well-defined task, which may be -parameterised by some of the key/value pairs in the step. For example, -`mkimg` would create a disk image file. In the above example it is a -raw disk image file, as opposed to some other format. The image is 4 -gigabytes in size. `mkfs` creates an ext4 filesystem in the image -file; in thie example there are no partitions. And so on. - -Steps may need to clean up after themselves. For example, a step that -mounts a filesystem will need to unmount it at the end of the image -creation. Also, if a later step fails, then the unmount needs to -happen as well. This is called a "teardown". Some steps are provided -by a plugin that handles the teardown automatically, others may need -to provide instructions for the teardown in the specification file. - -By providing well-defined steps that the user may combine as they -wish, vmdb2 gives great flexibility without much complexity, but at -the cost of forcing the user to write a longer specification file than -a single vmdeboostrap command line. - - -A happy path -============================================================================= - -The first case we look at is one for the happy path: a specification -with two echo steps, and nothing else. It's very simple, and nothing -goes wrong when executing it. In addition to the actual thing to do, -each step may also define a "teardown" thing to do. For example, if -the step mounts a filesystem, the teardown would unmount it. - - SCENARIO happy path - GIVEN a specification file called happy.vmdb containing - ... { - ... steps: [ - ... { echo: "foo", teardown: "foo_teardown" }, - ... { echo: "bar", teardown: "bar_teardown" } - ... ] - ... } - WHEN user runs vmdb2 -v happy.vmdb - THEN exit code is 0 - AND stdout contains "foo" followed by "bar" - AND stdout contains "bar" followed by "bar_teardown" - AND stdout contains "bar_teardown" followed by "foo_teardown" - - -Jinja2 templating in specification file values -============================================================================= - -Vmdb2 allows values in specification files to be processed by the -Jinja2 templating engine. This allows users to do thing such as write -specifications that use configuration values to determine what -happens. For our simple echo/error steps, we will write a rule that -outputs the image file name given by the user. A more realistic -specification file would instead do thing like create the file. - - SCENARIO jinja2 templating - GIVEN a specification file called j2.vmdb containing - ... { - ... steps: [ - ... { echo: "image is {{ output }}" }, - ... { echo: "bar" }, - ... ] - ... } - WHEN user runs vmdb2 -v --output=foo.img j2.vmdb - THEN exit code is 0 - AND stdout contains "image is foo.img" followed by "bar" - - -Error handling -============================================================================= - -Sometimes things do not quite go as they should. What does vmdb2 do -then? - - SCENARIO error handling - GIVEN a specification file called unhappy.vmdb containing - ... { - ... steps: [ - ... { echo: "foo", teardown: "foo_teardown" }, - ... { error: "yikes", teardown: "WAT?!" }, - ... { echo: "bar_step", teardown: "bar_teardown" } - ... ] - ... } - WHEN user runs vmdb2 -v unhappy.vmdb - THEN exit code is 1 - AND stdout contains "foo" followed by "yikes" - AND stdout contains "yikes" followed by "WAT?!" - AND stdout contains "WAT?!" followed by "foo_teardown" - AND stdout does NOT contain "bar_step" - AND stdout does NOT contain "bar_teardown" diff --git a/yarns/900-implements.yarn b/yarns/900-implements.yarn deleted file mode 100644 index f8bc328..0000000 --- a/yarns/900-implements.yarn +++ /dev/null @@ -1,41 +0,0 @@ -IMPLEMENTS for all scenario steps -============================================================================= - -This chapter contains the implementations for all scenario steps. - - IMPLEMENTS GIVEN a specification file called (\S+) containing (.+) - filename = get_next_match() - spec = get_next_match() - open(filename, 'w').write(spec) - - IMPLEMENTS WHEN user runs vmdb2 (.*) - args = get_next_match() - vmdb2 = os.path.join(srcdir, 'vmdb2') - exit, out, err = cliapp.runcmd_unchecked([vmdb2] + args.split()) - vars['exit'] = exit - vars['stdout'] = out - vars['stderr'] = err - - IMPLEMENTS THEN exit code is (\d+) - wanted = int(get_next_match()) - exit = vars['exit'] - print 'exit code', exit - print 'stdout:', vars['stdout'] - print 'stderr:', vars['stderr'] - assertEqual(exit, wanted) - - IMPLEMENTS THEN stdout contains "(.+)" followed by "(.+)" - first = get_next_match() - second = get_next_match() - stdout = vars['stdout'] - first_i = stdout.find(first) - assertGreaterThan(first_i, 0) - rest = stdout[first_i + len(first):] - second_i = rest.find(second) - assertGreaterThan(second_i, -1) - - IMPLEMENTS THEN stdout does NOT contain "(\S+)" - what = get_next_match() - stdout = vars['stdout'] - i = stdout.find(what) - assertEqual(i, -1) diff --git a/yarns/Makefile b/yarns/Makefile deleted file mode 100644 index b50e685..0000000 --- a/yarns/Makefile +++ /dev/null @@ -1,28 +0,0 @@ -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# =*= License: GPL-3+ =*= - - -all: yarns.html - -yarns.pdf: $(yarns) Makefile - pandoc --chapters --toc -o yarns.pdf *.yarn - -yarns.html: $(yarns) Makefile ugly.css - pandoc -H ugly.css -f markdown+smart --toc \ - --top-level-division=chapter --number-sections \ - -V geometry:lettersize \ - --standalone --self-contained -o yarns.html *.yarn diff --git a/yarns/lib.py b/yarns/lib.py deleted file mode 100644 index 1171387..0000000 --- a/yarns/lib.py +++ /dev/null @@ -1,10 +0,0 @@ -import os -import sys - -import cliapp -from yarnutils import * - -srcdir = os.environ['SRCDIR'] -datadir = os.environ['DATADIR'] - -vars = Variables(datadir) diff --git a/yarns/ugly.css b/yarns/ugly.css deleted file mode 100644 index 78cd374..0000000 --- a/yarns/ugly.css +++ /dev/null @@ -1,79 +0,0 @@ -<style> -html { - background: white; - font-family: serif; - margin-left: 3em; - margin-right: 2em; - margin-top: 2em; -} - -form#searchform { - font-family: monospace; - text-align: right; -} - -div.actions { - font-family: monospace; - text-align: right; -} - -div.actions ul, div.actions li { - display: inline; -} - -div.pageheader { - font-family: monospace; - margin-bottom: 2em; -} - -div.pageheader span.title { - display: block; - font-size: 200%; - font-weight: bold; - font-family: sans-serif; - margin-top: 0.5em; -} - -div#pagebody { -} - -div.pagefooter { - font-family: monospace; - margin-top: 3em; -} - -div#pagebody { -} - -div#TOC ul { - list-style: none; -} - -h1, h2, h3, h4, h5, h6 { - font-family: sans-serif; - font-weight: bold; - margin-top: 2em; -} - -h1 { - font-size: 150%; -} - -h2 { - font-size: 120%; -} - -h3 { - font-size: 100%; -} - -ul li, ol li { - margin-top: 0.5em; - margin-bottom: 0.5em; -} - -pre { - margin-left: 4em; -} - -</style> |