Merge branch 'rust-image' into 'main'

update things for going public

See merge request larswirzenius/ambient-ci!14

8 files changed, 265 insertions, 103 deletions

diff --git a/README.md b/README.md
index d4a05da..a03cb49 100644
--- a/README.md
+++ b/README.md
@@ -4,24 +4,86 @@ Ambient is a PRE-ALPHA level project to build a CI system. It's only
 ever been tested on Debian 12 (bookworm). To try this:
 
 - you need: vmdb2, ansible, QEMU, possibly more
-- build the VM image: `sudo ./runner.sh /var/tmp ambient.qcow2`
+- build the VM image:  
+  `sudo ./ambient-build-debian-image --image ~/ambient.qcow2--cache ~`
   - building an image requires root access, but you can re-use the
     image any number of times, and you only need to build it once
-  - the first argument is a directory where a cache file is created to
-    speed up future builds
-  - the second argument is the name of the finished VM image
-  - on my machine the first build takes minutes, subsequent builds
-    take about 30s
+  - the cache option names a directory where a cache file is created
+    to speed up future builds
+  - the image option is the name of the finished VM image
+  - this can take a few minutes
 - run a build: `./ambient-run --image ambient.qcow2 --log log --artifact output.tar test-project`
   - this builds the project in `test-project` in a VM, by running
     `test-project/.ambient-script`
   - this will write the build log to `log` and any artifacts produced
     by the build to `output.tar`
-  - on my machine the build takes about 30s
 
+## The virtual machine
+
+You need to prepare a virtual machine image to use with `ambient-run`.
+The provided `ambient-build-debian-image` builds a Debian one, but any
+operating system will work, if prepared the right way.
+
+The VM will get be run using QEMU (the `kvm` command), and probably
+only works for x86-64, for now. (This can be fixed, at a cost of
+speed.) The VM will be given two extra devices, one with the source
+files, and one for the output files. On Linux, these devices are `vdb`
+and `vdc`. The input device will contain a tar archive with the source
+tree. The build shall write a tar archive of the exported artifacts to
+the output device. The output device has a size limit, and the build
+can't write more than that. `ambient-run` extracts the tar archive
+from the output device.
+
+The operating system in the virtual machine must run the build by
+extracting the source tar, and run `.ambient-script` from there. The
+script must be given the output device as its only argument. See
+`ambient-run-script` in the source file for how the Debian image does
+it. See also `ambient.service` for the systemd unit that invokes the
+script. The build is non-interactive.
+
+The first serial port (`/dev/ttyS0` in the Debian image) can be used
+for a build log. It is captured to a file by `ambient-run` (see the
+`--log` option). For now, the log is unstructured and just contains
+the raw output from the serial port.
+
+
+## Building Rust programs
+
+The `rust.yml` extra playbook installs a Rust toolchain with `rustup`.
+A Rust project might have an `.ambient-script` like this (assuming the
+output binary is `helloworld`):
+
+~~~sh
+#!/bin/bash
+
+set -xeuo pipefail
+
+output="$1"
+
+export PATH="/root/.cargo/bin:$PATH"
+export CARGO_HOME=cargo-home
+
+cargo build
+
+tar -cf "$output" -C target/debug helloworld
+~~~
+
+The build in the VM has no network access at all. To provide
+dependencies, you can use `cargo fetch`:
+
+~~~sh
+$ mkdir cargo-home
+$ CARGO_HOME=cargo-home cargo fetch
+~~~
+
+This will mean the dependencies get downloaded into `cargo-home` in
+the source tree, and `ambient-run` will provide them to the build VM
+with the rest of the sources.
 
 ## Links
 
+These were an inspiration:
+
 * <https://asylum.madhouse-project.org/blog/2022/08/02/the-big-ci-hunt/>
 * <https://asylum.madhouse-project.org/blog/2022/08/05/the-ideal-ci-part-one/>
 * <https://asylum.madhouse-project.org/blog/2022/08/07/the-ideal-ci-part-two/>
diff --git a/ambient-build-debian-image b/ambient-build-debian-image
new file mode 100755
index 0000000..892d006
--- /dev/null
+++ b/ambient-build-debian-image
@@ -0,0 +1,126 @@
+#!/usr/bin/python3
+
+import argparse
+import os
+import shutil
+import subprocess
+import tempfile
+import yaml
+
+
+PROG = "ambient-build-debian-image"
+DESC = "build virtual machine image with Debian for Ambient"
+VERSION = "0.0"
+CACHE = os.path.expanduser("~/.cache/ambient")
+BASE_VMDB = "ambient.vmdb"
+BASE_PLAYBOOK = "ambient-playbook-base.yml"
+
+
+def parse_args():
+    p = argparse.ArgumentParser(prog=PROG, description=DESC)
+
+    p.add_argument(
+        "--version", action="version", version=VERSION, help="show version of program"
+    )
+
+    p.add_argument(
+        "--cache",
+        action="store",
+        help="set cache directory (default is %(default)s)",
+        default=CACHE,
+    )
+
+    p.add_argument(
+        "--playbook",
+        action="store",
+        help="extra Ansible playbook to use with vmdb to build image (default is none)",
+    )
+
+    p.add_argument(
+        "--image",
+        action="store",
+        help="filename for resulting image (required)",
+        required=True,
+    )
+    return p.parse_args()
+
+
+def load_yaml(filename):
+    with open(filename) as f:
+        return yaml.safe_load(f)
+
+
+def write_yaml(filename, obj):
+    with open(filename, "w") as f:
+        yaml.safe_dump(obj, stream=f, indent=4)
+
+
+def prepare_build_files(tmp, args):
+    vmdb_filename = os.path.join(tmp, "image.vmdb")
+    playbook_filename = os.path.join(tmp, "playbook.yml")
+    extra_playbook_filename = os.path.join(tmp, "extra.yml")
+
+    vmdb = load_yaml(BASE_VMDB)
+
+    if args.playbook is not None:
+        vmdb["steps"].append(
+            {
+                # note: the value MUST be the root file system tag in the VMDB file
+                "ansible": "/",
+                "playbook": args.playbook,
+            }
+        )
+        shutil.copyfile(args.playbook, extra_playbook_filename)
+
+    write_yaml(vmdb_filename, vmdb)
+    shutil.copyfile(BASE_PLAYBOOK, playbook_filename)
+
+    return vmdb_filename
+
+
+def build_image(tmp, vmdb, tarball, image):
+    raw = os.path.join(tmp, "image.img")
+    log = os.path.join(tmp, "log")
+
+    # Build image.
+    p = subprocess.run(
+        ["vmdb2", vmdb, "--output", raw, "--rootfs-tarball", tarball, "--log", log],
+        capture_output=True,
+    )
+    if p.returncode != 0:
+        with open(log, "r") as f:
+            log = f.read()
+        sys.stderr.write(f"{log}\nERROR: failed to build image")
+        sys.exit(1)
+
+    # Convert built image from raw to QCOW2 format.
+    p = subprocess.run(
+        ["qemu-img", "convert", "-f", "raw", "-O", "qcow2", raw, image],
+        capture_output=True,
+    )
+    if p.returncode != 0:
+        sys.stderr.write(f"{p.stderr}\nERROR: failed to convert image to QCOW2")
+        sys.exit(1)
+
+
+def main():
+    args = parse_args()
+    tarball = os.path.join(args.cache, "ambient.tar.gz")
+    tmp = tempfile.mkdtemp()
+
+    try:
+        vmdb = prepare_build_files(tmp, args)
+    except Exception as e:
+        sys.stderr.write("ERROR: failed to create build files: {e}\n")
+        shutil.rmtree(tmp)
+        sys.exit(1)
+
+    try:
+        build_image(tmp, vmdb, tarball, args.image)
+    except Exception as e:
+        sys.stderr.write(f"ERROR: failed to create build files: {e}\n")
+        shutil.rmtree(tmp)
+        sys.exit(1)
+
+
+main()
diff --git a/runner.yml b/ambient-playbook-base.yml
index 2fa26b4..2fa26b4 100644
--- a/runner.yml
+++ b/ambient-playbook-base.yml
diff --git a/ambient-run b/ambient-run
index be52226..f9b03e1 100755
--- a/ambient-run
+++ b/ambient-run
@@ -6,6 +6,7 @@ import os
 import shutil
 import subprocess
 import sys
+import tarfile
 import tempfile
 
 
@@ -18,11 +19,44 @@ def parse_args():
     p.add_argument("--verbose", action="store_true")
     p.add_argument("--image", required=True)
     p.add_argument("--log", required=True)
-    p.add_argument("--artifact", required=True)
+    p.add_argument("--artifact")
     p.add_argument("source_tree")
     return p.parse_args()
 
 
+def extract_tar_from_tar(filename, output):
+    N = 1024
+    BLOCK = 10 * 1024
+
+    with open(filename, "rb") as f:
+        tar = tarfile.open(fileobj=f, mode="r:")
+        while True:
+            info = tar.next()
+            if info is None:
+                break
+
+        # We've read to the end of the final tar file entry.
+        num_bytes = tar.fileobj.tell()
+
+        with open(output, "wb") as o:
+            # Copy first num_bytes of input file to output file.
+
+            f.seek(0)
+            n = 0
+            while n < num_bytes:
+                data = f.read(N)
+                if data is None:
+                    break
+                o.write(data)
+                n += len(data)
+
+            # Pad output file with zeroes until next block border.
+
+            while (n % BLOCK) != 0:
+                o.write(b"\0")
+                n += 1
+
+
 def main():
     args = parse_args()
 
@@ -40,7 +74,7 @@ def main():
         logging.debug(f"created {tmp}")
 
         image = os.path.join(tmp, "vm.qcow2")
-        output_drive = args.artifact
+        output_drive = os.path.join(tmp, "artifacts.tar")
         src_tar = os.path.join(tmp, "src.tar")
         vars = os.path.join(tmp, "vars.fd")
 
@@ -88,6 +122,10 @@ def main():
                 sys.stderr.write(p.stderr.decode())
                 raise Exception("qemu-system failed")
 
+        if args.artifact is not None:
+            logging.info("extract artifacts, if any")
+            extract_tar_from_tar(output_drive, args.artifact)
+
         logging.info("Build finished")
     except Exception as e:
         sys.stderr.write(f"{e}\n")
diff --git a/ambient.md b/ambient.md
index afda670..8d20a59 100644
--- a/ambient.md
+++ b/ambient.md
@@ -6,7 +6,8 @@ simple, sane, safe, secure, speedy, and
 supercalifragilisticexpialidocious. There are many existing such
 systems, but we feel none of them are excellent.
 
-For now, it is a command line tool that runs a build locally, in a VM.
+For now, it is a command line tool that runs a build locally, in a VM,
+without network access.
 
 ## Continuous integration concepts
 
@@ -16,7 +17,7 @@ definitions for Ambient. The goal here is being clear and unambiguous
 in the Ambient context, not to define terminology in the wider
 community of software developers.
 
-Note: this is currently aimed at building, testing, and publishing
+Note: Ambient is currently aimed at building, testing, and publishing
 software, but not deploying it. That may change, but it's possible
 that deployment will need its own kind of system rather than forcing a
 CI system to do it badly.
@@ -27,7 +28,7 @@ Ambient, or several may be combined into one. For thinking and
 discussing the system and its software architecture, we pretend that
 they're all separate programs.
 
-* **artifact** -- a file produced by a build step
+* **artifact** -- a file produced by a build
 
 * **artifact store** -- where artifacts are stored after a build step
   is finished
@@ -144,14 +145,14 @@ These are not in any kind of order.
   hardware capacity allows.
 
 * We want it to be easy to provide build workers, without having to
-  worry about the security of the worker hots, or the security of the
+  worry about the security of the worker host, or the security of the
   build artifacts.
 
-* If a job is going to fail for a reason that can be predicted before
-  it even starts, the job should not start. For example, if a build
-  step runs a shell command, the syntax should be checked before the
-  job starts. Obviously this is not possible in every case, but in the
-  common case it is.
+* If a build is going to fail for a reason that can be predicted
+  before it even starts, the job should not start. For example, if a
+  build step runs a shell command, the syntax should be checked before
+  the job starts. Obviously this is not possible in every case, but in
+  the common case it is.
 
 * Build failures should be easy to debug. Exactly what this means is
   unclear at the time of writing, but it should be a goal for all
@@ -160,7 +161,7 @@ These are not in any kind of order.
 * It's easy to host both client and server components.
  
 * It's possible, straightforward, and safe, for workers to require
-  payment to run a job. This needs to be done in a way that is
+  payment to run a build step. This needs to be done in a way that is
   unlikely to anyone being scammed.
 
 * Is integrated into major git hosting platforms (GitHub, GitLab,
@@ -228,9 +229,8 @@ We build and test in a local virtual machine.
 The VM has no network access at all. We provide the VM the project
 source code via a read-only virtual disk. We provide the VM with
 another virtual disk where it can store any artifacts that should
-persist. Both virtual disks will use file system that can be produced
-and read entirely using tools that run outside the operating system
-kernel, to avoid triggering a kernel bug.
+persist. Both virtual disks will contain no file system, but a tar
+archive.
 
 We provide the VM with a pre-determined amount of virtual disk. The
 project won't be able to use more.
@@ -247,71 +247,3 @@ We run the VM with a pre-determined amount of disk, number of CPUs,
 amount of memory. We fail the build if it exceeds a pre-determined
 time limit. We fail the build if the amount of output via the serial
 console exceeds a pre-determined limit.
-
-# MVP for local job runner for Ambient
-
-I think the first implementation step for Ambient will need to be a
-command line tool to run a job locally, but safely and securely.
-Something along the following lines:
-
-* Tool is tentatively called `ambient-run`.
-* It is invoked like this: `ambien-run .` where the argument, `DIR`,
-  is the root of the source tree to be built.
-* At startup, `ambient-run` reads its configuration file, and
-  `DIR/.ambient-ci.yml`, where `DIR` is the command line argument.
-* The configuration specifies the path to the base image for the build
-  VM, in qcow2 format
-* The `.ambient-ci.yml` file specifies fields:
-  - `build`: shell snippet to run to do the build
-  - `test`: shell snippet to run automated tests after the build
-* The tool works like this:
-  - copy the base image to a temporary file
-  - create a workspace disk image where the directory specified on the
-    command line is copied
-  - boot build VM using QEMU, with a serial port, and no networking
-  - via serial port, log in as root, then run the `build` and
-    `test` shell snippets, as if prefixed by `#!/bin/bash\nset -euo pipefail\n`
-  - capture output from the snippets into `ambient-run` stdout and stderr
-  - if either snippet fails, terminate job run and destroy the VM and
-    don't run remaining snippet, if any
-
-The only output is the "build log" captured from the serial port. No
-artifacts are exported for the MVP.
-
-The base image needs to be specially prepared:
-
-* `root` can log in without a password
-* the image has all the build tools and dependencies needed
-  pre-installed
-
-The workspace disk:
-
-* ISO disk image (so that it can be created with genisoimage, without
-  root privileges)
-* contains tar archive of the directory given on the command line
-* tar archive is extracted to `/workspace` before the build starts
-* all files owned by `root:root` and with the same mode bits as in the
-  source tree
-  - `mkdir -p /workspace`
-  - `tar -xf /mnt/src.tar.xz -C /workspace`
-  - `chdown -R 0:0 /workspace`
-
-Control of build VM via serial console:
-
-- use expect(1) to log in and get to prompt
-- at prompt use expect(1) to send commands to mount workspace, and to
-  upload shell script to run job
-
-This should allow me to do a "smoke test" build like this for Subplot,
-assuming the base image has the Rust toolchain installed.
-
-~~~yaml
-build: |
-  RUSTFLAGS="-D warnings" cargo clippy --workspace --all-targets -- -Dclippy::all
-  cargo build --workspace --all-targets
-test: |
-  cargo test --workspace
-~~~
-
-This would not be enough in general, but would be enough as a proof of
-concept and to get the Ambient implementation started.
diff --git a/runner.vmdb b/ambient.vmdb
index 3cd7067..3cd7067 100644
--- a/runner.vmdb
+++ b/ambient.vmdb
diff --git a/runner.sh b/runner.sh
deleted file mode 100755
index d037942..0000000
--- a/runner.sh
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-set -euo pipefail
-
-cachedir="$1"
-img="$2"
-
-if ! vmdb2 runner.vmdb --output runner.img --rootfs-tarball "$cachedir/runner.tar.gz" --log runner.log; then
-	cat runner.log 1>&2
-	exit 1
-fi
-
-qemu-img convert -f raw -O qcow2 runner.img "$img"
diff --git a/rust.yml b/rust.yml
new file mode 100644
index 0000000..1f9197a
--- /dev/null
+++ b/rust.yml
@@ -0,0 +1,17 @@
+- hosts: image
+  tasks:
+
+    - name: "install tools for Rust"
+      apt:
+        name:
+          - build-essential
+          - cmake
+          - curl
+
+    - name: "install rustup"
+      shell: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > /tmp/rustup.sh
+        sh /tmp/rustup.sh -y --no-modify-path        
+
+  vars:
+    ansible_python_interpreter: /usr/bin/python3