summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLars Wirzenius <liw@liw.fi>2020-12-13 12:40:16 +0200
committerLars Wirzenius <liw@liw.fi>2020-12-13 12:51:22 +0200
commit95048665bfec3d981c76d01df96d81dbf9597a3b (patch)
tree256f9d8fd5853f7af93a4e65486145803cdcea24
parent979628f86910890118b6f591633bd1e884b2cdf1 (diff)
downloadcontractor2-95048665bfec3d981c76d01df96d81dbf9597a3b.tar.gz
doc: add start of architecture and acceptance criteria document
-rw-r--r--contractor.md401
-rw-r--r--subplot/contractor.py25
-rw-r--r--subplot/contractor.yaml5
-rw-r--r--subplot/files.py158
-rw-r--r--subplot/files.yaml58
-rw-r--r--subplot/vendor/runcmd.py252
-rw-r--r--subplot/vendor/runcmd.yaml83
7 files changed, 982 insertions, 0 deletions
diff --git a/contractor.md b/contractor.md
new file mode 100644
index 0000000..c76dc62
--- /dev/null
+++ b/contractor.md
@@ -0,0 +1,401 @@
+<!-- meta data block is at the end of the file, because Emacs gets -->
+<!-- less confused that way -->
+
+# Introduction
+
+Software development is a security risk.
+
+Building software from source code and running it is a core activity
+of software development. Software developers do it on the machine they
+work on. Continuous integration systems do it on server. These are
+fundamentally the same. The process is roughly as follows:
+
+* install any dependencies
+* build the software
+* run the software to perform automated tests on it
+
+When the software is run, even if only a small unit of it, it can do
+anything that the person running the build can do. For example, it can
+do any and all of the following, unless constrained:
+
+* delete files
+* modify files
+* log into remote hosts using SSH
+* decrypt or sign files with PGP
+* send email
+* delete email
+* commit to version control repositories
+* do anything with the browser that a the person could do
+* run things as root using sudo
+* in general, cause mayhem and chaos
+
+Normally, a software developer can assume that the code they wrote
+themselves won't ever do any of that. They can even assume that people
+they work with make code that won't do any of that. In both cases,
+they may be wrong: mistakes happen. It's a well-guarded secret among
+programmers that they sometimes, even if rarely, make catastrophic
+mistakes.
+
+Accidents aside, mayhem and chaos may be intentional. Your own project
+may not have malware, and you may have vetted all your dependencies,
+and you trust them. But your dependencies have dependencies, which
+have further dependencies, which have dependencies of their own. You'd
+need to vet the whole dependency tree. Even decades ago, in the 1990s,
+this could easily be hundreds of thousands of lines of code, and
+modern systems a much larger. Note that build tools are themselves
+dependencies, as is the whole operating system. Any code that is invoked
+in the build process is a dependency.
+
+How certain are you that you can spot malicious code that's
+intentionally hidden and obfuscated?
+
+Are you prepared to vet any changes to any transitive dependencies?
+
+Does this really matter? Maybe it doesn't. If you can't ever do
+anything on your computer that would affect you or anyone else in a
+negative way, it probably doesn't matter. Most software developers are
+not in that position.
+
+This risk affects every operating system and every programming
+language. The degree in which it exists varies, a lot. Some
+programming language ecosystems seem more vulnerable than others: the
+nodejs/npm one, for example, values tiny and highly focused packages,
+which leads to immense dependency trees. The more direct or indirect
+dependencies there are, the higher the chance that one of them turns
+out to be bad.
+
+The risk also exists for more traditional languages, such as C. Few C
+programs have no dependencies. They all need a C compiler, which in
+turns requires an operating system, at least.
+
+The risk is there for both free software systems, and non-free ones.
+As an example, the Debian system is entirely free software, but it's
+huge: the Debian 10 (buster) release has over 50 thousand packages,
+maintained by thousands of people. While it's probable that none of
+those packages contains actual malware, it's not certain. Even if
+everyone who helps maintain is completely trustworthy, the amount of
+software in Debian is much too large for all code to be
+comprehensively reviewed.
+
+This is true for all operating systems that are not mere toys.
+
+The conclusion here is that to build software securely, we can't
+assume all code involved in the build to be secure. We need something
+more secure. The Contractor aims to be a possible solution.
+
+## Threat model
+
+This section collects a list of specific threats to consider.
+
+* accessing or modifying files not part of the build
+* excessive use build host resources
+ * e.g., CPU, GPU, RAM, disk, etc
+ * this might happen to make unauthorized use of the resources, or to
+ just be wasteful
+* excessive use of network bandwidth
+* attack on a networked target via a denial of service attack
+ * e.g., build joins a DDoS swarm, or sends fabricated SYN packets to
+ prevent target from working
+* attack on build host, or other host, via network intrusion
+ * e.g., port scanning, probing for known vulnerabilities
+* attack build host directly without network
+ * e.g., by breaching security isolation using build host kernel or
+ hardware vulnerabilities, or CI engine vulnerabilities
+ * this includes eavesdropping on the host, and stealing secrets
+
+## Status of this document
+
+Everything about the Contractor is in its early stages of thinking,
+sketching, experimentations, and planning. Nothing is nailed down yet.
+
+Pre-ALPHA. Don't trust anything. Anything you trust may be used
+against you. Anything may change.
+
+# Requirements
+
+This chapter discusses the requirements for the Contractor solution.
+The requirements are divided into two parts: one that's based on the
+threat model, and another for requirements that aren't about security.
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+document are to be interpreted as described in [RFC 2119][].
+
+[RFC 2119]: https://tools.ietf.org/html/rfc2119
+
+## Security requirements
+
+These requirements stem from the threat model above.
+
+* **FilesystemIsolation**: The Contractor MUST prevent the build from
+ accessing or modifying any files outside the build. Build tools and
+ libraries outside the source tree MUST be usable.
+
+* The Contractor MUST prevent the build from using more than the
+ user-specified amount of CPU time (**HardCPULimit**), disk space
+ (**HardDiskLimit**), or network bandwidth (**HardBandwidthLimit**).
+ Any attempt by the build to use more should fail. The Contractor
+ MUST fail the build if the limits are exceeded.
+
+* **HardRAMLimit**: The Contractor MUST prevent the build from using
+ more than the user-specified amount of RAM. The Contractor MAY fail
+ to build if the limit is exceeded, but is not required to do so.
+
+* **ConstrainNetworkAccess**: The Contractor MUST prevent the build
+ from accessing the network ourside the build environment in ways
+ that haven't been specifically allowed by the user. The contractor
+ SHOULD fail the build if it makes an attempt at such access. The
+ user MUST be able to specify which hosts to access, and using which
+ protocols.
+
+* **HostProtection**: The Contractor SHOULD attempt to protect the
+ host its running on from non-networked attacks performed by the
+ build. This includes vulnerabilities of the host's operating system
+ kernel, virtualisation solution, and hardware.
+
+## Non-security requirements
+
+* **AnyBuildOS**: Builds SHOULD be able to run in any operating system
+ that can be run as a virtual machine guest of the host operating
+ system.
+
+* **NoRoot**: Running the Contractor SHOULD NOT require root
+ privileges. It's OK to require sufficient privileges to use
+ virtualisation.
+
+* **DefaultBuilder**: The Contractor SHOULD be easy to set up and to
+ use. It should not require extensive configuration. Running a build
+ should be as easy as running **make**(1) on the command line. It
+ should be feasible to expect developers to use the Contractor for
+ their normal development work.
+
+
+# Architecture
+
+This chapter discusses the architecture of the solution, with
+particular emphasis on threat mitigation.
+
+The overall solution is use of nested virtual machines, running on a
+developer's host. The outer VM runs the Contractor itself, and is
+called "the manager VM". The inner VM runs the build, and is called
+the "worker VM". The manager VM controls the worker VM, proxies its
+external access, and prevents it from doing anything nefarious. The
+manager VM is managed by a command line tool. Developers only interact
+directly with the command line tool.
+
+~~~dot
+digraph "arch" {
+ labelloc=b;
+ labeljust=l;
+ dev [shape=octagon label="Developer"];
+ img [shape=tab label="VM image"];
+ src [shape=tab label="Source tree"];
+ ws [shape=tab label="Exported workspace"];
+ apt [shape=tab label="APT repository"];
+ subgraph cluster_host {
+ label="Host system \n (the vulnerable bit)";
+ contractor [label="Contractor CLI"];
+ subgraph cluster_contractor {
+ label="Manager VM \n (defence force)";
+ manager;
+ libvirt;
+ subgraph cluster_builder {
+ label="Worker VM \n (here be dragons)";
+ style=filled;
+ fillcolor="#dd0000";
+ guestos [label="Guest OS"];
+ }
+ }
+ }
+ dev -> contractor;
+ contractor -> manager;
+ contractor -> guestos;
+ img -> contractor;
+ ws -> contractor;
+ src -> contractor;
+ apt -> guestos;
+ manager -> libvirt;
+ libvirt -> guestos;
+ contractor -> ws;
+}
+~~~
+
+This high-level design is chosen for the following reasons:
+
+* it allows the build to happen in any operating system
+ (**AnyBuildOS**)
+* the Contractor is a VM and running it doesn't require root
+ privileges; it has root inside both VMs if needed; all the
+ complexity of setting things up so the worker VM works correctly are
+ contained the manager VM, and the user need do only minimal
+ configuration (**NoRoot**)
+* the command line tool for using the Contractor can be made to be as
+ easy as any build tool so that developer actually use it by default
+ (**DefaultBuilder**)
+* the manager VM can monitor and control the build (**HardCPULimit**,
+ **HardBandwidthLimit**)
+* the manager can supply the worker VM with only a specified amount of
+ RAM and disk space (**HardRAMLimit**, **HardDiskLimit**)
+* the manager can set up routing and firewalls so that the worker VM
+ cannot access the network, except via proxies provided by the outer
+ VM (**ConstrainNetworkAccess**)
+* the nested VMs provide a smaller attack surface than the Linux
+ kernel API, and this protects the host better than Linux container
+ technologies, although it doesn't do much to protect against
+ virtualisation or hardware vulnerabilities (**HostProtection**)
+
+## Build process
+
+The architecture leads to a build process that would work roughly like
+this:
+
+* the manager VM is already running
+* developer runs command line tool to do a build:
+ `contractor build foo.yaml`
+* command line tool copies the worker VM image into the manager VM
+* command line tool boots the worker VM
+* command line tool installs any build dependencies into the worker VM
+* command line tool copies a previously saved dump of the workspace
+ into the worker VM
+* command line tool copies the source code and build recipe into the
+ worker VM's workspace
+* command line tool runs build commands in the worker VM, in the
+ source tree
+* command line tool copies out the workspace into a local directory
+* command line tool reports to the developer build success or failure
+ and where build log and build artifacts are
+
+## Implementation sketch (FIXME: update)
+
+FIXME: write this
+
+
+
+# Acceptance criteria
+
+This chapter specifies acceptance criteria for the Contractor, as
+*scenarios*, which also define how the criteria are automatically
+verified.
+
+## Local use of the Contractor
+
+These scenarios use the Contractor locally, to make sure it can do
+things that don't require the VM.
+
+### Parse build spec
+
+Make sure the Contractor can read a build spec and dump it back out,
+as JSON. This exercises the parsing code. JSON output is chosen,
+instead of YAML, to make sure the program doesn't just copy input to
+output.
+
+~~~scenario
+given file dump.yaml
+when I invoke contractor dump dump.yaml
+then the JSON output matches dump.yaml
+~~~
+
+~~~{.file #dump.yaml .yaml .numberLines}
+worker-image: worker.img
+ansible:
+ - hosts: worker
+ remote_user: worker
+ become: true
+ tasks:
+ - apt:
+ name: build-essential
+ vars:
+ ansible_python_interpreter: /usr/bin/python3
+source: .
+workspace: workspace
+build: |
+ ./check
+~~~
+
+## Smoke tests
+
+These scenarios build a simple "hello, world" C application on a
+variety of guest systems, and verify the resulting binaries output the
+desired greeting. The goal of these scenarios is to ensure the various
+Contractor components fit together at least in the very basic case.
+
+### Debian smoke test
+
+This scenario checks that the developer can build a simple C program
+in the Contractor.
+
+~~~disabled-scenario
+given a working contractor
+and file hello.c
+and file hello.yaml
+and file worker.img from source directory
+when I run contractor build hello.yaml
+then exit code is 0
+then file ws/src/hello exists
+~~~
+
+~~~{.file #hello.c .c .numberLines}
+#include <stdio.h>
+
+int main()
+{
+ printf("hello, world\n");
+ return 0;
+}
+~~~
+
+~~~{.file #hello.yaml .yaml .numberLines}
+worker-image: worker.img
+ansible:
+ - hosts: worker
+ remote_user: worker
+ become: true
+ tasks:
+ - apt:
+ name: build-essential
+ vars:
+ ansible_python_interpreter: /usr/bin/python3
+source: .
+workspace: ws
+build: |
+ gcc hello.c -o hello
+ ./hello
+~~~
+
+
+
+---
+title: "Contractor: build software securely"
+author: "Lars Wirzenius"
+bindings:
+- subplot/contractor.yaml
+- subplot/vendor/runcmd.yaml
+- subplot/files.yaml
+functions:
+ - subplot/contractor.py
+ - subplot/vendor/runcmd.py
+ - subplot/files.py
+documentclass: report
+classes:
+ - c
+ - disabled-scenario
+abstract: |
+ Building software typically requires running code downloaded from
+ the Internet. Even when you're building your own software, you
+ usually depend on libraries and tools, which in turn may depend on
+ further things. It is becoming infeasible to vet the whole set of
+ software running during a build. If a build includes running local
+ tests (unit tests, some integration tests), the problem gets worse
+ in magnitude, if not quality.
+
+ Some software ecosystems are especially vulnerable to this (nodejs,
+ Python, Ruby, Go, Rust), but it's true for anything that has
+ dependencies on any code from outside its own code base, and even if
+ all the dependencies come from a trusted source, such as the
+ operating system vendor or a Linux distribution.
+
+ The Contractor is an attempt to be able to build software securely,
+ by leveraging virtual machine technology. It attempts to be
+ secure, convenient, and reasonably efficient.
+
+...
diff --git a/subplot/contractor.py b/subplot/contractor.py
new file mode 100644
index 0000000..a2569dd
--- /dev/null
+++ b/subplot/contractor.py
@@ -0,0 +1,25 @@
+import json
+import logging
+import os
+import yaml
+
+
+def run_contractor_dump(ctx, filename=None):
+ runcmd_run = globals()["runcmd_run"]
+ srcdir = globals()["srcdir"]
+ argv = [os.path.join(srcdir, "target", "debug", "contractor"), "dump", filename]
+ runcmd_run(ctx, argv)
+
+
+def stdout_json_matches_yaml_file(ctx, filename=None):
+ runcmd_get_stdout = globals()["runcmd_get_stdout"]
+ assert_dict_eq = globals()["assert_dict_eq"]
+
+ stdout = runcmd_get_stdout(ctx)
+ logging.debug(f"stdout: {stdout!r}")
+ obj_json = json.loads(stdout)
+ logging.info(f"object from stdout: {obj_json!r}")
+
+ obj_yaml = yaml.safe_load(open(filename))
+ logging.info(f"object from yaml: {obj_yaml!r}")
+ assert_dict_eq(obj_json, obj_yaml)
diff --git a/subplot/contractor.yaml b/subplot/contractor.yaml
new file mode 100644
index 0000000..7a512b7
--- /dev/null
+++ b/subplot/contractor.yaml
@@ -0,0 +1,5 @@
+- when: I invoke contractor dump {filename}
+ function: run_contractor_dump
+
+- then: the JSON output matches {filename}
+ function: stdout_json_matches_yaml_file
diff --git a/subplot/files.py b/subplot/files.py
new file mode 100644
index 0000000..ec37b9d
--- /dev/null
+++ b/subplot/files.py
@@ -0,0 +1,158 @@
+import logging
+import os
+import re
+import time
+
+
+def files_create_from_embedded(ctx, filename=None):
+ files_create_from_embedded_with_other_name(
+ ctx, filename_on_disk=filename, embedded_filename=filename
+ )
+
+
+def files_create_from_embedded_with_other_name(
+ ctx, filename_on_disk=None, embedded_filename=None
+):
+ get_file = globals()["get_file"]
+ with open(filename_on_disk, "wb") as f:
+ f.write(get_file(embedded_filename))
+
+
+def files_create_from_text(ctx, filename=None, text=None):
+ with open(filename, "w") as f:
+ f.write(text)
+
+
+def files_file_exists(ctx, filename=None):
+ assert_eq = globals()["assert_eq"]
+ assert_eq(os.path.exists(filename), True)
+
+
+def files_file_does_not_exist(ctx, filename=None):
+ assert_eq = globals()["assert_eq"]
+ assert_eq(os.path.exists(filename), False)
+
+
+def files_only_these_exist(ctx, filenames=None):
+ assert_eq = globals()["assert_eq"]
+ filenames = filenames.replace(",", "").split()
+ assert_eq(set(os.listdir(".")), set(filenames))
+
+
+def files_file_contains(ctx, filename=None, data=None):
+ assert_eq = globals()["assert_eq"]
+ with open(filename, "rb") as f:
+ actual = f.read()
+ actual = actual.decode("UTF-8")
+ assert_eq(data in actual, True)
+
+
+def files_file_matches_regex(ctx, filename=None, regex=None):
+ assert_eq = globals()["assert_eq"]
+ with open(filename) as f:
+ content = f.read()
+ m = re.search(regex, content)
+ if m is None:
+ logging.debug(f"files_file_matches_regex: no match")
+ logging.debug(f" filenamed: {filename}")
+ logging.debug(f" regex: {regex}")
+ logging.debug(f" content: {regex}")
+ logging.debug(f" match: {m}")
+ assert_eq(bool(m), True)
+
+
+def files_match(ctx, filename1=None, filename2=None):
+ assert_eq = globals()["assert_eq"]
+ with open(filename1, "rb") as f:
+ data1 = f.read()
+ with open(filename2, "rb") as f:
+ data2 = f.read()
+ assert_eq(data1, data2)
+
+
+def files_touch_with_timestamp(
+ ctx,
+ filename=None,
+ year=None,
+ month=None,
+ day=None,
+ hour=None,
+ minute=None,
+ second=None,
+):
+ t = (
+ int(year),
+ int(month),
+ int(day),
+ int(hour),
+ int(minute),
+ int(second),
+ -1,
+ -1,
+ -1,
+ )
+ ts = time.mktime(t)
+ _files_touch(filename, ts)
+
+
+def files_touch(ctx, filename=None):
+ _files_touch(filename, None)
+
+
+def _files_touch(filename, ts):
+ if not os.path.exists(filename):
+ open(filename, "w").close()
+ times = None
+ if ts is not None:
+ times = (ts, ts)
+ os.utime(filename, times=times)
+
+
+def files_mtime_is_recent(ctx, filename=None):
+ st = os.stat(filename)
+ age = abs(st.st_mtime - time.time())
+ assert age < 1.0
+
+
+def files_mtime_is_ancient(ctx, filename=None):
+ st = os.stat(filename)
+ age = abs(st.st_mtime - time.time())
+ year = 365 * 24 * 60 * 60
+ required = 39 * year
+ logging.debug(f"ancient? mtime={st.st_mtime} age={age} required={required}")
+ assert age > required
+
+
+def files_remember_metadata(ctx, filename=None):
+ meta = _files_remembered(ctx)
+ meta[filename] = _files_get_metadata(filename)
+ logging.debug("files_remember_metadata:")
+ logging.debug(f" meta: {meta}")
+ logging.debug(f" ctx: {ctx}")
+
+
+# Check that current metadata of a file is as stored in the context.
+def files_has_remembered_metadata(ctx, filename=None):
+ assert_eq = globals()["assert_eq"]
+ meta = _files_remembered(ctx)
+ logging.debug("files_has_remembered_metadata:")
+ logging.debug(f" meta: {meta}")
+ logging.debug(f" ctx: {ctx}")
+ assert_eq(meta[filename], _files_get_metadata(filename))
+
+
+def files_has_different_metadata(ctx, filename=None):
+ assert_ne = globals()["assert_ne"]
+ meta = _files_remembered(ctx)
+ assert_ne(meta[filename], _files_get_metadata(filename))
+
+
+def _files_remembered(ctx):
+ ns = ctx.declare("_files")
+ return ns.get("remembered-metadata", {})
+
+
+def _files_get_metadata(filename):
+ st = os.lstat(filename)
+ keys = ["st_dev", "st_gid", "st_ino", "st_mode", "st_mtime", "st_size", "st_uid"]
+ return {key: getattr(st, key) for key in keys}
diff --git a/subplot/files.yaml b/subplot/files.yaml
new file mode 100644
index 0000000..5777a8e
--- /dev/null
+++ b/subplot/files.yaml
@@ -0,0 +1,58 @@
+- given: file {filename}
+ function: files_create_from_embedded
+
+- given: file {filename_on_disk} from {embedded_filename}
+ function: files_create_from_embedded_with_other_name
+
+- given: file {filename} has modification time {year}-{month}-{day} {hour}:{minute}:{second}
+ function: files_touch_with_timestamp
+
+- when: I write "(?P<text>.*)" to file (?P<filename>\S+)
+ regex: true
+ function: files_create_from_text
+
+- when: I remember metadata for file {filename}
+ function: files_remember_metadata
+
+- when: I touch file {filename}
+ function: files_touch
+
+- then: file {filename} exists
+ function: files_file_exists
+
+- then: file {filename} does not exist
+ function: files_file_does_not_exist
+
+- then: only files (?P<filenames>.+) exist
+ function: files_only_these_exist
+ regex: true
+
+- then: file (?P<filename>\S+) contains "(?P<data>.*)"
+ regex: true
+ function: files_file_contains
+
+- then: file (?P<filename>\S+) matches regex /(?P<regex>.*)/
+ regex: true
+ function: files_file_matches_regex
+
+- then: file (?P<filename>\S+) matches regex "(?P<regex>.*)"
+ regex: true
+ function: files_file_matches_regex
+
+- then: files {filename1} and {filename2} match
+ function: files_match
+
+- then: file {filename} has same metadata as before
+ function: files_has_remembered_metadata
+
+- then: file {filename} has different metadata from before
+ function: files_has_different_metadata
+
+- then: file {filename} has changed from before
+ function: files_has_different_metadata
+
+- then: file {filename} has a very recent modification time
+ function: files_mtime_is_recent
+
+- then: file {filename} has a very old modification time
+ function: files_mtime_is_ancient
diff --git a/subplot/vendor/runcmd.py b/subplot/vendor/runcmd.py
new file mode 100644
index 0000000..a2564c6
--- /dev/null
+++ b/subplot/vendor/runcmd.py
@@ -0,0 +1,252 @@
+import logging
+import os
+import re
+import shlex
+import subprocess
+
+
+#
+# Helper functions.
+#
+
+# Get exit code or other stored data about the latest command run by
+# runcmd_run.
+
+
+def _runcmd_get(ctx, name):
+ ns = ctx.declare("_runcmd")
+ return ns[name]
+
+
+def runcmd_get_exit_code(ctx):
+ return _runcmd_get(ctx, "exit")
+
+
+def runcmd_get_stdout(ctx):
+ return _runcmd_get(ctx, "stdout")
+
+
+def runcmd_get_stdout_raw(ctx):
+ return _runcmd_get(ctx, "stdout.raw")
+
+
+def runcmd_get_stderr(ctx):
+ return _runcmd_get(ctx, "stderr")
+
+
+def runcmd_get_stderr_raw(ctx):
+ return _runcmd_get(ctx, "stderr.raw")
+
+
+def runcmd_get_argv(ctx):
+ return _runcmd_get(ctx, "argv")
+
+
+# Run a command, given an argv and other arguments for subprocess.Popen.
+#
+# This is meant to be a helper function, not bound directly to a step. The
+# stdout, stderr, and exit code are stored in the "_runcmd" namespace in the
+# ctx context.
+def runcmd_run(ctx, argv, **kwargs):
+ ns = ctx.declare("_runcmd")
+
+ # The Subplot Python template empties os.environ at startup, modulo a small
+ # number of variables with carefully chosen values. Here, we don't need to
+ # care about what those variables are, but we do need to not overwrite
+ # them, so we just add anything in the env keyword argument, if any, to
+ # os.environ.
+ env = dict(os.environ)
+ for key, arg in kwargs.pop("env", {}).items():
+ env[key] = arg
+
+ pp = ns.get("path-prefix")
+ if pp:
+ env["PATH"] = pp + ":" + env["PATH"]
+
+ logging.debug(f"runcmd_run")
+ logging.debug(f" argv: {argv}")
+ logging.debug(f" env: {env}")
+ p = subprocess.Popen(
+ argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, **kwargs
+ )
+ stdout, stderr = p.communicate("")
+ ns["argv"] = argv
+ ns["stdout.raw"] = stdout
+ ns["stderr.raw"] = stderr
+ ns["stdout"] = stdout.decode("utf-8")
+ ns["stderr"] = stderr.decode("utf-8")
+ ns["exit"] = p.returncode
+ logging.debug(f" ctx: {ctx}")
+ logging.debug(f" ns: {ns}")
+
+
+# Step: prepend srcdir to PATH whenever runcmd runs a command.
+def runcmd_helper_srcdir_path(ctx):
+ srcdir = globals()["srcdir"]
+ runcmd_prepend_to_path(ctx, srcdir)
+
+
+# Step: This creates a helper script.
+def runcmd_helper_script(ctx, filename=None):
+ get_file = globals()["get_file"]
+ with open(filename, "wb") as f:
+ f.write(get_file(filename))
+
+
+#
+# Step functions for running commands.
+#
+
+
+def runcmd_prepend_to_path(ctx, dirname=None):
+ ns = ctx.declare("_runcmd")
+ pp = ns.get("path-prefix", "")
+ if pp:
+ pp = f"{pp}:{dirname}"
+ else:
+ pp = dirname
+ ns["path-prefix"] = pp
+
+
+def runcmd_step(ctx, argv0=None, args=None):
+ runcmd_try_to_run(ctx, argv0=argv0, args=args)
+ runcmd_exit_code_is_zero(ctx)
+
+
+def runcmd_try_to_run(ctx, argv0=None, args=None):
+ argv = [shlex.quote(argv0)] + shlex.split(args)
+ runcmd_run(ctx, argv)
+
+
+#
+# Step functions for examining exit codes.
+#
+
+
+def runcmd_exit_code_is_zero(ctx):
+ runcmd_exit_code_is(ctx, exit=0)
+
+
+def runcmd_exit_code_is(ctx, exit=None):
+ assert_eq = globals()["assert_eq"]
+ assert_eq(runcmd_get_exit_code(ctx), int(exit))
+
+
+def runcmd_exit_code_is_nonzero(ctx):
+ runcmd_exit_code_is_not(ctx, exit=0)
+
+
+def runcmd_exit_code_is_not(ctx, exit=None):
+ assert_ne = globals()["assert_ne"]
+ assert_ne(runcmd_get_exit_code(ctx), int(exit))
+
+
+#
+# Step functions and helpers for examining output in various ways.
+#
+
+
+def runcmd_stdout_is(ctx, text=None):
+ _runcmd_output_is(runcmd_get_stdout(ctx), text)
+
+
+def runcmd_stdout_isnt(ctx, text=None):
+ _runcmd_output_isnt(runcmd_get_stdout(ctx), text)
+
+
+def runcmd_stderr_is(ctx, text=None):
+ _runcmd_output_is(runcmd_get_stderr(ctx), text)
+
+
+def runcmd_stderr_isnt(ctx, text=None):
+ _runcmd_output_isnt(runcmd_get_stderr(ctx), text)
+
+
+def _runcmd_output_is(actual, wanted):
+ assert_eq = globals()["assert_eq"]
+ wanted = bytes(wanted, "utf8").decode("unicode_escape")
+ logging.debug("_runcmd_output_is:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" wanted: {wanted!r}")
+ assert_eq(actual, wanted)
+
+
+def _runcmd_output_isnt(actual, wanted):
+ assert_ne = globals()["assert_ne"]
+ wanted = bytes(wanted, "utf8").decode("unicode_escape")
+ logging.debug("_runcmd_output_isnt:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" wanted: {wanted!r}")
+ assert_ne(actual, wanted)
+
+
+def runcmd_stdout_contains(ctx, text=None):
+ _runcmd_output_contains(runcmd_get_stdout(ctx), text)
+
+
+def runcmd_stdout_doesnt_contain(ctx, text=None):
+ _runcmd_output_doesnt_contain(runcmd_get_stdout(ctx), text)
+
+
+def runcmd_stderr_contains(ctx, text=None):
+ _runcmd_output_contains(runcmd_get_stderr(ctx), text)
+
+
+def runcmd_stderr_doesnt_contain(ctx, text=None):
+ _runcmd_output_doesnt_contain(runcmd_get_stderr(ctx), text)
+
+
+def _runcmd_output_contains(actual, wanted):
+ assert_eq = globals()["assert_eq"]
+ wanted = bytes(wanted, "utf8").decode("unicode_escape")
+ logging.debug("_runcmd_output_contains:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" wanted: {wanted!r}")
+ assert_eq(wanted in actual, True)
+
+
+def _runcmd_output_doesnt_contain(actual, wanted):
+ assert_ne = globals()["assert_ne"]
+ wanted = bytes(wanted, "utf8").decode("unicode_escape")
+ logging.debug("_runcmd_output_doesnt_contain:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" wanted: {wanted!r}")
+ assert_ne(wanted in actual, True)
+
+
+def runcmd_stdout_matches_regex(ctx, regex=None):
+ _runcmd_output_matches_regex(runcmd_get_stdout(ctx), regex)
+
+
+def runcmd_stdout_doesnt_match_regex(ctx, regex=None):
+ _runcmd_output_doesnt_match_regex(runcmd_get_stdout(ctx), regex)
+
+
+def runcmd_stderr_matches_regex(ctx, regex=None):
+ _runcmd_output_matches_regex(runcmd_get_stderr(ctx), regex)
+
+
+def runcmd_stderr_doesnt_match_regex(ctx, regex=None):
+ _runcmd_output_doesnt_match_regex(runcmd_get_stderr(ctx), regex)
+
+
+def _runcmd_output_matches_regex(actual, regex):
+ assert_ne = globals()["assert_ne"]
+ r = re.compile(regex)
+ m = r.search(actual)
+ logging.debug("_runcmd_output_matches_regex:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" regex: {regex!r}")
+ logging.debug(f" match: {m}")
+ assert_ne(m, None)
+
+
+def _runcmd_output_doesnt_match_regex(actual, regex):
+ assert_eq = globals()["assert_eq"]
+ r = re.compile(regex)
+ m = r.search(actual)
+ logging.debug("_runcmd_output_doesnt_match_regex:")
+ logging.debug(f" actual: {actual!r}")
+ logging.debug(f" regex: {regex!r}")
+ logging.debug(f" match: {m}")
+ assert_eq(m, None)
diff --git a/subplot/vendor/runcmd.yaml b/subplot/vendor/runcmd.yaml
new file mode 100644
index 0000000..48dde90
--- /dev/null
+++ b/subplot/vendor/runcmd.yaml
@@ -0,0 +1,83 @@
+# Steps to run commands.
+
+- given: helper script {filename} for runcmd
+ function: runcmd_helper_script
+
+- given: srcdir is in the PATH
+ function: runcmd_helper_srcdir_path
+
+- when: I run (?P<argv0>\S+)(?P<args>.*)
+ regex: true
+ function: runcmd_step
+
+- when: I try to run (?P<argv0>\S+)(?P<args>.*)
+ regex: true
+ function: runcmd_try_to_run
+
+# Steps to examine exit code of latest command.
+
+- then: exit code is {exit}
+ function: runcmd_exit_code_is
+
+- then: exit code is not {exit}
+ function: runcmd_exit_code_is_not
+
+- then: command is successful
+ function: runcmd_exit_code_is_zero
+
+- then: command fails
+ function: runcmd_exit_code_is_nonzero
+
+# Steps to examine stdout/stderr for exact content.
+
+- then: stdout is exactly "(?P<text>.*)"
+ regex: true
+ function: runcmd_stdout_is
+
+- then: "stdout isn't exactly \"(?P<text>.*)\""
+ regex: true
+ function: runcmd_stdout_isnt
+
+- then: stderr is exactly "(?P<text>.*)"
+ regex: true
+ function: runcmd_stderr_is
+
+- then: "stderr isn't exactly \"(?P<text>.*)\""
+ regex: true
+ function: runcmd_stderr_isnt
+
+# Steps to examine stdout/stderr for sub-strings.
+
+- then: stdout contains "(?P<text>.*)"
+ regex: true
+ function: runcmd_stdout_contains
+
+- then: "stdout doesn't contain \"(?P<text>.*)\""
+ regex: true
+ function: runcmd_stdout_doesnt_contain
+
+- then: stderr contains "(?P<text>.*)"
+ regex: true
+ function: runcmd_stderr_contains
+
+- then: "stderr doesn't contain \"(?P<text>.*)\""
+ regex: true
+ function: runcmd_stderr_doesnt_contain
+
+# Steps to match stdout/stderr against regular expressions.
+
+- then: stdout matches regex (?P<regex>.*)
+ regex: true
+ function: runcmd_stdout_matches_regex
+
+- then: stdout doesn't match regex (?P<regex>.*)
+ regex: true
+ function: runcmd_stdout_doesnt_match_regex
+
+- then: stderr matches regex (?P<regex>.*)
+ regex: true
+ function: runcmd_stderr_matches_regex
+
+- then: stderr doesn't match regex (?P<regex>.*)
+ regex: true
+ function: runcmd_stderr_doesnt_match_regex