From 95048665bfec3d981c76d01df96d81dbf9597a3b Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Sun, 13 Dec 2020 12:40:16 +0200 Subject: doc: add start of architecture and acceptance criteria document --- contractor.md | 401 +++++++++++++++++++++++++++++++++++++++++++++ subplot/contractor.py | 25 +++ subplot/contractor.yaml | 5 + subplot/files.py | 158 ++++++++++++++++++ subplot/files.yaml | 58 +++++++ subplot/vendor/runcmd.py | 252 ++++++++++++++++++++++++++++ subplot/vendor/runcmd.yaml | 83 ++++++++++ 7 files changed, 982 insertions(+) create mode 100644 contractor.md create mode 100644 subplot/contractor.py create mode 100644 subplot/contractor.yaml create mode 100644 subplot/files.py create mode 100644 subplot/files.yaml create mode 100644 subplot/vendor/runcmd.py create mode 100644 subplot/vendor/runcmd.yaml diff --git a/contractor.md b/contractor.md new file mode 100644 index 0000000..c76dc62 --- /dev/null +++ b/contractor.md @@ -0,0 +1,401 @@ + + + +# Introduction + +Software development is a security risk. + +Building software from source code and running it is a core activity +of software development. Software developers do it on the machine they +work on. Continuous integration systems do it on server. These are +fundamentally the same. The process is roughly as follows: + +* install any dependencies +* build the software +* run the software to perform automated tests on it + +When the software is run, even if only a small unit of it, it can do +anything that the person running the build can do. For example, it can +do any and all of the following, unless constrained: + +* delete files +* modify files +* log into remote hosts using SSH +* decrypt or sign files with PGP +* send email +* delete email +* commit to version control repositories +* do anything with the browser that a the person could do +* run things as root using sudo +* in general, cause mayhem and chaos + +Normally, a software developer can assume that the code they wrote +themselves won't ever do any of that. They can even assume that people +they work with make code that won't do any of that. In both cases, +they may be wrong: mistakes happen. It's a well-guarded secret among +programmers that they sometimes, even if rarely, make catastrophic +mistakes. + +Accidents aside, mayhem and chaos may be intentional. Your own project +may not have malware, and you may have vetted all your dependencies, +and you trust them. But your dependencies have dependencies, which +have further dependencies, which have dependencies of their own. You'd +need to vet the whole dependency tree. Even decades ago, in the 1990s, +this could easily be hundreds of thousands of lines of code, and +modern systems a much larger. Note that build tools are themselves +dependencies, as is the whole operating system. Any code that is invoked +in the build process is a dependency. + +How certain are you that you can spot malicious code that's +intentionally hidden and obfuscated? + +Are you prepared to vet any changes to any transitive dependencies? + +Does this really matter? Maybe it doesn't. If you can't ever do +anything on your computer that would affect you or anyone else in a +negative way, it probably doesn't matter. Most software developers are +not in that position. + +This risk affects every operating system and every programming +language. The degree in which it exists varies, a lot. Some +programming language ecosystems seem more vulnerable than others: the +nodejs/npm one, for example, values tiny and highly focused packages, +which leads to immense dependency trees. The more direct or indirect +dependencies there are, the higher the chance that one of them turns +out to be bad. + +The risk also exists for more traditional languages, such as C. Few C +programs have no dependencies. They all need a C compiler, which in +turns requires an operating system, at least. + +The risk is there for both free software systems, and non-free ones. +As an example, the Debian system is entirely free software, but it's +huge: the Debian 10 (buster) release has over 50 thousand packages, +maintained by thousands of people. While it's probable that none of +those packages contains actual malware, it's not certain. Even if +everyone who helps maintain is completely trustworthy, the amount of +software in Debian is much too large for all code to be +comprehensively reviewed. + +This is true for all operating systems that are not mere toys. + +The conclusion here is that to build software securely, we can't +assume all code involved in the build to be secure. We need something +more secure. The Contractor aims to be a possible solution. + +## Threat model + +This section collects a list of specific threats to consider. + +* accessing or modifying files not part of the build +* excessive use build host resources + * e.g., CPU, GPU, RAM, disk, etc + * this might happen to make unauthorized use of the resources, or to + just be wasteful +* excessive use of network bandwidth +* attack on a networked target via a denial of service attack + * e.g., build joins a DDoS swarm, or sends fabricated SYN packets to + prevent target from working +* attack on build host, or other host, via network intrusion + * e.g., port scanning, probing for known vulnerabilities +* attack build host directly without network + * e.g., by breaching security isolation using build host kernel or + hardware vulnerabilities, or CI engine vulnerabilities + * this includes eavesdropping on the host, and stealing secrets + +## Status of this document + +Everything about the Contractor is in its early stages of thinking, +sketching, experimentations, and planning. Nothing is nailed down yet. + +Pre-ALPHA. Don't trust anything. Anything you trust may be used +against you. Anything may change. + +# Requirements + +This chapter discusses the requirements for the Contractor solution. +The requirements are divided into two parts: one that's based on the +threat model, and another for requirements that aren't about security. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in [RFC 2119][]. + +[RFC 2119]: https://tools.ietf.org/html/rfc2119 + +## Security requirements + +These requirements stem from the threat model above. + +* **FilesystemIsolation**: The Contractor MUST prevent the build from + accessing or modifying any files outside the build. Build tools and + libraries outside the source tree MUST be usable. + +* The Contractor MUST prevent the build from using more than the + user-specified amount of CPU time (**HardCPULimit**), disk space + (**HardDiskLimit**), or network bandwidth (**HardBandwidthLimit**). + Any attempt by the build to use more should fail. The Contractor + MUST fail the build if the limits are exceeded. + +* **HardRAMLimit**: The Contractor MUST prevent the build from using + more than the user-specified amount of RAM. The Contractor MAY fail + to build if the limit is exceeded, but is not required to do so. + +* **ConstrainNetworkAccess**: The Contractor MUST prevent the build + from accessing the network ourside the build environment in ways + that haven't been specifically allowed by the user. The contractor + SHOULD fail the build if it makes an attempt at such access. The + user MUST be able to specify which hosts to access, and using which + protocols. + +* **HostProtection**: The Contractor SHOULD attempt to protect the + host its running on from non-networked attacks performed by the + build. This includes vulnerabilities of the host's operating system + kernel, virtualisation solution, and hardware. + +## Non-security requirements + +* **AnyBuildOS**: Builds SHOULD be able to run in any operating system + that can be run as a virtual machine guest of the host operating + system. + +* **NoRoot**: Running the Contractor SHOULD NOT require root + privileges. It's OK to require sufficient privileges to use + virtualisation. + +* **DefaultBuilder**: The Contractor SHOULD be easy to set up and to + use. It should not require extensive configuration. Running a build + should be as easy as running **make**(1) on the command line. It + should be feasible to expect developers to use the Contractor for + their normal development work. + + +# Architecture + +This chapter discusses the architecture of the solution, with +particular emphasis on threat mitigation. + +The overall solution is use of nested virtual machines, running on a +developer's host. The outer VM runs the Contractor itself, and is +called "the manager VM". The inner VM runs the build, and is called +the "worker VM". The manager VM controls the worker VM, proxies its +external access, and prevents it from doing anything nefarious. The +manager VM is managed by a command line tool. Developers only interact +directly with the command line tool. + +~~~dot +digraph "arch" { + labelloc=b; + labeljust=l; + dev [shape=octagon label="Developer"]; + img [shape=tab label="VM image"]; + src [shape=tab label="Source tree"]; + ws [shape=tab label="Exported workspace"]; + apt [shape=tab label="APT repository"]; + subgraph cluster_host { + label="Host system \n (the vulnerable bit)"; + contractor [label="Contractor CLI"]; + subgraph cluster_contractor { + label="Manager VM \n (defence force)"; + manager; + libvirt; + subgraph cluster_builder { + label="Worker VM \n (here be dragons)"; + style=filled; + fillcolor="#dd0000"; + guestos [label="Guest OS"]; + } + } + } + dev -> contractor; + contractor -> manager; + contractor -> guestos; + img -> contractor; + ws -> contractor; + src -> contractor; + apt -> guestos; + manager -> libvirt; + libvirt -> guestos; + contractor -> ws; +} +~~~ + +This high-level design is chosen for the following reasons: + +* it allows the build to happen in any operating system + (**AnyBuildOS**) +* the Contractor is a VM and running it doesn't require root + privileges; it has root inside both VMs if needed; all the + complexity of setting things up so the worker VM works correctly are + contained the manager VM, and the user need do only minimal + configuration (**NoRoot**) +* the command line tool for using the Contractor can be made to be as + easy as any build tool so that developer actually use it by default + (**DefaultBuilder**) +* the manager VM can monitor and control the build (**HardCPULimit**, + **HardBandwidthLimit**) +* the manager can supply the worker VM with only a specified amount of + RAM and disk space (**HardRAMLimit**, **HardDiskLimit**) +* the manager can set up routing and firewalls so that the worker VM + cannot access the network, except via proxies provided by the outer + VM (**ConstrainNetworkAccess**) +* the nested VMs provide a smaller attack surface than the Linux + kernel API, and this protects the host better than Linux container + technologies, although it doesn't do much to protect against + virtualisation or hardware vulnerabilities (**HostProtection**) + +## Build process + +The architecture leads to a build process that would work roughly like +this: + +* the manager VM is already running +* developer runs command line tool to do a build: + `contractor build foo.yaml` +* command line tool copies the worker VM image into the manager VM +* command line tool boots the worker VM +* command line tool installs any build dependencies into the worker VM +* command line tool copies a previously saved dump of the workspace + into the worker VM +* command line tool copies the source code and build recipe into the + worker VM's workspace +* command line tool runs build commands in the worker VM, in the + source tree +* command line tool copies out the workspace into a local directory +* command line tool reports to the developer build success or failure + and where build log and build artifacts are + +## Implementation sketch (FIXME: update) + +FIXME: write this + + + +# Acceptance criteria + +This chapter specifies acceptance criteria for the Contractor, as +*scenarios*, which also define how the criteria are automatically +verified. + +## Local use of the Contractor + +These scenarios use the Contractor locally, to make sure it can do +things that don't require the VM. + +### Parse build spec + +Make sure the Contractor can read a build spec and dump it back out, +as JSON. This exercises the parsing code. JSON output is chosen, +instead of YAML, to make sure the program doesn't just copy input to +output. + +~~~scenario +given file dump.yaml +when I invoke contractor dump dump.yaml +then the JSON output matches dump.yaml +~~~ + +~~~{.file #dump.yaml .yaml .numberLines} +worker-image: worker.img +ansible: + - hosts: worker + remote_user: worker + become: true + tasks: + - apt: + name: build-essential + vars: + ansible_python_interpreter: /usr/bin/python3 +source: . +workspace: workspace +build: | + ./check +~~~ + +## Smoke tests + +These scenarios build a simple "hello, world" C application on a +variety of guest systems, and verify the resulting binaries output the +desired greeting. The goal of these scenarios is to ensure the various +Contractor components fit together at least in the very basic case. + +### Debian smoke test + +This scenario checks that the developer can build a simple C program +in the Contractor. + +~~~disabled-scenario +given a working contractor +and file hello.c +and file hello.yaml +and file worker.img from source directory +when I run contractor build hello.yaml +then exit code is 0 +then file ws/src/hello exists +~~~ + +~~~{.file #hello.c .c .numberLines} +#include + +int main() +{ + printf("hello, world\n"); + return 0; +} +~~~ + +~~~{.file #hello.yaml .yaml .numberLines} +worker-image: worker.img +ansible: + - hosts: worker + remote_user: worker + become: true + tasks: + - apt: + name: build-essential + vars: + ansible_python_interpreter: /usr/bin/python3 +source: . +workspace: ws +build: | + gcc hello.c -o hello + ./hello +~~~ + + + +--- +title: "Contractor: build software securely" +author: "Lars Wirzenius" +bindings: +- subplot/contractor.yaml +- subplot/vendor/runcmd.yaml +- subplot/files.yaml +functions: + - subplot/contractor.py + - subplot/vendor/runcmd.py + - subplot/files.py +documentclass: report +classes: + - c + - disabled-scenario +abstract: | + Building software typically requires running code downloaded from + the Internet. Even when you're building your own software, you + usually depend on libraries and tools, which in turn may depend on + further things. It is becoming infeasible to vet the whole set of + software running during a build. If a build includes running local + tests (unit tests, some integration tests), the problem gets worse + in magnitude, if not quality. + + Some software ecosystems are especially vulnerable to this (nodejs, + Python, Ruby, Go, Rust), but it's true for anything that has + dependencies on any code from outside its own code base, and even if + all the dependencies come from a trusted source, such as the + operating system vendor or a Linux distribution. + + The Contractor is an attempt to be able to build software securely, + by leveraging virtual machine technology. It attempts to be + secure, convenient, and reasonably efficient. + +... diff --git a/subplot/contractor.py b/subplot/contractor.py new file mode 100644 index 0000000..a2569dd --- /dev/null +++ b/subplot/contractor.py @@ -0,0 +1,25 @@ +import json +import logging +import os +import yaml + + +def run_contractor_dump(ctx, filename=None): + runcmd_run = globals()["runcmd_run"] + srcdir = globals()["srcdir"] + argv = [os.path.join(srcdir, "target", "debug", "contractor"), "dump", filename] + runcmd_run(ctx, argv) + + +def stdout_json_matches_yaml_file(ctx, filename=None): + runcmd_get_stdout = globals()["runcmd_get_stdout"] + assert_dict_eq = globals()["assert_dict_eq"] + + stdout = runcmd_get_stdout(ctx) + logging.debug(f"stdout: {stdout!r}") + obj_json = json.loads(stdout) + logging.info(f"object from stdout: {obj_json!r}") + + obj_yaml = yaml.safe_load(open(filename)) + logging.info(f"object from yaml: {obj_yaml!r}") + assert_dict_eq(obj_json, obj_yaml) diff --git a/subplot/contractor.yaml b/subplot/contractor.yaml new file mode 100644 index 0000000..7a512b7 --- /dev/null +++ b/subplot/contractor.yaml @@ -0,0 +1,5 @@ +- when: I invoke contractor dump {filename} + function: run_contractor_dump + +- then: the JSON output matches {filename} + function: stdout_json_matches_yaml_file diff --git a/subplot/files.py b/subplot/files.py new file mode 100644 index 0000000..ec37b9d --- /dev/null +++ b/subplot/files.py @@ -0,0 +1,158 @@ +import logging +import os +import re +import time + + +def files_create_from_embedded(ctx, filename=None): + files_create_from_embedded_with_other_name( + ctx, filename_on_disk=filename, embedded_filename=filename + ) + + +def files_create_from_embedded_with_other_name( + ctx, filename_on_disk=None, embedded_filename=None +): + get_file = globals()["get_file"] + with open(filename_on_disk, "wb") as f: + f.write(get_file(embedded_filename)) + + +def files_create_from_text(ctx, filename=None, text=None): + with open(filename, "w") as f: + f.write(text) + + +def files_file_exists(ctx, filename=None): + assert_eq = globals()["assert_eq"] + assert_eq(os.path.exists(filename), True) + + +def files_file_does_not_exist(ctx, filename=None): + assert_eq = globals()["assert_eq"] + assert_eq(os.path.exists(filename), False) + + +def files_only_these_exist(ctx, filenames=None): + assert_eq = globals()["assert_eq"] + filenames = filenames.replace(",", "").split() + assert_eq(set(os.listdir(".")), set(filenames)) + + +def files_file_contains(ctx, filename=None, data=None): + assert_eq = globals()["assert_eq"] + with open(filename, "rb") as f: + actual = f.read() + actual = actual.decode("UTF-8") + assert_eq(data in actual, True) + + +def files_file_matches_regex(ctx, filename=None, regex=None): + assert_eq = globals()["assert_eq"] + with open(filename) as f: + content = f.read() + m = re.search(regex, content) + if m is None: + logging.debug(f"files_file_matches_regex: no match") + logging.debug(f" filenamed: {filename}") + logging.debug(f" regex: {regex}") + logging.debug(f" content: {regex}") + logging.debug(f" match: {m}") + assert_eq(bool(m), True) + + +def files_match(ctx, filename1=None, filename2=None): + assert_eq = globals()["assert_eq"] + with open(filename1, "rb") as f: + data1 = f.read() + with open(filename2, "rb") as f: + data2 = f.read() + assert_eq(data1, data2) + + +def files_touch_with_timestamp( + ctx, + filename=None, + year=None, + month=None, + day=None, + hour=None, + minute=None, + second=None, +): + t = ( + int(year), + int(month), + int(day), + int(hour), + int(minute), + int(second), + -1, + -1, + -1, + ) + ts = time.mktime(t) + _files_touch(filename, ts) + + +def files_touch(ctx, filename=None): + _files_touch(filename, None) + + +def _files_touch(filename, ts): + if not os.path.exists(filename): + open(filename, "w").close() + times = None + if ts is not None: + times = (ts, ts) + os.utime(filename, times=times) + + +def files_mtime_is_recent(ctx, filename=None): + st = os.stat(filename) + age = abs(st.st_mtime - time.time()) + assert age < 1.0 + + +def files_mtime_is_ancient(ctx, filename=None): + st = os.stat(filename) + age = abs(st.st_mtime - time.time()) + year = 365 * 24 * 60 * 60 + required = 39 * year + logging.debug(f"ancient? mtime={st.st_mtime} age={age} required={required}") + assert age > required + + +def files_remember_metadata(ctx, filename=None): + meta = _files_remembered(ctx) + meta[filename] = _files_get_metadata(filename) + logging.debug("files_remember_metadata:") + logging.debug(f" meta: {meta}") + logging.debug(f" ctx: {ctx}") + + +# Check that current metadata of a file is as stored in the context. +def files_has_remembered_metadata(ctx, filename=None): + assert_eq = globals()["assert_eq"] + meta = _files_remembered(ctx) + logging.debug("files_has_remembered_metadata:") + logging.debug(f" meta: {meta}") + logging.debug(f" ctx: {ctx}") + assert_eq(meta[filename], _files_get_metadata(filename)) + + +def files_has_different_metadata(ctx, filename=None): + assert_ne = globals()["assert_ne"] + meta = _files_remembered(ctx) + assert_ne(meta[filename], _files_get_metadata(filename)) + + +def _files_remembered(ctx): + ns = ctx.declare("_files") + return ns.get("remembered-metadata", {}) + + +def _files_get_metadata(filename): + st = os.lstat(filename) + keys = ["st_dev", "st_gid", "st_ino", "st_mode", "st_mtime", "st_size", "st_uid"] + return {key: getattr(st, key) for key in keys} diff --git a/subplot/files.yaml b/subplot/files.yaml new file mode 100644 index 0000000..5777a8e --- /dev/null +++ b/subplot/files.yaml @@ -0,0 +1,58 @@ +- given: file {filename} + function: files_create_from_embedded + +- given: file {filename_on_disk} from {embedded_filename} + function: files_create_from_embedded_with_other_name + +- given: file {filename} has modification time {year}-{month}-{day} {hour}:{minute}:{second} + function: files_touch_with_timestamp + +- when: I write "(?P.*)" to file (?P\S+) + regex: true + function: files_create_from_text + +- when: I remember metadata for file {filename} + function: files_remember_metadata + +- when: I touch file {filename} + function: files_touch + +- then: file {filename} exists + function: files_file_exists + +- then: file {filename} does not exist + function: files_file_does_not_exist + +- then: only files (?P.+) exist + function: files_only_these_exist + regex: true + +- then: file (?P\S+) contains "(?P.*)" + regex: true + function: files_file_contains + +- then: file (?P\S+) matches regex /(?P.*)/ + regex: true + function: files_file_matches_regex + +- then: file (?P\S+) matches regex "(?P.*)" + regex: true + function: files_file_matches_regex + +- then: files {filename1} and {filename2} match + function: files_match + +- then: file {filename} has same metadata as before + function: files_has_remembered_metadata + +- then: file {filename} has different metadata from before + function: files_has_different_metadata + +- then: file {filename} has changed from before + function: files_has_different_metadata + +- then: file {filename} has a very recent modification time + function: files_mtime_is_recent + +- then: file {filename} has a very old modification time + function: files_mtime_is_ancient diff --git a/subplot/vendor/runcmd.py b/subplot/vendor/runcmd.py new file mode 100644 index 0000000..a2564c6 --- /dev/null +++ b/subplot/vendor/runcmd.py @@ -0,0 +1,252 @@ +import logging +import os +import re +import shlex +import subprocess + + +# +# Helper functions. +# + +# Get exit code or other stored data about the latest command run by +# runcmd_run. + + +def _runcmd_get(ctx, name): + ns = ctx.declare("_runcmd") + return ns[name] + + +def runcmd_get_exit_code(ctx): + return _runcmd_get(ctx, "exit") + + +def runcmd_get_stdout(ctx): + return _runcmd_get(ctx, "stdout") + + +def runcmd_get_stdout_raw(ctx): + return _runcmd_get(ctx, "stdout.raw") + + +def runcmd_get_stderr(ctx): + return _runcmd_get(ctx, "stderr") + + +def runcmd_get_stderr_raw(ctx): + return _runcmd_get(ctx, "stderr.raw") + + +def runcmd_get_argv(ctx): + return _runcmd_get(ctx, "argv") + + +# Run a command, given an argv and other arguments for subprocess.Popen. +# +# This is meant to be a helper function, not bound directly to a step. The +# stdout, stderr, and exit code are stored in the "_runcmd" namespace in the +# ctx context. +def runcmd_run(ctx, argv, **kwargs): + ns = ctx.declare("_runcmd") + + # The Subplot Python template empties os.environ at startup, modulo a small + # number of variables with carefully chosen values. Here, we don't need to + # care about what those variables are, but we do need to not overwrite + # them, so we just add anything in the env keyword argument, if any, to + # os.environ. + env = dict(os.environ) + for key, arg in kwargs.pop("env", {}).items(): + env[key] = arg + + pp = ns.get("path-prefix") + if pp: + env["PATH"] = pp + ":" + env["PATH"] + + logging.debug(f"runcmd_run") + logging.debug(f" argv: {argv}") + logging.debug(f" env: {env}") + p = subprocess.Popen( + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, **kwargs + ) + stdout, stderr = p.communicate("") + ns["argv"] = argv + ns["stdout.raw"] = stdout + ns["stderr.raw"] = stderr + ns["stdout"] = stdout.decode("utf-8") + ns["stderr"] = stderr.decode("utf-8") + ns["exit"] = p.returncode + logging.debug(f" ctx: {ctx}") + logging.debug(f" ns: {ns}") + + +# Step: prepend srcdir to PATH whenever runcmd runs a command. +def runcmd_helper_srcdir_path(ctx): + srcdir = globals()["srcdir"] + runcmd_prepend_to_path(ctx, srcdir) + + +# Step: This creates a helper script. +def runcmd_helper_script(ctx, filename=None): + get_file = globals()["get_file"] + with open(filename, "wb") as f: + f.write(get_file(filename)) + + +# +# Step functions for running commands. +# + + +def runcmd_prepend_to_path(ctx, dirname=None): + ns = ctx.declare("_runcmd") + pp = ns.get("path-prefix", "") + if pp: + pp = f"{pp}:{dirname}" + else: + pp = dirname + ns["path-prefix"] = pp + + +def runcmd_step(ctx, argv0=None, args=None): + runcmd_try_to_run(ctx, argv0=argv0, args=args) + runcmd_exit_code_is_zero(ctx) + + +def runcmd_try_to_run(ctx, argv0=None, args=None): + argv = [shlex.quote(argv0)] + shlex.split(args) + runcmd_run(ctx, argv) + + +# +# Step functions for examining exit codes. +# + + +def runcmd_exit_code_is_zero(ctx): + runcmd_exit_code_is(ctx, exit=0) + + +def runcmd_exit_code_is(ctx, exit=None): + assert_eq = globals()["assert_eq"] + assert_eq(runcmd_get_exit_code(ctx), int(exit)) + + +def runcmd_exit_code_is_nonzero(ctx): + runcmd_exit_code_is_not(ctx, exit=0) + + +def runcmd_exit_code_is_not(ctx, exit=None): + assert_ne = globals()["assert_ne"] + assert_ne(runcmd_get_exit_code(ctx), int(exit)) + + +# +# Step functions and helpers for examining output in various ways. +# + + +def runcmd_stdout_is(ctx, text=None): + _runcmd_output_is(runcmd_get_stdout(ctx), text) + + +def runcmd_stdout_isnt(ctx, text=None): + _runcmd_output_isnt(runcmd_get_stdout(ctx), text) + + +def runcmd_stderr_is(ctx, text=None): + _runcmd_output_is(runcmd_get_stderr(ctx), text) + + +def runcmd_stderr_isnt(ctx, text=None): + _runcmd_output_isnt(runcmd_get_stderr(ctx), text) + + +def _runcmd_output_is(actual, wanted): + assert_eq = globals()["assert_eq"] + wanted = bytes(wanted, "utf8").decode("unicode_escape") + logging.debug("_runcmd_output_is:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" wanted: {wanted!r}") + assert_eq(actual, wanted) + + +def _runcmd_output_isnt(actual, wanted): + assert_ne = globals()["assert_ne"] + wanted = bytes(wanted, "utf8").decode("unicode_escape") + logging.debug("_runcmd_output_isnt:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" wanted: {wanted!r}") + assert_ne(actual, wanted) + + +def runcmd_stdout_contains(ctx, text=None): + _runcmd_output_contains(runcmd_get_stdout(ctx), text) + + +def runcmd_stdout_doesnt_contain(ctx, text=None): + _runcmd_output_doesnt_contain(runcmd_get_stdout(ctx), text) + + +def runcmd_stderr_contains(ctx, text=None): + _runcmd_output_contains(runcmd_get_stderr(ctx), text) + + +def runcmd_stderr_doesnt_contain(ctx, text=None): + _runcmd_output_doesnt_contain(runcmd_get_stderr(ctx), text) + + +def _runcmd_output_contains(actual, wanted): + assert_eq = globals()["assert_eq"] + wanted = bytes(wanted, "utf8").decode("unicode_escape") + logging.debug("_runcmd_output_contains:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" wanted: {wanted!r}") + assert_eq(wanted in actual, True) + + +def _runcmd_output_doesnt_contain(actual, wanted): + assert_ne = globals()["assert_ne"] + wanted = bytes(wanted, "utf8").decode("unicode_escape") + logging.debug("_runcmd_output_doesnt_contain:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" wanted: {wanted!r}") + assert_ne(wanted in actual, True) + + +def runcmd_stdout_matches_regex(ctx, regex=None): + _runcmd_output_matches_regex(runcmd_get_stdout(ctx), regex) + + +def runcmd_stdout_doesnt_match_regex(ctx, regex=None): + _runcmd_output_doesnt_match_regex(runcmd_get_stdout(ctx), regex) + + +def runcmd_stderr_matches_regex(ctx, regex=None): + _runcmd_output_matches_regex(runcmd_get_stderr(ctx), regex) + + +def runcmd_stderr_doesnt_match_regex(ctx, regex=None): + _runcmd_output_doesnt_match_regex(runcmd_get_stderr(ctx), regex) + + +def _runcmd_output_matches_regex(actual, regex): + assert_ne = globals()["assert_ne"] + r = re.compile(regex) + m = r.search(actual) + logging.debug("_runcmd_output_matches_regex:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" regex: {regex!r}") + logging.debug(f" match: {m}") + assert_ne(m, None) + + +def _runcmd_output_doesnt_match_regex(actual, regex): + assert_eq = globals()["assert_eq"] + r = re.compile(regex) + m = r.search(actual) + logging.debug("_runcmd_output_doesnt_match_regex:") + logging.debug(f" actual: {actual!r}") + logging.debug(f" regex: {regex!r}") + logging.debug(f" match: {m}") + assert_eq(m, None) diff --git a/subplot/vendor/runcmd.yaml b/subplot/vendor/runcmd.yaml new file mode 100644 index 0000000..48dde90 --- /dev/null +++ b/subplot/vendor/runcmd.yaml @@ -0,0 +1,83 @@ +# Steps to run commands. + +- given: helper script {filename} for runcmd + function: runcmd_helper_script + +- given: srcdir is in the PATH + function: runcmd_helper_srcdir_path + +- when: I run (?P\S+)(?P.*) + regex: true + function: runcmd_step + +- when: I try to run (?P\S+)(?P.*) + regex: true + function: runcmd_try_to_run + +# Steps to examine exit code of latest command. + +- then: exit code is {exit} + function: runcmd_exit_code_is + +- then: exit code is not {exit} + function: runcmd_exit_code_is_not + +- then: command is successful + function: runcmd_exit_code_is_zero + +- then: command fails + function: runcmd_exit_code_is_nonzero + +# Steps to examine stdout/stderr for exact content. + +- then: stdout is exactly "(?P.*)" + regex: true + function: runcmd_stdout_is + +- then: "stdout isn't exactly \"(?P.*)\"" + regex: true + function: runcmd_stdout_isnt + +- then: stderr is exactly "(?P.*)" + regex: true + function: runcmd_stderr_is + +- then: "stderr isn't exactly \"(?P.*)\"" + regex: true + function: runcmd_stderr_isnt + +# Steps to examine stdout/stderr for sub-strings. + +- then: stdout contains "(?P.*)" + regex: true + function: runcmd_stdout_contains + +- then: "stdout doesn't contain \"(?P.*)\"" + regex: true + function: runcmd_stdout_doesnt_contain + +- then: stderr contains "(?P.*)" + regex: true + function: runcmd_stderr_contains + +- then: "stderr doesn't contain \"(?P.*)\"" + regex: true + function: runcmd_stderr_doesnt_contain + +# Steps to match stdout/stderr against regular expressions. + +- then: stdout matches regex (?P.*) + regex: true + function: runcmd_stdout_matches_regex + +- then: stdout doesn't match regex (?P.*) + regex: true + function: runcmd_stdout_doesnt_match_regex + +- then: stderr matches regex (?P.*) + regex: true + function: runcmd_stderr_matches_regex + +- then: stderr doesn't match regex (?P.*) + regex: true + function: runcmd_stderr_doesnt_match_regex -- cgit v1.2.1