From f99f5c188693b877a927b61e686a556a7f15ecbd Mon Sep 17 00:00:00 2001 From: Daniel Silverstone Date: Fri, 3 Sep 2021 10:30:58 +0100 Subject: tests: Move Python tests out of share/ Signed-off-by: Daniel Silverstone --- share/python/lib/daemon.md | 189 --------------------------------- share/python/lib/files.md | 105 ------------------- share/python/lib/runcmd.md | 214 -------------------------------------- share/python/lib/runcmd_test.py | 15 --- share/python/lib/runcmd_test.yaml | 9 -- 5 files changed, 532 deletions(-) delete mode 100644 share/python/lib/daemon.md delete mode 100644 share/python/lib/files.md delete mode 100644 share/python/lib/runcmd.md delete mode 100644 share/python/lib/runcmd_test.py delete mode 100644 share/python/lib/runcmd_test.yaml (limited to 'share') diff --git a/share/python/lib/daemon.md b/share/python/lib/daemon.md deleted file mode 100644 index c7bb49f..0000000 --- a/share/python/lib/daemon.md +++ /dev/null @@ -1,189 +0,0 @@ -# Introduction - -The [Subplot][] library `daemon` for Python provides scenario steps -and their implementations for running a background process and -terminating at the end of the scenario. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the -`lib/daemon` library. The scenarios all have the same structure: run a -command, then examine the exit code, verify the process is running. - -# Daemon is started and terminated - -This scenario starts a background process, verifies it's started, and -verifies it's terminated after the scenario ends. - -~~~scenario -given there is no "/bin/sleep 12765" process -when I start "/bin/sleep 12765" as a background process as sleepyhead -then a process "/bin/sleep 12765" is running -when I stop background process sleepyhead -then there is no "/bin/sleep 12765" process -~~~ - - -# Daemon takes a while to open its port - -This scenario verifies that if the background process doesn't immediately start -listening on its port, the daemon library handles that correctly. We do this -with a helper script that waits 2 seconds before opening the port. The -lib/daemon code will wait for the script by repeatedly trying to connect. Once -successful, it immediately closes the port, which causes the script to -terminate. - -~~~scenario -given a daemon helper shell script slow-start-daemon.py -given there is no "slow-start-daemon.py" process -when I try to start "./slow-start-daemon.py" as slow-daemon, on port 8888 -then starting the daemon succeeds -when I stop background process slow-daemon -then there is no "slow-start-daemon.py" process -~~~ - -~~~{#slow-start-daemon.py .file .python .numberLines} -#!/usr/bin/env python3 - -import socket -import time - -time.sleep(2) - -s = socket.socket() -s.bind(("127.0.0.1", 8888)) -s.listen() - -(conn, _) = s.accept() -conn.recv(1) -s.close() - -print("OK") -~~~ - -# Daemon never opens the intended port - -This scenario verifies that if the background process never starts -listening on its port, the daemon library handles that correctly. - -~~~scenario -given there is no "/bin/sleep 12765" process -when I try to start "/bin/sleep 12765" as sleepyhead, on port 8888 -then starting daemon fails with "ConnectionRefusedError" -then a process "/bin/sleep 12765" is running -when I stop background process sleepyhead -then there is no "/bin/sleep 12765" process -~~~ - - -# Daemon stdout and stderr are retrievable - -Sometimes it's useful for the step functions to be able to retrieve -the stdout or stderr of of the daemon, after it's started, or even -after it's terminated. This scenario verifies that `lib/daemon` can do -that. - -~~~scenario -given a daemon helper shell script chatty-daemon.sh -given there is no "chatty-daemon" process -when I start "./chatty-daemon.sh" as a background process as chatty-daemon -when daemon chatty-daemon has produced output -when I stop background process chatty-daemon -then there is no "chatty-daemon" process -then daemon chatty-daemon stdout is "hi there\n" -then daemon chatty-daemon stderr is "hola\n" -~~~ - -We make for the daemon to exit, to work around a race condition: if -the test program retrieves the daemon's output too fast, it may not -have had time to produce it yet. - - -~~~{#chatty-daemon.sh .file .sh .numberLines} -#!/bin/bash - -set -euo pipefail - -trap 'exit 0' TERM - -echo hola 1>&2 -echo hi there -~~~ - -# Can specify additional environment variables for daemon - -Some daemons are configured through their environment rather than configuration -files. This scenario verifies that a step can set arbitrary variables in the -daemon's environment. - -~~~scenario -when I start "/usr/bin/env" as a background process as env, with environment {"custom_variable": "has a Value"} -when daemon env has produced output -when I stop background process env -then daemon env stdout contains "custom_variable=has a Value" -~~~ - -~~~scenario -given a daemon helper shell script env-with-port.py -when I try to start "./env-with-port.py 8765" as env-with-port, on port 8765, with environment {"custom_variable": "1337"} -when I stop background process env-with-port -then daemon env-with-port stdout contains "custom_variable=1337" -~~~ - -~~~scenario -given a daemon helper shell script env-with-port.py -when I start "./env-with-port.py 8766" as a background process as another-env-with-port, on port 8766, with environment {"subplot2": "000"} -when daemon another-env-with-port has produced output -when I stop background process another-env-with-port -then daemon another-env-with-port stdout contains "subplot2=000" -~~~ - -It's important that these new environment variables are not inherited by the -steps that follow. To verify that, we run one more scenario which *doesn't* set -any variables, but checks that none of the variables we mentioned above are -present. - -~~~scenario -when I start "/usr/bin/env" as a background process as env2 -when daemon env2 has produced output -when I stop background process env2 -then daemon env2 stdout doesn't contain "custom_variable=has a Value" -then daemon env2 stdout doesn't contain "custom_variable=1337" -then daemon env2 stdout doesn't contain "subplot2=000" -~~~ - -~~~{#env-with-port.py .file .python .numberLines} -#!/usr/bin/env python3 - -import os -import socket -import sys -import time - -for (key, value) in os.environ.items(): - print(f"{key}={value}") - -port = int(sys.argv[1]) -print(f"port is {port}") - -s = socket.socket() -s.bind(("127.0.0.1", port)) -s.listen() - -(conn, _) = s.accept() -conn.recv(1) -s.close() -~~~ - - ---- -title: Acceptance criteria for the lib/daemon Subplot library -author: The Subplot project -bindings: -- daemon.yaml -template: python -functions: -- daemon.py -- runcmd.py -... diff --git a/share/python/lib/files.md b/share/python/lib/files.md deleted file mode 100644 index 5638791..0000000 --- a/share/python/lib/files.md +++ /dev/null @@ -1,105 +0,0 @@ -# Introduction - -The [Subplot][] library `files` provides scenario steps and their -implementations for managing files on the file system during tests. -The library consists of a bindings file `lib/files.yaml` and -implementations in Python in `lib/files.py`. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the `files` -library. - -# Create on-disk files from embedded files - -Subplot allows the source document to embed test files, and the -`files` library provides steps to create real, on-disk files from -the embedded files. - -~~~scenario -given file hello.txt -then file hello.txt exists -and file hello.txt contains "hello, world" -and file other.txt does not exist -given file other.txt from hello.txt -then file other.txt exists -and files hello.txt and other.txt match -and only files hello.txt, other.txt exist -~~~ - -~~~{#hello.txt .file .numberLines} -hello, world -~~~ - - -# File metadata - -These steps create files and manage their metadata. - -~~~scenario -given file hello.txt -when I remember metadata for file hello.txt -then file hello.txt has same metadata as before - -when I write "yo" to file hello.txt -then file hello.txt has different metadata from before -~~~ - -# File modification time - -These steps manipulate and test file modification times. - -~~~scenario -given file foo.dat has modification time 1970-01-02 03:04:05 -then file foo.dat has a very old modification time - -when I touch file foo.dat -then file foo.dat has a very recent modification time -~~~ - - -# File contents - -These steps verify contents of files. - -~~~scenario -given file hello.txt -then file hello.txt contains "hello, world" -and file hello.txt matches regex "hello, .*" -and file hello.txt matches regex /hello, .*/ -~~~ - -# Directories - -There are also a large number of directory based steps and some directory -based behaviour available in creating files which are available in the files -library. - -```scenario -given a directory first -then directory first exists -and directory first is empty -and directory second does not exist -when I remove directory first -then directory first does not exist -when I create directory second -then directory second exists -and directory second is empty -given file second/third/hello.txt from hello.txt -then directory second is not empty -and directory second/third exists -and directory second/third is not empty -when I remove directory second -then directory second does not exist -``` - ---- -title: Acceptance criteria for the files Subplot library -author: The Subplot project -template: python -bindings: -- lib/files.yaml -functions: -- files.py -... diff --git a/share/python/lib/runcmd.md b/share/python/lib/runcmd.md deleted file mode 100644 index f165ad3..0000000 --- a/share/python/lib/runcmd.md +++ /dev/null @@ -1,214 +0,0 @@ -# Introduction - -The [Subplot][] library `runcmd` for Python provides scenario steps -and their implementations for running Unix commands and examining the -results. The library consists of a bindings file `lib/runcmd.yaml` and -implementations in Python in `lib/runcmd.py`. There is no Bash -version. - -[Subplot]: https://subplot.liw.fi/ - -This document explains the acceptance criteria for the library and how -they're verified. It uses the steps and functions from the -`lib/runcmd` library. The scenarios all have the same structure: run a -command, then examine the exit code, standard output (stdout for -short), or standard error output (stderr) of the command. - -The scenarios use the Unix commands `/bin/true` and `/bin/false` to -generate exit codes, and `/bin/echo` to produce stdout. To generate -stderr, they use the little helper script below. - -~~~{#err.sh .file .sh .numberLines} -#!/bin/sh -echo "$@" 1>&2 -~~~ - -# Check exit code - -These scenarios verify the exit code. To make it easier to write -scenarios in language that flows more naturally, there are a couple of -variations. - -## Successful execution - -~~~scenario -when I run /bin/true -then exit code is 0 -and command is successful -~~~ - -## Successful execution in a sub-directory - -~~~scenario -given a directory xyzzy -when I run, in xyzzy, /bin/pwd -then exit code is 0 -then command is successful -then stdout contains "/xyzzy" -~~~ - -## Failed execution - -~~~scenario -when I try to run /bin/false -then exit code is not 0 -and command fails -~~~ - -## Failed execution in a sub-directory - -~~~scenario -given a directory xyzzy -when I try to run, in xyzzy, /bin/false -then exit code is not 0 -and command fails -~~~ - -# Check we can prepend to $PATH - -This scenario verifies that we can add a directory to the beginning of -the PATH environment variable, so that we can have `runcmd` invoke a -binary from our build tree rather than from system directories. This -is especially useful for testing new versions of software that's -already installed on the system. - -~~~scenario -given executable script ls from ls.sh -when I prepend . to PATH -when I run ls -then command is successful -then stdout contains "custom ls, not system ls" -~~~ - -~~~{#ls.sh .file .sh .numberLines} -#!/bin/sh -echo "custom ls, not system ls" -~~~ - -# Check output has what we want - -These scenarios verify that stdout or stderr do have something we want -to have. - -## Check stdout is exactly as wanted - -Note that the string is surrounded by double quotes to make it clear -to the reader what's inside. Also, C-style string escapes are -understood. - -~~~scenario -when I run /bin/echo hello, world -then stdout is exactly "hello, world\n" -~~~ - -## Check stderr is exactly as wanted - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr is exactly "hello, world\n" -~~~ - -## Check stdout using sub-string search - -Exact string comparisons are not always enough, so we can verify a -sub-string is in output. - -~~~scenario -when I run /bin/echo hello, world -then stdout contains "world\n" -and exit code is 0 -~~~ - -## Check stderr using sub-string search - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr contains "world\n" -~~~ - -## Check stdout using regular expressions - -Fixed strings are not always enough, so we can verify output matches a -regular expression. Note that the regular expression is not delimited -and does not get any C-style string escaped decoded. - -~~~scenario -when I run /bin/echo hello, world -then stdout matches regex world$ -~~~ - -## Check stderr using regular expressions - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hello, world -then stderr matches regex world$ -~~~ - -# Check output doesn't have what we want to avoid - -These scenarios verify that the stdout or stderr do not -have something we want to avoid. - -## Check stdout is not exactly something - -~~~scenario -when I run /bin/echo hi -then stdout isn't exactly "hello, world\n" -~~~ - -## Check stderr is not exactly something - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr isn't exactly "hello, world\n" -~~~ - -## Check stdout doesn't contain sub-string - -~~~scenario -when I run /bin/echo hi -then stdout doesn't contain "world" -~~~ - -## Check stderr doesn't contain sub-string - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr doesn't contain "world" -~~~ - -## Check stdout doesn't match regular expression - -~~~scenario -when I run /bin/echo hi -then stdout doesn't match regex world$ - -~~~ - -## Check stderr doesn't match regular expressions - -~~~scenario -given helper script err.sh for runcmd -when I run sh err.sh hi -then stderr doesn't match regex world$ -~~~ - - ---- -title: Acceptance criteria for the lib/runcmd Subplot library -author: The Subplot project -template: python -bindings: -- lib/runcmd.yaml -- runcmd_test.yaml -- lib/files.yaml -functions: -- runcmd.py -- runcmd_test.py -- files.py -... diff --git a/share/python/lib/runcmd_test.py b/share/python/lib/runcmd_test.py deleted file mode 100644 index 4aa5f49..0000000 --- a/share/python/lib/runcmd_test.py +++ /dev/null @@ -1,15 +0,0 @@ -import os - - -def create_script_from_embedded(ctx, filename=None, embedded=None): - files_create_from_embedded_with_other_name = globals()[ - "files_create_from_embedded_with_other_name" - ] - - # Create the file. - files_create_from_embedded_with_other_name( - ctx, filename_on_disk=filename, embedded_file=embedded - ) - - # Make the new file executable. - os.chmod(filename, 0o755) diff --git a/share/python/lib/runcmd_test.yaml b/share/python/lib/runcmd_test.yaml deleted file mode 100644 index 2ad981e..0000000 --- a/share/python/lib/runcmd_test.yaml +++ /dev/null @@ -1,9 +0,0 @@ -- given: "executable script {filename} from {embedded}" - impl: - python: - function: create_script_from_embedded - -- when: "I prepend {dirname} to PATH" - impl: - python: - function: runcmd_prepend_to_path -- cgit v1.2.1