tests: Move Python tests out of share/

Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>

5 files changed, 532 insertions, 0 deletions

diff --git a/tests/python/daemon.md b/tests/python/daemon.md
new file mode 100644
index 0000000..285f9f8
--- /dev/null
+++ b/tests/python/daemon.md
@@ -0,0 +1,189 @@
+# Introduction
+
+The [Subplot][] library `daemon` for Python provides scenario steps
+and their implementations for running a background process and
+terminating at the end of the scenario.
+
+[Subplot]: https://subplot.liw.fi/
+
+This document explains the acceptance criteria for the library and how
+they're verified. It uses the steps and functions from the
+`lib/daemon` library. The scenarios all have the same structure: run a
+command, then examine the exit code, verify the process is running.
+
+# Daemon is started and terminated
+
+This scenario starts a background process, verifies it's started, and
+verifies it's terminated after the scenario ends.
+
+~~~scenario
+given there is no "/bin/sleep 12765" process
+when I start "/bin/sleep 12765" as a background process as sleepyhead
+then a process "/bin/sleep 12765" is running
+when I stop background process sleepyhead
+then there is no "/bin/sleep 12765" process
+~~~
+
+
+# Daemon takes a while to open its port
+
+This scenario verifies that if the background process doesn't immediately start
+listening on its port, the daemon library handles that correctly. We do this
+with a helper script that waits 2 seconds before opening the port. The
+lib/daemon code will wait for the script by repeatedly trying to connect. Once
+successful, it immediately closes the port, which causes the script to
+terminate.
+
+~~~scenario
+given a daemon helper shell script slow-start-daemon.py
+given there is no "slow-start-daemon.py" process
+when I try to start "./slow-start-daemon.py" as slow-daemon, on port 8888
+then starting the daemon succeeds
+when I stop background process slow-daemon
+then there is no "slow-start-daemon.py" process
+~~~
+
+~~~{#slow-start-daemon.py .file .python .numberLines}
+#!/usr/bin/env python3
+
+import socket
+import time
+
+time.sleep(2)
+
+s = socket.socket()
+s.bind(("127.0.0.1", 8888))
+s.listen()
+
+(conn, _) = s.accept()
+conn.recv(1)
+s.close()
+
+print("OK")
+~~~
+
+# Daemon never opens the intended port
+
+This scenario verifies that if the background process never starts
+listening on its port, the daemon library handles that correctly.
+
+~~~scenario
+given there is no "/bin/sleep 12765" process
+when I try to start "/bin/sleep 12765" as sleepyhead, on port 8888
+then starting daemon fails with "ConnectionRefusedError"
+then a process "/bin/sleep 12765" is running
+when I stop background process sleepyhead
+then there is no "/bin/sleep 12765" process
+~~~
+
+
+# Daemon stdout and stderr are retrievable
+
+Sometimes it's useful for the step functions to be able to retrieve
+the stdout or stderr of of the daemon, after it's started, or even
+after it's terminated. This scenario verifies that `lib/daemon` can do
+that.
+
+~~~scenario
+given a daemon helper shell script chatty-daemon.sh
+given there is no "chatty-daemon" process
+when I start "./chatty-daemon.sh" as a background process as chatty-daemon
+when daemon chatty-daemon has produced output
+when I stop background process chatty-daemon
+then there is no "chatty-daemon" process
+then daemon chatty-daemon stdout is "hi there\n"
+then daemon chatty-daemon stderr is "hola\n"
+~~~
+
+We make for the daemon to exit, to work around a race condition: if
+the test program retrieves the daemon's output too fast, it may not
+have had time to produce it yet.
+
+
+~~~{#chatty-daemon.sh .file .sh .numberLines}
+#!/bin/bash
+
+set -euo pipefail
+
+trap 'exit 0' TERM
+
+echo hola 1>&2
+echo hi there
+~~~
+
+# Can specify additional environment variables for daemon
+
+Some daemons are configured through their environment rather than configuration
+files. This scenario verifies that a step can set arbitrary variables in the
+daemon's environment.
+
+~~~scenario
+when I start "/usr/bin/env" as a background process as env, with environment {"custom_variable": "has a Value"}
+when daemon env has produced output
+when I stop background process env
+then daemon env stdout contains "custom_variable=has a Value"
+~~~
+
+~~~scenario
+given a daemon helper shell script env-with-port.py
+when I try to start "./env-with-port.py 8765" as env-with-port, on port 8765, with environment {"custom_variable": "1337"}
+when I stop background process env-with-port
+then daemon env-with-port stdout contains "custom_variable=1337"
+~~~
+
+~~~scenario
+given a daemon helper shell script env-with-port.py
+when I start "./env-with-port.py 8766" as a background process as another-env-with-port, on port 8766, with environment {"subplot2": "000"}
+when daemon another-env-with-port has produced output
+when I stop background process another-env-with-port
+then daemon another-env-with-port stdout contains "subplot2=000"
+~~~
+
+It's important that these new environment variables are not inherited by the
+steps that follow. To verify that, we run one more scenario which *doesn't* set
+any variables, but checks that none of the variables we mentioned above are
+present.
+
+~~~scenario
+when I start "/usr/bin/env" as a background process as env2
+when daemon env2 has produced output
+when I stop background process env2
+then daemon env2 stdout doesn't contain "custom_variable=has a Value"
+then daemon env2 stdout doesn't contain "custom_variable=1337"
+then daemon env2 stdout doesn't contain "subplot2=000"
+~~~
+
+~~~{#env-with-port.py .file .python .numberLines}
+#!/usr/bin/env python3
+
+import os
+import socket
+import sys
+import time
+
+for (key, value) in os.environ.items():
+    print(f"{key}={value}")
+
+port = int(sys.argv[1])
+print(f"port is {port}")
+
+s = socket.socket()
+s.bind(("127.0.0.1", port))
+s.listen()
+
+(conn, _) = s.accept()
+conn.recv(1)
+s.close()
+~~~
+
+
+---
+title: Acceptance criteria for the lib/daemon Subplot library
+author: The Subplot project
+bindings:
+- lib/daemon.yaml
+template: python
+functions:
+- lib/daemon.py
+- lib/runcmd.py
+...
diff --git a/tests/python/files.md b/tests/python/files.md
new file mode 100644
index 0000000..5c96a2c
--- /dev/null
+++ b/tests/python/files.md
@@ -0,0 +1,105 @@
+# Introduction
+
+The [Subplot][] library `files` provides scenario steps and their
+implementations for managing files on the file system during tests.
+The library consists of a bindings file `lib/files.yaml` and
+implementations in Python in `lib/files.py`.
+
+[Subplot]: https://subplot.liw.fi/
+
+This document explains the acceptance criteria for the library and how
+they're verified. It uses the steps and functions from the `files`
+library.
+
+# Create on-disk files from embedded files
+
+Subplot allows the source document to embed test files, and the
+`files` library provides steps to create real, on-disk files from
+the embedded files.
+
+~~~scenario
+given file hello.txt
+then file hello.txt exists
+and file hello.txt contains "hello, world"
+and file other.txt does not exist
+given file other.txt from hello.txt
+then file other.txt exists
+and files hello.txt and other.txt match
+and only files hello.txt, other.txt exist
+~~~
+
+~~~{#hello.txt .file .numberLines}
+hello, world
+~~~
+
+
+# File metadata
+
+These steps create files and manage their metadata.
+
+~~~scenario
+given file hello.txt
+when I remember metadata for file hello.txt
+then file hello.txt has same metadata as before
+
+when I write "yo" to file hello.txt
+then file hello.txt has different metadata from before
+~~~
+
+# File modification time
+
+These steps manipulate and test file modification times.
+
+~~~scenario
+given file foo.dat has modification time 1970-01-02 03:04:05
+then file foo.dat has a very old modification time
+
+when I touch file foo.dat
+then file foo.dat has a very recent modification time
+~~~
+
+
+# File contents
+
+These steps verify contents of files.
+
+~~~scenario
+given file hello.txt
+then file hello.txt contains "hello, world"
+and file hello.txt matches regex "hello, .*"
+and file hello.txt matches regex /hello, .*/
+~~~
+
+# Directories
+
+There are also a large number of directory based steps and some directory
+based behaviour available in creating files which are available in the files
+library.
+
+```scenario
+given a directory first
+then directory first exists
+and directory first is empty
+and directory second does not exist
+when I remove directory first
+then directory first does not exist
+when I create directory second
+then directory second exists
+and directory second is empty
+given file second/third/hello.txt from hello.txt
+then directory second is not empty
+and directory second/third exists
+and directory second/third is not empty
+when I remove directory second
+then directory second does not exist
+```
+
+---
+title: Acceptance criteria for the files Subplot library
+author: The Subplot project
+template: python
+bindings:
+- lib/files.yaml
+functions:
+- lib/files.py
+...
diff --git a/tests/python/runcmd.md b/tests/python/runcmd.md
new file mode 100644
index 0000000..68465a8
--- /dev/null
+++ b/tests/python/runcmd.md
@@ -0,0 +1,214 @@
+# Introduction
+
+The [Subplot][] library `runcmd` for Python provides scenario steps
+and their implementations for running Unix commands and examining the
+results. The library consists of a bindings file `lib/runcmd.yaml` and
+implementations in Python in `lib/runcmd.py`. There is no Bash
+version.
+
+[Subplot]: https://subplot.liw.fi/
+
+This document explains the acceptance criteria for the library and how
+they're verified. It uses the steps and functions from the
+`lib/runcmd` library. The scenarios all have the same structure: run a
+command, then examine the exit code, standard output (stdout for
+short), or standard error output (stderr) of the command.
+
+The scenarios use the Unix commands `/bin/true` and `/bin/false` to
+generate exit codes, and `/bin/echo` to produce stdout. To generate
+stderr, they use the little helper script below.
+
+~~~{#err.sh .file .sh .numberLines}
+#!/bin/sh
+echo "$@" 1>&2
+~~~
+
+# Check exit code
+
+These scenarios verify the exit code. To make it easier to write
+scenarios in language that flows more naturally, there are a couple of
+variations.
+
+## Successful execution
+
+~~~scenario
+when I run /bin/true
+then exit code is 0
+and command is successful
+~~~
+
+## Successful execution in a sub-directory
+
+~~~scenario
+given a directory xyzzy
+when I run, in xyzzy, /bin/pwd
+then exit code is 0
+then command is successful
+then stdout contains "/xyzzy"
+~~~
+
+## Failed execution
+
+~~~scenario
+when I try to run /bin/false
+then exit code is not 0
+and command fails
+~~~
+
+## Failed execution in a sub-directory
+
+~~~scenario
+given a directory xyzzy
+when I try to run, in xyzzy, /bin/false
+then exit code is not 0
+and command fails
+~~~
+
+# Check we can prepend to $PATH
+
+This scenario verifies that we can add a directory to the beginning of
+the PATH environment variable, so that we can have `runcmd` invoke a
+binary from our build tree rather than from system directories. This
+is especially useful for testing new versions of software that's
+already installed on the system.
+
+~~~scenario
+given executable script ls from ls.sh
+when I prepend . to PATH
+when I run ls
+then command is successful
+then stdout contains "custom ls, not system ls"
+~~~
+
+~~~{#ls.sh .file .sh .numberLines}
+#!/bin/sh
+echo "custom ls, not system ls"
+~~~
+
+# Check output has what we want
+
+These scenarios verify that stdout or stderr do have something we want
+to have.
+
+## Check stdout is exactly as wanted
+
+Note that the string is surrounded by double quotes to make it clear
+to the reader what's inside. Also, C-style string escapes are
+understood.
+
+~~~scenario
+when I run /bin/echo hello, world
+then stdout is exactly "hello, world\n"
+~~~
+
+## Check stderr is exactly as wanted
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hello, world
+then stderr is exactly "hello, world\n"
+~~~
+
+## Check stdout using sub-string search
+
+Exact string comparisons are not always enough, so we can verify a
+sub-string is in output.
+
+~~~scenario
+when I run /bin/echo hello, world
+then stdout contains "world\n"
+and exit code is 0
+~~~
+
+## Check stderr using sub-string search
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hello, world
+then stderr contains "world\n"
+~~~
+
+## Check stdout using regular expressions
+
+Fixed strings are not always enough, so we can verify output matches a
+regular expression. Note that the regular expression is not delimited
+and does not get any C-style string escaped decoded.
+
+~~~scenario
+when I run /bin/echo hello, world
+then stdout matches regex world$
+~~~
+
+## Check stderr using regular expressions
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hello, world
+then stderr matches regex world$
+~~~
+
+# Check output doesn't have what we want to avoid
+
+These scenarios verify that the stdout or stderr do not
+have something we want to avoid.
+
+## Check stdout is not exactly something
+
+~~~scenario
+when I run /bin/echo hi
+then stdout isn't exactly "hello, world\n"
+~~~
+
+## Check stderr is not exactly something
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hi
+then stderr isn't exactly "hello, world\n"
+~~~
+
+## Check stdout doesn't contain sub-string
+
+~~~scenario
+when I run /bin/echo hi
+then stdout doesn't contain "world"
+~~~
+
+## Check stderr doesn't contain sub-string
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hi
+then stderr doesn't contain "world"
+~~~
+
+## Check stdout doesn't match regular expression
+
+~~~scenario
+when I run /bin/echo hi
+then stdout doesn't match regex world$
+
+~~~
+
+## Check stderr doesn't match regular expressions
+
+~~~scenario
+given helper script err.sh for runcmd
+when I run sh err.sh hi
+then stderr doesn't match regex world$
+~~~
+
+
+---
+title: Acceptance criteria for the lib/runcmd Subplot library
+author: The Subplot project
+template: python
+bindings:
+- lib/runcmd.yaml
+- runcmd_test.yaml
+- lib/files.yaml
+functions:
+- lib/runcmd.py
+- runcmd_test.py
+- lib/files.py
+...
diff --git a/tests/python/runcmd_test.py b/tests/python/runcmd_test.py
new file mode 100644
index 0000000..4aa5f49
--- /dev/null
+++ b/tests/python/runcmd_test.py
@@ -0,0 +1,15 @@
+import os
+
+
+def create_script_from_embedded(ctx, filename=None, embedded=None):
+    files_create_from_embedded_with_other_name = globals()[
+        "files_create_from_embedded_with_other_name"
+    ]
+
+    # Create the file.
+    files_create_from_embedded_with_other_name(
+        ctx, filename_on_disk=filename, embedded_file=embedded
+    )
+
+    # Make the new file executable.
+    os.chmod(filename, 0o755)
diff --git a/tests/python/runcmd_test.yaml b/tests/python/runcmd_test.yaml
new file mode 100644
index 0000000..2ad981e
--- /dev/null
+++ b/tests/python/runcmd_test.yaml
@@ -0,0 +1,9 @@
+- given: "executable script {filename} from {embedded}"
+  impl:
+    python:
+      function: create_script_from_embedded
+
+- when: "I prepend {dirname} to PATH"
+  impl:
+    python:
+      function: runcmd_prepend_to_path