summaryrefslogtreecommitdiff
path: root/subplot.md
diff options
context:
space:
mode:
authorDaniel Silverstone <dsilvers@digital-scurf.org>2022-02-19 12:04:12 +0000
committerDaniel Silverstone <dsilvers@digital-scurf.org>2022-02-19 12:04:57 +0000
commit48d56b2cb4123db919c1a938766c19c94f7344da (patch)
tree23f67e7c316937776b587606eb992a6fa0dd905f /subplot.md
parent14f2349c1338b6bf6d1867d5d173c37b7782364d (diff)
downloadsubplot-48d56b2cb4123db919c1a938766c19c94f7344da.tar.gz
(doc): Add scenario runner comparison 'table'
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Diffstat (limited to 'subplot.md')
-rw-r--r--subplot.md26
1 files changed, 26 insertions, 0 deletions
diff --git a/subplot.md b/subplot.md
index 017cb5a..551a9c6 100644
--- a/subplot.md
+++ b/subplot.md
@@ -959,6 +959,32 @@ def file_ends_in_zero_newlines(ctx, filename=None):
assert_ne(content[-1], "\n")
~~~
+## Comparing the scenario runners
+
+Currently Subplot ships with three scenario runner templates. The
+Bash, Python, and Rust templates. The first two are fully self-contained
+and have a set of features dictated by the Subplot version. The
+latter is tied to how Cargo runs tests. Given that, this comparison
+is only considered correct against the version of Rust at the time of
+publishing a Subplot release. Newer versions of Rust may introduce
+additional functionality which we do not list here. Finally, we do
+not list features here which are considered fundamental, such as
+"runs all the scenarios" or "supports embedded files" since no template
+would be considered for release if it did not do these things. These
+are the differentiation points.
+
+```
+| Feature | Bash | Python | Rust |
+| ----------------------------- | ---------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------ |
+| Isolation model | Subprocess | Subprocess | Threads |
+| Parallelism | None | None | Threading |
+| Passing environment variables | CLI | CLI | Prefixed env vars |
+| Execution order | Fixed order | Randomised | Fixed order plus threading peturbation |
+| Run specific scenarios | Simple substring check | Simple substring check | Either exact _or_ simple substring check |
+| Diagnostic logging | Writes to stdout/stderr per normal shell | Supports comprehensive log file | Writes captured output to stdout/stderr on failure |
+| Stop-on-failure | Stops on first failure | Stops on first failure unless told not to | Runs all tests unless told not to |
+| Data dir integration | Cleans up only on full success | Cleans up each scenario unless told to save it | Cleans up each scenario with no option to save failure state |
+```
# Acceptance criteria for Subplot {#acceptance}