diff options
author | Daniel Silverstone <dsilvers@digital-scurf.org> | 2022-02-19 12:04:12 +0000 |
---|---|---|
committer | Daniel Silverstone <dsilvers@digital-scurf.org> | 2022-02-19 12:04:57 +0000 |
commit | 48d56b2cb4123db919c1a938766c19c94f7344da (patch) | |
tree | 23f67e7c316937776b587606eb992a6fa0dd905f /subplot.md | |
parent | 14f2349c1338b6bf6d1867d5d173c37b7782364d (diff) | |
download | subplot-48d56b2cb4123db919c1a938766c19c94f7344da.tar.gz |
(doc): Add scenario runner comparison 'table'
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Diffstat (limited to 'subplot.md')
-rw-r--r-- | subplot.md | 26 |
1 files changed, 26 insertions, 0 deletions
@@ -959,6 +959,32 @@ def file_ends_in_zero_newlines(ctx, filename=None): assert_ne(content[-1], "\n") ~~~ +## Comparing the scenario runners + +Currently Subplot ships with three scenario runner templates. The +Bash, Python, and Rust templates. The first two are fully self-contained +and have a set of features dictated by the Subplot version. The +latter is tied to how Cargo runs tests. Given that, this comparison +is only considered correct against the version of Rust at the time of +publishing a Subplot release. Newer versions of Rust may introduce +additional functionality which we do not list here. Finally, we do +not list features here which are considered fundamental, such as +"runs all the scenarios" or "supports embedded files" since no template +would be considered for release if it did not do these things. These +are the differentiation points. + +``` +| Feature | Bash | Python | Rust | +| ----------------------------- | ---------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------ | +| Isolation model | Subprocess | Subprocess | Threads | +| Parallelism | None | None | Threading | +| Passing environment variables | CLI | CLI | Prefixed env vars | +| Execution order | Fixed order | Randomised | Fixed order plus threading peturbation | +| Run specific scenarios | Simple substring check | Simple substring check | Either exact _or_ simple substring check | +| Diagnostic logging | Writes to stdout/stderr per normal shell | Supports comprehensive log file | Writes captured output to stdout/stderr on failure | +| Stop-on-failure | Stops on first failure | Stops on first failure unless told not to | Runs all tests unless told not to | +| Data dir integration | Cleans up only on full success | Cleans up each scenario unless told to save it | Cleans up each scenario with no option to save failure state | +``` # Acceptance criteria for Subplot {#acceptance} |