path: root/tutorial.mdwn

Introduction
=============================================================================

Subplot is software to help **capture and communicate acceptance
criteria** for software and systems, and **how they are verified**, in
a way that's understood by **all project stakeholders**.

Another way to express that is that Subplot is for **automated testing
of acceptance criteria**.

A simplified explanation is that Subplot provides a way to codify
an agreement of when a customer is required to pay for a project.

Acceptance testing differ from other integration testing by looking at
the software only from the user's point of view, not the developers'.
Because of this, Subplot produces two things: a standalone document,
aimed at the user, to describe the acceptance criteria, and a test suite
for developers to use to validate that the software meets those
acceptance criteria.

The document is meant to be a vehicle of communication between the
various stakeholders in a project, for specifying in detail what the
acceptance criteria are. The document both explains the criteria, and
how they're verified, and makes it possible to do the verification
automatically.

Subplot generates the document, and a test program to execute the
tests. Running the generated test program produces a test report which
can be used as evidence of passing the tests.

# Architecture

```dot
digraph "archiecture" {
md [label="document source\n(Markdown)"];
md [shape=box];

bindings [label="bindings file\n(YAML)"];
bindings [shape=box];

impl [label="step implementations\n(Python, Rust, ...)"]
impl [shape=box];

subplot [label="Subplot"];
subplot [shape=ellipse];

pdf [label="PDF document"]
pdf [shape=box];

testprog [label="Test program\n(generated)"]
testprog [shape=box];

report [label="Test report"]
report [shape=box];

md -> subplot;
bindings -> subplot;
impl -> subplot;
subplot -> pdf;
subplot -> testprog;
testprog -> report;
}
```

Example
=============================================================================

Here is a quick example of using Subplot. Let's imagine you are in
need of a new implementation of the **echo**(1) command line utility,
and you want to commission the implementation. You need to specify
acceptance tests for it. After negotations with the developers, and
their project manager, and your own sales and test teams, you end up
with the following requirements:

* If echo is run without arguments, it should write a newline
  character, and exit with a zero exit code.
* If echo is run with arguments, it should write each argument,
  separated by a space character, and then a newline character, and
  finally exit with a zero exit code.

[Markdown]: https://en.wikipedia.org/wiki/Markdown

These are specified for Subplot in a [Markdown][] document. A minimal
one for echo might look like this:

~~~~markdown
# No arguments

Run `echo` without arguments.

```scenario
when user runs echo without arguments
then exit code is 0
and standard output contains a newline
and standard error is empty
```

# Hello, world

Run `echo` to produce the output "hello, world".

```scenario
when user runs echo with arguments hello, world
then exit code is 0
and standard output contains "hello, world\n"
and standard error is empty
```
~~~~


Acceptance tests in Subplot are expressed as **scenarios**, which
roughly correspond to use cases and acceptance
requirements. A scenario has an overall structure of given/when/then:

* **given** some initial context (setup)
* **when** some action is taken (the thing to test)
* **then** the end result looks in a specific way (verify what happened)

These are embedded in markdown using **fenced code blocks** marked as
containing `scenario` text. Subplot extracts these and
generates a test program to execute them.

The scenario steps need to be formulated in a way that all
stakeholders in the project understand. This is crucial. The scenario
steps, and the Markdown document in general, is typically written by
the software architect or analyst of the developing organization, in
collaboration with the client, and approved by the client.

Subplot is not magic artificial intelligence. Each scenario step needs
to be implemented by programmers so that computers also understand
them. Each step corresponds to a function, written in a programming
language such as Python, and a YAML file **binds** the step to the
function, using a pattern to capture relevant parts of the step. The
pattern can be simple or a full PCRE regular expression. A bindings
file `echo.yaml` might look like this:

```yaml
- when: user runs echo without arguments
  function: run_echo_without_args

- when: user runs echo with arguments {args}
  function: run_echo_with_args

- then: exit code is {exit_code}
  function: exit_code_is_zero

- then: standard output contains a newline
  function: stdout_is_a_newline

- then: standard output contains "(?P<text>.*)"
  regex: true
  function: stdout_is_text

- then: standard error is empty
  function: stderr_is_empty
```

This means that for a step saying 'then standard output contains
"foo"', the Python function `stdout_is_text` is called and given the
string `foo` as an argument called `text`.

To join the Markdown file, the bindings file, and the step
implementations together, you also need a YAML metadata file:

~~~yaml
title: echo acceptance criteria
bindings:
- echo.yaml
impl:
  python:
  - echo.py
~~~

You would call this file something like `echo.subplot` to distinguish
it from the bindings file, `echo.yaml`.

The developers on the project, in collaboration with the testing team,
need to supply both the bindings and the implementation functions in
whatever language the test suite is written in (Python in this
example). Subplot produces the code that extracts the interesting
parts of scenario steps, and calls your functions in the right order,
plus any other scaffolding needed.


Installing Subplot
=============================================================================

To install Subplot, see the [download
page](https://subplot.tech/download/) on the Subplot website. We don't
replicate it here to avoid information getting stale.

Producing a document
=============================================================================

To produce a PDF document of the echo requirements, for you to review,
the following commands are needed:

```sh
subplot docgen echo.subplot -o echo.html
subplot docgen echo.subplot -o echo.pdf
```


Running the tests
=============================================================================

To generate the test program, and running it to produce the test
report, the following commands are needed:

```sh
subplot codegen echo.subplot -o test.py --run
```

The output of the command is the test report:

```{.numberLines}
srcdir /home/liw/pers/subplot/subplot
datadir /tmp/tmp.MeW2f5NS7n
scenario: No arguments
  step: when user runs echo without arguments
  step: then exit code is 0
  step: then standard output contains a newline
  step: then standard error is empty
  cleanup: 
scenario: Hello, world
  step: when user runs echo with arguments hello, world
  step: then exit code is 0
  step: then standard output contains "hello, world"
  step: then standard error is empty
  cleanup: 
OK, all scenarios finished successfully
```

Finally
=============================================================================

It would great if you could give us, Lars and Daniel, feedback on
Subplot and this tutorial.

Questions for before you go through the tutorial (if we are
interviewing you):

* What kinds of testing do you already do?
* Who are your usual stakeholders?
* What sorts of things would you like to test better?

Questions for after you go through the tutorial:

* What do you think of Subplot as a concept?
* What's good about Subplot?
* What's less good about Subplot?
* Did you have any problems following the tutorial?
* Do you see yourself using Subplot in the future?
* Do you see Subplot as filling a gap?


---
title: Subplot tutorial
...