1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
|
README for story testing
========================
Document History
----------------
**TODO: Remove this when finalised**
v1. liw wrote it
v2. kinnison updated based on conversation with liw
Introduction
------------
`story-test` is a story testing tool: you write a story describing
how a user uses your software and what should happen, and express,
using very lightweight syntax, the story in such a way that it can
be tested automatically. The story has a simple, but strict structure:
GIVEN some setup for the test
WHEN thing that is to be tested happens
THEN the post-conditions must be true
As an example, consider a very short test story for verifying that
a backup program works, at least for one simple case.
GIVEN some live data in a directory
AND an empty backup repository
WHEN a backup is made
THEN the data case be restored
(Note the addition of AND: you can have multiple GIVEN, WHEN, and
THEN statements.)
Stories are meant to be written in somewhat human readable language.
However, they are not free form text. In addition to the GIVEN/WHEN/THEN
structure, the text for each of the steps needs a computer-executable
implementation. This is done by using IMPLEMENTS. The backup story
from above might be implemented as follows:
IMPLEMENTS GIVEN some live data in a directory
rm -rf "$TESTDIR/data"
mkdir "$TESTDIR/data"
echo foo > "$TESTDIR/data/foo"
IMPLEMENTS GIVEN an empty backup repository
rm -rf "$TESTDIR/repo"
mkdir "$TESTDIR/repo"
IMPLEMENTS WHEN a backup is made
backup-program -r "$TESTDIR/repo" "$TESTDIR/data"
IMPLEMENTS THEN the data can be restored
mkdir "$TESTDIR/restored"
restore-program -r "$TESTDIR/repo" "$TESTDIR/restored"
diff -rq "$TESTDIR/data" "$TESTDIR/restored"
Each "IMPLEMENT GIVEN" (or WHEN, THEN) is followed by a regular
expression on the same line, and then a shell script that gets
executed to implement any step that matches the regular expression.
The implementation can extract data from the match as well: for
example, the regular expression might allow an amount to be specified.
The above example is a bit silly, of course: why go to the effort
to obfuscate the various steps? The answer is that the various
steps, implemented using IMPLEMENTS, can be combined in many
ways, to test different aspects of the program being tested.
Moreover, by making the step descriptions be human language
text, matched by regular expressions, most of the test can
hopefully be written, and understood, by non-programmers. Someone
who understands what a program should do, could write tests
to verify its behaviour. The implementations of the various
steps need to be implemented by a programmer, but given a
well-designed set of steps, with enough flexibility in their
implementation, that quite a good test suite can be written.
Test language specification
---------------------------
A test document is written in [Markdown][markdown], with block
quoted code blocks being interpreted specially. Each block
must follow the syntax defined here.
* Every step in a story is one line, and starts with a keyword.
* Each implementation (IMPLEMENTS) starts as a new block, and
continues until there is a block that starts with another
keyword.
The following keywords are defined.
* **STORY** starts a new story. The rest of the line is the name of
the story. The name is used for documentation and reporting
purposes only and has no semantic meaning. STORY MUST be the
first keyword in a story, with the exception of IMPLEMENTING.
The set of documents passed in a test run may define any number of
stories between them, but there must be at least one or it is a
test failure. The IMPLEMENTING sections are shared between the
documents and stories.
* **ASSUMING** defines a condition for the story. The rest of the
line is "matched text", which gets implemented by an
IMPLEMENTING section. If the code executed by the implementation
fails, the story is skipped.
* **GIVEN** prepares the world for the test to run. If
the implementation fails, the story fails.
* **WHEN** makes the change to the world that is to be tested.
If the code fails, the story fails.
* **THEN** verifies that the changes made by the GIVEN steps
did the right thing. If the code fails, the story fails.
* **FINALLY** specifies how to clean up after a story. If the
code fails, the story fails. All finally blocks get run either when
encountered in the story flow, or at the end of the story, regardless
of whether the story is failing or not.
* **AND** acts as ASSUMING, GIVEN, WHEN, THEN, or FINALLY: whichever
was used last. It must not be used unless the previous step was
one of those, or another AND.
* **IMPLEMENTING** is followed by one of ASSUMING, GIVEN, WHEN, or
THEN, and a PCRE regular expression, and then shell commands until
the end of the block quoted code block. Markdown is unclear whether
an empty line (no characters, not even whitespace) between two
block quoted code blocks starts a new one or not, so we resolve the
ambiguity by specifiying that a code block directly following a code
block is a continuation unless it starts with one of the story
testing keywords.
The shell commands get parenthesised parts of the match of the
regular expression as positional arguments (`$1` etc). For
example, if the regexp is "a (\d+) byte file", then `$1` gets
set to the number matchec by `\d+`.
The test runner creates a temporary directory, whose name is
given to the shell code in the `TESTDIR` environment variable.
The shell commands get invoked with `/bin/sh -eu`, and need to
be written accordingly. Be careful about commands that return
a non-zero exit code. There will be a library of shell functions
supplied which allow handling the testing of non-zero exit codes
cleanly. In addition functions for handling stdout and stderr will
be provided.
The code block of an IMPLEMENTING block fails if the shell
invocation exits with a non-zero exit code. Output to stderr is
not an indication of failure. Any output to stdout or stderr may
or may not be shown to the user.
Semantics:
* The name of each story (given with STORY) must be unique.
* All names will be normalised before use (whitespace collapse, leading
and trailing whitespace
* Every ASSUMING, GIVEN, WHEN, THEN, FINALLY must be matched by
exactly one IMPLEMENTING. The test runner checks this before running
any code.
* Every IMPLEMENTING may match any number of ASSUMING, GIVEN, WHEN,
THEN, or FINALLY. The test runner can warn if an IMPLEMENTING is unused.
* If ASSUMING fails, that story is skipped, and any FINALLY steps
are not run.
See also
--------
Wikipedia has an article on [Behaviour Driven Development][BDD],
which can provide background and further explanation to what this
tools tries to do.
[BDD]: https://en.wikipedia.org/wiki/Behavior-driven_development
[Markdown]: ...
TODO
----
* Add DEFINING, PRODUCING, if they turn out to be useful.
* Need something like ASSUMING, except fail the story if the
pre-condition is not true. Useful for testing that you can ssh
to localhost when flinging, for example.
**DJAS**: We think this might be 'REQUIRING' and it still does
not run the FINALLY group.
* Consider the ordering some more. IMPLEMENTING can come anywhere
but otherwise stories are defined as:
* STORY
* 0+: ASSUMING
* 1+:
* 1+: GIVEN
* 1+:
* 1+: WHEN
* 1+: THEN
* 0+: FINALLY
|
