summaryrefslogtreecommitdiff
path: root/README.yarn
diff options
context:
space:
mode:
authorLars Wirzenius <liw@liw.fi>2013-07-14 17:17:53 +0100
committerLars Wirzenius <liw@liw.fi>2013-07-14 17:17:53 +0100
commit6a05675ffdf619fc043c517cfcd782114b606174 (patch)
treed4512891e3746b14c73f6e1adc6b03c6c97872d5 /README.yarn
parent796c4e4182b810225816a2f090d00c21d708d3a3 (diff)
downloadcmdtest-6a05675ffdf619fc043c517cfcd782114b606174.tar.gz
Update README.yarn so it matches reality better
I now know of nothing that's inaccurate, but we'll see.
Diffstat (limited to 'README.yarn')
-rw-r--r--README.yarn95
1 files changed, 46 insertions, 49 deletions
diff --git a/README.yarn b/README.yarn
index ae857e0..49fcd62 100644
--- a/README.yarn
+++ b/README.yarn
@@ -1,5 +1,5 @@
-README for scenario testing
-===========================
+README for yarn, a scenario testing tool
+========================================
Introduction
------------
@@ -9,6 +9,7 @@ user uses your software and what should happen, and express, using
very lightweight syntax, the scenario in such a way that it can be tested
automatically. The scenario has a simple, but strict structure:
+ SCENARIO name of scenario
GIVEN some setup for the test
WHEN thing that is to be tested happens
THEN the post-conditions must be true
@@ -16,36 +17,37 @@ automatically. The scenario has a simple, but strict structure:
As an example, consider a very short test scenario for verifying that
a backup program works, at least for one simple case.
+ SCENARIO basic backup and restore
GIVEN some live data in a directory
AND an empty backup repository
WHEN a backup is made
- THEN the data case be restored
+ THEN the data can be restored
(Note the addition of AND: you can have multiple GIVEN, WHEN, and
THEN statements. The AND keyword makes the text be more readable.)
-Scenarios are meant to be written in somewhat human readable language.
+Scenarios are meant to be written in mostly human readable language.
However, they are not free form text. In addition to the GIVEN/WHEN/THEN
structure, the text for each of the steps needs a computer-executable
implementation. This is done by using IMPLEMENTS. The backup scenario
from above might be implemented as follows:
IMPLEMENTS GIVEN some live data in a directory
- rm -rf "$TESTDIR/data"
- mkdir "$TESTDIR/data"
- echo foo > "$TESTDIR/data/foo"
+ rm -rf "$DATADIR/data"
+ mkdir "$DATADIR/data"
+ echo foo > "$DATADIR/data/foo"
IMPLEMENTS GIVEN an empty backup repository
- rm -rf "$TESTDIR/repo"
- mkdir "$TESTDIR/repo"
+ rm -rf "$DATADIR/repo"
+ mkdir "$DATADIR/repo"
IMPLEMENTS WHEN a backup is made
- backup-program -r "$TESTDIR/repo" "$TESTDIR/data"
+ backup-program -r "$DATADIR/repo" "$DATADIR/data"
IMPLEMENTS THEN the data can be restored
- mkdir "$TESTDIR/restored"
- restore-program -r "$TESTDIR/repo" "$TESTDIR/restored"
- diff -rq "$TESTDIR/data" "$TESTDIR/restored"
+ mkdir "$DATADIR/restored"
+ restore-program -r "$DATADIR/repo" "$DATADIR/restored"
+ diff -rq "$DATADIR/data" "$DATADIR/restored"
Each "IMPLEMENT GIVEN" (or WHEN, THEN) is followed by a regular
expression on the same line, and then a shell script that gets executed
@@ -53,10 +55,13 @@ to implement any step that matches the regular expression. The
implementation can extract data from the match as well: for example,
the regular expression might allow a file size to be specified.
-The above example is a bit silly, of course: why go to the effort
-to obfuscate the various steps? The answer is that the various
-steps, implemented using IMPLEMENTS, can be combined in many
-ways, to test different aspects of the program being tested.
+The above example seems a bit silly, of course: why go to the effort
+to obfuscate the various steps? The answer is that the various steps,
+implemented using IMPLEMENTS, can be combined in many ways, to test
+different aspects of the program being tested. In effect, the IMPLEMENTS
+sections provide a vocabulary which the scenario writer can use to
+express a variety of usefully different scenarios, which together
+test all the aspects of the software that need to be tested.
Moreover, by making the step descriptions be human language
text, matched by regular expressions, most of the test can
@@ -75,6 +80,7 @@ quoted code blocks being interpreted specially. Each block
must follow the syntax defined here.
* Every step in a scenario is one line, and starts with a keyword.
+
* Each implementation (IMPLEMENTS) starts as a new block, and
continues until there is a block that starts with another
keyword.
@@ -93,7 +99,7 @@ The following keywords are defined.
* **ASSUMING** defines a condition for the scenario. The rest of the
line is "matched text", which gets implemented by an
IMPLEMENTS section. If the code executed by the implementation
- fails, the scenario is skipped.
+ fails, the scenario is skipped. (NOT yet implemented.)
* **GIVEN** prepares the world for the test to run. If
the implementation fails, the scenario fails.
@@ -104,23 +110,23 @@ The following keywords are defined.
* **THEN** verifies that the changes made by the GIVEN steps
did the right thing. If the code fails, the scenario fails.
-* **FINALLY** specifies how to clean up after a scenario. If the
- code fails, the scenario fails. All FINALLY blocks get run either when
- encountered in the scenario flow, or at the end of the scenario, regardless
- of whether the scenario is failing or not.
+* **FINALLY** specifies how to clean up after a scenario. If the code
+ fails, the scenario fails. All FINALLY blocks get run either when
+ encountered in the scenario flow, or at the end of the scenario,
+ regardless of whether the scenario is failing or not.
* **AND** acts as ASSUMING, GIVEN, WHEN, THEN, or FINALLY: whichever
was used last. It must not be used unless the previous step was
one of those, or another AND.
-* **IMPLEMENTS** is followed by one of ASSUMING, GIVEN, WHEN, or
- THEN, and a PCRE regular expression, and then shell commands until
- the end of the block quoted code block. Markdown is unclear whether
- an empty line (no characters, not even whitespace) between two
- block quoted code blocks starts a new one or not, so we resolve the
- ambiguity by specifiying that a code block directly following a code
- block is a continuation unless it starts with one of the scenario
- testing keywords.
+* **IMPLEMENTS** is followed by one of ASSUMING, GIVEN, WHEN, or THEN,
+ and a PCRE regular expression, all on one line, and then further
+ lines of shell commands until the end of the block quoted code
+ block. Markdown is unclear whether an empty line (no characters,
+ not even whitespace) between two block quoted code blocks starts a
+ new one or not, so we resolve the ambiguity by specifiying that a
+ code block directly following a code block is a continuation unless
+ it starts with one of the scenario testing keywords.
The shell commands get parenthesised parts of the match of the
regular expression as environment variables (`$MATCH_1` etc). For
@@ -128,14 +134,14 @@ The following keywords are defined.
set to the number matched by `\d+`.
The test runner creates a temporary directory, whose name is
- given to the shell code in the `TESTDIR` environment variable.
+ given to the shell code in the `DATADIR` environment variable.
The shell commands get invoked with `/bin/sh -eu`, and need to
- be written accordingly. Be careful about commands that return
- a non-zero exit code. There will be a library of shell functions
- supplied which allow handling the testing of non-zero exit codes
- cleanly. In addition functions for handling stdout and stderr will
- be provided.
+ be written accordingly. Be careful about commands that return a
+ non-zero exit code. There will eventually be a library of shell
+ functions supplied which allow handling the testing of non-zero
+ exit codes cleanly. In addition functions for handling stdout and
+ stderr will be provided.
The code block of an IMPLEMENTS block fails if the shell
invocation exits with a non-zero exit code. Output to stderr is
@@ -145,13 +151,13 @@ The following keywords are defined.
Semantics:
* The name of each scenario (given with SCENARIO) must be unique.
-* All names will be normalised before use (whitespace collapse, leading
- and trailing whitespace
+* All names of scenarios and steps will be normalised before use
+ (whitespace collapse, leading and trailing whitespace
* Every ASSUMING, GIVEN, WHEN, THEN, FINALLY must be matched by
exactly one IMPLEMENTS. The test runner checks this before running
any code.
* Every IMPLEMENTS may match any number of ASSUMING, GIVEN, WHEN,
- THEN, or FINALLY. The test runner can warn if an IMPLEMENTS is unused.
+ THEN, or FINALLY. The test runner may warn if an IMPLEMENTS is unused.
* If ASSUMING fails, that scenario is skipped, and any FINALLY steps
are not run.
@@ -174,13 +180,4 @@ TODO
to localhost when flinging, for example.
**DJAS**: We think this might be 'REQUIRING' and it still does
not run the FINALLY group.
-* Consider the ordering some more. IMPLEMENTS can come anywhere
- but otherwise scenarios are defined as:
- * SCENARIO
- * 0+: ASSUMING
- * 1+:
- * 1+: GIVEN
- * 1+:
- * 1+: WHEN
- * 1+: THEN
- * 0+: FINALLY
+