summaryrefslogtreecommitdiff
path: root/tutorial.md
blob: 67b73c0b8fb846dd4dd5847ea1daefd6d2506de1 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
---
title: "Fable tutorial"
author:
- The Fable project
- Lars Wirzenius
- Daniel Silverstone
date: work in progress
...

Introduction
=============================================================================

Fable is a tool for **automated acceptance testing**. Acceptance tests
define if a program (or other system) is accepable, from a user's or
client's or employer's point of view. Basically, in a commercial
consulting setting, acceptance tests determine if the client is
willing to pay the bill for the software having been developed.

Acceptance tests differ from other integration tests by looking at the
software _only_ from the user's point of view, not the developer's.
Because of this, Fable produces two things: a standalone document
aimed at the user to describe the accepance tests, and a report from
executing the automated tests.

The document is meant to be a vehicle of communication between the
various stakeholders in a project, when specifying in detail what the
acceptance criteria are. It both explains the criteria, and how
they're verified, and lets the verification to be done automatically.

Fable generates the document, and a test program to execute the tests.
The generated test program produces the test report.

```dot
md [label="document source\n(Markdown)"];
md [shape=box];

bindings [label="bindings file\n(YAML)"];
bindings [shape=box];

impl [label="step implementations\n(Python, Rust, ...)"]
impl [shape=box];

fable [label="Fable"];
fable [shape=ellipse];

pdf [label="PDF document"]
pdf [shape=box];

testprog [label="Test program\n(generated)"]
testprog [shape=box];

report [label="Test report"]
report [shape=box];

md -> fable;
bindings -> fable;
impl -> fable;
fable -> pdf;
fable -> testprog;
testprog -> report;
```

Example
=============================================================================

Here is a quick example of using Fable. Let's imagine you are in need
of a new implementation of the **echo**(1) command line utility, and
you want to commission the implementation. You need to specify
acceptance tests. After negotations with the developers, and their
project manager, and your own sales and test teams, you end up with
the following requirements:

* If echo is run without arguments, it should write a newline
  character, and exit with a zero exit code.
* If echo is run with arguments, it should write each argument,
  separated by a space character, and then a newline character, and
  finally exit with a zero exit code.

[Markdown]: https://en.wikipedia.org/wiki/Markdown

These are specified for Fable in a [Markdown][] document. A minimal
one for echo might look like this:

    No arguments
    =============================================================================

    Run `/bin/echo` without arguments.

    ```fable
    when user runs echo without arguments
    then exit code is 0
    and standard output contains a newline
    and standard error is empty
    ```

    Hello, world
    =============================================================================

    Run `/bin/echo` to produce the output "hello, world".

    ```fable
    when user runs echo with arguments hello, world
    then exit code is 0
    and standard output contains "hello, world"
    and standard error is empty
    ```

Acceptance tests in Fable are expresses as **scenarios**, which
roughly correspond to use cases and acceptance
requirements. A scenario has an overall structure of given/when/then:

* **given** some initial context (setup)
* **when** some action is taken (the thing to test)
* **then** the end result looks in a specific way (checks)

These are embedded in markdown using **fenced code blocks** marked as
containing `fable` text. Fable extracts these and generates a test
program to execute them. The scenario steps are formulated in a way
that all stakeholders in the project understand. This is crucial.

Fable is not magic artificial intelligence. Each scenario step needs
to be implemented by programmers so that computers also understand
them. Each step corresponds to a function, currently in Python, and a
YAML file **binds** the step to the function, using a regular
expression to capture relevant parts of the step. A binding file might
look like this:

    - when: user runs echo without arguments
      function: run_echo_without_args

    - when: user runs echo with arguments (?P<args>.+)
      function: run_echo_with_args

    - then: exit code is (?P<exit_code>\d+)
      function: exit_code_is_zero

    - then: standard output contains a newline
      function: stdout_is_a_newline

    - then: standard output contains "(?P<text>.*)"
      function: stdout_is_text

    - then: standard error is empty
      function: stderr_is_empty

This means that for a step saying 'then standard output contains
"foo"', the Python function `stdout_is_text` is called and given the
string `foo` as an argument.

You, or your test team, in collaboration with the development team,
need to supply the bindings and the Python functions. Fable produces
the code that extracts the interesting parts of scenario steps, and
calls your functions in the right order, plus any other scaffolding
needed.

Installing Fable
=============================================================================

Fable is not currently packaged for Debian. You need to clone the git
repository and need Python 3, and the Markdown parser installed, as
well as document typesetting tools, from Debian 10 (buster) or later:

```sh
sudo apt update
sudo apt install make git locales-all pandoc python3 \
  python3-pandocfilters python3-commonmark-bkrs python3-yaml \
  texlive-latex-base texlive-fonts-recommended \
  graphviz librsvg2-bin
git clone git://git.liw.fi/fable-poc
cd fable-poc
```

Note that `fable-poc` contains sample versions of the files for the
echo acceptance tests (`echo.md`, `echo.yaml`, `echo.py`, and
`echo-prelude.py`) so that you don't need to type them in yourself.


Producing a document
=============================================================================

To produce a PDF document of the echo requirements, for you to review,
the following commands are needed:

```sh
./ftt-docgen --html echo.md
./ftt-docgen --pdf echo.md
```

`echo.md` is the markdown input file. `echo.yaml` is the bindings, and
its name is inferred by `ftt-docgen`. `echo.pdf` and `echo.html` are
the formatted documents. The input files are included in the
`fable-poc` git repository.

Running the tests
=============================================================================

To generate the test program, and running it to produce the test
report, the following commands are needed:

```sh
./ftt-codegen echo.md --run
```

The output of the last command is the test report:

```
OK: No arguments
OK: Hello, world
```

`echo.yaml` and `echo.md` are again the bindings and markdown files.
Also, `echo.py` has the Python functions, needed at runtime, though
not referenced above. The names of `echo.py` and `echo.yaml` are
inferred by `ftt-codegen`.

Exercise
=============================================================================

Your mission, should you choose to accept it, is to write use Fable
to write an acceptance test suite for simple uses of the **cp**(1)
command. You can model it after the **echo**(1) one, and you get to
specify the actual acceptance criteria yourself, but at minimum you
need to check that a regular file can be copied to another name and
that the end result is the same.

Finally
=============================================================================

It would great if you could give us, Lars and Daniel, feedback on
Fable and this tutorial.

* What do you think of Fable as a concept?
* What's good about Fable?
* What's less good about Fable?
* Did you have any problems following the tutorial?
* Did you get through the tutorial successfully?
* Did you do the exercise?