summaryrefslogtreecommitdiff
path: root/fable-arch.md
blob: 2f29edfac68c02bdd2bfabd9bca8dc67744147db (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
---
documentclass: report
title: Acceptance testing using Fable
author:
- The Fable project
- Lars Wirzenius
- Daniel Silverstone
date: WORK IN PROGRESS
keywords:
- automated acceptance testing
- scenario testing
- gherkin
- cucumber
abstract: |
  Fable is a tool that supports acceptance testing in two ways: it is a
  way to write and run automated acceptance tests, and also
  presents the acceptance test suite to non-expert readers as a
  human-readable text document.

  This document explains Fable, its architecture, its input
  language, and specifies the acceptance criteria for Fable.
...

# Beware

This document describes a future that will be, not current status quo.
Fable is its initial, active development period, and we don't want to
update this document every time we make a change. Thus, be aware that
parts of this document is speculative and may describe a version of
Fable that does not yet exist.

# Introduction

Fable is a tool for acceptance testing of software. It helps describe
acceptance criteria for all stakeholders in a software project, and
also for computers. Such criteria may come from paying clients, users,
the developers, their employers, or elsewhere.

Fable is specifically meant for describing automated acceptance tests.
It takes a two-pronged approach, where it lets developers write
automated tests for all the acceptance criteria they have, and runs
the tests. On the other hand, Fable also produces a standalone
document, which describes the automated tests all stakeholders,
including non-technical ones, such as project managers and clients.

More concretely, Fable helps developers specify the automated
acceptance tests so that the tests can be executed, but also
understood without requiring programming knowledge to understand.

Fable is meant to be used to produce a document to facilitate
communication between various shareholders of the software being
developed.

Fable's overall working principle is that the tests are implemented and
documented in a number of source files, which by two Fable tools. One
tool produces a PDF or HTML document, for humans to read. The other
produces a test program, which executes the acceptaance tests.

## Motivation for Fable

Keeping track of requirements and acceptance criteria is necessary for
all but the simplest of software projects. Having all stakeholders in
a projects agree to them is crucial, as is that all agree how it is
verified that the software meets the acceptance criteria. Fable aims
to provide a way for documenting the shared understanding of what the
acceptance criteria are and how they can be checked automatically.

Stakeholders in a project may include:

* those who pay for the work to be done; this may be the employer of
  the developers for in-house projects ("customer")
* those who use the resulting systems, whether they pay for it or not
  ("user")
* those who install and configure the systems and keep them functional
  ("sysadmin")
* those who support the users ("support")
* those who develop the system in the first place ("developer")

All stakeholders need to understand the acceptance criteria, and how
the system is evaluated against them. In the simplest case, the
customer and the developer need to both understand and agree so that
the developer knows when the job is done, and the customer knows when
they need to pay their bill.

However, even when the various stakeholder roles all fall upon the
same person, or only on people who act as developers, the Fable
approach can be useful. A developer would understand acceptance
criteria expressed only in code, but doing so may take time and energy
that are not always available. The Fable approach aims to encourage
hiding unnecessary detail and documenting things in a way that is easy
to understand with little effort.

Unfortunately, this does mean that for a Fable output document to
be good and helpful, writing it will require effort and skill. No tool
can replace that.

# Requirements

This chapter lists requirements for Fable. These requirements are not
meant to be testable as such. For more specific, testable acceptance
criteria, see the later [chapter with acceptance tests for
Fable](#acceptance).

Each requirement here is given a unique mnemnoic id for easier
reference in discussions.

**UnderstadableTests**

: Acceptance tests should be possible to express in a way that's
  easily understood by non-programmers.

**EasyToWriteDocs**

: The markup language for writing documentation should be easy to
  write.

**AidsComprehension**

: The formatted human-readable documentation should use good layout
  and typography to enhance comprension.

**CodeSeparately**

: The code to implement the acceptance tests should not be embedded in
  the documentation source, but be in separate files. This makes it
  easier to edit without specialised tooling.

**AnyProgammingLanguage**

: The developers implementing the acceptance tests should be free to
  use a language they're familiar and comfortable with. Fable should
  not require them to use a specific language.

**FastTestExecution**

: Executing the acceptance tests should be fast.

**NoDeployment**

: The acceptance test tooling should assume the system under test is
  already deployed and available. Deploying is too big of a problem
  space to bring into the scope of acceptance testing, and there are
  already good tools for deployment.

**MachineParseableResults**

: The tests should produce a machine parseable result that can be
  archived, post-processed, and analyzed in ways that are of interest
  to the project using Fable. For example, to see trends in how long
  tests take, how often tests fail, to find regressions, and to find
  tests that don't provide value.

# Fable architecture

```dot
md [label="document source\n(Markdown)"];
md [shape=box];

bindings [label="bindings file\n(YAML)"];
bindings [shape=box];

impl [label="step implementations\n(Python, Rust, ...)"]
impl [shape=box];

fable [label="Fable"];
fable [shape=ellipse];

pdf [label="PDF document"]
pdf [shape=box];

testprog [label="Test program\n(generated)"]
testprog [shape=box];

report [label="Test report"]
report [shape=box];

md -> fable;
bindings -> fable;
impl -> fable;
fable -> pdf;
fable -> testprog;
testprog -> report;
```

Fable reads input files, and produces two outputs.
On the one hand, it outputs a program that executes the tests
specified in the input files. The person running Fable then runs the
test program to get a test report, with results of each of the test
scenarios. On the other hand it outputs a human-readable document (as
PDF or HTML), for communicating what is being tested.

Fable is able to produce the test program in various languages, using
a templating system to make it simple to add support for more
languages. Fable comes with support for Python and Rust. It's the
user's choice which language they're most comfortable with.

Acceptance tests are expressed to Fable in the form of test scenarios,
in which a sequence of actions are taken, and then the results are
checked. If the checks fail, the scenario fails.

Fable runs the scenarios concurrently (but see the USING keyword),
within the constrains of hardware resources. If Fable determines it
doesn't have all the resources to run all scenarios at once, it will
run fewer, but randomly chosen scenarios concurrently, to more likely
to detect unintentional dependencies between scenarios.

Note that Fable or the scenarios it runs aren't meant deploy the
software, or start services to run in the background.

# The Fable input language

Fable input consists of three types of files:

* one or more markdown files which document the acceptance tests
* a binding file (in YAML) which binds scenario steps to their
  implementations
* scenario step implementations, which are implemented in some
  programming language (e.g., Python or Rust); Fable will combine this
  code with some scaffolding provided by Fable itself

The input files for a simple acceptance test suite for Fable would be
divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming
step implementation in Python).

## Markdown files

[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks
[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language
[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software)

The Fable input language is markdown files using [fenced code blocks][]
with backticks. The code blocks must indicate that they contain Fable
language:

    ```fable
    given a service
    and I am Tomjon
    when I access the service
    then it's OK
    ```

Code blocks for the PlantUML or GraphViz dot markup languages are
also supported. These can be used for generating embedded images in
the documented produced by Fable.

Any unfenced code blocks (indented code blocks) are ignored by Fable,
as are fenced code blocks using unknown content types.

The Fable code generator understands the full Markdown language, by
ignoring everything except headings and its own code blocks. The Fable
document generator uses [Pandoc][] to produce standalone files, and
anything that Pandoc supports is OK to use.

[Pandoc]: https://pandoc.org/

Fable treats multiple Markdown files as one, as if they had been
concatenated with the **cat**(1) utility. Within the logical file,
normal Markdown and Pandoc markup can be used to structure the
document in any way that aids human understanding of the acceptance
test suite, which the caveat that chapter or section headings are used
by Fable to group code blocks into scenarios.

All code blocks for the same scenario must be grouped under a single
heading. Sub-headings are permitted within a scenario, but the next
heading at the same or a higher level will end the scenario. This
allows for scenarios to begin at any level, but not to leak into a
wider scope within the acceptance document. For example, a scenario
which starts after a level 2 heading may have subdivisions marked with
level 3 or below headings, but will end at the next level 2 or level 1
heading.

Within the Fable code blocks, Fable understands a special language,
derived from [Gherkin][], as defined by the [Cucumber][] testing tool.
The language understood by Fable has the following general structure:

* each logical line starts with a keyword at the beginning of the line
* logical lines may be broken into physical lines, by starting the
  continuation lines with one or more space or TAB characters; the
  physical line break and white space characters are preserved
* logical lines define steps in a test scenario
* the meaning and implementation of the steps are defined by other
  Fable input files
* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, and AND, with
  meanings defined below; keywords can be written in upper or lower
  case, or mixes, Fable doesn't care

The keywords have the following meanings:

**assuming**

:   A condition for the scenario. If an ASSUMING step fails, the
    scenario is skipped.

    This is used for skipping test scenarios that require specific
    software to be installed in the test environment, or access to
    external services, but which can't be required for all runs of the
    acceptance tests.

**using**

:   Indicate that the scenario uses a resource such as a
    database, that's constrained and can't be used by all scenarios if
    they run concurrently. When scenarios declare the resource, Fable can
    limit which scenarios run concurrently.

    For example, if several scenarios require uncontested use of the
    GPU, of which there is typically only one per machine, they can all
    declare "using the graphical processing unit", and Fable will run
    them one at a time.

    (This is an intentionally simplistic way of controlling concurrency.
    The goal is to be simple and correct rather then achievee maximal
    concurrency.)

    The actual management of resources belongs to the generated test
    program at runtime, not the Fable compiler.

**given**

:   Set up the test environment for the action (WHEN). This
    might create files, start a background process, or something like
    that. This also sets up the reversal of the setup, so that any
    background processes are stopped automatically after the scenario
    has ended. The setup and cleanup must succeed, or the scenario will
    fail.

    The cleanups are executed in the reverse order of the GIVENs, and
    they're done always, whether the scenario succeeds or not.

**when**

:   Perform the action that is being tested. This must succeed. This
    might, for example, execute a command line program, and capture
    its output and exit code.

**then**

:   Test the results of the action. This would examine the output and
    exit code of the program run in a WHEN step, or examine current
    content of the database, or whatever is needed.

**and**

:   This keyword exists to make scenarios "read" better in English.
    The keyword indicates that this step should use the same keyword
    as the previous step, whatever that keyword is. For example, a
    step "THEN output is empty" might be followed by "AND the exit
    code is 0" rather than "THEN the exit code is 0".

The order of steps is somewhat constrainted: first any ASSUMING steps,
then any USING steps, at least one WHEN must come before a THEN.

## Bindings

FIXME: The binding specification needs thought. This is just a sketch.

Binding files match scenario steps to functions that implement them,
using regular expressions. The bindings may also extract parts of the
steps, and pass them onto the functions as parameters.

Binding files are YAML files, with lists of bindings, each binding
being a dict. For example:

```yaml
- define:
    name: string
    exit_code: int

- pattern: given a service
  function: start_service
  cleanup: stop_service

- pattern: given I am (?P<name\S+)
  function: set_name
  produces: [name]

- pattern: when I access the service
  function: access_service
  requires: [name]
  produces: [exit_code]

- pattern: then it's OK
  function: check_access_was_ok
  requires: [exit_code]
```

In the example above, the "I am" step extracts the name of the user
from the step. It's type is declared, and the value is saved for use
by a later step.

The "I access" step expects the name to have been set by a previous
step. Fable will check that the name is set, and give an error if it
isn't, before any scenario runs. If name is set, it is given to the
function to be called as a function argument.

The "I access" step further sets the variable "exit_code", and the
"it's OK" step expects it to be set.

## Step implementations

Continuing the example from the previous section, the following Python
code might implement the functions:

```python
def start_service():
    ...

def set_name(**matches):
    ...

def access_service(name):
    ...
    return {
        'exit_code': 0,
    }

def check_access_was_ok(exit_code):
    assert exit_code == 0
```

With these bindings, Fable produces a Python program, which calls
these functions in order, and passes values between them via function
arguments and return values. The generated program will handle running
scenarios concurrently, and taking care of USING constraints, and
other resource constraints.

# Acceptance tests for Fable {#acceptance}

## The simplest possible scenario

This tests that Fable can run a simplest possible scenario
successfully. The test is based on generating the test program from an
input file, running the test program, and examining the log file.

```fable
given files simple.md, simple.yaml, and simple.py
when I run ftt-codegen --run simple.md
then log file says scenario "Simple" was run
and log file says step "given precondition foo" was run
and log file says step "when I do bar" was run
and log file says step "then bar was done" was run
and program finished successfully
```

### simple.md&mdash;markdown file

~~~~{#simple.md .markdown}
# Simple
This is the simplest possible test scenario

```fable
given precondition foo
when I do bar
then bar was done
```
~~~~

### simple.yaml&mdash;bindings file

```{#simple.yaml .yaml}
- given: precondition foo
  function: precond
- when: I do bar
  function: do
- then: bar was done
  function: was_done
```

### simple.py&mdash;Python file with functions

```{#simple.py .python}
state = {'done': False}
def precond(ctx):
    pass
def do(ctx):
    state['done'] = True
def was_done(ctx):
    assert state['done']
```

## Two scenarios in the same markdown file

This tests that Fable can run two successful scenarios in the same
Markdown file successfully.

```fable
given files two.md, two.yaml, and two.py
when I run ftt-codegen --run two.md
then log file says scenario "First" was run
and log file says step "given precondition foo" was run
and log file says step "when I do bar" was run
and log file says step "then bar was done" was run
then log file says scenario "Second" was run
and log file says step "given precondition foo" was run
and log file says step "when I do bar" was run
and log file says step "then bar was done" was run
and program finished successfully
```

### two.md&mdash;markdown file

~~~~{#two.md .markdown}
# First
This is the simplest possible test scenario

```fable
given precondition foo
when I do bar
then bar was done
```

# Second
This is another scenario.

```fable
given precondition foo
when I do bar
then bar was done
```
~~~~

### two.yaml&mdash;bindings file

```{#two.yaml .yaml}
- given: precondition foo
  function: precond
- when: I do bar
  function: do
- then: bar was done
  function: was_done
```

### two.md&mdash;Python file with functions

```{#two.py .python}
state = {'done': False}
def precond(ctx):
    pass
def do(ctx):
    state['done'] = True
def was_done(ctx):
    assert state['done']
```

## A scenario step fails

This tests that Fable can run handle a scenario step failing.

```fable
given files fail.md, fail.yaml, and fail.py
when I run ftt-codegen --run fail.md
then log file says scenario "Fail" was run
and log file says step "given precondition foo" was run
and log file says step "then bar was done" failed
and program finished with an error
```

### fail.md&mdash;markdown file

~~~~{#fail.md .markdown}
# Fail
This is a scenario that fails.

```fable
given precondition foo
then bar was done
```
~~~~

### fail.md&mdash;bindings file

```{#fail.yaml .yaml}
- given: precondition foo
  function: precond
- then: bar was done
  function: was_done
```

### fail.md&mdash;Python file with functions

```{#fail.py .python}
state = {'done': False}
def precond(ctx):
    pass
def do(ctx):
    state['done'] = True
def was_done(ctx):
    assert state['done']
```