---
title: Acceptance testing using Fable
author:
- The Fable project ()
- Lars Wirzenius
- Daniel Silverstone
date: WORK IN PROGRESS
keywords:
- automated acceptance testing
- scenario testing
- gherkin
- cucumber
abstract: |
Fable is a tool that supports acceptance testing in two ways: it is a
way to write and run automated acceptance tests, and also
presents the acceptance test suite to non-expert readers as a
human-readable text document.
This document explains Fable, its architecture, and its input
language.
...
# Introduction
Fable is a tool for acceptance testing of software. This means it helps
software developers to make sure their software fulfils the
acceptance criteria for the software. Such criteria may come from
users, the developers, their employers, or elsewhere.
Fable is specifically meant for automated acceptance testing. It takes a
two-pronged approach, where it lets developers write automated
tests for all the acceptance criteria they have, and runs the tests.
On the other hand, Fable also produces a PDF file, which documents the
automated tests for non-technical stakeholders, such as project
managers and clients.
More concretely, Fable helps developers implement and document their
automated acceptance tests in a way that, at the same time, helps the
developers automatically test their software, and write documentation
for the tests in way that doesn't require programming knowledge to
understand.
Fable is meant to be a tool for developers, who use it to produce a
document, which is meant to facilitate communication between various
shareholders of the software being developed.
Fable's overall working principle is that the tests are implemented and
documented in a number of source files, which Fable reads to execute
tests, and to produce a PDF document for non-developer consumption.
# Fable architecture
```dot
md [label="document source\n(Markdown)"];
md [shape=box];
bindings [label="bindings file\n(YAML)"];
bindings [shape=box];
impl [label="step implementations\n(Python, Rust, ...)"]
impl [shape=box];
fable [label="Fable"];
fable [shape=ellipse];
pdf [label="PDF document"]
pdf [shape=box];
testprog [label="Test program\n(generated)"]
testprog [shape=box];
report [label="Test report"]
report [shape=box];
md -> fable;
bindings -> fable;
impl -> fable;
fable -> pdf;
fable -> testprog;
testprog -> report;
```
Fable's architecture it to read input files, and produce two outputs.
On the one hand, it outputs a program that executes the tests
specified in the input files. The person running Fable then runs the
test program to get a test report, with results of each of the test
scenarios. On the other hand it outputs a human-readable document (as
PDF), for communicating what is being tested.
Fable is able to produce the test program in various languages, using
a templating system to make it simple to add support for more
languages. Fable comes with support for Python and Rust. It's the
user's choice which language they're most comfortable with.
Acceptance tests are expressed to Fable in the form of test scenarios,
in which a sequence of actions are taken, and then the results are
checked. If the checks fail, the scenario fails.
Fable runs the scenarios concurrently (but see the USING keyword),
within the constrains of hardware resources. If Fable determines it
doesn't have all the resources to run all scenarios at once, it will
run fewer, but randomly chosen scenarios concurrently, to more likely
to detect unintentional dependencies between scenarios.
# The Fable input language
Fable input consists of three types of files:
* one or more markdown files which document that acceptance tests
* a binding file (in YAML) which binds scenario steps to their
implementations
* scenario step implementations, which are implemented in some
programming language (e.g., Python or Rust); Fable will combine this
code with some scaffolding provided by Fable itself
The input files for a simple acceptance test suite for Fable would be
divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming
step implementation in Python).
## Markdown files
[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks
[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language
[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software)
The Fable input language is markdown files using [fenced code blocks][]
with backticks. The code blocks MUST indicate that they contain Fable
language:
```fable
given a service
and I am Tomjon
when I access the service
then it's OK
```
Any other code blocks are ignored by Fable and can be used for, say,
code examples.
FIXME: Later, we may want to add support for embedding
PlantUML or Graphviz Dot markup, and generate images from that
automatically. This would look like the example below, and Fable would
replace the code block with the generated image in the PDF output.
```dot
foo -> bar
```
Fable understands the full Markdown language, by ignoring everything
except its own code blocks and headings. It uses [Pandoc][] to produce
PDF files, and anything that Pandoc supports is OK to use.
[Pandoc]: https://pandoc.org/
Fable treats multiple Markdown files as one, as if they had been
concatenated with the **cat**(1) utility. Within the logical file,
normal Markdown and Pandoc markup can be used to structure the
document in any way that aids human understanding of the acceptance
test suite, which the caveat that chapter or section headings are used
by Fable to group code blocks into scenarios.
All code blocks for the same scenario MUST be grouped under a single
heading. Sub-headings are permitted within a scenario, but the next
heading at the same or a higher level will end the scenario. This
allows for scenarios to begin at any level, but not to leak into a
wider scope within the acceptance document. For example, a scenario
which starts after a level 2 heading may have subdivisions marked with
level 3 or below headings, but will end at the next level 2 or level 1
heading.
Within the Fable code blocks, Fable understands a special language,
derived from [Gherkin][], as defined by the [Cucumber][] testing tool.
The language understood by Fable has the following general structure:
* each logical line starts with a keyword at the beginning of the line
* logical lines may be broken into physical lines, by starting the
continuation lines with one or more space or TAB characters; the
physical line break and white space characters are preserved
* logical lines define steps in a test scenario
* the meaning and implementation of the steps are defined by other
Fable input files
* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with
meanings defined below; keywords can be written in upper or lower
case, or mixes, Fable doesn't care
The keywords have the following meanings:
* ASSUMING—a condition for the scenario; all ASSUMING steps MUST
be at the beginning of the scenario, and may not be preceded by
other steps; if the condition fails, the scenario is skipped.
This is used for skipping test scenarios that require specific
software to be installed in the test environment, or access to
external services, but which can't be required for all runs of the
acceptance tests.
All ASSUMING steps MUST come first in the scenario.
* USING—indicate that the scenario uses a resource such as a
database, that's constrained and can't be used by all scenarios if
they run concurrently. When scenarios declare the resource, Fable can
limit which scenarios run concurrently.
For example, if several scenarios require uncontested use of the
GPU, of which there is typically only one per machine, they can all
declare "using the graphical processing unit", and Fable will run
them one at a time.
(This is an intentionally simplistic way of controlling concurrency.
The goal is to be simple and correct rather then achievee maximal
concurrency.)
The actual management of resources belongs to the generated test
program at runtime, not the Fable compiler.
All USING steps MUST come at the beginning of the scenario, except
the ASSUMING steps may precede them.
* GIVEN—set up the test environment for the action (WHEN). This
might create files, start a background process, or something like
that. This also sets up the reversal of the setup, so that any
background processes are stopped automatically after the scenario
has ended. The setup and cleanup MUST succeed, or the scenario will
fail.
The cleanups are executed in the reverse order of the GIVENs, and
they're done always, whether the scenario succeeds or not.
* WHEN—perform the action that is being tested. This MUST
succeed. This might, for example, execute a command line program,
and capture its output and exit code.
* THEN—test the results of the action. This would examine the
output and exit code of the program run in a WHEN step, or examine
current content of the database, or whatever is needed.
* AND—this keyword exists to make scenarios "read" better in
English. The keyword indicates that this step should use the same
keyword as the previous step, whatever that keyword is. For example,
a step "THEN output is empty" might be followed by "AND the exit
code is 0" rather than "THEN the exit code is 0".
## Bindings
FIXME: The binding specification needs thought. This is just a sketch.
Binding files match scenario steps to functions that implement them,
using regular expressions. The bindings may also extract parts of the
steps, and pass them onto the functions as parameters.
Binding files are YAML files, with lists of bindings, each binding
being a dict. For example:
```yaml
- define:
name: string
exit_code: int
- pattern: given a service
function: start_service
cleanup: stop_service
- pattern: given I am (?P