summaryrefslogtreecommitdiff
path: root/fable-arch.md
diff options
context:
space:
mode:
Diffstat (limited to 'fable-arch.md')
-rw-r--r--fable-arch.md318
1 files changed, 318 insertions, 0 deletions
diff --git a/fable-arch.md b/fable-arch.md
new file mode 100644
index 0000000..eef69b2
--- /dev/null
+++ b/fable-arch.md
@@ -0,0 +1,318 @@
+---
+title: Acceptance testing using Fable
+author:
+- Lars Wirzenius
+- Daniel Silverstone
+date: WORK IN PROGRESS
+keywords:
+- automated acceptance testing
+- scenario testing
+- gherkin
+- cucumber
+abstract: |
+ Fable is a tool that supports acceptance testing in two ways: it is a
+ way to write and run automated acceptance tests, and also
+ presents the acceptance test suite to non-expert readers as a
+ human-readable text document.
+
+ This document explains Fable, its architecture, and its input
+ language.
+...
+
+# Introduction
+
+Fable is a tool for acceptance testing of software. This means it helps
+software developers to make sure their software fulfils the
+acceptance criteria for the software. Such criteria may come from
+users, the developers, their employers, or elsewhere.
+
+Fable is specifically meant for automated acceptance testing. It takes a
+two-pronged approach, where it lets developers write automated
+tests for all the acceptance criteria they have, and runs the tests.
+On the other hand, Fable also produces a PDF file, which documents the
+automated tests for non-technical stakeholders, such as project
+managers and clients.
+
+More concretely, Fable helps developers implement and document their
+automated acceptance tests in a way that, at the same time, helps the
+developers automatically test their software, and write documentation
+for the tests in way that doesn't require programming knowledge to
+understand.
+
+Fable is meant to be a tool for developers, who use it to produce a
+document, which is meant to facilitate communication between various
+shareholders of the software being developed.
+
+Fable's overall working principle is that the tests are implemented and
+documented in a number of source files, which Fable reads to execute
+tests, and to produce a PDF document for non-developer consumption.
+
+# Fable architecture
+
+```dot
+md [label="document source\n(Markdown)"];
+md [shape=box];
+
+bindings [label="bindings file\n(YAML)"];
+bindings [shape=box];
+
+impl [label="step implementations\n(Python, Rust, ...)"]
+impl [shape=box];
+
+fable [label="Fable"];
+fable [shape=ellipse];
+
+pdf [label="PDF document"]
+pdf [shape=box];
+
+testprog [label="Test program\n(generated)"]
+testprog [shape=box];
+
+report [label="Test report"]
+report [shape=box];
+
+md -> fable;
+bindings -> fable;
+impl -> fable;
+fable -> pdf;
+fable -> testprog;
+testprog -> report;
+```
+
+Fable's architecture it to read input files, and produce two outputs.
+On the one hand, it outputs a program that executes the tests
+specified in the input files. The person running Fable then runs the
+test program to get a test report, with results of each of the test
+scenarios. On the other hand it outputs a human-readable document (as
+PDF), for communicating what is being tested.
+
+Fable is able to produce the test program in various languages, using
+a templating system to make it simple to add support for more
+languages. Fable comes with support for Python and Rust. It's the
+user's choice which language they're most comfortable with.
+
+Acceptance tests are expressed to Fable in the form of test scenarios,
+in which a sequence of actions are taken, and then the results are
+checked. If the checks fail, the scenario fails.
+
+Fable runs the scenarios concurrently (but see the USING keyword),
+within the constrains of hardware resources. If Fable determines it
+doesn't have all the resources to run all scenarios at once, it will
+run fewer, but randomly chosen scenarios concurrently, to more likely
+to detect unintentional dependencies between scenarios.
+
+# The Fable input language
+
+Fable input consists of three types of files:
+
+* one or more markdown files which document that acceptance tests
+* a binding file (in YAML) which binds scenario steps to their
+ implementations
+* scenario step implementations, which are implemented in some
+ programming language (e.g., Python or Rust); Fable will combine this
+ code with some scaffolding provided by Fable itself
+
+The input files for a simple acceptance test suite for Fable would be
+divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming
+step implementation in Python).
+
+## Markdown files
+
+[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks
+[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language
+[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software)
+
+The Fable input language is markdown files using [fenced code blocks][]
+with backticks. The code blocks MUST indicate that they contain Fable
+language:
+
+ ```fable
+ given a service
+ and I am Tomjon
+ when I access the service
+ then it's OK
+ ```
+
+Any other code blocks are ignored by Fable and can be used for, say,
+code examples.
+
+FIXME: Later, we may want to add support for embedding
+PlantUML or Graphviz Dot markup, and generate images from that
+automatically. This would look like the example below, and Fable would
+replace the code block with the generated image in the PDF output.
+
+ ```dot
+ foo -> bar
+ ```
+
+Fable understands the full Markdown language, by ignoring everything
+except its own code blocks and headings. It uses [Pandoc][] to produce
+PDF files, and anything that Pandoc supports is OK to use.
+
+[Pandoc]: https://pandoc.org/
+
+Fable treats multiple Markdown files as one, as if they had been
+concatenated with the **cat**(1) utility. Within the logical file,
+normal Markdown and Pandoc markup can be used to structure the
+document in any way that aids human understanding of the acceptance
+test suite, which the caveat that chapter or section headings are used
+by Fable to group code blocks into scenarios.
+
+All code blocks for the same scenario MUST be grouped under a single
+heading. Sub-headings are permitted within a scenario, but the next
+heading at the same or a higher level will end the scenario. This
+allows for scenarios to begin at any level, but not to leak into a
+wider scope within the acceptance document. For example, a scenario
+which starts after a level 2 heading may have subdivisions marked with
+level 3 or below headings, but will end at the next level 2 or level 1
+heading.
+
+Within the Fable code blocks, Fable understands a special language,
+derived from [Gherkin][], as defined by the [Cucumber][] testing tool.
+The language understood by Fable has the following general structure:
+
+* each logical line starts with a keyword at the beginning of the line
+* logical lines may be broken into physical lines, by starting the
+ continuation lines with one or more space or TAB characters; the
+ physical line break and white space characters are preserved
+* logical lines define steps in a test scenario
+* the meaning and implementation of the steps are defined by other
+ Fable input files
+* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with
+ meanings defined below; keywords can be written in upper or lower
+ case, or mixes, Fable doesn't care
+
+The keywords have the following meanings:
+
+* ASSUMING—a condition for the scenario; all ASSUMING steps MUST
+ be at the beginning of the scenario, and may not be preceded by
+ other steps; if the condition fails, the scenario is skipped.
+
+ This is used for skipping test scenarios that require specific
+ software to be installed in the test environment, or access to
+ external services, but which can't be required for all runs of the
+ acceptance tests.
+
+ All ASSUMING steps MUST come first in the scenario.
+
+* USING—indicate that the scenario uses a resource such as a
+ database, that's constrained and can't be used by all scenarios if
+ they run concurrently. When scenarios declare the resource, Fable can
+ limit which scenarios run concurrently.
+
+ For example, if several scenarios require uncontested use of the
+ GPU, of which there is typically only one per machine, they can all
+ declare "using the graphical processing unit", and Fable will run
+ them one at a time.
+
+ (This is an intentionally simplistic way of controlling concurrency.
+ The goal is to be simple and correct rather then achievee maximal
+ concurrency.)
+
+ The actual management of resources belongs to the generated test
+ program at runtime, not the Fable compiler.
+
+ All USING steps MUST come at the beginning of the scenario, except
+ the ASSUMING steps may precede them.
+
+* GIVEN—set up the test environment for the action (WHEN). This
+ might create files, start a background process, or something like
+ that. This also sets up the reversal of the setup, so that any
+ background processes are stopped automatically after the scenario
+ has ended. The setup and cleanup MUST succeed, or the scenario will
+ fail.
+
+ The cleanups are executed in the reverse order of the GIVENs, and
+ they're done always, whether the scenario succeeds or not.
+
+* WHEN—perform the action that is being tested. This MUST
+ succeed. This might, for example, execute a command line program,
+ and capture its output and exit code.
+
+* THEN—test the results of the action. This would examine the
+ output and exit code of the program run in a WHEN step, or examine
+ current content of the database, or whatever is needed.
+
+* AND—this keyword exists to make scenarios "read" better in
+ English. The keyword indicates that this step should use the same
+ keyword as the previous step, whatever that keyword is. For example,
+ a step "THEN output is empty" might be followed by "AND the exit
+ code is 0" rather than "THEN the exit code is 0".
+
+## Bindings
+
+FIXME: The binding specification needs thought. This is just a sketch.
+
+Binding files match scenario steps to functions that implement them,
+using regular expressions. The bindings may also extract parts of the
+steps, and pass them onto the functions as parameters.
+
+Binding files are YAML files, with lists of bindings, each binding
+being a dict. For example:
+
+```yaml
+- define:
+ name: string
+ exit_code: int
+
+- pattern: given a service
+ function: start_service
+ cleanup: stop_service
+
+- pattern: given I am (?P<name\S+)
+ function: set_name
+ produces: [name]
+
+- pattern: when I access the service
+ function: access_service
+ requires: [name]
+ produces: [exit_code]
+
+- pattern: then it's OK
+ function: check_access_was_ok
+ requires: [exit_code]
+```
+
+In the example above, the "I am" step extracts the name of the user
+from the step. It's type is declared, and the value is saved for use
+by a later step.
+
+The "I access" step expects the name to have been set by a previous
+step. Fable will check that the name is set, and give an error if it
+isn't, before any scenario runs. If name is set, it is given to the
+function to be called as a function argument.
+
+The "I access" step further sets the variable "exit_code", and the
+"it's OK" step expects it to be set.
+
+## Step implementations
+
+Continuing the example from the previous section, the following Python
+code might implement the functions:
+
+```python
+def start_service():
+ ...
+
+def set_name(**matches):
+ ...
+
+def access_service(name):
+ ...
+ return {
+ 'exit_code': 0,
+ }
+
+def check_access_was_ok(exit_code):
+ assert exit_code == 0
+```
+
+With these bindings, Fable produces a Python program, which calls
+these functions in order, and passes values between them via function
+arguments and return values. The generated program will handle running
+scenarios concurrently, and taking care of USING constraints, and
+other resource constraints.
+
+# Acceptance tests for Fable
+
+FIXME: This needs to be written eventually, when Fable is implemented.