From 79b4286c82a9d6e9122a85790997dd873c49d046 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Tue, 4 Jun 2019 11:33:03 +0300 Subject: Add: fable-arch.md --- fable-arch.md | 318 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 fable-arch.md diff --git a/fable-arch.md b/fable-arch.md new file mode 100644 index 0000000..eef69b2 --- /dev/null +++ b/fable-arch.md @@ -0,0 +1,318 @@ +--- +title: Acceptance testing using Fable +author: +- Lars Wirzenius +- Daniel Silverstone +date: WORK IN PROGRESS +keywords: +- automated acceptance testing +- scenario testing +- gherkin +- cucumber +abstract: | + Fable is a tool that supports acceptance testing in two ways: it is a + way to write and run automated acceptance tests, and also + presents the acceptance test suite to non-expert readers as a + human-readable text document. + + This document explains Fable, its architecture, and its input + language. +... + +# Introduction + +Fable is a tool for acceptance testing of software. This means it helps +software developers to make sure their software fulfils the +acceptance criteria for the software. Such criteria may come from +users, the developers, their employers, or elsewhere. + +Fable is specifically meant for automated acceptance testing. It takes a +two-pronged approach, where it lets developers write automated +tests for all the acceptance criteria they have, and runs the tests. +On the other hand, Fable also produces a PDF file, which documents the +automated tests for non-technical stakeholders, such as project +managers and clients. + +More concretely, Fable helps developers implement and document their +automated acceptance tests in a way that, at the same time, helps the +developers automatically test their software, and write documentation +for the tests in way that doesn't require programming knowledge to +understand. + +Fable is meant to be a tool for developers, who use it to produce a +document, which is meant to facilitate communication between various +shareholders of the software being developed. + +Fable's overall working principle is that the tests are implemented and +documented in a number of source files, which Fable reads to execute +tests, and to produce a PDF document for non-developer consumption. + +# Fable architecture + +```dot +md [label="document source\n(Markdown)"]; +md [shape=box]; + +bindings [label="bindings file\n(YAML)"]; +bindings [shape=box]; + +impl [label="step implementations\n(Python, Rust, ...)"] +impl [shape=box]; + +fable [label="Fable"]; +fable [shape=ellipse]; + +pdf [label="PDF document"] +pdf [shape=box]; + +testprog [label="Test program\n(generated)"] +testprog [shape=box]; + +report [label="Test report"] +report [shape=box]; + +md -> fable; +bindings -> fable; +impl -> fable; +fable -> pdf; +fable -> testprog; +testprog -> report; +``` + +Fable's architecture it to read input files, and produce two outputs. +On the one hand, it outputs a program that executes the tests +specified in the input files. The person running Fable then runs the +test program to get a test report, with results of each of the test +scenarios. On the other hand it outputs a human-readable document (as +PDF), for communicating what is being tested. + +Fable is able to produce the test program in various languages, using +a templating system to make it simple to add support for more +languages. Fable comes with support for Python and Rust. It's the +user's choice which language they're most comfortable with. + +Acceptance tests are expressed to Fable in the form of test scenarios, +in which a sequence of actions are taken, and then the results are +checked. If the checks fail, the scenario fails. + +Fable runs the scenarios concurrently (but see the USING keyword), +within the constrains of hardware resources. If Fable determines it +doesn't have all the resources to run all scenarios at once, it will +run fewer, but randomly chosen scenarios concurrently, to more likely +to detect unintentional dependencies between scenarios. + +# The Fable input language + +Fable input consists of three types of files: + +* one or more markdown files which document that acceptance tests +* a binding file (in YAML) which binds scenario steps to their + implementations +* scenario step implementations, which are implemented in some + programming language (e.g., Python or Rust); Fable will combine this + code with some scaffolding provided by Fable itself + +The input files for a simple acceptance test suite for Fable would be +divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming +step implementation in Python). + +## Markdown files + +[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks +[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language +[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) + +The Fable input language is markdown files using [fenced code blocks][] +with backticks. The code blocks MUST indicate that they contain Fable +language: + + ```fable + given a service + and I am Tomjon + when I access the service + then it's OK + ``` + +Any other code blocks are ignored by Fable and can be used for, say, +code examples. + +FIXME: Later, we may want to add support for embedding +PlantUML or Graphviz Dot markup, and generate images from that +automatically. This would look like the example below, and Fable would +replace the code block with the generated image in the PDF output. + + ```dot + foo -> bar + ``` + +Fable understands the full Markdown language, by ignoring everything +except its own code blocks and headings. It uses [Pandoc][] to produce +PDF files, and anything that Pandoc supports is OK to use. + +[Pandoc]: https://pandoc.org/ + +Fable treats multiple Markdown files as one, as if they had been +concatenated with the **cat**(1) utility. Within the logical file, +normal Markdown and Pandoc markup can be used to structure the +document in any way that aids human understanding of the acceptance +test suite, which the caveat that chapter or section headings are used +by Fable to group code blocks into scenarios. + +All code blocks for the same scenario MUST be grouped under a single +heading. Sub-headings are permitted within a scenario, but the next +heading at the same or a higher level will end the scenario. This +allows for scenarios to begin at any level, but not to leak into a +wider scope within the acceptance document. For example, a scenario +which starts after a level 2 heading may have subdivisions marked with +level 3 or below headings, but will end at the next level 2 or level 1 +heading. + +Within the Fable code blocks, Fable understands a special language, +derived from [Gherkin][], as defined by the [Cucumber][] testing tool. +The language understood by Fable has the following general structure: + +* each logical line starts with a keyword at the beginning of the line +* logical lines may be broken into physical lines, by starting the + continuation lines with one or more space or TAB characters; the + physical line break and white space characters are preserved +* logical lines define steps in a test scenario +* the meaning and implementation of the steps are defined by other + Fable input files +* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with + meanings defined below; keywords can be written in upper or lower + case, or mixes, Fable doesn't care + +The keywords have the following meanings: + +* ASSUMING—a condition for the scenario; all ASSUMING steps MUST + be at the beginning of the scenario, and may not be preceded by + other steps; if the condition fails, the scenario is skipped. + + This is used for skipping test scenarios that require specific + software to be installed in the test environment, or access to + external services, but which can't be required for all runs of the + acceptance tests. + + All ASSUMING steps MUST come first in the scenario. + +* USING—indicate that the scenario uses a resource such as a + database, that's constrained and can't be used by all scenarios if + they run concurrently. When scenarios declare the resource, Fable can + limit which scenarios run concurrently. + + For example, if several scenarios require uncontested use of the + GPU, of which there is typically only one per machine, they can all + declare "using the graphical processing unit", and Fable will run + them one at a time. + + (This is an intentionally simplistic way of controlling concurrency. + The goal is to be simple and correct rather then achievee maximal + concurrency.) + + The actual management of resources belongs to the generated test + program at runtime, not the Fable compiler. + + All USING steps MUST come at the beginning of the scenario, except + the ASSUMING steps may precede them. + +* GIVEN—set up the test environment for the action (WHEN). This + might create files, start a background process, or something like + that. This also sets up the reversal of the setup, so that any + background processes are stopped automatically after the scenario + has ended. The setup and cleanup MUST succeed, or the scenario will + fail. + + The cleanups are executed in the reverse order of the GIVENs, and + they're done always, whether the scenario succeeds or not. + +* WHEN—perform the action that is being tested. This MUST + succeed. This might, for example, execute a command line program, + and capture its output and exit code. + +* THEN—test the results of the action. This would examine the + output and exit code of the program run in a WHEN step, or examine + current content of the database, or whatever is needed. + +* AND—this keyword exists to make scenarios "read" better in + English. The keyword indicates that this step should use the same + keyword as the previous step, whatever that keyword is. For example, + a step "THEN output is empty" might be followed by "AND the exit + code is 0" rather than "THEN the exit code is 0". + +## Bindings + +FIXME: The binding specification needs thought. This is just a sketch. + +Binding files match scenario steps to functions that implement them, +using regular expressions. The bindings may also extract parts of the +steps, and pass them onto the functions as parameters. + +Binding files are YAML files, with lists of bindings, each binding +being a dict. For example: + +```yaml +- define: + name: string + exit_code: int + +- pattern: given a service + function: start_service + cleanup: stop_service + +- pattern: given I am (?P