diff options
author | Lars Wirzenius <liw@liw.fi> | 2019-06-09 08:43:19 +0300 |
---|---|---|
committer | Lars Wirzenius <liw@liw.fi> | 2019-06-09 08:43:19 +0300 |
commit | 87c39a5b41ab5d775177524e88fbcf12a4c452c1 (patch) | |
tree | ec475e4862c2549a29746284d4ff90df592ca159 | |
parent | 7456120b2c7f0572e76fb81901413abc9092a482 (diff) | |
download | fable-87c39a5b41ab5d775177524e88fbcf12a4c452c1.tar.gz |
-rw-r--r-- | Makefile | 11 | ||||
-rw-r--r-- | README | 3 | ||||
-rw-r--r-- | arch.dot | 29 | ||||
-rw-r--r-- | fable.md | 290 | ||||
-rw-r--r-- | fable.yaml | 0 |
5 files changed, 2 insertions, 331 deletions
diff --git a/Makefile b/Makefile deleted file mode 100644 index 1392c21..0000000 --- a/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -.SUFFIXES: .md .pdf .dot .svg - -all: fable.pdf - -fable.pdf: Makefile fable.md arch.svg - -.md.pdf: - pandoc -o $@ $< - -.dot.svg: - dot -Tsvg -o $@ $< @@ -1,4 +1,5 @@ fable - automated acceptance testing tool ============================================================================= -FIXME +This used to have the arch doc. It is moved, for now, to +fable-poc.git. diff --git a/arch.dot b/arch.dot deleted file mode 100644 index 8c8e780..0000000 --- a/arch.dot +++ /dev/null @@ -1,29 +0,0 @@ -digraph arch { - md [label="document source\n(Markdown)"]; - md [shape=box]; - - bindings [label="bindings file\n(YAML)"]; - bindings [shape=box]; - - impl [label="step implementations\n(Python, Rust, ...)"] - impl [shape=box]; - - fable [label="Fable"]; - fable [shape=ellipse]; - - pdf [label="PDF document"] - pdf [shape=box]; - - testprog [label="Test program\n(generated)"] - testprog [shape=box]; - - report [label="Test report"] - report [shape=box]; - - md -> fable; - bindings -> fable; - impl -> fable; - fable -> pdf; - fable -> testprog; - testprog -> report; -} diff --git a/fable.md b/fable.md deleted file mode 100644 index e1158fa..0000000 --- a/fable.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: Acceptance testing using Fable -author: -- Lars Wirzenius -- Daniel Silverstone -date: WORK IN PROGRESS -keywords: -- automated acceptance testing -- scenario testing -- gherkin -- cucumber -abstract: | - Fable is a tool that supports acceptance testing in two ways: it is a - way to write and run automated acceptance tests, and also - presents the acceptance test suite to non-expert readers as a - human-readable text document. - - This document explains Fable, its architecture, and its input - language. -... - -# Introduction - -Fable is a tool for acceptance testing of software. This means it helps -software developers to make sure their software fulfils the -acceptance criteria for the software. Such criteria may come from -users, the developers, their employers, or elsewhere. - -Fable is specifically meant for automated acceptance testing. It takes a -two-pronged approach, where it lets developers write automated -tests for all the acceptance criteria they have, and runs the tests. -On the other hand, Fable also produces a PDF file, which documents the -automated tests for non-technical stakeholders, such as project -managers and clients. - -More concretely, Fable helps developers implement and document their -automated acceptance tests in a way that, at the same time, helps the -developers automatically test their software, and write documentation -for the tests in way that doesn't require programming knowledge to -understand. - -Fable is meant to be a tool for developers, who use it to produce a -document, which is meant to facilitate communication between various -shareholders of the software being developed. - -Fable's overall working principle is that the tests are implemented and -documented in a number of source files, which Fable reads to execute -tests, and to produce a PDF document for non-developer consumption. - -# Fable architecture - -![Fable architecture](arch.svg)\ - -Fable's architecture it to read input files, and produce two outputs. -On the one hand, it outputs a program that executes the tests -specified in the input files. The person running Fable then runs the -test program to get a test report, with results of each of the test -scenarios. On the other hand it outputs a human-readable document (as -PDF), for communicating what is being tested. - -Fable is able to produce the test program in various languages, using -a templating system to make it simple to add support for more -languages. Fable comes with support for Python and Rust. It's the -user's choice which language they're most comfortable with. - -Acceptance tests are expressed to Fable in the form of test scenarios, -in which a sequence of actions are taken, and then the results are -checked. If the checks fail, the scenario fails. - -Fable runs the scenarios concurrently (but see the USING keyword), -within the constrains of hardware resources. If Fable determines it -doesn't have all the resources to run all scenarios at once, it will -run fewer, but randomly chosen scenarios concurrently, to more likely -to detect unintentional dependencies between scenarios. - -# The Fable input language - -Fable input consists of three types of files: - -* one or more markdown files which document that acceptance tests -* a binding file (in YAML) which binds scenario steps to their - implementations -* scenario step implementations, which are implemented in some - programming language (e.g., Python or Rust); Fable will combine this - code with some scaffolding provided by Fable itself - -The input files for a simple acceptance test suite for Fable would be -divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming -step implementation in Python). - -## Markdown files - -[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks -[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language -[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) - -The Fable input language is markdown files using [fenced code blocks][] -with backticks. The code blocks MUST indicate that they contain Fable -language: - - ```fable - given a service - and I am Tomjon - when I access the service - then it's OK - ``` - -Any other code blocks are ignored by Fable and can be used for, say, -code examples. - -FIXME: Later, we may want to add support for embedding -PlantUML or Graphviz Dot markup, and generate images from that -automatically. This would look like the example below, and Fable would -replace the code block with the generated image in the PDF output. - - ```dot - foo -> bar - ``` - -Fable understands the full Markdown language, by ignoring everything -except its own code blocks and headings. It uses [Pandoc][] to produce -PDF files, and anything that Pandoc supports is OK to use. - -[Pandoc]: https://pandoc.org/ - -Fable treats multiple Markdown files as one, as if they had been -concatenated with the **cat**(1) utility. Within the logical file, -normal Markdown and Pandoc markup can be used to structure the -document in any way that aids human understanding of the acceptance -test suite, which the caveat that chapter or section headings are used -by Fable to group code blocks into scenarios. - -All code blocks for the same scenario MUST be grouped under a single -heading. Sub-headings are permitted within a scenario, but the next -heading at the same or a higher level will end the scenario. This -allows for scenarios to begin at any level, but not to leak into a -wider scope within the acceptance document. For example, a scenario -which starts after a level 2 heading may have subdivisions marked with -level 3 or below headings, but will end at the next level 2 or level 1 -heading. - -Within the Fable code blocks, Fable understands a special language, -derived from [Gherkin][], as defined by the [Cucumber][] testing tool. -The language understood by Fable has the following general structure: - -* each logical line starts with a keyword at the beginning of the line -* logical lines may be broken into physical lines, by starting the - continuation lines with one or more space or TAB characters; the - physical line break and white space characters are preserved -* logical lines define steps in a test scenario -* the meaning and implementation of the steps are defined by other - Fable input files -* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with - meanings defined below; keywords can be written in upper or lower - case, or mixes, Fable doesn't care - -The keywords have the following meanings: - -* ASSUMING—a condition for the scenario; all ASSUMING steps MUST - be at the beginning of the scenario, and may not be preceded by - other steps; if the condition fails, the scenario is skipped. - - This is used for skipping test scenarios that require specific - software to be installed in the test environment, or access to - external services, but which can't be required for all runs of the - acceptance tests. - - All ASSUMING steps MUST come first in the scenario. - -* USING—indicate that the scenario uses a resource such as a - database, that's constrained and can't be used by all scenarios if - they run concurrently. When scenarios declare the resource, Fable can - limit which scenarios run concurrently. - - For example, if several scenarios require uncontested use of the - GPU, of which there is typically only one per machine, they can all - declare "using the graphical processing unit", and Fable will run - them one at a time. - - (This is an intentionally simplistic way of controlling concurrency. - The goal is to be simple and correct rather then achievee maximal - concurrency.) - - The actual management of resources belongs to the generated test - program at runtime, not the Fable compiler. - - All USING steps MUST come at the beginning of the scenario, except - the ASSUMING steps may precede them. - -* GIVEN—set up the test environment for the action (WHEN). This - might create files, start a background process, or something like - that. This also sets up the reversal of the setup, so that any - background processes are stopped automatically after the scenario - has ended. The setup and cleanup MUST succeed, or the scenario will - fail. - - The cleanups are executed in the reverse order of the GIVENs, and - they're done always, whether the scenario succeeds or not. - -* WHEN—perform the action that is being tested. This MUST - succeed. This might, for example, execute a command line program, - and capture its output and exit code. - -* THEN—test the results of the action. This would examine the - output and exit code of the program run in a WHEN step, or examine - current content of the database, or whatever is needed. - -* AND—this keyword exists to make scenarios "read" better in - English. The keyword indicates that this step should use the same - keyword as the previous step, whatever that keyword is. For example, - a step "THEN output is empty" might be followed by "AND the exit - code is 0" rather than "THEN the exit code is 0". - -## Bindings - -FIXME: The binding specification needs thought. This is just a sketch. - -Binding files match scenario steps to functions that implement them, -using regular expressions. The bindings may also extract parts of the -steps, and pass them onto the functions as parameters. - -Binding files are YAML files, with lists of bindings, each binding -being a dict. For example: - -```yaml -- define: - name: string - exit_code: int - -- pattern: given a service - function: start_service - cleanup: stop_service - -- pattern: given I am (?P<name\S+) - function: set_name - produces: [name] - -- pattern: when I access the service - function: access_service - requires: [name] - produces: [exit_code] - -- pattern: then it's OK - function: check_access_was_ok - requires: [exit_code] -``` - -In the example above, the "I am" step extracts the name of the user -from the step. It's type is declared, and the value is saved for use -by a later step. - -The "I access" step expects the name to have been set by a previous -step. Fable will check that the name is set, and give an error if it -isn't, before any scenario runs. If name is set, it is given to the -function to be called as a function argument. - -The "I access" step further sets the variable "exit_code", and the -"it's OK" step expects it to be set. - -## Step implementations - -Continuing the example from the previous section, the following Python -code might implement the functions: - -```python -def start_service(): - ... - -def set_name(**matches): - ... - -def access_service(name): - ... - return { - 'exit_code': 0, - } - -def check_access_was_ok(exit_code): - assert exit_code == 0 -``` - -With these bindings, Fable produces a Python program, which calls -these functions in order, and passes values between them via function -arguments and return values. The generated program will handle running -scenarios concurrently, and taking care of USING constraints, and -other resource constraints. - -# Acceptance tests for Fable - -FIXME: This needs to be written eventually, when Fable is implemented. diff --git a/fable.yaml b/fable.yaml deleted file mode 100644 index e69de29..0000000 --- a/fable.yaml +++ /dev/null |