From 87c39a5b41ab5d775177524e88fbcf12a4c452c1 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Sun, 9 Jun 2019 08:43:19 +0300 Subject: Drop: fable arch doc, now that it's in fable-poc.git --- Makefile | 11 --- README | 3 +- arch.dot | 29 ------- fable.md | 290 ------------------------------------------------------------- fable.yaml | 0 5 files changed, 2 insertions(+), 331 deletions(-) delete mode 100644 Makefile delete mode 100644 arch.dot delete mode 100644 fable.md delete mode 100644 fable.yaml diff --git a/Makefile b/Makefile deleted file mode 100644 index 1392c21..0000000 --- a/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -.SUFFIXES: .md .pdf .dot .svg - -all: fable.pdf - -fable.pdf: Makefile fable.md arch.svg - -.md.pdf: - pandoc -o $@ $< - -.dot.svg: - dot -Tsvg -o $@ $< diff --git a/README b/README index 756b1f9..5ef7ce9 100644 --- a/README +++ b/README @@ -1,4 +1,5 @@ fable - automated acceptance testing tool ============================================================================= -FIXME +This used to have the arch doc. It is moved, for now, to +fable-poc.git. diff --git a/arch.dot b/arch.dot deleted file mode 100644 index 8c8e780..0000000 --- a/arch.dot +++ /dev/null @@ -1,29 +0,0 @@ -digraph arch { - md [label="document source\n(Markdown)"]; - md [shape=box]; - - bindings [label="bindings file\n(YAML)"]; - bindings [shape=box]; - - impl [label="step implementations\n(Python, Rust, ...)"] - impl [shape=box]; - - fable [label="Fable"]; - fable [shape=ellipse]; - - pdf [label="PDF document"] - pdf [shape=box]; - - testprog [label="Test program\n(generated)"] - testprog [shape=box]; - - report [label="Test report"] - report [shape=box]; - - md -> fable; - bindings -> fable; - impl -> fable; - fable -> pdf; - fable -> testprog; - testprog -> report; -} diff --git a/fable.md b/fable.md deleted file mode 100644 index e1158fa..0000000 --- a/fable.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -title: Acceptance testing using Fable -author: -- Lars Wirzenius -- Daniel Silverstone -date: WORK IN PROGRESS -keywords: -- automated acceptance testing -- scenario testing -- gherkin -- cucumber -abstract: | - Fable is a tool that supports acceptance testing in two ways: it is a - way to write and run automated acceptance tests, and also - presents the acceptance test suite to non-expert readers as a - human-readable text document. - - This document explains Fable, its architecture, and its input - language. -... - -# Introduction - -Fable is a tool for acceptance testing of software. This means it helps -software developers to make sure their software fulfils the -acceptance criteria for the software. Such criteria may come from -users, the developers, their employers, or elsewhere. - -Fable is specifically meant for automated acceptance testing. It takes a -two-pronged approach, where it lets developers write automated -tests for all the acceptance criteria they have, and runs the tests. -On the other hand, Fable also produces a PDF file, which documents the -automated tests for non-technical stakeholders, such as project -managers and clients. - -More concretely, Fable helps developers implement and document their -automated acceptance tests in a way that, at the same time, helps the -developers automatically test their software, and write documentation -for the tests in way that doesn't require programming knowledge to -understand. - -Fable is meant to be a tool for developers, who use it to produce a -document, which is meant to facilitate communication between various -shareholders of the software being developed. - -Fable's overall working principle is that the tests are implemented and -documented in a number of source files, which Fable reads to execute -tests, and to produce a PDF document for non-developer consumption. - -# Fable architecture - -![Fable architecture](arch.svg)\ - -Fable's architecture it to read input files, and produce two outputs. -On the one hand, it outputs a program that executes the tests -specified in the input files. The person running Fable then runs the -test program to get a test report, with results of each of the test -scenarios. On the other hand it outputs a human-readable document (as -PDF), for communicating what is being tested. - -Fable is able to produce the test program in various languages, using -a templating system to make it simple to add support for more -languages. Fable comes with support for Python and Rust. It's the -user's choice which language they're most comfortable with. - -Acceptance tests are expressed to Fable in the form of test scenarios, -in which a sequence of actions are taken, and then the results are -checked. If the checks fail, the scenario fails. - -Fable runs the scenarios concurrently (but see the USING keyword), -within the constrains of hardware resources. If Fable determines it -doesn't have all the resources to run all scenarios at once, it will -run fewer, but randomly chosen scenarios concurrently, to more likely -to detect unintentional dependencies between scenarios. - -# The Fable input language - -Fable input consists of three types of files: - -* one or more markdown files which document that acceptance tests -* a binding file (in YAML) which binds scenario steps to their - implementations -* scenario step implementations, which are implemented in some - programming language (e.g., Python or Rust); Fable will combine this - code with some scaffolding provided by Fable itself - -The input files for a simple acceptance test suite for Fable would be -divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming -step implementation in Python). - -## Markdown files - -[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks -[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language -[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) - -The Fable input language is markdown files using [fenced code blocks][] -with backticks. The code blocks MUST indicate that they contain Fable -language: - - ```fable - given a service - and I am Tomjon - when I access the service - then it's OK - ``` - -Any other code blocks are ignored by Fable and can be used for, say, -code examples. - -FIXME: Later, we may want to add support for embedding -PlantUML or Graphviz Dot markup, and generate images from that -automatically. This would look like the example below, and Fable would -replace the code block with the generated image in the PDF output. - - ```dot - foo -> bar - ``` - -Fable understands the full Markdown language, by ignoring everything -except its own code blocks and headings. It uses [Pandoc][] to produce -PDF files, and anything that Pandoc supports is OK to use. - -[Pandoc]: https://pandoc.org/ - -Fable treats multiple Markdown files as one, as if they had been -concatenated with the **cat**(1) utility. Within the logical file, -normal Markdown and Pandoc markup can be used to structure the -document in any way that aids human understanding of the acceptance -test suite, which the caveat that chapter or section headings are used -by Fable to group code blocks into scenarios. - -All code blocks for the same scenario MUST be grouped under a single -heading. Sub-headings are permitted within a scenario, but the next -heading at the same or a higher level will end the scenario. This -allows for scenarios to begin at any level, but not to leak into a -wider scope within the acceptance document. For example, a scenario -which starts after a level 2 heading may have subdivisions marked with -level 3 or below headings, but will end at the next level 2 or level 1 -heading. - -Within the Fable code blocks, Fable understands a special language, -derived from [Gherkin][], as defined by the [Cucumber][] testing tool. -The language understood by Fable has the following general structure: - -* each logical line starts with a keyword at the beginning of the line -* logical lines may be broken into physical lines, by starting the - continuation lines with one or more space or TAB characters; the - physical line break and white space characters are preserved -* logical lines define steps in a test scenario -* the meaning and implementation of the steps are defined by other - Fable input files -* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with - meanings defined below; keywords can be written in upper or lower - case, or mixes, Fable doesn't care - -The keywords have the following meanings: - -* ASSUMING—a condition for the scenario; all ASSUMING steps MUST - be at the beginning of the scenario, and may not be preceded by - other steps; if the condition fails, the scenario is skipped. - - This is used for skipping test scenarios that require specific - software to be installed in the test environment, or access to - external services, but which can't be required for all runs of the - acceptance tests. - - All ASSUMING steps MUST come first in the scenario. - -* USING—indicate that the scenario uses a resource such as a - database, that's constrained and can't be used by all scenarios if - they run concurrently. When scenarios declare the resource, Fable can - limit which scenarios run concurrently. - - For example, if several scenarios require uncontested use of the - GPU, of which there is typically only one per machine, they can all - declare "using the graphical processing unit", and Fable will run - them one at a time. - - (This is an intentionally simplistic way of controlling concurrency. - The goal is to be simple and correct rather then achievee maximal - concurrency.) - - The actual management of resources belongs to the generated test - program at runtime, not the Fable compiler. - - All USING steps MUST come at the beginning of the scenario, except - the ASSUMING steps may precede them. - -* GIVEN—set up the test environment for the action (WHEN). This - might create files, start a background process, or something like - that. This also sets up the reversal of the setup, so that any - background processes are stopped automatically after the scenario - has ended. The setup and cleanup MUST succeed, or the scenario will - fail. - - The cleanups are executed in the reverse order of the GIVENs, and - they're done always, whether the scenario succeeds or not. - -* WHEN—perform the action that is being tested. This MUST - succeed. This might, for example, execute a command line program, - and capture its output and exit code. - -* THEN—test the results of the action. This would examine the - output and exit code of the program run in a WHEN step, or examine - current content of the database, or whatever is needed. - -* AND—this keyword exists to make scenarios "read" better in - English. The keyword indicates that this step should use the same - keyword as the previous step, whatever that keyword is. For example, - a step "THEN output is empty" might be followed by "AND the exit - code is 0" rather than "THEN the exit code is 0". - -## Bindings - -FIXME: The binding specification needs thought. This is just a sketch. - -Binding files match scenario steps to functions that implement them, -using regular expressions. The bindings may also extract parts of the -steps, and pass them onto the functions as parameters. - -Binding files are YAML files, with lists of bindings, each binding -being a dict. For example: - -```yaml -- define: - name: string - exit_code: int - -- pattern: given a service - function: start_service - cleanup: stop_service - -- pattern: given I am (?P