From 4b6c89e424f09664178c5fdda574b44922d29567 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Fri, 13 Sep 2019 10:40:24 +0300 Subject: Add: copy of Fable's architecture document --- architecture.mdwn | 702 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 702 insertions(+) create mode 100644 architecture.mdwn (limited to 'architecture.mdwn') diff --git a/architecture.mdwn b/architecture.mdwn new file mode 100644 index 0000000..1116b41 --- /dev/null +++ b/architecture.mdwn @@ -0,0 +1,702 @@ +--- +documentclass: report +title: Acceptance testing using Fable +author: +- The Fable project +- Lars Wirzenius +- Daniel Silverstone +date: WORK IN PROGRESS +keywords: +- automated acceptance testing +- scenario testing +- gherkin +- cucumber +abstract: | + Fable is a tool that supports acceptance testing in two ways: it is a + way to write and run automated acceptance tests, and also + presents the acceptance test suite to non-expert readers as a + human-readable text document. + + This document explains Fable, its architecture, its input + language, and specifies the acceptance criteria for Fable. +... + +# Beware + +This document describes a future that will be, not current status quo. +Fable is its initial, active development period, and we don't want to +update this document every time we make a change. Thus, be aware that +parts of this document is speculative and may describe a version of +Fable that does not yet exist. + +# Introduction + +Fable is a tool for acceptance testing of software. It helps describe +acceptance criteria for all stakeholders in a software project, and +also for computers. Such criteria may come from paying clients, users, +the developers, their employers, or elsewhere. + +Fable is specifically meant for describing automated acceptance tests. +It takes a two-pronged approach, where it lets developers write +automated tests for all the acceptance criteria they have, and runs +the tests. On the other hand, Fable also produces a standalone +document, which describes the automated tests all stakeholders, +including non-technical ones, such as project managers and clients. + +More concretely, Fable helps developers specify the automated +acceptance tests so that the tests can be executed, but also +understood without requiring programming knowledge to understand. + +Fable is meant to be used to produce a document to facilitate +communication between various shareholders of the software being +developed. + +Fable's overall working principle is that the tests are implemented and +documented in a number of source files, which by two Fable tools. One +tool produces a PDF or HTML document, for humans to read. The other +produces a test program, which executes the acceptance tests. + +## A fairy tale of acceptance testing + +The king was upset. This naturally meant the whole court was in a +tizzy and chattering excitedly at each other, while trying to avoid +the royal wrath. + +"Who will rid me of this troublesome chore?" shouted the king, and +quaffed a flagon of wine. "And no killing of priests, this time!" + +The grand hall's doors were thrown open. The grand wizard stood in the +doorway, robe, hat, and staff everything, but quite still. After the +court became silent, the wizard strode confidently to stand before the +king. + +"What ails you, my lord?" + +The king looked upon the wizard, and took a deep breath. It does not +do to shout at wizards, for they control dragons, and even kings are +tasty morsels to the great beasts. + +"I am tired of choosing what to wear every day. Can't you do +something?" + +The wizard stoke his long, grey beard. He turned around, looked at the +magnificent outfits worn by members of the court. He turned back, and +looked at the king. + +"I believe I can fix this. Just to be clear, your beef is with having +to choose clothing, yes?" + +"Yes", said the king, "that's what I said. When will you be done?" + +The wizard raised his staff and brought it back down again, with a +loud bang. + +"Done" said the wizard, smugly. + +The king was amazed and started smiling, until he noticed that +everyone, including himself, was wearing identical burlap sacks and +nothing on their feet. His voice was high, whiny, like that of a +little child. + +"Oh no, that's not at all what I wanted! Change it back! Change it +back now!" + +The morale of this story is to be clear and precise in your acceptance +criteria, or you might get something than what you really, really +wanted. + +## Motivation for Fable + +Keeping track of requirements and acceptance criteria is necessary for +all but the simplest of software projects. Having all stakeholders in +a projects agree to them is crucial, as is that all agree how it is +verified that the software meets the acceptance criteria. Fable aims +to provide a way for documenting the shared understanding of what the +acceptance criteria are and how they can be checked automatically. + +Stakeholders in a project may include: + +* those who pay for the work to be done; this may be the employer of + the developers for in-house projects ("customer") +* those who use the resulting systems, whether they pay for it or not + ("user") +* those who install and configure the systems and keep them functional + ("sysadmin") +* those who support the users ("support") +* those who develop the system in the first place ("developer") + +All stakeholders need to understand the acceptance criteria, and how +the system is evaluated against them. In the simplest case, the +customer and the developer need to both understand and agree so that +the developer knows when the job is done, and the customer knows when +they need to pay their bill. + +However, even when the various stakeholder roles all fall upon the +same person, or only on people who act as developers, the Fable +approach can be useful. A developer would understand acceptance +criteria expressed only in code, but doing so may take time and energy +that are not always available. The Fable approach aims to encourage +hiding unnecessary detail and documenting things in a way that is easy +to understand with little effort. + +Unfortunately, this does mean that for a Fable output document to +be good and helpful, writing it will require effort and skill. No tool +can replace that. + +# Requirements + +This chapter lists requirements for Fable. These requirements are not +meant to be testable as such. For more specific, testable acceptance +criteria, see the later [chapter with acceptance tests for +Fable](#acceptance). + +Each requirement here is given a unique mnemnoic id for easier +reference in discussions. + +**UnderstadableTests** + +: Acceptance tests should be possible to express in a way that's + easily understood by non-programmers. + +**EasyToWriteDocs** + +: The markup language for writing documentation should be easy to + write. + +**AidsComprehension** + +: The formatted human-readable documentation should use good layout + and typography to enhance comprension. + +**CodeSeparately** + +: The code to implement the acceptance tests should not be embedded in + the documentation source, but be in separate files. This makes it + easier to edit without specialised tooling. + +**AnyProgammingLanguage** + +: The developers implementing the acceptance tests should be free to + use a language they're familiar and comfortable with. Fable should + not require them to use a specific language. + +**FastTestExecution** + +: Executing the acceptance tests should be fast. + +**NoDeployment** + +: The acceptance test tooling should assume the system under test is + already deployed and available. Deploying is too big of a problem + space to bring into the scope of acceptance testing, and there are + already good tools for deployment. + +**MachineParseableResults** + +: The tests should produce a machine parseable result that can be + archived, post-processed, and analyzed in ways that are of interest + to the project using Fable. For example, to see trends in how long + tests take, how often tests fail, to find regressions, and to find + tests that don't provide value. + +# Fable architecture + +```dot +md [label="document source\n(Markdown)"]; +md [shape=box]; + +bindings [label="bindings file\n(YAML)"]; +bindings [shape=box]; + +impl [label="step implementations\n(Python, Rust, ...)"] +impl [shape=box]; + +fable [label="Fable"]; +fable [shape=ellipse]; + +pdf [label="PDF document"] +pdf [shape=box]; + +testprog [label="Test program\n(generated)"] +testprog [shape=box]; + +report [label="Test report"] +report [shape=box]; + +md -> fable; +bindings -> fable; +impl -> fable; +fable -> pdf; +fable -> testprog; +testprog -> report; +``` + +Fable reads input files, and produces two outputs. +On the one hand, it outputs a program that executes the tests +specified in the input files. The person running Fable then runs the +test program to get a test report, with results of each of the test +scenarios. On the other hand it outputs a human-readable document (as +PDF or HTML), for communicating what is being tested. + +Fable is able to produce the test program in various languages, using +a templating system to make it simple to add support for more +languages. Fable comes with support for Python and Rust. It's the +user's choice which language they're most comfortable with. + +Acceptance tests are expressed to Fable in the form of test scenarios, +in which a sequence of actions are taken, and then the results are +checked. If the checks fail, the scenario fails. + +Fable runs the scenarios concurrently (but see the USING keyword), +within the constrains of hardware resources. If Fable determines it +doesn't have all the resources to run all scenarios at once, it will +run fewer, but randomly chosen scenarios concurrently, to more likely +to detect unintentional dependencies between scenarios. + +Note that Fable or the scenarios it runs aren't meant deploy the +software, or start services to run in the background. + +# The Fable input language + +Fable input consists of three types of files: + +* one or more markdown files which document the acceptance tests +* a binding file (in YAML) which binds scenario steps to their + implementations +* scenario step implementations, which are implemented in some + programming language (e.g., Python or Rust); Fable will combine this + code with some scaffolding provided by Fable itself + +The input files for a simple acceptance test suite for Fable would be +divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming +step implementation in Python). + +## Markdown files + +[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks +[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language +[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) + +The Fable input language is markdown files using [fenced code blocks][] +with backticks. The code blocks must indicate that they contain Fable +language: + + ```fable + given a service + and I am Tomjon + when I access the service + then it's OK + ``` + +Code blocks for the PlantUML or GraphViz dot markup languages are +also supported. These can be used for generating embedded images in +the documented produced by Fable. + +Any unfenced code blocks (indented code blocks) are ignored by Fable, +as are fenced code blocks using unknown content types. + +The Fable code generator understands the full Markdown language, by +ignoring everything except headings and its own code blocks. The Fable +document generator uses [Pandoc][] to produce standalone files, and +anything that Pandoc supports is OK to use. + +[Pandoc]: https://pandoc.org/ + +Fable treats multiple Markdown files as one, as if they had been +concatenated with the **cat**(1) utility. Within the logical file, +normal Markdown and Pandoc markup can be used to structure the +document in any way that aids human understanding of the acceptance +test suite, which the caveat that chapter or section headings are used +by Fable to group code blocks into scenarios. + +All code blocks for the same scenario must be grouped under a single +heading. Sub-headings are permitted within a scenario, but the next +heading at the same or a higher level will end the scenario. This +allows for scenarios to begin at any level, but not to leak into a +wider scope within the acceptance document. For example, a scenario +which starts after a level 2 heading may have subdivisions marked with +level 3 or below headings, but will end at the next level 2 or level 1 +heading. + +Within the Fable code blocks, Fable understands a special language, +derived from [Gherkin][], as defined by the [Cucumber][] testing tool. +The language understood by Fable has the following general structure: + +* each logical line starts with a keyword at the beginning of the line +* logical lines may be broken into physical lines, by starting the + continuation lines with one or more space or TAB characters; the + physical line break and white space characters are preserved +* logical lines define steps in a test scenario +* the meaning and implementation of the steps are defined by other + Fable input files +* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, and AND, with + meanings defined below; keywords can be written in upper or lower + case, or mixes, Fable doesn't care + +The keywords have the following meanings: + +**assuming** + +: A condition for the scenario. If an ASSUMING step fails, the + scenario is skipped. + + This is used for skipping test scenarios that require specific + software to be installed in the test environment, or access to + external services, but which can't be required for all runs of the + acceptance tests. + +**using** + +: Indicate that the scenario uses a resource such as a + database, that's constrained and can't be used by all scenarios if + they run concurrently. When scenarios declare the resource, Fable can + limit which scenarios run concurrently. + + For example, if several scenarios require uncontested use of the + GPU, of which there is typically only one per machine, they can all + declare "using the graphical processing unit", and Fable will run + them one at a time. + + (This is an intentionally simplistic way of controlling concurrency. + The goal is to be simple and correct rather then achievee maximal + concurrency.) + + The actual management of resources belongs to the generated test + program at runtime, not the Fable compiler. + +**given** + +: Set up the test environment for the action (WHEN). This + might create files, start a background process, or something like + that. This also sets up the reversal of the setup, so that any + background processes are stopped automatically after the scenario + has ended. The setup and cleanup must succeed, or the scenario will + fail. + + The cleanups are executed in the reverse order of the GIVENs, and + they're done always, whether the scenario succeeds or not. + +**when** + +: Perform the action that is being tested. This must succeed. This + might, for example, execute a command line program, and capture + its output and exit code. + +**then** + +: Test the results of the action. This would examine the output and + exit code of the program run in a WHEN step, or examine current + content of the database, or whatever is needed. + +**and** + +: This keyword exists to make scenarios "read" better in English. + The keyword indicates that this step should use the same keyword + as the previous step, whatever that keyword is. For example, a + step "THEN output is empty" might be followed by "AND the exit + code is 0" rather than "THEN the exit code is 0". + +The order of steps is somewhat constrainted: first any ASSUMING steps, +then any USING steps, at least one WHEN must come before a THEN. + +## Bindings + +FIXME: The binding specification needs thought. This is just a sketch. + +Binding files match scenario steps to functions that implement them, +using regular expressions. The bindings may also extract parts of the +steps, and pass them onto the functions as parameters. + +Binding files are YAML files, with lists of bindings, each binding +being a dict. For example: + +```yaml +- define: + name: string + exit_code: int + +- pattern: given a service + function: start_service + cleanup: stop_service + +- pattern: given I am (?P