summaryrefslogtreecommitdiff
path: root/architecture.mdwn
diff options
context:
space:
mode:
authorLars Wirzenius <liw@liw.fi>2019-09-13 10:58:24 +0300
committerLars Wirzenius <liw@liw.fi>2019-09-13 10:59:39 +0300
commit4cdc0222827529e3b3de2d9f3a63f865d98446ce (patch)
treefbebae630ee7d8810c4e75b91378976aa970e101 /architecture.mdwn
parent4b6c89e424f09664178c5fdda574b44922d29567 (diff)
downloadsubplot.liw.fi-4cdc0222827529e3b3de2d9f3a63f865d98446ce.tar.gz
Change: update arch doc for Subplot
Diffstat (limited to 'architecture.mdwn')
-rw-r--r--architecture.mdwn217
1 files changed, 101 insertions, 116 deletions
diff --git a/architecture.mdwn b/architecture.mdwn
index 1116b41..4c9a683 100644
--- a/architecture.mdwn
+++ b/architecture.mdwn
@@ -1,58 +1,38 @@
----
-documentclass: report
-title: Acceptance testing using Fable
-author:
-- The Fable project
-- Lars Wirzenius
-- Daniel Silverstone
-date: WORK IN PROGRESS
-keywords:
-- automated acceptance testing
-- scenario testing
-- gherkin
-- cucumber
-abstract: |
- Fable is a tool that supports acceptance testing in two ways: it is a
- way to write and run automated acceptance tests, and also
- presents the acceptance test suite to non-expert readers as a
- human-readable text document.
-
- This document explains Fable, its architecture, its input
- language, and specifies the acceptance criteria for Fable.
-...
-
-# Beware
+[[!meta title="Architecture of Subplot"]]
+
+**Beware**
This document describes a future that will be, not current status quo.
-Fable is its initial, active development period, and we don't want to
+Subplot is its initial, active development period, and we don't want to
update this document every time we make a change. Thus, be aware that
parts of this document is speculative and may describe a version of
-Fable that does not yet exist.
+Subplot that does not yet exist.
+
# Introduction
-Fable is a tool for acceptance testing of software. It helps describe
+Subplot is a tool for acceptance testing of software. It helps describe
acceptance criteria for all stakeholders in a software project, and
also for computers. Such criteria may come from paying clients, users,
the developers, their employers, or elsewhere.
-Fable is specifically meant for describing automated acceptance tests.
+Subplot is specifically meant for describing automated acceptance tests.
It takes a two-pronged approach, where it lets developers write
automated tests for all the acceptance criteria they have, and runs
-the tests. On the other hand, Fable also produces a standalone
+the tests. On the other hand, Subplot also produces a standalone
document, which describes the automated tests all stakeholders,
including non-technical ones, such as project managers and clients.
-More concretely, Fable helps developers specify the automated
+More concretely, Subplot helps developers specify the automated
acceptance tests so that the tests can be executed, but also
understood without requiring programming knowledge to understand.
-Fable is meant to be used to produce a document to facilitate
+Subplot is meant to be used to produce a document to facilitate
communication between various shareholders of the software being
developed.
-Fable's overall working principle is that the tests are implemented and
-documented in a number of source files, which by two Fable tools. One
+Subplot's overall working principle is that the tests are implemented and
+documented in a number of source files, which are processed by two Subplot tools. One
tool produces a PDF or HTML document, for humans to read. The other
produces a test program, which executes the acceptance tests.
@@ -102,19 +82,20 @@ little child.
back now!"
The morale of this story is to be clear and precise in your acceptance
-criteria, or you might get something than what you really, really
+criteria, and make sure everyone involved understands them the same
+way, or you might get something else than what you really, really
wanted.
-## Motivation for Fable
+## Motivation for Subplot
Keeping track of requirements and acceptance criteria is necessary for
all but the simplest of software projects. Having all stakeholders in
-a projects agree to them is crucial, as is that all agree how it is
-verified that the software meets the acceptance criteria. Fable aims
+a project agree to them is crucial, as is that all agree how it is
+verified that the software meets the acceptance criteria. Subplot aims
to provide a way for documenting the shared understanding of what the
acceptance criteria are and how they can be checked automatically.
-Stakeholders in a project may include:
+Stakeholders in any given project may include:
* those who pay for the work to be done; this may be the employer of
the developers for in-house projects ("customer")
@@ -122,7 +103,7 @@ Stakeholders in a project may include:
("user")
* those who install and configure the systems and keep them functional
("sysadmin")
-* those who support the users ("support")
+* those who support the users ("helpdesk")
* those who develop the system in the first place ("developer")
All stakeholders need to understand the acceptance criteria, and how
@@ -132,23 +113,23 @@ the developer knows when the job is done, and the customer knows when
they need to pay their bill.
However, even when the various stakeholder roles all fall upon the
-same person, or only on people who act as developers, the Fable
+same person, or only on people who act as developers, the Subplot
approach can be useful. A developer would understand acceptance
criteria expressed only in code, but doing so may take time and energy
-that are not always available. The Fable approach aims to encourage
+that are not always available. The Subplot approach aims to encourage
hiding unnecessary detail and documenting things in a way that is easy
to understand with little effort.
-Unfortunately, this does mean that for a Fable output document to
+Unfortunately, this does mean that for a Subplot output document to
be good and helpful, writing it will require effort and skill. No tool
can replace that.
# Requirements
-This chapter lists requirements for Fable. These requirements are not
+This chapter lists requirements for Subplot itself. These requirements are not
meant to be testable as such. For more specific, testable acceptance
criteria, see the later [chapter with acceptance tests for
-Fable](#acceptance).
+Subplot](#acceptance).
Each requirement here is given a unique mnemnoic id for easier
reference in discussions.
@@ -177,7 +158,7 @@ reference in discussions.
**AnyProgammingLanguage**
: The developers implementing the acceptance tests should be free to
- use a language they're familiar and comfortable with. Fable should
+ use a language they're familiar and comfortable with. Subplot should
not require them to use a specific language.
**FastTestExecution**
@@ -195,11 +176,11 @@ reference in discussions.
: The tests should produce a machine parseable result that can be
archived, post-processed, and analyzed in ways that are of interest
- to the project using Fable. For example, to see trends in how long
+ to the project using Subplot. For example, to see trends in how long
tests take, how often tests fail, to find regressions, and to find
tests that don't provide value.
-# Fable architecture
+# Subplot architecture
```dot
md [label="document source\n(Markdown)"];
@@ -211,8 +192,8 @@ bindings [shape=box];
impl [label="step implementations\n(Python, Rust, ...)"]
impl [shape=box];
-fable [label="Fable"];
-fable [shape=ellipse];
+subplot [label="Subplot"];
+subplot [shape=ellipse];
pdf [label="PDF document"]
pdf [shape=box];
@@ -223,51 +204,51 @@ testprog [shape=box];
report [label="Test report"]
report [shape=box];
-md -> fable;
-bindings -> fable;
-impl -> fable;
-fable -> pdf;
-fable -> testprog;
+md -> subplot;
+bindings -> subplot;
+impl -> subplot;
+subplot -> pdf;
+subplot -> testprog;
testprog -> report;
```
-Fable reads input files, and produces two outputs.
+Subplot reads input files, and produces two outputs.
On the one hand, it outputs a program that executes the tests
-specified in the input files. The person running Fable then runs the
-test program to get a test report, with results of each of the test
-scenarios. On the other hand it outputs a human-readable document (as
+specified in the input files. The person running the generated program
+gets a test report, with results of each of the test
+scenarios. On the other hand Subplot also outputs a human-readable document (as
PDF or HTML), for communicating what is being tested.
-Fable is able to produce the test program in various languages, using
+Subplot is able to produce the test program in various languages, using
a templating system to make it simple to add support for more
-languages. Fable comes with support for Python and Rust. It's the
+languages. Subplot comes with support for Python and Rust. It's the
user's choice which language they're most comfortable with.
-Acceptance tests are expressed to Fable in the form of test scenarios,
+Acceptance tests are expressed to Subplot in the form of test scenarios,
in which a sequence of actions are taken, and then the results are
checked. If the checks fail, the scenario fails.
-Fable runs the scenarios concurrently (but see the USING keyword),
-within the constrains of hardware resources. If Fable determines it
+Subplot runs the scenarios concurrently (but see the USING keyword),
+within the constrains of hardware resources. If Subplot determines it
doesn't have all the resources to run all scenarios at once, it will
run fewer, but randomly chosen scenarios concurrently, to more likely
to detect unintentional dependencies between scenarios.
-Note that Fable or the scenarios it runs aren't meant deploy the
+Note that Subplot or the scenarios it runs aren't meant deploy the
software, or start services to run in the background.
-# The Fable input language
+# The Subplot input language
-Fable input consists of three types of files:
+Subplot input consists of three types of files:
* one or more markdown files which document the acceptance tests
-* a binding file (in YAML) which binds scenario steps to their
+* a bindings file (in YAML) which binds scenario steps to their
implementations
* scenario step implementations, which are implemented in some
- programming language (e.g., Python or Rust); Fable will combine this
- code with some scaffolding provided by Fable itself
+ programming language (e.g., Python or Rust); Subplot will combine this
+ code with some scaffolding provided by Subplot itself
-The input files for a simple acceptance test suite for Fable would be
+The input files for a simple acceptance test suite for Subplot would be
divided into three files: `foo.md`, `foo.yaml`, and `foo.py` (assuming
step implementation in Python).
@@ -277,37 +258,38 @@ step implementation in Python).
[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language
[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software)
-The Fable input language is markdown files using [fenced code blocks][]
-with backticks. The code blocks must indicate that they contain Fable
+The Subplot input language is markdown files using [fenced code blocks][]
+with backticks. The code blocks must indicate that they contain Subplot
language:
- ```fable
+ ```subplot
given a service
and I am Tomjon
when I access the service
then it's OK
```
-Code blocks for the PlantUML or GraphViz dot markup languages are
+Code blocks for the PlantUML, GraphViz dot markup, and a custom
+roadmap markup languages are
also supported. These can be used for generating embedded images in
-the documented produced by Fable.
+the documented produced by Subplot.
-Any unfenced code blocks (indented code blocks) are ignored by Fable,
+Any unfenced code blocks (indented code blocks) are ignored by Subplot,
as are fenced code blocks using unknown content types.
-The Fable code generator understands the full Markdown language, by
-ignoring everything except headings and its own code blocks. The Fable
+The Subplot code generator understands the full Markdown language, by
+ignoring everything except headings and its own code blocks. The Subplot
document generator uses [Pandoc][] to produce standalone files, and
anything that Pandoc supports is OK to use.
[Pandoc]: https://pandoc.org/
-Fable treats multiple Markdown files as one, as if they had been
+Subplot treats multiple Markdown files as one, as if they had been
concatenated with the **cat**(1) utility. Within the logical file,
normal Markdown and Pandoc markup can be used to structure the
document in any way that aids human understanding of the acceptance
test suite, which the caveat that chapter or section headings are used
-by Fable to group code blocks into scenarios.
+by Subplot to group code blocks into scenarios.
All code blocks for the same scenario must be grouped under a single
heading. Sub-headings are permitted within a scenario, but the next
@@ -318,9 +300,9 @@ which starts after a level 2 heading may have subdivisions marked with
level 3 or below headings, but will end at the next level 2 or level 1
heading.
-Within the Fable code blocks, Fable understands a special language,
-derived from [Gherkin][], as defined by the [Cucumber][] testing tool.
-The language understood by Fable has the following general structure:
+Within the Subplot code blocks, Subplot understands a special language,
+inspired by [Gherkin][], as used by the [Cucumber][] testing tool.
+The language understood by Subplot has the following general structure:
* each logical line starts with a keyword at the beginning of the line
* logical lines may be broken into physical lines, by starting the
@@ -328,10 +310,10 @@ The language understood by Fable has the following general structure:
physical line break and white space characters are preserved
* logical lines define steps in a test scenario
* the meaning and implementation of the steps are defined by other
- Fable input files
+ Subplot input files
* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, and AND, with
meanings defined below; keywords can be written in upper or lower
- case, or mixes, Fable doesn't care
+ case, or mixes, Subplot doesn't care
The keywords have the following meanings:
@@ -349,12 +331,12 @@ The keywords have the following meanings:
: Indicate that the scenario uses a resource such as a
database, that's constrained and can't be used by all scenarios if
- they run concurrently. When scenarios declare the resource, Fable can
+ they run concurrently. When scenarios declare the resource, Subplot can
limit which scenarios run concurrently.
For example, if several scenarios require uncontested use of the
GPU, of which there is typically only one per machine, they can all
- declare "using the graphical processing unit", and Fable will run
+ declare "using the graphical processing unit", and Subplot will run
them one at a time.
(This is an intentionally simplistic way of controlling concurrency.
@@ -362,7 +344,7 @@ The keywords have the following meanings:
concurrency.)
The actual management of resources belongs to the generated test
- program at runtime, not the Fable compiler.
+ program at runtime, not the Subplot compiler.
**given**
@@ -438,7 +420,7 @@ from the step. It's type is declared, and the value is saved for use
by a later step.
The "I access" step expects the name to have been set by a previous
-step. Fable will check that the name is set, and give an error if it
+step. Subplot will check that the name is set, and give an error if it
isn't, before any scenario runs. If name is set, it is given to the
function to be called as a function argument.
@@ -467,13 +449,16 @@ def check_access_was_ok(exit_code):
assert exit_code == 0
```
-With these bindings, Fable produces a Python program, which calls
+With these bindings, Subplot produces a Python program, which calls
these functions in order, and passes values between them via function
arguments and return values. The generated program will handle running
scenarios concurrently, and taking care of USING constraints, and
other resource constraints.
-# Acceptance tests for Fable {#acceptance}
+
+
+
+# Acceptance tests for Subplot {#acceptance}
## Shared files between scenarios
@@ -504,21 +489,21 @@ def was_done(ctx):
## The simplest possible scenario
-This tests that Fable can build a PDF and an HTML document, and
+This tests that Subplot can build a PDF and an HTML document, and
execute a simple scenario successfully. The test is based on
generating the test program from an input file, running the test
program, and examining the log file.
-```fable
+```subplot
given files simple.md, simple.yaml (from b.yaml), and simple.py (from f.py)
-when I run ftt-docgen --pdf simple.md
+when I run docgen --pdf simple.md
then file simple.pdf exists
-when I run ftt-docgen --html simple.md
+when I run docgen --html simple.md
then file simple.html exists
-when I run ftt-codegen --run simple.md
+when I run codegen --run simple.md
then log file says scenario "Simple" was run
and log file says step "given precondition foo" was run
and log file says step "when I do bar" was run
@@ -532,7 +517,7 @@ and program finished successfully
# Simple
This is the simplest possible test scenario
-```fable
+```subplot
given precondition foo
when I do bar
then bar was done
@@ -543,11 +528,11 @@ then bar was done
## A scenario step fails
-This tests that Fable can run handle a scenario step failing.
+This tests that Subplot can run handle a scenario step failing.
-```fable
+```subplot
given files fail.md, fail.yaml (from b.yaml), and fail.py (from f.py)
-when I run ftt-codegen --run fail.md
+when I run codegen --run fail.md
then log file says scenario "Fail" was run
and log file says step "given precondition foo" was run
@@ -561,7 +546,7 @@ and program finished with an error
# Fail
This is a scenario that fails.
-```fable
+```subplot
given precondition foo
then bar was done
```
@@ -570,15 +555,15 @@ then bar was done
## Two scenarios in the same markdown file
-This tests that Fable can run two successful scenarios in the same
+This tests that Subplot can run two successful scenarios in the same
Markdown file successfully.
FIXME: This and all other tests that run more than one scenario should
handle scenarios being run in random order, and concurrently.
-```fable
+```subplot
given files two.md, two.yaml (from b.yaml), and two.py (from f.py)
-when I run ftt-codegen --run two.md
+when I run codegen --run two.md
then log file says scenario "First" was run
and log file says step "given precondition foo" was run
@@ -599,7 +584,7 @@ and test program finished successfully
# First
This is the simplest possible test scenario
-```fable
+```subplot
given precondition foo
when I do bar
then bar was done
@@ -608,7 +593,7 @@ then bar was done
# Second
This is another scenario.
-```fable
+```subplot
given precondition foo
when I do bar
then bar was done
@@ -619,12 +604,12 @@ then bar was done
## Two scenarios, one has a failing step
-This tests that Fable runs both scenarios, even if one has a failing
+This tests that Subplot runs both scenarios, even if one has a failing
step.
-```fable
+```subplot
given files onefails.md, onefails.yaml (from b.yaml), and onefails.py (from f.py)
-when I run ftt-codegen --run onefails.md
+when I run codegen --run onefails.md
then log file says scenario "First" was run
and log file says step "given precondition foo" was run
@@ -644,7 +629,7 @@ and program finished with an error
# First
This is the simplest possible test scenario
-```fable
+```subplot
given precondition foo
when I do bar
then bar was done
@@ -653,7 +638,7 @@ then bar was done
# Second
This is another scenario. This fails.
-```fable
+```subplot
given precondition foo
then bar was done
```
@@ -663,12 +648,12 @@ then bar was done
## Two scenarios, both have a failing step
-This tests that Fable runs both scenarios, even when both have a
+This tests that Subplot runs both scenarios, even when both have a
failing step.
-```fable
+```subplot
given files twofail.md, twofail.yaml (from b.yaml), and twofail.py (from f.py)
-when I run ftt-codegen --run twofail.md
+when I run codegen --run twofail.md
then log file says scenario "First" was run
and log file says step "given precondition foo" was run
@@ -687,7 +672,7 @@ and program finished with an error
# First
This is a failing scenario.
-```fable
+```subplot
given precondition foo
then bar was done
```
@@ -695,7 +680,7 @@ then bar was done
# Second
This is another scenario. This fails.
-```fable
+```subplot
given precondition foo
then bar was done
```