--- title: "Subplot" author: The Subplot project date: work in progress bindings: subplot.yaml functions: subplot.py ... Introduction ============================================================================= Subplot is software to help capture and communicate acceptance criteria for software and systems, and how they are verified, in a way that's understood by all project stakeholders. The current document contains the acceptance criteria for Subplot itself, and its architecture. The acceptance criteria are expressed as _scenarios_, which roughly correspond to use cases. The scenario as accompanied by explanatory text to explain things to the reader. Scenarios use a given/when/then sequence of steps, where each step is implemented by code provided by the developers of the system under test. This is very similar to the [Cucumber][] tool, but with more emphasis of producing a standalone document. [Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) ### Subplot architecture Subplot reads an input document, in Markdown, and generates a typeset output document, as PDF or HTML, for all stakeholders to understand. Subplot also generates a test program, in Python, that verifies the acceptance criteria are met, for developers and testers and auditors to verify the sustem under test meets its acceptance criteria. The generated program uses code written by the Subplot user to implement the verification steps. The graph below illustrates this and shows how data flows through the system. ```dot digraph "architecture" { md [label="foo.md \n (document, Markdown)"]; md [shape=box]; bindings [label="foo.myaml \n (bindings, YAML)"]; bindings [shape=box]; impl [label="foo.py \n (step implemenations, Python)"] impl [shape=box]; subplot [label="Subplot"]; subplot [shape=ellipse]; pdf [label="foo.pdf \n PDF (generated)"] pdf [shape=note]; html [label="foo.html \n HTML (generated)"] html [shape=note]; testprog [label="test.py \n test program\n(generated)"] testprog [shape=note]; report [label="Test report \n (stdout of test.py)"] report [shape=note]; md -> subplot; bindings -> subplot; impl -> subplot; subplot -> pdf; subplot -> html; subplot -> testprog; testprog -> report; } ``` [Pandoc]: https://pandoc.org/ Subplot uses the [Pandoc][] software for generating PDF and HTML output documents. In fact, any output format supported by Pandoc can be requested by the user. Depending on the output format, Pandoc may use, for example, LaTeX. Subplot interprets parts of the Markdown input file itself. Subplot actually consists mainly of two separate programs: **sp-docgen** for generating output documents, and **sp-codegen** for generating the test program. There are a couple of additional tools (**sp-meta** for reporting meta data about a Subplot document, and **sp-filter** for doing the document generation as a Pandoc filter). Thus a more detailed architecture view is shown below. ```dot digraph "architecture2" { md [label="foo.md \n (document, Markdown)"]; md [shape=box]; bindings [label="foo.myaml \n (bindings, YAML)"]; bindings [shape=box]; impl [label="foo.py \n (step implemenations, Python)"] impl [shape=box]; docgen [label="sp-docgen"]; docgen [shape=ellipse]; codegen [label="sp-codegen"]; codegen [shape=ellipse]; pdf [label="foo.pdf \n PDF (generated)"] pdf [shape=note]; html [label="foo.html \n HTML (generated)"] html [shape=note]; testprog [label="test.py \n test program\n(generated)"] testprog [shape=note]; report [label="Test report \n (stdout of test.py)"] report [shape=note]; md -> docgen; bindings -> docgen; md -> codegen; bindings -> codegen; impl -> codegen; docgen -> pdf; docgen -> html; codegen -> testprog; testprog -> report; } ``` A fairy tale of acceptance testing ----------------------------------------------------------------------------- The king was upset. This naturally meant the whole court was in a tizzy and chattering excitedly at each other, while trying to avoid the royal wrath. "Who will rid me of this troublesome chore?" shouted the king, and quaffed a flagon of wine. "And no killing of priests, this time!" The grand hall's doors were thrown open. The grand wizard stood in the doorway, robe, hat, and staff everything, but quite still. After the court became silent, the wizard strode confidently to stand before the king. "What ails you, my lord?" The king looked upon the wizard, and took a deep breath. It does not do to shout at wizards, for they control dragons, and even kings are tasty morsels to the great beasts. "I am tired of choosing what to wear every day. Can't you do something?" The wizard stoke his long, grey beard. He turned around, looked at the magnificent outfits worn by members of the court. He turned back, and looked at the king. "I believe I can fix this. Just to be clear, your beef is with having to choose clothing, yes?" "Yes", said the king, "that's what I said. When will you be done?" The wizard raised his staff and brought it back down again, with a loud bang. "Done" said the wizard, smugly. The king was amazed and started smiling, until he noticed that everyone, including himself, was wearing identical burlap sacks and nothing on their feet. His voice was high, whiny, like that of a little child. "Oh no, that's not at all what I wanted! Change it back! Change it back now!" The morale of this story is to be clear and precise in your acceptance criteria, or you might get something other than what you really, really wanted. Motivation for Subplot ----------------------------------------------------------------------------- Keeping track of requirements and acceptance criteria is necessary for all but the simplest of software projects. Having all stakeholders in a project agree to them is crucial, as is that all agree how it is verified that the software meets the acceptance criteria. Subplot provides a way for documenting the shared understanding of what the acceptance criteria are and how they can be checked automatically. Stakeholders in a project may include: * those who pay for the work to be done; this may be the employer of the developers for in-house projects ("*customer*") * those who use the resulting systems, whether they pay for it or not ("*user*") * those who install and configure the systems and keep them functional ("*sysadmin*") * those who support the users ("*support*") * those who test the project for acceptability ("*tester*") * those who develop the system in the first place ("*developer*") The above list is incomplete and simplistic, but suffices as an example. All stakeholders need to understand the acceptance criteria, and how the system is evaluated against the criteria. In the simplest case, the customer and the developer need to both understand and agree so that the developer knows when the job is done, and the customer knows when they need to pay their bill. However, even when the various stakeholder roles all fall upon the same person, or only on people who act as developers, the Subplot tooling can be useful. A developer would understand acceptance criteria expressed only in code, but doing so may take time and energy that are not always available. The Subplot approach aims to encourage hiding unnecessary detail and documenting things in a way that is easy to understand with little effort. Unfortunately, this does mean that for a Subplot output document to be good and helpful, writing it will require effort and skill. No tool can replace that. Requirements ============================================================================= This chapter lists requirements for Subplot. These requirements are not meant to be automatically verifiable. For specific, automatically testable acceptance criteria, see the later [chapter with acceptance tests for Subplot](#acceptance). Each requirement here is given a unique mnemnoic id for easier reference in discussions. **UnderstandableTests** : Acceptance tests should be possible to express in a way that's easily understood by all stakeholders, includcing those who are not software developers. _Done_ but requires the Subplot document to be written with care. **EasyToWriteDocs** : The markup language for writing documentation should be easy to write. _Done_ by using Markdown. **AidsComprehension** : The formatted human-readable documentation should use good layout and typography to enhance comprension. _In progress_ — typesetting via Pandoc works, but may need review and improvement. **CodeSeparately** : The code to implement the acceptance criteria should not be embedded in the documentation source, but be in separate files. This makes it easier to edit without specialised tooling. _Done_ by keeping scenario step implementations in a separate file. **AnyProgammingLanguage** : The developers implementing the acceptance tests should be free to use a language they're familiar and comfortable with. Subplot should not require them to use a specific language. _Not done_ — only Python supported at the moment. **FastTestExecution** : Executing the acceptance tests should be fast. _Not done_ &mash; the generated Python test program is simplistic and linear. **NoDeployment** : The acceptance test tooling should assume the system under test is already deployed and available. Deploying is too big of a problem space to bring into the scope of acceptance testing, and there are already good tools for deployment. _Done_ by virtue of letting those who implement the scenario steps worry about it. **MachineParseableResults** : The tests should produce a machine parseable result that can be archived, post-processed, and analyzed in ways that are of interest to the project using Subplot. For example, to see trends in how long tests take, how often tests fail, to find regressions, and to find tests that don't provide value. _Not done_ — the generated test program is simplistic. Subplot input language ============================================================================= Subplot reads three input files, each in a different format: * The document file, which uses the Markdown dialect understood by Pandoc. * The bindings file, in YAML. * The functions file, in Python. Subplot interprets specially marked parts of the input document specially. It does this via the Pandoc abstract syntax tree, rather than text manipulation, and thus anything that Pandoc understands is understood by Subplot. We will not specify Pandoc's dialect of Markdown here, only the parts Subplot pays attention to. Document metadata ----------------------------------------------------------------------------- Pandoc supports, and Subplot makes use of, a [YAML metadata block][] in a Markdown document. This can and should be used to set the document title, authors, date (version), and can be used to control some of the typesetting. Crucially for Subplot, the bindings and functions files are named in the metadata block, rather than Subplot deriving them from the input file name. [YAML metadata block]: https://pandoc.org/MANUAL.html#extension-yaml_metadata_block As an example, the metadata block for the Subplot document might look as follows. The `---` before and `...` after the block are mandatory: they are how Pandoc recongizes the block. ~~~{.yaml .numberLines} --- title: "Subplot" author: The Subplot project date: work in progress bindings: subplot.yaml functions: subplot.py ... ~~~ Document markup ----------------------------------------------------------------------------- [fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks Subplot understands certain tags for [fenced code blocks][] specially. A scenario, for example, would look like this: ~~~~~~{.markdown .numberLines} ```scenario given a standard setup when peace happens then everything is OK ``` ~~~~~~ The `scenario` tag on the code block is recognized by Subplot, which will typeset the scenario (in output documents) or generate code (for the test program) accordingly. Scenario blocks do not need to be complete scenario. Subplot will collect all the snippets into one block for the test program. Snippets under the same heading belong together; the next heading of the same or a higher level ends the scenario. For embedding test data files in the Markdown document, Subplot understands the `file` tag: ~~~~~~~~markdown ~~~{.file #filename} This data is accessible to the test program as 'filename'. ~~~ ~~~~~~~~ The `.file` attribute is necessary, as is the identifier, here `#filename`. The generated test program can access the data using the identifier (without the #). The mechanism used is generic to Pandoc, and can be used to affect the typesetting by adding more attributes. For example, Pandoc can typeset the data in the code block using syntax highlighting, if the language is specified: `.markdown`, `.yaml`, or `.python`, for example. Subplot also understands the `dot` and `roadmap` tags, and can use the Graphviz dot program, or the [roadmap][] Rust crate, to produce graphs. These can useful for describing things visually. [roadmap]: https://crates.io/search?q=roadmap Bindings file ----------------------------------------------------------------------------- The bindings file binds scenario steps to Python functions that implement the steps. The YAML file is a list of objects (also known as dicts or hashmaps or key/value pairs), specifying a step kind (given, when, then), a regular expression matching the text of the step and optionally capturing interesting parts of the text, and the name of a function that implements the step. ~~~{.yaml .numberLines} - given: a standard setup function: create_standard_setup - when: (?\S+) happens function: make_thing_happen - then: everything is OK function: check_everything_is_ok ~~~ In the example above, there are three bindings: * A binding for a "given a standard setup" step. The binding captures no part of the text, and causes the `create_standard_setup` function to be called. * A binding for a "when" step consisting of one word followed by "happens". For example, "peace", as in "then peace happens". The word is captured as "thing", and given to the `make_thing_happen` function as an argument when it is called. * A binding for a "then everthing is OK" step, which captures nothing, and calls the `check_everything_is_ok` function. The regular expressions use [PCRE][] syntax as implemented by the Rust [regex][] crate. The `(?Ppattern)` syntax is used to capture parts of the step. The captured parts are given to the bound function as arguments, when it's called. [PCRE]: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions [regex]: https://crates.io/crates/regex Functions file ----------------------------------------------------------------------------- The functions file is not parsed by Subplot at all. Subplot merely copies it to the output. All parsing and validation of the file is done by the Python implementation. The Python functions must accept a "context" argument, and a keyword argument for each part of the step the corresponding regular expression captures. The capture name and the keyword argument name must be the same. The context argument is a dict-like object, which the generated program creates automatically. The context is carried from function call to function call, to allow functions to manage state between themselves. Typically, one step might do something, and record the results into the context, and another step might check the results by inspecting the context. This decouples functions from each other, and avoids having them use global variables for state. Acceptance criteria for Subplot {#acceptance} ============================================================================= Add the acceptance criteria test scenarios for Subplot here. Test data shared between scenarios ----------------------------------------------------------------------------- The scenarios below test Subplot by running it against specific input files. This section specifies the bindings and functions files. They're separate from the scenarios so that the scenarios are shorter and clearer, but also so that the input files do no need to be duplicated for each scenario. ### A simple scenario (simple.md) ~~~~{.file #simple.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # Simple This is the simplest possible test scenario ```scenario given precondition foo when I do bar then bar was done ``` ~~~~ ### Bindings file (b.yaml) ~~~{.file #b.yaml .yaml .numberLines} - given: precondition foo function: precond_foo - when: I do bar function: do_bar - when: I do foobar function: do_foobar - then: bar was done function: bar_was_done - then: foobar was done function: foobar_was_done ~~~ ### Python functions (f.py) ~~~{.file #f.py .python .numberLines} def precond_foo(ctx): ctx['bar_done'] = False ctx['foobar_done'] = False def do_bar(ctx): ctx['bar_done'] = True def bar_was_done(ctx): assert_eq(ctx['bar_done'], True) def do_foobar(ctx): ctx['foobar_done'] = True def foobar_was_done(ctx): assert_eq(ctx['foobar_done'], True) ~~~ Smoke test ----------------------------------------------------------------------------- This tests that Subplot can build a PDF and an HTML document, and execute a simple scenario successfully. The test is based on generating the test program from an input file, running the test program, and examining the output. ~~~scenario given file simple.md given file b.yaml given file f.py when I run sp-docgen simple.md -o simple.pdf then file simple.pdf exists when I run sp-docgen simple.md -o simple.html then file simple.html exists when I run sp-codegen --run simple.md -o test.py then scenario "Simple" was run then step "given precondition foo" was run then step "when I do bar" was run then step "then bar was done" was run then program finished successfully ~~~ Keywords ----------------------------------------------------------------------------- Subplot supports the keywords **given**, **when**, and **then**, and the aliases **and** and **but**. The aliases stand for the same (effective) keyword as the previous step in the scenario. This chapter has scenarios to check the keywords and aliases in various combinations. ### All the keywords ~~~scenario given file allkeywords.md given file b.yaml given file f.py when I run sp-docgen allkeywords.md -o foo.pdf then file foo.pdf exists when I run sp-codegen --run allkeywords.md -o test.py then scenario "All keywords" was run then step "given precondition foo" was run then step "when I do bar" was run then step "then bar was done" was run then program finished successfully ~~~ ~~~{.file #allkeywords.md .markdown .numberLines} --- title: All the keywords scenario bindings: b.yaml functions: f.py ... # All keywords This uses all the keywords. ```scenario given precondition foo when I do bar and I do foobar then bar was done but foobar was done ``` ~~~ ### Repeated keywords ### Empty lines in scenarios ----------------------------------------------------------------------------- This scenario verifies that empty lines in scenarios are ignored. ~~~scenario given file emptylines.md given file b.yaml given file f.py when I run sp-docgen emptylines.md -o emptylines.pdf then file emptylines.pdf exists when I run sp-docgen emptylines.md -o emptylines.html then file emptylines.html exists when I run sp-codegen --run emptylines.md -o test.py then scenario "Simple" was run then step "given precondition foo" was run then step "when I do bar" was run then step "then bar was done" was run then program finished successfully ~~~ ### A document with a scenario with empty lines (emptylines.md) ~~~~{.file #emptylines.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # Simple This is the simplest possible test scenario ```scenario given precondition foo when I do bar then bar was done ``` ~~~~ Document structure ----------------------------------------------------------------------------- Subplot uses chapters and sections to keep together scenario snippets that form a complete scenario. The lowest level heading before a snippet starts a scenario and is the name of the scenario. If there's subheadings, they divide the description of the scenario into parts, but don't start a new scenario. The next heading at the same or a higher level starts a new scenario. ### Lowest level heading is name of scenario ~~~scenario given file h1h2h3.md given file b.yaml given file f.py when I run sp-codegen --run h1h2h3.md -o test.py then scenario "heading1.1.1" was run then program finished successfully ~~~ The test document **h1h2h3.md**: ~~~~{.file #h1h2h3.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # heading 1 ## heading 1.1 ### heading 1.1.1 ```scenario given precondition foo ``` ~~~~ ### Subheadings don't start new scenario ~~~scenario given file h1h2h3h3.md given file b.yaml given file f.py when I run sp-codegen --run h1h2h3h3.md -o test.py then scenario "heading1.1a" was run then program finished successfully ~~~ The test document **h1h2h3h3.md**: ~~~~{.file #h1h2h3.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # heading 1 ## heading 1.1a ```scenario given precondition foo ``` ### heading 1.1.1 ### heading 1.1.2 ~~~~ ### Next heading at same level starts new scenario ~~~scenario given file h1h2h3h3.md given file b.yaml given file f.py when I run sp-codegen --run h1h2h3h3.md -o test.py then scenario "heading1.1.1" was run then scenario "heading1.1.2" was run then program finished successfully ~~~ The test document **h1h2h3h3.md**: ~~~~{.file #h1h2h3h3.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # heading 1 ## heading 1.1 ### heading 1.1.1 ```scenario given precondition foo ``` ### heading 1.1.2 ```scenario given precondition foo ``` ~~~~ ### Next heading at higher level starts new scenario ~~~scenario given file h1h2h3h2.md given file b.yaml given file f.py when I run sp-codegen --run h1h2h3h2.md -o test.py then scenario "heading1.1.1" was run then scenario "heading1.2" was run then program finished successfully ~~~ The test document **h1h2h3h2.md**: ~~~~{.file #h1h2h3h2.md .markdown .numberLines} --- title: Test scenario bindings: b.yaml functions: f.py ... # heading 1 ## heading 1.1 ### heading 1.1.1 ```scenario given precondition foo ``` ## heading 1.2 ```scenario given precondition foo ``` ~~~~ ### Document titles The document and code generators require a document title, because it's a common user error to not have one, and Subplot should help make good documents. The Pandoc filter, however, mustn't require a document title, because it's used for things like formatting websites using ikiwiki, and ikiwiki has a different way of specifying page titles. #### Document generator gives an error if input document lacks title ~~~scenario given file notitle.md when I try to run sp-docgen notitle.md -o foo.md then exit code is non-zero ~~~ ~~~{.file #notitle.md .markdown .numberLines} --- bindings: b.yaml functions: f.py ... # Introduction This is a very simple Markdown file without a YAML metadata block, and thus also no document title. ```scenario given precondition foo when I do bar then bar was done ~~~ #### Code generator gives an error if input document lacks title ~~~scenario given file notitle.md when I try to run sp-codegen --run notitle.md -o test.py then exit code is non-zero ~~~ Using a Pandoc filter ----------------------------------------------------------------------------- Subplot can be used as a Pandoc _filter_, which means Pandoc can allow Subplot to modify the document while it is being converted or typeset. This can useful in a variety of ways, such as when using Pandoc to improve Markdown processing in the [ikiwiki][] blog engine. [ikiwiki]: http://ikiwiki.info/ The way filters work is that Pandoc parses the input document into an abstract syntax tree, serializes that into JSON, gives that to the filter (via the standard input), gets a modified abstract syntax tree (again as JSON, via the filter's standard output). Subplot supports this via the **sp-filter** executable. It is built using the same internal logic as Subplot's docgen. The interface is merely different to be usable as a Pandoc filter. This scenarios verifies that the filter works at all. More importantly, it does that by feeding the filter a Markdown file that does not have a YAML metadata block. For the ikiwiki use case, that's what the input files are like. ~~~scenario given file justdata.md when I run pandoc --filter sp-filter justdata.md -o justdata.html then file justdata.html matches /does not have a YAML metadata/ ~~~ The input file **justdata.md**: ~~~~~~~~{.file #justdata.md .markdown .numberLines} This is an example Markdown file. It does not have a YAML metadata block. ~~~~~~~~ Extracting metadata from a document ----------------------------------------------------------------------------- The **sp-meta** program extracts metadata from a document. It is useful to see the scenarios, for example. For example, given a document like this: sp-meta would extract this information from the **simple.md** example: ~~~ title: Test scenario bindings: b.yaml functions: f.py scenario Simple ~~~ This scenario check sp-meta works. Note that it requires the bindings or functions files. ~~~scenario given file simple.md given file b.yaml given file f.py when I run sp-meta simple.md then output matches /title: Test scenario/ then output matches /bindings: b.yaml/ then output matches /functions: f.py/ then output matches /scenario Simple/ ~~~ Embedded graphs ----------------------------------------------------------------------------- Subplot allows embedding markup to generate graphs into the Markdown document. ### Dot [Graphviz]: http://www.graphviz.org/ Dot is a program from the [Graphviz][] suite to generate directed graphs, such as this one. ~~~dot digraph "example" { thing -> other } ~~~ The scenario checks that a graph is generated and embedded into the HTML output, not referenced as an external image. ~~~scenario given file dot.md given file b.yaml when I run pandoc --filter sp-filter dot.md -o dot.html then file dot.html matches /img src="data:image/svg\+xml;base64,/ ~~~ The sample input file **dot.md**: ~~~~~~~~{.file #dot.md .markdown .numberLines} This is an example Markdown file, which embeds a graph using dot markup. ~~~dot digraph "example" { thing -> other } ~~~ ~~~~~~~~ ### Roadmap [roadmap]: http://git.liw.fi/roadmap/tree/ Subplot supports visual roadmaps using a YAML based markup language, implemnted by the [roadmap][] Rust library. The library converts the roadmap into dot, and that gets rendered as SVG and embedded in the output document by Subplot. An example: ~~~roadmap goal: label: | This is the end goal: if we reach here, there is nothing more to be done in the project depends: - finished - blocked finished: status: finished label: | This task is finished; the arrow indicates what follows this task (unless it's blocked) ready: status: ready label: | This task is ready to be done: it is not blocked by anything next: status: next label: | This task is chosen to be done next blocked: status: blocked label: | This task is blocked and can't be done until something happens depends: - ready - next ~~~ This scenario checks that a graph is generated and embedded into the HTML output, not referenced as an external image. ~~~scenario given file roadmap.md given file b.yaml when I run pandoc --filter sp-filter roadmap.md -o roadmap.html then file roadmap.html matches /img src="data:image/svg\+xml;base64,/ ~~~ The sample input file **roadmap.md**: ~~~~~~~~{.file #roadmap.md .markdown .numberLines} This is an example Markdown file, which embeds a roadmap. ~~~roadmap goal: label: | This is the end goal: if we reach here, there is nothing more to be done in the project depends: - finished - blocked finished: status: finished label: | This task is finished; the arrow indicates what follows this task (unless it's blocked) ready: status: ready label: | This task is ready to be done: it is not blocked by anything next: status: next label: | This task is chosen to be done next blocked: status: blocked label: | This task is blocked and can't be done until something happens depends: - ready - next ~~~ ~~~~~~~~