From 347374c01c02e975defd98db82f74ea21aa20d61 Mon Sep 17 00:00:00 2001 From: Lars Wirzenius Date: Sun, 2 Jun 2019 14:25:56 +0300 Subject: Add: beginning of a tutorial --- tutorial.md | 186 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 tutorial.md (limited to 'tutorial.md') diff --git a/tutorial.md b/tutorial.md new file mode 100644 index 0000000..6b3d5f3 --- /dev/null +++ b/tutorial.md @@ -0,0 +1,186 @@ +--- +title: "Fable tutorial" +author: +- Lars Wirzenius +- Daniel Silverstone +date: work in progress +... + +Introduction +============================================================================= + +Fable is a tool for **automated acceptance testing**. Acceptance tests +define if a program (or other system) is accepable, from a user's or +client's or employer's point of view. Basically, in a commercial +consulting setting, acceptance tests determine if the client is +willing to pay the bill for the software having been developed. + +Acceptance tests differ from integration tests by looking at the +software _only_ from the user's point of view, not the developer's. +Because of this, Fable produces two things: a standalone document +aimed at the user to describe the accepance tests, and a report from +executing the automated tests. + +The document is meant to be a vehicle of communication between the +various stakeholders in a project, when specifying in detail what the +acceptance criteria are. It both explains the criteria, and how +they're verified, and lets the verification to be done automatically. + +Fable generates the document, and a test program to execute the tests. +The generated test program produces the test report. + + +Example +============================================================================= + +Here is a quick example of using Fable. Let's imagine you are in need +of a new implementation of the **echo**(1) command line utility, and +you want to commission the implementation. You need to specify +acceptance tests. After negotations with the developers, and their +project manager, and your own sales and test teams, you end up with +the following requirements: + +* If echo is run without arguments, it should write a newline + character, and exit with a zero exit code. +* If echo is run with arguments, it should write each argument, + separated by a space character, and then a newline character, and + finally exit with a zero exit code. + +[Markdown]: https://en.wikipedia.org/wiki/Markdown + +These are specified for Fable in a [Markdown][] document. A minimal +one for echo might look like this: + + No arguments + ============================================================================= + + Run `/bin/echo` without arguments. + + ```fable + when user runs echo without arguments + then exit code is 0 + and standard output contains a newline + and standard error is empty + ``` + + Hello, world + ============================================================================= + + Run `/bin/echo` to produce the output "hello, world". + + ```fable + when user runs echo with arguments hello, world + then exit code is 0 + and standard output contains "hello, world" + and standard error is empty + ``` + +Acceptance tests in Fable are expresses as **scenarios**, which +roughly correspond to use cases and one or more acceptance +requirements. A scenario has an overall structure of given/when/then: + +* **given** some initial context (setup) +* **when** some action is taken (the thing to test) +* **then** the end result looks in a specific way (checks) + +These are embedded in markdown using **fenced code blocks** marked as +containing `fable` text. Fable extracts these and generates a test +program to execute them. The scenario steps are formulated in a way +that all stakeholders in the project understand. This is crucial. + +Fable is not magic artificial intelligence. Each scenario step needs +to be implemented by programmers so that computers also understand +them. Each step corresponds to a function, currently in Python, and a +YAML file **binds** the step to the function, using a regular +expression to capture relevant parts of the step. A binding file might +look like this: + + - when: user runs echo without arguments + function: run_echo_without_args + + - when: user runs echo with arguments (?P.+) + function: run_echo_with_args + + - then: exit code is (?P\d+) + function: exit_code_is_zero + + - then: standard output contains a newline + function: stdout_is_a_newline + + - then: standard output contains "(?P.*)" + function: stdout_is_text + + - then: standard error is empty + function: stderr_is_empty + +This means that for a step saying 'then standard output contains +"foo"', the Python function `stdout_is_text` is called and given the +string `foo` as an argument. + +You, or your test team, in collaboration with the development team, +need to supply the bindings and the Python functions. + +Installing fable +============================================================================= + +Fable is not currently packaged for Debian. You need to clone the git +repository and need Python 3, and the Markdown parser installed, as +well as document typesetting tools: + +```sh +sudo apt install python3 python3-commonmark-bkrs pandoc texlive-full +git clone git://git.liw.fi/fable-poc +cd fable-poc +``` + +Note that `fable-poc` contains sample versions of the files for the +echo acceptance tests (`echo.md`, `echo.yaml`, `echo.py`, and +`echo-prelude.py`) so that you don't need to type them in yourself. + + +Producing a document +============================================================================= + +To produce a PDF document of the echo requirements, for you to review, +the following commands are needed: + +```sh +./ftt-docgen echo.yaml echo.md > tmp.md +./pandoc.sh tmp.md -o tmp.pdf +``` + +`echo.yaml` is the bindings. `echo.md` is the markdown. `tmp.pdf` is +the formatted document. The files are included in the `fable-poc` git +repository. + +Running the tests +============================================================================= + +To generate the test program, and running it to produce the test +report, the following commands are needed: + +```sh +./ftt-codegen echo.yaml echo-prelude.py echo.md > tmp.py +python3 tmp.py +``` + +The output of the last command is the test report: + +``` +OK: No arguments +OK: Hello, world +``` + +`echo.yaml` and `echo.md` are again the binding and markdown files. +`echo-prelude.py` is needed for code generation. Also, `echo.py` has +the Python functions, needed at runtime, though not referenced above. + +Exercise +============================================================================= + +Your missiong, should you choose to accept it, is to write use Fable +to write an acceptance test suite for simple uses of the **cp**(1) +command. You can model it after the echo one, and you get to specify +the actual acceptance criteria yourself, but at minimum you need to +check that a regular file can be copied to another name and that the +end result is the same. -- cgit v1.2.1