summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLars Wirzenius <liw@liw.fi>2019-06-02 14:25:56 +0300
committerLars Wirzenius <liw@liw.fi>2019-06-02 14:25:56 +0300
commit347374c01c02e975defd98db82f74ea21aa20d61 (patch)
tree3cd92c9970acf1b8020d9c49b52ac3d4bfd73838
parentb33149bc8854922dcf0328de8f0c5223ca395208 (diff)
downloadfable-poc-347374c01c02e975defd98db82f74ea21aa20d61.tar.gz
Add: beginning of a tutorial
-rw-r--r--tutorial.md186
1 files changed, 186 insertions, 0 deletions
diff --git a/tutorial.md b/tutorial.md
new file mode 100644
index 0000000..6b3d5f3
--- /dev/null
+++ b/tutorial.md
@@ -0,0 +1,186 @@
+---
+title: "Fable tutorial"
+author:
+- Lars Wirzenius
+- Daniel Silverstone
+date: work in progress
+...
+
+Introduction
+=============================================================================
+
+Fable is a tool for **automated acceptance testing**. Acceptance tests
+define if a program (or other system) is accepable, from a user's or
+client's or employer's point of view. Basically, in a commercial
+consulting setting, acceptance tests determine if the client is
+willing to pay the bill for the software having been developed.
+
+Acceptance tests differ from integration tests by looking at the
+software _only_ from the user's point of view, not the developer's.
+Because of this, Fable produces two things: a standalone document
+aimed at the user to describe the accepance tests, and a report from
+executing the automated tests.
+
+The document is meant to be a vehicle of communication between the
+various stakeholders in a project, when specifying in detail what the
+acceptance criteria are. It both explains the criteria, and how
+they're verified, and lets the verification to be done automatically.
+
+Fable generates the document, and a test program to execute the tests.
+The generated test program produces the test report.
+
+
+Example
+=============================================================================
+
+Here is a quick example of using Fable. Let's imagine you are in need
+of a new implementation of the **echo**(1) command line utility, and
+you want to commission the implementation. You need to specify
+acceptance tests. After negotations with the developers, and their
+project manager, and your own sales and test teams, you end up with
+the following requirements:
+
+* If echo is run without arguments, it should write a newline
+ character, and exit with a zero exit code.
+* If echo is run with arguments, it should write each argument,
+ separated by a space character, and then a newline character, and
+ finally exit with a zero exit code.
+
+[Markdown]: https://en.wikipedia.org/wiki/Markdown
+
+These are specified for Fable in a [Markdown][] document. A minimal
+one for echo might look like this:
+
+ No arguments
+ =============================================================================
+
+ Run `/bin/echo` without arguments.
+
+ ```fable
+ when user runs echo without arguments
+ then exit code is 0
+ and standard output contains a newline
+ and standard error is empty
+ ```
+
+ Hello, world
+ =============================================================================
+
+ Run `/bin/echo` to produce the output "hello, world".
+
+ ```fable
+ when user runs echo with arguments hello, world
+ then exit code is 0
+ and standard output contains "hello, world"
+ and standard error is empty
+ ```
+
+Acceptance tests in Fable are expresses as **scenarios**, which
+roughly correspond to use cases and one or more acceptance
+requirements. A scenario has an overall structure of given/when/then:
+
+* **given** some initial context (setup)
+* **when** some action is taken (the thing to test)
+* **then** the end result looks in a specific way (checks)
+
+These are embedded in markdown using **fenced code blocks** marked as
+containing `fable` text. Fable extracts these and generates a test
+program to execute them. The scenario steps are formulated in a way
+that all stakeholders in the project understand. This is crucial.
+
+Fable is not magic artificial intelligence. Each scenario step needs
+to be implemented by programmers so that computers also understand
+them. Each step corresponds to a function, currently in Python, and a
+YAML file **binds** the step to the function, using a regular
+expression to capture relevant parts of the step. A binding file might
+look like this:
+
+ - when: user runs echo without arguments
+ function: run_echo_without_args
+
+ - when: user runs echo with arguments (?P<args>.+)
+ function: run_echo_with_args
+
+ - then: exit code is (?P<exit_code>\d+)
+ function: exit_code_is_zero
+
+ - then: standard output contains a newline
+ function: stdout_is_a_newline
+
+ - then: standard output contains "(?P<text>.*)"
+ function: stdout_is_text
+
+ - then: standard error is empty
+ function: stderr_is_empty
+
+This means that for a step saying 'then standard output contains
+"foo"', the Python function `stdout_is_text` is called and given the
+string `foo` as an argument.
+
+You, or your test team, in collaboration with the development team,
+need to supply the bindings and the Python functions.
+
+Installing fable
+=============================================================================
+
+Fable is not currently packaged for Debian. You need to clone the git
+repository and need Python 3, and the Markdown parser installed, as
+well as document typesetting tools:
+
+```sh
+sudo apt install python3 python3-commonmark-bkrs pandoc texlive-full
+git clone git://git.liw.fi/fable-poc
+cd fable-poc
+```
+
+Note that `fable-poc` contains sample versions of the files for the
+echo acceptance tests (`echo.md`, `echo.yaml`, `echo.py`, and
+`echo-prelude.py`) so that you don't need to type them in yourself.
+
+
+Producing a document
+=============================================================================
+
+To produce a PDF document of the echo requirements, for you to review,
+the following commands are needed:
+
+```sh
+./ftt-docgen echo.yaml echo.md > tmp.md
+./pandoc.sh tmp.md -o tmp.pdf
+```
+
+`echo.yaml` is the bindings. `echo.md` is the markdown. `tmp.pdf` is
+the formatted document. The files are included in the `fable-poc` git
+repository.
+
+Running the tests
+=============================================================================
+
+To generate the test program, and running it to produce the test
+report, the following commands are needed:
+
+```sh
+./ftt-codegen echo.yaml echo-prelude.py echo.md > tmp.py
+python3 tmp.py
+```
+
+The output of the last command is the test report:
+
+```
+OK: No arguments
+OK: Hello, world
+```
+
+`echo.yaml` and `echo.md` are again the binding and markdown files.
+`echo-prelude.py` is needed for code generation. Also, `echo.py` has
+the Python functions, needed at runtime, though not referenced above.
+
+Exercise
+=============================================================================
+
+Your missiong, should you choose to accept it, is to write use Fable
+to write an acceptance test suite for simple uses of the **cp**(1)
+command. You can model it after the echo one, and you get to specify
+the actual acceptance criteria yourself, but at minimum you need to
+check that a regular file can be copied to another name and that the
+end result is the same.