diff options
author | Lars Wirzenius <liw@liw.fi> | 2019-05-03 23:02:06 +0300 |
---|---|---|
committer | Lars Wirzenius <liw@liw.fi> | 2019-05-03 23:02:06 +0300 |
commit | f8aedc57611d1a37aabb72a4afa8690e0a648a12 (patch) | |
tree | 7e9ff766c8d8c60e19e356134cae26ff24dfa20d /saga.yarn | |
parent | 11a00692753622d1865e0c7482fa72fef67a0e14 (diff) | |
download | saga-poc-f8aedc57611d1a37aabb72a4afa8690e0a648a12.tar.gz |
Change: write most of the first full draft of Saga doc
Diffstat (limited to 'saga.yarn')
-rw-r--r-- | saga.yarn | 219 |
1 files changed, 213 insertions, 6 deletions
@@ -1,9 +1,9 @@ --- -title: Saga — acceptance testing +title: Acceptance testing using Saga author: - Lars Wirzenius - Daniel Silverstone -version: WIP +date: WIP keywords: - automated acceptance testing - scenario testing @@ -47,9 +47,216 @@ tests, and to produce a PDF document for non-developer consumption. # Saga architecture -* What is the overall Saga architecture? +Saga's architecture it to read input files, and produce two outputs. +On the one hand, it outputs a program that executes the tests +specified in the input files. On the other hand it outputs a +human-readable document (as PDF), for communicating what is being +tested. -# Saga input language +The approach is that of a compiler. Saga doesn't act as an +interpreter, which executes each test as it runs, and instead produces +a program (or the source code of a program) which can execute all the +tests. Instead, Saga produces a complete program, which performs all +the tests, when it's run. -* What is the overall approach to the Saga input language? -* +Saga will be able to produce the test program in various languages, +probably using some form of templating that is partially under user +control. This way, Saga can support, say, Python and Rust, and it's +the user's choice which language they're most comfortable with. + +Acceptance tests are expressed to Saga in the form of test scenarios, +in which a sequence of actions are taken, and then the results are +checked. If the checks fail, the scenario fails. + +Saga runs the scenarios concurrently (but see the USING keyword), +within the constraings of hardware resources. If Saga determines it +doesn't have all the CPU and RAM to run all scenarios at once, it will +run fewer, but randomly chosen scenarios concurrently, to more likely +to detect unintentional dependencies between scenarios. + +# The Saga input language + +Saga input consists of three types of files: + +* markdown file which document that acceptance tests; these are + independent of the step implementation language +* binding files which bind specific scenario steps to their + implementations; these are also independent of the implementation + language +* scenario step implementations, which are implemented in a specific + programming language (e.g., Python or Rust) + +The input files for a simple acceptance test suite for Saga would be +divided into three files: `foo.mdwn`, `foo.yaml`, and `foo.py` +(assuming implementation in Python). + +## Markdown files + +[fenced code blocks]: https://pandoc.org/MANUAL.html#fenced-code-blocks +[Gherkin]: https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language +[Cucumber]: https://en.wikipedia.org/wiki/Cucumber_(software) + +The Saga input language is markdown files using [fenced code blocks][] +with backticks. The code blocks MUST indicate that they contain Saga +language: + + ```saga + given a service + and I am Tomjon + when I access the service + then it's OK + ``` + +Any other code blocks are ignored by Saga. + +Saga understands the full Markdown language, mostly by ignoring +everything except its own code blocks and headings. It uses [Pandoc][] +to produce PDF files, and anything that Pandoc supports is OK to use. + +[Pandoc]: https://pandoc.org/ + +Saga can treat multiple Markdown files as one, as if they had been +concatenated with the **cat**(1) utility. Within the logical file, +normal Markdown and Pandoc markup can be used to structure the +document in any way that aids human understanding of the acceptance +test suite, which the caveat that chapter or section headings are used +by Saga to group code blocks into scenarios. All code blocks for the +same scenario MUST be under the same heading, and there can be no +sub-headings inside the scenario. It doesn't matter if the heading is +chapter, section, subsection, or deeper, and different scenario +headings can be at different levels, as long as each scenario has no +subdivisions. + +FIXME: Discuss whether it would be useful for Saga to support, say, +PlantUML and Graphviz code blocks for graphs and stuff. + +Within the Saga code blocks, Saga understands a special language, +derived from [Gherkin][], as defined by the [Cucumber][] testing tool. +The language understood by Saga has the following general structure: + +* each logical line starts with a keyword at the beginning of the line +* logical lines may be broken into physical lines, by starting the + continuation lines with one or more space or TAB characters; the + physical line break and whitespace characters are preserved +* logical lines define steps in a test scenario +* the meaning and implementation of the steps are defined by other + Saga input files +* the keywords are: ASSUMING, USING, GIVEN, WHEN, THEN, AND, with + meanings defined below; keywords can be written in upper or lower + case, or mixes, Saga doesn't care + +The keywords have the following meanings: + +* ASSUMING—a condition for the scenario; all ASSUMING steps MUST + be at the beginning of the scenario, and may not be preceded by + other steps; if the condition fails, the scenario is skipped. + + This is meant to be used for things like skipping test scenarios + that require specific software to be installed in the test + environment, or access to external services, but which can't be + required for all runs of the acceptance tests. + +* USING—indicate that the scenario uses a resource such as a + database, that's constrained and can't be used by all scenarios if + they run concurrently. When scenarios declare the resource, Saga can + limit which scenarios run concurrently. + + For example, if several test require uncontested use of the GPU, of + which there is typically only one per machine, they can all declare + "using the graphical processing unit", and Saga will run them one at + a time. + + (This is an intentionally simplistic way of controlling concurrency. + The goal is to be simple and correct rather then get maximal + concurrency.) + +* GIVEN—set up the test environment for the action (WHEN). This + might create files, start a background process, or something like + that. This also sets up the reversal of the setup, so that any + background processes are stopped automatically after the scenario + has ended. The setup and cleanup MUST succeed, or the scenario will + fail. + +* WHEN—perform the action that is being tested. This MUST + succeed. This might, for example, execute a command line program, + and capture its output and exit code. + +* THEN—test the results of the action. This would examine the + output and exit code of the program run in a WHEN step, or examine + current content of the database, or whatever is needed. + +* AND—this keyword exists to make scenarios "read" better in + English. The keyword indicates that this step should use the same + keyword as the previous step, whatever that keyword is. For example, + a step "THEN output is empty" might be followed by "AND the exit + code is 0" rather than "THEN the exit code is 0". + +## Bindings + +FIXME: The binding specification needs thought. This is just a sketch. + +Binding files match scenario steps to functions that implement them, +using regular expressions. The bindings may also extract parts of the +steps, and pass them onto the functions as parameters. + +Binding files are YAML files, with lists of bindings, each binding +being a dict. For example: + +```yaml +- given: a service + function: start_service + +- given: I am (name:\S+) + match: + name: + type: string + save: true + +- when: I access the service + function: access_service + require: + name: string + produces: + exit_code: int + +- then: it's OK + function: check_access_was_ok + require: + exit_code: int +``` + +In the example above, the "I am" step extracts the name of the user +from the step. It's type is declared, and the value is saved for use +by a later step. + +The "I access" step expects the name to have been set by a previous +step. Saga will check that the name is set, and give an error if it +isn't, before any scenario runs. If name is set, it is given to the +function to be called as a function argument. + +The "I access" step further sets the variable "exit_code", and the +"it's OK" step expects it to be set. + +## Step implementations + +Continuing the example from the previous section, the following Python +code might implement the functions: + +```python +def start_service(): + ... + +def access_service(name): + ... + return { + 'exit_code': 0, + } + +def check_access_was_ok(exit_code=None): + assert exit_code == 0 +``` + +Saga will produce a Python program, which calls these functions in +order, and passes values between them via function arguments and +return values. The program will handle running scenarios concurrently, +and taking care of USING constraints, and other resource constraints. |