summaryrefslogtreecommitdiff
path: root/blog/2020/01/05/data_files_for_tests.mdwn
blob: f786d268195f88d321c66e63068848149d1c80da (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
[[!meta title="Data files for tests"]]
[[!tag pondering]]
[[!meta date="2020-01-05 15:30"]]

I've started to sketch out the first Subplot document for Subplot.
The goal is to have Subplot to test itself. The approach is to run the
program with specific inputs, and examine the results. The inputs will
be the Markdown, YAML, and Python files that Subplot needs to do its
thing. This raises the question how those files should be stored. A
few possible approaches:

1. Provide the files adjacent to the subplot.md file that is the
   Subplot self-testing file. This would mean the acceptance tests use
   test data that's not evident in the output document of Subplot,
   making them hard to examine unless one has access to the document
   source code.

1. Embed the file data into the scenarios. This is what Subplot's
   predecessor yarn did. It tends to result in tediously long
   scenarios. Tedious to read, tedious to write, tedious to change. On
   the other hand, this would make the test data evident and not
   require changes to current Subplot code.

1. Embed the file data in the Markdown document, and have Subplot
   extract it for the generated test program. This would require
   changing Subplot to support that, but seems otherwise ideal.
   Markdown, as implemented Pandoc, is fairly good at this, and Pandoc
   can even do syntax highlighting if, for example, the data is in a
   YAML format.

I'm leaning strongly towards the last approach. It seems like the best
approach for editing comfort, while keeping the test data evident to
all stakeholders.

The syntax would look something like below, which is an example of a
test data file containing a Subplot document.

~~~~~~~~markdown
~~~~~{.file .markdown numberLines}
---
title: Test scenario
bindings: b.yaml
functions: f.py
...

# Simple
This is the simplest possible test scenario

```subplot
given precondition foo
when I do bar
then bar was done
```
~~~~~
~~~~~~~~

Pandoc allows attributes to fenced code blocks. Above, we declare the
block is a file, in markdown, and its lines should be numbered.
Subplot would require the `.file` attribute to recognize it as a file.
The rest are optional, but make Pandoc typeset the file nicely.