1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
|
README for cmdtest
==================
Sometimes it would be nice to test Unix command line tools as black
boxes. This is the start of a tool for doing that. Not written yet.
Each test consists a directory with several fuiles,
some of which are optional:
* `setup`: shell commands to run before the test, one per line
* `teardown`: shell commands to run after the test, one per line
* `args`: arguments to the command, i.e., everything beyond `argv[0]`,
one argument per line (`argv[0]` is given with `--command=CMD` option)
* `stdin`: the contents is fed to the command via the standard input;
if not given, `/dev/null` is used instead
* `stdout`: the expected output from stdout (default is no output)
* `stderr`: the expected output from stderr (default is no output)
* `exit`: one line giving the expected exit code of the command (zero, if
not given)
* `verify`: shell commands to verify the output, one per line; if this
exists, then `stdout`, `stderr`, and `exit` are not checked automatically
In addition to a single `setup` file, you can provide many shell scripts
name `setup-*`, and these will be executed one by one. Ditto for `teardown`.
Instead of a single `args` file, you can provide `args-*` files, and
corresponding `stdout-*`, `stderr-*`, and `exit-*` files, or
`verify-*` files. This allows you to share the setup and teardown
scaffolding for several tests, without having to duplicate them between
test directories.
You may use the following environment variables:
* `TESTDIR`: a temporary directory (created and removed automatically),
where you can create files indiscriminately
A simple test:
sorts-correctly/
stdin # "xyz\nabc\ndef\n"
stdout # "abc\ndef\nxyz\n"
Invocation:
cmdtest --command=./mysort sorts-correctly
A more complicated test:
sorts-correctly/
stdin-empty # ""
stdout-empty # ""
stdin-one-line # "abc\n"
stdout-one-line # "abc\n"
stdin-two-lines-sorted # "abc\ndef\n"
stdout-two-lines-sorted # "abc\ndef\n"
stdin-two-lines-unsorted # "def\nabc\n"
stdout-two-lines-unsorted # "abc\ndef\n"
This will make cmdtest run four tests: it will run the command to
be tested with the standard input coming from `stdin-empty` and comparing
the output with `stdout-empty`, and repeating that with the other pairs
of files.
If cmdtest finds a problem, it shows the "diff -u" output of the expected
and actual outputs.
---
sorts-empty-file.stdin # ""
sorts-empty-file.stdout # ""
sorts-one-line.stdin # "abc\n"
sorts-one-line.stdout # "abc\n"
sorts-two-sorted-lines.stdin # "abc\ndef\n"
sorts-two-sorted-lines.stdout # "abc\ndef\n"
sorts-two-unsorted-lines.stdin # "def\nabc\n"
sorts-two-unsorted-lines.stdout # "abc\ndef\n"
---
`cmdtest` black box tests Unix command line tools.
Roughly, it is given a command line and input files, and the expected output,
and it verifies that the command line produces the expected output.
If not, it reports a problem, and shows the differences.
The command line arguments, inputs, and outputs are given in files
that follow a naming convention. The test report is written to the
standard output.
Each test case consists of:
* a set of command line arguments, not including the command name (`foo.args`)
* the file fed to standard input (`foo.stdin`)
* the expected output to the standard output (`foo.stdout`)
* the expected output to the standard error (`foo.stderr`)
* the expected exit code (`foo.exit`)
* a shell script to run before the test (`foo.setup`)
* a shell script to run after the test (`foo.teardown`)
Usually, a single test is not enough. All tests can be put into the
same directory, and they may share some setup and teardown code:
* a shell script to run once, before all tests (`setup-once`)
* a shell script to run before each test (`setup`)
* a shell script to run after each test (`teardown`)
* a shell script to run after all tests (`teardown-once`)
`cmdtest` is given the name of the directory with all the tests,
or several such directories, and it does the following:
* execute `setup-once`
* for each test case (unique prefix `foo`):
- execute `setup`
- execute `foo.setup`
- execute the command, giving it command line arguments from
`foo.args`, and redirecting standard input to come from `foo.stdin`
- capture standard output and error and exit codes
- execute `foo.teardown`
- execute `teardown`
- report result of test: does exit code match `foo.exit`, standard
output match `foo.stdout`, and standard error match `foo.stderr`?
* execute `teardown-once`
All of these files are optional. If a setup/teardown script is missing,
it is simply not executed. If one of the standard input, output, or
error files is missing, it is treated as if it were empty. If the
exit code file is missing, it is treated as if it specified an exit
code of zero.
The actual command is given to `cmdtest` separately. This allows
testing various implementations of the same command, for example
various checksum utilities, or different versions of the same command.
If `foo.args` would hardcode the command name, this would be hard
to achieve.
The shell scripts may use the following environment variables:
* `DATADIR`: a temporary directory where files may be created by the test
|
