1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
README for cmdtest
==================
`cmdtest` black box tests Unix command line tools.
Roughly, it is given a command line and input files, and the expected output,
and it verifies that the command line produces the expected output.
If not, it reports a problem, and shows the differences.
The command line arguments, inputs, and outputs are given in files
that follow a naming convention. The test report is written to the
standard output.
Each test case consists of:
* a set of command line arguments, not including the command name (`foo.args`)
- each argument is on its own line
* the file fed to standard input (`foo.stdin`)
* the expected output to the standard output (`foo.stdout`)
* the expected output to the standard error (`foo.stderr`)
* the expected exit code (`foo.exit`)
* a shell script to run before the test (`foo.setup`)
* a shell script to run after the test (`foo.teardown`)
Usually, a single test is not enough. All tests can be put into the
same directory, and they may share some setup and teardown code:
* a shell script to run once, before all tests (`setup-once`)
* a shell script to run before each test (`setup`)
* a shell script to run after each test (`teardown`)
* a shell script to run after all tests (`teardown-once`)
`cmdtest` is given the name of the directory with all the tests,
or several such directories, and it does the following:
* execute `setup-once`
* for each test case (unique prefix `foo`):
- execute `setup`
- execute `foo.setup`
- execute the command, giving it command line arguments from
`foo.args`, and redirecting standard input to come from `foo.stdin`
- capture standard output and error and exit codes
- execute `foo.teardown`
- execute `teardown`
- report result of test: does exit code match `foo.exit`, standard
output match `foo.stdout`, and standard error match `foo.stderr`?
* execute `teardown-once`
All of these files are optional. If a setup/teardown script is missing,
it is simply not executed. If one of the standard input, output, or
error files is missing, it is treated as if it were empty. If the
exit code file is missing, it is treated as if it specified an exit
code of zero.
The actual command is given to `cmdtest` separately. This allows
testing various implementations of the same command, for example
various checksum utilities, or different versions of the same command.
If `foo.args` would hardcode the command name, this would be hard
to achieve.
The shell scripts may use the following environment variables:
* `DATADIR`: a temporary directory where files may be created by the test
In addition, the `foo.args` files can use Pythonic string expansion:
* `%(datadir)s` for the same value as `$DATADIR`
|
