1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
|
# Introduction
The [Subplot][] library `runcmd` for Python provides scenario steps
and their implementations for running Unix commands and examining the
results. The library consists of a bindings file `lib/runcmd.yaml` and
implementations in Python in `lib/runcmd.py` or in the Rust subplotlib
step library. There is no Bash version.
[Subplot]: https://subplot.liw.fi/
This document explains the acceptance criteria for the library and how
they're verified. It uses the steps and functions from the
`lib/runcmd` library. The scenarios all have the same structure: run a
command, then examine the exit code, standard output (stdout for
short), or standard error output (stderr) of the command.
The scenarios use the Unix commands `true` and `false` to
generate exit codes, and `echo` to produce stdout. To generate
stderr, they use the little helper script below.
~~~{#err.sh .file .sh .numberLines}
#!/bin/sh
echo "$@" 1>&2
~~~
# Check exit code
These scenarios verify the exit code. To make it easier to write
scenarios in language that flows more naturally, there are a couple of
variations.
## Successful execution
~~~scenario
when I run true
then exit code is 0
and command is successful
~~~
## Successful execution in a sub-directory
~~~scenario
given a directory xyzzy
when I run, in xyzzy, pwd
then exit code is 0
then command is successful
then stdout contains "/xyzzy"
~~~
## Failed execution
~~~scenario
when I try to run false
then exit code is not 0
and command fails
~~~
## Failed execution in a sub-directory
~~~scenario
given a directory xyzzy
when I try to run, in xyzzy, false
then exit code is not 0
and command fails
~~~
# Check we can prepend to $PATH
This scenario verifies that we can add a directory to the beginning of
the PATH environment variable, so that we can have `runcmd` invoke a
binary from our build tree rather than from system directories. This
is especially useful for testing new versions of software that's
already installed on the system.
~~~scenario
given executable script ls from ls.sh
when I prepend . to PATH
when I run ls
then command is successful
then stdout contains "custom ls, not system ls"
~~~
~~~{#ls.sh .file .sh .numberLines}
#!/bin/sh
echo "custom ls, not system ls"
~~~
# Check output has what we want
These scenarios verify that stdout or stderr do have something we want
to have.
## Check stdout is exactly as wanted
Note that the string is surrounded by double quotes to make it clear
to the reader what's inside. Also, C-style string escapes are
understood.
~~~scenario
when I run echo hello, world
then stdout is exactly "hello, world\n"
~~~
## Check stderr is exactly as wanted
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hello, world
then stderr is exactly "hello, world\n"
~~~
## Check stdout using sub-string search
Exact string comparisons are not always enough, so we can verify a
sub-string is in output.
~~~scenario
when I run echo hello, world
then stdout contains "world\n"
and exit code is 0
~~~
## Check stderr using sub-string search
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hello, world
then stderr contains "world\n"
~~~
## Check stdout using regular expressions
Fixed strings are not always enough, so we can verify output matches a
regular expression. Note that the regular expression is not delimited
and does not get any C-style string escaped decoded.
~~~scenario
when I run echo hello, world
then stdout matches regex world$
~~~
## Check stderr using regular expressions
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hello, world
then stderr matches regex world$
~~~
# Check output doesn't have what we want to avoid
These scenarios verify that the stdout or stderr do not
have something we want to avoid.
## Check stdout is not exactly something
~~~scenario
when I run echo hi
then stdout isn't exactly "hello, world\n"
~~~
## Check stderr is not exactly something
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hi
then stderr isn't exactly "hello, world\n"
~~~
## Check stdout doesn't contain sub-string
~~~scenario
when I run echo hi
then stdout doesn't contain "world"
~~~
## Check stderr doesn't contain sub-string
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hi
then stderr doesn't contain "world"
~~~
## Check stdout doesn't match regular expression
~~~scenario
when I run echo hi
then stdout doesn't match regex world$
~~~
## Check stderr doesn't match regular expressions
~~~scenario
given helper script err.sh for runcmd
when I run sh err.sh hi
then stderr doesn't match regex world$
~~~
|
