1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
---
title: "`obnam-benchmark`---tool to run benchmarks"
author: "The Obnam project"
documentclass: report
bindings:
- subplot/benchmark.yaml
- lib/files.yaml
- lib/runcmd.yaml
impls:
python:
- subplot/benchmark.py
- lib/files.py
- lib/runcmd.py
...
# Introduction
Obnam is a backup program. It aims to have good performance. To
achieve that, we run benchmarks, and we automate that. To allow
flexibility in what benchmark scenarios we have, we have a tool that
reads benchmark specifications from a file and runs them. This
document is the description of that tool, and its acceptance criteria.
# Overview
The `obnam-benchmark` tool runs a set of specified benchmarks for one
version of Obnam. It works as follows:
1. Read the specification file.
2. Build the specified Obnam version.
3. Start the Obnam server with an empty chunk directory.
4. Run each specified benchmark, and take measurements.
5. Stop the Obnam server. Clean up.
6. Store measurements in a form that can be processed later.
A benchmark specification file is in YAML format. It has a list of
individual benchmarks. Each benchmark has a name, and a description of
how to prepare the test data for each backup generation. Example:
```{.yaml .numberLines}
- benchmark: maildir
backups:
- changes:
- create:
files: 100000
file_size: 0
- rename:
files: 1000
- changes: []
- benchmark: video-footage
backups:
- changes:
- create:
files: 1000
filee_size: 1G
- changes: []
```
The example above specifies two benchmarks: "maildir", and
"video-footage". The names are not interpreted by `obnam-benchmark`,
they are for human consumption only. The `backups` field specifies a
list of changes to the test data; the benchmark tool runs a backup for
each item in the list. Each change can create, copy, delete, or rename
files, compared to the previous backup. The created files will have
the specified size, and the content is randomly generated,
non-repetitive, binary junk. The files will be stored in a directory
tree avoiding very large numbers of files per directory.
By specifying what kind of data is generated, and how it changes from
backup to backup, and how many backups are made, the specification
file can describe synthetic benchmarks for different use cases.
# Acceptance criteria
## Generate random data for a backup
_Requirement: The benchmark tool can generate random test data._
We verify this by having the benchmark generate some test data, using
a subcommand for this purpose. If the data doesn't compress well, we
assume it's sufficiently random. We can safely assume that the same
code is used to generate test data for benchmarks, though of course
the scenario can't verify that. Instead, it can be verified via manual
code inspection.
~~~scenario
given an installed Rust program obnam-benchmark
when I run obnam-benchmark generate-junk 123000 junk.dat
when I run gzip junk.dat
then file junk.dat.gz is at least 100100 bytes long
~~~
## Parses a specification file
_Requirement: The benchmark tool can parse a specification file._
We verify this by having the tool output a YAML specification file as
JSON. Given the formats are different, this will check that the tool
actually parses the file. We use an input file that contains aspects
of benchmark specification that are not just YAML, with the assumption
that the tool uses a YAML parser that works well, so that we only need
to care about things that are specific to the benchmark tool.
Specifically, we verify that the tool parses file sizes such as "1K"
correctly.
```scenario
given an installed Rust program obnam-benchmark
given file spec.yaml
given file expected.json
when I run obnam-benchmark spec spec.yaml --output spec.json
then JSON files spec.json and expected.json match
```
```{#spec.yaml .yaml .file .numberLines}
benchmarks:
- benchmark: foo
backups:
- changes: []
```
```{#expected.json .yaml .file .numberLines}
{
"benchmarks": [
{
"benchmark": "foo",
"backups": [
{
"changes": []
}
]
}
]
}
```
## Smoke test to run a trivial benchmark
_Requirement: The benchmark tool can benchmarks at all._
We verify this by running a trivial benchmark, which backs up an
empty directory.
~~~scenario
given an installed Rust program obnam-benchmark
given file smoke.yaml
when I run obnam-benchmark run smoke.yaml --output smoke.json
then file smoke.json is valid JSON
~~~
```{#smoke.yaml .yaml .file .numberLines}
benchmarks:
- benchmark: smoke
backups:
- changes: []
```
|
