# Introduction

Obnam is a backup program. It aims to have good performance. To
achieve that, we run benchmarks, and we automate that. To allow
flexibility in what benchmark scenarios we have, we have a tool that
reads benchmark specifications from a file and runs them. This
document is the description of that tool, and its acceptance criteria.

# Overview

The `obnam-benchmark` tool runs a set of specified benchmarks for one
version of Obnam. It works as follows:

1. Read the specification file.
2. Build the specified Obnam version.
3. Start the Obnam server with an empty chunk directory.
4. Run each specified benchmark, and take measurements.
5. Stop the Obnam server. Clean up.
6. Store measurements in a form that can be processed later.

A benchmark specification file is in YAML format. It has a list of
individual benchmarks. Each benchmark has a name, and a description of
how to prepare the test data for each backup generation. Example:

```{.yaml .numberLines}
- benchmark: maildir
  backups:
  - changes:
    - create:
        files: 100000
        file_size: 0
    - rename:
        files: 1000
  - changes: []
- benchmark: video-footage
  backups:
  - changes:
    - create:
        files: 1000
        filee_size: 1G
  - changes: []
```

The example above specifies two benchmarks: "maildir", and
"video-footage". The names are not interpreted by `obnam-benchmark`,
they are for human consumption only. The `backups` field specifies a
list of changes to the test data; the benchmark tool runs a backup for
each item in the list. Each change can create, copy, delete, or rename
files, compared to the previous backup. The created files will have
the specified size, and the content is randomly generated,
non-repetitive, binary junk. The files will be stored in a directory
tree avoiding very large numbers of files per directory.

By specifying what kind of data is generated, and how it changes from
backup to backup, and how many backups are made, the specification
file can describe synthetic benchmarks for different use cases.

# Acceptance criteria

## Generate random data for a backup

_Requirement: The benchmark tool can generate random test data._

We verify this by having the benchmark generate some test data, using
a subcommand for this purpose. If the data doesn't compress well, we
assume it's sufficiently random. We can safely assume that the same
code is used to generate test data for benchmarks, though of course
the scenario can't verify that. Instead, it can be verified via manual
code inspection.

~~~scenario
given an installed Rust program obnam-benchmark
when I run obnam-benchmark generate-junk 123000 junk.dat
when I run gzip junk.dat
then file junk.dat.gz is at least 100100 bytes long
~~~


## Parses a specification file

_Requirement: The benchmark tool can parse a specification file._

We verify this by having the tool output a YAML specification file as
JSON. Given the formats are different, this will check that the tool
actually parses the file. We use an input file that contains aspects
of benchmark specification that are not just YAML, with the assumption
that the tool uses a YAML parser that works well, so that we only need
to care about things that are specific to the benchmark tool.

Specifically, we verify that the tool parses file sizes such as "1K"
correctly.

```scenario
given an installed Rust program obnam-benchmark
given file spec.yaml
given file expected.json
when I run obnam-benchmark spec spec.yaml --output spec.json
then JSON files spec.json and expected.json match
```

```{#spec.yaml .yaml .file .numberLines}
benchmarks:
- benchmark: foo
  backups:
  - changes: []
```

```{#expected.json .yaml .file .numberLines}
{
    "benchmarks": [
        {
            "benchmark": "foo",
            "backups": [
                {
                    "changes": []
                }
            ]
        }
    ]
}
```


## Smoke test to run a trivial benchmark

_Requirement: The benchmark tool can benchmarks at all._

We verify this by running a trivial benchmark, which backs up an
empty directory.

~~~scenario
given an installed Rust program obnam-benchmark
given file smoke.yaml
when I run obnam-benchmark run smoke.yaml --output smoke.json
then file smoke.json is valid JSON
~~~

```{#smoke.yaml .yaml .file .numberLines}
benchmarks:
- benchmark: smoke
  backups:
  - changes: []
```


## Benchmark with several backups

_Requirement: The benchmark tool can benchmarks with more than one
backup._

We verify this by running a benchmark with three backup generations.

~~~scenario
given an installed Rust program obnam-benchmark
given file three.yaml
when I run obnam-benchmark run three.yaml --output three.json
then file three.json is valid JSON
~~~

```{#three.yaml .yaml .file .numberLines}
benchmarks:
- benchmark: three
  backups:
  - changes:
      - create:
          files: 1
          file_size: 1024
  - changes:
      - create:
          files: 2
          file_size: 2048
  - changes:
      - create:
          files: 4
          file_size: 4096
```