summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--Cargo.toml8
-rw-r--r--NEWS6
-rw-r--r--README.md51
-rw-r--r--RELEASE.md1
-rw-r--r--ansible/files/obnam.service4
-rw-r--r--ansible/hosts2
-rw-r--r--ansible/obnam-server.yml41
-rwxr-xr-xcheck11
-rw-r--r--client.yaml4
-rw-r--r--debian/changelog2
-rwxr-xr-xdebian/rules4
-rwxr-xr-x[-rw-r--r--]list_new_release_tags0
-rw-r--r--obnam.md531
-rw-r--r--src/backup_progress.rs51
-rw-r--r--src/backup_reason.rs12
-rw-r--r--src/backup_run.rs162
-rw-r--r--src/bin/obnam-server.rs52
-rw-r--r--src/bin/obnam.rs16
-rw-r--r--src/chunk.rs17
-rw-r--r--src/chunker.rs14
-rw-r--r--src/chunkmeta.rs5
-rw-r--r--src/client.rs160
-rw-r--r--src/cmd/backup.rs98
-rw-r--r--src/cmd/get_chunk.rs5
-rw-r--r--src/cmd/list.rs5
-rw-r--r--src/cmd/list_files.rs11
-rw-r--r--src/cmd/mod.rs3
-rw-r--r--src/cmd/restore.rs113
-rw-r--r--src/cmd/show_config.rs7
-rw-r--r--src/cmd/show_gen.rs10
-rw-r--r--src/error.rs51
-rw-r--r--src/fsentry.rs51
-rw-r--r--src/fsiter.rs47
-rw-r--r--src/generation.rs83
-rw-r--r--src/genlist.rs12
-rw-r--r--src/index.rs70
-rw-r--r--src/indexedstore.rs71
-rw-r--r--src/policy.rs2
-rw-r--r--src/server.rs48
-rw-r--r--src/store.rs18
-rw-r--r--subplot/client.py1
-rw-r--r--subplot/client.yaml3
-rw-r--r--subplot/data.py68
-rw-r--r--subplot/data.yaml23
-rw-r--r--subplot/server.py21
-rw-r--r--subplot/server.yaml3
-rw-r--r--tutorial.md279
47 files changed, 1792 insertions, 465 deletions
diff --git a/Cargo.toml b/Cargo.toml
index 9cf5b52..d36c450 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,12 +1,18 @@
[package]
name = "obnam"
-version = "0.1.2"
+version = "0.2.2"
authors = ["Lars Wirzenius <liw@liw.fi>"]
edition = "2018"
+description = "a backup program"
+license = "GPL-3.0-or-later"
+homepage = "https://obnam.org/"
+repository = "https://gitlab.com/larswirzenius/obnam"
+
[dependencies]
anyhow = "1"
bytes = "0.5"
+bytesize = "1"
chrono = "0.4"
dirs = "3"
indicatif = "0.15"
diff --git a/NEWS b/NEWS
index 78b12be..cdd50df 100644
--- a/NEWS
+++ b/NEWS
@@ -6,7 +6,9 @@ of Obnam, the backup software. The software is technically called
ended in 2017 with version number 1.22.
-## Obnam2 version 0.2, not yet released
+## Obnam2 version 0.2.2, released 2021-01-29
-This if the first release of Obnam2. It can just barely make and
+This is the first release of Obnam2. It can just barely make and
restore backups. It's ready for a light trial, but not for real use.
+There's no encryption, and backups can't be deleted yet. Restores of
+the entire backup work.
diff --git a/README.md b/README.md
index af9d6e1..97c4484 100644
--- a/README.md
+++ b/README.md
@@ -2,54 +2,9 @@
Obnam2 is a project to develop a backup system.
-You probably want to read the [obnam.md](obnam.md) subplot file.
-
-## Client installation
-
-See instructions at <https://obnam.org/download/> for installing the
-client. It's not duplicated here to avoid having to keep the
-information in sync in two places.
-
-## Server installation
-
-To install the Obnam server component, you need a Debian host with
-sufficient disk space, and Ansible installed locally. Run the
-following commands in the Obnam source tree, replacing
-`obnam.example.com` with the domain name of your server:
-
-```sh
-$ cd ansible
-$ printf '[obnam-server]\nobnam.example.com\n' > hosts
-$ ansible-playbook -i hosts obnam-server.yml -e domain=obnam.example.com
-```
-
-The above gets a free TLS certificate from [Let's Encrypt][], but only
-works if the server is accessible from the public Internet. For a
-private host use the following instead:
-
-```sh
-$ cd ansible
-$ printf '[obnam-server]\nprivate-vm\n' > hosts
-$ ansible-playbook -i hosts obnam-server.yml
-```
-
-This uses a pre-created self-signed certificate from
-`files/server.key` and `files/server.pem` and is probably only good
-for trying out Obnam. You may want to generate your own certificates
-instead.
-
-To create a self-signed certificate, something like the following
-command might work, using [OpenSSL]:
-
-```sh
-$ openssl req -x509 -newkey rsa:4096 -passout pass:hunter2 \
- -keyout key.pem -out cert.pem -days 365 -subj /CN=localhost
-```
-
-
-[Let's Encrypt]: https://letsencrypt.org/
-[OpenSSL]: https://www.openssl.org/
-
+For installation instructions and a quick start guide, see
+[tutorial.md](tutorial.md). For more details on goals, requirements, and
+implementation details, see the [obnam.md](obnam.md) subplot file.
## Legalese
diff --git a/RELEASE.md b/RELEASE.md
index c9f886d..3e59694 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -15,6 +15,7 @@ Follow these steps to make a release of Obnam2.
* create a signed, annotated git tag `vX.Y.Z` for version X.Y.Z for
the release commit
* push tag to `gitlab.com` and `git.liw.fi`
+* publish Obnam crate to crates.io.
* announce new release
- obnam.org blog, possibly other blogs
- `#obnam` IRC channel
diff --git a/ansible/files/obnam.service b/ansible/files/obnam.service
index 9d933aa..f30eb60 100644
--- a/ansible/files/obnam.service
+++ b/ansible/files/obnam.service
@@ -1,10 +1,10 @@
[Unit]
Description=Obnam server
+ConditionPathExists=/etc/obnam/server.yaml
[Service]
Type=simple
-ConditionPathExists=/etc/obnam/server.yaml
-ExecStart=/usr/bin/obnam-server /etc/obnam/server.yaml
+ExecStart=/bin/obnam-server /etc/obnam/server.yaml
[Install]
WantedBy=multi-user.target
diff --git a/ansible/hosts b/ansible/hosts
index 253d739..136086c 100644
--- a/ansible/hosts
+++ b/ansible/hosts
@@ -1,2 +1,2 @@
-[obnam-server]
+[server]
obnam-server
diff --git a/ansible/obnam-server.yml b/ansible/obnam-server.yml
index 426ca74..8837f1b 100644
--- a/ansible/obnam-server.yml
+++ b/ansible/obnam-server.yml
@@ -1,9 +1,15 @@
- hosts: server
remote_user: root
tasks:
+ - name: add APT signing key for the Obnam package repository
+ copy:
+ content: |
+ {{ apt_signing_key }}
+ dest: /etc/apt/trusted.gpg.d/obnam.asc
+
- name: add Obnam package repository to APT
apt_repository:
- repo: "deb [trusted=yes] http://ci-prod-controller.vm.liw.fi/debian unstable-ci main"
+ repo: "deb http://ci-prod-controller.vm.liw.fi/debian unstable-ci main"
- name: refresh APT package lists and upgrade all installed packages
apt:
@@ -94,3 +100,36 @@
vars:
tls_key_path: "/var/lib/dehydrated/certs/{{ domain }}/privkey.pem"
tls_cert_path: "/var/lib/dehydrated/certs/{{ domain }}/cert.pem"
+
+
+ apt_signing_key: |
+ -----BEGIN PGP PUBLIC KEY BLOCK-----
+
+ mQINBFrLO7kBEADdz6mHstYmKU5Dp6OSjxWtWaqTDOX1sJdmmaIK/9EKVIH0Maxp
+ 5kvVO5G6mULLAjv/kLG0MxasHPrq8I2A/y8AqKAGVL8QelwLjQMIFZ30/VbGQPHS
+ +T5TZXEnoQtNce1GUhFwJ38ZyjjwHBFV9tSec7rZ2Q3YeM3nNnGPf6DacXGfEOPO
+ HIN4sXAN2hzNXNjKRzTIvxQseb6nr7afUh/SlZ3yhQOCrIzmYlD7tP9WJe7ofL0p
+ JY4pDQYw8rT6nC2BE/ioemh84kERCT1vCe+OVFlSRuMlqfEv+ZpKQ+itOmPDQ/lM
+ jpUm1K2hrW/lWpxT/ZxHKo/w1K36J5WshgMZxfUu5BMCL9LMqMcrXNhNjDMfxDMM
+ 3yBPOvQ4ls6fecOZ/bsFo1p8VzMk/w/eG8vPs5yuNa5XxN95yFMXoOHGb5Xbu8D4
+ 6yiW+Af70LbiSNpGdmNdneiGB2fY38NxBukPw5u3S5qG8HedSmMr1RvSr5kHoAAe
+ UbOY+BYaaKsTAT7+1skUW1o3FJSqoRKCHAzTsMWC6zzhR8hRn7jVrrguH1hGbqq5
+ TZSCFQZExuTJ7uXrTLG0WoBXIjB5wWNcSeXn8myUWYB51nJNF4tJBouZOz9JwWGl
+ kiAQkrHnBttLQWdW9FyjbIoTZMtpvVx+m6ObGTGdGL1cNlLAvWprMXGc+QARAQAB
+ tDJJY2sgQVBUIHJlcG9zaXRvcnkgc2lnbmluZyBrZXkgKDIwMTgpIDxsaXdAbGl3
+ LmZpPokCTgQTAQgAOBYhBKL1uyDoXyxUH3O717Wr+TZVS6PGBQJayzu5AhsDBQsJ
+ CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJELWr+TZVS6PGB5QQANTcikhRUHwt9N4h
+ dGc/Hp6CbqdshMoWlwpFskttoVDxQG5OAobuZl5XyzGcmja1lT85RGkZFfbca0IZ
+ LnXOLLSAu51QBkXNaj4OhjK/0uQ+ITrvL6RQSXNgHiUTR/W2XD1GIUq6nBqe2GSN
+ 31S1baYKKVj5QIMsi7Dq8ls3BBXuPCE+xTSaNmGWjes2t9pPidcRvxsksCLY1qgw
+ P1GFXBeMkBQ29kBP87SUL15SIk7OiQLlEURCy5iRls5rt/YEsdEpRWIb0Tm5Nrjv
+ 2M3VM+iBhfNXTwj0rJ34mlycF1qQmA7YcTEobT7z587GPY0VWzBpQUnEQj7rQWPM
+ cDYY0b+I6kQ8VKOaL4wVAtE98d7HzFIrIrwhTKufnrWrVDPYsmLZ+LPC1jiF7JBD
+ SR6Vftb+SdDR9xoE1yRuXbC6IfoW+5/qQNrdQ2mm9BFw5jOonBqchs18HTTf3441
+ 6SWwP9fY3Vi+IZphPPi0Gf85oMStgnv/Wnw6LacEL32ek39Desero/D8iGLZernK
+ Q2mC9mua5A/bYGVhsNWyURNFkKdbFa+/wW3NfdKYyZnsSfo+jJ2luNewrhAY7Kod
+ GWXTer9RxzTGA3EXFGvNr+BBOOxSj0SfWTl0Olo7J5dnxof+jLAUS1VHpceHGHps
+ GSJSdir7NkZidgwoCPA7BTqsb5LN
+ =dXB0
+ -----END PGP PUBLIC KEY BLOCK-----
+
diff --git a/check b/check
index 65514cc..79b73ee 100755
--- a/check
+++ b/check
@@ -4,14 +4,13 @@
set -eu
-quiet=-q
hideok=chronic
if [ "$#" -gt 0 ]
then
case "$1" in
verbose | -v | --verbose)
- quiet=
hideok=
+ shift
;;
esac
fi
@@ -21,10 +20,10 @@ got_cargo_cmd()
cargo --list | grep " $1 " > /dev/null
}
-cargo build --all-targets $quiet
-got_cargo_cmd clippy && cargo clippy $quiet
-got_cargo_cmd fmt && cargo fmt -- --check
-$hideok cargo test $quiet
+$hideok cargo build --all-targets
+got_cargo_cmd clippy && $hideok cargo clippy
+got_cargo_cmd fmt && $hideok cargo fmt -- --check
+$hideok cargo test
sp-docgen obnam.md -o obnam.html
sp-docgen obnam.md -o obnam.pdf
diff --git a/client.yaml b/client.yaml
index dd60c9c..7fa935b 100644
--- a/client.yaml
+++ b/client.yaml
@@ -1,3 +1,5 @@
server_url: https://localhost:8888
-root: /home/liw/tmp/Foton
+verify_tls_cert: false
+roots:
+ - /home/liw/tmp/watch-and-review
log: obnam.log
diff --git a/debian/changelog b/debian/changelog
index ed26379..136f47c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,4 +1,4 @@
-obnam (0.1.3-1) unstable; urgency=low
+obnam (0.2.2-1) unstable; urgency=low
* Initial packaging. This is not intended to be uploaded to Debian, so
no closing of an ITP bug.
diff --git a/debian/rules b/debian/rules
index 6675ae9..ecfca56 100755
--- a/debian/rules
+++ b/debian/rules
@@ -8,8 +8,8 @@ override_dh_auto_build:
override_dh_auto_install:
cargo install --path=. --root=debian/obnam
- rm -f debian/obnam/.crates.toml
- rm -f debian/obnam/.crates2.json
+ find debian/obnam -name '.crates*' -delete
+ find debian/obnam/bin -type f ! -name 'obnam*' -delete
override_dh_auto_test:
./check
diff --git a/list_new_release_tags b/list_new_release_tags
index 90994ac..90994ac 100644..100755
--- a/list_new_release_tags
+++ b/list_new_release_tags
diff --git a/obnam.md b/obnam.md
index accfca2..240dcf0 100644
--- a/obnam.md
+++ b/obnam.md
@@ -95,6 +95,40 @@ outcomes.
[Subplot]: https://subplot.liw.fi/
+# Threat model
+
+This chapter discusses the various threats against backups. Or it
+will. For now it's very much work in progress. This version of the
+chapter is only meant to get threat modeling started by having the
+simplest possible model that is in any way useful.
+
+## Backed up data is readable by server operator
+
+This threat is about the operator of the backup server being able to
+read the data backed up by any user of the server. We have to assume
+that the operator can read any file and can also eavesdrop all network
+traffic. The operator can even read all physical and virtual memory on
+the server.
+
+The mitigation strategy is to encrypt the data before it is sent to
+the server. If the server never receives cleartext data, the operator
+can't read it.
+
+Backups have four kinds of data:
+
+* actual contents of live data files
+* metadata about live data files, as stored on the client file system,
+ such as the name, ownership, or size of each file
+* metadata about the contents of live data, such as its cryptographic
+ checksum
+* metadata about the backup itself
+
+For now, we are concerned about the first two kinds. The rest will be
+addressed later.
+
+The mitigation technique against this threat is to encrypt the live
+data and its metadata before uploading it to the server.
+
# Software architecture
## Effects of requirements
@@ -154,66 +188,6 @@ requirements and notes how they affect the architecture.
access that.
-## On SFTP versus HTTPS
-
-Obnam1 supported using a standard SFTP server as a backup repository,
-and this was a popular feature. This section argues against supporting
-SFTP in Obnam2.
-
-The performance requirement for network use means favoring protocols
-such as HTTPS, or even QUIC, rather than SFTP.
-
-SFTP works on top of SSH. SSH provides a TCP-like abstraction for
-SFTP, and thus multiple SFTP connections can run over the same SSH
-connection. However, SSH itself uses a single TCP connection. If that
-TCP connection has a dropped packet, all traffic over the SSH
-connections, including all SFTP connections, waits until TCP
-re-transmits the lost packet and re-synchronizes itself.
-
-With multiple HTTP connections, each on its own TCP connection, a
-single dropped packet will not affect other HTTP transactions. Even
-better, the new QUIC protocol doesn't use TCP.
-
-The modern Internet is to a large degree designed for massive use of
-the world wide web, which is all HTTP, and adopting QUIC. It seems
-wise for Obnam to make use of technologies that have been designed
-for, and proven to work well with concurrency and network problems.
-
-Further, having used SFTP with Obnam1, it is not always an easy
-protocol to use. Further, if there is a desire to have controlled
-sharing of parts of one client's data with another, this would require
-writing a custom SFTP service, which seems much harder to do than
-writing a custom HTTP service. From experience, a custom HTTP service
-is easy to do. A custom SFTP service would need to shoehorn the
-abstractions it needs into something that looks more or less like a
-Unix file system.
-
-The benefit of using SFTP would be that a standard SFTP service could
-be used, if partial data sharing between clients is not needed. This
-would simplify deployment and operations for many. However, it doesn't
-seem important enough to warrant the implementation effort.
-
-Supporting both HTTP and SFTP would be possible, but also much more
-work and against the desire to keep things simple.
-
-## On "btrfs send" and similar constructs
-
-The btrfs and ZFS file systems, and possibly others, have a way to
-mark specific states of the file system and efficiently generate a
-"delta file" of all the changes between the states. The delta can be
-transferred elsewhere, and applied to a copy of the file system. This
-can be quite efficient, but Obnam won't be built on top of such a
-system.
-
-On the one hand, it would force the use of specific file systems:
-Obnam would no be able to back up data on, say, an ext4 file system,
-which seems to be the most popular one by far.
-
-Worse, it also for the data to be restored to the same type of file
-system as where the live data was originally. This onerous for people
-to do.
-
-
## Overall shape
It seems fairly clear that a simple shape of the software architecture
@@ -266,8 +240,8 @@ The responsibilities of the server are roughly:
The responsibilities of the client are roughly:
* split live data into chunks, upload them to server
-* store metadata of live data files in a file, which represents a
- backup generation, store that too as chunks on the server
+* store metadata of live data files in a generation file (an SQLite
+ database), store that too as chunks on the server
* retrieve chunks from server when restoring
* let user manage sharing of backups with other clients
@@ -282,6 +256,221 @@ RSA-signed JSON Web Tokens. The server is configured to trust specific
public keys. The clients have the private keys and generate the tokens
themselves.
+## Logical structure of backups
+
+For each backup (generation) the client stores, on the server, exactly
+one _generation chunk_. This is a chunk that is specially marked as a
+generation, but is otherwise not special. The generation chunk content
+is a list of identifiers for chunks that form an SQLite database.
+
+The SQLite database lists all the files in the backup, as well as
+their metadata. For each file, a list of chunk identifiers are listed,
+for the content of the file. The chunks may be shared between files in
+the same backup or different backups.
+
+File content data chunks are just blobs of data with no structure.
+They have no reference to other data chunks, or to files or backups.
+This makes it easier to share them between files.
+
+Let's look at an example. In the figure below there are three backups,
+each using three chunks for file content data. One chunk, "data chunk
+3", is shared between all three backups.
+
+~~~pikchr
+GEN1: ellipse "Backup 1" big big
+move 200%
+GEN2: ellipse "Backup 2" big big
+move 200%
+GEN3: ellipse "Backup 3" big big
+
+arrow from GEN1.e right to GEN2.w
+arrow from GEN2.e right to GEN3.w
+
+arrow from GEN1.s down 100%
+DB1: box "SQLite" big big
+
+arrow from DB1.s left 20% then down 100%
+C1: file "data" big big "chunk 1" big big
+
+arrow from DB1.s right 0% then down 70% then right 100% then down 30%
+C2: file "data" big big "chunk 2" big big
+
+arrow from DB1.s right 20% then down 30% then right 200% then down 70%
+C3: file "data" big big "chunk 3" big big
+
+
+
+arrow from GEN2.s down 100%
+DB2: box "SQLite" big big
+
+arrow from DB2.s left 20% then down 100% then down 0.5*C3.height then to C3.e
+
+arrow from DB2.s right 0% then down 70% then right 100% then down 30%
+C4: file "data" big big "chunk 4" big big
+
+arrow from DB2.s right 20% then down 30% then right 200% then down 70%
+C5: file "data" big big "chunk 5" big big
+
+
+
+
+arrow from GEN3.s down 100%
+DB3: box "SQLite" big big
+
+arrow from DB3.s left 50% then down 100% then down 1.5*C3.height \
+ then left until even with C3.s then up to C3.s
+
+arrow from DB3.s right 20% then down 100%
+C6: file "data" big big "chunk 6" big big
+
+arrow from DB3.s right 60% then down 70% then right 100% then down 30%
+C7: file "data" big big "chunk 7" big big
+~~~
+
+
+## On SFTP versus HTTPS
+
+Obnam1 supported using a standard SFTP server as a backup repository,
+and this was a popular feature. This section argues against supporting
+SFTP in Obnam2.
+
+The performance requirement for network use means favoring protocols
+such as HTTPS, or even QUIC, rather than SFTP.
+
+SFTP works on top of SSH. SSH provides a TCP-like abstraction for
+SFTP, and thus multiple SFTP connections can run over the same SSH
+connection. However, SSH itself uses a single TCP connection. If that
+TCP connection has a dropped packet, all traffic over the SSH
+connections, including all SFTP connections, waits until TCP
+re-transmits the lost packet and re-synchronizes itself.
+
+With multiple HTTP connections, each on its own TCP connection, a
+single dropped packet will not affect other HTTP transactions. Even
+better, the new QUIC protocol doesn't use TCP.
+
+The modern Internet is to a large degree designed for massive use of
+the world wide web, which is all HTTP, and adopting QUIC. It seems
+wise for Obnam to make use of technologies that have been designed
+for, and proven to work well with concurrency and network problems.
+
+Further, having used SFTP with Obnam1, it is not always an easy
+protocol to use. Further, if there is a desire to have controlled
+sharing of parts of one client's data with another, this would require
+writing a custom SFTP service, which seems much harder to do than
+writing a custom HTTP service. From experience, a custom HTTP service
+is easy to do. A custom SFTP service would need to shoehorn the
+abstractions it needs into something that looks more or less like a
+Unix file system.
+
+The benefit of using SFTP would be that a standard SFTP service could
+be used, if partial data sharing between clients is not needed. This
+would simplify deployment and operations for many. However, it doesn't
+seem important enough to warrant the implementation effort.
+
+Supporting both HTTP and SFTP would be possible, but also much more
+work and against the desire to keep things simple.
+
+## On "btrfs send" and similar constructs
+
+The btrfs and ZFS file systems, and possibly others, have a way to
+mark specific states of the file system and efficiently generate a
+"delta file" of all the changes between the states. The delta can be
+transferred elsewhere, and applied to a copy of the file system. This
+can be quite efficient, but Obnam won't be built on top of such a
+system.
+
+On the one hand, it would force the use of specific file systems:
+Obnam would no be able to back up data on, say, an ext4 file system,
+which seems to be the most popular one by far.
+
+Worse, it also for the data to be restored to the same type of file
+system as where the live data was originally. This onerous for people
+to do.
+
+
+## On content addressable storage
+
+[content-addressable storage]: https://en.wikipedia.org/wiki/Content-addressable_storage
+[git version control system]: https://git-scm.com/
+
+It would be possible to use the cryptographic checksum ("hash") of the
+contents of a chunk as its identifier on the server side, also known
+as [content-addressable storage][]. This would simplify de-duplication
+of chunks. However, it also has some drawbacks:
+
+* it becomes harder to handle checksum collisions
+* changing the checksum algorithm becomes harder
+
+In 2005, the author of [git version control system][] chose the
+content addressable storage model, using the SHA1 checksum algorithm.
+At the time, the git author considered SHA1 to be reasonably strong
+from a cryptographic and security point of view, for git. In other
+words, given the output of SHA1, it was difficult to deduce what the
+input was, or to find another input that would give the same output,
+known as a checksum collision. It is still difficult to deduce the
+input, but manufacturing collisions is now feasible, with some
+constraints. The git project has spent years changing the checksum
+algorithm.
+
+Collisions are problematic for security applications of checksum
+algorithms in general. Checksums are used, for example, in storing and
+verifying passwords: the cleartext password is never stored, and
+instead a checksum of it is computed and stored. To verify a later
+login attempt a new checksum is computed from the newly entered
+password from the attempt. If the checksums match, the password is
+accepted.[^passwords] This means that if an attacker can find _any_ input that
+gives the same output for the checksum algorithm used for password
+storage, they can log in as if they were a valid user, whether the
+password they have is the same as the real one.
+
+[^passwords]: In reality, storing passwords securely is much more
+ complicated than described here.
+
+For backups, and version control systems, collisions cause a different
+problem: they can prevent the correct content from being stored. If
+two files (or chunks) have the same checksum, only one will be stored.
+If the files have different content, this is a problem. A backup
+system should guard against this possibility.
+
+As an extreme and rare, but real, case consider a researcher of
+checksum algorithms. They've spent enormous effort to produce two
+distinct files that have the same checksum. They should be able make a
+backup of the files, and restore them, and not lose one. They should
+not have to know that their backup system uses the same checksum
+algorithm they are researching, and have to guard against the backup
+system getting the files confused. (Backup systems should be boring
+and just always work.)
+
+Attacks on security-sensitive cryptographic algorithms only get
+stronger by time. It is therefore necessary for Obnam to be able to
+easily change the checksum algorithm it uses, without disruption for
+user. To achieve this, Obnam does not use content-addressable storage.
+
+Obnam will (eventually, as this hasn't been implemented yet) allow
+storing multiple checksums for each chunk. It will use the strongest
+checksum available for a chunk. Over time, the checksums for chunks
+can be replaced with stronger ones. This will allow Obnam to migrate
+to a stronger algorithm when attacks against the current one become
+too scary.
+
+## On pull versus push backups
+
+Obnam only does push backups. This means the client runs on the host
+where the live data is, and sends it to the server.
+
+Backups could also be pulled, in that the server reaches out tot he
+host where the live data is, retrieves the data, and stores it on the
+server. Obnam does not do this, due to the hard requirement that live
+data never leaves its host in cleartext.
+
+The reason pull backups are of interest in many use cases is because
+they allow central administration of backups, which can simplify
+things a lot in a large organization. Central backup administration
+can be achieved with Obnam in a more complicated way: the installation
+and configuration of live data hosts is done in a central fashion
+using configuration management, and if necessary, backups can be
+triggered on each host by having the server reach out and run the
+Obnam client.
# File metadata
@@ -616,6 +805,10 @@ The server has the following API for managing chunks:
* `GET /chunks?sha256=xyzzy` &ndash; find chunks on the server whose
metadata indicates their contents has a given SHA256 checksum
* `GET /chunks?generation=true` &ndash; find generation chunks
+* `GET /chunks?data=True` &ndash; find chunks with file data
+ - this is meant for testing only
+ - it excludes generation chunks, and chunks used to store the
+ generation's SQLite file
HTTP status codes are used to indicate if a request succeeded or not,
using the customary meanings.
@@ -703,6 +896,7 @@ when I POST data.dat to /chunks, with chunk-meta: {"sha256":"abc"}
then HTTP status code is 201
and content-type is application/json
and the JSON body has a field chunk_id, henceforth ID
+and server has 1 file chunks
~~~
We must be able to retrieve it.
@@ -820,6 +1014,75 @@ and chunk-meta is {"sha256":"abc","generation":null,"ended":null}
and the body matches file data.dat
~~~
+# Acceptance criteria for the Obnam client
+
+The scenarios in chapter verify that the Obnam client works as it
+should, when it is used independently of an Obnam chunk server.
+
+## Client shows its configuration
+
+This scenario verifies that the client can show its current
+configuration, with the `obnam config` command. The configuration is
+stored as YAML, but the command outputs JSON, to make sure it doesn't
+just copy the configuration file to the output.
+
+~~~scenario
+given an installed obnam
+and file config.yaml
+and JSON file config.json converted from YAML file config.yaml
+when I run obnam --config config.yaml config
+then stdout, as JSON, matches file config.json
+~~~
+
+~~~{#config.yaml .file .yaml .numberLines}
+roots: [live]
+server_url: https://backup.example.com
+verify_tls_cert: true
+~~~
+
+
+## Client requires https
+
+This scenario verifies that the client rejects a configuration with a
+server URL using `http:` instead of `https:`.
+
+
+~~~scenario
+given an installed obnam
+and file http.yaml
+when I try to run obnam --config http.yaml config
+then command fails
+then stderr contains "https:"
+~~~
+
+~~~{#http.yaml .file .yaml .numberLines}
+roots: [live]
+server_url: http://backup.example.com
+verify_tls_cert: true
+~~~
+
+## Client refuses a self-signed certificate
+
+This scenario verifies that the client refuses to connect to a server
+if the server's TLS certificate is self-signed. The test server set up
+by the scenario uses self-signed certificates.
+
+~~~scenario
+given an installed obnam
+and a running chunk server
+and a client config based on ca-required.yaml
+and a file live/data.dat containing some random data
+when I try to run obnam --config ca-required.yaml backup
+then command fails
+then stderr contains "self signed certificate"
+~~~
+
+~~~{#ca-required.yaml .file .yaml .numberLines}
+verify_tls_cert: true
+roots: [live]
+~~~
+
+
# Acceptance criteria for Obnam as a whole
The scenarios in this chapter apply to Obnam as a whole: the client
@@ -848,7 +1111,8 @@ then files live.yaml and rest.yaml match
~~~
~~~{#smoke.yaml .file .yaml .numberLines}
-root: live
+verify_tls_cert: false
+roots: [live]
~~~
@@ -862,7 +1126,8 @@ anything.
All these scenarios use the following configuration file.
~~~{#metadata.yaml .file .yaml .numberLines}
-root: live
+verify_tls_cert: false
+roots: [live]
~~~
### Modification time
@@ -919,6 +1184,28 @@ given a manifest of the directory live restored in rest in rest.yaml
then files live.yaml and rest.yaml match
~~~
+## Set chunk size
+
+This scenario verifies that the user can set the chunk size in the
+configuration file. The chunk size only affects the chunks of live
+data.
+
+~~~scenario
+given an installed obnam
+given a running chunk server
+given a client config based on tiny-chunk-size.yaml
+given a file live/data.dat containing "abc"
+when I run obnam --config tiny-chunk-size.yaml backup
+then server has 3 file chunks
+~~~
+
+~~~{#tiny-chunk-size.yaml .file .yaml .numberLines}
+verify_tls_cert: false
+roots: [live]
+chunk_size: 1
+~~~
+
+
## Backup or not for the right reason
The decision of whether to back up a file or keep the version in the
@@ -996,6 +1283,28 @@ when I invoke obnam --config smoke.yaml get-chunk <GEN>
then command fails
~~~
+## Irregular files
+
+This scenario verifies that Obnam backs up and restores files that
+aren't regular files, directories, or symbolic links. Specifically,
+Unix domain sockets and named pipes (FIFOs). However, block and
+character device nodes are not tested, as that would require running
+the test suite with `root` permissions and that would be awkward.
+
+~~~scenario
+given an installed obnam
+and a running chunk server
+and a client config based on smoke.yaml
+and a file live/data.dat containing some random data
+and a Unix socket live/socket
+and a named pipe live/pipe
+and a manifest of the directory live in live.yaml
+when I run obnam --config smoke.yaml backup
+when I invoke obnam --config smoke.yaml restore latest rest
+given a manifest of the directory live restored in rest in rest.yaml
+then files live.yaml and rest.yaml match
+~~~
+
## Tricky filenames
Obnam needs to handle all filenames the underlying operating and file
@@ -1016,6 +1325,28 @@ given a manifest of the directory live restored in rest in rest.yaml
then files live.yaml and rest.yaml match
~~~
+## Unreadable file
+
+This scenario verifies that Obnam will back up all files of live data,
+even if one of them is unreadable. By inference, we assume this means
+other errors on individual files also won't end the backup
+prematurely.
+
+
+~~~scenario
+given an installed obnam
+and a running chunk server
+and a client config based on smoke.yaml
+and a file live/data.dat containing some random data
+and a file live/bad.dat containing some random data
+and file live/bad.dat has mode 000
+when I run obnam --config smoke.yaml backup
+then backup generation is GEN
+when I invoke obnam --config smoke.yaml restore <GEN> rest
+then file live/data.dat is restored to rest
+then file live/bad.dat is not restored to rest
+~~~
+
## Restore latest generation
This scenario verifies that the latest backup generation can be
@@ -1039,11 +1370,81 @@ given a manifest of the directory live restored in rest in rest.yaml
then files second.yaml and rest.yaml match
~~~
+## Back up multiple directories
+
+This scenario verifies that Obnam can back up more than one directory
+at a time.
+
+
+~~~scenario
+given an installed obnam
+and a running chunk server
+and a client config based on roots.yaml
+and a file live/one/data.dat containing some random data
+and a file live/two/data.dat containing some random data
+and a manifest of the directory live/one in one.yaml
+and a manifest of the directory live/two in two.yaml
+when I run obnam --config roots.yaml backup
+then backup generation is GEN
+when I invoke obnam --config roots.yaml restore <GEN> rest
+given a manifest of the directory live/one restored in rest in rest-one.yaml
+given a manifest of the directory live/two restored in rest in rest-two.yaml
+then files one.yaml and rest-one.yaml match
+then files two.yaml and rest-two.yaml match
+~~~
+
+~~~{#roots.yaml .file .yaml .numberLines}
+roots:
+- live/one
+- live/two
+~~~
+# Acceptance criteria for backup encryption
+
+This chapter outlines scenarios, to be implemented later, for
+verifying that Obnam properly encrypts the backups. These scenarios
+verify only encryption aspects of Obnam.
+
+## Backup without passphrase fails
+
+Verify that trying to backup without having set a passphrase fails
+with an error message that clearly identifies the lack of a
+passphrase.
+
+## A passphrase can be set
+
+Set a passphrase. Verify that it's stored in a file that is only
+readable by it owner. Verify that a backup can be made.
+
+## A passphrase stored insecurely is rejected
+
+Verify that a backup fails if the file where the passphrase is stored
+is readable by anyone but its owner. Verify that the error message
+explains that the backup failed due to the passphrase file insecurity.
+
+## The passphrase can be changed
+
+Verify that the passphrase can be changed and that backups made before
+the change can no longer be restored. (Later, this requirement will be
+re-evaluated, but this is simple and gets us started.)
+
+## The passphrase is not on server in cleartext
+
+Verify that after the passphrase has been set, and a backup has been
+made, the passphrase is not stored in cleartext on the server.
+
+## A backup is encrypted
+
+Verify that the backup repository does not contain the backed up data
+in cleartext.
+
<!-- -------------------------------------------------------------------- -->
+# Colophon
+
+This manual is edited in Markdown and typeset using [Subplot][].
---
title: "Obnam2&mdash;a backup system"
@@ -1053,6 +1454,7 @@ bindings:
- subplot/server.yaml
- subplot/client.yaml
- subplot/data.yaml
+ - subplot/vendored/files.yaml
- subplot/vendored/runcmd.yaml
template: python
functions:
@@ -1060,6 +1462,7 @@ functions:
- subplot/client.py
- subplot/data.py
- subplot/vendored/daemon.py
+ - subplot/vendored/files.py
- subplot/vendored/runcmd.py
classes:
- json
diff --git a/src/backup_progress.rs b/src/backup_progress.rs
index 6c1d3e6..6035eff 100644
--- a/src/backup_progress.rs
+++ b/src/backup_progress.rs
@@ -6,13 +6,33 @@ pub struct BackupProgress {
}
impl BackupProgress {
- pub fn new() -> Self {
+ pub fn initial() -> Self {
let progress = if true {
ProgressBar::new(0)
} else {
ProgressBar::hidden()
};
let parts = vec![
+ "initial backup",
+ "elapsed: {elapsed}",
+ "files: {pos}",
+ "current: {wide_msg}",
+ "{spinner}",
+ ];
+ progress.set_style(ProgressStyle::default_bar().template(&parts.join("\n")));
+ progress.enable_steady_tick(100);
+
+ Self { progress }
+ }
+
+ pub fn incremental() -> Self {
+ let progress = if true {
+ ProgressBar::new(0)
+ } else {
+ ProgressBar::hidden()
+ };
+ let parts = vec![
+ "incremental backup",
"{wide_bar}",
"elapsed: {elapsed}",
"files: {pos}/{len}",
@@ -25,6 +45,32 @@ impl BackupProgress {
Self { progress }
}
+ pub fn upload_generation() -> Self {
+ let progress = ProgressBar::new(0);
+ let parts = vec![
+ "uploading new generation metadata",
+ "elapsed: {elapsed}",
+ "{spinner}",
+ ];
+ progress.set_style(ProgressStyle::default_bar().template(&parts.join("\n")));
+ progress.enable_steady_tick(100);
+
+ Self { progress }
+ }
+
+ pub fn download_generation(gen_id: &str) -> Self {
+ let progress = ProgressBar::new(0);
+ let parts = vec!["{msg}", "elapsed: {elapsed}", "{spinner}"];
+ progress.set_style(ProgressStyle::default_bar().template(&parts.join("\n")));
+ progress.enable_steady_tick(100);
+ progress.set_message(&format!(
+ "downloading previous generation metadata: {}",
+ gen_id
+ ));
+
+ Self { progress }
+ }
+
pub fn files_in_previous_generation(&self, count: u64) {
self.progress.set_length(count);
}
@@ -35,6 +81,9 @@ impl BackupProgress {
pub fn found_live_file(&self, filename: &Path) {
self.progress.inc(1);
+ if self.progress.length() < self.progress.position() {
+ self.progress.set_length(self.progress.position());
+ }
self.progress
.set_message(&format!("{}", filename.display()));
}
diff --git a/src/backup_reason.rs b/src/backup_reason.rs
index 218857c..f785dea 100644
--- a/src/backup_reason.rs
+++ b/src/backup_reason.rs
@@ -8,7 +8,9 @@ pub enum Reason {
IsNew,
Changed,
Unchanged,
- Error,
+ GenerationLookupError,
+ FileError,
+ Unknown,
}
impl Reason {
@@ -18,7 +20,9 @@ impl Reason {
"new" => Reason::IsNew,
"changed" => Reason::Changed,
"unchanged" => Reason::Unchanged,
- _ => Reason::Error,
+ "genlookuperror" => Reason::GenerationLookupError,
+ "fileerror" => Reason::FileError,
+ _ => Reason::Unknown,
}
}
}
@@ -39,7 +43,9 @@ impl fmt::Display for Reason {
Reason::IsNew => "new",
Reason::Changed => "changed",
Reason::Unchanged => "unchanged",
- Reason::Error => "error",
+ Reason::GenerationLookupError => "genlookuperror",
+ Reason::FileError => "fileerror",
+ Reason::Unknown => "unknown",
};
write!(f, "{}", reason)
}
diff --git a/src/backup_run.rs b/src/backup_run.rs
index e3bfc5a..ea5888a 100644
--- a/src/backup_run.rs
+++ b/src/backup_run.rs
@@ -1,82 +1,139 @@
use crate::backup_progress::BackupProgress;
use crate::backup_reason::Reason;
use crate::chunkid::ChunkId;
-use crate::client::{BackupClient, ClientConfig};
+use crate::client::{BackupClient, ClientConfig, ClientError};
+use crate::error::ObnamError;
use crate::fsentry::FilesystemEntry;
-use crate::generation::LocalGeneration;
+use crate::fsiter::{FsIterError, FsIterResult};
+use crate::generation::{LocalGeneration, LocalGenerationError};
use crate::policy::BackupPolicy;
use log::{info, warn};
+use std::path::Path;
-pub struct BackupRun {
- client: BackupClient,
- policy: BackupPolicy,
+pub struct InitialBackup<'a> {
+ client: &'a BackupClient,
buffer_size: usize,
progress: BackupProgress,
}
-impl BackupRun {
- pub fn new(config: &ClientConfig, buffer_size: usize) -> anyhow::Result<Self> {
- let client = BackupClient::new(&config.server_url)?;
- let policy = BackupPolicy::new();
- let progress = BackupProgress::new();
+pub struct IncrementalBackup<'a> {
+ client: &'a BackupClient,
+ policy: BackupPolicy,
+ buffer_size: usize,
+ progress: Option<BackupProgress>,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum BackupError {
+ #[error(transparent)]
+ ClientError(#[from] ClientError),
+
+ #[error(transparent)]
+ FsIterError(#[from] FsIterError),
+
+ #[error(transparent)]
+ LocalGenerationError(#[from] LocalGenerationError),
+}
+
+pub type BackupResult<T> = Result<T, BackupError>;
+
+impl<'a> InitialBackup<'a> {
+ pub fn new(config: &ClientConfig, client: &'a BackupClient) -> BackupResult<Self> {
+ let progress = BackupProgress::initial();
Ok(Self {
client,
- policy,
- buffer_size,
+ buffer_size: config.chunk_size,
progress,
})
}
- pub fn client(&self) -> &BackupClient {
- &self.client
+ pub fn drop(&self) {
+ &self.progress.finish();
}
- pub fn progress(&self) -> &BackupProgress {
- &self.progress
- }
-
- pub fn backup_file_initially(
+ pub fn backup(
&self,
- entry: anyhow::Result<FilesystemEntry>,
- ) -> anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)> {
+ entry: FsIterResult<FilesystemEntry>,
+ ) -> BackupResult<(FilesystemEntry, Vec<ChunkId>, Reason)> {
match entry {
- Err(err) => Err(err.into()),
+ Err(err) => {
+ warn!("backup: there was a problem: {:?}", err);
+ self.progress.found_problem();
+ Err(err.into())
+ }
Ok(entry) => {
let path = &entry.pathbuf();
info!("backup: {}", path.display());
self.progress.found_live_file(path);
- let ids = self
- .client
- .upload_filesystem_entry(&entry, self.buffer_size)?;
- Ok((entry.clone(), ids, Reason::IsNew))
+ backup_file(&self.client, &entry, &path, self.buffer_size, Reason::IsNew)
}
}
}
+}
+
+impl<'a> IncrementalBackup<'a> {
+ pub fn new(config: &ClientConfig, client: &'a BackupClient) -> BackupResult<Self> {
+ let policy = BackupPolicy::new();
+ Ok(Self {
+ client,
+ policy,
+ buffer_size: config.chunk_size,
+ progress: None,
+ })
+ }
+
+ pub fn start_backup(&mut self, old: &LocalGeneration) -> Result<(), ObnamError> {
+ let progress = BackupProgress::incremental();
+ progress.files_in_previous_generation(old.file_count()? as u64);
+ self.progress = Some(progress);
+ Ok(())
+ }
+
+ pub fn client(&self) -> &BackupClient {
+ self.client
+ }
+
+ pub fn drop(&self) {
+ if let Some(progress) = &self.progress {
+ progress.finish();
+ }
+ }
+
+ pub fn fetch_previous_generation(
+ &self,
+ genid: &str,
+ oldname: &Path,
+ ) -> Result<LocalGeneration, ObnamError> {
+ let progress = BackupProgress::download_generation(genid);
+ let old = self.client().fetch_generation(genid, &oldname)?;
+ progress.finish();
+ Ok(old)
+ }
- pub fn backup_file_incrementally(
+ pub fn backup(
&self,
- entry: anyhow::Result<FilesystemEntry>,
+ entry: FsIterResult<FilesystemEntry>,
old: &LocalGeneration,
- ) -> anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)> {
+ ) -> BackupResult<(FilesystemEntry, Vec<ChunkId>, Reason)> {
match entry {
Err(err) => {
warn!("backup: {}", err);
- self.progress.found_problem();
- Err(err)
+ self.found_problem();
+ Err(BackupError::FsIterError(err))
}
Ok(entry) => {
let path = &entry.pathbuf();
info!("backup: {}", path.display());
- self.progress.found_live_file(path);
+ self.found_live_file(path);
let reason = self.policy.needs_backup(&old, &entry);
match reason {
- Reason::IsNew | Reason::Changed | Reason::Error => {
- let ids = self
- .client
- .upload_filesystem_entry(&entry, self.buffer_size)?;
- Ok((entry.clone(), ids, reason))
+ Reason::IsNew
+ | Reason::Changed
+ | Reason::GenerationLookupError
+ | Reason::Unknown => {
+ backup_file(&self.client, &entry, &path, self.buffer_size, reason)
}
- Reason::Unchanged | Reason::Skipped => {
+ Reason::Unchanged | Reason::Skipped | Reason::FileError => {
let fileno = old.get_fileno(&entry.pathbuf())?;
let ids = if let Some(fileno) = fileno {
old.chunkids(fileno)?
@@ -89,4 +146,33 @@ impl BackupRun {
}
}
}
+
+ fn found_live_file(&self, path: &Path) {
+ if let Some(progress) = &self.progress {
+ progress.found_live_file(path);
+ }
+ }
+
+ fn found_problem(&self) {
+ if let Some(progress) = &self.progress {
+ progress.found_problem();
+ }
+ }
+}
+
+fn backup_file(
+ client: &BackupClient,
+ entry: &FilesystemEntry,
+ path: &Path,
+ chunk_size: usize,
+ reason: Reason,
+) -> BackupResult<(FilesystemEntry, Vec<ChunkId>, Reason)> {
+ let ids = client.upload_filesystem_entry(&entry, chunk_size);
+ match ids {
+ Err(err) => {
+ warn!("error backing up {}, skipping it: {}", path.display(), err);
+ Ok((entry.clone(), vec![], Reason::FileError))
+ }
+ Ok(ids) => Ok((entry.clone(), ids, reason)),
+ }
}
diff --git a/src/bin/obnam-server.rs b/src/bin/obnam-server.rs
index 19f2e99..d4513fa 100644
--- a/src/bin/obnam-server.rs
+++ b/src/bin/obnam-server.rs
@@ -4,11 +4,12 @@ use obnam::chunk::DataChunk;
use obnam::chunkid::ChunkId;
use obnam::chunkmeta::ChunkMeta;
use obnam::indexedstore::IndexedStore;
-use serde::{Deserialize, Serialize};
+use obnam::server::{Config, ConfigError};
+use serde::Serialize;
use std::collections::HashMap;
use std::default::Default;
use std::net::{SocketAddr, ToSocketAddrs};
-use std::path::{Path, PathBuf};
+use std::path::PathBuf;
use std::sync::Arc;
use structopt::StructOpt;
use tokio::sync::Mutex;
@@ -82,51 +83,6 @@ async fn main() -> anyhow::Result<()> {
Ok(())
}
-#[derive(Debug, Deserialize, Clone)]
-pub struct Config {
- pub chunks: PathBuf,
- pub address: String,
- pub tls_key: PathBuf,
- pub tls_cert: PathBuf,
-}
-
-#[derive(Debug, thiserror::Error)]
-enum ConfigError {
- #[error("Directory for chunks {0} does not exist")]
- ChunksDirNotFound(PathBuf),
-
- #[error("TLS certificate {0} does not exist")]
- TlsCertNotFound(PathBuf),
-
- #[error("TLS key {0} does not exist")]
- TlsKeyNotFound(PathBuf),
-
- #[error("server address can't be resolved")]
- BadServerAddress,
-}
-
-impl Config {
- pub fn read_config(filename: &Path) -> anyhow::Result<Config> {
- let config = std::fs::read_to_string(filename)?;
- let config: Config = serde_yaml::from_str(&config)?;
- config.check()?;
- Ok(config)
- }
-
- pub fn check(&self) -> anyhow::Result<()> {
- if !self.chunks.exists() {
- return Err(ConfigError::ChunksDirNotFound(self.chunks.clone()).into());
- }
- if !self.tls_cert.exists() {
- return Err(ConfigError::TlsCertNotFound(self.tls_cert.clone()).into());
- }
- if !self.tls_key.exists() {
- return Err(ConfigError::TlsKeyNotFound(self.tls_key.clone()).into());
- }
- Ok(())
- }
-}
-
pub async fn create_chunk(
store: Arc<Mutex<IndexedStore>>,
meta: String,
@@ -188,6 +144,8 @@ pub async fn search_chunks(
}
if key == "generation" && value == "true" {
store.find_generations().expect("SQL lookup failed")
+ } else if key == "data" && value == "true" {
+ store.find_file_chunks().expect("SQL lookup failed")
} else if key == "sha256" {
store.find_by_sha256(value).expect("SQL lookup failed")
} else {
diff --git a/src/bin/obnam.rs b/src/bin/obnam.rs
index e9f30ca..c163695 100644
--- a/src/bin/obnam.rs
+++ b/src/bin/obnam.rs
@@ -2,12 +2,10 @@ use log::{debug, error, info, LevelFilter};
use log4rs::append::file::FileAppender;
use log4rs::config::{Appender, Config, Logger, Root};
use obnam::client::ClientConfig;
-use obnam::cmd::{backup, get_chunk, list, list_files, restore, show_generation};
+use obnam::cmd::{backup, get_chunk, list, list_files, restore, show_config, show_generation};
use std::path::{Path, PathBuf};
use structopt::StructOpt;
-const BUFFER_SIZE: usize = 1024 * 1024;
-
fn main() -> anyhow::Result<()> {
let opt = Opt::from_args();
let config_file = match opt.config {
@@ -15,26 +13,25 @@ fn main() -> anyhow::Result<()> {
Some(ref path) => path.to_path_buf(),
};
let config = ClientConfig::read_config(&config_file)?;
- if let Some(ref log) = config.log {
- setup_logging(&log)?;
- }
+ setup_logging(&config.log)?;
info!("client starts");
debug!("{:?}", opt);
let result = match opt.cmd {
- Command::Backup => backup(&config, BUFFER_SIZE),
+ Command::Backup => backup(&config),
Command::List => list(&config),
Command::ShowGeneration { gen_id } => show_generation(&config, &gen_id),
Command::ListFiles { gen_id } => list_files(&config, &gen_id),
Command::Restore { gen_id, to } => restore(&config, &gen_id, &to),
Command::GetChunk { chunk_id } => get_chunk(&config, &chunk_id),
+ Command::Config => show_config(&config),
};
if let Err(ref e) = result {
- error!("{}", e);
+ error!("command failed: {}", e);
eprintln!("ERROR: {}", e);
- return result;
+ result?
}
info!("client ends successfully");
@@ -84,6 +81,7 @@ enum Command {
#[structopt()]
chunk_id: String,
},
+ Config,
}
fn setup_logging(filename: &Path) -> anyhow::Result<()> {
diff --git a/src/chunk.rs b/src/chunk.rs
index 4917b60..a67ed8c 100644
--- a/src/chunk.rs
+++ b/src/chunk.rs
@@ -30,12 +30,25 @@ pub struct GenerationChunk {
chunk_ids: Vec<ChunkId>,
}
+/// All the errors that may be returned for `GenerationChunk` operations.
+#[derive(Debug, thiserror::Error)]
+pub enum GenerationChunkError {
+ #[error(transparent)]
+ Utf8Error(#[from] std::str::Utf8Error),
+
+ #[error(transparent)]
+ SerdeJsonError(#[from] serde_json::Error),
+}
+
+/// A result from a chunk operation.
+pub type GenerationChunkResult<T> = Result<T, GenerationChunkError>;
+
impl GenerationChunk {
pub fn new(chunk_ids: Vec<ChunkId>) -> Self {
Self { chunk_ids }
}
- pub fn from_data_chunk(chunk: &DataChunk) -> anyhow::Result<Self> {
+ pub fn from_data_chunk(chunk: &DataChunk) -> GenerationChunkResult<Self> {
let data = chunk.data();
let data = std::str::from_utf8(data)?;
Ok(serde_json::from_str(data)?)
@@ -53,7 +66,7 @@ impl GenerationChunk {
self.chunk_ids.iter()
}
- pub fn to_data_chunk(&self) -> anyhow::Result<DataChunk> {
+ pub fn to_data_chunk(&self) -> GenerationChunkResult<DataChunk> {
let json = serde_json::to_string(self)?;
Ok(DataChunk::new(json.as_bytes().to_vec()))
}
diff --git a/src/chunker.rs b/src/chunker.rs
index 145b1db..f424833 100644
--- a/src/chunker.rs
+++ b/src/chunker.rs
@@ -9,6 +9,14 @@ pub struct Chunker {
handle: std::fs::File,
}
+#[derive(Debug, thiserror::Error)]
+pub enum ChunkerError {
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+}
+
+pub type ChunkerResult<T> = Result<T, ChunkerError>;
+
impl Chunker {
pub fn new(chunk_size: usize, handle: std::fs::File) -> Self {
let mut buf = vec![];
@@ -20,7 +28,7 @@ impl Chunker {
}
}
- pub fn read_chunk(&mut self) -> anyhow::Result<Option<(ChunkMeta, DataChunk)>> {
+ pub fn read_chunk(&mut self) -> ChunkerResult<Option<(ChunkMeta, DataChunk)>> {
let mut used = 0;
loop {
@@ -44,9 +52,9 @@ impl Chunker {
}
impl Iterator for Chunker {
- type Item = anyhow::Result<(ChunkMeta, DataChunk)>;
+ type Item = ChunkerResult<(ChunkMeta, DataChunk)>;
- fn next(&mut self) -> Option<anyhow::Result<(ChunkMeta, DataChunk)>> {
+ fn next(&mut self) -> Option<ChunkerResult<(ChunkMeta, DataChunk)>> {
match self.read_chunk() {
Ok(None) => None,
Ok(Some((meta, chunk))) => Some(Ok((meta, chunk))),
diff --git a/src/chunkmeta.rs b/src/chunkmeta.rs
index 37e2ed5..fc5ffff 100644
--- a/src/chunkmeta.rs
+++ b/src/chunkmeta.rs
@@ -67,7 +67,10 @@ impl ChunkMeta {
/// Is this a generation chunk?
pub fn is_generation(&self) -> bool {
- matches!(self.generation, Some(true))
+ match self.generation {
+ Some(true) => true,
+ _ => false,
+ }
}
/// When did this generation end?
diff --git a/src/client.rs b/src/client.rs
index 515b8c9..7d1613c 100644
--- a/src/client.rs
+++ b/src/client.rs
@@ -1,39 +1,94 @@
use crate::checksummer::sha256;
use crate::chunk::DataChunk;
-use crate::chunk::GenerationChunk;
-use crate::chunker::Chunker;
+use crate::chunk::{GenerationChunk, GenerationChunkError};
+use crate::chunker::{Chunker, ChunkerError};
use crate::chunkid::ChunkId;
use crate::chunkmeta::ChunkMeta;
-use crate::error::ObnamError;
use crate::fsentry::{FilesystemEntry, FilesystemKind};
-use crate::generation::{FinishedGeneration, LocalGeneration};
+use crate::generation::{FinishedGeneration, LocalGeneration, LocalGenerationError};
use crate::genlist::GenerationList;
-use anyhow::Context;
+use bytesize::MIB;
use chrono::{DateTime, Local};
use log::{debug, error, info, trace};
use reqwest::blocking::Client;
-use serde::Deserialize;
+use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fs::File;
use std::io::prelude::*;
use std::path::{Path, PathBuf};
-#[derive(Debug, Deserialize, Clone)]
+const DEFAULT_CHUNK_SIZE: usize = MIB as usize;
+const DEVNULL: &str = "/dev/null";
+
+#[derive(Debug, Serialize, Deserialize, Clone)]
+struct TentativeClientConfig {
+ server_url: String,
+ verify_tls_cert: Option<bool>,
+ chunk_size: Option<usize>,
+ roots: Vec<PathBuf>,
+ log: Option<PathBuf>,
+}
+
+#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct ClientConfig {
pub server_url: String,
- pub root: PathBuf,
- pub log: Option<PathBuf>,
+ pub verify_tls_cert: bool,
+ pub chunk_size: usize,
+ pub roots: Vec<PathBuf>,
+ pub log: PathBuf,
}
+#[derive(Debug, thiserror::Error)]
+pub enum ClientConfigError {
+ #[error("server_url is empty")]
+ ServerUrlIsEmpty,
+
+ #[error("No backup roots in config; at least one is needed")]
+ NoBackupRoot,
+
+ #[error("server URL doesn't use https: {0}")]
+ NotHttps(String),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+
+ #[error(transparent)]
+ SerdeYamlError(#[from] serde_yaml::Error),
+}
+
+pub type ClientConfigResult<T> = Result<T, ClientConfigError>;
+
impl ClientConfig {
- pub fn read_config(filename: &Path) -> anyhow::Result<Self> {
+ pub fn read_config(filename: &Path) -> ClientConfigResult<Self> {
trace!("read_config: filename={:?}", filename);
- let config = std::fs::read_to_string(filename)
- .with_context(|| format!("reading configuration file {}", filename.display()))?;
- let config = serde_yaml::from_str(&config)?;
+ let config = std::fs::read_to_string(filename)?;
+ let tentative: TentativeClientConfig = serde_yaml::from_str(&config)?;
+
+ let config = ClientConfig {
+ server_url: tentative.server_url,
+ roots: tentative.roots,
+ verify_tls_cert: tentative.verify_tls_cert.or(Some(false)).unwrap(),
+ chunk_size: tentative.chunk_size.or(Some(DEFAULT_CHUNK_SIZE)).unwrap(),
+ log: tentative.log.or(Some(PathBuf::from(DEVNULL))).unwrap(),
+ };
+
+ config.check()?;
Ok(config)
}
+
+ fn check(&self) -> Result<(), ClientConfigError> {
+ if self.server_url.is_empty() {
+ return Err(ClientConfigError::ServerUrlIsEmpty);
+ }
+ if !self.server_url.starts_with("https://") {
+ return Err(ClientConfigError::NotHttps(self.server_url.to_string()));
+ }
+ if self.roots.is_empty() {
+ return Err(ClientConfigError::NoBackupRoot);
+ }
+ Ok(())
+ }
}
#[derive(Debug, thiserror::Error)]
@@ -46,21 +101,54 @@ pub enum ClientError {
#[error("Server does not have generation {0}")]
GenerationNotFound(String),
+
+ #[error(transparent)]
+ GenerationChunkError(#[from] GenerationChunkError),
+
+ #[error(transparent)]
+ LocalGenerationError(#[from] LocalGenerationError),
+
+ #[error(transparent)]
+ ChunkerError(#[from] ChunkerError),
+
+ #[error("Server response did not have a 'chunk-meta' header for chunk {0}")]
+ NoChunkMeta(ChunkId),
+
+ #[error("Wrong checksum for chunk {0}, got {1}, expected {2}")]
+ WrongChecksum(ChunkId, String, String),
+
+ #[error(transparent)]
+ ReqwestError(#[from] reqwest::Error),
+
+ #[error(transparent)]
+ ReqwestToStrError(#[from] reqwest::header::ToStrError),
+
+ #[error(transparent)]
+ SerdeJsonError(#[from] serde_json::Error),
+
+ #[error(transparent)]
+ SerdeYamlError(#[from] serde_yaml::Error),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
}
+pub type ClientResult<T> = Result<T, ClientError>;
+
pub struct BackupClient {
client: Client,
base_url: String,
}
impl BackupClient {
- pub fn new(base_url: &str) -> anyhow::Result<Self> {
+ pub fn new(config: &ClientConfig) -> ClientResult<Self> {
+ info!("creating backup client with config: {:#?}", config);
let client = Client::builder()
- .danger_accept_invalid_certs(true)
+ .danger_accept_invalid_certs(!config.verify_tls_cert)
.build()?;
Ok(Self {
client,
- base_url: base_url.to_string(),
+ base_url: config.server_url.to_string(),
})
}
@@ -68,19 +156,23 @@ impl BackupClient {
&self,
e: &FilesystemEntry,
size: usize,
- ) -> anyhow::Result<Vec<ChunkId>> {
- info!("upload entry: {:?}", e);
+ ) -> ClientResult<Vec<ChunkId>> {
+ let path = e.pathbuf();
+ info!("uploading {:?}", path);
let ids = match e.kind() {
- FilesystemKind::Regular => self.read_file(e.pathbuf(), size)?,
+ FilesystemKind::Regular => self.read_file(&path, size)?,
FilesystemKind::Directory => vec![],
FilesystemKind::Symlink => vec![],
+ FilesystemKind::Socket => vec![],
+ FilesystemKind::Fifo => vec![],
};
+ info!("upload OK for {:?}", path);
Ok(ids)
}
- pub fn upload_generation(&self, filename: &Path, size: usize) -> anyhow::Result<ChunkId> {
+ pub fn upload_generation(&self, filename: &Path, size: usize) -> ClientResult<ChunkId> {
info!("upload SQLite {}", filename.display());
- let ids = self.read_file(filename.to_path_buf(), size)?;
+ let ids = self.read_file(filename, size)?;
let gen = GenerationChunk::new(ids);
let data = gen.to_data_chunk()?;
let meta = ChunkMeta::new_generation(&sha256(data.data()), &current_timestamp());
@@ -89,7 +181,7 @@ impl BackupClient {
Ok(gen_id)
}
- fn read_file(&self, filename: PathBuf, size: usize) -> anyhow::Result<Vec<ChunkId>> {
+ fn read_file(&self, filename: &Path, size: usize) -> ClientResult<Vec<ChunkId>> {
info!("upload file {}", filename.display());
let file = std::fs::File::open(filename)?;
let chunker = Chunker::new(size, file);
@@ -105,7 +197,7 @@ impl BackupClient {
format!("{}/chunks", self.base_url())
}
- pub fn has_chunk(&self, meta: &ChunkMeta) -> anyhow::Result<Option<ChunkId>> {
+ pub fn has_chunk(&self, meta: &ChunkMeta) -> ClientResult<Option<ChunkId>> {
trace!("has_chunk: url={:?}", self.base_url());
let req = self
.client
@@ -136,7 +228,7 @@ impl BackupClient {
Ok(has)
}
- pub fn upload_chunk(&self, meta: ChunkMeta, chunk: DataChunk) -> anyhow::Result<ChunkId> {
+ pub fn upload_chunk(&self, meta: ChunkMeta, chunk: DataChunk) -> ClientResult<ChunkId> {
let res = self
.client
.post(&self.chunks_url())
@@ -155,11 +247,7 @@ impl BackupClient {
Ok(chunk_id)
}
- pub fn upload_gen_chunk(
- &self,
- meta: ChunkMeta,
- gen: GenerationChunk,
- ) -> anyhow::Result<ChunkId> {
+ pub fn upload_gen_chunk(&self, meta: ChunkMeta, gen: GenerationChunk) -> ClientResult<ChunkId> {
let res = self
.client
.post(&self.chunks_url())
@@ -178,7 +266,7 @@ impl BackupClient {
Ok(chunk_id)
}
- pub fn upload_new_file_chunks(&self, chunker: Chunker) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn upload_new_file_chunks(&self, chunker: Chunker) -> ClientResult<Vec<ChunkId>> {
let mut chunk_ids = vec![];
for item in chunker {
let (meta, chunk) = item?;
@@ -195,7 +283,7 @@ impl BackupClient {
Ok(chunk_ids)
}
- pub fn list_generations(&self) -> anyhow::Result<GenerationList> {
+ pub fn list_generations(&self) -> ClientResult<GenerationList> {
let url = format!("{}?generation=true", &self.chunks_url());
trace!("list_generations: url={:?}", url);
let req = self.client.get(&url).build()?;
@@ -212,7 +300,7 @@ impl BackupClient {
Ok(GenerationList::new(finished))
}
- pub fn fetch_chunk(&self, chunk_id: &ChunkId) -> anyhow::Result<DataChunk> {
+ pub fn fetch_chunk(&self, chunk_id: &ChunkId) -> ClientResult<DataChunk> {
info!("fetch chunk {}", chunk_id);
let url = format!("{}/{}", &self.chunks_url(), chunk_id);
@@ -227,7 +315,7 @@ impl BackupClient {
let headers = res.headers();
let meta = headers.get("chunk-meta");
if meta.is_none() {
- let err = ObnamError::NoChunkMeta(chunk_id.clone());
+ let err = ClientError::NoChunkMeta(chunk_id.clone());
error!("fetching chunk {} failed: {}", chunk_id, err);
return Err(err.into());
}
@@ -241,7 +329,7 @@ impl BackupClient {
let actual = sha256(&body);
if actual != meta.sha256() {
let err =
- ObnamError::WrongChecksum(chunk_id.clone(), actual, meta.sha256().to_string());
+ ClientError::WrongChecksum(chunk_id.clone(), actual, meta.sha256().to_string());
error!("fetching chunk {} failed: {}", chunk_id, err);
return Err(err.into());
}
@@ -251,14 +339,14 @@ impl BackupClient {
Ok(chunk)
}
- fn fetch_generation_chunk(&self, gen_id: &str) -> anyhow::Result<GenerationChunk> {
+ fn fetch_generation_chunk(&self, gen_id: &str) -> ClientResult<GenerationChunk> {
let chunk_id = ChunkId::from_str(gen_id);
let chunk = self.fetch_chunk(&chunk_id)?;
let gen = GenerationChunk::from_data_chunk(&chunk)?;
Ok(gen)
}
- pub fn fetch_generation(&self, gen_id: &str, dbname: &Path) -> anyhow::Result<LocalGeneration> {
+ pub fn fetch_generation(&self, gen_id: &str, dbname: &Path) -> ClientResult<LocalGeneration> {
let gen = self.fetch_generation_chunk(gen_id)?;
// Fetch the SQLite file, storing it in the named file.
diff --git a/src/cmd/backup.rs b/src/cmd/backup.rs
index da7298f..94b8761 100644
--- a/src/cmd/backup.rs
+++ b/src/cmd/backup.rs
@@ -1,15 +1,20 @@
-use crate::backup_run::BackupRun;
-use crate::client::ClientConfig;
+use crate::backup_progress::BackupProgress;
+use crate::backup_run::{IncrementalBackup, InitialBackup};
+use crate::chunkid::ChunkId;
+use crate::client::{BackupClient, ClientConfig};
+use crate::error::ObnamError;
use crate::fsiter::FsIterator;
use crate::generation::NascentGeneration;
+use bytesize::MIB;
use log::info;
+use std::path::{Path, PathBuf};
use std::time::SystemTime;
use tempfile::NamedTempFile;
-pub fn backup(config: &ClientConfig, buffer_size: usize) -> anyhow::Result<()> {
- let runtime = SystemTime::now();
+const SQLITE_CHUNK_SIZE: usize = MIB as usize;
- let run = BackupRun::new(config, buffer_size)?;
+pub fn backup(config: &ClientConfig) -> Result<(), ObnamError> {
+ let runtime = SystemTime::now();
// Create a named temporary file. We don't meed the open file
// handle, so we discard that.
@@ -27,39 +32,74 @@ pub fn backup(config: &ClientConfig, buffer_size: usize) -> anyhow::Result<()> {
dbname
};
- let genlist = run.client().list_generations()?;
- let file_count = {
- let iter = FsIterator::new(&config.root);
- let mut new = NascentGeneration::create(&newname)?;
-
- match genlist.resolve("latest") {
- None => {
- info!("fresh backup without a previous generation");
- new.insert_iter(iter.map(|entry| run.backup_file_initially(entry)))?;
- }
- Some(old) => {
- info!("incremental backup based on {}", old);
- let old = run.client().fetch_generation(&old, &oldname)?;
- run.progress()
- .files_in_previous_generation(old.file_count()? as u64);
- new.insert_iter(iter.map(|entry| run.backup_file_incrementally(entry, &old)))?;
- }
+ let client = BackupClient::new(config)?;
+ let genlist = client.list_generations()?;
+ let file_count = match genlist.resolve("latest") {
+ Err(_) => {
+ let run = InitialBackup::new(config, &client)?;
+ let count = initial_backup(&config.roots, &newname, &run)?;
+ count
+ }
+ Ok(old) => {
+ let mut run = IncrementalBackup::new(config, &client)?;
+ let count = incremental_backup(&old, &config.roots, &newname, &oldname, &mut run)?;
+ count
}
- run.progress().finish();
- new.file_count()
};
// Upload the SQLite file, i.e., the named temporary file, which
// still exists, since we persisted it above.
- let gen_id = run.client().upload_generation(&newname, buffer_size)?;
- println!("status: OK");
- println!("duration: {}", runtime.elapsed()?.as_secs());
- println!("file-count: {}", file_count);
- println!("generation-id: {}", gen_id);
+ let progress = BackupProgress::upload_generation();
+ let gen_id = client.upload_generation(&newname, SQLITE_CHUNK_SIZE)?;
+ progress.finish();
// Delete the temporary file.q
std::fs::remove_file(&newname)?;
std::fs::remove_file(&oldname)?;
+ report_stats(&runtime, file_count, &gen_id)?;
+
+ Ok(())
+}
+
+fn report_stats(runtime: &SystemTime, file_count: i64, gen_id: &ChunkId) -> Result<(), ObnamError> {
+ println!("status: OK");
+ println!("duration: {}", runtime.elapsed()?.as_secs());
+ println!("file-count: {}", file_count);
+ println!("generation-id: {}", gen_id);
Ok(())
}
+
+fn initial_backup(
+ roots: &[PathBuf],
+ newname: &Path,
+ run: &InitialBackup,
+) -> Result<i64, ObnamError> {
+ info!("fresh backup without a previous generation");
+
+ let mut new = NascentGeneration::create(&newname)?;
+ for root in roots {
+ let iter = FsIterator::new(root);
+ new.insert_iter(iter.map(|entry| run.backup(entry)))?;
+ }
+ Ok(new.file_count())
+}
+
+fn incremental_backup(
+ old: &str,
+ roots: &[PathBuf],
+ newname: &Path,
+ oldname: &Path,
+ run: &mut IncrementalBackup,
+) -> Result<i64, ObnamError> {
+ info!("incremental backup based on {}", old);
+
+ let old = run.fetch_previous_generation(old, oldname)?;
+ run.start_backup(&old)?;
+ let mut new = NascentGeneration::create(&newname)?;
+ for root in roots {
+ let iter = FsIterator::new(root);
+ new.insert_iter(iter.map(|entry| run.backup(entry, &old)))?;
+ }
+ Ok(new.file_count())
+}
diff --git a/src/cmd/get_chunk.rs b/src/cmd/get_chunk.rs
index bf653ff..385c4d5 100644
--- a/src/cmd/get_chunk.rs
+++ b/src/cmd/get_chunk.rs
@@ -1,10 +1,11 @@
use crate::chunkid::ChunkId;
use crate::client::BackupClient;
use crate::client::ClientConfig;
+use crate::error::ObnamError;
use std::io::{stdout, Write};
-pub fn get_chunk(config: &ClientConfig, chunk_id: &str) -> anyhow::Result<()> {
- let client = BackupClient::new(&config.server_url)?;
+pub fn get_chunk(config: &ClientConfig, chunk_id: &str) -> Result<(), ObnamError> {
+ let client = BackupClient::new(config)?;
let chunk_id: ChunkId = chunk_id.parse().unwrap();
let chunk = client.fetch_chunk(&chunk_id)?;
diff --git a/src/cmd/list.rs b/src/cmd/list.rs
index 8766e34..a3f059b 100644
--- a/src/cmd/list.rs
+++ b/src/cmd/list.rs
@@ -1,7 +1,8 @@
use crate::client::{BackupClient, ClientConfig};
+use crate::error::ObnamError;
-pub fn list(config: &ClientConfig) -> anyhow::Result<()> {
- let client = BackupClient::new(&config.server_url)?;
+pub fn list(config: &ClientConfig) -> Result<(), ObnamError> {
+ let client = BackupClient::new(config)?;
let generations = client.list_generations()?;
for finished in generations.iter() {
diff --git a/src/cmd/list_files.rs b/src/cmd/list_files.rs
index a69c3df..ec3e52e 100644
--- a/src/cmd/list_files.rs
+++ b/src/cmd/list_files.rs
@@ -5,7 +5,7 @@ use crate::error::ObnamError;
use crate::fsentry::{FilesystemEntry, FilesystemKind};
use tempfile::NamedTempFile;
-pub fn list_files(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<()> {
+pub fn list_files(config: &ClientConfig, gen_ref: &str) -> Result<(), ObnamError> {
// Create a named temporary file. We don't meed the open file
// handle, so we discard that.
let dbname = {
@@ -14,13 +14,10 @@ pub fn list_files(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<()> {
dbname
};
- let client = BackupClient::new(&config.server_url)?;
+ let client = BackupClient::new(config)?;
let genlist = client.list_generations()?;
- let gen_id: String = match genlist.resolve(gen_ref) {
- None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()),
- Some(id) => id,
- };
+ let gen_id: String = genlist.resolve(gen_ref)?;
let gen = client.fetch_generation(&gen_id, &dbname)?;
for file in gen.files()? {
@@ -38,6 +35,8 @@ fn format_entry(e: &FilesystemEntry, reason: Reason) -> String {
FilesystemKind::Regular => "-",
FilesystemKind::Directory => "d",
FilesystemKind::Symlink => "l",
+ FilesystemKind::Socket => "s",
+ FilesystemKind::Fifo => "p",
};
format!("{} {} ({})", kind, e.pathbuf().display(), reason)
}
diff --git a/src/cmd/mod.rs b/src/cmd/mod.rs
index 8f08668..fc517be 100644
--- a/src/cmd/mod.rs
+++ b/src/cmd/mod.rs
@@ -15,3 +15,6 @@ pub use get_chunk::get_chunk;
pub mod show_gen;
pub use show_gen::show_generation;
+
+pub mod show_config;
+pub use show_config::show_config;
diff --git a/src/cmd/restore.rs b/src/cmd/restore.rs
index d783a70..b394d7d 100644
--- a/src/cmd/restore.rs
+++ b/src/cmd/restore.rs
@@ -1,21 +1,24 @@
-use crate::client::BackupClient;
+use crate::backup_reason::Reason;
use crate::client::ClientConfig;
+use crate::client::{BackupClient, ClientError};
use crate::error::ObnamError;
use crate::fsentry::{FilesystemEntry, FilesystemKind};
-use crate::generation::LocalGeneration;
+use crate::generation::{LocalGeneration, LocalGenerationError};
use indicatif::{ProgressBar, ProgressStyle};
-use libc::{fchmod, futimens, timespec};
+use libc::{chmod, mkfifo, timespec, utimensat, AT_FDCWD};
use log::{debug, error, info};
-use std::fs::File;
+use std::ffi::CString;
use std::io::prelude::*;
use std::io::Error;
+use std::os::unix::ffi::OsStrExt;
use std::os::unix::fs::symlink;
-use std::os::unix::io::AsRawFd;
+use std::os::unix::net::UnixListener;
+use std::path::StripPrefixError;
use std::path::{Path, PathBuf};
use structopt::StructOpt;
use tempfile::NamedTempFile;
-pub fn restore(config: &ClientConfig, gen_ref: &str, to: &Path) -> anyhow::Result<()> {
+pub fn restore(config: &ClientConfig, gen_ref: &str, to: &Path) -> Result<(), ObnamError> {
// Create a named temporary file. We don't meed the open file
// handle, so we discard that.
let dbname = {
@@ -24,20 +27,20 @@ pub fn restore(config: &ClientConfig, gen_ref: &str, to: &Path) -> anyhow::Resul
dbname
};
- let client = BackupClient::new(&config.server_url)?;
+ let client = BackupClient::new(config)?;
let genlist = client.list_generations()?;
- let gen_id: String = match genlist.resolve(gen_ref) {
- None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()),
- Some(id) => id,
- };
+ let gen_id: String = genlist.resolve(gen_ref)?;
info!("generation id is {}", gen_id);
let gen = client.fetch_generation(&gen_id, &dbname)?;
info!("restoring {} files", gen.file_count()?);
let progress = create_progress_bar(gen.file_count()?, true);
for file in gen.files()? {
- restore_generation(&client, &gen, file.fileno(), file.entry(), &to, &progress)?;
+ match file.reason() {
+ Reason::FileError => (),
+ _ => restore_generation(&client, &gen, file.fileno(), file.entry(), &to, &progress)?,
+ }
}
for file in gen.files()? {
if file.entry().is_dir() {
@@ -68,6 +71,32 @@ struct Opt {
to: PathBuf,
}
+#[derive(Debug, thiserror::Error)]
+pub enum RestoreError {
+ #[error("Could not create named pipe (FIFO) {0}")]
+ NamedPipeCreationError(PathBuf),
+
+ #[error(transparent)]
+ ClientError(#[from] ClientError),
+
+ #[error(transparent)]
+ LocalGenerationError(#[from] LocalGenerationError),
+
+ #[error(transparent)]
+ StripPrefixError(#[from] StripPrefixError),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+
+ #[error(transparent)]
+ SerdeYamlError(#[from] serde_yaml::Error),
+
+ #[error(transparent)]
+ NulError(#[from] std::ffi::NulError),
+}
+
+pub type RestoreResult<T> = Result<T, RestoreError>;
+
fn restore_generation(
client: &BackupClient,
gen: &LocalGeneration,
@@ -75,7 +104,7 @@ fn restore_generation(
entry: &FilesystemEntry,
to: &Path,
progress: &ProgressBar,
-) -> anyhow::Result<()> {
+) -> RestoreResult<()> {
info!("restoring {:?}", entry);
progress.set_message(&format!("{}", entry.pathbuf().display()));
progress.inc(1);
@@ -85,17 +114,19 @@ fn restore_generation(
FilesystemKind::Regular => restore_regular(client, &gen, &to, fileid, &entry)?,
FilesystemKind::Directory => restore_directory(&to)?,
FilesystemKind::Symlink => restore_symlink(&to, &entry)?,
+ FilesystemKind::Socket => restore_socket(&to, &entry)?,
+ FilesystemKind::Fifo => restore_fifo(&to, &entry)?,
}
Ok(())
}
-fn restore_directory(path: &Path) -> anyhow::Result<()> {
+fn restore_directory(path: &Path) -> RestoreResult<()> {
debug!("restoring directory {}", path.display());
std::fs::create_dir_all(path)?;
Ok(())
}
-fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> anyhow::Result<()> {
+fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> RestoreResult<()> {
let to = restored_path(entry, to)?;
match entry.kind() {
FilesystemKind::Directory => restore_metadata(&to, entry)?,
@@ -107,7 +138,7 @@ fn restore_directory_metadata(entry: &FilesystemEntry, to: &Path) -> anyhow::Res
Ok(())
}
-fn restored_path(entry: &FilesystemEntry, to: &Path) -> anyhow::Result<PathBuf> {
+fn restored_path(entry: &FilesystemEntry, to: &Path) -> RestoreResult<PathBuf> {
let path = &entry.pathbuf();
let path = if path.is_absolute() {
path.strip_prefix("/")?
@@ -123,7 +154,7 @@ fn restore_regular(
path: &Path,
fileid: i64,
entry: &FilesystemEntry,
-) -> anyhow::Result<()> {
+) -> RestoreResult<()> {
debug!("restoring regular {}", path.display());
let parent = path.parent().unwrap();
debug!(" mkdir {}", parent.display());
@@ -140,7 +171,7 @@ fn restore_regular(
Ok(())
}
-fn restore_symlink(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> {
+fn restore_symlink(path: &Path, entry: &FilesystemEntry) -> RestoreResult<()> {
debug!("restoring symlink {}", path.display());
let parent = path.parent().unwrap();
debug!(" mkdir {}", parent.display());
@@ -154,10 +185,29 @@ fn restore_symlink(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> {
Ok(())
}
-fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()> {
+fn restore_socket(path: &Path, entry: &FilesystemEntry) -> RestoreResult<()> {
+ debug!("creating Unix domain socket {:?}", path);
+ UnixListener::bind(path)?;
+ restore_metadata(path, entry)?;
+ Ok(())
+}
+
+fn restore_fifo(path: &Path, entry: &FilesystemEntry) -> RestoreResult<()> {
+ debug!("creating fifo {:?}", path);
+ let filename = path_to_cstring(path);
+ match unsafe { mkfifo(filename.as_ptr(), 0) } {
+ -1 => {
+ return Err(RestoreError::NamedPipeCreationError(path.to_path_buf()));
+ }
+ _ => restore_metadata(path, entry)?,
+ }
+ Ok(())
+}
+
+fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> RestoreResult<()> {
debug!("restoring metadata for {}", entry.pathbuf().display());
- let handle = File::open(path)?;
+ debug!("restoring metadata for {:?}", path);
let atime = timespec {
tv_sec: entry.atime(),
@@ -170,28 +220,35 @@ fn restore_metadata(path: &Path, entry: &FilesystemEntry) -> anyhow::Result<()>
let times = [atime, mtime];
let times: *const timespec = &times[0];
+ let path = path_to_cstring(path);
+
// We have to use unsafe here to be able call the libc functions
// below.
unsafe {
- let fd = handle.as_raw_fd(); // FIXME: needs to NOT follow symlinks
-
- debug!("fchmod");
- if fchmod(fd, entry.mode()) == -1 {
+ debug!("chmod {:?}", path);
+ if chmod(path.as_ptr(), entry.mode()) == -1 {
let error = Error::last_os_error();
- error!("fchmod failed on {}", path.display());
+ error!("chmod failed on {:?}", path);
return Err(error.into());
}
- debug!("futimens");
- if futimens(fd, times) == -1 {
+ debug!("utimens {:?}", path);
+ if utimensat(AT_FDCWD, path.as_ptr(), times, 0) == -1 {
let error = Error::last_os_error();
- error!("futimens failed on {}", path.display());
+ error!("utimensat failed on {:?}", path);
return Err(error.into());
}
}
Ok(())
}
+fn path_to_cstring(path: &Path) -> CString {
+ let path = path.as_os_str();
+ let path = path.as_bytes();
+ let path = CString::new(path).unwrap();
+ path
+}
+
fn create_progress_bar(file_count: i64, verbose: bool) -> ProgressBar {
let progress = if verbose {
ProgressBar::new(file_count as u64)
diff --git a/src/cmd/show_config.rs b/src/cmd/show_config.rs
new file mode 100644
index 0000000..b4f4cdc
--- /dev/null
+++ b/src/cmd/show_config.rs
@@ -0,0 +1,7 @@
+use crate::client::ClientConfig;
+use crate::error::ObnamError;
+
+pub fn show_config(config: &ClientConfig) -> Result<(), ObnamError> {
+ println!("{}", serde_json::to_string_pretty(config)?);
+ Ok(())
+}
diff --git a/src/cmd/show_gen.rs b/src/cmd/show_gen.rs
index d355389..c7a4bdd 100644
--- a/src/cmd/show_gen.rs
+++ b/src/cmd/show_gen.rs
@@ -5,7 +5,7 @@ use crate::fsentry::FilesystemKind;
use indicatif::HumanBytes;
use tempfile::NamedTempFile;
-pub fn show_generation(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<()> {
+pub fn show_generation(config: &ClientConfig, gen_ref: &str) -> Result<(), ObnamError> {
// Create a named temporary file. We don't meed the open file
// handle, so we discard that.
let dbname = {
@@ -14,14 +14,10 @@ pub fn show_generation(config: &ClientConfig, gen_ref: &str) -> anyhow::Result<(
dbname
};
- let client = BackupClient::new(&config.server_url)?;
+ let client = BackupClient::new(config)?;
let genlist = client.list_generations()?;
- let gen_id: String = match genlist.resolve(gen_ref) {
- None => return Err(ObnamError::UnknownGeneration(gen_ref.to_string()).into()),
- Some(id) => id,
- };
-
+ let gen_id: String = genlist.resolve(gen_ref)?;
let gen = client.fetch_generation(&gen_id, &dbname)?;
let files = gen.files()?;
diff --git a/src/error.rs b/src/error.rs
index d368763..73d4a66 100644
--- a/src/error.rs
+++ b/src/error.rs
@@ -1,25 +1,46 @@
-use crate::chunkid::ChunkId;
-use std::path::PathBuf;
+use crate::backup_run::BackupError;
+use crate::client::{ClientConfigError, ClientError};
+use crate::cmd::restore::RestoreError;
+use crate::generation::{LocalGenerationError, NascentError};
+use crate::genlist::GenerationListError;
+use std::time::SystemTimeError;
+use tempfile::PersistError;
use thiserror::Error;
-/// Define all the kinds of errors any part of this crate can return.
+/// Define all the kinds of errors that functions corresponding to
+/// subcommands of the main program can return.
#[derive(Debug, Error)]
pub enum ObnamError {
- #[error("Can't find backup '{0}'")]
- UnknownGeneration(String),
+ #[error(transparent)]
+ GenerationListError(#[from] GenerationListError),
- #[error("Generation has more than one file with the name {0}")]
- TooManyFiles(PathBuf),
+ #[error(transparent)]
+ ClientError(#[from] ClientError),
- #[error("Server response did not have a 'chunk-meta' header for chunk {0}")]
- NoChunkMeta(ChunkId),
+ #[error(transparent)]
+ ClientConfigError(#[from] ClientConfigError),
- #[error("Wrong checksum for chunk {0}, got {1}, expected {2}")]
- WrongChecksum(ChunkId, String, String),
+ #[error(transparent)]
+ BackupError(#[from] BackupError),
- #[error("Chunk is missing: {0}")]
- MissingChunk(ChunkId),
+ #[error(transparent)]
+ NascentError(#[from] NascentError),
- #[error("Chunk is in store too many times: {0}")]
- DuplicateChunk(ChunkId),
+ #[error(transparent)]
+ LocalGenerationError(#[from] LocalGenerationError),
+
+ #[error(transparent)]
+ RestoreError(#[from] RestoreError),
+
+ #[error(transparent)]
+ PersistError(#[from] PersistError),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+
+ #[error(transparent)]
+ SystemTimeError(#[from] SystemTimeError),
+
+ #[error(transparent)]
+ SerdeJsonError(#[from] serde_json::Error),
}
diff --git a/src/fsentry.rs b/src/fsentry.rs
index eae11b4..570877a 100644
--- a/src/fsentry.rs
+++ b/src/fsentry.rs
@@ -1,9 +1,11 @@
+use log::{debug, error};
use serde::{Deserialize, Serialize};
use std::ffi::OsString;
use std::fs::read_link;
use std::fs::{FileType, Metadata};
use std::os::linux::fs::MetadataExt;
use std::os::unix::ffi::OsStringExt;
+use std::os::unix::fs::FileTypeExt;
use std::path::{Path, PathBuf};
/// A file system entry.
@@ -37,10 +39,34 @@ pub struct FilesystemEntry {
symlink_target: Option<PathBuf>,
}
+#[derive(Debug, thiserror::Error)]
+pub enum FsEntryError {
+ #[error("Unknown file kind {0}")]
+ UnknownFileKindCode(u8),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+}
+
+pub type FsEntryResult<T> = Result<T, FsEntryError>;
+
#[allow(clippy::len_without_is_empty)]
impl FilesystemEntry {
- pub fn from_metadata(path: &Path, meta: &Metadata) -> anyhow::Result<Self> {
+ pub fn from_metadata(path: &Path, meta: &Metadata) -> FsEntryResult<Self> {
let kind = FilesystemKind::from_file_type(meta.file_type());
+ let symlink_target = if kind == FilesystemKind::Symlink {
+ debug!("reading symlink target for {:?}", path);
+ let target = match read_link(path) {
+ Ok(x) => x,
+ Err(err) => {
+ error!("read_link failed: {}", err);
+ return Err(err.into());
+ }
+ };
+ Some(target)
+ } else {
+ None
+ };
Ok(Self {
path: path.to_path_buf().into_os_string().into_vec(),
kind: FilesystemKind::from_file_type(meta.file_type()),
@@ -50,11 +76,7 @@ impl FilesystemEntry {
mtime_ns: meta.st_mtime_nsec(),
atime: meta.st_atime(),
atime_ns: meta.st_atime_nsec(),
- symlink_target: if kind == FilesystemKind::Symlink {
- Some(read_link(path)?)
- } else {
- None
- },
+ symlink_target,
})
}
@@ -106,6 +128,8 @@ pub enum FilesystemKind {
Regular,
Directory,
Symlink,
+ Socket,
+ Fifo,
}
impl FilesystemKind {
@@ -116,6 +140,10 @@ impl FilesystemKind {
FilesystemKind::Directory
} else if file_type.is_symlink() {
FilesystemKind::Symlink
+ } else if file_type.is_socket() {
+ FilesystemKind::Socket
+ } else if file_type.is_fifo() {
+ FilesystemKind::Fifo
} else {
panic!("unknown file type {:?}", file_type);
}
@@ -126,15 +154,19 @@ impl FilesystemKind {
FilesystemKind::Regular => 0,
FilesystemKind::Directory => 1,
FilesystemKind::Symlink => 2,
+ FilesystemKind::Socket => 3,
+ FilesystemKind::Fifo => 4,
}
}
- pub fn from_code(code: u8) -> anyhow::Result<Self> {
+ pub fn from_code(code: u8) -> FsEntryResult<Self> {
match code {
0 => Ok(FilesystemKind::Regular),
1 => Ok(FilesystemKind::Directory),
2 => Ok(FilesystemKind::Symlink),
- _ => Err(Error::UnknownFileKindCode(code).into()),
+ 3 => Ok(FilesystemKind::Socket),
+ 4 => Ok(FilesystemKind::Fifo),
+ _ => Err(FsEntryError::UnknownFileKindCode(code).into()),
}
}
}
@@ -153,6 +185,9 @@ mod test {
fn file_kind_regular_round_trips() {
one_file_kind_round_trip(FilesystemKind::Regular);
one_file_kind_round_trip(FilesystemKind::Directory);
+ one_file_kind_round_trip(FilesystemKind::Symlink);
+ one_file_kind_round_trip(FilesystemKind::Socket);
+ one_file_kind_round_trip(FilesystemKind::Fifo);
}
fn one_file_kind_round_trip(kind: FilesystemKind) {
diff --git a/src/fsiter.rs b/src/fsiter.rs
index a40ad34..f59fb64 100644
--- a/src/fsiter.rs
+++ b/src/fsiter.rs
@@ -1,13 +1,27 @@
-use crate::fsentry::FilesystemEntry;
-use log::info;
+use crate::fsentry::{FilesystemEntry, FsEntryError};
+use log::{debug, error};
use std::path::Path;
-use walkdir::{IntoIter, WalkDir};
+use walkdir::{DirEntry, IntoIter, WalkDir};
/// Iterator over file system entries in a directory tree.
pub struct FsIterator {
iter: IntoIter,
}
+#[derive(Debug, thiserror::Error)]
+pub enum FsIterError {
+ #[error(transparent)]
+ WalkError(#[from] walkdir::Error),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+
+ #[error(transparent)]
+ FsEntryError(#[from] FsEntryError),
+}
+
+pub type FsIterResult<T> = Result<T, FsIterError>;
+
impl FsIterator {
pub fn new(root: &Path) -> Self {
Self {
@@ -17,21 +31,30 @@ impl FsIterator {
}
impl Iterator for FsIterator {
- type Item = Result<FilesystemEntry, anyhow::Error>;
+ type Item = FsIterResult<FilesystemEntry>;
fn next(&mut self) -> Option<Self::Item> {
- match self.iter.next() {
+ let next = self.iter.next();
+ debug!("walkdir found: {:?}", next);
+ match next {
None => None,
- Some(Ok(entry)) => {
- info!("found {}", entry.path().display());
- Some(new_entry(&entry))
- }
+ Some(Ok(entry)) => Some(new_entry(&entry)),
Some(Err(err)) => Some(Err(err.into())),
}
}
}
-fn new_entry(e: &walkdir::DirEntry) -> anyhow::Result<FilesystemEntry> {
- let meta = e.metadata()?;
- let entry = FilesystemEntry::from_metadata(e.path(), &meta)?;
+fn new_entry(e: &DirEntry) -> FsIterResult<FilesystemEntry> {
+ let path = e.path();
+ let meta = std::fs::metadata(path);
+ debug!("metadata for {:?}: {:?}", path, meta);
+ let meta = match meta {
+ Ok(meta) => meta,
+ Err(err) => {
+ error!("failed to get metadata: {}", err);
+ return Err(err.into());
+ }
+ };
+ let entry = FilesystemEntry::from_metadata(path, &meta)?;
+ debug!("FileSystemEntry for {:?}: {:?}", path, entry);
Ok(entry)
}
diff --git a/src/generation.rs b/src/generation.rs
index 8a15363..4655c17 100644
--- a/src/generation.rs
+++ b/src/generation.rs
@@ -1,8 +1,9 @@
use crate::backup_reason::Reason;
+use crate::backup_run::{BackupError, BackupResult};
use crate::chunkid::ChunkId;
use crate::fsentry::FilesystemEntry;
use rusqlite::Connection;
-use std::path::Path;
+use std::path::{Path, PathBuf};
/// An identifier for a file in a generation.
type FileId = i64;
@@ -17,8 +18,25 @@ pub struct NascentGeneration {
fileno: FileId,
}
+#[derive(Debug, thiserror::Error)]
+pub enum NascentError {
+ #[error(transparent)]
+ LocalGenerationError(#[from] LocalGenerationError),
+
+ #[error(transparent)]
+ BackupError(#[from] BackupError),
+
+ #[error(transparent)]
+ RusqliteError(#[from] rusqlite::Error),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+}
+
+pub type NascentResult<T> = Result<T, NascentError>;
+
impl NascentGeneration {
- pub fn create<P>(filename: P) -> anyhow::Result<Self>
+ pub fn create<P>(filename: P) -> NascentResult<Self>
where
P: AsRef<Path>,
{
@@ -35,7 +53,7 @@ impl NascentGeneration {
e: FilesystemEntry,
ids: &[ChunkId],
reason: Reason,
- ) -> anyhow::Result<()> {
+ ) -> NascentResult<()> {
let t = self.conn.transaction()?;
self.fileno += 1;
sql::insert_one(&t, e, self.fileno, ids, reason)?;
@@ -45,8 +63,8 @@ impl NascentGeneration {
pub fn insert_iter<'a>(
&mut self,
- entries: impl Iterator<Item = anyhow::Result<(FilesystemEntry, Vec<ChunkId>, Reason)>>,
- ) -> anyhow::Result<()> {
+ entries: impl Iterator<Item = BackupResult<(FilesystemEntry, Vec<ChunkId>, Reason)>>,
+ ) -> NascentResult<()> {
let t = self.conn.transaction()?;
for r in entries {
let (e, ids, reason) = r?;
@@ -110,6 +128,23 @@ pub struct LocalGeneration {
conn: Connection,
}
+#[derive(Debug, thiserror::Error)]
+pub enum LocalGenerationError {
+ #[error("Generation has more than one file with the name {0}")]
+ TooManyFiles(PathBuf),
+
+ #[error(transparent)]
+ RusqliteError(#[from] rusqlite::Error),
+
+ #[error(transparent)]
+ SerdeJsonError(#[from] serde_json::Error),
+
+ #[error(transparent)]
+ IoError(#[from] std::io::Error),
+}
+
+pub type LocalGenerationResult<T> = Result<T, LocalGenerationError>;
+
pub struct BackedUpFile {
fileno: FileId,
entry: FilesystemEntry,
@@ -140,7 +175,7 @@ impl BackedUpFile {
}
impl LocalGeneration {
- pub fn open<P>(filename: P) -> anyhow::Result<Self>
+ pub fn open<P>(filename: P) -> LocalGenerationResult<Self>
where
P: AsRef<Path>,
{
@@ -148,23 +183,23 @@ impl LocalGeneration {
Ok(Self { conn })
}
- pub fn file_count(&self) -> anyhow::Result<i64> {
+ pub fn file_count(&self) -> LocalGenerationResult<i64> {
Ok(sql::file_count(&self.conn)?)
}
- pub fn files(&self) -> anyhow::Result<Vec<BackedUpFile>> {
+ pub fn files(&self) -> LocalGenerationResult<Vec<BackedUpFile>> {
Ok(sql::files(&self.conn)?)
}
- pub fn chunkids(&self, fileno: FileId) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn chunkids(&self, fileno: FileId) -> LocalGenerationResult<Vec<ChunkId>> {
Ok(sql::chunkids(&self.conn, fileno)?)
}
- pub fn get_file(&self, filename: &Path) -> anyhow::Result<Option<FilesystemEntry>> {
+ pub fn get_file(&self, filename: &Path) -> LocalGenerationResult<Option<FilesystemEntry>> {
Ok(sql::get_file(&self.conn, filename)?)
}
- pub fn get_fileno(&self, filename: &Path) -> anyhow::Result<Option<FileId>> {
+ pub fn get_fileno(&self, filename: &Path) -> LocalGenerationResult<Option<FileId>> {
Ok(sql::get_fileno(&self.conn, filename)?)
}
}
@@ -172,15 +207,16 @@ impl LocalGeneration {
mod sql {
use super::BackedUpFile;
use super::FileId;
+ use super::LocalGenerationError;
+ use super::LocalGenerationResult;
use crate::backup_reason::Reason;
use crate::chunkid::ChunkId;
- use crate::error::ObnamError;
use crate::fsentry::FilesystemEntry;
use rusqlite::{params, Connection, OpenFlags, Row, Transaction};
use std::os::unix::ffi::OsStrExt;
use std::path::Path;
- pub fn create_db(filename: &Path) -> anyhow::Result<Connection> {
+ pub fn create_db(filename: &Path) -> LocalGenerationResult<Connection> {
let flags = OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE;
let conn = Connection::open_with_flags(filename, flags)?;
conn.execute(
@@ -197,7 +233,7 @@ mod sql {
Ok(conn)
}
- pub fn open_db(filename: &Path) -> anyhow::Result<Connection> {
+ pub fn open_db(filename: &Path) -> LocalGenerationResult<Connection> {
let flags = OpenFlags::SQLITE_OPEN_READ_WRITE;
let conn = Connection::open_with_flags(filename, flags)?;
conn.pragma_update(None, "journal_mode", &"WAL")?;
@@ -210,7 +246,7 @@ mod sql {
fileno: FileId,
ids: &[ChunkId],
reason: Reason,
- ) -> anyhow::Result<()> {
+ ) -> LocalGenerationResult<()> {
let json = serde_json::to_string(&e)?;
t.execute(
"INSERT INTO files (fileno, filename, json, reason) VALUES (?1, ?2, ?3, ?4)",
@@ -236,7 +272,7 @@ mod sql {
Ok((fileno, json, reason))
}
- pub fn file_count(conn: &Connection) -> anyhow::Result<FileId> {
+ pub fn file_count(conn: &Connection) -> LocalGenerationResult<FileId> {
let mut stmt = conn.prepare("SELECT count(*) FROM files")?;
let mut iter = stmt.query_map(params![], |row| row.get(0))?;
let count = iter.next().expect("SQL count result (1)");
@@ -244,7 +280,7 @@ mod sql {
Ok(count)
}
- pub fn files(conn: &Connection) -> anyhow::Result<Vec<BackedUpFile>> {
+ pub fn files(conn: &Connection) -> LocalGenerationResult<Vec<BackedUpFile>> {
let mut stmt = conn.prepare("SELECT * FROM files")?;
let iter = stmt.query_map(params![], |row| row_to_entry(row))?;
let mut files = vec![];
@@ -256,7 +292,7 @@ mod sql {
Ok(files)
}
- pub fn chunkids(conn: &Connection, fileno: FileId) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn chunkids(conn: &Connection, fileno: FileId) -> LocalGenerationResult<Vec<ChunkId>> {
let mut stmt = conn.prepare("SELECT chunkid FROM chunks WHERE fileno = ?1")?;
let iter = stmt.query_map(params![fileno], |row| Ok(row.get(0)?))?;
let mut ids: Vec<ChunkId> = vec![];
@@ -267,14 +303,17 @@ mod sql {
Ok(ids)
}
- pub fn get_file(conn: &Connection, filename: &Path) -> anyhow::Result<Option<FilesystemEntry>> {
+ pub fn get_file(
+ conn: &Connection,
+ filename: &Path,
+ ) -> LocalGenerationResult<Option<FilesystemEntry>> {
match get_file_and_fileno(conn, filename)? {
None => Ok(None),
Some((_, e, _)) => Ok(Some(e)),
}
}
- pub fn get_fileno(conn: &Connection, filename: &Path) -> anyhow::Result<Option<FileId>> {
+ pub fn get_fileno(conn: &Connection, filename: &Path) -> LocalGenerationResult<Option<FileId>> {
match get_file_and_fileno(conn, filename)? {
None => Ok(None),
Some((id, _, _)) => Ok(Some(id)),
@@ -284,7 +323,7 @@ mod sql {
fn get_file_and_fileno(
conn: &Connection,
filename: &Path,
- ) -> anyhow::Result<Option<(FileId, FilesystemEntry, String)>> {
+ ) -> LocalGenerationResult<Option<(FileId, FilesystemEntry, String)>> {
let mut stmt = conn.prepare("SELECT * FROM files WHERE filename = ?1")?;
let mut iter =
stmt.query_map(params![path_into_blob(filename)], |row| row_to_entry(row))?;
@@ -296,7 +335,7 @@ mod sql {
if iter.next() == None {
Ok(Some((fileno, entry, reason)))
} else {
- Err(ObnamError::TooManyFiles(filename.to_path_buf()).into())
+ Err(LocalGenerationError::TooManyFiles(filename.to_path_buf()).into())
}
}
}
diff --git a/src/genlist.rs b/src/genlist.rs
index 10c614e..5eec248 100644
--- a/src/genlist.rs
+++ b/src/genlist.rs
@@ -5,6 +5,12 @@ pub struct GenerationList {
list: Vec<FinishedGeneration>,
}
+#[derive(Debug, thiserror::Error)]
+pub enum GenerationListError {
+ #[error("Unknown generation: {0}")]
+ UnknownGeneration(String),
+}
+
impl GenerationList {
pub fn new(gens: Vec<FinishedGeneration>) -> Self {
let mut list = gens.clone();
@@ -16,7 +22,7 @@ impl GenerationList {
self.list.iter()
}
- pub fn resolve(&self, genref: &str) -> Option<String> {
+ pub fn resolve(&self, genref: &str) -> Result<String, GenerationListError> {
let gen = if self.list.is_empty() {
None
} else if genref == "latest" {
@@ -36,8 +42,8 @@ impl GenerationList {
}
};
match gen {
- None => None,
- Some(gen) => Some(gen.id().to_string()),
+ None => Err(GenerationListError::UnknownGeneration(genref.to_string())),
+ Some(gen) => Ok(gen.id().to_string()),
}
}
}
diff --git a/src/index.rs b/src/index.rs
index d527839..9386e73 100644
--- a/src/index.rs
+++ b/src/index.rs
@@ -17,8 +17,27 @@ pub struct Index {
metas: HashMap<ChunkId, ChunkMeta>,
}
+/// All the errors that may be returned for `Index`.
+#[derive(Debug, thiserror::Error)]
+pub enum IndexError {
+ /// Index does not have a chunk.
+ #[error("The repository index does not have chunk {0}")]
+ MissingChunk(ChunkId),
+
+ /// Index has chunk more than once.
+ #[error("The repository index duplicates chunk {0}")]
+ DuplicateChunk(ChunkId),
+
+ /// An error from SQLite.
+ #[error(transparent)]
+ SqlError(#[from] rusqlite::Error),
+}
+
+/// A result from an `Index` operation.
+pub type IndexResult<T> = Result<T, IndexError>;
+
impl Index {
- pub fn new<P: AsRef<Path>>(dirname: P) -> anyhow::Result<Self> {
+ pub fn new<P: AsRef<Path>>(dirname: P) -> IndexResult<Self> {
let filename = dirname.as_ref().join("meta.db");
let conn = if filename.exists() {
sql::open_db(&filename)?
@@ -34,28 +53,32 @@ impl Index {
})
}
- pub fn insert_meta(&mut self, id: ChunkId, meta: ChunkMeta) -> anyhow::Result<()> {
+ pub fn insert_meta(&mut self, id: ChunkId, meta: ChunkMeta) -> IndexResult<()> {
let t = self.conn.transaction()?;
sql::insert(&t, &id, &meta)?;
t.commit()?;
Ok(())
}
- pub fn get_meta(&self, id: &ChunkId) -> anyhow::Result<ChunkMeta> {
+ pub fn get_meta(&self, id: &ChunkId) -> IndexResult<ChunkMeta> {
sql::lookup(&self.conn, id)
}
- pub fn remove_meta(&mut self, id: &ChunkId) -> anyhow::Result<()> {
+ pub fn remove_meta(&mut self, id: &ChunkId) -> IndexResult<()> {
sql::remove(&self.conn, id)
}
- pub fn find_by_sha256(&self, sha256: &str) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn find_by_sha256(&self, sha256: &str) -> IndexResult<Vec<ChunkId>> {
sql::find_by_256(&self.conn, sha256)
}
- pub fn find_generations(&self) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn find_generations(&self) -> IndexResult<Vec<ChunkId>> {
sql::find_generations(&self.conn)
}
+
+ pub fn all_chunks(&self) -> IndexResult<Vec<ChunkId>> {
+ sql::find_chunk_ids(&self.conn)
+ }
}
#[cfg(test)]
@@ -132,14 +155,14 @@ mod test {
}
mod sql {
+ use super::{IndexError, IndexResult};
use crate::chunkid::ChunkId;
use crate::chunkmeta::ChunkMeta;
- use crate::error::ObnamError;
use log::error;
use rusqlite::{params, Connection, OpenFlags, Row, Transaction};
use std::path::Path;
- pub fn create_db(filename: &Path) -> anyhow::Result<Connection> {
+ pub fn create_db(filename: &Path) -> IndexResult<Connection> {
let flags = OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE;
let conn = Connection::open_with_flags(filename, flags)?;
conn.execute(
@@ -155,14 +178,14 @@ mod sql {
Ok(conn)
}
- pub fn open_db(filename: &Path) -> anyhow::Result<Connection> {
+ pub fn open_db(filename: &Path) -> IndexResult<Connection> {
let flags = OpenFlags::SQLITE_OPEN_READ_WRITE;
let conn = Connection::open_with_flags(filename, flags)?;
conn.pragma_update(None, "journal_mode", &"WAL")?;
Ok(conn)
}
- pub fn insert(t: &Transaction, chunkid: &ChunkId, meta: &ChunkMeta) -> anyhow::Result<()> {
+ pub fn insert(t: &Transaction, chunkid: &ChunkId, meta: &ChunkMeta) -> IndexResult<()> {
let chunkid = format!("{}", chunkid);
let sha256 = meta.sha256();
let generation = if meta.is_generation() { 1 } else { 0 };
@@ -174,12 +197,12 @@ mod sql {
Ok(())
}
- pub fn remove(conn: &Connection, chunkid: &ChunkId) -> anyhow::Result<()> {
+ pub fn remove(conn: &Connection, chunkid: &ChunkId) -> IndexResult<()> {
conn.execute("DELETE FROM chunks WHERE id IS ?1", params![chunkid])?;
Ok(())
}
- pub fn lookup(conn: &Connection, id: &ChunkId) -> anyhow::Result<ChunkMeta> {
+ pub fn lookup(conn: &Connection, id: &ChunkId) -> IndexResult<ChunkMeta> {
let mut stmt = conn.prepare("SELECT * FROM chunks WHERE id IS ?1")?;
let iter = stmt.query_map(params![id], |row| row_to_meta(row))?;
let mut metas: Vec<ChunkMeta> = vec![];
@@ -189,20 +212,20 @@ mod sql {
eprintln!("lookup: meta={:?}", meta);
metas.push(meta);
} else {
- let err = ObnamError::DuplicateChunk(id.clone());
+ let err = IndexError::DuplicateChunk(id.clone());
error!("{}", err);
return Err(err.into());
}
}
if metas.len() == 0 {
eprintln!("lookup: no hits");
- return Err(ObnamError::MissingChunk(id.clone()).into());
+ return Err(IndexError::MissingChunk(id.clone()).into());
}
let r = metas[0].clone();
Ok(r)
}
- pub fn find_by_256(conn: &Connection, sha256: &str) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn find_by_256(conn: &Connection, sha256: &str) -> IndexResult<Vec<ChunkId>> {
let mut stmt = conn.prepare("SELECT id FROM chunks WHERE sha256 IS ?1")?;
let iter = stmt.query_map(params![sha256], |row| row_to_id(row))?;
let mut ids = vec![];
@@ -213,7 +236,7 @@ mod sql {
Ok(ids)
}
- pub fn find_generations(conn: &Connection) -> anyhow::Result<Vec<ChunkId>> {
+ pub fn find_generations(conn: &Connection) -> IndexResult<Vec<ChunkId>> {
let mut stmt = conn.prepare("SELECT id FROM chunks WHERE generation IS 1")?;
let iter = stmt.query_map(params![], |row| row_to_id(row))?;
let mut ids = vec![];
@@ -224,7 +247,18 @@ mod sql {
Ok(ids)
}
- pub fn row_to_meta(row: &Row) -> rusqlite::Result<ChunkMeta> {
+ pub fn find_chunk_ids(conn: &Connection) -> IndexResult<Vec<ChunkId>> {
+ let mut stmt = conn.prepare("SELECT id FROM chunks WHERE generation IS 0")?;
+ let iter = stmt.query_map(params![], |row| row_to_id(row))?;
+ let mut ids = vec![];
+ for x in iter {
+ let x = x?;
+ ids.push(x);
+ }
+ Ok(ids)
+ }
+
+ fn row_to_meta(row: &Row) -> rusqlite::Result<ChunkMeta> {
let sha256: String = row.get(row.column_index("sha256")?)?;
let generation: i32 = row.get(row.column_index("generation")?)?;
let meta = if generation == 0 {
@@ -236,7 +270,7 @@ mod sql {
Ok(meta)
}
- pub fn row_to_id(row: &Row) -> rusqlite::Result<ChunkId> {
+ fn row_to_id(row: &Row) -> rusqlite::Result<ChunkId> {
let id: String = row.get(row.column_index("id")?)?;
Ok(ChunkId::from_str(&id))
}
diff --git a/src/indexedstore.rs b/src/indexedstore.rs
index 0366013..f2d1831 100644
--- a/src/indexedstore.rs
+++ b/src/indexedstore.rs
@@ -1,8 +1,9 @@
-use crate::chunk::DataChunk;
+use crate::chunk::{DataChunk, GenerationChunk, GenerationChunkError};
use crate::chunkid::ChunkId;
use crate::chunkmeta::ChunkMeta;
-use crate::index::Index;
-use crate::store::Store;
+use crate::index::{Index, IndexError};
+use crate::store::{Store, StoreError};
+use std::collections::HashSet;
use std::path::Path;
/// A store for chunks and their metadata.
@@ -14,43 +15,83 @@ pub struct IndexedStore {
index: Index,
}
+/// All the errors that may be returned for `IndexStore`.
+#[derive(Debug, thiserror::Error)]
+pub enum IndexedError {
+ /// An error from Index.
+ #[error(transparent)]
+ IndexError(#[from] IndexError),
+
+ #[error(transparent)]
+ GenerationChunkError(#[from] GenerationChunkError),
+
+ /// An error from Store.
+ #[error(transparent)]
+ SqlError(#[from] StoreError),
+}
+
+/// A result from an `Index` operation.
+pub type IndexedResult<T> = Result<T, IndexedError>;
+
impl IndexedStore {
- pub fn new(dirname: &Path) -> anyhow::Result<Self> {
+ pub fn new(dirname: &Path) -> IndexedResult<Self> {
let store = Store::new(dirname);
let index = Index::new(dirname)?;
Ok(Self { store, index })
}
- pub fn save(&mut self, meta: &ChunkMeta, chunk: &DataChunk) -> anyhow::Result<ChunkId> {
+ pub fn save(&mut self, meta: &ChunkMeta, chunk: &DataChunk) -> IndexedResult<ChunkId> {
let id = ChunkId::new();
self.store.save(&id, meta, chunk)?;
self.insert_meta(&id, meta)?;
Ok(id)
}
- fn insert_meta(&mut self, id: &ChunkId, meta: &ChunkMeta) -> anyhow::Result<()> {
+ fn insert_meta(&mut self, id: &ChunkId, meta: &ChunkMeta) -> IndexedResult<()> {
self.index.insert_meta(id.clone(), meta.clone())?;
Ok(())
}
- pub fn load(&self, id: &ChunkId) -> anyhow::Result<(DataChunk, ChunkMeta)> {
+ pub fn load(&self, id: &ChunkId) -> IndexedResult<(DataChunk, ChunkMeta)> {
Ok((self.store.load(id)?, self.load_meta(id)?))
}
- pub fn load_meta(&self, id: &ChunkId) -> anyhow::Result<ChunkMeta> {
- self.index.get_meta(id)
+ pub fn load_meta(&self, id: &ChunkId) -> IndexedResult<ChunkMeta> {
+ Ok(self.index.get_meta(id)?)
}
- pub fn find_by_sha256(&self, sha256: &str) -> anyhow::Result<Vec<ChunkId>> {
- self.index.find_by_sha256(sha256)
+ pub fn find_by_sha256(&self, sha256: &str) -> IndexedResult<Vec<ChunkId>> {
+ Ok(self.index.find_by_sha256(sha256)?)
}
- pub fn find_generations(&self) -> anyhow::Result<Vec<ChunkId>> {
- self.index.find_generations()
+ pub fn find_generations(&self) -> IndexedResult<Vec<ChunkId>> {
+ Ok(self.index.find_generations()?)
+ }
+
+ pub fn find_file_chunks(&self) -> IndexedResult<Vec<ChunkId>> {
+ let gen_ids = self.find_generations()?;
+
+ let mut sql_chunks: HashSet<ChunkId> = HashSet::new();
+ for id in gen_ids {
+ let gen_chunk = self.store.load(&id)?;
+ let gen = GenerationChunk::from_data_chunk(&gen_chunk)?;
+ for sqlite_chunk_id in gen.chunk_ids() {
+ sql_chunks.insert(sqlite_chunk_id.clone());
+ }
+ }
+
+ let all_chunk_ids = self.index.all_chunks()?;
+ let file_chunks = all_chunk_ids
+ .iter()
+ .filter(|id| !sql_chunks.contains(id))
+ .map(|id| id.clone())
+ .collect();
+
+ Ok(file_chunks)
}
- pub fn remove(&mut self, id: &ChunkId) -> anyhow::Result<()> {
- self.index.remove_meta(id).unwrap();
+ pub fn remove(&mut self, id: &ChunkId) -> IndexedResult<()> {
+ self.index.remove_meta(id)?;
self.store.delete(id)?;
Ok(())
}
diff --git a/src/policy.rs b/src/policy.rs
index 032b851..8a65e09 100644
--- a/src/policy.rs
+++ b/src/policy.rs
@@ -42,7 +42,7 @@ impl BackupPolicy {
"needs_backup: lookup in old generation returned error, ignored: {:?}: {}",
new_name, err
);
- Reason::Error
+ Reason::GenerationLookupError
}
};
debug!(
diff --git a/src/server.rs b/src/server.rs
index 4d5880e..01a6958 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -4,6 +4,52 @@ use crate::chunkmeta::ChunkMeta;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::default::Default;
+use std::path::{Path, PathBuf};
+
+#[derive(Debug, Deserialize, Clone)]
+pub struct Config {
+ pub chunks: PathBuf,
+ pub address: String,
+ pub tls_key: PathBuf,
+ pub tls_cert: PathBuf,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum ConfigError {
+ #[error("Directory for chunks {0} does not exist")]
+ ChunksDirNotFound(PathBuf),
+
+ #[error("TLS certificate {0} does not exist")]
+ TlsCertNotFound(PathBuf),
+
+ #[error("TLS key {0} does not exist")]
+ TlsKeyNotFound(PathBuf),
+
+ #[error("server address can't be resolved")]
+ BadServerAddress,
+}
+
+impl Config {
+ pub fn read_config(filename: &Path) -> anyhow::Result<Config> {
+ let config = std::fs::read_to_string(filename)?;
+ let config: Config = serde_yaml::from_str(&config)?;
+ config.check()?;
+ Ok(config)
+ }
+
+ pub fn check(&self) -> anyhow::Result<()> {
+ if !self.chunks.exists() {
+ return Err(ConfigError::ChunksDirNotFound(self.chunks.clone()).into());
+ }
+ if !self.tls_cert.exists() {
+ return Err(ConfigError::TlsCertNotFound(self.tls_cert.clone()).into());
+ }
+ if !self.tls_key.exists() {
+ return Err(ConfigError::TlsKeyNotFound(self.tls_key.clone()).into());
+ }
+ Ok(())
+ }
+}
/// Result of creating a chunk.
#[derive(Debug, Serialize)]
@@ -50,7 +96,7 @@ impl SearchHits {
self.map.insert(id.to_string(), meta);
}
- pub fn from_json(s: &str) -> anyhow::Result<Self> {
+ pub fn from_json(s: &str) -> Result<Self, serde_json::Error> {
let map = serde_json::from_str(s)?;
Ok(SearchHits { map })
}
diff --git a/src/store.rs b/src/store.rs
index e6cc71f..fca2c13 100644
--- a/src/store.rs
+++ b/src/store.rs
@@ -1,7 +1,6 @@
use crate::chunk::DataChunk;
use crate::chunkid::ChunkId;
use crate::chunkmeta::ChunkMeta;
-use anyhow::Context;
use std::path::{Path, PathBuf};
/// Store chunks, with metadata, persistently.
@@ -13,6 +12,12 @@ pub struct Store {
dir: PathBuf,
}
+/// An error from a `Store` operation.
+pub type StoreError = std::io::Error;
+
+/// A result from an `Store` operation.
+pub type StoreResult<T> = Result<T, StoreError>;
+
impl Store {
/// Create a new Store to represent on-disk storage of chunks.x
pub fn new(dir: &Path) -> Self {
@@ -38,14 +43,11 @@ impl Store {
}
/// Save a chunk into a store.
- pub fn save(&self, id: &ChunkId, meta: &ChunkMeta, chunk: &DataChunk) -> anyhow::Result<()> {
+ pub fn save(&self, id: &ChunkId, meta: &ChunkMeta, chunk: &DataChunk) -> StoreResult<()> {
let (dir, metaname, dataname) = &self.filenames(id);
if !dir.exists() {
- let res = std::fs::create_dir_all(dir).into();
- if let Err(_) = res {
- return res.with_context(|| format!("creating directory {}", dir.display()));
- }
+ std::fs::create_dir_all(dir)?;
}
std::fs::write(&metaname, meta.to_json())?;
@@ -54,7 +56,7 @@ impl Store {
}
/// Load a chunk from a store.
- pub fn load(&self, id: &ChunkId) -> anyhow::Result<DataChunk> {
+ pub fn load(&self, id: &ChunkId) -> StoreResult<DataChunk> {
let (_, _, dataname) = &self.filenames(id);
let data = std::fs::read(&dataname)?;
let data = DataChunk::new(data);
@@ -62,7 +64,7 @@ impl Store {
}
/// Delete a chunk from a store.
- pub fn delete(&self, id: &ChunkId) -> anyhow::Result<()> {
+ pub fn delete(&self, id: &ChunkId) -> StoreResult<()> {
let (_, metaname, dataname) = &self.filenames(id);
std::fs::remove_file(&metaname)?;
std::fs::remove_file(&dataname)?;
diff --git a/subplot/client.py b/subplot/client.py
index d450b4c..0e724b8 100644
--- a/subplot/client.py
+++ b/subplot/client.py
@@ -20,6 +20,7 @@ def configure_client(ctx, filename=None):
config = yaml.safe_load(config)
config["server_url"] = ctx["server_url"]
+ logging.debug(f"client config {filename}: {config}")
with open(filename, "w") as f:
yaml.safe_dump(config, stream=f)
diff --git a/subplot/client.yaml b/subplot/client.yaml
index b1f9b19..eef4714 100644
--- a/subplot/client.yaml
+++ b/subplot/client.yaml
@@ -4,6 +4,9 @@
- given: "a client config based on {filename}"
function: configure_client
+- when: "I invoke obnam --config {filename} config"
+ function: run_obnam_config
+
- when: "I invoke obnam --config {filename} restore <{genid}> {todir}"
function: run_obnam_restore
diff --git a/subplot/data.py b/subplot/data.py
index a24cd0c..0c1a4a5 100644
--- a/subplot/data.py
+++ b/subplot/data.py
@@ -1,16 +1,31 @@
+import json
import logging
import os
import random
+import socket
+import yaml
-def create_file_with_random_data(ctx, filename=None):
- N = 128
- data = "".join(chr(random.randint(0, 255)) for i in range(N)).encode("UTF-8")
+def create_file_with_given_data(ctx, filename=None, data=None):
+ logging.debug(f"creating file {filename} with {data!r}")
dirname = os.path.dirname(filename) or "."
- logging.debug(f"create_file_with_random_data: dirname={dirname}")
os.makedirs(dirname, exist_ok=True)
- with open(filename, "wb") as f:
- f.write(data)
+ open(filename, "wb").write(data.encode("UTF-8"))
+
+
+def create_file_with_random_data(ctx, filename=None):
+ N = 128
+ data = "".join(chr(random.randint(0, 255)) for i in range(N))
+ create_file_with_given_data(ctx, filename=filename, data=data)
+
+
+def create_unix_socket(ctx, filename=None):
+ fd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ fd.bind(filename)
+
+
+def create_fifo(ctx, filename=None):
+ os.mkfifo(filename)
def create_nonutf8_filename(ctx, dirname=None):
@@ -49,6 +64,20 @@ def _create_manifest_of_directory(ctx, dirname=None, manifest=None):
open(manifest, "w").write(stdout)
+def file_is_restored(ctx, filename=None, restored=None):
+ filename = os.path.join(restored, "./" + filename)
+ exists = os.path.exists(filename)
+ logging.debug(f"restored? {filename} {exists}")
+ assert exists
+
+
+def file_is_not_restored(ctx, filename=None, restored=None):
+ filename = os.path.join(restored, "./" + filename)
+ exists = os.path.exists(filename)
+ logging.debug(f"restored? {filename} {exists}")
+ assert not exists
+
+
def files_match(ctx, first=None, second=None):
assert_eq = globals()["assert_eq"]
@@ -57,3 +86,30 @@ def files_match(ctx, first=None, second=None):
logging.debug(f"files_match: f:\n{f}")
logging.debug(f"files_match: s:\n{s}")
assert_eq(f, s)
+
+
+def convert_yaml_to_json(ctx, yaml_name=None, json_name=None):
+ with open(yaml_name) as f:
+ obj = yaml.safe_load(f)
+ with open(json_name, "w") as f:
+ json.dump(obj, f)
+
+
+def match_stdout_to_json_file(ctx, filename=None):
+ runcmd_get_stdout = globals()["runcmd_get_stdout"]
+ assert_eq = globals()["assert_eq"]
+
+ stdout = runcmd_get_stdout(ctx)
+ stdout = json.loads(stdout.strip())
+ obj = json.load(open(filename))
+ logging.debug(f"match_stdout_to_json_file: stdout={stdout!r}")
+ logging.debug(f"match_stdout_to_json_file: file={obj!r}")
+
+ for key in obj:
+ if key not in stdout:
+ logging.error(f"{key} not in stdout")
+ assert key in stdout
+
+ if stdout[key] != obj[key]:
+ logging.error(f"stdout value for key is not what was exptected")
+ assert_eq(stdout[key], obj[key])
diff --git a/subplot/data.yaml b/subplot/data.yaml
index 7659319..0e13abd 100644
--- a/subplot/data.yaml
+++ b/subplot/data.yaml
@@ -1,11 +1,15 @@
-- given: >
- a file (?P<filename>\\S+) containing "(?P<data>.*)"
- regex: true
+- given: a file {filename} containing "{data:text}"
function: create_file_with_given_data
- given: "a file {filename} containing some random data"
function: create_file_with_random_data
+- given: "a Unix socket {filename}"
+ function: create_unix_socket
+
+- given: "a named pipe {filename}"
+ function: create_fifo
+
- given: "a file in {dirname} with a non-UTF8 filename"
function: create_nonutf8_filename
@@ -21,5 +25,14 @@
- given: a manifest of the directory {dirname} restored in {restored} in {manifest}
function: create_manifest_of_restored
-- then: files {first} and {second} match
- function: files_match
+- given: "JSON file {json_name} converted from YAML file {yaml_name}"
+ function: convert_yaml_to_json
+
+- then: "stdout, as JSON, matches file {filename}"
+ function: match_stdout_to_json_file
+
+- then: "file {filename} is restored to {restored}"
+ function: file_is_restored
+
+- then: "file {filename} is not restored to {restored}"
+ function: file_is_not_restored
diff --git a/subplot/server.py b/subplot/server.py
index 289e181..df594f7 100644
--- a/subplot/server.py
+++ b/subplot/server.py
@@ -5,8 +5,6 @@ import random
import re
import requests
import shutil
-import socket
-import time
import urllib3
import yaml
@@ -35,7 +33,9 @@ def start_chunk_server(ctx):
"address": f"localhost:{port}",
}
- server_binary = os.path.abspath(os.path.join(srcdir, "target", "debug", "obnam-server"))
+ server_binary = os.path.abspath(
+ os.path.join(srcdir, "target", "debug", "obnam-server")
+ )
filename = "config.yaml"
yaml.safe_dump(config, stream=open(filename, "w"))
@@ -44,11 +44,7 @@ def start_chunk_server(ctx):
ctx["server_url"] = f"https://{config['address']}"
daemon_start_on_port(
- ctx,
- name="obnam-server",
- path=server_binary,
- args=filename,
- port=port,
+ ctx, name="obnam-server", path=server_binary, args=filename, port=port
)
@@ -138,6 +134,15 @@ def json_body_matches(ctx, wanted=None):
assert_eq(body.get(key, "not.there"), wanted[key])
+def server_has_n_file_chunks(ctx, n=None):
+ assert_eq = globals()["assert_eq"]
+ n = int(n)
+ url = f"{ctx['server_url']}/chunks?data=true"
+ _request(ctx, requests.get, url)
+ num_chunks = len(ctx["http.json"])
+ assert_eq(n, num_chunks)
+
+
# Make an HTTP request.
def _request(ctx, method, url, headers=None, data=None):
r = method(url, headers=headers, data=data, verify=False)
diff --git a/subplot/server.yaml b/subplot/server.yaml
index 68f8f0c..60f8a44 100644
--- a/subplot/server.yaml
+++ b/subplot/server.yaml
@@ -43,3 +43,6 @@
- then: "the body matches file {filename}"
function: body_matches_file
+
+- then: "server has {n:int} file chunks"
+ function: server_has_n_file_chunks
diff --git a/tutorial.md b/tutorial.md
new file mode 100644
index 0000000..4f84380
--- /dev/null
+++ b/tutorial.md
@@ -0,0 +1,279 @@
+With the help of this tutorial, you're going to set up Obnam, make your first
+backup, and check that you can restore files from it.
+
+In Obnam, **a client** is a computer whose data is being backed up, and **a
+server** is a computer that holds the backup. A single computer can serve both
+roles, but don't put your backup onto the same disk as you're backing up; if
+that disk breaks, the backup won't do you any good. Consider using
+an USB-attached disk, or better yet, some network-attached storage.
+
+
+
+# Setting up a server
+
+For this, you'll need:
+
+* Git and Ansible installed on your local machine
+* a Debian host with plenty of space to keep your backups (this can be the same,
+ local machine)
+
+On your local machine, clone the Obnam repository:
+
+```
+$ git clone https://gitlab.com/larswirzenius/obnam.git
+$ cd obnam/ansible
+```
+
+The next command depends on where your Obnam server is hosted:
+
+- if the server is accessible from the Internet, run the following commands,
+ replacing `obnam.example.com` with the domain name of the host:
+
+ ```
+ $ printf '[server]\nobnam.example.com\n' > hosts
+ $ ansible-playbook -i hosts obnam-server.yml -e domain=obnam.example.com
+ ```
+
+ The above gets a free TLS certificate from [Let's Encrypt][].
+
+- if it's a private server or just the same machine as the Obnam client, run the
+ following:
+
+
+ ```
+ $ printf '[server]\nprivate-vm\n' > hosts
+ $ ansible-playbook -i hosts obnam-server.yml
+ ```
+
+ This uses a pre-created self-signed certificate from `files/server.key` and
+ `files/server.pem`, and is probably only good for trying out Obnam. You may
+ want to generate your own certificates instead, e.g. using [OpenSSL]
+ something like this:
+
+ ```
+ $ openssl req -x509 -newkey rsa:4096 -passout pass:hunter2 \
+ -keyout key.pem -out cert.pem -days 365 -subj /CN=localhost
+ ```
+
+ Put the generated keys into `/etc/obnam` (the location can be configured
+ with `tls_key` and `tls_cert` keys in `/etc/obnam/server.yaml`, which we
+ are about to describe).
+
+Check that the server is installed and running:
+
+```
+$ sudo systemctl is-active obnam
+active
+```
+
+Ansible created a directory, `/src/obnam/chunks`, that will contain the
+backed-up data. If you want to use a different directory, you have to stop the
+service, move the existing directory to a new location, and update Obnam's
+configuration:
+
+```
+$ sudo systemctl stop obnam
+$ sudo mv /srv/obnam /the/new/location/
+$ sudoedit /etc/obnam/server.yaml
+```
+
+In the editor, you'll see something like this:
+
+```
+address: 0.0.0.0:443
+chunks: /srv/obnam/chunks
+tls_key: /etc/obnam/server.key
+tls_cert: /etc/obnam/server.pem
+```
+
+Paths to TLS files might be different if you're using Let's Encrypt. Anyway, you
+have to edit `chunks` key to point at the new location. Once you're done, save
+the file and start the server again:
+
+```
+$ sudo systemctl start obnam
+$ sudo systemctl is-active obnam
+active
+```
+
+Half the job done, another half to go! Let's set up a client now.
+
+[Let's Encrypt]: https://letsencrypt.org/
+[OpenSSL]: https://www.openssl.org/
+
+
+
+# Setting up a client
+
+There is a Debian package built by CI from every commit. It works on Debian 10
+(buster) and later. You can run a script to install it:
+
+```
+$ curl -s https://gitlab.com/larswirzenius/obnam/-/raw/main/install-debian.sh | sudo bash
+```
+
+If you'd rather not download a script from the Internet and run it as
+root (kudos!), you can do the same steps manually. Add the following
+to `/etc/apt/sources.list.d/obnam.list`:
+
+```
+deb http://ci-prod-controller.vm.liw.fi/debian unstable-ci main
+```
+
+Then save the following PGP public key as `/etc/apt/trusted.gpg.d/obnam.asc`:
+
+```
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBFrLO7kBEADdz6mHstYmKU5Dp6OSjxWtWaqTDOX1sJdmmaIK/9EKVIH0Maxp
+5kvVO5G6mULLAjv/kLG0MxasHPrq8I2A/y8AqKAGVL8QelwLjQMIFZ30/VbGQPHS
++T5TZXEnoQtNce1GUhFwJ38ZyjjwHBFV9tSec7rZ2Q3YeM3nNnGPf6DacXGfEOPO
+HIN4sXAN2hzNXNjKRzTIvxQseb6nr7afUh/SlZ3yhQOCrIzmYlD7tP9WJe7ofL0p
+JY4pDQYw8rT6nC2BE/ioemh84kERCT1vCe+OVFlSRuMlqfEv+ZpKQ+itOmPDQ/lM
+jpUm1K2hrW/lWpxT/ZxHKo/w1K36J5WshgMZxfUu5BMCL9LMqMcrXNhNjDMfxDMM
+3yBPOvQ4ls6fecOZ/bsFo1p8VzMk/w/eG8vPs5yuNa5XxN95yFMXoOHGb5Xbu8D4
+6yiW+Af70LbiSNpGdmNdneiGB2fY38NxBukPw5u3S5qG8HedSmMr1RvSr5kHoAAe
+UbOY+BYaaKsTAT7+1skUW1o3FJSqoRKCHAzTsMWC6zzhR8hRn7jVrrguH1hGbqq5
+TZSCFQZExuTJ7uXrTLG0WoBXIjB5wWNcSeXn8myUWYB51nJNF4tJBouZOz9JwWGl
+kiAQkrHnBttLQWdW9FyjbIoTZMtpvVx+m6ObGTGdGL1cNlLAvWprMXGc+QARAQAB
+tDJJY2sgQVBUIHJlcG9zaXRvcnkgc2lnbmluZyBrZXkgKDIwMTgpIDxsaXdAbGl3
+LmZpPokCTgQTAQgAOBYhBKL1uyDoXyxUH3O717Wr+TZVS6PGBQJayzu5AhsDBQsJ
+CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJELWr+TZVS6PGB5QQANTcikhRUHwt9N4h
+dGc/Hp6CbqdshMoWlwpFskttoVDxQG5OAobuZl5XyzGcmja1lT85RGkZFfbca0IZ
+LnXOLLSAu51QBkXNaj4OhjK/0uQ+ITrvL6RQSXNgHiUTR/W2XD1GIUq6nBqe2GSN
+31S1baYKKVj5QIMsi7Dq8ls3BBXuPCE+xTSaNmGWjes2t9pPidcRvxsksCLY1qgw
+P1GFXBeMkBQ29kBP87SUL15SIk7OiQLlEURCy5iRls5rt/YEsdEpRWIb0Tm5Nrjv
+2M3VM+iBhfNXTwj0rJ34mlycF1qQmA7YcTEobT7z587GPY0VWzBpQUnEQj7rQWPM
+cDYY0b+I6kQ8VKOaL4wVAtE98d7HzFIrIrwhTKufnrWrVDPYsmLZ+LPC1jiF7JBD
+SR6Vftb+SdDR9xoE1yRuXbC6IfoW+5/qQNrdQ2mm9BFw5jOonBqchs18HTTf3441
+6SWwP9fY3Vi+IZphPPi0Gf85oMStgnv/Wnw6LacEL32ek39Desero/D8iGLZernK
+Q2mC9mua5A/bYGVhsNWyURNFkKdbFa+/wW3NfdKYyZnsSfo+jJ2luNewrhAY7Kod
+GWXTer9RxzTGA3EXFGvNr+BBOOxSj0SfWTl0Olo7J5dnxof+jLAUS1VHpceHGHps
+GSJSdir7NkZidgwoCPA7BTqsb5LN
+=dXB0
+-----END PGP PUBLIC KEY BLOCK-----
+```
+
+After that, run the following commands to install Obnam:
+
+```
+$ sudo apt update
+$ sudo apt install obnam
+```
+
+Now verify that everything is installed correctly:
+
+```
+$ obnam --version
+obnam-backup 0.2.2
+```
+
+The version might be different, but at least there should **not** be any errors.
+
+
+
+# Making a backup
+
+To create a backup, client needs to know just two things: where the backup
+server is, and where the live data is. To tell it about that, create a file
+`~/.config/obnam/obnam.yaml` with contents like this:
+
+```yaml
+server_url: https://obnam.example.com:443
+roots:
+ - /home/joe
+ - /home/ann
+ - /etc
+ - /var/spool
+```
+
+Adjust the server address to match what you previously configured on the server.
+The `roots` key is a list of all the directories that Obnam should back up. Make
+sure that the roots are accessible to the user who would be doing the backup —
+the user has to be able to read their contents to back them up.
+
+With that, you're ready to make your first backup! Run the following command,
+and watch Obnam go through all the files in your roots:
+
+```
+$ obnam backup
+elapsed: 7s
+files: 3422/0
+current: /home/ann/music/Beethoven/1.flac
+```
+
+Depending on how much data you have under the roots, this might take a while.
+But once Obnam is done, it will print out a report like this:
+
+```
+status: OK
+duration: 85
+file-count: 1223
+generation-id: 3905a0ad-9971-413c-ac81-ca8587c5f8c2
+```
+
+That's how you know you've got a backup! Hold off the celebration, though; the
+backups are only as good as your ability to use them, so let's check if you can
+recover the files you just backed up.
+
+
+
+# Restoring a backup
+
+Let's imagine that your disk crapped out. In that case, you probably want to
+just grab the latest backup. In other cases, you might find that a file you
+thought useless and deleted long ago is actually important. To restore it, you
+need to find the backup that still has it. So the first thing you do is get
+a list of all your backups with `obnam list`:
+
+```
+$ obnam list
+6d35e3dd-3264-4269-a9d3-74fbd354c90e 2021-01-13 02:32:50.482465724 +0300
+e4387899-d1dd-4e42-bc57-f56e6097d235 2021-01-14 02:36:00.029561204 +0300
+9acde8d9-c167-4ad0-86b6-560c711713e1 2021-01-18 02:45:56.865274252 +0300
+708db71e-d863-47e6-92c3-679041e25c8e 2021-01-20 02:49:50.664349817 +0300
+0f3a63d0-d992-42ff-ab77-7e2457745a40 2021-01-22 03:00:56.902063598 +0300
+028ce888-4a5b-438c-978c-0812646165cf 2021-02-07 16:18:19.008757980 +0300
+481bb25f-5377-4e41-b824-4e60fda8f01c 2021-02-08 19:04:44.072710112 +0300
+5067e10e-2d4d-4ff4-a9a0-568ed008dd2c 2021-02-11 20:26:06.589610566 +0300
+3905a0ad-9971-413c-ac81-ca8587c5f8c2 2021-02-12 22:35:20.431081194 +0300
+```
+
+That second-to-last backup, 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c, looks like
+it's old enough. Let's see what files it contains:
+
+```
+$ obnam list-files 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c
+```
+
+You might need to `grep` the result to check for specific files. Anyway, suppose
+this backup it exactly what you need. Let's restore it to a directory called
+"yesterday":
+
+```
+$ obnam restore 5067e10e-2d4d-4ff4-a9a0-568ed008dd2c yesterday
+```
+
+Obnam will print out a progress bar and some stats. Once the restoration is
+done, you can look under `yesterday/` to find the file you needed. Easy!
+
+Now you're prepared for the worst. (Unless *both* your primary and backup disks
+break. Or your backup server is inaccessible. Or there is no electrical grid
+anymore to power your devices. Or the zombies are trying to break in,
+distracting you from reading this tutorial. Look up "disaster recovery
+planning"—oh right, no electricity.)
+
+
+# Where to go from here
+
+Obnam is still at the alpha stage, so it's likely that the instructions above
+didn't quite work for you. If so, please [open issues][issue-tracker] and help
+us improve Obnam!
+
+If you're interested in more details, and especially in how Obnam works
+internally, take a look at [obnam.md](obnam.md) Subplot file. It not just
+explains things, but also contains acceptance criteria and tests for them. Great
+stuff!
+
+
+[issue-tracker]: https://gitlab.com/larswirzenius/obnam/-/issues