| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The previous commit introduced a function to create FilesystemEntry
values from arbitrary data. Previously one could only be created from
std::fs::Metadata. This complicated our own testing, which (now) needs
to construct an arbitrary entry structure. However, while the function
added in the last commit was straightforward, it had 11 arguments, and
that's hard to keep track of. Replace that function with an
EntryBuilder struct, for clarity.
Sponsored-by: author
|
|
The test passes. We create a FilesystemEntry with a length field
containing u64::MAX, store that into a generation, and read it back.
This works. The entry is serialised into JSON for storing in SQLite,
and this proves we can handle any u64 value in an entry. serde_json
deals with it fine, and we don't need to worry about it.
Sponsored-by: author
|
|
This is more friendly towards anyone wanting to use the output in a script.
Sponsored-by: author
|
|
When making a backup, use the same checksum for any chunks it re-uses
or creates. This is for performance: if we allowed two checksums to be
used, we would have to compute the checksum for a chunk twice, and
potentially look up both on the server. This is just a lot of work.
Instead, we use only one. The trade-off here is that when (not if) the
user wants to switch to a new checksum type, they'll have to do a full
backup, uploading all their data to the server, even when it's already
there, just with a different checksum. Hopefully this will be rare.
Full backups always use the built-in, hardcoded default checksum, and
incremental backups use whatever the previous backup used. The default
is still SHA256, but this commit add code to support BLAKE2 if we
decide to switch that as a default. It's also easy to add support for
others, now. BLAKE2 was added to verify that Obnam can actually handle
the checksum changing (manual test: not in the test suite).
I don't think users need to be offered even the option of choosing a
checksum algorithm to use. When one cares about both security and
performance, choosing a checksum requires specialist, expert
knowledge. Obnam developers should choose the default. Giving users a
knob they can twiddle just makes it that much harder to configure and
use Obnam. If the choice Obnam developers have made is shown to be
sub-optimal, it seems better to change the default for everyone,
rather than hope that every user changes their configuration to gain
the benefit.
Experience has shown that people mostly don't change the default
configuration, and that they are especially bad at choosing well when
security is a concern.
(Obnam is free software. Expert users can choose their checksum by
changing the source code. I'm not fundamentally limiting anyone's
freedom or choice here.)
Users can switch to a new default algorithm by triggering a full
backup with the new "obnam backup --full".
Sponsored-by: author
|
|
The way this is currently implemented resulted in a lot of code
duplication in src/generation.rs. This should be refactored later. My
first attempt to do it by adding a trait for a schema variant failed.
Sponsored-by: author
|
|
Sponsored-by: author
|
|
Sponsored-by: author
|
|
Sponsored-by: author
|
|
This is clearer and less error prone.
Sponsored-by: author
|
|
When opening a local generation, check that it's compatible with the
current version of Obnam.
Sponsored-by: author
|
|
Also, make it an error for a public symbol to not be documented.
Sponsored-by: author
|
|
Sponsored-by: author
|
|
Previously an error from, say, a missing backup root directory was
reported to the user as a warning. Turn it into an error. However,
errors reading a file or directory inside the backup root should still
be just a warning.
Sponsored-by: author
|
|
Sponsored-by: author
|
|
Add a new mandatory database table "meta" to the SQLite database the
stores information about the files in a backup generation. The idea is
for future versions of the Obnam client to be able to be able to
restore from backups made by older -- or newer -- versions of Obnam,
as far as is reasonable.
Add the `obnam gen-info` command to show information about the
generation metadata.
Sponsored-by: author
|
|
It was only used by a test function, which is now changed to not use it.
Add comment to the test function that it's too complicated and things
need refactoring. However, that probably needs to wait for new
abstractions.
Sponsored-by: author
|
|
This makes the code more explicit, which is good for now, and is a
step towards making it all use async. There will be a need to refactor
this further with better abstractions, once async works.
Sponsored-by: author
|
|
This is a step towards getting rid of insert_iter entirely, which
would make it easier to make `obnam backup` use async.
I originally split insert_iter so I could use a single transaction for
inserting many rows, but it seems to not be needed for speed after
all. I've benchmarked backing up a large file with and without this
change, and there's no real difference. I've not benchmarked with a
large number of files.
Even if there's a performance hit from using multiple transactions, my
hope is that by being able to use more CPUs/threads for backing up
will outweigh that by far.
Sponsored-by: author
|
|
This means a ChunkId can't be used instead.
Sponsored-by: author
|
|
This will make it harder to accidentally use a string. Can still be
confused with a chunk id.
Sponsored-by: author
|
|
Mostly
https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow.
Sponsored-by: author
|
|
In the following commits, we'll use this to check if a tag existed
before.
|
|
I do not plan to simplify the `T` in the return type of
`get_file_and_fileno` because that function is only ever called from
inside the module; it doesn't seem worthwhile to introduce a new type
there.
|
|
|
|
|
|
|
|
This should make it easier to introduce async, later.
|
|
All unclear error messages should now be clearer. For example, all the
ones related to a file mention the file name and the attempted
operation that failed.
|
|
|
|
|
|
|
|
This adds the machinery. We have to keep the compiled SQL query while
the iterator is in use, so we wrap it in an `SqlResults` struct which
the iterator borrows.
|
|
`LocalGeneration::sql::files()` runs an SQL query, iterates over the
results and collects the rows into a `Vec`. This can fail at any step:
the query might fail to run, or one of the rows might fail to be fetched
or processed.
Right now, we lump all those failures into a `Result` that wraps the
whole return value. This is only possible because we process each row
before returning. Once `Vec` is replaced by an iterator, we won't have
that luxury anymore, so we now wrap each individual element into its own
`Result` (as well as wrapping the whole vector into a `Result` of its
own).
|
|
|
|
|
|
|
|
|
|
Previously, we either ignored it or aborted the backup. Neither is
good. Now we ignore the problem, except to show a warning at the end
of the backup run.
|
|
This means that a function that parses step bindings can't return an
error that the document is missing a title. Such an error return would
be nonsensical, and we use the Rust type system to prevent it, at a
small cost of being a bit verbose.
Additional benefit is that the library portion of Obnam doesn't return
anyhow::Result values anymore.
|
|
The struct is easier to use right.
|
|
This commit also splits up the src/cmd/backup.rs module into other,
smaller, more cohesive modules that are easier to understand and use.
|
|
|
|
This changes SQL schema.
|
|
|
|
This speeds things up a lot compared to iterating over all files.
|
|
|
|
This keeps all the SQL related functions closer together, making it
easier to make changes to them.
|
|
This uses the previous, latest generation as a guideline to see what
is new or changed.
|
|
This splits the use of NascentGeneration to more cohesive "new
generation being built" versus "existing generation being restored".
|
|
Various part of Obnam will need to deal with lists of generations.
Abstract this.
|
