Obnam NEWS ========== This file summarizes changes between releases of Obnam. Version 0.16, released 2011-03-21; a BETA release ------------------------------------------------- * This is the first BETA release. Obnam should now be feature complete for real use. Performance is lacking and there are many bugs remaining. There are no known bugs that would corrupt backed up data, or prevent its recovery. * Add encryption support. See the manual page for how to use it. Version 0.15.1, released 2011-03-21; an ALPHA release ---------------------------------------------------- * Fix `setup.py` to not import `obnamlib`, so it works when building under pbuilder on Debian. Meh. Version 0.15, released 2011-03-21; an ALPHA release ---------------------------------------------------- Bugs fixed: * Manual page GPL copyright blurb is now properly marked up as a comment. (Thanks, Joey Hess.) * README now links to python-lru correctly. (Thanks, Erik Johansson.) Improvements and other changes: * Filenames and directories are backed up in sorted order. This should make it easier to know how far obnam's gotten. * The location where backups are stored is now called the repository, instead of the store. Suggested by Joey Hess. * The repository and the target directory for restored data are now both created by Obnam, if they don't already exist. Suggested by Joey Hess. * Better control of logging, using the new `--trace` option. * Manual page now explains making backups a little better. * Default value for `--lru-size` reduced to 500, for great improvement in memory used, without, it seems, much decrease in speed. * `obnam verify` now reports success explicitly. Based on question from Joey Hess. * `obnam verify` now accepts both non-option arguments and the `--root` option. Suggested by Joey Hess. * `obnam forget` now accepts "generation specifiers", not just numeric generation ids. This means that `obnam forget latest` works. * I/O statistics are logged more systematically. * `obnam force-lock` introduced, to allow breaking a lock left behind if obnam crashes. But it never does, of course. (Well, except if there's a bug, like when a file changes at the wrong moment.) * `obnam genids` introduced, to list generation ids without any other data. The old command `obnam generations` still works, and lists other info about each generation as well, but that's sometimes bad for scripting. * The `--dump-memory-profile` option now accepts the value `simple`, for reporting basic memory use. It has such a small impact that it's the default. * Obnam now stores the version of the on-disk format in the repository. This should allow it to handle repositories created by a different version and act suitably (hopefully without wiping all your backups). Version 0.14, released 2010-12-29; an ALPHA release ---------------------------------------------------- This version is capable of backing up my laptop's home directory. It is, however, still an ALPHA release, and you should not rely on it as your sole form of backup. It is also slow. But if you're curious, now would be a good time to try it out a bit. Bug fixes: * `COPYING` now contains GPL version 3, instead of 2. The code was licensed under version 3 already. (Thank you Greg Grossmeier.) * The manual page now uses `-` and `\-` correctly. * `obnam forget` now actually removes data that is no longer used by any generation. * When backing up a new generation, if any of the root directories for the backup got dropped by the user, they are now also removed from the backup generation. Old generations obviously still have them. * Only the per-client B-tree forest should have multiple trees. Now this actually happens, whereas previously sometimes a very large number of new trees would be created in some forests. (What's good for rain forests is not good for saving disk space.) * When recursing through directory trees, obnam no longer follows symlinks to directories. * obnam no longer creates a missing backup store when backing up to a local disk. It never did this when backing up via sftp. (This saves me from figuring out which of `store`, `stor`, and `sorte` is the real directory.) New features and stuff: * `blackboxtest` has been rewritten to use Python's `unittest` framework, rather than a homegrown bad re-implementation of some of it. * `obnam ls` interprets arguments as "genspecs" rather than generation identifiers. This means `obnam ls latest` works, and now `latest` is also the default if you don't give any spec. * `run-benchmarks` now outputs results into a git checkout of , an ikiwiki instance hosted by . The script also puts the results into a suitable sub-directory, adds a page for the RSS feed of benchmark results, and updates the report page that summarizes all stored results. * There is now a 100 GiB benchmark. * Clients are now called clients, instead of hosts. This terminology should be clearer. * The list of clients now stores a random integer identifier for each client (unique within the store). The identifier is used as the name of the per-client B-tree directory, rather than the hostname of the client. This should prevent a teeny tiny bit of information leakage. It also makes debugging things much harder. * Various refactorings and prettifications of the code has happened. For example, several classes have been split off from the `store.py` module. This has also resulted in much better test coverage for those classes. * The per-client trees (formerly GenerationStore, now ClientMetadataTree) have a more complicated key now: 4 parts, not 3. This makes it easier to keep separate data about files, and other data that needs to be stored per-generation, such as what the generation id is. * `find-duplicate-chunks`, a tool for finding duplicate chunks of data in a files in a directory tree, was added to the tree. I have used it to find out if is worthwhile to do duplicate chunk removal at all. (It is, at least for my data.) Also, it can be used to find good values for chunk sizes for duplicate detection. * The whole way in which obnam does de-duplication got re-designed and re-implemented. This is tricky stuff, when there is more than one client. * `SftpFS` now uses a hack copied from bzrlib, to use openssh if it is available, and paramiko only if it is not. This speeds up sftp data transfers quite a bit. (Where bzrlib supports more than just openssh, we don't, since I have no way to test the other stuff. Patches welcome.) * The way lists of chunk ids are stored for files got changed. Now we store several ids per list item, which is faster and also saves some space in the B-tree nodes. Also, it is now possible to append to the list, which means the caller does not need to first gather a list of all ids. Such a list gets quite costly when the file is quite big (e.g., in the terabyte size). * New `--dump-memory-profile` option was added to help do memory profiling with meliae or heapy have been added. (Obnam's memory consumption finally got annoying enough that I did something about it.) Removed stuff: * The functional specification was badly outdated, and has been removed. I decided to stop kidding myself that I would keep it up to date. * The store design document has been removed from the store tree. The online version at is the canonical version, and is actually kept up to date. * The benchmark specification has likewise been replaced with . Version 0.13, released 2010-07-13; an ALPHA release ---------------------------------------------------- * Bug fix: a mistake in 0.12 caused checkpoints to happen after each file after the first checkpoint. Now they happen at the right intervals again. * Upload speed is now displayed during backups. * Obnam now tells the kernel that it shouldn't cache data it reads or writes. It is not likely that data being backed up is going to be needed again any time soon, so there's no point in caching it. (The posix_fadvise call is used for this.) * New --lru-size option sets size of LRU cache for nodes in memory. The obnam default is large enough to suit large backups. This uses more memory, but is faster than btree's small default of 100. Version 0.12, released 2010-07-11; an ALPHA release ---------------------------------------------------- * NOTE: This version makes incompatible changes to the way data is stored on-disk. Backups made with older versions are NOT supported. Sorry. * The run-benchmark script has dropped some smaller sizes (they're too fast to be interesting), and adds a 10 GiB test size. * Various speed optimizations. Most importantly, the way file metadata (results of lstat(2)) are encoded has changed. This is the incompatible change from above. It's much faster now, though. * Preliminary support for using SFTP for the backup store added. Hasn't been used much yet, so might well be very buggy. Version 0.11, released 2010-07-05; an ALPHA release ---------------------------------------------------- * Speed optimizations: - chunk identifiers are now sequential, except for the first one, or when there's a collision - chunks are now stored in a more sensible directory hierarchy (instead of one per directory, on average) - adding files to a directory in the backup store is now faster - only store a file's metadata that if it is changed * New --exclude=regexp option to exclude files based on pathnames * Obnam now makes checkpoints during backups. If a backup is aborted in the middle and then re-started, it will continue from the latest checkpoint rather than from the beginning of the previous backup run. - New option --checkpoint to set the interval between checkpoints. Defaults to 1 GiB. * Options for various B-tree settings. This is mostly useful for finding the optimal set of defaults, but may be useful in other situations for some people. - New options --chunk-group-size, --chunk-size, --node-size, --upload-queue-size. * Somewhat better progress reporting during backups. Version 0.10, released 2010-06-29; an ALPHA release --------------------------------------------------- * Rewritten from scratch. * Old NEWS file entries removed (see bzr if you're interested).