diff options
| author | Lars Wirzenius <liw@liw.fi> | 2014-04-22 18:49:48 +0100 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2014-04-22 18:49:48 +0100 |
| commit | fb313c774c1edaae5d014793086ebbe174aac567 (patch) | |
| tree | afeac1aa5c74ad63a9075b72ef703d8009cdaac2 | |
| parent | 83c1e57e2f36d0604cfe7b16a068dd7b9a47bf87 (diff) | |
| download | obnam.org-fb313c774c1edaae5d014793086ebbe174aac567.tar.gz | |
Add preliminary content, from old site
238 files changed, 8072 insertions, 12 deletions
diff --git a/1.0.mdwn b/1.0.mdwn new file mode 100644 index 0000000..a49a246 --- /dev/null +++ b/1.0.mdwn @@ -0,0 +1,328 @@ +[[!meta title="Obnam 1.0 (backup software); a story in many words"]] + +**tl;dr**: Version 1.0 of [Obnam](http://liw.fi/obnam/), my +snapshotting, de-duplicating, encrypting backup program is released. +See the end of this announcement for the details. + +Where we see the hero in his formative years; parental influence +---------------------------------------------------------------- + +From the very beginning, my computing life has involved backups. + +In 1984, when I was 14, +[my father](http://www.kolumbus.fi/arnow/) was an independent +telecommunications consultant, which meant he needed a personal computer +for writing reports. He bought a +[Luxor ABC-802](http://en.wikipedia.org/wiki/ABC_800#ABC_802), +a Swedish computer with a Z80 microprocessor and two floppy drives. + +My father also taught me how to use it. When I needed to +save files, he gave me not one, but two floppies, and explained +that I should store my files one one, and then copy them to the +other one every now and then. + +Later on, over the years, I've made backups from a hard disk +(30 megabytes!) to +a stack of floppies, to a tape drive installed into +a floppy interface (400 megabytes!), to a DAT drive, and various other media. +It was always a bit tedious. + +The start of the quest; lengthy justification for NIH +----------------------------------------------------- + +In 2004, I decided to do a full backup, by burning a copy of all my +files onto CD-R disks. It took me most of the day. Afterwards, I sat +admiring the large stack of disks, and realized that I would not ever +do that again. I'm too lazy for that. That I had done it once was an +aberration in the space-time continuum. + +Switching to DVD-Rs instead CD-Rs would reduce to the number of disks to +burn, but not enough: it would still take a stack of them. +I needed something much better. + +I had a little experience with tape drives, and that was enough to convince +me that I didn't want them. Tape drives are expensive hardware, +and the tapes also cost money. If the drive goes bad, you have to get +a compatible one, or all your backups are toast. The price per gigabyte +was coming down fast for hard drives, and it was clear that they were +about to be very competitive with tapes for price. + +I looked for backup programs that I could use for disk based backups. +`rsync`, of course, was the obvious choice, but there were others. +I ended up doing what many geeks do: I wrote my own wrapper around +`rsync`. There's hundred, possibly thousands, of such wrappers around +the Internet. + +I also got the idea that doing a startup to provide online backup +space would be a really cool thing. However, I didn't really do +anything about that until 2007. More on that later. + +The `rsync` wrapper script I wrote used hardlinked directory trees +to provide a backup history, though not in the smart way that +[backuppc](http://backuppc.sourceforge.net/) does it. +The hardlinks were wonderful, because they were +cheap, and provided de-duplication. They were also quite cumbersome, +when I needed to move my backups to a new disk the first time. It +turned out that a lot of tools deal very badly with directory trees +with large numbers of hardlinks. + +I also decided I wanted encrypted backups. This led me to find +[duplicity](http://duplicity.nongnu.org/), which is a nice program +that does encrypted backups, but I had issues with some of its +limitations. To fix those limitations, I would have had to re-design +and possibly re-implement the entire program. The biggest limitation +was that it treated backups as full backup, plus a sequence of +incremental backups, which were deltas against the previous backup. + +Delta based incrementals make sense for tape drives. You run a full +backup once, then incremental deltas for every day. When enough time +has passed since the full backup, you do a new full backup, and then +future incrementals are based on that. Repeat forever. + +I decided that this makes no sense for disk based backups. If I already +have backed up a file, there's no point in making me backup it again, +since it's already there on the same hard disk. It makes even less +sense for online backups, since doing a new full backup would require +me to transmit all the data all over again, even though it's already +on the server. + +The first battle +---------------- + +I could not find a program that did what I wanted to do, and like +every good [NIHolic](http://en.wikipedia.org/wiki/Not_Invented_Here), +I started writing my own. + +After various aborted attempts, I started for real in 2006. Here is +the first commit message: + + revno: 1 + committer: Lars Wirzenius <liw@iki.fi> + branch nick: wibbr + timestamp: Wed 2006-09-06 18:35:52 +0300 + message: + Initial commit. + +`wibbr` was the placeholder name for Obnam until we came up with +something better. We was myself and Richard Braakman, who was going +to be doing the backup startup with me. We eventually founded the +company near the end of 2006, and started doing business in 2007. + +However, we did not do very much business, and ran out of money in +September 2007. We ended the backup startup experiment. +That's when I took a job with Canonical, and Obnam became a hobby +project of mine: I still wanted a good backup tool. + +In September 2007, Obnam was working, but it was not very good. +For example, it was quite slow and wasteful of backup space. + +That version of Obnam used deltas, based on the `rsync` algorithm, to +backup only changes. It did not require the user to do full and +incremental backups manually, but essentially created an endless +sequence of incrementals. It was possible to remove any generation, +and Obnam would manage the deltas as necessary, keeping the ones +needed for the remaining generations, and removing the rest. +Obnam made it look as if each generation was independent of each other. + +The wasteful part was the way in which metadata about files was +stored: each generation stored the full list of filenames and their +permissions and other inode fields. This turned out to be bigger +than my daily delta. + +The lost years; getting lost in the forest +------------------------------------------ + +For the next two years, I did a little work on Obnam, but I did not +make progress very fast. I changed the way metadata was stored, for +example, but I picked another bad way of doing it: the new way was +essentially building a tree of directory and file nodes, and any +unchanged subtrees were shared between generations. This reduced the +space overhead per generation, but made it quite slow to look up +the metadata for any one file. + +The final battle; finding cows in the forest +-------------------------------------------- + +In 2009 I decided to leave Canonical and after that, my Obnam hobby +picked up in speed again. Below is a table of the number of commits +per year, from the very first commit (`bzr log -n0 | +awk '/timestamp:/ { print $3}' | sed 's/-.*//' | uniq -c | +awk '{ print $2, $1 }' | tac`): + + 2006 466 + 2007 353 + 2008 402 + 2009 467 + 2010 616 + 2011 790 + 2012 282 + +During most of 2010 and 2011 I was unemployed, and happily hacking +Obnam, while moving to another country twice. I don't recommend that +as a way to hack on hobby projects, but it worked for me. + +After Canonical, I decided to tackle the way Obnam stores data from +a new angle. Richard told me about the copy-on-write (or COW) B-trees that +btrfs uses, originally designed by Ohad Rodeh +(see [his paper](http://liw.fi/larch/ohad-btrees-shadowing-clones.pdf) +for details), +and I started reading about that. It turned out that +they're pretty ideal for backups: each B-tree stores data about +one generation. To start a new generation, you clone the previous +generation's B-tree, and make any modifications you need. + +I implemented the B-tree library myself, in Python. +I wanted something that +was flexible about how and where I stored data, which the btrfs +implementation did not seem to give me. (Also, I worship at the +altar of NIH.) + +With the B-trees, doing file deltas from the previous generation +no longer made any sense. I realized that it was, in any case, a +better idea to store file data in chunks, and re-use chunks in +different generations as needed. This makes it much easier to +manage changes to files: with deltas, you need to keep a long chain +of deltas and apply many deltas to reconstruct a particular version. +With lists of chunks, you just get the chunks you need. + +The spin-off franchise; lost in a maze of dependencies, all alike +----------------------------------------------------------------- + +In the process of developing Obnam, I have split off a number of +helper programs and libraries: + +* [genbackupdata](http://liw.fi/genbackupdata/) + generates reproducible test data for backups +* [seivot](http://liw.fi/seivot/) + runs benchmarks on backup software (although only Obnam for now) +* [cliapp](http://liw.fi/cliapp/) + is a Python framework for command line applications +* [cmdtest](http://liw.fi/cmdtest/) + runs black box tests for Unix command line applications +* [summain](http://liw.fi/summain/) + makes diff-able file manifests (`md5sum` on steroids), + useful for verifying that files are restored correctly +* [tracing](http://liw.fi/tracing/) + allows run-time selectable debug log messages that is really + fast during normal production runs when messages are not printed + +I have found it convenient to keep these split off, since I've been +able to use them in other projects as well. However, it turns out that +those installing Obnam don't like this: it would probably make sense to +have a fat release with Obnam and all dependencies, but I haven't bothered +to do that yet. + +The blurb; readers advised about blatant marketing +-------------------------------------------------- + +The strong points of Obnam are, I think: + +* **Snapshot** backups, similar to btrfs snapshot subvolumes. + Every generation looks like a complete snapshot, + so you don't need to care about full versus incremental backups, or + rotate real or virtual tapes. + The generations share data as much as possible, + so only changes are backed up each time. +* Data **de-duplication**, across files, and backup generations. If the + backup repository already contains a particular chunk of data, it will + be re-used, even if it was in another file in an older backup + generation. This way, you don't need to worry about moving around large + files, or modifying them. +* **Encrypted** backups, using GnuPG. + +Backups may be stored on local hard disks (e.g., USB drives), any +locally mounted network file shares (NFS, SMB, almost anything with +remotely Posix-like semantics), or on any SFTP server you have access to. + +What's not so strong is backing up online over SFTP, particularly with +long round trip times to the server, or many small files to back up. +That performance is Obnam's weakest part. I hope to fix that in the future, +but I don't want to delay 1.0 for it. + +The big news; readers sighing in relief +--------------------------------------- + +I am now ready to release version 1.0 of Obnam. Finally. It's been +a long project, much longer than I expected, and much longer than +was really sensible. However, it's ready now. It's not bug free, and +it's not as fast as I would like, but it's time to declare it ready +for general use. If nothing else, this will get more people to use +it, and they'll find the remaining problems faster than I can do on +my own. + +I have packaged Obnam for Debian, and it is in `unstable`, and will +hopefully get into `wheezy` before the Debian freeze. I provide +packages built for `squeeze` on my own repository, +see the [download](http://liw.fi/obnam/download/) page. + +The changes in the 1.0 release compared to the previous one: + +* Fixed bug in finding duplicate files during a backup generation. + Thanks to Saint Germain for reporting the problem. +* Changed version number to 1.0. + +The future; not including winning lottery numbers +------------------------------------------------- + +I expect to get a flurry of bug reports in the near future as new people +try Obnam. It will take a bit of effort dealing with that. Help is, of +course, welcome! + +After that, I expect to be mainly working on Obnam performance for the +foreseeable future. There may also be a FUSE filesystem interface for +restoring from backups, and a continous backup version of Obnam. Plus +other features, too. + +I make no promises about how fast new features +and optimizations will happen: Obnam is a hobby project for me, and I +work on it only in my free time. Also, I have a bunch of things that +are on hold until I get Obnam into shape, and I may decide to do one +of those things before the next big Obnam push. + +Where; the trail of an errant hacker +------------------------------------ + +I've developed Obnam in a number of physical locations, and I thought +it might be interesting to list them: +Espoo, Helsinki, Vantaa, Kotka, Raahe, Oulu, Tampere, Cambridge, Boston, +Plymouth, London, Los Angeles, Auckland, Wellington, Christchurch, +Portland, New York, Edinburgh, Manchester, San Giorgio di Piano. +I've also hacked on Obnam in trains, on planes, and once on a ship, +but only for a few minutes on the ship before I got seasick. + +Thank you; sincerely +-------------------- + +* Richard Braakman, for helping me with ideas, feedback, and some + code optimizations, and for doing the startup with me. Even though + he has provided little code, he's Obnam's most significant contributor + so far. +* [Chris Cormack](http://blog.bigballofwax.co.nz/), for helping to build + Obnam for Ubuntu. I no longer use Ubuntu at all, so it's a big help to + not have to worry about building and testing packages for it. +* [Daniel Silverstone](http://www.digital-scurf.org/), for spending a + Saturday with me hacking Obnam, and rewriting the way repository file + filters work (compression, encryption), thus making them not suck. +* [Tapani Tarvainen](http://tapani.tarvainen.info/) for running Obnam for + serious amounts of real data, and for being patient while I fixed things. +* [Soile Mottisenkangas](http://docstory.fi/) for believing in me, and + helping me overcome periods of despair. +* Everyone else who has tried Obnam and reported bugs or provided any + other feedback. I apologize for not listing everyone. + +SEE ALSO +-------- + +* [Obnam home page](http://liw.fi/obnam/) + - [tutorial](http://liw.fi/obnam/tutorial/) + - [support](http://liw.fi/obnam/status/) + - [NEWS](http://liw.fi/obnam/NEWS/) + - [README](http://liw.fi/obnam/README/) + - [manual page](http://liw.fi/obnam/obnam.1.txt) + - [design documents](http://liw.fi/obnam/development/) + - [Debian QA package page](http://packages.qa.debian.org/o/obnam.html) + - [bugs](http://liw.fi/obnam/bugs/) + - [bugs in Debian](http://bugs.debian.org/obnam) +* [Other projects of mine (many are dependencies of + Obnam)](http://liw.fi/tag/program/) + diff --git a/NEWS.mdwn b/NEWS.mdwn new file mode 100644 index 0000000..1e0fd47 --- /dev/null +++ b/NEWS.mdwn @@ -0,0 +1,979 @@ +Obnam NEWS +========== + +This file summarizes changes between releases of Obnam. + +Version 1.7.4, released 2014-03-31 +-------------------------------- + +* The manual is now dual-licensed under GNU GPL v3 or later, and + Creative Commons CC-BY-SA 4.0. + +* The 1.7.3 release never went out. Let's pretend it wasn't even + tagged in git, and everyone will be happy. + +Bug fixes: + +* Obnam FUSE got another bug fix from Valery Yundin, to fix a bug I + introduced in 1.7. Reading big files via `obnam mount` should now + work better. + +* Fix count of backed up files. It used to always count directories. + Reported by Alberto Fuentes as Debian bug + [742384](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=742384). + +* `obnam diff latest` would fail due to a programming error. Reported + by Junyx. + +Version 1.7.2, released 2014-03-22 +-------------------------------- + +Bug fixes: + +* Fix another bug in the FUSE plugin's file reading code, found during + the release process of 1.7.2. + +Version 1.7.1, released 2014-03-22 +-------------------------------- + +* The `dump-repo` command now outputs JSON instead of YAML. The + dependency on PyYAML is no longer. + +Bug fixes: + +* Nemo Inis found a bug in the FUSE plugin (`obnam mount`), where + Obnam would return the wrong data when the program reading the file + didn't read the whole file from the beginning in one read(2) system + call. + +* The test suite now skips tests that require use of extended + attributes in the `user` namespace. This should allow the test suite + to be run on more build servers run by various distributions. + +Version 1.7, released 2014-03-15 +-------------------------------- + +WARNING: This release has had fairly large parts of the internals +re-written. There shouldn't be any externally visible changes due to +that, but there is a chance of bugs. Be careful. Make a copy of your +backup repository before upgrading, if you can. + +* The `convert5to6` subcommand has been removed. If you need to + convert from a pre-1.0 backup repository, and haven't done so yet, + please use Obnam version 1.6.1 or earlier to do so. + +* A new `backup-finished` hook is provided by the backup plugin, so + that other plugins may do processing at the end of a backup, such as + report the successful backup to a monitoring system. Patch by + Enrico Tröger. + +* The FUSE plugin can now refresh its view, by having the user read + the `.pid` file. Patch by Valery Yundin. + +* New option `--always-restore-setuid` to always restore setuid/setgid + flags in permissions, even if the restore is not being run by `root` + or the owner of the files (as recorded in the backup). + +* New option `--exclude-from` allows exclusion patterns to be given in + a separate file (one per line), instead of in a configuration file + or on the command line. Patch by Enrico Tröger. + +* A start of a manual for Obnam. This will gain more content with + new releases. The current versions is mainly an edited version of + Lars's blog posts about backups, plus the Obnam tutorial from the + Obnam homepage. See <http://code.liw.fi/obnam/manual/> for + rendered versions (PDF, HTML). + +* Most of the error messages Obnam produces now have a unique error + code: `ERROR: R0B15DX: Cannot find requested generation for client + havelock` for example. More error messages will gain error codes in + future releases. The error codes are meant to be easy to search for, + and will allow error messages to be translated in the future. + +* The `obnam-benchmark` program got rewritten so that it'll do + something useful, but at the same time, it is no longer useful as a + general tool. It is now expected to be run from the Obnam source + tree (a cloned git repository), and isn't installed anymore. + +* The log file now includes information about the transfer overhead to + the repository. Overhead is all the bytes that are not file content + data: filenames, permission bits, extended attributes, etc, plus + Obnam internal bookkeeping. + +* `obnam verify` now shows progress both based on number of files and + amount of data. + +Bug fixes: + +* Obnam now doesn't remove chunks that are shared between clients. + Previously, this would sometimes happen, because only the first + client would correctly record itself as using a chunk. Now all + clients do that. + +* Obnam now creates a `trustdb.gpg` in the temporary GNUPGHOME it uses + during encryption operations. From version 2.0.22 (or thereabouts), + `gpg` insists on having a `trustdb.gpg` in the GNUPGHOME it uses. + +* When backing up a large file, and making a checkpoint generation in + the middle of it, Obnam would say "continuing backup" after the + checkpoint was finished, instead of saying the name of the file. + This is now fixed. + +Internal changes: + +* The `obnamlib.Error` exception class has been replaced by the + `obnamlib.ObnamError` class, which derives from the new + `obnamlib.StructuredError` class. All new exceptions will need + to be derived from `obnamlib.Error` in the future. Also, due to the + way `StructuredError` works, it is now necessary to create a new + exception class for each kind of error. This gives us unique the + error codes mentioned above. + +* The old `obnamlib.Repository` class is gone, and replaced with the + `obnamlib.RepositoryInterface` class, which gets implemented for + each repository format (there is only one, for now, but there will + be more). + +Version 1.6.1, released 2013-11-30 +---------------------------------- + +* Fix Debian package dependencies correctly. + +Version 1.6, released 2013-11-30 +-------------------------------- + +* Stop logging paramiko exceptions that get converted into another + type of exception by the SFTP plugin in Obnam. + +* `obnam-benchmark` can now use an installed version of larch. + Patch by Lars Kruse. + +* Obnam has been ported to FreeBSD by Itamar Turner-Trauring + of HybridCluster. + +* Backup progress reporting now reports scanned file data, not just + backed up file data. This will hopefully be less confusing to people. + +* The `list-keys`, `client-keys`, and `list-toplevels` commands now + obey a new option, `--key-details`, to show the usernames attached + to each public key. Patch by Lars Kruse. + +* New option `--ssh-command` to set the command Obnam runs + when invoking ssh. patch by Lars Kruse. + +* `obnam clients` can now be used without being an existing client. + Patch by Itamar Turner-Trauring. + +* New option `--ssh-host-keys-check` to better specify how SSH + host keys should be checked. Patch by Itamar Turner-Trauring. + +Bug fixes: + +* Fix`"obnam list-toplevels` so it doesn't give an error when it's + unable to read the per-client directory of another client, when + encryption is used. Fix by Lars Kruse. + +* Fix the encryption plugin to give a better error message when it + looks for client directories but fails to find them. Fix by + Lars Kruse. + +* `obnam list-toplevels` got confused when the repository contained + extra files, such as "lock" (left there by a previous, crashed Obnam + run). It no longer does. Fix by Lars Kruse. + +* The SFTP plugin now handles another error code (EACCESS) when writing + a file and the directory it should go into not existing. Patch by + Armin Größlinger. + +* Obnam's manual page now explains about breaking long logical lines + into multiple physical ones. + +* The `/~/` path prefix in SFTP URLs works again, at least with + sufficiently new versions of Paramiko (1.7.7.1 in Debian wheezy is + OK). Reported by Lars Kruse. + +* The Nagios plugin to report errors in a way Nagios expects. + Patch by Martijn Grendelman. + +* The Nagios plugin for Obnam now correctly handles the case + where a backup repository for a client exists, but does not have + a backup yet. Patch by Lars Kruse. + +* `obnam ls` now handles trailing slashes in filename arguments. + Reported by Biltong. + +* When restoring a backup, Obnam will now continue past errors, + instead of aborting with the first one. Patch by Itamar + Turner-Trauring. + +Version 1.5, released 2013-08-08 +-------------------------------- + +Bug fixes: + +* Terminal progress reporting now updated only every 0.1 seconds, + instead of 0.01 seconds, to reduce terminal emulator CPU usage. + Reported by Neal Becker. +* Empty exclude patterns are ignored. Previously, a configuration file + line such as "exclude = foo, bar," (note trailing comma) would result + in an empty pattern, which would match everything, and therefore + nothing would be backed up. Reported by Sharon Kimble. +* A FUSE plugin to access (read-only) data from the backup repository + has been added. Written by Valery Yundin. + +Version 1.4, released 2013-03-16 +-------------------------------- + +* The``ls` command now takes filenames as (optional) arguments, instead + of a list of generations. Based on patch by Damien Couroussé. +* Even more detailed progress reporting during a backup. +* Add --fsck-skip-generations option to tell fsck to not check any + generation metadata. +* The default log level is now INFO, instead of DEBUG. This is to be + considered a quantum leap in the continuing rise of the maturity level + of the software. (Actually, the change is there just to save some + disk space and I/O for people who don't want to be involved in Obnam + development and don't want to have massive log files.) +* The default sizes for the `lru-size` and `upload-queue-size` settings + have been reduced, to reduce the memory impact of Obnam. +* `obnam restore` now reports transfer statistics at the end, similarly + to what `obnam backup` does. Suggested by "S. B.". + +Bug fixes: + +* If listing extended attributes for a filesystem that does not support + them, Obnam no longer crashes, just silently does not backup extended + attributes. Which aren't there anyway. +* A bug in handling stat lookup errors was fixed. Reported by + Peter Palfrader. Symptom: `AttributeError: 'exceptions.OSError' + object has no attribute 'st_ino'` in an error message or log file. +* A bug in a restore crashing when failing to set extended attributes + on the restored file was fixed. Reported by "S. B.". +* Made it clearer what is happening when unlocking the repository due to + errors, and fixed it so that a failure to unlock is also an error. + Reported by andrewsh. +* The dependency on Larch is now for 1.20121216 or newer, since that is + needed for fsck to work. +* The manual page did not document the client name arguments to the + `add-key` and `remove-key` subcommands. Reported by Lars Kruse. +* Restoring symlinks as root would fail. Reported and fixed by + David Fries. +* Only set ssh user/port if explicitily requested, otherwise let ssh + select them. Reported by Michael Goetze, fixed by David Fries. +* Fix problem with old version of paramiko and chdir. Fixed by Nick Altmann. +* Fix problems with signed vs unsigned values for struct stat fields. + Reported by Henning Verbeek. + +Version 1.3, released 2012-12-16 +-------------------------------- + +* When creating files in the backup repository, Obnam tries to avoid NFS + synchronisation problems by first writing a temporary file and then + creating a hardlink to the actual filename. This works badly on filesystems + that do not allow hard links, such as VFAT. If creating the hardlink + fails, Obnam now further tries to use the `open(2)` system call with + the `O_EXCL` flag to create the target file. This should allow things + to work with both NFS and VFAT. +* More detailed progress reporting during the backup. +* Manual page now covers the diff subcommand. Patch by Peter Valdemar Mørch. +* Speed optimisation patch for backing up files in inode numbering order, + from Christophe Vu-Brugier. +* A setuid or setgid bit is now not restored if Obnam is not used by root + or the same user as the owner of the restored file. +* Many new settings to control "obnam fsck", mainly to reduce the amount + of checking being done in order to make it faster. However, fsck is + has lost some features (checks), which will be added back in a future + release. +* More frequent fsck progress reporting. Some speed optimisations to fsck. + +Bug fixes: + +* Empty values for extended attributes are now backed up correctly. + Previously they would cause an infinite loop. +* Extended attributes without values are now ignored. This is different + from attributes with empty values. Reported by Vladimir Elisseev. +* An empty port number in sftp URLs is now handled correctly. Found based + on report by Anton Shevtsov. +* A bad performance bug when backing up full systems (starting from the + filesystem root directory) has been fixed. At the beginning of each + generation, Obnam removes any directories that are not part of the + current backup roots. This is necessary so that if you change the + backup roots, the old stuff doesn't hang around forever. However, when + the backup root is the filesystem root, due to the now-fixed bug Obnam + would first remove everything, and then back it up all over again. This + "worked", but was quite slow. Thanks to Nix for reporting the problem. +* Obnam now runs GnuPG explicitly with the "no text mode" setting, to override + a "text mode" setting in the user's configuration. The files Obnam encrypts + need to be treated as binary, not text files. Reported by Robin Sheat. +* A shared B-tree concurrency bug has been fixed: If another instance of + Obnam was modifying a shared B-tree, Obnam would crash and abort a backup, + possibly leaving lock files lying around. Now a failure to look up a chunk + via its checksum is ignored, and the backup continues. +* Bugs in how Python OSError exceptions were being raises have been fixed. + Error messages should now be somewhat clearer. +* Unset or wrongly set variable "full" fixed in "obnam diff". Reported + by ROGERIO DE CARVALHO BASTOS and patched by Peter Valdemar Mørch. +* Setuid and setgid bits are now restored correctly, when restore happens + as root. Reported by Pavel Kokolemin. +* Obnam now complains if no backup roots have been specfied. + +Version 1.2, released 2012-10-06 +-------------------------------- + +* Added a note to `--node-size` that it only affects new B-trees. + Thanks, Michael Brown. +* New `obnam diff` subcommand to show differences (added/removed/modified + files) between two generations, by Peter Valdemar Mørch. +* `obnam backup` now logs the names of files that are getting backed up + at the INFO level rather than DEBUG. +* The command synopsises for backup, restore, and verify commands now + make it clearer that Obnam only accepts directories, not individual + files, as arguments. (For now.) +* The output from the `show` plugin can now be redirected with the + `--output=FILE` option. Affected subcommands: `clients`, `generations`, + `genids`, `ls`, `diff`, `nagios-last-backup-age`. + +Bug fixes: + +* Notify user of errors during backups. +* The SFTP plugin now manages to deal with repository paths starting + with `/~/` which already exist without crashing. +* Character and block device nodes are now restored correctly. + Thanks to Martin Dummer for the bug report. +* The symmteric key for a toplevel repository directory is re-encrypted + when a public key is added or removed to the toplevel using the + `add-key` or `remove-key` subcommands. +* Manual page typo fix. Thanks, Steve Kemp. + +Version 1.1, released 2012-06-30 +-------------------------------- + +* Mark the `--small-files-in-btree` settings as deprecated. +* Obnam now correctly checks that `--repository` is set. +* Options in `--help` output are now grouped in random senseless ways + rather than being in one randomly ordered group. +* Manual page clarification for `--root` and `verify`. Thanks, Saint Germain. +* Remove outdated section from manual page explaining that there is not + format conversion. Thanks, Elrond of Samba-TNG. +* Added missing information about specifying a user in sftp URLs. Thanks, + Joey Hess, for pointing it out. +* Manual page clarification on `--keep` from Damien Couroussé. +* Make `obnam forget` report which generations it would remove without + `--pretend`. Thanks, Neal Becker, for the suggestion. + +Version 1.0, released 2012-06-01 +-------------------------------- + +* Fixed bug in finding duplicate files during a backup generation. + Thanks to Saint Germain for reporting the problem. +* Changed version number to 1.0. + +Version 0.30, released 2012-05-30; a RELEASE CANDIDATE +------------------------------------------------------ + +Only bug fixes, and only in the test suite. + +* Fix test case problem when `$TMPDIR` lacks `user_xattr`. The extended + attributes test won't succeed in that case, and it's pointless to run it. +* Fix test case problem when `$TMPDIR` lacks nanosecond timestamps for + files. The test case now ignores such timestamps, making the test pass + anyway. The timestamp accuracy is not important for this test. + +Version 0.29, released 2012-05-27; a RELEASE CANDIDATE +------------------------------------------------------ + +* "obnam backup" now writes performance statistics at the end of a backup run. + Search the log for "Backup performance statistics" (INFO level). +* "obnam verify" now continues past the first error. Thanks to Rafał Gwiazda + for requesting this. +* Add an `obnam-viewprof` utility to translate Python profiling output + into human readable text form. +* Bug fix: If a file's extended attributes have changed in any way, the change + is now backed up. +* "obnam fsck" is now a bit faster. +* The shared directories in the repository are now locked only during updates, + allowing more efficient concurrent backups between several computers. +* Obnam now gives a better error message when a backup root is not a + directory. Thanks to Edward Allcutt for reporting the error + (<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654211>). +* The output format of "obnam ls" has changed. It now has one line per + file, and includes the full pathname of the file, rather mimicking the + output of "ls -lAR". Thanks to Edward Allcutt for the suggestion + (<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655095>). +* A few optimizations to sftp speed. Small files are still slow. + +Version 0.28, released 2012-05-10; a BETA release +------------------------------------------------- + +* `force-lock` should now remove all locks. +* Out-of-space errors in the repository now terminate the backup process. + Previously, Obnam would continue, ignoring the failure to write. If you + make space in the repository and restart Obnam, it will continue from + the previous checkpoint. +* The convert5to6 black box test now works even if run by other people + than liw. +* "obnam backup" now uses a single SFTP connection to the backup repository, + rather than opening a new one after each checkpoint generation. Thanks to + weinzwang for reporting the problem. +* "obnam verify" now obeys the `--quiet` option. +* "obnam backup" no longer counts chunks already in the repository in the + uploaded amount of data. + +Version 0.27, released 2012-04-30; a BETA release +------------------------------------------------- + +* The repository format has again changed in an incompatible manner, + so you will need to re-backup everything again. Alternatively, you can + try the new `convert5to6` subcommand. See the manual page for details. + Make sure you have a copy of the repository before converting, the + code is new and may be buggy. +* New option `--small-files-in-btree` enables Obnam to store the contents + of small files in the per-client B-tree. This is not the default, at + least yet, since it's impact on real life performance is unknown, but + it should make things go a bit faster for high latency repository + connections. +* Some SFTP related speed optimizations. +* Data filtering is now strictly stable and priority-ordered, ensuring that + compression always happens before encryption etc. +* Repository metadata is never filtered, so that we can be sure that + in future if when we add backwards-compatibility we can detect the format + without worrying about any other filtering which might occur. +* Forcing of locks is now unconditional and across the entire repository. +* Uses the larch 0.30 read-only mode to fix a bug where opening a B-tree + rolls back changes someone else is making, even if we only use the tree + to read stuff from. +* "obnam backup" will now exit with a non-zero exit code if there were + any errors during a backup, and the problematic files were skipped. + Thanks, Peter Palfrader, for reporting the bug. +* "obnam forget" is now a bit faster. +* Hash collisions for filenames are now handled. + +Version 0.26, released 2012-03-26; a BETA release +------------------------------------------------- + +* Clients now lock the parts of the backup repository they're using, + while making any changes, so that multiple clients can work at the + same time without corrupting the repository. +* Now depends on a larch 0.28, which uses journalling to avoid on-disk + inconsistencies and corruption during crashes. +* Compression and encryption can now be used together. + +Version 0.25, released 2012-02-18; a BETA release +------------------------------------------------- + +* Log files are now created with permissions that allow only the owner + to read or write them. This fixes a privacy leak. +* The `nagios-last-backup-age` subcommand is useful for setting up Nagios + (or similar systems) to check that backups get run properly. Thanks to + Peter Palfrader for the patch. +* Some clarification on how the forget policy works, prompted by questions + from Peter Palfrader. +* New settings `ssh-known-hosts` (for choosing which file to check for + known host keys), `strict-ssh-host-keys` (for disallowing unknown host + keys), and `ssh-key` (for choosing which key file to use for SSH + connections) allow better and safer use of ssh. +* Checkpoints will now happen even in the middle of files (but between + chunks). +* The `--pretend` option now works for backups as well. + +BUG FIXES: + +* `obnam ls` now shows the correct timestamps for generations. + Thanks, Anders Wirzenius. + +Version 0.24.1, released 2011-12-24; a BETA release +------------------------------------------------- + +BUG FIX: + +* Fix test case for file timestamps with sub-second resolution. Not all + filesystems have that, so the test case has been changed to accept lack + of sub-second timestamps. + +Version 0.24, released 2011-12-18; a BETA release +------------------------------------------------- + +USER VISIBLE CHANGES + +* The way file timestamps (modification and access times) have changed, + to fix inaccuracies introduced by the old way. Times are now stored + as two integers giving full seconds and nanoseconds past the full + second, instead of the weird earlier system that was imposed by Python's + use of floating point for the timestamps. This causes the repository + format version to be bumped, resulting in a need to start over with an + empty repository. +* Extended file attributes are now backed up from and restored to local + filesystems. They are neither backed up, nor restored for live data + accessed over SFTP. +* If the `--exclude` regular expression is wrong, Obnam now gives an + error message and then ignores the regexp, rather than crashing. +* There is now a compression plugin, enabled with `--compress-with=gzip`. +* De-duplication mode can now be chosen by the user: the new + `--deduplicate` setting can be one of `never` (fast, but uses more space); + `verify` (slow, but handles hash collisions gracefully); and + `fatalist` (fast, but lossy, if there is a hash collision). `fatalist` + is the default mode. +* Restores now obey the `--dry-run` option. Thanks to Peter Palfreder for + the bug report. +* New option `--verify-randomly` allows you to check only a part of the + backup, instead of everything. +* Verify now has some progress reporting. +* Forget is now much faster. +* Forget now has progress reporting. It is not fast enough to do without, + sorry. +* Backup now removes any checkpoint generations it created during a backup + run, if it succeeds without errors. + +BUG FIXES: + +* Now works with a repository on sshfs. Thanks to Dafydd Harries for + reporting the problem. +* Now depends on a newer version of the larch library, fixing a problem + when the Obnam default node size changes and an existing repository + has a different size. +* User and group names for sftp live data are no longer queried from the + local system. Instead, they're marked as unknown. + +Version 0.23, released 2011-10-02; a BETA release +------------------------------------------------- + +USER VISIBLE CHANGES: + +* `restore` now shows a progress bar. +* `fsck` now has more useful progress reporting, and does more checking, + including the integrity of the contents of file content. +* `fsck` now also checks the integrity of the B-trees in the repository, + so that it is not necessary to run `fsck-larch` manually anymore. This + works remotely as well, whereas `fsck-larch` only worked on B-trees + on the local filesystem. +* `force-lock` now gives a warning if the client does not exist in the + repository. +* Subcommands for encryption now give a warning if encryption key is not + given. +* The `--fsck-fix` option will now instruct `obnam fsck` to try to fix + problems found. For this release, it only means fixing B-tree missing + node problems, but more will follow. +* The default sizes have been changed for B-tree nodes (256 KiB) + and file contents chunks (1 MiB), based on benchmarking. +* SFTP protocol use has been optimized, which should result in some + more speed. This also highlights the need to change obnam so it can + do uploads in the background. +* If a client does not exist in the repository, `force-lock` now gives + a warning to the user, rather than ignoring it silently. + +DEVELOPER CHANGES: + +* New `--sftp-delay=100` option can be used to simulate SFTP backups over + networks with long round trip times. +* `obnam-benchmark` can now use `--sftp-delay` and other changes to make + it more useful. + +INTERNAL CHANGES: + +* Got rid of terminal status plugin. Now, the `Application` class provides + a `ttystatus.TerminalStatus` instance instead, in the `ts` attribute. + Other plugings are supposed to use that for progress reporting and + messaging to the user. +* The `posix_fadvise` system call is used only if available. This should + improve Obnam's portability a bit. + +Version 0.22, released 2011-08-25; a BETA release +------------------------------------------------- + +USER VISIBLE CHANGES: + +* Obnam now reports its current configuration in the log file at startup. + This will hopefully remove one round of "did you use the --foo option?" + questions between developers and bug reporters. + +BUG FIXES: + +* The repository is now unlocked on exit only if it is still locked. +* A wrongly caught `GeneratorExit` is now dealt with properly. +* Keyboard interrupts are logged, so they don't show up as anonymous errors. + +CHANGES RELEVANT TO DEVELOPERS ONLY: + +* `setup.py` has been enhanced to work more like the old `Makefile` did: + `clean` removes more artifacts. Instructions in `README` have been updated + to point at `setup.py`. +* Compiler warning about `_XOPEN_SOURCE` re-definition fixed. +* Tests are now again run during a Debian package build. + +Version 0.21, released 2011-08-23; a BETA release +------------------------------------------------- + +USER VISIBLE CHANGES: + +* Obnam will now unlock the repository if there's an error during a backup. + For the most part, the `force-lock` operation should now be unnecessary, + but it's still there in case it's useful some day. + +BUG FIXES: + +* Negative timestamps for files now work. Thanks to Jamil Djadala + for reporting the bug. +* The documentation for --checkpoint units fixed. Thanks, user weinzwang + from IRC. +* The connections to the repository and live data filesystem are now + properly closed. This makes benchmark read/write statistics be correct. + +Version 0.20.1, released 2011-08-11; a BETA release +------------------------------------------------- + +BUG FIXES: + +* More cases of Unicode strings versus plain strings in filenames + over SFTP fixed. Thanks to Tapani Tarvainen. + +Version 0.20, released 2011-08-09; a BETA release +------------------------------------------------- + +BUG FIXES: + +* Non-ASCII filenames over SFTP root now work. (Thanks, Tapani Tarvainen, + for the reproducible bug report.) +* The count of files while making a backup now counts all files found, + not just those backed up. The old behavior was confusing people. + +USER VISIBLE CHANGES: + +* The output of `obnam ls` now formats the columns a little prettier, + so that wide values do not cause misalignment. +* The error message when trying to use an encrypted repository without + encryption is now better (and suggests missing encryption being the + reason). Thanks, chrysn. +* Obnam now supports backing up of Unix sockets. + +Version 0.19, released 2011-08-03; a BETA release +------------------------------------------------- + +INCOMPATIBILITY CHANGES: + +* We now require version 0.21 of the `larch` library, and this requires + bumping the repository format. This means old backup repositories can't + be used with this version, and you need to back up everything again. + (Please tell me when this becomes a problem.) + +BUG FIXES: + +* Found one more place where a file going missing during a backup may + cause a crash. +* Typo in error message about on-disk formats fixed. + (Thanks, Tapani Tarvainen.) +* The `--trace` option works again. +* `fcntl.F_SETFL` does not seem to work on file descriptors for files + owned by root that are read-only to the user running obnam. Worked + around by ignoring any problems with setting the flags. +* The funnest bug in this release: if no log file was specified with `--log`, + the current working directory was excluded from the backup. + +USER VISIBLE CHANGES: + +* `obnam(1)` manual page now discusses how configuration files are used. +* The manual page describes problems using sftp to access live data. +* The documentation for `--no-act` was clarified to say it only works + for `forget. (Thanks, Daniel Silverstone.) +* `obnam-benchmark` now has a manual page. +* The backup plugin logs files it excludes, so the user can find out what's + going on. A confused user is an unhappy user. + +INTERNAL STUFF: + +* Tracing statements added to various parts of the code, to help debug + mysterious problems. +* All exceptions are derived from `obnamlib.AppException` or + `obnamlib.Error`, and those are derived from `cliapp.AppException`, + so that the user gets nicer error messages than Python stack traces. +* `blackboxtests` is no longer run under fakeroot, because Debian packages + are built under fakeroot, and fakeroot within fakeroot causes trouble. + However, the point of running tests under fakeroot was to make sure + certain kinds of bugs are caught, and since Debian package building runs + the tests anyway, the test coverage is not actually diminished. +* The `Makefile` has new targets `fast-check` and `network-tests`. The + latter runs tests over sftp to localhost. + +Version 0.18, released 2011-07-20; a BETA release +------------------------------------------------- + +* The repository format has again changed in an incompatible manner, + so you will need to re-backup everything again. (If this is a problem, + tell me, and I'll consider adding backwards compatibility before 1.0 + is released.) +* New option `--exclude-caches` allows automatic exclusion of cache + directories that are marked as such. +* Obnam now makes files in the repository be read-only, so that they're + that much harder to delete by mistake. +* Error message about files that can't be backed up now mentions the + correct file. +* Bugfix: unreadable files and directories no longer cause the backup + to fail. The problems are reported, but the backup continues. + Thanks to Jeff Epler for reporting the bug. +* Speed improvement from Jeff Epler for excluding files from backups. +* Various other speed improvements. +* Bugfix: restoring symlinks now works even if the symlink is restored + before its target. Also, the permissions of the symlink (rather than its + target) are now restored correctly. Thanks to Jeff Epler for an + exemplary bug report. +* New option `--one-file-system`, from Jeff Epler. +* New benchmarking tool `obnam-benchmark`, which is more flexible than + the old `run-benchmark`. +* When encrypting/decrypting data with GnuPG, temporary files are no + longer used. +* When verifying, `.../foo` and `.../foo/` now work the same way. +* New option `--symmetric-key-bits`. +* The chunk directory uses more hierarchy levels, and the way chunks + are stored there is now user-configurable (but you'll get into trouble + if you don't always use the same configuration). This should speed + things up a bit once the number of chunks grows very large. +* New `--chunkids-per-group` option, for yet more knobs to tweak when + searching for optimal performance. +* Local files are now opened using `O_NOATIME` so they can be backed + up without affecting timestamps. +* Now uses the `cliapp` framework for writing command line applications. + The primary user-visible effect is that the manpage now has an + accurate list of options. +* Bugfix: Obnam now again reports VFS I/O statistics. +* Bugfix: Obnam can again back up live data that is accessed using sftp. + Thanks to Tapani Tarvainen for reporting the problem. + +Version 0.17, released 2011-05-21; a BETA release +------------------------------------------------- + +* This is the second BETA release. +* The `run-benchmark` script now works with the new version of `seivot`. + The only benchmark size is one gibibyte, for now, because Obnam's too + slow to do big ones in reasonable time. As an aside, the benchmark + script got rewritten in Python, so it can be made more flexible. +* Benchmarks are run using encrypted backups. +* The kernel buffer cache is dropped before each obnam run, so the + benchmark result is more realistic (read: slower). +* Obnam now rotates its logs. See `--log-max` and `--log-keep` options + in the manual page. The default location for the log file is now + `~/.cache/obnam/obnam.log` for people, and + `/var/log/obnam.log` for root. +* Obnam now restores sparse files correctly. +* There have been some speed improvements to Obnam. +* The `--repository` option now has the shorter alias `-r`, since it + gets used so often. +* `obnam force-lock` now merely gives an error message, instead of a + Python stack trace, if the repository does not exist. +* Obnam now does not crash if files go missing during a backup, or can't + be read, or there are other problems with them. It will report the + problem, but then continue as if it had never heard of the file. +* Obnam now supports FIFO files. +* Obnam now verifies checksums when it restores files. +* Obnam now stores the checksum for the whole file, not just the checksum + for each chunk of its contents. +* Obnam's own log file is automatically excluded from backups. +* Obnam now stores and restores file timestamps to full accuracy, + instead of truncating them to whole seconds. +* The format of the backup repository has changed in an incompatible way, + and Obnam will now refuse to use an old repository. This means you + will need to use an old version to restore from them, and need to + re-backup everything. Sorry. + +Version 0.16, released 2011-07-17; a BETA release +------------------------------------------------- + +* This is the first BETA release. Obnam should now be feature complete + for real use. Performance is lacking and there are many bugs remaining. + There are no known bugs that would corrupt backed up data, or prevent + its recovery. +* Add encryption support. See the manual page for how to use it. + +Version 0.15.1, released 2011-03-21; an ALPHA release +---------------------------------------------------- + +* Fix `setup.py` to not import `obnamlib`, so it works when building under + pbuilder on Debian. Meh. + +Version 0.15, released 2011-03-21; an ALPHA release +---------------------------------------------------- + +Bugs fixed: + +* Manual page GPL copyright blurb is now properly marked up as a comment. + (Thanks, Joey Hess.) +* README now links to python-lru correctly. (Thanks, Erik Johansson.) + +Improvements and other changes: + +* Filenames and directories are backed up in sorted order. This should + make it easier to know how far obnam's gotten. +* The location where backups are stored is now called the repository, + instead of the store. Suggested by Joey Hess. +* The repository and the target directory for restored data are now + both created by Obnam, if they don't already exist. Suggested by + Joey Hess. +* Better control of logging, using the new `--trace` option. +* Manual page now explains making backups a little better. +* Default value for `--lru-size` reduced to 500, for great improvement + in memory used, without, it seems, much decrease in speed. +* `obnam verify` now reports success explicitly. Based on question + from Joey Hess. +* `obnam verify` now accepts both non-option arguments and the `--root` + option. Suggested by Joey Hess. +* `obnam forget` now accepts "generation specifiers", not just numeric + generation ids. This means that `obnam forget latest` works. +* I/O statistics are logged more systematically. +* `obnam force-lock` introduced, to allow breaking a lock left behind + if obnam crashes. But it never does, of course. (Well, except if there's + a bug, like when a file changes at the wrong moment.) +* `obnam genids` introduced, to list generation ids without any other data. + The old command `obnam generations` still works, and lists other info + about each generation as well, but that's sometimes bad for scripting. +* The `--dump-memory-profile` option now accepts the value `simple`, for + reporting basic memory use. It has such a small impact that it's the + default. +* Obnam now stores the version of the on-disk format in the repository. + This should allow it to handle repositories created by a different + version and act suitably (hopefully without wiping all your backups). + +Version 0.14, released 2010-12-29; an ALPHA release +---------------------------------------------------- + +This version is capable of backing up my laptop's home directory. +It is, however, still an ALPHA release, and you should not rely on +it as your sole form of backup. It is also slow. But if you're +curious, now would be a good time to try it out a bit. + +Bug fixes: + +* `COPYING` now contains GPL version 3, instead of 2. The code was + licensed under version 3 already. (Thank you Greg Grossmeier.) +* The manual page now uses `-` and `\-` correctly. +* `obnam forget` now actually removes data that is no longer used by + any generation. +* When backing up a new generation, if any of the root directories for + the backup got dropped by the user, they are now also removed from + the backup generation. Old generations obviously still have them. +* Only the per-client B-tree forest should have multiple trees. Now this + actually happens, whereas previously sometimes a very large number of + new trees would be created in some forests. (What's good for rain + forests is not good for saving disk space.) +* When recursing through directory trees, obnam no longer follows + symlinks to directories. +* obnam no longer creates a missing backup store when backing up to + a local disk. It never did this when backing up via sftp. (This + saves me from figuring out which of `store`, `stor`, and `sorte` + is the real directory.) + +New features and stuff: + +* `blackboxtest` has been rewritten to use Python's `unittest` + framework, rather than a homegrown bad re-implementation of some of it. +* `obnam ls` interprets arguments as "genspecs" rather than generation + identifiers. This means `obnam ls latest` works, and now `latest` is + also the default if you don't give any spec. +* `run-benchmarks` now outputs results into a git checkout of + <http://braawi.org/>, an ikiwiki instance hosted by + <http://www.branchable.com/>. The script also puts the results into + a suitable sub-directory, adds a page for the RSS feed of benchmark + results, and updates the report page that summarizes all stored results. +* There is now a 100 GiB benchmark. +* Clients are now called clients, instead of hosts. This terminology should + be clearer. +* The list of clients now stores a random integer identifier for each client + (unique within the store). The identifier is used as the name of the + per-client B-tree directory, rather than the hostname of the client. + This should prevent a teeny tiny bit of information leakage. It also + makes debugging things much harder. +* Various refactorings and prettifications of the code has happened. + For example, several classes have been split off from the `store.py` + module. This has also resulted in much better test coverage for those + classes. +* The per-client trees (formerly GenerationStore, now ClientMetadataTree) + have a more complicated key now: 4 parts, not 3. This makes it easier + to keep separate data about files, and other data that needs to be + stored per-generation, such as what the generation id is. +* `find-duplicate-chunks`, a tool for finding duplicate chunks of data + in a files in a directory tree, was added to the tree. I have used it + to find out if is worthwhile to do duplicate chunk removal at all. + (It is, at least for my data.) Also, it can be used to find good + values for chunk sizes for duplicate detection. +* The whole way in which obnam does de-duplication got re-designed and + re-implemented. This is tricky stuff, when there is more than one client. +* `SftpFS` now uses a hack copied from bzrlib, to use openssh if it is + available, and paramiko only if it is not. This speeds up sftp data + transfers quite a bit. (Where bzrlib supports more than just openssh, + we don't, since I have no way to test the other stuff. Patches welcome.) +* The way lists of chunk ids are stored for files got changed. Now we store + several ids per list item, which is faster and also saves some space + in the B-tree nodes. Also, it is now possible to append to the list, + which means the caller does not need to first gather a list of all ids. + Such a list gets quite costly when the file is quite big (e.g., in the + terabyte size). +* New `--dump-memory-profile` option was added to help do memory profiling + with meliae or heapy have been added. (Obnam's memory consumption finally + got annoying enough that I did something about it.) + +Removed stuff: + +* The functional specification was badly outdated, and has been removed. + I decided to stop kidding myself that I would keep it up to date. +* The store design document has been removed from the store tree. + The online version at <http://braawi.org/obnam/ondisk/> is the + canonical version, and is actually kept up to date. +* The benchmark specification has likewise been replaced with + <http://braawi.org/obnam/benchmarkspec/>. + + +Version 0.13, released 2010-07-13; an ALPHA release +---------------------------------------------------- + +* Bug fix: a mistake in 0.12 caused checkpoints to happen after each + file after the first checkpoint. Now they happen at the right intervals + again. +* Upload speed is now displayed during backups. +* Obnam now tells the kernel that it shouldn't cache data it reads or + writes. It is not likely that data being backed up is going to be + needed again any time soon, so there's no point in caching it. + (The posix_fadvise call is used for this.) +* New --lru-size option sets size of LRU cache for nodes in memory. + The obnam default is large enough to suit large backups. This uses more + memory, but is faster than btree's small default of 100. + +Version 0.12, released 2010-07-11; an ALPHA release +---------------------------------------------------- + +* NOTE: This version makes incompatible changes to the way data is stored + on-disk. Backups made with older versions are NOT supported. Sorry. +* The run-benchmark script has dropped some smaller sizes (they're too + fast to be interesting), and adds a 10 GiB test size. +* Various speed optimizations. Most importantly, the way file metadata + (results of lstat(2)) are encoded has changed. This is the incompatible + change from above. It's much faster now, though. +* Preliminary support for using SFTP for the backup store added. Hasn't + been used much yet, so might well be very buggy. + + +Version 0.11, released 2010-07-05; an ALPHA release +---------------------------------------------------- + +* Speed optimizations: + - chunk identifiers are now sequential, except for the first one, or + when there's a collision + - chunks are now stored in a more sensible directory hierarchy (instead + of one per directory, on average) + - adding files to a directory in the backup store is now faster + - only store a file's metadata that if it is changed +* New --exclude=regexp option to exclude files based on pathnames +* Obnam now makes checkpoints during backups. If a backup is aborted + in the middle and then re-started, it will continue from the latest + checkpoint rather than from the beginning of the previous backup run. + - New option --checkpoint to set the interval between checkpoints. + Defaults to 1 GiB. +* Options for various B-tree settings. This is mostly useful for finding + the optimal set of defaults, but may be useful in other situations for + some people. + - New options --chunk-group-size, --chunk-size, --node-size, + --upload-queue-size. +* Somewhat better progress reporting during backups. + + +Version 0.10, released 2010-06-29; an ALPHA release +--------------------------------------------------- + +* Rewritten from scratch. +* Old NEWS file entries removed (see bzr if you're interested). diff --git a/README.mdwn b/README.mdwn new file mode 100644 index 0000000..11be4c3 --- /dev/null +++ b/README.mdwn @@ -0,0 +1,176 @@ +Obnam, a backup program +======================= + +Obnam is a backup program. + + +Home page +--------- + +The Obnam home page is at <http://liw.fi/obnam/>, see there +for more information. + + +Installation +------------ + +The source tree contains packaging for Debian. Run `debuild -us -uc -i.git` to +build an installation package. + +On other systems, using the `setup.py` file should work: run +"python setup.py --help" for advice. If not, please report a bug. +(I've only tested `setup.py` enough for to build the Debian package.) + +You need Python 2.6 or 2.7 (Python 3 is not yet supported). You also +need to install my Python B-tree library, and some of my other +libraries and tools, which you can get from: + +* <http://liw.fi/larch/> +* <http://liw.fi/ttystatus/> +* <http://liw.fi/coverage-test-runner/> (for automatic tests) +* <http://liw.fi/tracing/> +* <http://liw.fi/cliapp/> +* <http://liw.fi/genbackupdata/> +* <http://liw.fi/summain/> +* <http://liw.fi/cmdtest/> +* <http://liw.fi/seivot/> (for benchmarks) + +You also need third party libraries: + +* paramiko: <http://www.lag.net/paramiko/> + +See debian/control for the full set of build dependencies and runtime +dependencies on a Debian system. (That set actually gets tested. The +above list is maintained manually and may get out of date from time +to time.) + +Use +--- + +To get a quick help summary of options: + + ./obnam --help + +To make a backup: + + ./obnam backup --repository /tmp/mybackup $HOME + +For more information, see the manual page: + + man -l obnam.1 + + +Hacking +------- + +Obnam source code is stored in git for version control purposes; +you can get a copy as follows: + + git clone git://git.liw.fi/obnam + +The 'master' branch is the main development one. Any bug fixes and +features should be developed in a dedicated branch, which gets merged +to master when the changes are done and considered good. + +To build and run automatic tests: + + ./check + ./check --fast # unit tests only, no black box tests + ./check --network # requires ssh access to localhost + +`check` is a wrapper around `python setup.py`, but since using that +takes several steps, the script makes things easier. + +You need my CoverageTestRunner to run tests, see above for where to get it. +A couple of scripts exist to run benchmarks and profiles: + + ./metadata-speed 10000 + ./obnam-benchmark --size=1m/100k --results /tmp/benchmark-results + viewprof /tmp/benchmark-results/*/*backup-0.prof + seivots-summary /tmp/benchmark-results/*/*.seivot | less -S + +There are two kinds of results: Python profiling output, and `.seivot` +files. + +For the former, `viewprof` is a little helper script I wrote, +around the Python pstats module. +You can use your own, or get mine from extrautils +(<http://liw.fi/extrautils/>). Running the benchmarks under profiling +makes them a little slower (typically around 10% for me, when I've +compared), but that's OK: the absolute numbers of the benchmarks are +less important than the relative ones. It's nice to be able to look at +the profiler output, if a benchmark is surprisingly slow, without +having to re-run it. + +`seivots-summary` is a tool to display summaries of the measurements +made during a benchmark run. `seivot` is the tool that makes the +measurements. I typically save a number of benchmark results, so that +I can see how my changes affect performance over time. + +If you make any changes, I welcome patches, either as plain diffs, +`git format-patch --cover-letter` mails, or public repositories I can +merge from. + +The code layout is roughly like this: + + obnamlib/ # all the real code + obnamlib/plugins/ # the plugin code (see pluginmgr.py) + obnam # script to invoke obnam + _obnammodule.c # wrapper around some system calls + +In obnamlib, every code module has a corresponding test module, +and "make check" uses CoverageTestRunner to run them pairwise. For +each pair, test coverage must be 100% or the test will fail. +Mark statements that should not be included in coverage test with +"# pragma: no cover", if you really, really can't write a test. +without-tests lists modules that have no test modules. + +If you want to make a new release of Obnam, I recommend following +my release checklist: <http://liw.fi/obnam/release/>. + +Feedback +-------- + +I welcome bug fixes, enhancements, bug reports, suggestions, requests, +and other feedback. I prefer e-mail the mailing list: +see <http://vlists.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net> +for instructions. + +It would be helpful if you can run `make clean check` before submitting +a patch, but it is not strictly required. + + +Legal stuff +----------- + +Most of the code is written by Lars Wirzenius. (Please provide patches +so that can change.) + +This entire work is covered by the GNU General Public +License, version 3 or later. + +> Copyright 2010-2013 Lars Wirzenius +> +> This program is free software: you can redistribute it and/or modify +> it under the terms of the GNU General Public License as published by +> the Free Software Foundation, either version 3 of the License, or +> (at your option) any later version. +> +> This program is distributed in the hope that it will be useful, +> but WITHOUT ANY WARRANTY; without even the implied warranty of +> MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +> GNU General Public License for more details. +> +> You should have received a copy of the GNU General Public License +> along with this program. If not, see <http://www.gnu.org/licenses/>. + +A copy of the GPL is included in the file `COPYING` in the source tree. + +The manual (all the contents of the `manual` subdirectory) is +additionally licensed under a Creative Commons Attribution 4.0 +International License. You can choose whether to use the GPL or the CC +license for the manual. + +A copy of the Creative Commons license is included in the file +`CC-BY-SA-4.0.txt` in the source tree, and can be viewed online +at <http://creativecommons.org/licenses/by-sa/4.0/legalcode>. diff --git a/bug-reporting.mdwn b/bug-reporting.mdwn new file mode 100644 index 0000000..8316bed --- /dev/null +++ b/bug-reporting.mdwn @@ -0,0 +1,29 @@ +If you have a problem with Obnam, +and you want to report it (please do!), +including the following information is helpful +and makes it easier to figure out what the problem is. + +* What is the problem? +* The version of Obnam and Larch you're using, and how you installed it. +* The exact command line you used. Copy-paste it instead of + typing it again into the mail. +* If there's an error message, copy-paste that into the mail. +* The output of `obnam --dump-config`, which includes the full configuration. + Include it as an attachment. +* If you can reproduce the problem while running with `--log=obnam.log` + and `--trace=obnamlib` options, include a suitable amount from the + end of the log file. The suitable amount may depend on the situation, + but if you give the last two hundred lines, and it's + not enough, we'll ask for more. +* The output of the `env` command, in the same terminal window in which + you ran Obnam. (Again, as an attachment.) +* If your bug is about performance, please run Obnam under profiling, + and mail Lars or the [[obnam mailing list|contact]] the profiling file. + To run Obnam under profiling, install the Python profile + (`python-profiler` package in Debian/Ubuntu), and set + the `OBNAM_PROFILE` environment variable to the name of the file + with the profiling output (that's the file you should send by mail). + For example: `OBNAM_PROFILE=obnam.prof obnam backup` would run the + backup under the profiler, and write the result to `obnam.prof`. + +Thanks! diff --git a/bugs.mdwn b/bugs.mdwn new file mode 100644 index 0000000..99e3e58 --- /dev/null +++ b/bugs.mdwn @@ -0,0 +1,32 @@ +Open bugs in Obnam +================== + +If you have a problem with Obnam, please send mail to the mailing list. +Don't add one to this list. +See the [[contact]] page for information about the list. This wiki page +is meant to help developers keep track of confirmed bugs, and not as +a support channel. (This has changed on 2012-07-08.) + +See [[bug-reporting]] page for hints on what to include in a bug report, +if you're unsure. + +See also bugs that are [[done]], and bugs. + +See also: + +* [[done]] +* [[non-wishlist]] (and also not performance related) +* [[wishlist bugs only|wishlist-only]] +* [[performance bugs only|performance-only]] +* [Bugs in Debian](http://bugs.debian.org/obnam) + +[[!inline pages="page(obnam/bugs/*) and + !obnam/bugs/*/* and + !obnam/bugs/done and + !obnam/bugs/non-wishlist and + !obnam/bugs/wishlist-only and + !obnam/bugs/performance-only and + !link(obnam/bugs/done)" + show=0 + postform=no ]] + diff --git a/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn b/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn new file mode 100644 index 0000000..aa4c203 --- /dev/null +++ b/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn @@ -0,0 +1,24 @@ +>On Sun, Feb 26, 2012 at 02:12:18AM -0600, Adam Porter wrote: +>> On Sat, Feb 25, 2012 at 04:00, Lars Wirzenius wrote: +>> > Bzr now has a fix for combining compression and encryption. :) +>> +>> Thanks for fixing that. Will I need to backup from scratch when I +>> start using compression, or can compressed and uncompressed data be +>> mixed in a repo? +> +>Right now you need to re-backup everything, I think. If you open a +>bug about this, I'll make it work without having to do that. + +--- + +As requested. :) It seems like it would be a good idea for each chunk to know whether it's compressed and/or encrypted. It's possible that some backup runs might intentionally skip compression (e.g. MP3s or JPGs won't benefit much from gzip). + +I also wonder, how would Obnam handle it if a user accidentally mixed encrypted and unencrypted data in a repo? + +--- + +As of repository format 6 (Obnam 0.27?) the system should transparently decompress data even if you don't ask it to, and should be able to detect data and read uncompressed data if you ask it to compress. This is part of the 'stable filters' work. + +As such, later allowing mixing of uncompressed/compressed data should be easy. + +[[done]] diff --git a/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn b/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn new file mode 100644 index 0000000..7d51f39 --- /dev/null +++ b/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Currently, there seems to be no easy way to forget all (or all but the newest) checkpoint generations. Something like + + obnam --keep 1c forget + +would be nice. + +-- weinzwang diff --git a/bugs/Add_more_content_browsing_possibilities.mdwn b/bugs/Add_more_content_browsing_possibilities.mdwn new file mode 100644 index 0000000..66d40e6 --- /dev/null +++ b/bugs/Add_more_content_browsing_possibilities.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-wishlist]] + +It would be very useful to have a simple way to get a list of the generations a given file appears in, preferably with the relevant metadata. + +A way to walk through the backup repository would also be useful - think smbclient-like functionality; or maybe as already mentioned a fuse filesystem. + +--- + +I think the FUSE filesystem will cover all of the above. This is thus a duplicate request. --liw + +[[done]] diff --git a/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn b/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn new file mode 100644 index 0000000..036aae4 --- /dev/null +++ b/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn @@ -0,0 +1,25 @@ +obnam forget doesn't update its cli status. It says + + forgetting generation 0/39 + +for a few minutes, then updates to + + forgetting generation 39/39 + +prior to finishing. The generations are removed as expected though. + +--weinzwang + +*Update:* When obnam automatically removes snapshot generations after finishing a backup, the status line disappears altogether. --weinzwang + +*Update:* I'm nitpicking here, but the status line is a bit misleading. When it says "forgetting generation 0/1" it's actually forgetting generation 1/1. Maybe something like "forgetting generations: 0/1 done" would be better. --weinzwang + + +--- + +I confirm that I can see this too. --liw + +--- + +This should be fixed in [[ttystatus]] 0.17, just released and uploaded. --liw +[[done]] diff --git a/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn b/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn new file mode 100644 index 0000000..5341e76 --- /dev/null +++ b/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn @@ -0,0 +1,18 @@ +While on the commandline, you can enter multiple roots to be backed up, like: + obnam --config=/etc/obnam.cnf backup sftp://user@host/etc sftp://user@host/home + +The same is not possible when setting the root in a configfile. Setting multiple root= entries simply overrides the previous ones and only picks up the last one; setting them all on one line, space-delimited, like on commandline gets interpreted as one large filename with spaces. + +--- + +This turns out to be a problem in the config file: you need to set +all the roots in one root= line, instead of having one root= per +root. + + [config] + root = /home/you, /home/them + +That'll work. I've added something to the manual page to explain that +better. Sorry about the confusion. --liw + +[[done]] diff --git a/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn b/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn new file mode 100644 index 0000000..2e491dd --- /dev/null +++ b/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn @@ -0,0 +1,9 @@ +Just started with obnam. While backing up files via sftp works fine, when I'm trying to backup local files I'm getting only errors similar to the one in the post title. + +--- + +There's not nearly enough information there to investigate the problem. Please see +the <http://liw.fi/obnam/bug-reporting/> page and then send the relevant information to the +Obnam mailing list (see <http://liw.fi/obnam/contact/> for details). Thanks. + +[[done]] diff --git a/bugs/Excessive_Memory_usage.mdwn b/bugs/Excessive_Memory_usage.mdwn new file mode 100644 index 0000000..2a278c6 --- /dev/null +++ b/bugs/Excessive_Memory_usage.mdwn @@ -0,0 +1,45 @@ +I'm running obnam (1st run) on a decent sized (2.8T) volume. It's been running for days and beyond being slow, it's using a good bit of memory. This does not seem reasonable. Is there something I should tweak? + +Thanks + +(unloaded switched gigE between server/client) + +<pre> +PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND +20848 root 20 0 1745m 1.4g 1620 R 70 68.9 8558:22 obnam + +# more .obnam.conf +[config] +repository: sftp://xxx/yyy/obnam/ +log-level: warning +log: /root/obnam.log +log-max: 10M +weak-random: true +lock-timeout: 10 +compress-with: deflate +</pre> + + +--- + + +I have not benchmarked Obnam's memory usage for a while, so I don't know if that is unexpectedly large. +It's certainly larger than I would like, though. You can try different sizes of the following settings: + +* `node-size` +* `upload-queue-size` +* `lru-size` + +If you change the node size, you'll need to remove the repository and start over, since it only affects new +instances of the in-repository data structures, so it's probably best to start by trying different values for the +other two settings. + +--liw + + +-- + +I've reduced the default values for lru-size and upload-queue-size. Obnam should now be +using a lot less memory. --liw + +[[done]] diff --git a/bugs/Flexibilize_path_handling_when_doing_backups.mdwn b/bugs/Flexibilize_path_handling_when_doing_backups.mdwn new file mode 100644 index 0000000..678f8ee --- /dev/null +++ b/bugs/Flexibilize_path_handling_when_doing_backups.mdwn @@ -0,0 +1,36 @@ +[[!tag obnam-wishlist]] + +First of all, I realise that Obnam stores full paths because it is necessary for saving every file in the system, even when belonging to different users. + +However, for certain cases where just a backup of a directory is needed, this could be flexibilized, letting the backup store only the path that is given in the command line, following rsync's spirit. + +When does this show up? For example, when migrating from any other backup system, the easier way would be to dump all the generations from the older backup system, one by one, to a temporal place. For each generation, Obnam is run in order to replicate the same history. However, since Obnam stores full paths, the path to the temporary directory used for the migration is also stored. This can happen in production servers, where making the conversion into the original directory where the data belongs to is not possible. + +Rickard Nilsson suggested on the mailing list to have a "root" option that could be used for stripping the first part of the path. The stripped part would be the one not mentioned in the command line. That way, the backup will have a path computed like this: root + given_path. + +~$ obnam backup --repository=/media/backups/... --root=/ mydata + +...would be stored into /mydata instead of in /home/${USER}/mydata + +Thank you for taking this into consideration! + + +--- + +If this gets implemented, I suggest the following: + +* The `Repository` class will provide a hook for mangling the pathnames. +* The hook will get the pathname as it exists in live data and will return the pathname to store in the backup. +* The hook will be called at every point where live data pathnames are used by `Repository`. +* Someone writes a plugin that adds the suitable functionality. + +--liw + +--- +For the fun of it I added a mangle_filanem() method to Repository. What I quickly learned: If you backup /root/bar the process backs up "/", "/root", "/foo/bar". In reality you only want "bar" in the backup. + +So either the mangling hook is allowed to drop paths entirely. But this feels very crude. + +I propose not to change Repository and think of Repository just getting virtual paths from its callers. So instead the functions calling into Repository should be changed. In this case the backup command. I have stopped here. + +-- Elrond diff --git a/bugs/FreeBSD_support.mdwn b/bugs/FreeBSD_support.mdwn new file mode 100644 index 0000000..801eb2b --- /dev/null +++ b/bugs/FreeBSD_support.mdwn @@ -0,0 +1,41 @@ +## gid.diff +## no-llistxattr.diff +## o_noatime.diff +## symlink.diff +## utimensat.diff + +How I am supposed to attach a patch here? + +--- + +You need to click on the word "Attachments" below the editing area. --liw + +-- + +I can't: +prohibited by allowed_attachments (user is not an admin) + +I sent you the patches by mail, sunday. + +Best regards + + +---- + +Oh, right, I haven't enabled everyone to add attachments. Sorry about that. + +I've responded by e-mail to the patches. Summary: I'm happy to accept patches for portability, but since I don't run FreeBSD (or the kFreeBSD kernel on Debian), I want the patches to apply cleanly and be otherwise acceptable: if I need to hack on the patches to make them acceptable to me, I don't know if they'll work for FreeBSD anymore. These patches achieved portability by disabling tests and features on Linux, which is not acceptable, so now I'm waiting for updated patches. + +Meanwhile, since this is best handled via e-mail (since patches can't be attached here), I'll mark this +bug as [[done]]. + +---- + +Could these patches be made public? I'd like to get obnam working on my FreeBSD server doing its backups and minimizing duplication of work would be ideal. Thanks. -- mathstuf + +--- + +I'm afraid the patches were not particularly useful as a basis for further work. Most of them +concentrated on disabling features or functionality for everyone, which is not how porting +should happen. You'd be better off grabbing the current version of Obnam and its +dependencies and running "./check" in the Obnam source tree. --liw diff --git a/bugs/Ignores_invalid_command.mdwn b/bugs/Ignores_invalid_command.mdwn new file mode 100644 index 0000000..5778f85 --- /dev/null +++ b/bugs/Ignores_invalid_command.mdwn @@ -0,0 +1,8 @@ +I accidentally ran "obnam list-clients" instead of "obnam clients", and it didn't give an error, it just returned. + +--- + +I confirm this. This turns out to be an error in the [[cliapp]] framework, and is now fixed there, and the +fix will be included in the next release. --liw + +[[done]] diff --git a/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn b/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn new file mode 100644 index 0000000..b35239b --- /dev/null +++ b/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn @@ -0,0 +1,22 @@ +[[!tag obnam-wishlist]] + +I'm using trickle to restrict obnam to 40 KB/sec. I just resumed a backup +a few minutes ago, and obnam currently says the rate is 145 KB/sec, and +that it's transferred 111 MB. I'm guessing it's counting the entire size +of resumed files instead of the actual bytes transferred since resuming. +Whatever the cause, it's definitely off. I've not noticed it being off +when not resuming. + +--- + +So, there were at least a couple of problems here: + +1. Obnam was counting duplicate chunks as uploaded: when obnam found a + chunk was a duplicate of an existing one, it didn't upload it, but + did count it as uploaded. This has now been fixed. +1. Obnam was showing the average upload speed over the entire backup + run. It now shows the average for the last ten seconds. + +Thus, I think this bug is fixed now. [[done]] (If not, please tell me!) + +--liw diff --git a/bugs/Local_cache_of_remote_repo_metadata.mdwn b/bugs/Local_cache_of_remote_repo_metadata.mdwn new file mode 100644 index 0000000..e7d6e9e --- /dev/null +++ b/bugs/Local_cache_of_remote_repo_metadata.mdwn @@ -0,0 +1,12 @@ +[[!tag obnam-wishlist]] + +Could Obnam keep a local cache of remote repository metadata, like Duplicity does? I was looking through an Obnam log and noticed how many remote file accesses it does for each file it backs up. If it cached that data locally, it could do it much faster. The last-modified time of the repository could be stored in the remote repo, and if it matches the local cache, a lot of time could be saved. + +--- + +That is indeed a good idea. Unfortunately, the correctness of caches is often tricky, so I've been putting off implementing this until more important things work first. Also, not caching metadata forces me to do other things to make Obnam fast. But I'd like to do the caching too, some day. --liw + +--- + +[[done]] duplicate, and also an old wishlist bug. Keeping this open +isn't making this happen. --liw diff --git a/bugs/Log_performance_results_to_logfile.mdwn b/bugs/Log_performance_results_to_logfile.mdwn new file mode 100644 index 0000000..7bbc178 --- /dev/null +++ b/bugs/Log_performance_results_to_logfile.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +This is a wishlist item. + +If obnam is run with the --quiet option, then the performance details (time taken, speed etc) is lost. It'd be nice if that was logged to the log file. I'd suggest at a NOTICE level. + +-- + +thank you for the excellent suggestion. I've now implemented it in bzr, +so this'll show up in the next Obnam release. [[done]] --liw diff --git a/bugs/Missing___34__e__34___in_commandline_output.mdwn b/bugs/Missing___34__e__34___in_commandline_output.mdwn new file mode 100644 index 0000000..f0dc41b --- /dev/null +++ b/bugs/Missing___34__e__34___in_commandline_output.mdwn @@ -0,0 +1,5 @@ +During a backup, obnam regularly informs me that it's "making chckpoint". I think adding an "e" would probably solve this :) + +--weinzwang + +[[done]] diff --git a/bugs/Mutiple_bugs..mdwn b/bugs/Mutiple_bugs..mdwn new file mode 100644 index 0000000..2058135 --- /dev/null +++ b/bugs/Mutiple_bugs..mdwn @@ -0,0 +1,94 @@ +Obnam version: 1.0 + +Environment: backing up from multiple servers (each has its' own repo) to an NFS-shared storage. + +OS: OpenSuSE 11.3 + +--- + +0) From time to time I get strange locks, which prevents subsequent backups with lock timeout message, though, I am SURE that I'm running only one backup per-client at a time, and each client has dedicated obnam repo. How those locks happen to appear?... noone knows, but they do, for this or that obnam repo (I'm backing up 17 servers with it) + +--- + +1) When trying to fsck with fix: + + /it/bin/obnam/bin/obnam -r /data/backup/mlogin4.smware.local/obnam --fsck-fix fsck + Traceback (most recent call last): + File "/it/bin/obnam/lib/python/site-packages/cliapp/app.py", line 172, in _run + self.process_args(args) + File "/it/bin/obnam/lib/python/site-packages/obnamlib/app.py", line 158, in process_args + cliapp.Application.process_args(self, args) + File "/it/bin/obnam/lib/python/site-packages/cliapp/app.py", line 407, in process_args + method(args[1:]) + File "/it/bin/obnam/lib/python/site-packages/obnamlib/plugins/fsck_plugin.py", line 306, in fsck + for more in reversed(list(work.do() or [])): + File "/it/bin/obnam/lib/python/site-packages/obnamlib/plugins/fsck_plugin.py", line 250, in do + work.do() + File "/it/bin/obnam/lib/python/site-packages/larch/fsck.py", line 107, in do + tracing.trace('fixed it: %s' % new_node.keys()) + UnboundLocalError: local variable 'new_node' referenced before assignment + +That is strange ) + + +--- + +It would be much more helpful if you filed separate bugs as separate bugs. + + +--- +They are related. These two bugs deal with repo inconsistent state, which obnam leaves behind when something "wrong" happends with it. +On of the errors right before stalled locks are left looks like this (while backing up /local_disk/vpodrezov/ directory): + + <...> + + file_id = self.get_file_id(self.tree, filename) + File "/it/bin/obnam/lib/python/site-packages/obnamlib/clientmetadatatree.py", line 180, in get_file_id + raise KeyError('%s does not yet have a file-id' % pathname) + KeyError: '/local_disk/vpodrezov/save/0019_29052012/os/linux-2.6.36/arch/x86/boot/.svn/tmp/text-base does not yet have a file-id' + +And then if we try to backup to the repo, it ends with Lock timeout. + +------------ + + +Another error seen is like + + ERROR: Node 534798 cannot be found in the node store 3662754038122990765 + +And then the same, Lock timeout. No way to correct this (fsck --fsck-fix just doesnt' work). + +--- + +I'm confused: are you trying to fix lock timeouts with fsck instead of force-lock? + +Also, that's a nice, long traceback, but I think it would be more helpful if you made a debug log and posted that. + +--- + +What I want to say is that while backing up live filesystems (when files are being changed), from time to time obnam leaves its' repos in such a state, that no further use for backup is possble (errors, resulting in further locks, which are NOT ALWAYS may be bypassed with force lock (I see multiple lock files around the obnam repo tree, which force-lock doesn't find or know about)) + +Btw, one of the errors after which I get lock timeouts is (/net/backup/data/backup/mlogin1.smware.local/obnam/ is obnam's repo): + + ERROR: /net/backup/data/backup/mlogin1.smware.local/obnam/3662754038122990765/new/nodes/65/0/0: File exists + +// + +Sure, I'll go on then on the mailing list with debug log if it helps, but there are deffinitely problems with backing up live filesystems =(. + + +---- + +From discussion on mailing list: + +The unbound variable error has been fixed. + +The other errors seem to be caused by something in the repository having become corrupt, and +it is now not really possible to get Obnam working properly with it anymore. Ouch. And sorry. It's +no longer possible to see what went wrong either, it seems. It may have been some other error, +or it may be a problem in Obnam. Since this bug no longer seems actionable, I'm marking +it closed, but that doesn't mean eveyrthing's fine. I've made a note to my TODO that I need to +change Obnam to be more easy to debug about these problems in the future (no idea how to +achieve that yet, but we'll see). --liw + +[[done]] diff --git a/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn b/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn new file mode 100644 index 0000000..f51a84d --- /dev/null +++ b/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn @@ -0,0 +1,136 @@ +Just read about obnam on lwn and decided to try it! +Installed the version on the ubuntu ppa. +Unfortunately, I cannot execute the basic backup example: + + obnam backup --repository Backup/Obnam-backup Documents/BibFiles + Traceback (most recent call last): + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run + self.process_args(args) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 127, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args + method(args[1:]) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup + self.add_client(client_name) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 140, in add_client + if client_name not in self.repo.list_clients(): + File "/usr/lib/python2.7/dist-packages/obnamlib/repo.py", line 201, in list_clients + listed = set(self.clientlist.list_clients()) + File "/usr/lib/python2.7/dist-packages/obnamlib/clientlist.py", line 78, in list_clients + if self.init_forest() and self.forest.trees: + File "/usr/lib/python2.7/dist-packages/obnamlib/repo_tree.py", line 63, in init_forest + vfs=self.fs) + File "/usr/lib/python2.7/dist-packages/larch/forest.py", line 167, in open_forest + assert allow_writes is not None + AssertionError + +Here is the log: + + 2012-06-06 12:59:50 INFO obnam version 0.24.1 starts + 2012-06-06 12:59:51 DEBUG Current configuration: + [config] + output = + log = obnam.log + log-level = debug + log-max = 0 + log-keep = 10 + log-mode = 0600 + dump-memory-profile = simple + repository = /home/callegar/Backup/Obnam-backup + client-name = mandrago + node-size = 262144 + chunk-size = 1048576 + upload-queue-size = 1024 + lru-size = 500 + trace = obnamlib + idpath-depth = 3 + idpath-bits = 12 + idpath-skip = 13 + quiet = False + pretend = False + compress-with = none + encrypt-with = + keyid = + weak-random = False + symmetric-key-bits = + root = + exclude = + exclude-caches = False + one-file-system = False + checkpoint = 1073741824 + chunkids-per-group = 1024 + deduplicate = fatalist + sftp-delay = 0 + to = + generation = latest + keep = + fsck-fix = False + verify-randomly = 0 + + + 2012-06-06 12:59:51 INFO Obnam 0.24.1 starts + 2012-06-06 12:59:51 INFO Backup starts + 2012-06-06 12:59:51 INFO Checkpoints every 1073741824 bytes + 2012-06-06 12:59:51 DEBUG Exclude pattern: obnam.log + 2012-06-06 12:59:51 DEBUG opening repository (create=True) + 2012-06-06 12:59:51 DEBUG vfs_local.py:55:__init__: baseurl=/home/callegar /Backup/Obnam-backup + 2012-06-06 12:59:51 DEBUG vfs_local.py:56:__init__: create=True + 2012-06-06 12:59:51 INFO VFS: __init__: baseurl=/home/callegar/Backup/Obnam-backup + 2012-06-06 12:59:51 DEBUG vfs_local.py:64:reinit: baseurl=/home/callegar/Backup/Obnam-backup + 2012-06-06 12:59:51 DEBUG vfs_local.py:65:reinit: create=True + 2012-06-06 12:59:51 DEBUG clientlist.py:49:__init__: new ClientList + 2012-06-06 12:59:51 DEBUG chunklist.py:37:__init__: new ChunkList + 2012-06-06 12:59:51 DEBUG checksumtree.py:34:__init__: new ChecksumTree name=chunksums + 2012-06-06 12:59:51 DEBUG vfs_local.py:192:open: pathname=/home/callegar/Backup/Obnam-backup/metadata/format + 2012-06-06 12:59:51 DEBUG vfs_local.py:193:open: mode=rb + 2012-06-06 12:59:51 DEBUG repo_tree.py:53:init_forest: initializing forest dirname=clientlist + 2012-06-06 12:59:51 CRITICAL Traceback (most recent call last): + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run + self.process_args(args) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 127, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args + method(args[1:]) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup + self.add_client(client_name) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 140, in add_client + if client_name not in self.repo.list_clients(): + File "/usr/lib/python2.7/dist-packages/obnamlib/repo.py", line 201, in list_clients + listed = set(self.clientlist.list_clients()) + File "/usr/lib/python2.7/dist-packages/obnamlib/clientlist.py", line 78, in list_clients + if self.init_forest() and self.forest.trees: + File "/usr/lib/python2.7/dist-packages/obnamlib/repo_tree.py", line 63, in init_forest + vfs=self.fs) + File "/usr/lib/python2.7/dist-packages/larch/forest.py", line 167, in open_forest + assert allow_writes is not None + AssertionError + +---- + +The Python stack trace is awful for the user, but it's there as a development aid. If a user sees it, something is badly wrong in Obnam. + +Are you sure you have the right version of Obnam? How do you check that? Obnam is supposed to report the right version number in the log file or with "obnam --version", and if it isn't 1.0 then something's wrong. + +Do you have the right version of the Larch Python library installed? That stack trace looks like there's an old version installed. Are you running on Debian? Ubuntu? Somewhere else? The Ubuntu PPA is in the process of being updated to fix a few problems with wrong versions. If you're on either Ubuntu or Debian, what is the output of "dpkg -l obnam python-larch"? + +--liw + +---- + +Currently Chris's ppa (the one you're referring to in "download" page) seems broken. On ppa's page, it reports build fail for obnam package, in particular. Using that ppa and installing obnam, installs ubuntu precise packaged obnam: + + # LANGUAGE=C apt-cache policy obnam + obnam: + Installed: 0.24.1-1 + Candidate: 0.24.1-1 + Version table: + *** 0.24.1-1 0 + 500 http://it.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages + 100 /var/lib/dpkg/status +--negro + +--- + +The PPA seems to be fixed now. Can you check whether it now works for you? Thanks. --liw + +Haven't heard back, but assuming this is OK. [[done]] --liw diff --git a/bugs/Operation_not_permitted.mdwn b/bugs/Operation_not_permitted.mdwn new file mode 100644 index 0000000..20874d2 --- /dev/null +++ b/bugs/Operation_not_permitted.mdwn @@ -0,0 +1,54 @@ +I try to backup my mail directory to a USB disc with + + % obnam --log=obnam.log --repository=/media/HD-PCU2/ backup ~/Mail + 00h00m00s 1 files; 0 B (0 B/s) setting upERROR: Operation not permitted + +The USB disc is mounted with the following options + + /dev/sdb1 on /media/HD-PCU2 type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000, shortname=mixed,dmask=0077,utf8=1,showexec,flush) + +The obnam.log contains + + 2012-08-16 14:18:19 INFO Backup starts + 2012-08-16 14:18:19 INFO Checkpoints every 1073741824 bytes + 2012-08-16 14:18:19 DEBUG Exclude pattern: obnam.log + 2012-08-16 14:18:19 DEBUG opening repository (create=True) + 2012-08-16 14:18:19 INFO VFS: __init__: baseurl=/media/HD-PCU2/ + 2012-08-16 14:18:19 CRITICAL Traceback (most recent call last): + File "/home/friedrich/local/lib/python2.7/site-packages/cliapp/app.py", line 169, in _run + self.process_args(args) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/app.py", line 174, in process_args + cliapp.Application.process_args(self, args) + File "/home/friedrich/local/lib/python2.7/site-packages/cliapp/app.py", line 418, in process_args + method(args[1:]) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/plugins/backup_plugin.py", line 253, in + self.add_client(client_name) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/plugins/backup_plugin.py", line 311, in + self.repo.lock_root() + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/repo.py", line 266, in lock_root + self.lockmgr.lock(['.']) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/lockmgr.py", line 81, in lock + self._lock_one(dirname) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/lockmgr.py", line 65, in _lock_one + self._fs.lock(lock_name, self.data) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/repo.py", line 70, in lock + self.fs.lock(filename, data) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/vfs_local.py", line 106, in lock + self.write_file(lockname, data) + File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/vfs_local.py", line 274, in write_file + os.link(tempname, path) + OSError: [Errno 1] Operation not permitted + +Do you have an idea what's the problem with the USB disc? A backup into my home directry works fine. + + +--- + + +Obnam currently does not work with FAT filesystems, since it needs the +filesystem to support hardlinks. --liw + +--- + +This is now fixed in bzr. --liw +[[done]] diff --git a/bugs/Other_compression_methods__63__.mdwn b/bugs/Other_compression_methods__63__.mdwn new file mode 100644 index 0000000..f4f1732 --- /dev/null +++ b/bugs/Other_compression_methods__63__.mdwn @@ -0,0 +1,16 @@ +[[!tag obnam-wishlist]] + +Would Obnam benefit from being able to use other compression tools, like bzip2, p7zip, and xz? + +--- + +It would! However, implementing this is slightly tricky until I make Obnam can run external tools +in the background, rather than waiting for them to finish. The compression it uses right now is +done with the Python standard library `gzip` module, so it doesn't require executing an +external program, so the issue doesn't rise. However, we need the backgrounding feature +anyway, since encryption uses an external tool, and so there's a big performance impact +from using encryption. After backgrounding works, adding arbitrary compression filters will +be easy. --liw + +[[done]] this is an ancient wishlist bug, and keeping it open isn't helping it happen, +but it is making it harder to use the bug list. --liw diff --git a/bugs/Performance_impact_of_backup_generations__63__.mdwn b/bugs/Performance_impact_of_backup_generations__63__.mdwn new file mode 100644 index 0000000..dda828c --- /dev/null +++ b/bugs/Performance_impact_of_backup_generations__63__.mdwn @@ -0,0 +1,55 @@ +[[!tag obnam-1.0-blocker]] + +I'm not sure what to do with this or if it even qualifies as a bug but here we go... + +I back up my notebook using obnam for a while now so I have about 10 clean generations, the repo is 118G big. One of the files in the backup is a 32G VM file that changes slightly between backups, but not during backups. Until a few days ago this worked really well and fast. + +Then I tried to do a partial backup via a very slow line, with snapshots every 5MB. This was interrupted (ctrl+c) after a few hours. + +The next day I continued the backup using the local network. I forgot to change the snapshot interval, so it was still 5MB. I expected the backup to finish within a few minutes as it did in the past, but after crawling through the first couple of GB of the VM file at 50MB/s the speed went down to below 1MB/s, doing a huge amount of IO on the local machine. I haven't manged to complete a backup since. + +Nearly everything I try to do with the repo now takes a lot of time. Listing the huge amount of generations takes over an hour (it lists at a rate of about 1 generation every 10 seconds or so). Listing only the genids is still very quick. + +I'm not sure what goes wrong here. Is the massive amount of generations killing the performance? + +-- weinzwang + +--- + +The number of generations shouldn't impact the speed of backups. It does have an impact on +the speed of removing generations, so if you backup stalls at the end when it removes checkpoints, +then you can avoid that with the `--leave-checkpoints` option. + +Could you check logs to see where time is spent? Or even run with profiling enabled when you +list generations? `OBNAM_PROFILE=obnam.prof obnam generations` should do that. Then use +a Python script like the following to get it in cleartext: + + import pstats + import sys + + if len(sys.argv) not in [2, 3]: + sys.stderr.write('Usage: viewprof foo.prof [sort-order]\n') + sys.exit(1) + + if len(sys.argv) == 3: + order = sys.argv[2] + else: + order = 'cumulative' + + p = pstats.Stats(sys.argv[1]) + p.strip_dirs() + p.sort_stats(order) + p.print_stats() + p.print_callees() + +Then mail me the output, and I'll have a look. Thanks. --liw + +--- + +Since this problematic repository doesn't work with recent versions of obnam due to +on-disk format changes, I can't reproduce this anymore. I haven't encountered this effect +with any other repositories I have, so I think the bug can be closed. --weinzwang + +-- + +Ack, closing bug. [[done]] --liw diff --git a/bugs/SSH_connections_left_open.mdwn b/bugs/SSH_connections_left_open.mdwn new file mode 100644 index 0000000..d1c829e --- /dev/null +++ b/bugs/SSH_connections_left_open.mdwn @@ -0,0 +1,19 @@ +[[!tag obnam-1.0-blocker]] + +Obnam 0.26 opens 2 ssh connections at the beginning of a backup and another one for each snapshot generation it creates. All of these connections stay open until obnam terminates. For a long-running backup session that generates a lot of snapshot generations this might become a problem since for every ssh connection there are 3 processes running on the server. + +This behaviour seems to have been introduced with obnam 0.26. The previous version opened one ssh connection at the beginning of a backup and used it for everything. (Side note: This made it feasible to use password authentication, which is not possible anymore.) --weinzwang + + +--- + +I think this is related to the ssh locking changes in 0.26, and that the current bzr trunk has fixes for it. +If you could verify and report, that would be excellent. You'll need the new larch 0.19 version, though. --liw + +--- + +The problem is still the same with obnam 0.27. Every snapshot triggers a ssh authentication and the sshd processes pile up on the server. When obnam finishes, all connections are closed. --weinzwang + +-- + +Found and fixed in bzr now. [[done]] --liw diff --git a/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn b/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn new file mode 100644 index 0000000..1353ba7 --- /dev/null +++ b/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn @@ -0,0 +1,6 @@ +I've already configured ~/.ssh/config to have a "backup" alias, with a corresponding SSH key to use, port number, and various other configuration. I'd like to just tell obnam to use sftp://backup/... and have obnam use that alias and configuration, but obnam doesn't seem to want to do that: at a minimum, it appears to ignore the port number I specified in ~/.ssh/config, so I can't easily tell what else it ignored. + +--- + +For the same reasons as discussed in +[[the other bug|option_to_not_use_paramiko__63__]], I'll mark this as [[done]]. --liw diff --git a/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn b/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn new file mode 100644 index 0000000..2a62a70 --- /dev/null +++ b/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn @@ -0,0 +1,55 @@ +[[!tag obnam-wishlist]] + +Normally, GPG can encrypt to a key using only the public key, without needing the private key available. I'd like to have the same model for obnam encrypted backups: perform a backup to a key using only the public key, without having the private key available. That way, I could perform backups of various systems using public keys whose corresponding private keys I keep entirely offline and secured. + +--- + +Unfortunately, Obnam requires the secret in order to do backups. It needs to download, and decrypt, +some metadata from the backup repository in order to add new backups there. See the [[obnam/ondisk]] +and [[obnam/encryption]] pages for details. --liw + +--- + +I've read those pages, but as far as I can tell, the ability to decrypt part of the repository doesn't seem like an inherent requirement to meet obnam's goals, just an implementation detail. And as far as I can tell, without some change in this area, obnam does not support letting a client perform backups without giving that client full access to previous backups. + +Use case: I want to back up to a storage system that I trust for availability but not for privacy; thus, I need encryption, and I can't do "pull" backups since I don't trust the storage server with access to the system that needs backing up. However, I also want the backups to protect me from security breaches, by giving me the ability to follow the standard advice for how to deal with a system break-in: nuke everything and restore from the last-known good backup. How can I have a "last-known good backup" if the backed-up system can access the old backups? + +--- + +I understand the desire to do what you want to do, but I haven't found a technical solution to do it. If you can +come up with one, then write it up and send me a pointer. --liw + +--- + +It'd be nice if Obnam could do this, but in the meantime, I think Duplicity can work this way. --AP + +--- + +If Obnam had a local metadata cache, could this be implemented? If Obnam had the metadata locally, could it do a quick verification that the metadata is the same as that on the remote repository, and then backup to the remote repository "blindly", using only the local cache as a guide? --AP + +--- + +A concrete existence proof: tarsnap does this. You can do tarsnap backups with only a restricted public key, and that key need not also provide access to decrypt old backups. -- Josh + +--- + +I've re-opened this bug since there's ongoing discussion. I'll have a think and respond later. --liw + +--- + +Having thought about this... + +* Obnam will probably eventually get a local cache for stuff from the repository. + However, this is not enough: we can't assume the cache is complete, since + other clients will be making changes to the repository as well. +* Obnam will thus need to read, and decrypt, files from the repository. +* Having access to the PGP private key is thus necessary for Obnam. + +I haven't looked at how duplicity and tarsnap do things. If someone comes up +with a design for Obnam that can do this, without sacrificing things such as +de-duplication across clients, please e-mail the Obnam mailing list with +suggestions. Thanks! + +--liw + +[[done]] diff --git a/bugs/Unhelpful_error_for_missing_key.mdwn b/bugs/Unhelpful_error_for_missing_key.mdwn new file mode 100644 index 0000000..9845930 --- /dev/null +++ b/bugs/Unhelpful_error_for_missing_key.mdwn @@ -0,0 +1,24 @@ +[[!tag obnam-wishlist]] + +I decided to try running fsck on a remote repo from a user account that doesn't have the GPG key used to encrypt with. I got this error, which isn't helpful. I think, at least, it should mention the possibility that there is a missing key. + +<pre>ERROR: Unknown filter tag encountered: '\x8c\r\x04\x03\x03\x02n\x7fX8\xa5\xd7\xf1\xbd`\xc9[\xba\xc0\x11kU\xd0\x8f\xf3\r\x97J@\x0c\xc9\x11\xaan\xefxx\x89\xe9\x9b\xac\x96\xce \x1cD{:\x02\x10I\xda\xbb;\x0e\xac\xd3d9\x18(\x0c\xe4\x8a\xf89\xed\x9a!\x85\xd0t\xd0\xaa^0\xee\x17{\xe3\x9d\xa3\x89O\xc6\x04$c\x96\xee\xf3\xbb{.|\xd5\xb0\x96Z\xb3\x11\x0f&B\xf0'</pre> + +--- + +Someone else also reported this: + +Minor improvement: using Obnam 1.0 on Debian Unstable, the error message when using an encrypted repository without specifying any key is not helpful. Please consider using a human-readable error message to warn the user that the key is missing (or wrong). + +Here is an example of error message when using "obnam fsck" without the key: + + Traceback (most recent call last): + [lots of debug info] + File "/usr/lib/python2.7/dist-packages/obnamlib/hooks.py", line 122, in run_filter_read + tag, content = data.split("\0", 1) + ValueError: need more than 1 value to unpack + + +--- + +Fixed in git now. [[done]] --liw diff --git a/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn b/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn new file mode 100644 index 0000000..d2791ec --- /dev/null +++ b/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn @@ -0,0 +1,29 @@ +hello, + +My backup fails with the following message at the end of Python stack trace : "ValueError: need more than 1 value to unpack" + +Steps to reach the error state : + +* did a first succesful backup of a small part of the backup source +* started a complete backup of the source +* the backup ended because of network shutdown +* restarted the backup, but could not proceed because of the previous lock remaining settled +* did obnam force-lock +* restarted backup, got this error + +overview of my configuration: + +* backup over sftp +* gpg encryption + +I will send you the log files in a separate email. + +thanks! +Damien + + +--- + +This is fixed in version 1.1. --liw + +[[done]] diff --git a/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn b/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn new file mode 100644 index 0000000..6e9d7a6 --- /dev/null +++ b/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn @@ -0,0 +1,22 @@ +[[!tag obnam-wishlist]] + +One of the common (?) use cases of obnam is to backup laptops and workstations. Sometimes we grab ISOs etc and have them lying around, but they shouldn't be backed up. + +It'd be handy if obnam had a couple of extra options. One to warn about large files, the other to skip large files. They are not mutually exclusive. + +This would have saved me backing up a 700MB ISO. Ouch. + +---- +Great idea. --AP + +----------- +I second that great idea :) leto + + +--- + +[[done]] + +While I agree it is a nice idea, keeping the wishlist bug open clearly +isn't making this happen, and long lists of bugs are tedious to work +with. --liw diff --git a/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn b/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn new file mode 100644 index 0000000..32256d7 --- /dev/null +++ b/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn @@ -0,0 +1,599 @@ +Hi, + +I'm using obnam 1.0 and try to backup files / directories. obnam is installed on Debian Squeeze with the package frovided on http://liw.fi. No ~/.obnam.conf file is used. The obnam repository and the files to backup are stored on a partition that is mounted from an encrypted partition (cryptsetup, device-mapper). When trying to backup with + +cd /mnt/crypt/obnam + +obnam backup /mnt/crypt/myfiles/test.data + +obnam complains, that /mnt/crypt/myfiles/test.data is not available (No such file or directory). But the file is definitely available (less -f /mnt/crypt/myfiles/test.data shows data). When I comment out + +line 91: #raise OSError(err, os.strerror(err), self.cwd) + +in the file obnamlib/vfs_local.py, the backup works fine. I don't understand what the purpose of this line is (I'm not a python programmer), but for me it produces errors. Maybe you can find a solution ;-) + +--- + +Hi: there is at least one obvious bug here, which is that Obnam does not complain when you don't give an explicit backup repository (it uses the current working directory instead). I'll fix that. However, you seem to be working around that with the cd command. (Edit: --repository must now be set explicitly.) + +The line you commented out is essential. It makes sure that the repository exists. + +I'm not sure what the real error is that makes things not work for you. Could you e-mail me the log file, which has a lot extra information I could use for debugging. Thanks. + +--liw + +--- + +I am having this same problem. I had to wipe my obnam repo from March--32 GB, uploaded slowly :(--since the repo format changed. So before starting a whole, new, full backup, I decided to do a few little tests. I first backed up a small directory to a local directory and tried a few operations. Then I rsynced the repo to a remote server and ran fsck (obnam's, of course) on it. It came out fine. Listing clients and generations and keys and "ls" all worked fine. + +So then I tried to make a new backup of the same, small, local directory. It failed with: + + ERROR: sftp://me@host/~/obnam: No such file + +This was quite strange, since all the other ops worked. Finally I made a full log and here's what I see: + +<pre> +2012-06-21 03:05:51 INFO obnam version 1.0 starts +2012-06-21 03:05:51 DEBUG Current configuration: +[config] +output = +log = /tmp/obnam.log +log-level = debug +log-max = 0 +log-keep = 10 +log-mode = 0600 +dump-memory-profile = simple +repository = sftp://me@host/~/obnam +client-name = mysystem +node-size = 262144 +chunk-size = 1048576 +upload-queue-size = 1024 +lru-size = 500 +trace = +idpath-depth = 3 +idpath-bits = 12 +idpath-skip = 13 +quiet = False +pretend = False +pretend-time = +lock-timeout = 60 +crash-limit = 0 +compress-with = none +root = +exclude = +exclude-caches = False +one-file-system = False +checkpoint = 1073741824 +chunkids-per-group = 1024 +deduplicate = fatalist +leave-checkpoints = False +small-files-in-btree = False +warn-age = 27h +critical-age = 27h +sftp-delay = 0 +ssh-key = +strict-ssh-host-keys = False +ssh-known-hosts = /home/me/.ssh/known_hosts +pure-paramiko = False +to = +generation = latest +keep = +fsck-fix = False +verify-randomly = 0 +encrypt-with = mykeyid +keyid = +weak-random = False +symmetric-key-bits = + + +2012-06-21 03:05:51 INFO Backup starts +2012-06-21 03:05:51 INFO Checkpoints every 1073741824 bytes +2012-06-21 03:05:51 DEBUG Exclude pattern: /tmp/obnam.log +2012-06-21 03:05:51 DEBUG opening repository (create=True) +2012-06-21 03:05:51 INFO VFS: __init__: baseurl=sftp://me@host/~/obnam +2012-06-21 03:05:51 DEBUG executing openssh: ['ssh', '-oForwardX11=no', '-oForwardAgent=no', '-oClearAllForwardings=yes', '-oProtocol=2', '-p', '22', '-l', 'b122569', '-s', '-o', 'UserKnownHostsFile=/home/me/.ssh/known_hosts', 'host', 'sftp'] +2012-06-21 03:05:52 INFO [chan obnam SSHChannelAdapter] Opened sftp connection (server version 3) +2012-06-21 03:05:52 DEBUG [chan obnam SSHChannelAdapter] mkdir('obnam', 511) +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] stat('obnam') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] normalize('obnam') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/./lock', 'wx') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/./lock', 'wx') -> 00000000 +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata/format') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -> 00000000 +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/metadata', 511) +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', 'wx') +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', 'wx') -> 00000000 +2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/metadata/format') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', '/path/to/my/user/obnam/metadata/format') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') +2012-06-21 03:05:54 DEBUG Initializing Journal for clientlist +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') +2012-06-21 03:05:54 DEBUG Automatically rolling back remaining changes +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/') +2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/metadata') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/metadata', 'r') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/metadata', 'r') -> 00000000 +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/key', 'r') +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/key', 'r') -> 00000000 +2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5', 'r') +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5', 'r') -> 00000000 +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0', 'r') +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0', 'r') -> 00000000 +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata/format') +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') +2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -> 00000000 +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/2', 'r') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/2', 'r') -> 00000000 +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete') +2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes', 511) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0', 511) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0', 511) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0', 511) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', 'wx') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', 'wx') -> 00000000 +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', '/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/refcounts', 511) +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', 'wx') +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', 'wx') -> 00000000 +2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', '/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes', 511) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0', 511) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0', 511) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0', 511) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', 'wx') +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', 'wx') -> 00000000 +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', '/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') +2012-06-21 03:06:04 DEBUG LRUCache <larch.lru.LRUCache object at 0xb71a8e0c>: hits=0 misses=2 +2012-06-21 03:06:04 DEBUG LRUCache <larch.lru.LRUCache object at 0xb71a83ac>: hits=3 misses=2 +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new', 511) +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', 'wx') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', 'wx') -> 00000000 +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/metadata') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', '/path/to/my/user/obnam/clientlist/new/metadata') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/nodes/0/0/0/6') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6', '/path/to/my/user/obnam/clientlist/nodes/0/0/0/6') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0', '/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/metadata') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/metadata', '/path/to/my/user/obnam/clientlist/metadata') +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/./lock') +2012-06-21 03:06:09 DEBUG opening repository (create=False) +2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] stat('/path/to/my/user/obnam/obnam') +2012-06-21 03:06:09 CRITICAL Traceback (most recent call last): + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run + self.process_args(args) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 158, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args + method(args[1:]) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 226, in backup + self.add_client(client_name) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 293, in add_client + self.repo = self.app.open_repository(repofs=self.repo.fs.fs) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 187, in open_repository + repofs.reinit(repopath) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/sftp_plugin.py", line 58, in helper + raise OSError(e.errno, e.strerror or str(e), filename) +OSError: [Errno 2] No such file: 'sftp://me@host/~/obnam' +</pre> + +Near the end of the log, it tries to stat('/path/to/my/user/obnam/obnam'), which doesn't exist. Where did the extra "/obnam" come from? + +Also, from my cursory look at the log, it seems like the backup succeeded before it failed, yet when I run "generations" I don't see any new ones. Will I need to run "fsck" to remove orphaned nodes or files? + +Thanks for obnam! Looking forward to using it full-time. + +--AP + +--- + +AP, what was your command line? I wasn't able to reproduce this myself, so I need your help to do so. --liw + +--- + +That happened to me yesterday when setting up a my new offsite repository. +When obnam is asked to create a new repository (notice the "opening repository (create=True)" in the log above) but the repository directory already exists, then it fails with the file not found error. +I rmdir'd my empty directory on the server (and I also replaced the "~" with a fully expanded path, not sure if that's related to the error) and it started working. + +--nemoinis + +--- + +Ok, let me try this again. Hopefully this will make it clear. :) --AP + +<pre> +$ sftp me@server +Connected to server. +sftp> ls +Maildir logs obnam +sftp> ^D + +$ time obnam --log=/tmp/obnam.log --log-level=debug backup ~/Dropbox/Zim +ERROR: sftp://me@server/~/obnam/: No such file + +$ cat .obnam.conf +[config] +repository = sftp://me@server/~/obnam/ +encrypt-with = MYKEY + +$ cat /tmp/obnam.log +2012-07-07 13:47:40 INFO obnam version 1.0 starts +2012-07-07 13:47:40 DEBUG Current configuration: +[config] +output = +log = /tmp/obnam.log +log-level = debug +log-max = 0 +log-keep = 10 +log-mode = 0600 +dump-memory-profile = simple +repository = sftp://me@server/~/obnam/ +client-name = laptop +node-size = 262144 +chunk-size = 1048576 +upload-queue-size = 1024 +lru-size = 500 +trace = +idpath-depth = 3 +idpath-bits = 12 +idpath-skip = 13 +quiet = False +pretend = False +pretend-time = +lock-timeout = 60 +crash-limit = 0 +compress-with = none +root = +exclude = +exclude-caches = False +one-file-system = False +checkpoint = 1073741824 +chunkids-per-group = 1024 +deduplicate = fatalist +leave-checkpoints = False +small-files-in-btree = False +warn-age = 27h +critical-age = 27h +sftp-delay = 0 +ssh-key = +strict-ssh-host-keys = False +ssh-known-hosts = /home/me/.ssh/known_hosts +pure-paramiko = False +to = +generation = latest +keep = +fsck-fix = False +verify-randomly = 0 +encrypt-with = MYKEY +keyid = +weak-random = False +symmetric-key-bits = + + +2012-07-07 13:47:40 INFO Backup starts +2012-07-07 13:47:40 INFO Checkpoints every 1073741824 bytes +2012-07-07 13:47:40 DEBUG Exclude pattern: /tmp/obnam.log +2012-07-07 13:47:40 DEBUG opening repository (create=True) +2012-07-07 13:47:40 INFO VFS: __init__: baseurl=sftp://me@server/~/obnam/ +2012-07-07 13:47:40 DEBUG executing openssh: ['ssh', '-oForwardX11=no', '-oForwardAgent=no', '-oClearAllForwardings=yes', '-oProtocol=2', '-p', '22', '-l', 'me', '-s', '-o', 'UserKnownHostsFile=/home/me/.ssh/known_hosts', 'server', 'sftp'] +2012-07-07 13:47:42 INFO [chan obnam SSHChannelAdapter] Opened sftp connection (server version 3) +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] mkdir('obnam/', 511) +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] stat('obnam/') +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] normalize('obnam/') +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/./lock', 'wx') +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/./lock', 'wx') -> 00000000 +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata/format') +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') +2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -> 00000000 +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata') +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/metadata', 511) +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', 'wx') +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', 'wx') -> 00000000 +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/metadata/format') +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', '/path/to/me/obnam/metadata/format') +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') +2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist') +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') +2012-07-07 13:47:44 DEBUG Initializing Journal for clientlist +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') +2012-07-07 13:47:44 DEBUG Automatically rolling back remaining changes +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/') +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/') +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/metadata') +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/metadata', 'r') +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/metadata', 'r') -> 00000000 +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/key', 'r') +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/key', 'r') -> 00000000 +2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/6', 'r') +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/6', 'r') -> 00000000 +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts/refcounts-0') +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/refcounts/refcounts-0', 'r') +2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/refcounts/refcounts-0', 'r') -> 00000000 +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata/format') +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -> 00000000 +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/2', 'r') +2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/2', 'r') -> 00000000 +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0/6') +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete') +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes', 511) +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0', 511) +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0/0', 511) +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0', 511) +2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', 'wx') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', 'wx') -> 00000000 +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', '/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/refcounts', 511) +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', 'wx') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', 'wx') -> 00000000 +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') +2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', '/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes', 511) +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0', 511) +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0/0', 511) +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0', 511) +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', 'wx') +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', 'wx') -> 00000000 +2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', '/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') +2012-07-07 13:47:56 DEBUG LRUCache <larch.lru.LRUCache object at 0x9491e6c>: hits=0 misses=2 +2012-07-07 13:47:56 DEBUG LRUCache <larch.lru.LRUCache object at 0x9491f8c>: hits=3 misses=2 +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new', 511) +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', 'wx') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', 'wx') -> 00000000 +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] close(00000000) +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/metadata') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', '/path/to/me/obnam/clientlist/new/metadata') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/') +2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/') +2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes') +2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0') +2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/nodes/0/0/0/6') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') +2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/nodes/0/0/0/7') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7', '/path/to/me/obnam/clientlist/nodes/0/0/0/7') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') +2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/refcounts/refcounts-0') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0', '/path/to/me/obnam/clientlist/refcounts/refcounts-0') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') +2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') +2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/metadata') +2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/metadata', '/path/to/me/obnam/clientlist/metadata') +2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/./lock') +2012-07-07 13:48:03 DEBUG opening repository (create=False) +2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] stat('/path/to/me/obnam/obnam/') +2012-07-07 13:48:03 CRITICAL Traceback (most recent call last): + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run + self.process_args(args) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 158, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args + method(args[1:]) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 226, in backup + self.add_client(client_name) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 293, in add_client + self.repo = self.app.open_repository(repofs=self.repo.fs.fs) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 187, in open_repository + repofs.reinit(repopath) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/sftp_plugin.py", line 58, in helper + raise OSError(e.errno, e.strerror or str(e), filename) +OSError: [Errno 2] No such file: 'sftp://me@server/~/obnam/' +</pre> + + +--- + +I've fixed this now. It turns out that the SFTP plugin did not handle +re-initialization repository paths that start with `/~/`. This will be +included in the next release. --liw + +[[done]] diff --git a/bugs/abort-if-sftp-breaks.mdwn b/bugs/abort-if-sftp-breaks.mdwn new file mode 100644 index 0000000..f004de3 --- /dev/null +++ b/bugs/abort-if-sftp-breaks.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +Obnam does not currently seem to notice when the sftp connection +breaks. It should, and it should then abort the backup. +--liw diff --git a/bugs/acl-xattr.mdwn b/bugs/acl-xattr.mdwn new file mode 100644 index 0000000..0cd7130 --- /dev/null +++ b/bugs/acl-xattr.mdwn @@ -0,0 +1,15 @@ +Obnam should provide xattr and ACL support. --liw + +I only need to support extended attributes, since ACLs are implemented +in terms of those. Good, this makes things simpler. I can just read the +list of xattrs on backup, and set it on restore, and I can treat the +list as a string blob. It seems that in Linux, one needs to do a syscall +for each key/value pair for a file, but there's a library function that +operates on multiple pairs at once, and on Irix they're syscalls. Those +functions have a fairly complicated API, though. Might want to keep +things simple, at least initially. --liw + +This is now implemented for local filesystems. Paramiko does not seem +to support them for SFTP, so that will have to not work. --liw + +[[done]] diff --git a/bugs/all-filesystem-types.mdwn b/bugs/all-filesystem-types.mdwn new file mode 100644 index 0000000..2961ad7 --- /dev/null +++ b/bugs/all-filesystem-types.mdwn @@ -0,0 +1,5 @@ +Obnam needs tests for using every filesystem type available as live +data or repository. + +Not sure how to arrange that without root access, but there's a need +to do that. diff --git a/bugs/arch-diagram.mdwn b/bugs/arch-diagram.mdwn new file mode 100644 index 0000000..06aa8c3 --- /dev/null +++ b/bugs/arch-diagram.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +It would be good to have an architecture diagram of the internals of +obnam, to make life easier for new code contributors. --liw + +[[done]] old wishlist bug, closing. --liw diff --git a/bugs/avoid-hash-collisions.mdwn b/bugs/avoid-hash-collisions.mdwn new file mode 100644 index 0000000..eacb9d7 --- /dev/null +++ b/bugs/avoid-hash-collisions.mdwn @@ -0,0 +1,17 @@ +It would be good for Obnam to optionally be able to verify that hash +collisions do not happen. This requires downloading the data with +the same hash, and comparing the bytes, which can be quite expensive. +However, backup programs should be reliable, and the user should be +able to choose reliability over speed. + +Three modes are needed: + +* assume hash collisions do not happen (deduplication, max speed, max risk) +* verify that hash collisions do not happen + (deduplication, no risk, performance hit) +* do not deduplicate (max speed, no risk) + +--liw + + +[[done]] diff --git a/bugs/avoid-stat-for-unchanged-directory.mdwn b/bugs/avoid-stat-for-unchanged-directory.mdwn new file mode 100644 index 0000000..e7900ec --- /dev/null +++ b/bugs/avoid-stat-for-unchanged-directory.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-wishlist]] + +This is an idea for optimizing Obnam. + +Store MD5 of string containing names + relevant metadata of all +files in a directory, then re-compute that when backing up a +directory: if checksums are same, then no file in the directory +has changed, and there's no need to check each of them separately, +saving many tree lookups + +Consider only non-dirs, since subsubdir can change without it being visible +at grandparent level. Thus, recursing is always necessary. + +--liw + diff --git a/bugs/backup-plugin-reporting.mdwn b/bugs/backup-plugin-reporting.mdwn new file mode 100644 index 0000000..e54df15 --- /dev/null +++ b/bugs/backup-plugin-reporting.mdwn @@ -0,0 +1,4 @@ +Fix obnam backup plugin to report when it's doing something else than +backing up a file. +--liw +[[done]] diff --git a/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn b/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn new file mode 100644 index 0000000..524b329 --- /dev/null +++ b/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +make "obnam backup --pretend" print a list of would-be-backed-up files +to stdout instead of tty + + +[[done]] I have decided against this. --liw diff --git a/bugs/backup-progress-unchanged-chunks.mdwn b/bugs/backup-progress-unchanged-chunks.mdwn new file mode 100644 index 0000000..bfa940a --- /dev/null +++ b/bugs/backup-progress-unchanged-chunks.mdwn @@ -0,0 +1,4 @@ +When Obnam is backing up a large file with many unchanged chunks, the +progress reporting should indicate this to the user. + +[[done]] -- this should be done now. --liw diff --git a/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn b/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn new file mode 100644 index 0000000..23bc3a2 --- /dev/null +++ b/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655229> for details. + +[[done]] diff --git a/bugs/bad-message-for-regexp-syntax-error.mdwn b/bugs/bad-message-for-regexp-syntax-error.mdwn new file mode 100644 index 0000000..624cd27 --- /dev/null +++ b/bugs/bad-message-for-regexp-syntax-error.mdwn @@ -0,0 +1,27 @@ +If an exclude regexp has a syntax error, the error message is awful: + + > peter@dewsbx07:/media/GreenDrive/Backups$ obnam backup $HOME + > Traceback (most recent call last): + > File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 141, in _run + > self.process_args(args) + > File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 123, in + > process_args + > cliapp.Application.process_args(self, args) + > File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 316, in + > process_args + > method(args[1:]) + > File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", + > line 95, in backup + > self.compile_exclusion_patterns() + > File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", + > line 136, in compile_exclusion_patterns + > for x in self.app.settings['exclude']] + > File "/usr/lib/python2.6/re.py", line 190, in compile + > return _compile(pattern, flags) + > File "/usr/lib/python2.6/re.py", line 245, in _compile + > raise error, v # invalid expression + > error: nothing to repeat + +--liw + +[[done]] diff --git a/bugs/baserock.mdwn b/bugs/baserock.mdwn new file mode 100644 index 0000000..36e5574 --- /dev/null +++ b/bugs/baserock.mdwn @@ -0,0 +1,3 @@ +Obnam hasn't been ported to Baserock. + +[[done]] -- I'll port it when I need it, and wishlist bugs are a burden. --liw diff --git a/bugs/benchmark-fsck.mdwn b/bugs/benchmark-fsck.mdwn new file mode 100644 index 0000000..05b96d4 --- /dev/null +++ b/bugs/benchmark-fsck.mdwn @@ -0,0 +1,5 @@ +`obnam-benchmark` should benchmark also the fsck operation. --liw + +next version of [[/seivot]] will do this. --liw + +[[done]] diff --git a/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn b/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn new file mode 100644 index 0000000..574b27a --- /dev/null +++ b/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn @@ -0,0 +1,6 @@ +When running `obnam-benchmark`, the restore elapsed timers end at +00:00:00. Why? --liw + +It's not a bug: it's reporting remaining time, not elapsed time. Oops. +--liw +[[done]] diff --git a/bugs/better-way-to-upgrade-format-versions.mdwn b/bugs/better-way-to-upgrade-format-versions.mdwn new file mode 100644 index 0000000..35edef9 --- /dev/null +++ b/bugs/better-way-to-upgrade-format-versions.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +Instead of in-place conversions, which are error prone and clunky, +a better way would be nice. Maybe some kind of dump/undump pair, +using a streamable format? diff --git a/bugs/bgproc.mdwn b/bugs/bgproc.mdwn new file mode 100644 index 0000000..be834fd --- /dev/null +++ b/bugs/bgproc.mdwn @@ -0,0 +1,103 @@ +[[!tag obnam-wishlist]] + +Obnam should do some processing in the background, for example uploading +of data to the backup repository. This would allow better use of the +bottleneck resource (network). Below is a journal entry with my thoughts +on how to implement that. It may be out of date by now, but we'll see. +I have a Python module to simplify the use of `multiprocessing` to do +jobs in the background (which avoids the Python global interpreter lock, +in case that matters). --liw + +--- + +Here's a design for Obnam concurrency that came to me the other +day while walking. + +The core of Obnam (and larch) is quite synchronous: read data from +file, read B-tree nodes, push chunks and B-tree nodes into repository. +Some of that can be parallelized, but not easily: it's already tricky +code, and making it even more tricky is going to require very strong +justification. + +Things like encrypting and decrypting files need to be done in parallel +with other things, for speed. These things are not really in the +core, and indeed are provided by plugins. + +So here's a way to them in parallel: + +* the core code stays synchronous, the way it is now +* whenever larch code needs to read a B-tree node, + it blocks until it gets it +* the node is read, synchronously, from wherever, and put into + a background processing queue (using Python multiprocess) +* the code that waits for the node to be processed polls the queue, + and handles any other background jobs that happen to finish while + it waits, and returns the desired node when it gets it +* when larch writes a node (after it gets pushed out + of the upload queue inside larch), it is put into a background + processing queue +* at the same time, if there were any finished background jobs, they're + handled (written to repo) +* at the end of the run, the main loop makes sure any pending background + jobs finish and are handled + +There's a complication that the B-tree code may need a node that is +not yet written to the repository, since it is still going through +a background processing queue. + +I'm going to need to restructure how hooks process files that +are written to or read from the repository. Writing should happen +asynchronously: files are put in a queue and processed in the +background, and then written to the actual repository when background +processing is finished. Reading needs to happen synchronously, since +there's a B-tree call waiting for them, but to handle the case of +needing a node that is still being processed +in the background, we need to keep track of what nodes are in the +background, and wait for them to be done before reading them. + +Reading would thus be something like this, implemented in the +`Repository` class: + + while wanted file is in write queue: + process a write queue result + + read file from repository + process file data through hooks + return file + +The write queue is more complicated (again handled somehow in the +`Repository` class): + +* a `multiprocessing.Queue` instance for holding pending jobs + - a job is a (pathname, file contents) pair +* another `Queue` instance for holding unhandled results + - (pathname, file contents) pair, where the contents may have changed +* a `set` for holding file identifiers (paths) that have been put into + the pending jobs queue, but not yet processed from the results queue + +Each plugin can provide one or more Unix commands (filters) through +which the file contents gets piped. The background processes run each +filter in turn, giving the output of the previous one as input to the +next one. + +To handle a result from a background job, the following needs to be done: + +* remove the pathname from the `set` +* write the filtered file contents into the repository + +To implement this, I'll do this: + +* All changes should be in `HookedFS` +* `write_file` and `overwrite_file` put things into the pending jobs queue, + and also call a new method `handle_background_results` +* `cat` gets changed to wait for files in the write queue, calling + `handle_background_results` +* `handle_background_results` will do what is needed + +This design isn't optimal, since writing things to the repository +isn't being done in parallel with other things, but I'll tackle that +problem later. + + +[[done]] this clearly isn't happening, so closing the old wishlist +bug --liw diff --git a/bugs/block-device-backup.mdwn b/bugs/block-device-backup.mdwn new file mode 100644 index 0000000..d570402 --- /dev/null +++ b/bugs/block-device-backup.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +Obnam could do with a mode in which it backs up the data from a block +device, instead of the device node. If the block device contains a +filesystem, it should backup only the parts of the device that are +used by the filesystem, and skip unused parts. That could be used for +backing up disk images as well. diff --git a/bugs/c-lstat.mdwn b/bugs/c-lstat.mdwn new file mode 100644 index 0000000..fc9399c --- /dev/null +++ b/bugs/c-lstat.mdwn @@ -0,0 +1,9 @@ +Obnam is using the Python os.lstat method to get metadata from files. +Python returns timestamps as floats, which are awkward to store on +disk. It would be good to switch to a C extension that does the lstat +system call and returns timestamps as (pairs of) integers instead. +This would allow full timestamp precision. + +--liw + +[[done]] diff --git a/bugs/chunk-size-based-on-data-type-and-size.mdwn b/bugs/chunk-size-based-on-data-type-and-size.mdwn new file mode 100644 index 0000000..4fcfc33 --- /dev/null +++ b/bugs/chunk-size-based-on-data-type-and-size.mdwn @@ -0,0 +1,16 @@ +[[!tag obnam-wishlist]] + +Would it be beneficial to choose the size of data chunks based on the +type of data, or its size? Text data might benefit from small chunks, +whereas video data might benefit from large chunks, for example. +Possibly the size of a file would be a sufficient indicator. + +The goal would be to reduce the number of chunks, without reducing +the effectiveness of de-duplication. + +This needs to be measurements of real data to see if there are interesting +parameters. + +--liw + +[[done]] interesting ideas, but old wishlist bug. --liw diff --git a/bugs/chunks-dir-init-has-no-lock.mdwn b/bugs/chunks-dir-init-has-no-lock.mdwn new file mode 100644 index 0000000..36402d6 --- /dev/null +++ b/bugs/chunks-dir-init-has-no-lock.mdwn @@ -0,0 +1,14 @@ +[[!tag obnam-1.0-blocker]] + +When Obnam initializes the `chunks` directory, for encryption, it is +done without locking. That's a race condition: two clients started +at the same time against an empty directory may result in both of +them creating the same files. + +Normal use of the `chunks` directory does not need locking: clients +are free to create new chunks files there, as long as they take care +to not overwrite existing files. + +--liw + +[[done]] in bzr diff --git a/bugs/clients-list-should-include-ids.mdwn b/bugs/clients-list-should-include-ids.mdwn new file mode 100644 index 0000000..00232f8 --- /dev/null +++ b/bugs/clients-list-should-include-ids.mdwn @@ -0,0 +1,13 @@ +Obnam needs a way to show the id of each client, so one can, say, +safely remove the toplevel directory for the client. + +--- + +Obnam now has an internal interface API for accessing repositories, +and it doesn't expose the client id anymore. So this needs some +thought: should we expose the id (even though it's not otherwise +needed)? And what about future repository interface API +implementations that don't need a client id at all anymore? I suspect +it isn't going to useful enough to do this. If I get other uses +cases, I'll re-consider. [[done]], for now. --liw + diff --git a/bugs/clients-subcommand-for-non-clients.mdwn b/bugs/clients-subcommand-for-non-clients.mdwn new file mode 100644 index 0000000..28024d6 --- /dev/null +++ b/bugs/clients-subcommand-for-non-clients.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +The "obnam clients" command should work even for non-clients (modulo +encryption), so that you can get a list of the client names without +having to guess one. --liw + +Fixed now. [[done]] --liw diff --git a/bugs/commit-progress-reporting.mdwn b/bugs/commit-progress-reporting.mdwn new file mode 100644 index 0000000..2123a4c --- /dev/null +++ b/bugs/commit-progress-reporting.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +Improve Obnam's progress reporting during committing. + +- how much work will there be to do the commit? +- how many files to upload? +- how many files to move in journal? diff --git a/bugs/committing-changes-takes-long-without-feedback.mdwn b/bugs/committing-changes-takes-long-without-feedback.mdwn new file mode 100644 index 0000000..3aa744c --- /dev/null +++ b/bugs/committing-changes-takes-long-without-feedback.mdwn @@ -0,0 +1,8 @@ +When Obnam is committing changes (at end, or during checkpoint), +there can be a long time when nothing seems to be happening. It +needs to provide some feedback of what's going on. + + +-- + +[[done]] --liw diff --git a/bugs/compression-and-encryption-together-fails.mdwn b/bugs/compression-and-encryption-together-fails.mdwn new file mode 100644 index 0000000..3de0071 --- /dev/null +++ b/bugs/compression-and-encryption-together-fails.mdwn @@ -0,0 +1,46 @@ +Trying to create a new backup using both compression and encryption fails. The command I use is: + + obnam --compress-with=gzip --encrypt-with=ABCD1234 --repository /tmp/asd backup $HOME/stuff + +The error message printed: “error: Error -3 while decompressing data: incorrect header check” +You may view the whole stack trace at http://p.nnev.de/2368 because I couldn’t figure out how to add it properly with the formatting options. + +--xeen + +-- + +The above stack trace: + + Traceback (most recent call last): + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 144, in _run + self.process_args(args) + File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 127, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 350, in process_args + method(args[1:]) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup + self.add_client(client_name) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 143, in add_client + self.repo.add_client(client_name) + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 321, in add_client + if client_name in self.list_clients(): + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 200, in list_clients + self.check_format_version() + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 311, in check_format_version + on_disk = self.get_format_version() + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 282, in get_format_version + data = self.fs.cat('metadata/format') + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 66, in cat + repo=self.repo, toplevel=toplevel) + File "/usr/lib/python2.6/dist-packages/obnamlib/hooks.py", line 109, in call + return self.hooks[name].call_callbacks(*args, **kwargs) + File "/usr/lib/python2.6/dist-packages/obnamlib/hooks.py", line 73, in call_callbacks + data = callback(data, *args, **kwargs) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/compression_plugin.py", line 45, in toplevel_read_data + return zlib.decompress(data) + error: Error -3 while decompressing data: incorrect header check + + +--liw + +Fixed in bzr. [[done]] --liw diff --git a/bugs/compression-methods.mdwn b/bugs/compression-methods.mdwn new file mode 100644 index 0000000..967e391 --- /dev/null +++ b/bugs/compression-methods.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +Supporting multiple compression methods would be nice. + +Also one idea is to allow using one compression method (or no compression) while actually doing the backup, and then have a separate tool, to be run on otherwise idle time, recompress the data using some other, slower algorithm to conserve disk space without making the backup process slower. + +-- SLi + +[[done]] while this would be useful, it hasn't happened, and making +the bug list longer is hampering other things. --liw diff --git a/bugs/crash_when_started_without_--log_option.mdwn b/bugs/crash_when_started_without_--log_option.mdwn new file mode 100644 index 0000000..cbccc5d --- /dev/null +++ b/bugs/crash_when_started_without_--log_option.mdwn @@ -0,0 +1,25 @@ +obnam 0.23 crashes when started without --log option. Here is what it spits out on the console: + + user@host:~$ obnam backup ./ + CRITICAL:root:Traceback (most recent call last): + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 132, in _run + self.setup_logging() + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 279, in setup_logging + handler = logging.NullHandler() + AttributeError: 'module' object has no attribute 'NullHandler' + + Traceback (most recent call last): + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 132, in _run + self.setup_logging() + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 279, in setup_logging + handler = logging.NullHandler() + AttributeError: 'module' object has no attribute 'NullHandler' + +-- weinzwang + +This is actually a bug in cliapp 0.22, which switched to using +`logging.NullHandler`. That only exists in Python 2.7, and you're +using 2.6. Workaround: `obnam --log=/dev/null`. I'll release a new +cliapp soon, it's already fixed in bzr. --liw + +[[done]] diff --git a/bugs/crashing-early-leads-to-assertion-error.mdwn b/bugs/crashing-early-leads-to-assertion-error.mdwn new file mode 100644 index 0000000..bda5d11 --- /dev/null +++ b/bugs/crashing-early-leads-to-assertion-error.mdwn @@ -0,0 +1,39 @@ +> `<SLi> liw, I run obnam backup for a small while (probably not +> until even the first checkpoint) on a new repository, then ctrl-c +> and say obnam fsck, I get AssertionError on assert 'key_size' in +> ns_temp.get_metadata_keys().` + +Two things: + +* using `assert` is wrong in the code, it should raise an exception that + results in a humane error message +* crashing before the first checkpoint should not result in an error + in the first place + +--liw + +I've fixed the assertion in larch (it's now a more user-friendly +error message). The crash still occurs, but I'm not sure if it's +reasonable for Obnam to try to correct that. The same error may +happen due to other reasons, and automatically fixing things is +too likely to break things. So I'll leave the bug open, but take +it off the 1.0 blockers list. + +--liw + +I can reproduce this, by instrumenting the code so that +it crashes after having backed up the first file. The error printed +correctly identifies that the B-tree is broken. However, it is +broken in a useful way. The B-tree is going to be so broken it +does not have much useful information. There are some things that +could be done (e.g., if no key size is stored in the B-tree, add +one), but that only helps in specific cases, and I don't think +that's useful enough. Thus, I think I will just give up for this +kind of early corruption and have people start over. Sorry. +(I'd fix this if I had lots of extra time, but I don't. If someone +comes upon this bug later on, and makes a good patch, I'd be +happy to accept it, of course.) + +--liw + +[[done]] diff --git a/bugs/ctime.mdwn b/bugs/ctime.mdwn new file mode 100644 index 0000000..6b6433e --- /dev/null +++ b/bugs/ctime.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Obnam should, arguably, use ctime changes to trigger backups, so that +if a file's size and mtime are the same, because whatever fool program +modified a file reset the mtime, obnam will still backup the changed +data. + +I have code for this, but it requires a repository format change, +and breaks the upgrade from format 5 to 6. --liw diff --git a/bugs/deduplication-stats.mdwn b/bugs/deduplication-stats.mdwn new file mode 100644 index 0000000..71a689e --- /dev/null +++ b/bugs/deduplication-stats.mdwn @@ -0,0 +1,22 @@ +[[!tag obnam-wishlist]] + +It would be nice if Obnam could tell you how much you've saved thanks +to the de-duplication. --liw + +From discussion on the mailing list ([archive](http://vlists.pepperfish.net/pipermail/obnam-flarn.net/2012-June/000120.html): + +* how much space will be freed if I remove this generation? +* or a set of generations? +* how much space has been saved by de-duplicated chunks in the whole repo? + +Also, how much space has been saved with compression (alone or with encryption? + +Also, store per-generation data in the generation for faster retrieval. + +* Bytes added by this generation. +* Number of added/changed/removed files in this generation. + +--liw + +[[done]] if keeping the wishlist bug open would help make this happen, +it would have happened already. --liw diff --git a/bugs/default-exclusions.mdwn b/bugs/default-exclusions.mdwn new file mode 100644 index 0000000..c22483a --- /dev/null +++ b/bugs/default-exclusions.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +From Enrico: It might be good to have a way for Obnam to automatically +exclude certain kinds of common stuff, such as web browser caches, +Liferea caches, etc. This should be easy to enable, and should be +off by default (safe defaults are important). diff --git a/bugs/delete-files-from-generations.mdwn b/bugs/delete-files-from-generations.mdwn new file mode 100644 index 0000000..5530859 --- /dev/null +++ b/bugs/delete-files-from-generations.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +If you accidentally backup some large or sensitive files, but don't want +to delete all the generations they're in, it would be handy for Obnam to +be able to delete just the specific files from the generations, and leave +the rest. diff --git a/bugs/disable-paramiko-logging.mdwn b/bugs/disable-paramiko-logging.mdwn new file mode 100644 index 0000000..5747efb --- /dev/null +++ b/bugs/disable-paramiko-logging.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +It would be good to have obnam provide an option to enable/disable +paramiko logging. It is often voluminous and might not be helpful +when reading logs for debugging. --liw + + +This doesn't seem to be a problem anymore. [[done]] --liw diff --git a/bugs/disk-full.mdwn b/bugs/disk-full.mdwn new file mode 100644 index 0000000..ca8d892 --- /dev/null +++ b/bugs/disk-full.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-1.0-blocker]] + +How does Obnam deal with a full disk for the repository? + +- create a disk image of suitable size and create an ext4 fs on it +- mount the disk image and use it as a repository +- backup something bigger than what will fit into the disk image +- how does obnam handle the error? + +--liw + + +Obnam now aborts properly on out-of-space errors. --liw + +[[done]] diff --git a/bugs/do-not-cross-bind-mounts.mdwn b/bugs/do-not-cross-bind-mounts.mdwn new file mode 100644 index 0000000..8bf7850 --- /dev/null +++ b/bugs/do-not-cross-bind-mounts.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +When --one-file-system is used, it would be nice to not cross +bind-mounts. No idea how to figure that out, but it must be +possible. --liw + +You could look at the inode numbers for . and ./foodir/.. and check they're the same? -- kinnison + +The inode check will not work if foodir is a symlink. --mathstuf diff --git a/bugs/document-black-box-testing.mdwn b/bugs/document-black-box-testing.mdwn new file mode 100644 index 0000000..5bcbc38 --- /dev/null +++ b/bugs/document-black-box-testing.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-wishlist]] + +Review obnam black box tests and document how you're supposed to write them. + +- hacking section in README + + +--- + +On second though, I'll just let people read yarn documentation, +these days. --liw [[done]] diff --git a/bugs/document-how-to-check-encryption-is-being-used.mdwn b/bugs/document-how-to-check-encryption-is-being-used.mdwn new file mode 100644 index 0000000..1c44e3d --- /dev/null +++ b/bugs/document-how-to-check-encryption-is-being-used.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +Document how to check that Obnam is actually encrypting anything. --liw + + +This is in the manual now +(http://code.liw.fi/obnam/manual/manual.html#checking-if-a-repository-uses-encryption). +[[done]] --liw diff --git a/bugs/document-repo-upgrade.mdwn b/bugs/document-repo-upgrade.mdwn new file mode 100644 index 0000000..345b8b5 --- /dev/null +++ b/bugs/document-repo-upgrade.mdwn @@ -0,0 +1,9 @@ +Although there is currently no repository upgrade method, +the manpage should at least document that you need to start +over from scratch. + +Better yet, make a way to upgrade, of course. + +--liw + +[[done]] (see [[repo-upgrades-missing]] diff --git a/bugs/does-not-pass-test-suite-on-fedora17.mdwn b/bugs/does-not-pass-test-suite-on-fedora17.mdwn new file mode 100644 index 0000000..3f63ee0 --- /dev/null +++ b/bugs/does-not-pass-test-suite-on-fedora17.mdwn @@ -0,0 +1,9 @@ +Make Obnam pass its test suite (esp. test-locking) on Fedora 17. + +- yum install gcc python-sphinx python-devel + +--- + +I believe this is now the case, since there are Fedora packages. +[[done]] +--liw diff --git a/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn b/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn new file mode 100644 index 0000000..d621452 --- /dev/null +++ b/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn @@ -0,0 +1,29 @@ + <andrewsh> 20:53:15> liw: hi, I have strange issue with obnam's locks + <andrewsh> 20:53:35> I had OSError: [Errno 2] No such file or + directory: '/media/box' + <andrewsh> 20:53:56> which is itself okay, a remote mount was + inaccessible at that moment + <andrewsh> 20:54:05> obnam said, 2012-08-29 21:10:19 INFO Unlocking + client because of error + <andrewsh> 20:54:20> however, it did not unlock the repository + really + <andrewsh> 20:54:33> and the next time it gave me LockFail: + Lock timeout + <andrewsh> 21:01:47> and this is obnam 1.0 if that matters + <andrewsh> yes + <andrewsh> liw: + <andrewsh> well, no + <andrewsh> well, both :D + <andrewsh> remote mount wasn't the repo, remote mount was what + I wanted to back up + <andrewsh> repo was on the local disk + <andrewsh> unavailability of a remote mount to back up caused + an interruption + <andrewsh> but despite telling me it's unlocking the repo, obnam + didn't do this + + +--- + +This should now be fixed in bzr. --liw +[[done]] diff --git a/bugs/done.mdwn b/bugs/done.mdwn new file mode 100644 index 0000000..6fa70a6 --- /dev/null +++ b/bugs/done.mdwn @@ -0,0 +1,11 @@ +Closed bugs for Obnam +===================== + +[[!inline pages="page(obnam/bugs/*) and + !obnam/bugs/*/* and + !obnam/bugs/done and + link(obnam/bugs/done)" + show=0 + sort=mtime + archive=yes]] + diff --git a/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn b/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn new file mode 100644 index 0000000..08ce0f7 --- /dev/null +++ b/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn @@ -0,0 +1,298 @@ +(I can't figure out how to simply mark a block of text as +code or monospaced, other than indenting every line manually. +The FormattingHelp page isn't all that helpful.) + +> I've reformatted. Indenting by four spaces is the markdown way, +> or you can use the pre element in HTML (markdown and Branchable +> allow a subset of HTML to be used, including pre). --liw + +When I try to build Obnam 1.0 I get this error. I noticed that the +same build error occurs in Chris's PPA, and so Obnam isn't available +on Precise or Quantal. + +---- + +<pre> +rm -rf build +cp -a test-gpghome temp.gpghome +env GNUPGHOME=temp.gpghome python setup.py check --fast +running check +run unit tests +Running test 167/387: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' +Running test 168/387: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' +Running test 387/387: test_removes_chunk (chunklist_tests.ChunkListTests + +OK +121 excluded statements +22 excluded modules +Time: 5.1 s +run black box tests +FAIL: root-is-symlink: got exit code 1, expected 0 +41/42 tests OK, 1 failures +ERROR: Command '['cmdtest', 'tests']' returned non-zero exit status 1 +</pre> + +---- + +I don't know why root-is-symlink.script is failing. There is output, +and I guess there is not supposed to be? + +(Is there really no way to use preformatted text in Markdown other +than indenting every line? This seems like a step backwards. +What's wrong with `[pre][/pre]` ?) + +<pre> +$ cat root-is-symlink.stdout-actual +--- /tmp/tmp16rwh9/data/data.summain 2012-06-09 16:39:12.000000000 -0500 ++++ /tmp/tmp16rwh9/data/restored.summain 2012-06-09 16:39:12.000000000 -0500 +@@ -11,127 +11,3 @@ + Group: me + Target: data.real + +-Name: 0 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 40775 +-Ino: 2 +-Dev: 1 +-Nlink: 3 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +- +-Name: 0/0 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 40775 +-Ino: 3 +-Dev: 1 +-Nlink: 3 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +- +-Name: 0/0/0 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 40775 +-Ino: 4 +-Dev: 1 +-Nlink: 2 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +- +-Name: 0/0/0/0 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 5 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: 7b2f423343cc5a4b5afbd8792b84aeee3b1eef68 +- +-Name: 0/0/0/1 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 6 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: 99b6a6eac04a1dee31e07245c60bf6ff60c5dbb7 +- +-Name: 0/0/0/2 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 7 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: ca26a73aaf98ba4634794952c29bc02bed536529 +- +-Name: 0/0/0/3 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 8 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: d60b38536692c387932ffca6b59a235fa7744c22 +- +-Name: 0/0/0/4 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 9 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: e5213924a2ce010a667605951a2abee56f2373da +- +-Name: 0/0/0/5 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 10 +-Dev: 1 +-Nlink: 1 +-Size: 16384 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: f1837f15230109d673bf741afc0bb8e8cd343538 +- +-Name: 0/0/0/6 +-Mtime: 2012-06-09 21:39:11.000000 +0000 +-Mode: 100664 +-Ino: 11 +-Dev: 1 +-Nlink: 1 +-Size: 1696 +-Uid: 1000 +-Username: me +-Gid: 1000 +-Group: me +-SHA1: 578a819d61ad07e996900a236cbc33b70d47cd75 +</pre> + + +--- + +The PPA now has 1.0 for Ubuntu, so I assume the problem is fixed. +I seem to recall it was a problem of a too-old dependency. Are you +still having this problem? --liw + +--- + +Thanks for reformatting my stuff. + +I still see the build failure in the PPA for Precise and Quantal: + +https://launchpad.net/~chris-bigballofwax/+archive/obnam-ppa/+packages + +It hasn't been attempted for 10 days, but I don't see any upgradable Python packages on my system, so I'm not sure it would help. + +On the other hand, I just ran debian/rules on 1.0 again, and now the output is different. Maybe I was doing it wrong before? It seems to complete, but it also says "FAILED" and that it exited non-zero. I am confused. :) + +<pre> +debian/rules +python setup.py build_ext -i +running build_ext +building 'obnamlib._obnam' extension +creating build +creating build/temp.linux-i686-2.7 +gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _obnammodule.c -o build/temp.linux-i686-2.7/_obnammodule.o +gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-i686-2.7/_obnammodule.o -o /home/me/Projects/Python/obnam/trunk/obnamlib/_obnam.so +rm -rf build +cp -a test-gpghome temp.gpghome +env GNUPGHOME=temp.gpghome python setup.py check --fast +running check +run unit tests +Running test 167/774: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' +Running test 168/774: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' +Running test 554/774: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' +Running test 555/774: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' +Running test 774/774: test_removes_chunk (chunklist_tests.ChunkListTests + +FAILED + +Statements missed by per-module tests: + Module Missed statements + obnam-1.0/obnamlib/checksumtree.py 34-39, 42, 45, 48-53, 56-63, 66-71, 75-81 + obnam-1.0/obnamlib/checksumtree.py 34-39, 42, 45, 48-53, 56-63, 66-71, 75-81 + obnam-1.0/obnamlib/chunklist.py 37-42, 45, 48-51, 54-57, 60-63 + obnam-1.0/obnamlib/chunklist.py 37-42, 45, 48-51, 54-57, 60-63 + obnam-1.0/obnamlib/clientlist.py 49-59, 62, 65, 68-69, 72, 75, 78-84, 87-93, 96-99, 102-114, 117-122, 125-132, 135-142 + obnam-1.0/obnamlib/clientlist.py 49-59, 62, 65, 68-69, 72, 75, 78-84, 87-93, 96-99, 102-114, 117-122, 125-132, 135-142 + obnam-1.0/obnamlib/clientmetadatatree.py 73-82, 85-86, 90-95, 99, 124-131, 135, 139-140, 144, 148, 152, 157-158, 163-180, 185-204, 207, 210, 213-224, 227-228, 231-232, 236-250, 253-261, 264-271, 274-277, 280-285, 288-292, 295, 298-299, 304-307, 310-311, 315, 318-323, 326-328, 331, 334-369, 372-376, 379-387, 390-432, 435-445, 448-459, 462-463, 466-469, 472-474, 477-491, 494-509, 514-517, 522-525 + obnam-1.0/obnamlib/clientmetadatatree.py 73-82, 85-86, 90-95, 99, 124-131, 135, 139-140, 144, 148, 152, 157-158, 163-180, 185-204, 207, 210, 213-224, 227-228, 231-232, 236-250, 253-261, 264-271, 274-277, 280-285, 288-292, 295, 298-299, 304-307, 310-311, 315, 318-323, 326-328, 331, 334-369, 372-376, 379-387, 390-432, 435-445, 448-459, 462-463, 466-469, 472-474, 477-491, 494-509, 514-517, 522-525 + obnam-1.0/obnamlib/encryption.py 29-36, 44, 47-49, 52-54, 57, 79-97, 102, 107, 113-129, 134, 149-151, 154-157, 160-161, 165, 168-175, 178-180, 183, 186, 189-192, 195, 198, 201-212, 222-229, 234-237, 248 + obnam-1.0/obnamlib/encryption.py 29-36, 44, 47-49, 52-54, 57, 79-97, 102, 107, 113-129, 134, 149-151, 154-157, 160-161, 165, 168-175, 178-180, 183, 186, 189-192, 195, 198, 201-212, 222-229, 234-237, 248 + obnam-1.0/obnamlib/forget_policy.py 49-73, 76-93, 111-118 + obnam-1.0/obnamlib/forget_policy.py 49-73, 76-93, 111-118 + obnam-1.0/obnamlib/hooks.py 44-45, 54-60, 64-65, 69-71, 79-81, 99-100, 103-107, 110-111, 114, 117-123, 126-137, 145-146, 155-156, 160-161, 165-168, 172-175, 179, 183, 187 + obnam-1.0/obnamlib/hooks.py 44-45, 54-60, 64-65, 69-71, 79-81, 99-100, 103-107, 110-111, 114, 117-123, 126-137, 145-146, 155-156, 160-161, 165-168, 172-175, 179, 183, 187 + obnam-1.0/obnamlib/lockmgr.py 28-34, 37-43, 52-54, 57, 61-70, 73, 77-85, 89-90 + obnam-1.0/obnamlib/lockmgr.py 28-34, 37-43, 52-54, 57, 61-70, 73, 77-85, 89-90 + obnam-1.0/obnamlib/metadata.py 80-83, 86, 89, 92, 100-109, 167-192, 207-220, 242-265, 275-316 + obnam-1.0/obnamlib/metadata.py 80-83, 86, 89, 92, 100-109, 167-192, 207-220, 242-265, 275-316 + obnam-1.0/obnamlib/pluginbase.py 25 + obnam-1.0/obnamlib/pluginbase.py 25 + obnam-1.0/obnamlib/repo.py 47-49, 52, 55-57, 62-66, 70, 73-78, 81-86, 128-156, 159, 164-167, 174-178, 187-189, 193, 197, 201, 206-211, 215-216, 220-221, 225-226, 230-231, 235-236, 240-241, 245-246, 250-251, 261-272, 276-282, 286-300, 309-322, 326-329, 339-341, 348-353, 363-367, 371, 382-396, 402-406, 410-414, 424-448, 452-462, 466-476, 480-492, 496-497, 501-502, 511-519, 530-573, 577-580, 589-590, 594-595, 600-605, 609-611, 615-616, 619, 632-655, 666-673, 677-678, 682-683, 692-693, 697-704, 717-725, 729-730, 739-740, 749-750, 765-779, 790-803 + obnam-1.0/obnamlib/repo.py 47-49, 52, 55-57, 62-66, 70, 73-78, 81-86, 128-156, 159, 164-167, 174-178, 187-189, 193, 197, 201, 206-211, 215-216, 220-221, 225-226, 230-231, 235-236, 240-241, 245-246, 250-251, 261-272, 276-282, 286-300, 309-322, 326-329, 339-341, 348-353, 363-367, 371, 382-396, 402-406, 410-414, 424-448, 452-462, 466-476, 480-492, 496-497, 501-502, 511-519, 530-573, 577-580, 589-590, 594-595, 600-605, 609-611, 615-616, 619, 632-655, 666-673, 677-678, 682-683, 692-693, 697-704, 717-725, 729-730, 739-740, 749-750, 765-779, 790-803 + obnam-1.0/obnamlib/sizeparse.py 25, 31, 37, 61, 64-66, 69-79 + obnam-1.0/obnamlib/sizeparse.py 25, 31, 37, 61, 64-66, 69-79 + obnam-1.0/obnamlib/vfs_local.py 34-39, 42-45, 55-63, 76-91, 94, 97-101, 104-109, 114-116, 119, 122-124, 127-129, 132-137, 157, 160, 185-186, 189-197, 200-203, 206, 209-212, 215-227, 230, 233, 236-238, 241-243, 246-248, 251-253, 256-267, 270-279, 282-286, 289-306, 309, 312-319 + obnam-1.0/obnamlib/vfs_local.py 34-39, 42-45, 55-63, 76-91, 94, 97-101, 104-109, 114-116, 119, 122-124, 127-129, 132-137, 157, 160, 185-186, 189-197, 200-203, 206, 209-212, 215-227, 230, 233, 236-238, 241-243, 246-248, 251-253, 256-267, 270-279, 282-286, 289-306, 309, 312-319 + +Modules missing test modules: + obnam-1.0/setup.py + obnam-1.0/test-plugins/wrongversion_plugin.py + obnam-1.0/test-plugins/oldhello_plugin.py + obnam-1.0/test-plugins/aaa_hello_plugin.py + obnam-1.0/test-plugins/hello_plugin.py + obnam-1.0/obnamlib/__init__.py + obnam-1.0/obnamlib/vfs.py + obnam-1.0/obnamlib/app.py + obnam-1.0/obnamlib/repo_tree.py + obnam-1.0/obnamlib/plugins/__init__.py + obnam-1.0/obnamlib/plugins/restore_plugin.py + obnam-1.0/obnamlib/plugins/verify_plugin.py + obnam-1.0/obnamlib/plugins/forget_plugin.py + obnam-1.0/obnamlib/plugins/show_plugin.py + obnam-1.0/obnamlib/plugins/force_lock_plugin.py + obnam-1.0/obnamlib/plugins/sftp_plugin.py + obnam-1.0/obnamlib/plugins/vfs_local_plugin.py + obnam-1.0/obnamlib/plugins/encryption_plugin.py + obnam-1.0/obnamlib/plugins/fsck_plugin.py + obnam-1.0/obnamlib/plugins/compression_plugin.py + obnam-1.0/obnamlib/plugins/backup_plugin.py + obnam-1.0/obnamlib/plugins/convert5to6_plugin.py + +0 failures, 0 errors +242 excluded statements +22 excluded modules +22 missing test modules + +Slowest tests: + 0.2 s test_latest_returns_only_generation (repo_tests.RepositoryGenspecTests + 0.3 s test_counts_files_in_first_generation (clientmetadatatree_tests.Client + 0.3 s test_latest_returns_newest_generation (repo_tests.RepositoryGenspecTes + 0.4 s test_removing_client_that_has_data_removes_the_data_as_well (repo_test + 0.4 s test_latest_returns_newest_generation (repo_tests.RepositoryGenspecTes + 0.4 s test_find_chunks_finds_what_put_chunk_puts (repo_tests.RepositoryChunk + 0.4 s test_removing_generation_works (repo_tests.RepositoryClientTests) + 0.6 s test_removing_generation_works (repo_tests.RepositoryClientTests) + 0.6 s test_removing_only_second_generation_works (repo_tests.RepositoryClien + 0.9 s test_removing_only_second_generation_works (repo_tests.RepositoryClien +Time: 15.1 s +ERROR: Command '['python', '-m', 'CoverageTestRunner', '--ignore-missing-from=without-tests']' returned non-zero exit status 1 +make: *** [override_dh_auto_test] Error 1 +</pre> + +--- + +Ah, I must have misunderstood the Launchpad page. Let's see what Chris can find out. --liw + +--- + +I don't know what or if anything was fixed, but 1.0 is in the PPA now and working. Thanks. diff --git a/bugs/encrypt-fails-gpg-batchmode.mdwn b/bugs/encrypt-fails-gpg-batchmode.mdwn new file mode 100644 index 0000000..838a15f --- /dev/null +++ b/bugs/encrypt-fails-gpg-batchmode.mdwn @@ -0,0 +1,6 @@ +See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649769>. --liw + +The gpg agent problem is sorted, but there's a mysterious crash. +Does not seem important enough at this point to be a 1.0 blocker. --liw + +[[done]] diff --git a/bugs/encryption-improvement-suggestion.mdwn b/bugs/encryption-improvement-suggestion.mdwn new file mode 100644 index 0000000..7a49354 --- /dev/null +++ b/bugs/encryption-improvement-suggestion.mdwn @@ -0,0 +1,37 @@ +[[!tag obnam-wishlist]] + + <zeri> liw: I tried out obnam yesterday and noticed you guys + simply call gpg -c --batch for the symmetric encryption part + <zeri> liw: also if I am not misstaken you sort of have a master + key for the repository (64 bit hex number encrypted for all keys + in the repository that is used as passphrase) + <zeri> liw: since you do not sepcify the + s2k-algo,s2k-mode,s2k-count configuration options of gpg as + well as the compression-algo option a gpg.conf that is usually + considered good will slow down obnam to a couple of bytes/sec + <zeri> liw: since keyderivation is of no use if a sufficiently + big random secret is used you might want to consider specifying + --s2k-mode 1 to disable most of the keystrengthening in gpg and + simply hash the password once with a salt ... speeding up the + encryption of every block at least by one order of magnitude + <zeri> (default behaviour is the compute a hash chain of at least + 1024 length up to 65011712 which was in my gpg.conf) + <zeri> also I didn't check whether you do compression in obnam + but gpg can do that for you as well but it was turned off in my + gpg.conf (I used gpg primarily for large tar balls where the one + time overhead doesn't matter) + <zeri> liw: then again I might be totally wrong and overlooked + some switch to do all that without chaning the code :) + <zeri> oh and I didn't emphasise this yet ... this hash chain + i talked about before is computed for every "chunk" which were + between a couple of bytes and 16k in my test ... the effort to + compute this hashchain (to optain encryption/authentication keys) + exceeds the effort to ecrypt 16k with any blockcipher by far + I suppose + <zeri> and it's sole purpose is to prevent weak passwords for + being guesed in short time (since the computational effort to + test a password is equivalent to computing this hash chain thus + slowing bruteforce down by the factor of the length of the chain) + <zeri> "64 bit hex number encrypted" << that should have been 64 + digits :) + diff --git a/bugs/errors-during-backup-should-cause-exit-1.mdwn b/bugs/errors-during-backup-should-cause-exit-1.mdwn new file mode 100644 index 0000000..26a6b0f --- /dev/null +++ b/bugs/errors-during-backup-should-cause-exit-1.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-1.0-blocker]] + +If there's an error during a backup, even if Obnam continues to back up +another file, the exit code should still be 1. + +(Thanks, weasel.) + +--liw + +[[done]] in current bzr. diff --git a/bugs/estimate-backup-time.mdwn b/bugs/estimate-backup-time.mdwn new file mode 100644 index 0000000..d345f41 --- /dev/null +++ b/bugs/estimate-backup-time.mdwn @@ -0,0 +1,5 @@ +It would be cool if Obnam could do a very quick estimate of the amount +of new data to be backed up, without having to perform a complete new +backup. --liw + +This is [[done]] now. --liw diff --git a/bugs/estimate-to-be-freed-space.mdwn b/bugs/estimate-to-be-freed-space.mdwn new file mode 100644 index 0000000..b17ef44 --- /dev/null +++ b/bugs/estimate-to-be-freed-space.mdwn @@ -0,0 +1,16 @@ +[[!tag obnam-wishlist]] + + +It would be nice for Obnam to have a tool to answer the question +"how much space will be freed if I remove these generations?" + + - need to find list of chunks that are used only by the specified gens + - perhaps also count B-tree reduction? cound nodes that are unshared by + the relevant trees, or only shared by the trees to be deleted + - however, the B-trees are going to be a fraction (a few percent) of + the size of the chunk data, so they're not really worth it + +--liw + + +[[done]] ancient wishlist bug, closing. --liw diff --git a/bugs/exclude-after-readlink.mdwn b/bugs/exclude-after-readlink.mdwn new file mode 100644 index 0000000..db5ea4f --- /dev/null +++ b/bugs/exclude-after-readlink.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +Excluding files based on `readlink $file` would be nice to have. As an example, anything that symlinks to somewhere under $HOME/code/dotfiles is handled by my dotfiles system. --mathstuf + +I could see making the backup plugin have hooks to extend exclusions +so that an other plugin could provide this functionality. +However, it looks fairly specific, so I don't think I'll write a plugin myself. +Anyone who does, please send patches to the mailing list, I'll be happy +to consider them. +[[done]] - -liw diff --git a/bugs/exclude-based-on-mime-type.mdwn b/bugs/exclude-based-on-mime-type.mdwn new file mode 100644 index 0000000..1de5e1d --- /dev/null +++ b/bugs/exclude-based-on-mime-type.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +Exclude based on mime type of file. + +[[done]] -- not written, but keeping a wishlist bug open clearly +isn't making it happen. --liw diff --git a/bugs/exclude-based-on-size.mdwn b/bugs/exclude-based-on-size.mdwn new file mode 100644 index 0000000..b942350 --- /dev/null +++ b/bugs/exclude-based-on-size.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +Write Obnam plugin to exclude files based on size. + + +[[done]] -- not written, but keeping wishlist bug open isn't making +it happening. I'll write this if I really need it. --liw diff --git a/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn b/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn new file mode 100644 index 0000000..91a8ab4 --- /dev/null +++ b/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn @@ -0,0 +1,108 @@ +Obnam version 1.0 + +Host: Gentoo amd64 + +When excluding a file that already exists on the backup, the file is not removed from the last backup, but it is kept, without being updated. + +The file excluded should not appear on generations created with the exclude filter. + +How to reproduce: + + ############################################# + #!/bin/sh + + rm -rf /tmp/test + mkdir -p /tmp/test/backup + touch /tmp/test/backup/file1 + touch /tmp/test/backup/file2 + + # 1st run, exclude file1 + # only file2 is present on backup + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup + obnam -r /tmp/test/repo ls + + # 2nd run + # none excluded, both files present on backup + obnam -r /tmp/test/repo backup /tmp/test/backup + obnam -r /tmp/test/repo ls + + # 3rd run, modify files and backup with exclude=file1 + # both files present, file2 is updated on backup, file1 is neither updated, or removed + echo "000" > /tmp/test/backup/file1 + echo "000" > /tmp/test/backup/file2 + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup + obnam -r /tmp/test/repo ls + + cd / + rm -rf /tmp/test/backup + obnam -r /tmp/test/repo restore /tmp/test/backup + ls -lha /tmp/test/backup + ############################################# + +OUTPUT: + + + rm -rf /tmp/test + + mkdir -p /tmp/test/backup + + touch /tmp/test/backup/file1 + + touch /tmp/test/backup/file2 + + # 1st run, exclude file1 + # only file2 is present on backup + + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup + Backed up 2 files, uploaded 0.0 B in 0s at 0.0 B/s average speed + + obnam -r /tmp/test/repo ls + Generation 2 (2012-06-20 09:23:41 - 2012-06-20 09:23:41) + drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / + drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp + drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test + drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup + -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2 + + # 2nd run + # none excluded, both files present on backup + + obnam -r /tmp/test/repo backup /tmp/test/backup + Backed up 3 files, uploaded 0.0 B in 0s at 0.0 B/s average speed + + obnam -r /tmp/test/repo ls + Generation 5 (2012-06-20 09:23:41 - 2012-06-20 09:23:41) + drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / + drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp + drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test + drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup + -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1 + -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2 + + # 3rd run, modify files and backup with exclude=file1 + # both files present, file2 is updated on backup, file1 is neither updated, or removed + + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup + Backed up 2 files, uploaded 4.0 B in 0s at 13.8 B/s average speed + + obnam -r /tmp/test/repo ls + Generation 8 (2012-06-20 09:23:42 - 2012-06-20 09:23:42) + drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / + drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp + drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test + drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup + -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1 + -rw-rw---- 1 jordi jordi 4 2012-06-20 07:23:41 /tmp/test/backup/file2 + + # when restoring, they appear, the new file2, and the old file1. + + rm -rf /tmp/test/backup + + cd / + + obnam -r /tmp/test/repo restore /tmp/test/backup + --h--m--s 4 files 4 B (100 %) 1.4 KiB/s /tmp/test/backup + + ls -lha /tmp/test/backup + total 4,0K + drwxrwx--- 2 jordi jordi 80 jun 20 09:23 . + drwxrwx--- 4 jordi jordi 80 jun 20 09:23 .. + -rw-rw---- 1 jordi jordi 0 jun 20 09:23 file1 + -rw-rw---- 1 jordi jordi 4 jun 20 09:23 file2 + + +--- + +A workaround can be found in +<http://listmaster.pepperfish.net/pipermail/obnam-flarn.net/2013-May/001929.html>. + + +--- + +This is now fixed in git master. [[done]] --liw diff --git a/bugs/fd-leak.mdwn b/bugs/fd-leak.mdwn new file mode 100644 index 0000000..1abfbd0 --- /dev/null +++ b/bugs/fd-leak.mdwn @@ -0,0 +1,4 @@ +Ben Kelly reported on August 31, 2012, that he's seeing crashes +due to file descriptor leaks. See list mail archive for logs and +suggested patches. I have not been able to reproduce this, however. +--liw diff --git a/bugs/filename-hash-collisions.mdwn b/bugs/filename-hash-collisions.mdwn new file mode 100644 index 0000000..e114f20 --- /dev/null +++ b/bugs/filename-hash-collisions.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-1.0-blocker]] + +Obnam does not handle hash collisions in filenames. If two different +filenames have the same hash, Obnam assumes they're the same filename. + +This should be possible to test by using a very weak filename hash +function. + +--liw + +[[done]] in bzr. diff --git a/bugs/force-lock_currently_doesn__39__t_work.mdwn b/bugs/force-lock_currently_doesn__39__t_work.mdwn new file mode 100644 index 0000000..6c0bf7a --- /dev/null +++ b/bugs/force-lock_currently_doesn__39__t_work.mdwn @@ -0,0 +1,47 @@ +[[!tag obnam-wishlist]] + +obnam force-lock currently doesn't work. As a workaround, remove the lockfiles (all files named lock inside the repository) by hand. + + find [repository path] -name lock -exec rm '{}' \; + +--weinzwang + +--- + +I confirm that I see this too. This bug exists because I changed how Obnam uses locks: it now locks +each directory properly, instead of just the per-client directory. However, I haven't fixed "force-lock" to +deal with other locks, so now it's not possible to force the locks for other directories than the per-client +one. This is awkward. + +To fix this, Obnam needs to know that it can safely remove the locks. There's two cases: + +* the lock was created by some other client; in this case, the user (not Obnam automatically) + needs to decide if it is safe to remove the lock: just running "obnam force-lock" should not + do that, instead the user should provide an option like "--really-force-locks" or something +* the lock was created by the same client, i.e., Obnam running on the same host; in this case, + if the Obnam process no longer exists, the lock can be safely removed, otherwise the locks + should not be removed (again, unless "--really-force-locks" is used) + +To implement this, we need Obnam to store the hostname and process id of the Obnam +instance that created the lock, preferably in a way that does not leak sensitive information +easily (don't store the client name in cleartext, but the md5sum of it, or something). + +--liw + +--- + +As of 0.27, force-locks unconditionally breaks locks, but the lock files will +contain sufficient information to allow us to be more intelligent about the +breaking of locks in the future. + +--kinnison + +-- + +This is not good enough -- I'd like obnam to be able to break locks more kindly -- +but it's good enough for 1.0, I think, so removing the blocker tag. --liw + +--- + +Making the lock breaking more benign and intelligent is a +wishlist. Adding tag. --liw diff --git a/bugs/forget-is-too-slow.mdwn b/bugs/forget-is-too-slow.mdwn new file mode 100644 index 0000000..2b37342 --- /dev/null +++ b/bugs/forget-is-too-slow.mdwn @@ -0,0 +1,8 @@ +`obnam forget` is currently too slow to be usable. It might be much +faster if it removed all data for all generations to be removed at +once, rather than generation-by-generation. Needs experimentation and +benchmarking. + +--liw + +[[done]] diff --git a/bugs/forget-progress-reporting.mdwn b/bugs/forget-progress-reporting.mdwn new file mode 100644 index 0000000..b063f88 --- /dev/null +++ b/bugs/forget-progress-reporting.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +It would be nice if "obnam forget" could have more detailed progress +reporting than at the granularity of one generation. --liw + +[[done]] this is semi-duplicate, and in any case a wishlist bug that's +been open for a very long time. --liw diff --git a/bugs/forget_progress_reporting_broken.mdwn b/bugs/forget_progress_reporting_broken.mdwn new file mode 100644 index 0000000..39b2392 --- /dev/null +++ b/bugs/forget_progress_reporting_broken.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +The progress reporting for "obnam forget" seems to be broken. When +forgetting more than two generations, the progress display is stuck at + + forgetting generations: 2/64 done + +until the very end. It's updated to 64/64 right before finishing. + +-- weinzwang diff --git a/bugs/fsck-does-not-lock.mdwn b/bugs/fsck-does-not-lock.mdwn new file mode 100644 index 0000000..7186f2b --- /dev/null +++ b/bugs/fsck-does-not-lock.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-1.0-blocker]] + +`obnam fsck` does not lock anything in the repository, but it should. +--liw + +[[done]] in bzr diff --git a/bugs/fsck-should-fix-refcount-errors.mdwn b/bugs/fsck-should-fix-refcount-errors.mdwn new file mode 100644 index 0000000..338a324 --- /dev/null +++ b/bugs/fsck-should-fix-refcount-errors.mdwn @@ -0,0 +1,6 @@ +obnam fsck needs to be able to fix refcount errors; larch needs to be able +to fix this + +-- + +[[done]] --liw diff --git a/bugs/fsck-should-not-require-user-to-be-known-client.mdwn b/bugs/fsck-should-not-require-user-to-be-known-client.mdwn new file mode 100644 index 0000000..22187c6 --- /dev/null +++ b/bugs/fsck-should-not-require-user-to-be-known-client.mdwn @@ -0,0 +1,11 @@ +obnam fsck should perhaps not require user to be a known client + +--- + + liw@havelock$ ./obnam -r t.repo --client=nope fsck + Checking 57/57: extra chunks + liw@havelock$ ./obnam -r t.repo clients + havelock + liw@havelock$ + +[[done]] --liw diff --git a/bugs/fsck-should-reconstruct-chunk-btrees.mdwn b/bugs/fsck-should-reconstruct-chunk-btrees.mdwn new file mode 100644 index 0000000..77e68d2 --- /dev/null +++ b/bugs/fsck-should-reconstruct-chunk-btrees.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +fsck has all the necessary information it needs to reconstruct the +chunksums and chunklist B-trees in a repository. It should do so, +at least when requested. + +--liw + +[[done]] ancient wishlist bug --liw diff --git a/bugs/fsck-should-remove-extra-files.mdwn b/bugs/fsck-should-remove-extra-files.mdwn new file mode 100644 index 0000000..1c4e9de --- /dev/null +++ b/bugs/fsck-should-remove-extra-files.mdwn @@ -0,0 +1,4 @@ +[[!tag obnam-wishlist]] + +Make obnam fsck remove extraneous files (e.g., tmp*). +--liw diff --git a/bugs/fsck-workitem-unification.mdwn b/bugs/fsck-workitem-unification.mdwn new file mode 100644 index 0000000..259aeda --- /dev/null +++ b/bugs/fsck-workitem-unification.mdwn @@ -0,0 +1,5 @@ +Combine larch and obnam fsck workitems. Merge the two current WorkItem +classes, put them into larch, and adapt both code bases accordingly. +--liw + +[[done]] diff --git a/bugs/fsck_gives_warnings_for_simple_cases.mdwn b/bugs/fsck_gives_warnings_for_simple_cases.mdwn new file mode 100644 index 0000000..0b0889b --- /dev/null +++ b/bugs/fsck_gives_warnings_for_simple_cases.mdwn @@ -0,0 +1,21 @@ +The fsck command gives warnings in some cases. I have one case, where it is easy to reproduce: + + host:/dev/shm% mkdir obnam 0 + host:/dev/shm% obnam -r /dev/shm/obnam backup /dev/shm/0 + Backed up 1 files, uploaded 0.0 B in 0s at 0.0 B/s average speed + host:/dev/shm% obnam -r /dev/shm/obnam fsck + warning: node 1: node must have keys + warning: node 1: node must have keys + Checking 15/14: extra chunks + + Version used: debian/stable/backports 1.0-1~bpo60+1 + +I guess this is a minor issue, but still would be nice, if fsck would be mostly calm. + +--- + +I confirm that I can see the same error. --liw + +-- + +[[done]] --liw diff --git a/bugs/fsck_needs_too_much_memory.mdwn b/bugs/fsck_needs_too_much_memory.mdwn new file mode 100644 index 0000000..cf2e147 --- /dev/null +++ b/bugs/fsck_needs_too_much_memory.mdwn @@ -0,0 +1,18 @@ +[[!tag obnam-1.0-blocker]] + +obnam fsck's memory needs are so big I'm not able to check my repo: + + [6402316.939346] Out of memory: Kill process 18036 (obnam) score 906 or sacrifice child + [6402316.939685] Killed process 18036 (obnam) total-vm:2754908kB, anon-rss:1876312kB, file-rss:640kB + +--weinzwang + +I've pushed some optimizations in bzr, and they'll end up in the next release. +Could you check, before I release, whether that was enough? I don't have access +to a machine for a while which has enough disk space to test this properly. +--liw + +I'm happy to report that with obnam from bzr, my repo can now be checked without +problems. Props to you liw! --weinzwang + +Marking bug as [[done]]. Thanks! --liw diff --git a/bugs/fsck_progress_counts_wrong.mdwn b/bugs/fsck_progress_counts_wrong.mdwn new file mode 100644 index 0000000..092b08b --- /dev/null +++ b/bugs/fsck_progress_counts_wrong.mdwn @@ -0,0 +1,10 @@ +Reported-By: Neal Becker + + obnam fsck + Checking 5/4: client list + +--liw + +--- + +[[done]] --liw diff --git a/bugs/fstype-include-exclude-mdwn b/bugs/fstype-include-exclude-mdwn new file mode 100644 index 0000000..837f36e --- /dev/null +++ b/bugs/fstype-include-exclude-mdwn @@ -0,0 +1,13 @@ +Implement --exclude-fstype and --include-fstype for Obnam. 2011-08 +whenever obnam backup is crossing a filesystem boundary (mount point), +it looks up the type of the filesystem beyond the boundary, and if it +matches an excluded filesystem type, but does not match an included +one, then it won't recurse beyond the mount point (it will back up +the actual directory, using metadata of the root of the mounted +filesystem) + +- in backup_plugin.py:can_be_backed_up, add tests after the test for + one-file-system + +--liw + diff --git a/bugs/fuse-without-generations.mdwn b/bugs/fuse-without-generations.mdwn new file mode 100644 index 0000000..0f95a11 --- /dev/null +++ b/bugs/fuse-without-generations.mdwn @@ -0,0 +1,4 @@ +It seems `obnam mount` (the FUSE plugin) can't handle a client without +non-checkpoint generations. This is unfortunate, even if it is fairly +unlikely to happen. Should be easy enough to fix. +--liw diff --git a/bugs/fuse.mdwn b/bugs/fuse.mdwn new file mode 100644 index 0000000..62a0ee9 --- /dev/null +++ b/bugs/fuse.mdwn @@ -0,0 +1,16 @@ +[[!tag obnam-wishlist]] + +It would be good to have a FUSE filesystem for restoring data. --liw + +--- + +I didn't get very far today, but here's a draft: <http://p.sipsolutions.net/d9720a80c9cc5e99.txt>, there's still a lot of XXX in there but maybe it helps somebody get started. I probably won't have much time to work on it. + +You start it with "obnam fuse /mountpoint", or you can do "obnam fuse -- -h" to get fuse help. Need the "--" so obnam doesn't parse the "-h" itself ... + +--Johannes + +---- + +There is a FUSE plugin in Obnam now. --liw +[[done]] diff --git a/bugs/generation-descriptions.mdwn b/bugs/generation-descriptions.mdwn new file mode 100644 index 0000000..09e8cc7 --- /dev/null +++ b/bugs/generation-descriptions.mdwn @@ -0,0 +1,17 @@ +[[!tag obnam-wishlist]] + +S.B. suggests that backup generations have an optional description. + +> 2. Named generations -- There are certain generations that are more +> important than others. Some are automatically created by Obnam itself, +> some are routinely scheduled, and some were explicitly created. For +> example, I always run Obam immediately before traveling with my laptop +> in case it gets stolen or broken. The same goes for backups before +> major system upgrades. It would be nice to have something +> approximately analogous to the Windows "restore point" functionality, +> which has a description field. Sometimes they are only automatically +> created system checkpoints. But if the user explicitly creates a new +> restore point, he can add the description "before traveling to Europe" +> or "before upgrading OS" or whatever. Similarly, the automatic backup +> script could be programmed to label it as "cron backup". + diff --git a/bugs/generation-locking.mdwn b/bugs/generation-locking.mdwn new file mode 100644 index 0000000..50df37c --- /dev/null +++ b/bugs/generation-locking.mdwn @@ -0,0 +1,12 @@ +[[!tag obnam-wishlist]] + +S.B. suggests that generations could be tagged so they aren't automatically +deleted. + +> 3. Unforgettable generations -- In scenarios similar to the above, I +> would also find it useful to be able to mark certain important +> generations as "unforgettable". That way, when I run an automatic time +> based forget command, I can be sure that it will preserve certain +> milestone generations, even if they weren't the last generation of the +> month or the week or the day or whatever. + diff --git a/bugs/generation-time-stamps-wrong.mdwn b/bugs/generation-time-stamps-wrong.mdwn new file mode 100644 index 0000000..ab9199d --- /dev/null +++ b/bugs/generation-time-stamps-wrong.mdwn @@ -0,0 +1,9 @@ +From Anders Wirzenius: + + Generation 8 (1970-01-01 02:00:00 - 1970-01-01 02:00:00) <------ + ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ + +The timestamps are clearly wrong. It's as if Obnam is storing a zero +for the timestamps (zero UTC would be 02:00 Finnish time). --liw + +[[done]] diff --git a/bugs/glob-exclusion.mdwn b/bugs/glob-exclusion.mdwn new file mode 100644 index 0000000..a3e96a2 --- /dev/null +++ b/bugs/glob-exclusion.mdwn @@ -0,0 +1,12 @@ +[[!tag obnam-wishlist]] + +From Enrico: A way to exclude files with globs instead of regular +expressions would be nice sometimes. `--exclude-glob="*.o"` or +something like that. + +--- + +Having thought of this: I'd rather not do this myself. Globs are not +powerful enough for my own use, and I'd rather not make two kinds +of patterns. I'd be happy to accept a patch for this, but keeping the +wishlist bug open isn't going to make that happen. [[done]] --liw diff --git a/bugs/gpg-options.mdwn b/bugs/gpg-options.mdwn new file mode 100644 index 0000000..91cba84 --- /dev/null +++ b/bugs/gpg-options.mdwn @@ -0,0 +1,11 @@ +Obnam should have a `gnupg-home` setting to set the GnuPG home directory +(`~/.gnupg`), or possibly a `gnupg-option` string list setting to pass +any options to GnuPG. +--liw + +The home directory can be set with `$GNUPGHOME` environment variable. +Additional settings can be set in the config in that directory. An +Obnam setting seems unnecessary, therefore. I'll add it if I am told +of a specific example of why it's useful. --liw + +[[done]] diff --git a/bugs/gpg-passphrase.mdwn b/bugs/gpg-passphrase.mdwn new file mode 100644 index 0000000..e749ae8 --- /dev/null +++ b/bugs/gpg-passphrase.mdwn @@ -0,0 +1,40 @@ +[[!tag obnam-wishlist]] +Obnam should, optionally, ask for a gpg passphrase, for the key specified +with --encrypt-with, so that a user without a gpg agent will be able to +do encrypted backups. Obnam should read the passphrase if its +ask-passphrase setting is true, and it has access to a terminal. +It should not have a setting for the passphrase itself, just for +reading it from a terminal (just so that people who don't know +better don't put their passphrase in a config file or similar). + +Those running obnam from cron will need to have a passphraseless +key, since there's no way to give obnam a passphrase in that case, +without storing it in the crontab or a config file, and then it's +no better than not having a passphrase. + +See [Debian +bug #649769](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649769). + +--liw + +From my understanding, having a symmetric passphrase stored in a config file is not useless at all. My purpose in encrypting the backup data is to prevent the remote server from having my data in plain-view; or if I back it up to an external drive, I wouldn't want it to be accessible to anyone who picks it up. But if someone gains access to my config file, he'll have direct access to all of my data anyway--he wouldn't need to access my backups. + +If I use a passphrase, then if my house burns down and I lose everything, I can get a new computer and download my data and decrypt it with my passphrase--which is long enough to be unfeasible to crack, yet completely memorized by me. + +If I use a key, then if my house burns down and I don't have a working copy of my key outside my house, my backups are totally useless, and I really HAVE lost everything. (Sure, I should take precautions to keep from losing my key--but things happen.) + +--Adam + +It's possible to get obnam to request a passphrase when running from cron: + +1. Ensure 'use-agent' is enabled in ~/.gnupg/gpg.conf. +2. Ensure the gpg-agent is running, and GPG_AGENT_INFO is set in your regular environment. Note that if obnam already asks for an enccryption passphrase when run normally, then 1 & 2 are already correctly set. +3. Ensure the environment obnam is called from in cron is exporting GPG_AGENT_INFO correctly. This means you must set and export the GPG_AGENT_INFO environment variable in your cron script. gpg writes this information to ~/.gnupg/gpg-agent-info-$(hostname), so in your cron script you must have: + + source "~/.gnupg/gpg-agent-info-$(hostname)" && export GPG_AGENT_INFO + +Then call obnam as normal. + +This will only work on a desktop system where there is someone to notice that a pinentry window has popped up. However it looks like there may be a way to forward the gpg-agent socket over ssh, and thus run obnam with encryption from cron on a headless remote machine (<a href="http://superuser.com/questions/161973/how-can-i-forward-a-gpg-key-via-ssh-agent">See here</a>). You'd probably have to store the private key on the remote machine though.. so not sure how useful that would be. + +--Scott diff --git a/bugs/help-output-groups.mdwn b/bugs/help-output-groups.mdwn new file mode 100644 index 0000000..b5f4e2c --- /dev/null +++ b/bugs/help-output-groups.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +It would be nice of `obnam --help` would group the large list of options +in some sensible way. --liw + +[[done]] diff --git a/bugs/helper-script-for-bug-report-data-gathering.mdwn b/bugs/helper-script-for-bug-report-data-gathering.mdwn new file mode 100644 index 0000000..6f3dfa0 --- /dev/null +++ b/bugs/helper-script-for-bug-report-data-gathering.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-wishlist]] + +Would be nice to have a +script to gather information for reporting a bug about obnam? + +- Python version +- versions of obnam, larch, cliapp, and any other of my projects + obnam uses +- config dump +- is _obnam available? +- mount point, file system types (/proc/mounts on linux) +- linux kernel version, or other uname -a output + +[[done]] -- no script exists, but keeping the wishlist bug open isn't +helping it. I'll write one when the pain becomes too much. --liw diff --git a/bugs/inefficient-metadata.mdwn b/bugs/inefficient-metadata.mdwn new file mode 100644 index 0000000..0a953bf --- /dev/null +++ b/bugs/inefficient-metadata.mdwn @@ -0,0 +1,12 @@ +Obnam seems to be storing metadata quite inefficiently. I did this: + +* create a directory tree with about 1.3 million empty files +* back that up + +The data is about 500 megs (directory entries); the repository is +about 82 gigabytes. + +Where is the space used? Can we store it more efficiently and if we +do, does that have an impact on runtime? + +--liw diff --git a/bugs/insufficient-locking.mdwn b/bugs/insufficient-locking.mdwn new file mode 100644 index 0000000..4726d1f --- /dev/null +++ b/bugs/insufficient-locking.mdwn @@ -0,0 +1,5 @@ +Obnam does not currently do locking properly. It locks the per-client +directory, but not the shared directories. --liw + + +[[done]] diff --git a/bugs/integer-too-big.mdwn b/bugs/integer-too-big.mdwn new file mode 100644 index 0000000..9162ad5 --- /dev/null +++ b/bugs/integer-too-big.mdwn @@ -0,0 +1,36 @@ +Fix "long too large to convert to int" problem found by Jo. + + Traceback (most recent call last): + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 135, in _run + self.process_args(args) + File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 116, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 275, in process_args + method(args[1:]) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 80, in backup + self.backup_roots(roots) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 136, in backup_roots + self.backup_metadata(pathname, metadata) + File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 250, in backup_metadata + self.repo.create(pathname, metadata) + File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 509, in create + encoded = obnamlib.encode_metadata(metadata) + File "/usr/lib/python2.6/dist-packages/obnamlib/metadata.py", line 200, in encode_metadata + len(metadata.md5 or '')) + error: long too large to convert to int + +- hypothesis: overflows int on the way to the struct, on 32-bit +- wrote script to test hypothesis, but proved it wrong: the limit where + struct gives error is 2**64-1, as is proper, so the problem is something + else +- talk to directhex to extract more information, possibly add logging to + pinpoint the value + +--liw + + +Can't seem to reproduce this, but it was hopefully related to things +now replaced by our own lstat wrapper. If you see this, re-open and +add a way to reproduce. --liw + +[[done]] diff --git a/bugs/keep-bugs-in-ikiwiki.mdwn b/bugs/keep-bugs-in-ikiwiki.mdwn new file mode 100644 index 0000000..5ed976f --- /dev/null +++ b/bugs/keep-bugs-in-ikiwiki.mdwn @@ -0,0 +1,4 @@ +I want to move my bug tracking from SD to ikiwiki. --liw + +[[done]] + diff --git a/bugs/keep-chunk-dirs-locked-less.mdwn b/bugs/keep-chunk-dirs-locked-less.mdwn new file mode 100644 index 0000000..c0e00fd --- /dev/null +++ b/bugs/keep-chunk-dirs-locked-less.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +For better performance, it would be good if Obnam could keep the shared +chunks B-trees locked for shorter periods of time. This can be implemented +by only updating them at the end of the backup generation (including +checkpoints), rather than updating along the way when making a backup. +--liw + +[[done]] diff --git a/bugs/keep-hours-buggy.mdwn b/bugs/keep-hours-buggy.mdwn new file mode 100644 index 0000000..f886c4a --- /dev/null +++ b/bugs/keep-hours-buggy.mdwn @@ -0,0 +1,14 @@ +From mail: + +> I think there is problem in keep policy, +> see attached file, +> --keep=Nh always do same as --keep=Nd + +Seems to be true. Need cmdtest test case for reproduction. + +--- + +Actually, this seems to be mainly a documentation problem. Or an +indication that the system is too complicated, since so many people +understand it wrong. But [[done]], though I'll try to improve the +explanation in the manual. --liw diff --git a/bugs/keep-per-client-lock-across-snapshots.mdwn b/bugs/keep-per-client-lock-across-snapshots.mdwn new file mode 100644 index 0000000..02ffa1c --- /dev/null +++ b/bugs/keep-per-client-lock-across-snapshots.mdwn @@ -0,0 +1,7 @@ +Obnam should not release the per-client lock after doing a snapshot. +It should release the shared toplevel locks, though. --liw + +-- + +I have forgotten why this would be useful. If I remember it again, I'll +re-open. For now, [[done]]. --liw diff --git a/bugs/keeps-dev-urandom-open.mdwn b/bugs/keeps-dev-urandom-open.mdwn new file mode 100644 index 0000000..1a2b43f --- /dev/null +++ b/bugs/keeps-dev-urandom-open.mdwn @@ -0,0 +1,6 @@ +why does obnam keep /dev/urandom open, even though nothing should be +using it? it's open whether using encryption or not + +-- + +This seems to be due to the stdlib. Ignoring. [[done]] --liw diff --git a/bugs/key-managements.mdwn b/bugs/key-managements.mdwn new file mode 100644 index 0000000..7dd53e1 --- /dev/null +++ b/bugs/key-managements.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +There needs to be tools and documentation for key managment +with Obnam. + +How does not one replace one's key, or subkey, when it expires? diff --git a/bugs/lacks-compression-plugin.mdwn b/bugs/lacks-compression-plugin.mdwn new file mode 100644 index 0000000..754b031 --- /dev/null +++ b/bugs/lacks-compression-plugin.mdwn @@ -0,0 +1,15 @@ +Obnam could do with a plugin that just compresses data, without +encrypting it. Encryption might be unwanted, or unnecessary, for +example when backing to a USB drive that uses whole-disk encryption. +However, saving space by compressing data with gzip or xz is still +a reasonable thing to do. + +The compression plugin would be similar to the existing encryption +plugin, but be triggered by a different setting. + +* `compress-with`: program to compress files with; + default to gzip, but can be any program + +--liw + +[[done]] diff --git a/bugs/larch-journal-processing-progress-reporting.mdwn b/bugs/larch-journal-processing-progress-reporting.mdwn new file mode 100644 index 0000000..8b494ce --- /dev/null +++ b/bugs/larch-journal-processing-progress-reporting.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +When larch is processing a journal (committing or deleting at +startup, committing at end), obnam should be showing useful progress +reporting for that. diff --git a/bugs/live-data-backup-restore-benchmark-failure.mdwn b/bugs/live-data-backup-restore-benchmark-failure.mdwn new file mode 100644 index 0000000..7d12007 --- /dev/null +++ b/bugs/live-data-backup-restore-benchmark-failure.mdwn @@ -0,0 +1,10 @@ +Tried to run obnam benchmark with real data and verification on xander. + +- failed: no restored files +- see /home/liw/tmp/benchmark-with-real-data-failed-to-restore +- investigate why + +--liw + +I no longer have access to this machine, and cannot debug the issue +anymore. Meh. --liw [[done]] diff --git a/bugs/llistxattr-fail-fatal.mdwn b/bugs/llistxattr-fail-fatal.mdwn new file mode 100644 index 0000000..8f7973e --- /dev/null +++ b/bugs/llistxattr-fail-fatal.mdwn @@ -0,0 +1,14 @@ +When obnam backs up from .gvfs or another filesystem that does not +support the llistxattr system call it crashes with this traceback: + + > File "/usr/lib/python2.7/dist-packages/obnamlib/metadata.py", line 130, in get_xattrs_as_blob + > names = fs.llistxattr(filename) + > File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 165, in llistxattr + > raise OSError((ret, os.strerror(ret), filename)) + > OSError: (95, 'Operation not supported', '/home/ben/.gvfs') + +Obnam should report the error, but continue the backup. + +-- + +[[done]] --liw diff --git a/bugs/local-temp-cache.mdwn b/bugs/local-temp-cache.mdwn new file mode 100644 index 0000000..d74349e --- /dev/null +++ b/bugs/local-temp-cache.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655094> for details. + +[[done]] no point in keeping this open in multiple places. --liw diff --git a/bugs/lock-key-in-ram.mdwn b/bugs/lock-key-in-ram.mdwn new file mode 100644 index 0000000..969b0c7 --- /dev/null +++ b/bugs/lock-key-in-ram.mdwn @@ -0,0 +1,17 @@ +[[!tag obnam-wishlist]] + +Joey asked: + +<joeyh> have you done anything in obnam to deal with it needing to keep +the symmetric key, decrypted, in RAM? + +<joeyh> yeah, it's tough. probably could be avoided by having gpg decrypt +the passphrase and pipe it to the encrypting gpg .. but then gpg would +constantly be using the public key + +It might be possible to have a C extension that holds the symmetric +key, locks it into RAM, and feeds it to gpg whenever necessary, +via a file descriptor. + +--liw + diff --git a/bugs/lock-notification.mdwn b/bugs/lock-notification.mdwn new file mode 100644 index 0000000..7257e33 --- /dev/null +++ b/bugs/lock-notification.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-wishlist]] + +If obnam is waiting for a lock to expire, it should say so, maybe with a hint to use force-lock to remove if obnam crashed/was killed (otherwise the user is left wondering what obnam is doing). + +--- + +This will be fixed with the next version of [[ttystatus]], which has a functioning `flush` method. --liw + +-- + +[[done]] --liw diff --git a/bugs/lock-timeout-error-does-not-identify-lock.mdwn b/bugs/lock-timeout-error-does-not-identify-lock.mdwn new file mode 100644 index 0000000..aecc43f --- /dev/null +++ b/bugs/lock-timeout-error-does-not-identify-lock.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +When obnam gives an error for a lock timeout, it should specify what +lock file failed. --liw + +[[done]] in bzr diff --git a/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn b/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn new file mode 100644 index 0000000..5cd6409 --- /dev/null +++ b/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn @@ -0,0 +1,5 @@ +It would be nice to see, which files are being backed up when reducing the loglevel from DEBUG to INFO + +--- + +That seems reasonable. [[done]] --liw diff --git a/bugs/ls-output-formats.mdnw b/bugs/ls-output-formats.mdnw new file mode 100644 index 0000000..281d436 --- /dev/null +++ b/bugs/ls-output-formats.mdnw @@ -0,0 +1,3 @@ +[[!tag obnam-wishlist]] + +See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655229> for info. diff --git a/bugs/memory-profiling-everywhere.mdwn b/bugs/memory-profiling-everywhere.mdwn new file mode 100644 index 0000000..7b17e56 --- /dev/null +++ b/bugs/memory-profiling-everywhere.mdwn @@ -0,0 +1,4 @@ +All Obnam operations should be calling the memory profiling operations, +so that all operations can be profiled when needed. The profiling functions +should go into cliapp, however. --liw +[[done]] diff --git a/bugs/missing-progress-reporting.mdwn b/bugs/missing-progress-reporting.mdwn new file mode 100644 index 0000000..5979b17 --- /dev/null +++ b/bugs/missing-progress-reporting.mdwn @@ -0,0 +1,18 @@ +The following operations should have progress reporting added: + + obnam [options] forget + obnam [options] fsck + obnam [options] restore + obnam [options] verify + +--liw + +restore and fsck already have progress reporting. +forget and verify need it. +--liw + +verify now has progress reporting. forget still needs it. --liw + +forget now has progress reporting too. --liw + +[[done]] diff --git a/bugs/missing-v6-restore-test.mdwn b/bugs/missing-v6-restore-test.mdwn new file mode 100644 index 0000000..fefc0cb --- /dev/null +++ b/bugs/missing-v6-restore-test.mdwn @@ -0,0 +1,3 @@ +Add an Obnam cmdtest to verify v6 repos can be restored from. + +[[done]] in git master. --liw diff --git a/bugs/more-rsync-like.mdwn b/bugs/more-rsync-like.mdwn new file mode 100644 index 0000000..fc2e9f1 --- /dev/null +++ b/bugs/more-rsync-like.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Moving forward in small increments than individual chunks would allow +more rsync-like behavior, and more chance of finding duplicate data. +This might be worthwhile, for some users, some of the time. It should +be configurable, though, since it's also potentially going to be a +big performance problem. --liw + +[[done]] ancient wishlist bug --liw diff --git a/bugs/multiple-checksums.mdwn b/bugs/multiple-checksums.mdwn new file mode 100644 index 0000000..5585681 --- /dev/null +++ b/bugs/multiple-checksums.mdwn @@ -0,0 +1,27 @@ +[[!tag obnam-wishlist]] + +From Joey Hess: + +> My take on this is that, by choosing to use a tool that uses hashes, I +> am giving up (near-)absolute certainty for speed, or space, or whatever. +> So it's important that the hash type be good at collision resistance (for +> example, no two likely filenames should hash the same; "/etc/passwd" +> should only tend to collide with blobs that are very unlike a filename). +> It's also important that the tool be upfront about using hashes, and +> about what hash it uses. And if it's not designed to allow swapping the +> hash out when it gets broken, I will trust it less (hello git). + +Ah, the replacement of hash functions is an interesting problem. + +For pathnames, it's not at all important, I think, except perhaps for +performance, since pathnames will be compared byte-by-byte instead of +by hashes. + +For file data, replacing is easy, if one is willing to back up everything +from scratch. Supporting several hashes in the same backup store is a +little bit more work, but not a whole lot: instead of having just one +tree for mapping checksums to chunk identifiers, one would have one per +checksum algorithm. + +--liw + diff --git a/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn b/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn new file mode 100644 index 0000000..6201a06 --- /dev/null +++ b/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn @@ -0,0 +1,31 @@ +The documentation for "exclude" says that it could be used multiple times: + +>--exclude=EXCLUDE +> regular expression for pathnames to exclude from backup (can be used multiple times) + +It works when used multiple times on the command line. However, when using a configuration with: + + [config] + exclude = /foo + exclude = /bar + exclude = /baz + +...only the last item "/baz" is actually excluded from the backup. + +--- + +The documentation doesn't actually say this anywhere (oops, sorry), so you couldn't know: the right way of doing that in the config file is to use one exclude line and separate values with commas (spaces optional). + + [config] + exclude = /foo, /bar, /baz + +That should work. + +Using multiple exclude-lines in the config would be better, but Python's ConfigParser library doesn't really deal well with that. + +I'll mark this bug closed when the obnam manpage explains how to do this with config files. + +--liw + +It's now in the manual page in bzr and fix will be included in the next +release. [[done]] --liw diff --git a/bugs/multirepository.mdwn b/bugs/multirepository.mdwn new file mode 100644 index 0000000..84593e5 --- /dev/null +++ b/bugs/multirepository.mdwn @@ -0,0 +1,53 @@ +[[!tag obnam-wishlist]] + +Obnam should support multiple repositories, to be chosen at invocation time. + +- all repositories configured in config files +- nicknames for repositories so it's easy to choose +- --repository should accept nicknames +- choose many repositories for one run +- use all available repositories by default + +--liw + +--- +Could this also be achieved by running Obnam from a wrapper script that uses a different repository for each run? Could Obnam be run in parallel instances backing up the same data to different repos? Is that possible now? --*Adam* + +--- + +Adam, it can certainly be done by using wrapper scripts (I've been doing that), and while I haven't +actually tried it, there should be no problem with backing up to multiple repositories concurrently, +though you may need to fiddle with the configs so that they use different log files. --liw + +--- + +After some thinking, I think I don't want nicknames for repositories, I want "profiles".Here's a concrete +suggestion: + + [config] + encrypt-with = CAFEF00D + profile = all + log = /var/log/obnam/obnam-%(profile)s.log + + [profile "online"] + repository = sftp://liw@personal.backup.server.example.com/~/repo/ + use-if = ping -c1 personal.backup.server.example.com + + [profile "usb-drive"] + repository = /media/Drum/obnam-repo/ + use-if = test -d /media/Drum/obnam-repo/ + + [profile "at-work"] + repository = /mnt/backups/ + use-if = ping -c1 fs.work.example.com + pre-command = sudo mount /mnt/backups + post-command = sudo umount /mnt/backups + +- if --profile=all, then iterate automatically over all profiles +- otherwise, use only the chosen profiles +- some day: run some/all profiles in parallel in one obnam instance; initially, user may run parallel obnam instances +- log file should embed profile name somehow +- should profile be selected based on user too? that can be done with "use-if = test $USER = liw"; better support can be added later, if there's a need + +--liw + diff --git a/bugs/needs-porting-to-rhel-sl-centos.mdwn b/bugs/needs-porting-to-rhel-sl-centos.mdwn new file mode 100644 index 0000000..3959236 --- /dev/null +++ b/bugs/needs-porting-to-rhel-sl-centos.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-wishlist]] + +Obnam should work on RHEL, Scientific Linux, and CentOS, and other +popular distros such as Fedora, Ubuntu. Help porting these would +be welcome. --liw + + +There's a Fedora port, at least. Wishing isn't going make other ports happen, +though: users who want it on their platform, and are willing to do the work, is +what will make ports happen. Thus, keeping this bug open isn't useful. +[[done]] --liw diff --git a/bugs/no-chattr-support.mdwn b/bugs/no-chattr-support.mdwn new file mode 100644 index 0000000..b0381ef --- /dev/null +++ b/bugs/no-chattr-support.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Obnam does not support the ext2/3/4 chattr attributes. It should back them up and +set them on restore, when possible. + +In addition, it should support the d attribute to exclude +files from being backed up. + +--liw diff --git a/bugs/non-linux-file-types.mdwn b/bugs/non-linux-file-types.mdwn new file mode 100644 index 0000000..32349d7 --- /dev/null +++ b/bugs/non-linux-file-types.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Obnam should support non-linux file types: + +* <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html> +* <http://www.gnu.org/s/hello/manual/libc/Testing-File-Type.html> + +--liw + diff --git a/bugs/non-wishlist.mdwn b/bugs/non-wishlist.mdwn new file mode 100644 index 0000000..a117b0f --- /dev/null +++ b/bugs/non-wishlist.mdwn @@ -0,0 +1,11 @@ +[[!inline pages="page(obnam/bugs/*) and + !obnam/bugs/*/* and + !obnam/bugs/done and + !obnam/bugs/wishlist-only and + !obnam/bugs/performance-only and + !tagged(obnam-wishlist) and + !tagged(obnam-performance) and + !link(obnam/bugs/done)" + show=0 + postform=no]] + diff --git a/bugs/obnam-benchmark-seivot-branch.mdwn b/bugs/obnam-benchmark-seivot-branch.mdwn new file mode 100644 index 0000000..65fa976 --- /dev/null +++ b/bugs/obnam-benchmark-seivot-branch.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +The `--seivot-branch` option should not be required for +`./run-benchmark`. --liw + + +--- + +[[done]] now. --liw diff --git a/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn b/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn new file mode 100644 index 0000000..6d1d412 --- /dev/null +++ b/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn @@ -0,0 +1,13 @@ +force-lock in obnam 0.27-1.1 does not remove: + +./chunks/lock +or +./2275793072750507400/lock + +on a remote sftp:// repository. + +--- + +This is now fixed in bzr. --liw + +[[done]] diff --git a/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn b/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn new file mode 100644 index 0000000..29464d1 --- /dev/null +++ b/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +Currently, obnam fsck reports chunks that are unused: + + chunk 16541095925909528379 not used by anyone + +but doesn't do anything about it. There should be an option to +remove those unused chunks from the repository. --weinzwang diff --git a/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn b/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn new file mode 100644 index 0000000..2c13c25 --- /dev/null +++ b/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn @@ -0,0 +1,23 @@ +When I abort a backup using ctrl-c I get a stale lockfile on the sftp-Server + +Current (1.0) behavior: + + bart ~ # obnam backup --root=/etc + Enter passphrase for key '/home/mschiff/.ssh/id_dsa': + 00h00m03s 1286 files; 0 B (0 B/s) /etc/runit/runsvdir/all/getty-tty2 ^C Killed by signal 2. + (here I had hit ctrl-c) + bart ~ # obnam backup --root=/etc + Enter passphrase for key '/home/mschiff/.ssh/id_dsa': + ERROR: Lock timeout + bart ~ # + +--- + +Yeah. Obnam does not always clean up the lock files when it crashes. For some crashes it's not even possible. You can use the "obnam force-lock" command to remove the lock files, if you're sure they are not from another client using the same repository. --liw + +mschiff: You consider pressing ctrl-c to abort a backup a "crash" ? + +It's a user-induced crash, yes. It's one of the situations where Obnam could clean up, and it probably should. I believe there is a bug open about that already. --liw + + +[[done]] diff --git a/bugs/obnam_squeeze_package_dependencies.mdwn b/bugs/obnam_squeeze_package_dependencies.mdwn new file mode 100644 index 0000000..d15e65a --- /dev/null +++ b/bugs/obnam_squeeze_package_dependencies.mdwn @@ -0,0 +1,57 @@ +The obname package for Debian squeeze (version 0.27-1.1) has a dependency on python (>= 2.6.6-7~), but squeeze only provides python 2.6.6-3+squeeze6. + +--- + +I can't see this. From my code.liw.fi repository: + +> Depends: libc6 (>= 2.6), python2.7 | python2.6, python (>= 2.6.6-7~), python (<< 2.8), python-larch (>= 0.30~), python-ttystatus (>= 0.15), python-paramiko, python-tracing (>= 0.2), python-cliapp (>= 0.18) + +Can you point me at the .deb you found, and the URL you found it at? +I had some trouble building the 0.27 release, so I may have botched +it and put a bad version somewhere and if so, I'd like to fix that. Thanks. +--liw + +--- + +I'm referring to http://code.liw.fi/debian/pool/main/o/obnam/obnam_0.27-1.1_i386.deb, which is the package pulled in on squeeze. + +This is my /etc/apt/sources.list: + + deb http://ftp.debian.org/debian/ squeeze main + deb http://security.debian.org/ squeeze/updates main + deb http://ftp.debian.org/debian/ squeeze-updates main + deb http://code.liw.fi/debian squeeze main + +The real trouble is that there's no dedicated squeeze package for obnam 0.27. +There were dedicated squeeze packages for earlier versions of obnam (e.g. obnam_0.26-1~squeeze1_i386.deb). + +http://code.liw.fi/debian/dists/squeeze/main/binary-i386/Packages pulls in obname 0.27-1.1, which has a dependency on python (>= 2.6.6-7~), while Debian squeeze (6.0.4) only provides python 2.6.6-3+squeeze6: + + # apt-cache policy python obnam + python: + Installed: 2.6.6-3+squeeze6 + Candidate: 2.6.6-3+squeeze6 + Version table: + *** 2.6.6-3+squeeze6 0 + 990 http://ftp.de.debian.org/debian/ squeeze/main i386 Packages + 100 /var/lib/dpkg/status + obnam: + Installed: (none) + Candidate: 0.27-1.1 + Version table: + 0.27-1.1 0 + 500 http://code.liw.fi/debian/ squeeze/main i386 Packages + +Therefore, it's not possible to install obnam 0.27-1.1 on squeeze: + + The following packages have unmet dependencies: + obnam : Depends: python (>= 2.6.6-7~) but 2.6.6-3+squeeze6 is to be installed + E: Broken packages + +The fix would be to build a package for squeeze again, e.g. obnam_0.27-1.1~squeeze1_i386.deb. + +--flight + +--- + +Fixed with the 0.28 packages. [[done]] --flight diff --git a/bugs/obnammodule-malloc.mdwn b/bugs/obnammodule-malloc.mdwn new file mode 100644 index 0000000..731ccb2 --- /dev/null +++ b/bugs/obnammodule-malloc.mdwn @@ -0,0 +1,6 @@ +Review `_obnammodule.c`'s use of malloc: every return value should be +checked and if there's a failure, the process should abort. + +--- + +Fixed in git master. [[done]] --liw diff --git a/bugs/option_to_not_use_paramiko__63__.mdwn b/bugs/option_to_not_use_paramiko__63__.mdwn new file mode 100644 index 0000000..deabafe --- /dev/null +++ b/bugs/option_to_not_use_paramiko__63__.mdwn @@ -0,0 +1,11 @@ +Could we please have an option to force obnam to not use paramiko but instead openssh or libssh2-1? I'd like to get around the whole .ssh/config debacle and other nasty surprises I had when using duplicity, and keep using known and trusted software. + +--- + +Switching away from paramiko will require a fair amount of work. For other reasons, I'm going to be looking at doing so, but it's not just a matter of "using openssh". There is, for example, no Python API for openssh. There's other Python APIs for dealing with ssh, which I'm going to investigate, but it's not a one-evening-hack kind of thing. + +Also, given that ~/.ssh/config is a configuration file for openssh, and not all programs that use the ssh protocol, I think it's unrealistic to require every program to obey it. + +If and when I switch Obnam away from paramiko, it will not be an optional thing. Having to support multiple ways of doing ssh is unnecessary complexity, so I will only support one implementation at a time. Sorry. As such, I'll mark this bug as [[done]]. + +--liw diff --git a/bugs/oserror-stino.mdwn b/bugs/oserror-stino.mdwn new file mode 100644 index 0000000..bae2734 --- /dev/null +++ b/bugs/oserror-stino.mdwn @@ -0,0 +1,37 @@ +Robin reports: + + Traceback (most recent call last): + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 190, in _run + self.process_args(args) + File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 173, in process_args + cliapp.Application.process_args(self, args) + File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 537, in process_args + method(args[1:]) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 286, in backup + self.backup_roots(roots) + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 396, in backup_roots + for pathname, metadata in self.find_files(absroot): + File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 477, in find_files + for pathname, st in self.fs.scan_tree(root, ok=self.can_be_backed_up): + File "/usr/lib/python2.7/dist-packages/obnamlib/vfs.py", line 234, in scan_tree + pairs = self.listdir2(dirname) + File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 359, in listdir2 + return sorted(result, key=lambda st: st[1].st_ino) + File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 359, in <lambda> + return sorted(result, key=lambda st: st[1].st_ino) + AttributeError: 'exceptions.OSError' object has no attribute 'st_ino' + +Also: + + $ apt-cache policy obnam + obnam: + Geïnstalleerd: 1.3-1ubuntu1 + Kandidaat: 1.3-1ubuntu1 + Versietabel: + *** 1.3-1ubuntu1 0 + 500 http://ppa.launchpad.net/chris-bigballofwax/obnam-ppa/ubuntu/ quantal/main amd64 Packages + + +--- + +This is fixed in Obnam 1.4. [[done]] --liw diff --git a/bugs/performance-only.mdwn b/bugs/performance-only.mdwn new file mode 100644 index 0000000..6c20413 --- /dev/null +++ b/bugs/performance-only.mdwn @@ -0,0 +1,8 @@ +[[!inline pages="page(obnam/bugs/*) and + !obnam/bugs/*/* and + !obnam/bugs/done and + tagged(obnam-performance) and + !link(obnam/bugs/done)" + show=0 + postform=no]] + diff --git a/bugs/performance-with-many-hardlinks.mdwn b/bugs/performance-with-many-hardlinks.mdwn new file mode 100644 index 0000000..d30639a --- /dev/null +++ b/bugs/performance-with-many-hardlinks.mdwn @@ -0,0 +1,14 @@ +[[!tag obnam-performance]] + +How does Obnam perform if the live data has massive numbers of hardlinks? +Such hardlink trees are not rare, but it would be good to at least know +how Obnam handles them. Perform a benchmark comparing three cases: + +1. a very large number (one million?) of small files, all unique +2. same, but all files identical +3. same, but all files hardlinks to the same content + +Ideally, Obnam should perform about the same for all. --liw + + +[[done]] really ancient wishlist bug --liw diff --git a/bugs/precise-checkpoints.mdwn b/bugs/precise-checkpoints.mdwn new file mode 100644 index 0000000..be6623d --- /dev/null +++ b/bugs/precise-checkpoints.mdwn @@ -0,0 +1,5 @@ +Obnam should do checkpoint generations at precise times, not just between +files; this will result in partial files, but that's ok, the next backup +run will continue from the partial file. --liw + +[[done]] diff --git a/bugs/pretend-does-not-work-on-empty-repo.mdwn b/bugs/pretend-does-not-work-on-empty-repo.mdwn new file mode 100644 index 0000000..a7f58fa --- /dev/null +++ b/bugs/pretend-does-not-work-on-empty-repo.mdwn @@ -0,0 +1,5 @@ +`obnam backup --dry-run` doesn't work if there is no generation yet. +--liw + + +[[done]] --liw diff --git a/bugs/refactor-repository-class.mdwn b/bugs/refactor-repository-class.mdwn new file mode 100644 index 0000000..8363aa9 --- /dev/null +++ b/bugs/refactor-repository-class.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +The Repository class is quite big, and could perhaps be refactored to +become more manageable. +--liw + +I've merged changes to a replacement for the Repository class. +[[done]] +--liw diff --git a/bugs/regression_on_debian_squeeze_python_dependancy.mdwn b/bugs/regression_on_debian_squeeze_python_dependancy.mdwn new file mode 100644 index 0000000..3a3d679 --- /dev/null +++ b/bugs/regression_on_debian_squeeze_python_dependancy.mdwn @@ -0,0 +1,20 @@ +obnam package can not be installed into Debian Squeeze due dependency error: + + $ sudo apt-get install obnam + ... + The following packages have unmet dependencies: + obnam : Depends: python (>= 2.6.6-7~) but 2.6.6-3+squeeze7 is to be installed + E: Broken packages + +This is duplicate of this (once resolved) bug: +http://liw.fi/obnam/bugs/obnam_squeeze_package_dependencies/ + +I used this repo as instructed on tutorial: + deb http://code.liw.fi/debian squeeze main + +BR, +ilkka.tengvall iki.fi + +--- + +This is fixed now. [[done]] --liw diff --git a/bugs/remove-checkpoint-generations.mdwn b/bugs/remove-checkpoint-generations.mdwn new file mode 100644 index 0000000..7037a1b --- /dev/null +++ b/bugs/remove-checkpoint-generations.mdwn @@ -0,0 +1,14 @@ +After Obnam finishes a proper generation, and that is committed, +it should remove all the checkpoint generations from the current +backup run. + +Currently, removing generations is so slow that this is impractical, +but eventually it should be done. + +--liw + + +forget is now slow, so added code to remove checkpoints made during +a backup run, if it is successful. --liw + +[[done]] diff --git a/bugs/remove-client-not-in-manpage.mdwn b/bugs/remove-client-not-in-manpage.mdwn new file mode 100644 index 0000000..79b8161 --- /dev/null +++ b/bugs/remove-client-not-in-manpage.mdwn @@ -0,0 +1,4 @@ +"obnam remove-client" is not in the manpage. It also currently +doesn't work unless you use encryption. --liw + +[[done]] -- it is in the 1.6.1 manpage --liw diff --git a/bugs/remove-clients.mdwn b/bugs/remove-clients.mdwn new file mode 100644 index 0000000..6b0f85e --- /dev/null +++ b/bugs/remove-clients.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +Obnam needs a way to remove clients from the repository. The current +remove-client command just deals with encryption. + +Suggested-by: Daniel Silverstone diff --git a/bugs/rename-client.mdwn b/bugs/rename-client.mdwn new file mode 100644 index 0000000..7a39d02 --- /dev/null +++ b/bugs/rename-client.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +Obnam needs to be able to rename a client. + +[[done]] this is a duplicate bug. --liw diff --git a/bugs/rename-clients.mdwn b/bugs/rename-clients.mdwn new file mode 100644 index 0000000..192c0a3 --- /dev/null +++ b/bugs/rename-clients.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +Obnam needs a way to rename clients in the client list. + +Suggested-by: Daniel Silverstone diff --git a/bugs/repo-checksum-use-digest-not-hexdigest.mdwn b/bugs/repo-checksum-use-digest-not-hexdigest.mdwn new file mode 100644 index 0000000..4b3d1fb --- /dev/null +++ b/bugs/repo-checksum-use-digest-not-hexdigest.mdwn @@ -0,0 +1,7 @@ +Currently `obnamlib.Repository.checksum` returns a hex-encoded checksum. +This wastes space, since we could just use a raw byte string of the +digest. However, fixing this requires changing the repository format, +and it's not worth for the minor benefit. When the format needs to +change anyway, then this should be done, too. --liw + +[[done]] diff --git a/bugs/repo-upgrades-missing.mdwn b/bugs/repo-upgrades-missing.mdwn new file mode 100644 index 0000000..ea15e7f --- /dev/null +++ b/bugs/repo-upgrades-missing.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +Obnam needs to be able to upgrade a repository to the current format +version. --liw + +There is now a convert5to6 command and I'll add something for each +new repository format in the future. [[done]] --liw diff --git a/bugs/report-actual-transferred-bytes.mdwn b/bugs/report-actual-transferred-bytes.mdwn new file mode 100644 index 0000000..9387f15 --- /dev/null +++ b/bugs/report-actual-transferred-bytes.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +obnam backup should report actual transferred bytes in addition to live +data bytes, for reporting of overhead, and should calculate speed with +overhead too + +--- + +Fixed in git master now. --liw +[[done]] diff --git a/bugs/restartable-restore.mdwn b/bugs/restartable-restore.mdwn new file mode 100644 index 0000000..0e81839 --- /dev/null +++ b/bugs/restartable-restore.mdwn @@ -0,0 +1,7 @@ +[[!tag obnam-wishlist]] + +Restore should be able to continue a restore if it gets interrupted. +Restore should, perhaps optionally, skip files that already exist, +and continue partial files. --liw + +This is better done with the FUSE plugin and, say, rsync. [[done]] --liw diff --git a/bugs/restore-exclude-option.mdwn b/bugs/restore-exclude-option.mdwn new file mode 100644 index 0000000..6f1903e --- /dev/null +++ b/bugs/restore-exclude-option.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +It'd be handy if "obnam restore" could exclude parts of the +data, based on similar patterns as "obnam backup" does. --liw + +--- + +This is better done by using the FUSE plugin, now that it exists. +--liw +[[done]] diff --git a/bugs/restore-gid-when-possible.mdwn b/bugs/restore-gid-when-possible.mdwn new file mode 100644 index 0000000..4213b2f --- /dev/null +++ b/bugs/restore-gid-when-possible.mdwn @@ -0,0 +1,15 @@ +It is sometimes possible for a non-root to restore group ids (when obnam +process is in the group already). Should obnam try to do that? (Based +on suggestion from Gonéri Le Bouder.) --liw + +Seems like obnam should just try to restore whatever it has, and produce an error if it can't.-- Josh + +GNU Tar has the same issue. I should probably make Obnam have the same behavior as tar. I +expect that'll give people the fewest nasty surprises. Now I just need to figure out what tar does. --liw + +GNU Tar does not restore the group even if the user is in the group, +unless the user is root. I will keep this behavior with Obnam, since +it should be the least risky surprise: there is the least chance of +a privilege escalation from this. --liw + +[[done]] diff --git a/bugs/restore-skip-files.mdwn b/bugs/restore-skip-files.mdwn new file mode 100644 index 0000000..aa9cf87 --- /dev/null +++ b/bugs/restore-skip-files.mdwn @@ -0,0 +1,11 @@ +[[!tag obnam-wishlist]] + +It would be practical if "obnam restore" could skip files based +on various criteria, at least pathname. + + +--- + +Rather than add a complicated language for excluding what to restore, +using the FUSE plugin seems like an obviously better way to do this. +[[done]] --liw diff --git a/bugs/restore-timestamp-trouble.mdwn b/bugs/restore-timestamp-trouble.mdwn new file mode 100644 index 0000000..2a70a16 --- /dev/null +++ b/bugs/restore-timestamp-trouble.mdwn @@ -0,0 +1,6 @@ +Why does restore verification fail with small timestamp differences +when seivot runs summain with mtime enabled? +--liw + +[[done]] thanks to avoiding `os.lstat` and using our own lstat(2) +wrapper. diff --git a/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn b/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn new file mode 100644 index 0000000..50c5016 --- /dev/null +++ b/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn @@ -0,0 +1,44 @@ +Create repository using: +obnam --compress-with=gzip --repository --repository /tmp/backup/ backup "$HOME/" + +Try to restore: +obnam restore --repository /tmp/backup/ --to=/tmp/restore "$HOME/test" + +Error message: +ERROR: Invalid repository format version ('x\x9c3\xe5\x02\x00\x00v\x00@') -- forgot encryption? + +Either the compression used should be autodetected (maybe saved with metadata?) or the error message should be adjusted (using obnam 0.25) + +--xeen + +--- + +I confirm that I see the bug. And I agree that Obnam should automatically detect +when compression has been used and that the error message is awful. + +--- + + +Proposed solution (involving re-formatting the repository, for fun and profit) + +1. Alter the metadata store to not be filtered + * Alter Repository to not use encryption filter in the metadata files + * Alter HookedFS to allow you to ask that data filters be bypassed + * Alter Repository to ask for bypass on metadata files (format) +2. Alter the data filtering hooks to work as follows: + * When filtering for write, the output data format should always be filtername\0data. This chains for each write filter. + * When filtering for read, we read the filter name, apply its inverse on the data and then repeat, chaining until we read \0data (an empty filter name) + +Side effects: + +* Ordering of filters can change at will, because reading data is always deterministic on the stored data. +* Data is leaked about (at best) the top level filter (which might be encryption). + * Note: Encrypted repositories are obvious anyway so this should not be considered a blocker +* All files increase in size by some number of (at minimum 1) bytes. +* Repository metadata is now not affected by filters, so we can always guarantee to be able to read it and produce useful warnings/errors in case of format mismatches. + +--- + +Should be fixed in repository format 6 + +[[done]] diff --git a/bugs/ro-ops-while-rw-runs-crash.mdwn b/bugs/ro-ops-while-rw-runs-crash.mdwn new file mode 100644 index 0000000..50f76d4 --- /dev/null +++ b/bugs/ro-ops-while-rw-runs-crash.mdwn @@ -0,0 +1,9 @@ + <weasel> runing 'generations' while a backup or forget just is being run, + results in a backtrace + <weasel> probably should say 'repository locked', if it cannot provide the + information if work is in progress + +I wasn't able to reproduce this, at least not yet. I'll close this until +there's a traceback or recipe for reproducing. --liw + +[[done]] diff --git a/bugs/rolling-checksum-patches-need-review.mdwn b/bugs/rolling-checksum-patches-need-review.mdwn new file mode 100644 index 0000000..8422030 --- /dev/null +++ b/bugs/rolling-checksum-patches-need-review.mdwn @@ -0,0 +1,10 @@ +[[!tag obnam-wishlist]] + +* Review <Thingol> 01:53:50> http://doriath.cz/obnam/rollingsum-split-2.patch + ...faster, but I don't like it much. :-) + * <Thingol> http://doriath.cz/obnam/rollingsum-split-2.patch ...but maybe + a different rolling checksum function (and/or a different window size) + would be better, it needs more testing. + + +[[done]] -- I never got around to doing this. --liw diff --git a/bugs/root-is-file-not-dir-error-message.mdwn b/bugs/root-is-file-not-dir-error-message.mdwn new file mode 100644 index 0000000..cc20192 --- /dev/null +++ b/bugs/root-is-file-not-dir-error-message.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654211> for info. + +[[done]] diff --git a/bugs/rsync_as_transport__63__.mdwn b/bugs/rsync_as_transport__63__.mdwn new file mode 100644 index 0000000..abfea83 --- /dev/null +++ b/bugs/rsync_as_transport__63__.mdwn @@ -0,0 +1,20 @@ +SFTP is very slow because of the per-command, per-file latency. rsync, on the other hand, works much more quickly for transferring large numbers of small files. I don't know how the rsync protocol works, but I assume it basically batches files together into a stream, ala tar, and lets the rsync process on the receiving end split them up. + +I am considering using a local repo and rsyncing it to a remote server rather than directly backing up with SFTP. After running a few simple tests, it seems like it should work fine. Is this a good idea, assuming I have enough disk space for a local repo? + +Even better, could Obnam somehow use rsync as a transport instead of SFTP? Maybe it could write small files locally and then rsync them to the remote repo; and for reading it could rsync small files to a local directory. It could do it in batches based on total filesize, or, perhaps more usefully, based on total number of files (backend files, not source files). Doing it this way wouldn't require much disk space--a few megs, at most, I'd think--as opposed to keeping a local copy of the entire repo. + + +--- + +Backing up to a local repository and rsyncing that to the server will work just fine. After you've rsynced you can then either continue using the local repository and rsyncing it to the server for updates, or switch completely to using the remote one only. You can't mix the two approaches, however, since that will just make Obnam confused. + +As for using rsync as a transport I don't see that as a realistic thing at this time, +but it's an innovative idea. Thanks. + +Obnam does not know beforehand which files it will need from the repository, and it needs to be able to create files in the repository concurrently with other instances of Obnam, and these defeat any benefit from rsync at this time. However, a repository-side Obnam server process might be the way to go, but there's other approaches to be used as well, such as caching files locally in a way that is safe against concurrent use of repository by multiple clients. I have those ideas written down elsewhere, so I'll mark this bug report as done. + +--liw + +[[done]] + diff --git a/bugs/salsa-tins.mdwn b/bugs/salsa-tins.mdwn new file mode 100644 index 0000000..450174e --- /dev/null +++ b/bugs/salsa-tins.mdwn @@ -0,0 +1,36 @@ +[[!tag obnam-performance]] + +Problem: If chunk size is reasonably large (say, a megabyte), then +most files will be smaller, and the repository ends up with a large +number of identical files. + +Idea: collect chunks into groups, called "salsa tins". + +- salsa tin = list of chunks +- salsa tin has an id +- chunk id = salsa tin id + suitable number of extra bits for + index into list +- chunk id may be 64 bits total, or 64+32, or whatever seems convenient +- no chunk gets stored alone, only in salsa tins + +This lets a client put things into the repository at will, without +synchronisation or locking beyond what the filesystem provides +(exclusive creation of files). + + +--- + +Having multiple chunks in a single file complicates the logic for +managing files in the repository, and deleting unused chunks. + +Therefore, an alternative idea: instead of shoving multiple chunks +into one file, allow files to use parts of chunks. Currently a +file's metadata lists the chunks that have its contents. Change +this to be a list of (chunk id, offset, length) triplets, where +offset and length specify a part of a chunk. This way, a client can +create one chunk that contains the data of many small files, and +they can all just use the relevant part of the chunk. Managing +removal of those files is easy: it is the current code without +modification. + +--liw diff --git a/bugs/seek-over-holes.mdwn b/bugs/seek-over-holes.mdwn new file mode 100644 index 0000000..9638b70 --- /dev/null +++ b/bugs/seek-over-holes.mdwn @@ -0,0 +1,4 @@ +[[!tag obnam-wishlist]] + +If a file is sparse, and has a large hole, it would be good to skip over +it with `SEEK_HOLE` and `SEEK_DATA`. --liw diff --git a/bugs/setup-too-long-without-feedback.mdwn b/bugs/setup-too-long-without-feedback.mdwn new file mode 100644 index 0000000..9944f7a --- /dev/null +++ b/bugs/setup-too-long-without-feedback.mdwn @@ -0,0 +1,3 @@ +setting up takes too long without any feedback to show what's happening + +[[done]] --liw diff --git a/bugs/sftp-access-to-live-data-crashes.mdwn b/bugs/sftp-access-to-live-data-crashes.mdwn new file mode 100644 index 0000000..d2806c3 --- /dev/null +++ b/bugs/sftp-access-to-live-data-crashes.mdwn @@ -0,0 +1,35 @@ +Mysterious logging errors with sftp access to live data. + + sftp-root and sftp-repo run on xander: mysterious logging errors + Traceback (most recent call last): + File "/usr/lib/python2.6/logging/__init__.py", line 776, in emit + msg = self.format(record) + File "/usr/lib/python2.6/logging/__init__.py", line 654, in format + return fmt.format(record) + File "/usr/lib/python2.6/logging/__init__.py", line 436, in format + record.message = record.getMessage() + File "/usr/lib/python2.6/logging/__init__.py", line 306, in getMessage + msg = msg % self.args + TypeError: not enough arguments for format string + +* happened with repo on localfs, live data on sftp (sftp2) +* running with both on localfs (sftp3): worked fine +* so there's something wrong, possibly in paramiko +* tried a script to transfer 140 gigs of data with paramiko, but this failed, + the transfer speed is very slow (less than 1 MiB/s) + +--liw + +--- + +Removing 1.0 blocker tag: access to live data over sftp is limited enough +that it's not something I want to guarantee for 1.0. Fixing the limitations +may require patching paramiko or replacing it with another sftp +implementation. --liw + + + +--- + +I can't reproduce this with the current version of paramiko on Debian wheezy +(1.7.7.1-3.1) and the current obnam (1.3). [[done]] --liw diff --git a/bugs/sftp-get-put-for-transfer-for-speed.mdwn b/bugs/sftp-get-put-for-transfer-for-speed.mdwn new file mode 100644 index 0000000..afe0c44 --- /dev/null +++ b/bugs/sftp-get-put-for-transfer-for-speed.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-performance]] + +Would it be faster to use the sftp put and get methods instead of the +current open/read/write/close code for transferring files to and from +the repository? diff --git a/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn b/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn new file mode 100644 index 0000000..3626452 --- /dev/null +++ b/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn @@ -0,0 +1,34 @@ +[[!tag obnam-wishlist]] + +I've done a fair bit of work to optimise away a whole bunch of round +trips over sftp. For example, renaming a file over sftp can fail, if +the target name already exists. So we try to remove it first. The +pretty way of doing that would be to check if the target exists before +trying to remove it. However, that would mean an extra round trip, +when the target does exist. + +Pretty: + + 1. does target exist? + 2. if it exists, remove + 3. rename + +Ugly: + + 1. try to remove target, and ignore a failure due to target not + existing + 2. rename + +Unfortunately, doing things the ugly way results in the log file +containing a lot of tracebacks from paramiko. They're benign, but +they look scary, and they fill the log with useless text. + +Reported-By: Jordi Marqués + + +Could obnam just catch the exception and therefore stop the scary tracebacks from appearing in the logfile? - Andrew Ruthven + + +This is fixed now. It was a problem with Obnam debugging output, +not paramiko's fault at all. --liw [[done]] + diff --git a/bugs/sftp-too-limited.mdwn b/bugs/sftp-too-limited.mdwn new file mode 100644 index 0000000..0b3d608 --- /dev/null +++ b/bugs/sftp-too-limited.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-wishlist]] + +Obnam is currently using paramiko as the SFTP implementation. It is a +bit more limited than the SFTP protocol is, and so some stuff that Obnam +should be doing, such as restoring hardlinks across SFTP, are not +possible. There may also be some bugs with regards to timestamp handling. + +Possible fixes: + +* patch paramiko to support more of SFTP +* switch to twisted's conch or libssh or <http://pypi.python.org/pypi/ssh/> + or python-ssh2 + +--liw + diff --git a/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn b/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn new file mode 100644 index 0000000..e2af811 --- /dev/null +++ b/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-1.0-blocker]] + +`VFS.scan_tree` uses `listdir2`, for performance, and that does `lstat` +for each file. If the `lstat` fails, `listdir2` fails, and `scan_tree` +pretends the entire directory is empty. That's a problem: it should +skip only the items for which `lstat` failed. + +FUSE can cause the `lstat` to fail. + +Thanks, weasel, for spotting the problem and trying things until we found +the reason why his home directory was getting ignored. + +--liw + +[[done]] in current bzr diff --git a/bugs/small-chunks-in-btree.mdwn b/bugs/small-chunks-in-btree.mdwn new file mode 100644 index 0000000..268835c --- /dev/null +++ b/bugs/small-chunks-in-btree.mdwn @@ -0,0 +1,11 @@ +Would it be more efficient to store small amounts of data directly in +the B-tree rather than as a separate chunk? + +--liw + +Some quick experimentation indicates that this can be useful indeed. +It reduces the amount of round trips Obnam needs to do to the repository, +which matters a lot for sftp acces. It's now implemented in bzr and will +be in the next release. --liw + +[[done]] diff --git a/bugs/small-chunks-in-separate-btree.mdwn b/bugs/small-chunks-in-separate-btree.mdwn new file mode 100644 index 0000000..6421963 --- /dev/null +++ b/bugs/small-chunks-in-separate-btree.mdwn @@ -0,0 +1,12 @@ +[[!tag obnam-wishlist]] + +An idea for speed improvement: store small chunks not in the +per-client B-tree, but a separate per-client B-tree. I suspect +they are currently making the metadata too sparse in the per-client +B-tree. However, this needs to be measured. --liw + +-- + +This is not going to be acceptable. Even small chunks need to be +useable for de-duplication. Need a better idea. [[done]] with this idea. +--liw diff --git a/bugs/specify-ssh-key-to-use.mdwn b/bugs/specify-ssh-key-to-use.mdwn new file mode 100644 index 0000000..b6643e9 --- /dev/null +++ b/bugs/specify-ssh-key-to-use.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +obnam should allow specifying the ssh key for sftp to use + + +--- + +This is [[done]] now. --liw diff --git a/bugs/specify-ssh-key.mdwn b/bugs/specify-ssh-key.mdwn new file mode 100644 index 0000000..05932af --- /dev/null +++ b/bugs/specify-ssh-key.mdwn @@ -0,0 +1,7 @@ +Obnam needs a way to specify which ssh key (key file) to use. +Cf. `ssh -i` option. --liw + +At the moment, obnam only uses ssh agent for authentication. That +should be fixed. --liw + +[[done]] diff --git a/bugs/struct_calcsize_vs_len_struct_pack.mdwn b/bugs/struct_calcsize_vs_len_struct_pack.mdwn new file mode 100644 index 0000000..6f77c4c --- /dev/null +++ b/bugs/struct_calcsize_vs_len_struct_pack.mdwn @@ -0,0 +1,68 @@ +A small performance improvement on startup may be possible by replacing len(key('',0,0)) with struct.calcsize(self.fmt): + + python -m timeit -s \ + 'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\ + 'struct.calcsize(fmt)' + 1000000 loops, best of 3: 0.201 usec per loop + +vs + + python -m timeit -s \ + 'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\ + 'len(struct.pack(fmt,"",0,0))' + 1000000 loops, best of 3: 0.753 usec per loop + +I can't attach files, so I can't attach the bundle I think you prefer, but here's the diff, and I'll email you the bundle as an attachment. + + === modified file 'obnamlib/checksumtree.py' + --- obnamlib/checksumtree.py 2011-07-26 13:23:55 +0000 + +++ obnamlib/checksumtree.py 2013-02-08 07:09:22 +0000 + @@ -33,7 +33,7 @@ + upload_queue_size, lru_size, hooks): + tracing.trace('new ChecksumTree name=%s' % name) + self.fmt = '!%dsQQ' % checksum_length + - key_bytes = len(self.key('', 0, 0)) + + key_bytes = struct.calcsize(self.fmt) + obnamlib.RepositoryTree.__init__(self, fs, name, key_bytes, node_size, + upload_queue_size, lru_size, hooks) + self.keep_just_one_tree = True + + === modified file 'obnamlib/chunklist.py' + --- obnamlib/chunklist.py 2011-07-26 13:23:55 +0000 + +++ obnamlib/chunklist.py 2013-02-08 07:09:22 +0000 + @@ -35,14 +35,15 @@ + + def __init__(self, fs, node_size, upload_queue_size, lru_size, hooks): + tracing.trace('new ChunkList') + - self.key_bytes = len(self.key(0)) + + self.fmt = '!Q' + + self.key_bytes = struct.calcsize(self.fmt) + obnamlib.RepositoryTree.__init__(self, fs, 'chunklist', self.key_bytes, + node_size, upload_queue_size, + lru_size, hooks) + self.keep_just_one_tree = True + + def key(self, chunk_id): + - return struct.pack('!Q', chunk_id) + + return struct.pack(self.fmt, chunk_id) + + def add(self, chunk_id, checksum): + tracing.trace('chunk_id=%s', chunk_id) + + === modified file 'obnamlib/clientlist.py' + --- obnamlib/clientlist.py 2011-07-26 13:23:55 +0000 + +++ obnamlib/clientlist.py 2013-02-08 07:09:22 +0000 + @@ -49,7 +49,7 @@ + tracing.trace('new ClientList') + self.hash_len = len(self.hashfunc('')) + self.fmt = '!%dsQB' % self.hash_len + - self.key_bytes = len(self.key('', 0, 0)) + + self.key_bytes = struct.calcsize(self.fmt) + self.minkey = self.hashkey('\x00' * self.hash_len, 0, 0) + self.maxkey = self.hashkey('\xff' * self.hash_len, obnamlib.MAX_ID, + self.SUBKEY_MAX) + + +--- + +Thank you. This has been merged now. [[done]] --liw diff --git a/bugs/symlink-as-backup-root.mdwn b/bugs/symlink-as-backup-root.mdwn new file mode 100644 index 0000000..d5b4f29 --- /dev/null +++ b/bugs/symlink-as-backup-root.mdwn @@ -0,0 +1,24 @@ +From Lars Kruse: + +> Thus the situation seems to be that obnam stores all relevant chunks, +> but due to the symlink pointing to something outside of the repository +> I cannot restore + +From liw: This is obviously a bug. Obnam should either give an error +if the backup root is a symlink, or back it up as a symlink, not +recurse into the directory and then backup that and only then ruin +everything by backing up the symlink as a symlink. + +--- + +This should now be fixed to give an error message when you backup. +I can't currently give access to the hidden files easily, I'm afraid, +but if you change your backup root to be the directory pointed at by +the symlink, and run a backup, it should go fairly fast and give you +access to your files. + +Later, I may consider allowing the root to be a symlink to a +directory, but that'll require bigger changes than this error message, +I'm afraid. Patches to implement this would be welcome. + +[[done]] --liw diff --git a/bugs/symlink-as-root.mdwn b/bugs/symlink-as-root.mdwn new file mode 100644 index 0000000..90d052f --- /dev/null +++ b/bugs/symlink-as-root.mdwn @@ -0,0 +1,5 @@ +Obnam follows a symlink as the backup root, when backing up, but does +not let you restore the target of the symlink. It should not follow +the symlink when it is root, for consistency. --liw + +[[done]] now --liw diff --git a/bugs/tarball-with-everything.mdwn b/bugs/tarball-with-everything.mdwn new file mode 100644 index 0000000..e19334a --- /dev/null +++ b/bugs/tarball-with-everything.mdwn @@ -0,0 +1,28 @@ +[[!tag obnam-wishlist]] + +From Chris: + +> python2.6 is available in epel, so it should be simple. I've hit the +> "In order to install package foo, you need package bar". +> +> I'm sure this is a simple thing to do if I actually read the +> instructions, but for those of us who can't be bothered, a single +> tarball that just worked would be nice. +> +> Ideally one that could install in my homespace for testing purposes. + +From liw: + +I think that might be workable. It's almost all pure Python, +and can be run directly from the source directories, so a tarball +with all the source projects included, plus a little script to set +up PATH and PYTHONPATH would work. + + +-- + +I've been thinking about this, and I'm afraid I don't want to put in the +effort to do this. Installation is a bit of a bother, but having packages +for the main Linux distros seems like the best way to solve this. +[[done]] +--liw diff --git a/bugs/test-against-old-format-versions.mdwn b/bugs/test-against-old-format-versions.mdwn new file mode 100644 index 0000000..aedd086 --- /dev/null +++ b/bugs/test-against-old-format-versions.mdwn @@ -0,0 +1,8 @@ +The test suite should verify that Obnam works in the proper way when +it encounters a repository with the old format version. Whatever that +proper way might be. --liw + +The current tests already verify that the format version is correct, +and that's all I want to support right now. --liw + +[[done]] diff --git a/bugs/test-for-full-filesystem.mdwn b/bugs/test-for-full-filesystem.mdwn new file mode 100644 index 0000000..740a74c --- /dev/null +++ b/bugs/test-for-full-filesystem.mdwn @@ -0,0 +1 @@ +Obnam has no test for what happens when the filesystem fills up. diff --git a/bugs/test-suite-leave-temp-files.mdwn b/bugs/test-suite-leave-temp-files.mdwn new file mode 100644 index 0000000..ea1831d --- /dev/null +++ b/bugs/test-suite-leave-temp-files.mdwn @@ -0,0 +1,5 @@ +The obnam test suite leaves temporary files lying around. They should +be cleaned up. --liw + +I can't seem to reproduce this, except by killing the test suite in +the middle, and in that case it's OK to leave temp file. [[done]] --liw diff --git a/bugs/time_based_checkpoint_generations.mdwn b/bugs/time_based_checkpoint_generations.mdwn new file mode 100644 index 0000000..3d00f72 --- /dev/null +++ b/bugs/time_based_checkpoint_generations.mdwn @@ -0,0 +1,18 @@ +[[!tag obnam-wishlist]] + +obnam should accept SIZE values as well as TIME values for the --checkpoint option. For instance: + + obnam backup $HOME --checkpoint=5min + +This would be very handy for connections which vary a lot in speed and quality. + +-- weinzwang + + +While I agree this would be useful, I am afraid I haven't made any +move towards implementing it, and as such, keeping the bug open isn't +helping me manage the bug list. If anyone really wants this, sending +a patch (even a proof of concept one) would be effective, though. +--liw + +[[done]] diff --git a/bugs/too-slow.mdwn b/bugs/too-slow.mdwn new file mode 100644 index 0000000..f08c022 --- /dev/null +++ b/bugs/too-slow.mdwn @@ -0,0 +1,19 @@ +[[!tag obnam-1.0-blocker]] + +Obnam should be fast enough that it can back up my laptop's home +directory, overnight, encrypted, to an unencrypted USB hard disk. +That's 140 gigabytes of data, much of music and videos and photographs, +in 12 hours. --liw + +I can now back up 200+ gigabytes of data without encryption, to an +encrypted USB drive, in four hours. --liw + +However, removing a large number of checkpoints takes much too long, +making the total backup time much too long. --liw + +Did a full backup to a fresh repository on a USB drive. 375 gigabytes of +data, about 200 thousand files, backup took almost exactly 8 hours, +not including cleaning up checkpoints, since there was a file that +went missing during the backup. Deleted the checkpoints manually: +that took another 15 minutes. That's fast enough, marking the bug [[done]]. +--liw diff --git a/bugs/too_much_test_coverage_exclusion.mdwn b/bugs/too_much_test_coverage_exclusion.mdwn new file mode 100644 index 0000000..5f1b01f --- /dev/null +++ b/bugs/too_much_test_coverage_exclusion.mdwn @@ -0,0 +1,5 @@ +Too much code is excluded from test coverage. For example, as much code as possible in plugins should be unit tested (so coverage can be meaningfully tested). --liw + +--- + +I've reviewed the code now, and most of the plugins have only code that is for that plugin, and it's usually not easily unit testable in any case. [[done]] --liw diff --git a/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn b/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn new file mode 100644 index 0000000..ac0cf06 --- /dev/null +++ b/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn @@ -0,0 +1,23 @@ +I have a daily backup with obnam scheduled using a cronjob. The job succeeds without problems, however the cron daemon mails the following output to me every day: + + tput: No value for $TERM and no -T specified + +I looked for the source of this message, but neither obnam nor python know about the "-T" parameter. So I tried to wrap the job in a shell script that sets the TERM environment variable if not set like this: + + if [ -z "$TERM" ]; then + export TERM=linux + fi + +But that didn't help either. Any ideas how to fix this? + +--Christian + +--- + +tput is a shell command (see "man tput"). Obnam does not use it in any way, not even +indirectly, as far as I can see. What I suspect is happening is that when you invoke commands +from your crontab, it invokes your shell, and that runs something like `.bashrc`, and you +have something that calls tput. + +As such, unless shown otherwise, I'll assume this is not a bug in Obnam, sorry. --liw +[[done]] diff --git a/bugs/transient-problems-warning.mdwn b/bugs/transient-problems-warning.mdwn new file mode 100644 index 0000000..b22321b --- /dev/null +++ b/bugs/transient-problems-warning.mdwn @@ -0,0 +1,13 @@ +From Neal Becker: + +> Looks like a file disappeared during backup. No big deal, but probably +> obnam should handle this a little better (warning, rather than CRITICAL) ? + + +--- + +Obnam can't know if the problem is transient or not, I think, so it's +correct that it reports it as an error rather than warning. However, +it should not abort in that case, and I don't think it does, now. --liw + +[[done]] diff --git a/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn b/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn new file mode 100644 index 0000000..5759dc8 --- /dev/null +++ b/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn @@ -0,0 +1,16 @@ +[[!tag obnam-wishlist]] + +The Idea is to change or extend the --exclude-caches feature so that one can configure which filename to look for that will make obnam skip the directory + +--- + +Changing `--exclude-caches` seems wrong to me: it has a specific purpose (to implement the cache directory tagging spec, <http://www.bford.info/cachedir/spec.html>). + +Adding a new option to ignore directories that contain a specific file (or directory) would be fine. + +--liw + +--- + +bwh points out that the owner of the directory and the tag file should +be the same. diff --git a/bugs/ugly-enoent-error-message.mdwn b/bugs/ugly-enoent-error-message.mdwn new file mode 100644 index 0000000..4097f6e --- /dev/null +++ b/bugs/ugly-enoent-error-message.mdwn @@ -0,0 +1,24 @@ +[[!tag obnam-wishlist]] + +ugly error message from obnam: + + 00h28m12s 42015 files; 108.21 MiB (224.7 KiB/s) + /home/liw/.mozilla/firefox/td0gERROR: Cannot back up + /home/liw/.mozilla/firefox/td0gy7bo.default/urlclassifier3.sqlite-journal: + (2, 'No such file or directory', + '/home/liw/.mozilla/firefox/td0gy7bo.default/urlclassifier3.sqlite-journal') + +- should start error message on new line +- should not print Python tuple + +--liw + +--- + +I believe the tuple printing is now fixed. --liw + +--- + +The problem with error messages getting printed out in the middle of +progress output has now been fixed in git master. --liw +[[done]] diff --git a/bugs/ugly-sshexception-error-message.mdwn b/bugs/ugly-sshexception-error-message.mdwn new file mode 100644 index 0000000..893146b --- /dev/null +++ b/bugs/ugly-sshexception-error-message.mdwn @@ -0,0 +1,15 @@ +[[!tag obnam-wishlist]] + +The following is ugly: + + File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 667, in _read_response + raise SSHException('Server connection dropped: %s' % (str(e),)) + SSHException: Server connection dropped: + Exception OSError: OSError(32, 'Broken pipe') in <bound method SFTPFile.__del__ of <paramiko.SFTPFile object at 0x18861cd0>> ignored + +There should not be a traceback, but a proper error message. + + +--- + +I believe this is fixed now. --liw [[done]] diff --git a/bugs/use-cliapp-setup_logging-overrides.mdwn b/bugs/use-cliapp-setup_logging-overrides.mdwn new file mode 100644 index 0000000..c477577 --- /dev/null +++ b/bugs/use-cliapp-setup_logging-overrides.mdwn @@ -0,0 +1,6 @@ +cliapp now allows overriding parts of the logging setup, and obnam should +use that as soon as cliapp has been released. + +--- + +Fixed in git master. [[done]] --liw diff --git a/bugs/use-cliapps-plugin-manager.mdwn b/bugs/use-cliapps-plugin-manager.mdwn new file mode 100644 index 0000000..4da8a86 --- /dev/null +++ b/bugs/use-cliapps-plugin-manager.mdwn @@ -0,0 +1,4 @@ +Obnam should be using cliapp's plugin manager, instead of its +own. --liw + +Fixed now. [[done]] --liw diff --git a/bugs/use-existing-node-size.mdwn b/bugs/use-existing-node-size.mdwn new file mode 100644 index 0000000..1c6b241 --- /dev/null +++ b/bugs/use-existing-node-size.mdwn @@ -0,0 +1,7 @@ +When using an existing repository, Obnam should use the existing node +size even if the current Obnam `--node-size` setting is different. +It might log a warning message about it, but failing work is not +acceptable. + + +[[done]] diff --git a/bugs/use-fsync.mdwn b/bugs/use-fsync.mdwn new file mode 100644 index 0000000..faa3dbf --- /dev/null +++ b/bugs/use-fsync.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +Obnam should, at least optionally, use fsync or other methods to ensure +that everything gets committed to disk by the kernel by the end of a +backup run. --liw + +I want this to not have a huge performance impact, though. Learning +from the lessons of dpkg, sqlite/liferea/firefox, etc, and using fsync/fdatasync +and `sync_file_range` in the right ways is going to be necessary. --liw diff --git a/bugs/username-groupname-over-sftp-wrong.mdwn b/bugs/username-groupname-over-sftp-wrong.mdwn new file mode 100644 index 0000000..452d428 --- /dev/null +++ b/bugs/username-groupname-over-sftp-wrong.mdwn @@ -0,0 +1,11 @@ +When the backup root (i.e., live data) is accessed over sftp, obnam +queries the _local_ system for the names of the user and group. This +is going to result in bad data. In this case, unless it can query +the remote server, it should just not store the username and groupname +at all. --liw + +Fixing this is going to require an obnam server mode, where the remote +side is also running obnam, and we can query that for stuff. But then +SFTP as a whole will be unnecessary. --liw + +[[done]] diff --git a/bugs/utimensat-undefined-symbol.mdwn b/bugs/utimensat-undefined-symbol.mdwn new file mode 100644 index 0000000..323dd5e --- /dev/null +++ b/bugs/utimensat-undefined-symbol.mdwn @@ -0,0 +1,25 @@ +[[!tag obnam-wishlist]] + +Obnam apparently depends on "utimensat", which only appears in more recent versions of glibc. Unfortunately, CentOS5's glibc does not have this routine, and we are not upgrading to CentOS6 just yet. + + [hash@hydrogen]$ uname -a + Linux hydrogen.localdomain 2.6.18-274.18.1.el5 #1 SMP Thu Feb 9 12:45:44 EST 2012 x86_64 x86_64 x86_64 GNU/Linux + +Trying too import obnam produces this error: + + >>> import obnamlib._obnam + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ImportError: /usr/local/lib/python2.6/site-packages/obnamlib/_obnam.so: undefined symbol: utimensat + >>> + +-- hash + +--- + +`utimensat` was added to the Linux kernel in version 2.6.22 (2007), and to glibc in version 2.6 (2007). It is specified in Posix 2008. It's needed to set sub-second timestamps on files on restore. As a workaround, you can change the `_obnammodule.c` to call `utimes` instead. If you can provide a clean patch, I'll be happy to add it to Obnam proper. + +--liw + +No patch has appeared, closing bug. [[done]] --liw +(Patch will of course always be useful, please send one to the mailing list.) diff --git a/bugs/verify-pick-randomly.mdwn b/bugs/verify-pick-randomly.mdwn new file mode 100644 index 0000000..89085ea --- /dev/null +++ b/bugs/verify-pick-randomly.mdwn @@ -0,0 +1,9 @@ +Currently, `obnam verify` compares all files in the backup with all files +in live data. Often it would be enough to just verify a few files, to +sample the data rather than do a full verification. Obnam should be able +to pick a desired number of files randomly from the backup and compare +those. + +--liw + +[[done]] diff --git a/bugs/verify-progress-should-be-bytes.mdwn b/bugs/verify-progress-should-be-bytes.mdwn new file mode 100644 index 0000000..44744a9 --- /dev/null +++ b/bugs/verify-progress-should-be-bytes.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +When "obnam verify" shows progress, it's based on number of files. +It should be based on number of bytes instead, for better progress +reporting. --liw + +--- + +[[done]] in git master now. --liw diff --git a/bugs/verify-should-continue-past-error.mdwn b/bugs/verify-should-continue-past-error.mdwn new file mode 100644 index 0000000..5dd063c --- /dev/null +++ b/bugs/verify-should-continue-past-error.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +Suggested by Rafał Gwiazda: "obnam verify" should report all errors, +instead of terminating after the first problem found. + +--liw + +[[done]] diff --git a/bugs/verify-stricter.mdwn b/bugs/verify-stricter.mdwn new file mode 100644 index 0000000..5d8a0df --- /dev/null +++ b/bugs/verify-stricter.mdwn @@ -0,0 +1,29 @@ +[[!tag obnam-wishlist]] + +On Tue, Mar 11, 2014 at 04:34:13PM +0100, Thomas Schwinge wrote: + +> Alternatively, wouldn't it make sense to change to, or at least have +> available, a mode where verify works not based on the data in the backup, +> but instead of the actual user data? Wouldn't doing it in this way build +> a yet greater confidence that obnam has backed up everything alright? +> That is, instead of traversing the data in the backup and verifying +> against the corresponding user data, it'd traverse the user data (as when +> doing a backup) and verify against the data in the backup. + +I think you're right. I think we should have "obnam verify" traverse +through both the backup generation and the live data, and report as +follows: + +* if a file exists only in the backup, but not in the live data +* if a file exists only in the live data, but not in the backup + - also indicate whether the file matches (current) exclusion + rules, since that matters: a file that is in live data but is + not excluded is different from an excluded one +* if a file exists in both the backup and the live data, but is + different in any way + +I don't have time to work on this at the moment, but I'll add it to +the list of bugs in http://liw.fi/obnam/bugs/ and would be happy to +review and merge a patch for this. + +--liw diff --git a/bugs/vfs-split-for-live-data-repo-access.mdwn b/bugs/vfs-split-for-live-data-repo-access.mdwn new file mode 100644 index 0000000..9165ea9 --- /dev/null +++ b/bugs/vfs-split-for-live-data-repo-access.mdwn @@ -0,0 +1,8 @@ +[[!tag obnam-wishlist]] + +Should obnam split VFS for repo access and live data access? repo needs +so much less, after all. This would perhaps also make it easier to add support +for things such as S3. -liw + +[[done]] I have decided against this. It's unnecessary complication in +the implementation, I think. --liw diff --git a/bugs/warn-about-common-client-names.mdwn b/bugs/warn-about-common-client-names.mdwn new file mode 100644 index 0000000..0dd3727 --- /dev/null +++ b/bugs/warn-about-common-client-names.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-wishlist]] + +It would be cool if Obnam could warn about common client names, +such as `localhost` or `localhost.localdomain`. + +[[done]] closing wishlist bug since it isn't making the thing happen. --liw diff --git a/bugs/whitelist-backups.mdwn b/bugs/whitelist-backups.mdwn new file mode 100644 index 0000000..ea61394 --- /dev/null +++ b/bugs/whitelist-backups.mdwn @@ -0,0 +1,9 @@ +[[!tag obnam-wishlist]] + +The ability to do whitelist-based backups would be nice. This would make it easy to separate different chunks of $HOME as different clients (e.g., just game saves as a "games" client). --mathstuf + +--- + +Whitelists tend to be dangerous: you always forget something. This makes me +reluctanct to implement them. That said, you can achieve this with the PCRE +regular expressions Obnam uses. [[done]] --liw diff --git a/bugs/whole-file-checksums.mdwn b/bugs/whole-file-checksums.mdwn new file mode 100644 index 0000000..9a2fe26 --- /dev/null +++ b/bugs/whole-file-checksums.mdwn @@ -0,0 +1,5 @@ +[[!tag obnam-wishlist]] + +It would be good for Obnam to do the whole-file checksum with a different +checksum algorithm, or by using a suitable salt, to catch problems with +single-chunk files, e.g., when there is a hash collision. --liw diff --git a/bugs/wishlist-only.mdwn b/bugs/wishlist-only.mdwn new file mode 100644 index 0000000..d69a690 --- /dev/null +++ b/bugs/wishlist-only.mdwn @@ -0,0 +1,8 @@ +[[!inline pages="page(obnam/bugs/*) and + !obnam/bugs/*/* and + !obnam/bugs/done and + tagged(obnam-wishlist) and + !link(obnam/bugs/done)" + show=0 + postform=no]] + diff --git a/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn b/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn new file mode 100644 index 0000000..13215e8 --- /dev/null +++ b/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn @@ -0,0 +1,17 @@ +[[!tag obnam-wishlist]] + +As a system administrator I'd like to see data on how the memory and cpu usage of different obnam operations depend on the number of files and their sizes before adopting obnam for my own systems. + +I imagine that in the simplest case you could use something like + + for total_size in 50M 1G 50G 400G 1T 2T 3T; do + for file_size in 1k 2k 4k 8k 16k 32k 1M 64M; do + generate_filesystem $total_size $file_size original/ + /usr/bin/time -f "%e real %P rss" backup original/ backup/ + done + done + +to get cpu and memory usage of backup operations. Repeat same for obnam fsck if you are afraid that its memory usage could depend on total_size or file_size. + +[[done]] A very old wishlist bug. Closing this, since keeping it open +clearly isn't making it happen. --liw diff --git a/bugs/xattr-changes-do-not-trigger-backup.mdwn b/bugs/xattr-changes-do-not-trigger-backup.mdwn new file mode 100644 index 0000000..d001376 --- /dev/null +++ b/bugs/xattr-changes-do-not-trigger-backup.mdwn @@ -0,0 +1,6 @@ +[[!tag obnam-1.0-blocker]] + +It looks as if changes to xattrs (but nothing else) will not +trigger a backup in Obnam. They should. --liw + +[[done]] diff --git a/contact.mdwn b/contact.mdwn new file mode 100644 index 0000000..3e48ccf --- /dev/null +++ b/contact.mdwn @@ -0,0 +1,16 @@ +Contact +------- + +* [Mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net) +* `#obnam` on `irc.oftc.net` for IRC discussions about Obnam + - the mailing list is strongly preferred for asking help with Obnam + - Lars rarely has time to discuss anything serious on IRC + - patches to the mailing list only; post them as patches, not pointers to + code on some website, though an additional pointer to a branch + that can be merged is fine +* [Lars's blog posts about Obnam](http://blog.liw.fi/tag/obnam/) + and [btree](http://blog.liw.fi/tag/btree/). +* Also, [Lars's private journal entries](http://liw.fi/obnam/journal-dump/) + about obnam and btree. +* [Sharon Kimble's blog posts about Obnam](http://www.sharons.org.uk/tag/obnam/) +* If all else fails, [[/contact]] gives contact information for Lars. diff --git a/development.mdwn b/development.mdwn new file mode 100644 index 0000000..157a50a --- /dev/null +++ b/development.mdwn @@ -0,0 +1,40 @@ +Development +----------- + +**Contributions are welcome.** Every level of contribution is most +appreciated: bug reports, spelling and grammar fixes, code patches, +questions on how to get started, offers to run a continuous integration +server, etc. See [[README]] for information +on how to modify the code. and ask on the mailing list or on IRC for +anything else. If anything's unclear, ask! + +Things to do, if you want to help: + +* Review manual page. It is easy for documentation to get + out of date. Pointing out problems is helpful. +* Code review, both of obnam and the larch btree library. +* Benchmarking. +* Add support for translations. +* More! Add here your own ideas. + +Other stuff: + +* [[Obnam on-disk data structures|ondisk]] +* Repository [[encryption]] and [[signing]] +* Repository [[locking]] +* [[bugs]] + - see [code.liw.fi](http://liw.fi/code/) for instructions +* Benchmarks: + * [[specification|obnam/benchmarkspec]] + * [[recent benchmark results|benchmark-summary.txt]] +* [[Obnam release procedure|obnam/release]] + +Dependencies: + +* Python 2 +* paramiko +* larch: <http://liw.fi/larch/> +* python-lru: <http://liw.fi/lru/> +* ttystatus: <http://liw.fi/ttystatus/> +* CoverageTestRunner: <http://liw.fi/coverage-test-runner/> + (you only need this for running the test suite) diff --git a/donate.mdwn b/donate.mdwn new file mode 100644 index 0000000..331c838 --- /dev/null +++ b/donate.mdwn @@ -0,0 +1,45 @@ +[[!meta title="Donations"]] + +<pre> +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +Hi! I wrote Obnam, a backup program (http://liw.fi/obnam/). If you +like the program and would like donate a little to help me afford to +buy tools and services for developing it further, you can send me +money or gifts: + +* Amazon wishlist: + http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv +* Paypal: liw@liw.fi +* Bitcoin: bitcoin:1FUKDBy79cC6iTaiyg5i1JVBCemiVdb5QC + +Obnam does not run on donations: it currently mainly runs on my free +time. Donations are not required in any way: you're more than welcome +to use it without donating, and in any case I prefer a well-formulated +bug report (see http://liw.fi/obnam/bug-reporting/) or a patch to fix +a bug or add a feature. No bug is too small to be reported. + +I can also arrange to do Obnam development for specific features, or +support, for a fee, if you or your company needs something that isn't +happening otherwise. + +Lars Wirzenius, 2014-02-09 +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.12 (GNU/Linux) + +iQEcBAEBCAAGBQJS943KAAoJEIahkFub41rmf1kH/jefKNjYhfKE7xbzZgR1IzXJ +tJIC0a6YTqe+yXNPHOdsCj348v2TTcDl0l7106XprKDos02YZSDGqPbkqlxwF6DH +cPKw7ZAvosagmbRsW/8dctmDeDi5ejvAzUuSbDFiYDYK5f1OyhozVX8VIMvJXCEj +5OSGXzzYtpBuCNvB594u6E5FD3aE/MCT/5HTdnNhAfPK5Gn1bxEsDizB7DOIWD9D +ItAchaeSqTJJ7ODkFB7goRNd/yJiKYcTW3aK3NmuATavq92DWIwvhGTmK5AwbZS/ +mBvmzSLDpqw1cgAJpFCYK9yMBmvZ53okiBFxwuFSXY9FYI8mc8B9HsWK9szycd8= +=tbcm +-----END PGP SIGNATURE----- +</pre> + +(See [[donate.txt]] for a plain text version.) + +--- + +[[!img liw-bitcoin-qr-obnam.png]] diff --git a/donate.txt b/donate.txt new file mode 100644 index 0000000..69fd91b --- /dev/null +++ b/donate.txt @@ -0,0 +1,35 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +Hi! I wrote Obnam, a backup program (http://liw.fi/obnam/). If you +like the program and would like donate a little to help me afford to +buy tools and services for developing it further, you can send me +money or gifts: + +* Amazon wishlist: + http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv +* Paypal: liw@liw.fi +* Bitcoin: bitcoin:1FUKDBy79cC6iTaiyg5i1JVBCemiVdb5QC + +Obnam does not run on donations: it currently mainly runs on my free +time. Donations are not required in any way: you're more than welcome +to use it without donating, and in any case I prefer a well-formulated +bug report (see http://liw.fi/obnam/bug-reporting/) or a patch to fix +a bug or add a feature. No bug is too small to be reported. + +I can also arrange to do Obnam development for specific features, or +support, for a fee, if you or your company needs something that isn't +happening otherwise. + +Lars Wirzenius, 2014-02-09 +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1.4.12 (GNU/Linux) + +iQEcBAEBCAAGBQJS943KAAoJEIahkFub41rmf1kH/jefKNjYhfKE7xbzZgR1IzXJ +tJIC0a6YTqe+yXNPHOdsCj348v2TTcDl0l7106XprKDos02YZSDGqPbkqlxwF6DH +cPKw7ZAvosagmbRsW/8dctmDeDi5ejvAzUuSbDFiYDYK5f1OyhozVX8VIMvJXCEj +5OSGXzzYtpBuCNvB594u6E5FD3aE/MCT/5HTdnNhAfPK5Gn1bxEsDizB7DOIWD9D +ItAchaeSqTJJ7ODkFB7goRNd/yJiKYcTW3aK3NmuATavq92DWIwvhGTmK5AwbZS/ +mBvmzSLDpqw1cgAJpFCYK9yMBmvZ53okiBFxwuFSXY9FYI8mc8B9HsWK9szycd8= +=tbcm +-----END PGP SIGNATURE----- diff --git a/download.mdwn b/download.mdwn new file mode 100644 index 0000000..10f93e6 --- /dev/null +++ b/download.mdwn @@ -0,0 +1,25 @@ +Download +-------- + +* [direct download](http://code.liw.fi/debian/pool/main/o/obnam/); + includes source code, and Debian source and binary packages for + i386, and amd64 architectures for the releases of Debian I happen + to support +* [using apt](http://liw.fi/code/) (currently squeeze, wheezy, and + unstable supported): + + `deb http://code.liw.fi/debian wheezy main` + +* For Ubuntu, see [Chris's + PPA](https://launchpad.net/~chris-bigballofwax/+archive/obnam-ppa/). +* For Gentoo, Obnam is in the official portage. +* For openSUSE, see Valery Yundin's repository: + <http://download.opensuse.org/repositories/home:/Vayun/> + - spec files: + <https://build.opensuse.org/project/show?project=home%3AVayun> +* from version control: + + `git clone git://git.liw.fi/obnam` + +The code is licensed under GNU General Public License version 3, or later. + diff --git a/encryption.mdwn b/encryption.mdwn new file mode 100644 index 0000000..7c8ba64 --- /dev/null +++ b/encryption.mdwn @@ -0,0 +1,384 @@ +[[!meta title="Encryption support in Obnam"]] + +Obnam needs to support encryption of the backup store. This document +describes requirements for and design of how Obnam will be using +encryption. It is currently a **DRAFT** and feedback is very much +welcome. + +[[!toc ]] + + +Goal +---- + +The goal of encryption in Obnam is to avoid unauthorized parties from +gaining access to backed up data. If the backup store is on a server +on the Internet, and someone breaks into the server, the data should +be safe from prying eyes. Likewise, if the backup store is on a USB +hard disk, and the disk gets stolen, the data should not leak. + +This document does not worry about the data as it is being transferred. +The data is encrypted on the machine running obnam, +which reads the live data and stores it in a backup store, +and there may be up to three machines involved, and the links between +them are assumed to be over sftp or another sensible protocol. +(In other words, obnam may be running on one machine, the live data +may be on another one, and the backup store on a third one.) + +For the purpose of this document, the server hosting the backup store +is considered to be unauthorized to access the data, or do anything +to the data except store it. This means that if the server needs to +things like removing unwanted backup generations, it needs to be +explicitly authorized to do so, by giving it encryption keys. By +default, it should not have that access, and will see only encrypted +data. + + +Requirements +------------ + +* All data in the backup store MUST be encrypted. + Filenames are not encrypted, but Obnam uses random filenames whenever + it can already. +* The backup store does not need to store any backup keys. It MUST be + enough for the keys to be stored in the clients only. +* For disaster recovery, it may be necessary to be able to access the + backup store only using a passphrase. This should be supported by the + design, but use of this feature MUST be optional. +* Clients can replace public keys used with the backup store at any time, + without having to re-backup anything. After keys have been replaced, the + old keys MUST NOT be able to access data in the backup store. +* The design MUST support use of multiple keys. Each client MUST be able to + use their own key, and disallow anyone else from accessing their data, + even if the backup store is shared. It should be possible for the server + to have its own key that it uses for forgetting old generations, running + fsck on the entire store, and so on. + +The encryption keys for accessing the backup store are not used for +authenticating access to the backup store, to allow flexibility in +selecting store providers. + + +Encryption methods +------------------ + +There are two relevant encryption methods for Obnam: + +* public key cryptography +* symmetric cryptography + +With public keys, there is a optional passphrase for the secret key. +With symmetric encryption, the passphrase is the key. +The user needs to manage the secret key and its passphrase in a suitable +manner. + + +Design +------ + +All data in the backup store is stored either in B-tree nodes, +or in a special directory tree for chunks of file data. Each B-tree +is stored in its own directory tree, and each node is stored in its +own file. We'll call each of these directory +trees a _toplevel_. Thus the root of a backup store consists of some +number of toplevel directories. + +The backup store is used by some set of clients, plus perhaps the +backup server. We'll call each of these a user. Each user may have +its own public/private key pair, or they may be shared between +clients. This is up to the backup admins. The design assumes they +all have unique key pairs. + +Some toplevels are private for a particular client. Others are +shared. + +Every file in a toplevel is encrypted using symmetric encryption. +The symmetric encryption key is encrypted with the public +key of every user who needs access to the toplevel. In other words, +to have access to contents of the toplevel, a user needs to be able +to decrypt the symmetric encryption key using their own key. + +To stop a user from having access to a toplevel, the symmetric encryption key +file gets re-encrypted with every other key except the unwanted one. +This does not prevent an attacker who has previously stored a +local copy of the decrypted symmetric encryption key. To stop that, every file +in the toplevel needs to be re-encrypted with a new symmetric +encryption key. + +The encrypted symmetric encryption key is stored inside +the toplevel, using a well-known name. This is the only file not +encrypted with the symmetric encryption key. + +The public keys for users who should have access to the toplevel +are also stored inside the toplevel, in a file encrypted with the +symmetric encryption key. + +The symmetric encryption keys are generated randomly, +and are of sufficient length that brute-forcing them will not be +realistic. Perhaps something like 256 bits from /dev/random. + +To give a user access to the repository, the repository admin needs +to add the user's public key to the shared toplevels: client list, +chunks, and chunk checksum data. A client may remove itself from +the repository, or an admin may do that. "Admin" in this context +is anyone with access to all the shared toplevels. + + +Example +------- + + chunks/ -- toplevel for file data chunks + key -- symmetric encryption key + userkeys -- public keys of all users of this toplevel + * -- all other files + +To set up toplevel for encryption: + +* generate a suitable amount of random data to use as symmetric encryption key +* create list of public keys to have access to toplevel +* encrypt public key list using symmetric encryption key +* upload encrypted public key list +* encrypt symmetric encryption key with every public key in list +* upload encrypted symmetric encryption key + +To add a file to the toplevel: + +* download `key` file +* decrypt `key` file using user's private key, get key text +* encrypt file using symmetric encryption key +* upload encrypted file + +To use a file in the toplevel: + +* download `key` file +* decrypt `key` using user's private key +* download desired file +* decrypt file using symmetric encryption key + + +Discussion +---------- + +The simplest approach would be to only use public key encryption, +but this makes it difficult to change the keys. Changing the keys +is necessary to handle scenarios like giving access to the shared +toplevels to a new user with a new key pair. Otherwise the +symmetric encryption key +needs to be distributed to every user, and re-distributed +if it ever changes, and this is cumbersome. It would also be possible +to re-encrypt everything in the toplevel for every new user, but +that is laughably inefficient. However, it would be acceptably +simple to support the scenario of distributing the symmetric encryption key to +every user, if the backup admin thinks storing it on the backup +server even in encrypted form is too risky. + +I have removed [[data signing|signing]] from this spec, on the suggestion of +Daniel Kahn Gilmor. Data signing will be dealt with separately. + +I am going to assume that any public keys being used are generated +by the user, not by obnam. + +I am not an encryption expert. I will not be implementing my own +encryption code, and do not even want to choose the specific +algorithms or key formats. I will be using GnuPG for all encryption +operations, because it is well-known and well-respected, and lets +me outsource all thinking. + + +Implementation outline +---------------------- + +General repository I/O operations (these correspond to the +`mkdir`, `write_file`, and `cat` operations in the Obnam VFS layer): + + def repo_mkdir(pathname): + # create a new directory + + def repo_write_file(pathname, contents): + # write a file to the repository (pathname is relative to repo) + + def repo_read_file(pathname): + # read contents of a file in the repository (name is relative to repo) + +General encryption routines: + + def generate_symmetric_key(): + # return N random bits to be used as a symmetric encryption key + + def encrypt_with_symmetric_key(data, symmetric_key): + # return data encrypted using symmetric encryption + + def decrypt_with_symmetric_key(encrypted, symmetric_key): + # return data after it has been decrypted using symmetric encryption + + def encrypt_with_pubkeys(data, pubkeys): + # return data after it has been encrypted for all of the given + # public keys + + def decrypt_with_secret_key(encrypted, secret_key): + # decrypt encrypted data using a secret key; this will fail unless + # the data was encrypted using the public key corresponding to the + # the secret key + +Keyring handling in memory: + + def create_empty_keyring(): + ... + + def add_to_keyring(keyring, key): + ... + + def keyring_contains(keyring, key): + ... + + def remove_from_keyring(keyring, key): + ... + + def encode_keyring(keyring): + # Return form of keyring that can be stored on disk. + + def decode_keyring(encoded): + # Inverse of encode_keyring. + +Create a new toplevel: + + def create_toplevel(name, pubkeys): + repo_mkdir(name) + + symmetric_key = generate_symmetric_key() + encrypted = encrypt_with_pubkeys(symmetric_key, pubkeys) + repo_write_file(name + '/key', encrypted) + + keyring = create_empty_keyring() + for pubkey in pubkeys: + add_to_keyring(keyring, pubkey) + encoded = encode_keyring(keyring) + encrypted = encrypt_symmetric(encoded, symmetric_key) + repo_write_file(name + '/userkeys', encrypted) + +Reading and writing files in a toplevel: + + def get_symmetric_key(toplevel, secret_key): + encoded = repo_read_file(toplevel + '/key') + return decrypt_with_secret_key(encoded, secret_key) + + def toplevel_read_file(toplevel, filename, secret_key): + symmetric_key = get_symmetric_key(toplevel, secret_key) + encoded = repo_read_file(toplevel + '/' + filename) + return decrypt_with_symmetric_key(encoded, symmetric_key) + + def toplevel_write_file(toplevel, filename, cleartext, secret_key): + symmetric_key = get_symmetric_key(toplevel, secret_key) + encoded = encrypt_with_symmetric_key(cleartext, symmetric_key) + repo_write_file(toplevel + '/' + filename, encoded) + +Manage keys for a toplevel: + + def read_keyring(toplevel, name, secret_key): + encoded = toplevel_read_file(toplevel, name, secret_key) + return decode_keyring(encoded) + + def write_keyring(toplevel, name, keyring, secret_key): + encoded = encode_keyring(keyring) + toplevel_write_file(toplevel, name, encoded, secret_key) + + def add_to_userkeys(toplevel, public_key, secret_key): + userkeys = read_keyring(toplevel, 'userkeys', secret_key) + if not keyring_contains(userkeys, public_key): + add_to_keyring(userkeys, public_key) + write_keyring(toplevel, 'userkeys', userkeys, secret_key) + + def remove_from_userkeys(toplevel, public_key, secret_key): + userkeys = read_keyring(toplevel, 'userkeys', secret_key) + if keyring_contains(userkeys, public_key): + remove_from_keyring(userkeys, public_key) + write_keyring(toplevel, 'userkeys', userkeys, secret_key) + +Repository client management: + + def add_client(client_public_key, admin_secret_key): + add_to_userkeys('metadata', client_public_key, admin_secret_key) + add_to_userkeys('clientlist', client_public_key, admin_secret_key) + add_to_userkeys('chunks', client_public_key, admin_secret_key) + add_to_userkeys('chunksums', client_public_key, admin_secret_key) + # client will add itself to the clientlist and create its own toplevel + + def remove_client(client_public_key, admin_secret_key): + # client may remove itself, since it has access to the symmetric keys + # we assume the client-specific toplevel has already been removed + remove_from_userkeys('chunksums', client_public_key, admin_secret_key) + remove_from_userkeys('chunks', client_public_key, admin_secret_key) + remove_from_userkeys('clientlist', client_public_key, admin_secret_key) + remove_from_userkeys('metadata', client_public_key, admin_secret_key) + + +Hooks in Obnam +-------------- + +Obnam's Repository class needs to have a pair of hooks for modifying +data before it gets written to the repository, and after it has been +read. These modifications should be each other's inverse functions. +Apart from encryption, these hooks could be used for error correction +codes for data in the store, and perhaps other things. The repository +should just provide and call the hooks, and not otherwise concern +itself with encryption. + +These hooks are not needed at the VFS layer, since it is not necessary +to decrypt live data, nor encrypt data that is being restored. + +The hooks correspond to `create_toplevel`, `toplevel_read_file`, +and `toplevel_write_file` above. However, to allow chains of callbacks +for the hooks, instead of the encryption callback writing the +data out to the repository, it should return it instead. The next +callback will get the encrypted data, and add, say, error correction +codes to it. Finally, when all callbacks are done, the encrypted and +error-corrected blob gets written to the repository. + +Thus, Repository should provide the following hooks: + +* `repo-create-toplevel(name)`: called whenever the repository has created + a new toplevel directory +* `repo-write(toplevel, filename, data)`: called by the repository + prepare data to be written to the repository +* `repo-read(toplevel, filename, data)`: called by repository for data that + has been read from the repository + +The hook subsystem needs to have a way to order callbacks, and for +each callback to return a modified form of the data for the next +callback to process (instead of the next callback processing the +original data). + +Callback ordering is important so that encryption always happens +before ECC encoding: there's no point in ECC if it happens before +encryption. + +Since the universe of likely Obnam plugins is small, and it can be +assumed that plugin authors co-operate, we can achieve ordering +most simply by having an optional integer _order_ argument to +the hook registration method. + + def add_callback(self, name, callback, order=None): + +Any callbacks without an explicit order will be put at the end +of the callback chain, and all others to the beginning, sorted +by the order argument into increasing order. + +We can the arrange for the callback registrations for encryption +and ECC to use appropriate ordering: + + hooks.add_callback('repo-write', encrypt, order=1000) + hooks.add_callback('repo-write', ecc, order=2000) + +(Reverse the ordering for `repo-read`, of course.) + +Arranging for a hook to be able to modify the data is a bit +trickier. Ideally, it could just return the new data, but the +general purpose nature of the hook subsystem means that it does +not know what the arguments for a hook are. + +Thanks +------ + +Thank you to Richard Braakman, Peter Palfrader, Jaakko Niemi, +and Daniel Kahn Gillmor for feedback. Any problems that remain +are my fault. diff --git a/faq.mdwn b/faq.mdwn new file mode 100644 index 0000000..46adeda --- /dev/null +++ b/faq.mdwn @@ -0,0 +1,3 @@ +[[!meta title="Obnam FAQ"]] + +[[!inline pages="page(obnam/faq/*)" archive=yes template=titlepage]] diff --git a/faq/checksum-safety.mdwn b/faq/checksum-safety.mdwn new file mode 100644 index 0000000..cab4802 --- /dev/null +++ b/faq/checksum-safety.mdwn @@ -0,0 +1,35 @@ +[[!meta title="Checksum collisions and safety"]] + +Obnam is using the MD5 checksum algorithm for recognising duplicate +data chunks. MD5 has a reputation for being unsafe: people have +constructed files that are different, but result in the same MD5 +checksum. This is true. + +Every checksum algorithm can have collisions. Changing Obnam to, say, +SHA1, SHA2, or the as yet unreleased SHA3 would not remove the chance +of collisions. It would reduce the chance of accidental collisions, +but the chance of those is already so small with MD5 that it can be +disregarded. Or put in another way, if you care about the chance of +accidental MD5 collisions, you should be caring about accidental SHA1, +SHA2, or SHA3 collisions as well. + +Apart from accidental collisions, there are two cases where you should +worry about checksum collisions (regardless of algorithm). + +First, if you're into researching checksum collisions, you're likely +to have files that cause checksum collisions, and in that case, if you +restore after a catastrophe, you probably want to get the files back +intact, rather having Obnam confuse one with the other. + +Second, if you have an enemy who wishes to corrupt your backed up +data, they may replace some of the backed up data with other data that +has the same checksum. This way, when you restore, your data is +corrupted without Obnam noticing. + +For both of these cases, you can instruct Obnam to **verify** that +chunks of data with the same checksum actually are the same data, +instead of relying on the checksum alone. This is as safe as it can +be, but it has a big performance impact. It causes Obnam to have to +read from the repository (possibly downloading it from your backup +server) all the data you are backing up. You'll still benefit from the +de-duplication, however, so your repository size will be smaller. diff --git a/faq/dedup.mdwn b/faq/dedup.mdwn new file mode 100644 index 0000000..d70ad20 --- /dev/null +++ b/faq/dedup.mdwn @@ -0,0 +1,68 @@ +[[!meta title="De-duplication does not work very well for me"]] + +On some kinds of large files, Obnam's de-duplication does not work +very well, even though it should. For example, MySQL dump files +from successive days are mostly the same data, but Obnam does badly +with them. Below is an explanation of how the Obnam de-duplication +works, and why it works badly in some cases. + +Obnam does de-duplication by splitting up file data into chunks, +and storing those individually. If two files have the same data, +Obnam re-uses the already backed up chunk. So far, so good. However, +due to performance issues, Obnam currently only notices chunks when +they start at integer multiples of the chunk size. + +For example, assume a chunk size of 4 bytes, and the following two +files: + + file 1: AAAABBBBCCCC + file 2: BBBBCCCCAAAA + +In this case, Obnam will easily notice that there are three chunks +("AAAA", "BBBB", and "CCCC"), and will store them only once in the +backup repository. However, consider the following file: + + file 3: xAAAABBBBCCCC + +File 3 is identical to file 1, except that a new byte has been +inserted into the file. This makes Obnam look at file 3 as four +chunks: "xAAA", "ABBB", "BCCC", and "C". None of these chunks +match the chunks already in the backup repository. Thus, Obnam +thinks they're all new. + +There is no technical reason why Obnam could not notice that file 3 +only has one inserted byte. However, doing so would require a very +large number of lookups in the repository, and thus would be quite +slow. There may be better ways of noticing the minute difference, +and perhaps someday one of them will be implemented in Obnam. + +Note that Obnam does not do a "diff" (or "xdelta" or other such +approach) to notice differences between successive versions of +files. Doing so would make backup generations be dependent on each +other, and re-introduce "full" versus "incremental" backups in a +way that is not acceptable. + +With SQL dumps of databases, there are often small changes at +the beginning of of the file, or in the middle of the file, which +makes Obnam's de-duplication work very badly, even if the data as +such has only changed a tiny bit. + +Unfortunately, I don't know of a trick that would make the SQL +dumps work better with Obnam. In any case, you should not have +to munge your live data to suit Obnam: Obnam needs to be able +to deal with whatever data you have. Until Obnam's de-duplication +becomes better, though, perhaps someone would have a workaround? + +The best idea, untested, I have is to keep the first SQL dump, +in the live data, and then do a new dump before each backup, diff +the two dumps, delete the new dump, and then run the backup. This +way, each successive Obnam backup generation will have two files +(the original SQL dump, and the diff), and you'll need to apply +the diff to get the real dump you need to restore your database. +Does that make sense to anyone? + +See also the mailing list thread: + +* [Start](http://vlists.pepperfish.net/pipermail/obnam-flarn.net/2013-January/000510.html) +* [Lars's explanation](http://vlists.pepperfish.net/pipermail/obnam-flarn.net/2013-January/000511.html) +* [Workaround with rdiff](http://vlists.pepperfish.net/pipermail/obnam-flarn.net/2013-January/000516.html) diff --git a/faq/name.mdwn b/faq/name.mdwn new file mode 100644 index 0000000..c20a0b8 --- /dev/null +++ b/faq/name.mdwn @@ -0,0 +1,9 @@ +[[!meta title="What does the name 'Obnam' mean?"]] + +Back in 2006, when Obnam started, we wanted to have a name that was +short, reasonably pronounceable by major cultures, and did not have +too many Google hits. After much trying, we failed to come up with +anything good. However, without a name, you can't even start the +project, because what will you call its source directory? So we +chose the OBligatory NAMe. + diff --git a/faq/private-key-for-backup.mdwn b/faq/private-key-for-backup.mdwn new file mode 100644 index 0000000..d84cde5 --- /dev/null +++ b/faq/private-key-for-backup.mdwn @@ -0,0 +1,8 @@ +[[!meta title="Why does Obnam want my PGP private key passphrase?"]] + +When doing an incremental backup, Obnam reads metadata back from the backup +repository to determine what it needs to back up. For example, names of +files and when they were last modified. The metadata is also encrypted, +and Obnam needs to decrypt it to be able to do an incremental backup. +That is why Obnam needs the passphrase. + diff --git a/faq/profiling.mdwn b/faq/profiling.mdwn new file mode 100644 index 0000000..2732708 --- /dev/null +++ b/faq/profiling.mdwn @@ -0,0 +1,8 @@ +[[!meta title="How can I profile obnam if things get slow?"]] + +If the `OBNAM_PROFILE` environment variable contiains a file name, the profiling data gets stored there, and can be viewed later with `obnam-viewprof`: + + OBNAM_PROFILE=obnam.prof obnam ... + obnam-viewprof obnam.prof | less + +Performance issues that are not related to a particular setup can also be observed using the [[obnam-benchmark|obnam-benchmark.1.txt]] tool. diff --git a/faq/restore-on-other-client.mdwn b/faq/restore-on-other-client.mdwn new file mode 100644 index 0000000..8bee78e --- /dev/null +++ b/faq/restore-on-other-client.mdwn @@ -0,0 +1,5 @@ +[[!meta title="How do I restore on another computer?"]] + +Use `--client-name` and set it to the name of the computer that made +the original backup. + diff --git a/faq/safe-repo-sharing.mdwn b/faq/safe-repo-sharing.mdwn new file mode 100644 index 0000000..9bbbef9 --- /dev/null +++ b/faq/safe-repo-sharing.mdwn @@ -0,0 +1,11 @@ +[[!meta title="Can Alice and Bob share repository without sharing file data?"]] + +This doesn't work particularly well. Obnam does not currently support it +at all. On a conceptual level, it could work, by turning all de-duplication +off, and having Alice and Bob encrypt their own chunks only with their +own, separate keys. + +However, it would be simpler to just have a separate repository for +each of them. That would also prevent either of them to remove the +other's files. + diff --git a/faq/why-nonstd-format.mdwn b/faq/why-nonstd-format.mdwn new file mode 100644 index 0000000..ee07440 --- /dev/null +++ b/faq/why-nonstd-format.mdwn @@ -0,0 +1,17 @@ +[[!meta title="Why is Obnam not using standard file formats, such as tar or cpio?"]] + +The standard file formats do not support the kind of advanced features +that Obnam is built for. For example, block level de-duplication, +or abolishing the difference between full and incremental generations. +It would be possible to build those features on top, say, tar, but +it would mean the tar is just a container for odd bits of data, which +still requires a custom program to restore from. + +If you're worried that you might lose your copy of Obnam, and be unable +to restore, two points: + +* You can store a copy of Obnam (in source and binary versions) in your + backup repository. +* Obnam is free software, and it's in Debian. You're pretty much guaranteed + to be able to find a copy and build a new binary. + @@ -1,4 +1,5 @@ -[[!meta title="Obnam backup program"]] +[[!meta title="Obnam - backup program"]] +[[!tag program]] Obnam is an **easy, secure backup program.** Backups can be stored on local hard disks, @@ -40,8 +41,8 @@ Documentation ------------- * The full manual (currently a work in progress): available as - [a web page](http://code.liw.fi/obnam/manual/manual.html) and - [PDF](http://code.liw.fi/obnam/manual/manual.pdf). + [a web page](http://code.liw.fi/obnam/manual/obnam-manual.en.html) and + [PDF](http://code.liw.fi/obnam/manual/obnam-manual.en.pdf). - The Obnam test suite is also meant to be useful for the more technical users of Obnam to read: [web page](http://code.liw.fi/obnam/yarns.html), diff --git a/liw-bitcoin-qr-obnam.png b/liw-bitcoin-qr-obnam.png Binary files differnew file mode 100644 index 0000000..528744c --- /dev/null +++ b/liw-bitcoin-qr-obnam.png diff --git a/locking.mdwn b/locking.mdwn new file mode 100644 index 0000000..bceafbd --- /dev/null +++ b/locking.mdwn @@ -0,0 +1,158 @@ +[[!meta title="Locking in Obnam repositories"]] + +Obnam supports multiple clients backing up to the same repository at +the same time. It further allows the clients to share data, so that +if Alice has backed up a large file, and Bob has the same file, he +does not need to actually upload the file a second time to the backup +repository. See [[Obnam on-disk data structures|ondisk]] for more details. + +For Alice and Bob not to overwrite each others' changes to shared +data, there needs to be some locking, so that only one of them can +modify the data structures at the same time. + +The repository has six different classes of data: + +* metadata about the repository (shared, needs locking) + - currently this is the version number of the repository format +* a list of clients (shared, needs locking) +* file data chunks (shared, but does not need locking) +* encryption keys for chunks (shared, needs locking) +* shared metadata about file chunks (shared, needs locking) +* per-client file metadata (not shared, but does need locking) + +Another way to consider the data is as follows: + +* shared chunk storage +* shared and per-client B-trees, and chunk encryption keys +* repository metadata + +We can discuss the locking for each of these groups separately + +Shared chunk storage +-------------------- + +The shared chunk storage consists of a directory tree, where chunks are +stored in files named after the chunk identifier. When a client is +making a backup, they only ever add chunks, and they do that by picking +a chunk identifier by random, and creating a file with that name. If +the file creation fails, the chunk id was already in use, and the client +picks a new one by random. + +This means that the filesystem takes care of atomicity, and Obnam does +not need to care about that. Adding chunks is a lock-free operation even +when multiple clients are backing up at the same time. + +Read-only access to the chunks also does not require any locking, +obviously. + +The operations that can delete chunks (forget, fsck) can also do without +locking the chunk storage. They lock the relevant B-trees, so that other +clients do not at the same time try to change the state of the repository, +and this protects the shared chunks as well. + +When encryption is used, the encryption keys need to be updated for new +clients, and this needs locking, but this is handled separately from the +actual chunks. + +Shared and per-client B-trees, and chunk encryption keys +-------------------------------------------------------- + +There are several shared B-trees: the list of clients, the mapping of +chunk identifiers to chunk sums, and the mapping of chunk sums to +chunk ids. There is one B-tree per client. From a locking point of +view, they can be treated identically: if nothing else, even the +per-client tree can be considered shared, when the backup server is +doing a repository-wide fsck, for example. + +The encryption keys for chunk storage are not a B-tree, but they can be +handled as if they chunks directory is a B-tree, from a locking point of +view. + +Read-only access to the B-trees should not be locked. This is not entirely +safe: Alice may do read-only access (e.g., during a restore), while +Bob is doing a destructive operation (e.g., forget). However, the +read-only client can treat any failures due to concurrent access +the same way it treats other failures: it can try to restore as much +as it can, or it can give up. Locking does not help against bad sectors +on the repository. Not having to lock for read-only access will allow +faster access and avoids having a long-running read-only operation locking +out new backups. + +Destructive access will need locking. Thus, each B-tree will have a +lock file at its root (called 'lock'): existence of the file indicates +the tree is locked. Only one writer can have a lock at the same time. + +The client list B-tree only needs to be changed when a client is added +or removed from the tree, or its encryption details are changed. The +other shared B-trees need to be changed during every backup run of every +client. The per-client B-trees need to be locked during backup and forget +operations for that client. All B-trees will need to be locked while +an fsck runs. (Per-client B-trees need to be locked to protect against +concurrent backup, forget, and fsck operations, for example.) + +Each B-tree is locked separately, by creating a file called `lock` +inside its directory. + +Repository metadata +------------------- + +The repository metadata (the directory `metadata` at the root of +the repository, not to be confused with the similarly named files inside +B-tree directories), may need to be locked against writes. For example, +a future version of Obnam may provide a way to upgrade the repository +format to a new version, and this requires locking against any changes +to the repository. + +The repository metadata will be considered locked implicitly when the +client list B-tree is locked. + +Avoiding deadlocks and reducing lock contention on shared stuff +--------------------------------------------------------------- + +To avoid deadlocks, every client must create locks in the same order. + +1. The client list B-tree. +1. Any per-client B-trees, in the order of their directory basenames. +1. Any other shared B-trees, in the order of their directory basenames. + +(Directory basenames are ordered byte by byte, using unsigned bytes.) + +There are several scenarios in which a client may require a lock: + +* adding a new client + - lock client list, add new client to it + - create, then lock per-client B-tree + - lock the shared directories, then add the client's encryption key to them + - unlock everything +* making backups for a client + - lock per-client B-tree, add new metadata + - lock chunklist and chunksums B-trees, update with info about new chunks, + unlock (possibly do this several times, for checkpoints) + - unlock per-client B-tree +* forgetting backups for a client + - lock per-client B-tree, remove generations, gather list of chunks + that may need to be deleted, unlock + - lock chunklist and chunksums, update about chunks that aren't used + by the client anymore, possibly remove chunks from disk, unlock +* fsck for repository + - lock everything, fsck, unlock everything + +When making backups, a client will pool its updates to the shared B-trees +until it is ready to do all of them in a batch, typically at the end +of a backup, or at a checkpoint generation. This way, it can keep +the shared lock for only a short while. If the client crashes before +it can update the shared B-trees, it is unfortunate, but not +catastrophic: it will have stored some chunks in the repository that +cannot be used for de-duplication, but no actual data is lost. + +Reliability and scalability testing of locking +---------------------------------------------- + +The reliability of locking will be tested by writing a script to do +small backup generations, and running it concurrently many times +against the same repository. The script will verify that each client +can restore every generation successfully after all generations have +been backed up. Also, repository wide fsck will be run. + +The amount of data to back up in each generation shall vary for +each client, so that they are less likely to get into lockstep. diff --git a/obnam.1.txt b/obnam.1.txt new file mode 100644 index 0000000..c8b23eb --- /dev/null +++ b/obnam.1.txt @@ -0,0 +1,810 @@ +OBNAM(1) OBNAM(1) + + + +NAME + obnam - make, restore, and manipulate backups + +SYNOPSIS + obnam [--always-restore-setuid] [--checkpoint=SIZE] [--chunk-size=SIZE] + [--chunkids-per-group=NUM] [--client-name=CLIENT-NAME] + [--compress-with=PROGRAM] [--config=FILE] [--crash-limit=COUNTER] + [--critical-age=AGE] [--deduplicate=MODE] [--dump-config] + [--dump-memory-profile=METHOD] [--dump-repo-file-metadata] + [--dump-setting-names] [--encrypt-with=ENCRYPT-WITH] + [--exclude=EXCLUDE] [--exclude-caches] [--exclude-from=FILE] + [--fsck-fix] [--fsck-ignore-chunks] [--fsck-ignore-client=NAME] + [--fsck-last-generation-only] [--fsck-skip-dirs] [--fsck-skip-files] + [--fsck-skip-generations] [--fsck-skip-per-client-b-trees] + [--fsck-skip-shared-b-trees] [--fuse-opt=FUSE] + [--generate-manpage=TEMPLATE] [--generation=GENERATION] [-h] [--help] + [--help-all] [--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH] + [--idpath-skip=IDPATH-SKIP] [--keep=KEEP] [--key-details] + [--keyid=KEYID] [--leave-checkpoints] [--list-config-files] + [--lock-timeout=TIMEOUT] [--log=FILE] [--log-keep=N] + [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE] + [--lru-size=SIZE] [--memory-dump-interval=SECONDS] + [--no-always-restore-setuid] [--no-default-configs] + [--no-dump-repo-file-metadata] [--no-exclude-caches] [--no-fsck-fix] + [--no-fsck-ignore-chunks] [--no-fsck-last-generation-only] + [--no-fsck-skip-dirs] [--no-fsck-skip-files] + [--no-fsck-skip-generations] [--no-fsck-skip-per-client-b-trees] + [--no-fsck-skip-shared-b-trees] [--no-key-details] + [--no-leave-checkpoints] [--no-one-file-system] [--no-pretend] + [--no-dry-run] [--no-no-act] [--no-pure-paramiko] [--no-quiet] + [--no-small-files-in-btree] [--no-strict-ssh-host-keys] [--no-verbose] + [--no-weak-random] [--node-size=SIZE] [--one-file-system] + [--output=FILE] [--pretend] [--dry-run] [--no-act] + [--pretend-time=TIMESTAMP] [--pure-paramiko] [--quiet] [-r=REPOSITORY] + [--repository=REPOSITORY] [--root=ROOT] [--sftp-delay=SFTP-DELAY] + [--small-files-in-btree] [--ssh-command=EXECUTABLE] + [--ssh-host-keys-check=VALUE] [--ssh-key=FILENAME] + [--ssh-known-hosts=FILENAME] [--strict-ssh-host-keys] + [--symmetric-key-bits=BITS] [--testing-fail-matching=REGEXP] [--to=TO] + [--trace=TRACE] [--upload-queue-size=SIZE] [--verbose] + [--verify-randomly=N] [--version] [--warn-age=AGE] [--weak-random] + + obnam [options] add-key [CLIENT-NAME]... + obnam [options] backup [DIRECTORY]... + obnam [options] client-keys + obnam [options] clients + obnam [options] diff [GENERATION1]GENERATION2 + obnam [options] dump-repo + obnam [options] force-lock + obnam [options] forget [GENERATION]... + obnam [options] fsck + obnam [options] generations + obnam [options] genids + obnam [options] help + obnam [options] help-all + obnam [options] list-keys + obnam [options] list-toplevels + obnam [options] ls [FILE]... + obnam [options] mount [ROOT] + obnam [options] nagios-last-backup-age + obnam [options] remove-client [CLIENT-NAME]... + obnam [options] remove-key [CLIENT-NAME]... + obnam [options] restore [DIRECTORY]... + obnam [options] verify [DIRECTORY]... + +DESCRIPTION + obnam makes, restores, manipulates, and otherwise deals with backups. + It can store backups on a local disk or to a server via sftp. Every + backup generation looks like a fresh snapshot, but is really incremen- + tal: the user does not need to worry whether it's a full backup or not. + Only changed data is backed up, and if a chunk of data is already + backed up in another file, that data is re-used. + + The place where backed up data is placed is called the backup reposito- + ry. A repository may be, for example, a directory on an sftp server, + or a directory on a USB hard disk. A single repository may contain + backups from several clients. Their data will intermingle as if they + were using separate repositories, but if one client backs up a file, + the others may re-use the data. + + obnam command line syntax consists of a command possibly followed by + arguments. The commands are list below. + + o backup makes a new backup. The first time it is run, it makes a + full backup, after that an incremental one. + + o restore is the opposite of a backup. It copies backed up data + from the backup repository to a target directory. You can re- + store everything in a generation, or just selected files. + + o clients lists the clients that are backed up to the repository. + + o generations lists every backup generation for a given client, + plus some metadata about the generation. + + o genids lists the identifier for every backup generation for a + given client. No other information is shown. This can be use- + ful for scripting. + + o ls lists the contents of a given generation, similar to ls -lAR. + + o verify compares data in the backup with actual user data, and + makes sures they are identical. It is most useful to run imme- + diately after a backup, to check that it actually worked. It + can be run at any time, but if the user data has changed, veri- + fication fails even though the backup is OK. + + o forget removes backup generations that are no longer wanted, so + that they don't use disk space. Note that after a backup gener- + ation is removed the data can't be restored anymore. You can + either specify the generations to remove by listing them on the + command line, or use the --keep option to specify a policy for + what to keep (everything else will be removed). + + o fsck checks the internal consistency of the backup repository. + It verifies that all clients, generations, directories, files, + and all file contents still exists in the backup repository. It + may take quite a long time to run. + + o force-lock removes a lock file for a client in the repository. + You should only force a lock if you are sure no-one is accessing + that client's data in the repository. A dangling lock might + happen, for example, if obnam loses its network connection to + the backup repository. + + o client-keys lists the encryption key associated with each + client. + + o list-keys lists the keys that can access the repository, and + which toplevel directories each key can access. Some of the + toplevel directories are shared between clients, others are spe- + cific to a client. + + o list-toplevels is like list-keys, but lists toplevels and which + keys can access them. + + o add-key adds an encryption key to the repository. By default, + the key is added only to the shared toplevel directories, but it + can also be added to specific clients: list the names of the + clients on the command line. They key is given with the --keyid + option. Whoever has access to the secret key corresponding to + the key id can access the backup repository (the shared + toplevels plus specified clients). + + o remove-key removes a key from the shared toplevel directories, + plus any clients specified on the command line. + + o nagios-last-backup-age is a check that exits with non-zero re- + turn if a backup age exceeds a certain threshold. It is suit- + able for use as a check plugin for nagios. Thresholds can be + given the --warn-age and --critical-age options. + + o diff compares two generations and lists files differing between + them. Every output line will be prefixed either by a plus sign + (+) for files that were added, a minus sign (-) for files that + have been removed or an asterisk (*) for files that have + changed. If only one generation ID is specified on the command + line that generation will be compared with its direct predeces- + sor. If two IDs have been specified, all changes between those + two generations will be listed. + + o mount makes the backup repository available via a read-only FUSE + filesystem. This means you can look at backed up data using + normal tools, such as your GUI file manager, or command line + tools such as ls(1), diff(1), and cp(1). You can't make new + backups with the mount subcommand, but you can restore data eas- + ily. + + You need to have the FUSE utilities and have permission to use + FUSE for this to work. The details will vary between operating + systems; in Debian, install the package fuse and add yourself to + the fuse group (you may need to log out and back in again). + + Making backups + When you run a backup, obnam uploads data into the backup repository. + The data is divided into chunks, and if a chunk already exists in the + backup repository, it is not uploaded again. This allows obnam to deal + with files that have been changed or renamed since the previous backup + run. It also allows several backup clients to avoid uploading the same + data. If, for example, everyone in the office has a copy of the same + sales brochures, only one copy needs to be stored in the backup reposi- + tory. + + Every backup run is a generation. In addition, obnam will make check- + point generations every now and then. These are exactly like normal + generations, but are not guaranteed to be a complete snapshot of the + live data. If the backup run needs to be aborted in the middle, the + next backup run can continue from the latest checkpoint, avoiding the + need to start completely over. + + If one backup run drops a backup root directory, the older generations + will still keep it: nothing changes in the old generations just because + there is a new one. If the root was dropped by mistake, it can be + added back and the next backup run will re-use the existing data in the + backup repository, and will only back up the file metadata (filenames, + permissions, etc). + + Verifying backups + What good is a backup system you cannot rely on? How can you rely on + something you cannot test? The obnam verify command checks that data + in the backup repository matches actual user data. It retrieves one or + more files from the repository and compares them to the user data. + This is essentialy the same as doing a restore, then comparing restored + files with the original files using cmp(1), but easier to use. + + By default verification happens on all files. You can also specify the + files to be verified by listing them on the command line. You should + specify the full paths to the files, not relative to the current direc- + tory. + + The output lists files that fail verification for some reason. If you + verify everything, it is likely that some files (e.g., parent directo- + ries of backup root) may have changed without it being a problem. Note + that you will need to specify the whole path to the files or directo- + ries to be verified, not relative to the backup root. You still need + to specify at least one of the backup roots on the command line or via + the --root option so that obnam will find the filesystem, in case it is + a remote one. + + URL syntax + Whenever obnam accepts a URL, it can be either a local pathname, or an + sftp URL. An sftp URL has the following form: + + sftp://[user@]domain[:port]/path + + where domain is a normal Internet domain name for a server, user is + your username on that server, port is an optional port number, and path + is a pathname on the server side. Like bzr(1), but unlike the sftp URL + standard, the pathname is absolute, unless it starts with /~/ in which + case it is relative to the user's home directory on the server. + + See the EXAMPLE section for examples of URLs. + + You can use sftp URLs for the repository, or the live data (root), but + note that due to limitations in the protocol, and its implementation in + the paramiko library, some things will not work very well for accessing + live data over sftp. Most importantly, the handling of of hardlinks is + rather suboptimal. For live data access, you should not end the URL + with /~/ and should append a dot at the end in this special case. + + Generation specifications + When not using the latest generation, you will need to specify which + one you need. This will be done with the --generation option, which + takes a generation specification as its argument. The specification is + either the word latest, meaning the latest generation (also the de- + fault), or a number. See the generations command to see what genera- + tions are available, and what their numbers are. + + Policy for keeping and removing backup generations + The forget command can follow a policy to automatically keep some and + remove other backup generations. The policy is set with the + --keep=POLICY option. + + POLICY is comma-separated list of rules. Each rule consists of a count + and a time period. The time periods are h, d, w, m, and y, for hour, + day, week, month, and year. + + A policy of 30d means to keep the latest backup for each day when a + backup was made, and keep the last 30 such backups. Any backup matched + by any policy rule is kept, and any backups in between will be removed, + as will any backups older than the oldest kept backup. + + As an example, assume backups are taken every hour, on the hour: at + 00:00, 01:00, 02:00, and so on, until 23:00. If the forget command is + run at 23:15, with the above policy, it will keep the backup taken at + 23:00 on each day, and remove every other backup that day. It will al- + so remove backups older than 30 days. + + If backups are made every other day, at noon, forget would keep the 30 + last backups, or 60 days worth of backups, with the above policy. + + Note that obnam will only inspect timestamps in the backup repository, + and does not care what the actual current time is. This means that if + you stop making new backups, the existing ones won't be removed auto- + matically. In essence, obnam pretends the current time is just after + the latest backup when forget is run. + + The rules can be given in any order, but will be sorted to ascending + order of time period before applied. (It is an error to give two rules + for the same period.) A backup generation is kept if it matches any + rule. + + For example, assume the same backup frequence as above, but a policy of + 30d,52w. This will keep the newest daily backup for each day for thir- + ty days, and the newest weekly backup for 52 weeks. Because the hourly + backups will be removed daily, before they have a chance to get saved + by a weekly rule, the effect is that the 23:00 o'clock backup for each + day is saved for a month, and the 23:00 backup on Sundays is saved for + a year. + + If, instead, you use a policy of 72h,30d,52w, obnam would keep the last + 72 hourly backups, and the last backup of each calendar day for 30 + days, and the last backup of each calendar week for 52 weeks. If the + backup frequency was once per day, obnam would keep the backup of each + calendar hour for which a backup was made, for 72 such backups. In + other words, it would effectively keep the the last 72 daily backups. + + Sound confusing? Just wonder how confused the developer was when writ- + ing the code. + + If no policy is given, forget will keep everything. + + A typical policy might be 72h,7d,5w,12m, which means: keep the last 72 + hourly backups, the last 7 daily backups, the last 5 weekly backups and + the last 12 monthly backups. If the backups are systematically run on + an hourly basis, this will mean keeping hourly backups for three days, + daily backups for a week, weekly backups for a month, and monthly back- + ups for a year. + + The way the policy works is a bit complicated. Run forget with the + --pretend option to make sure you're removing the right ones. + + Using encryption + obnam can encrypt all the data it writes to the backup repository. It + uses gpg(1) to do the encryption. You need to create a key pair using + gpg --gen-key (or use an existing one), and then tell obnam about it + using the --encrypt-with option. + + Configuration files + obnam will look for configuration files in a number of locations. See + the FILES section for a list. All files are treated as if they were + one with the contents of all files catenated. + + The files are in INI format, and only the [config] section is used (any + other sections are ignored). + + The long names of options are used as keys for configuration variables. + Any setting that can be set from the command line can be set in a con- + figuration file, in the [config] section. + + For example, the options in the following command line: + + obnam --repository=/backup --exclude='.wav$' backup + + could be replaced with the following configuration file: + + [config] + repository: /backup + exclude: .wav$ + + (You can use either foo=value or foo: value syntax in the files.) + + The only unusual thing about the files is the way options that can be + used many times are expressed. All values are put in a single logical + line, separated by commas (and optionally spaces as well). For exam- + ple: + + [config] + exclude = foo, bar, \.mp3$ + + The above has three values for the exclude option: any files that con- + tain the words foo or bar anywhere in the fully qualified pathname, or + files with names ending with a period and mp3 (because the exclusions + are regular expressions). + + A long logical line can be broken into several physical ones, by start- + ing a new line at white space, and indenting the continuation lines: + + [config] + exclude = foo, + bar, + \.mp3$ + + The above example adds three exclusion patterns. + + Multiple clients and locking + obnam supports sharing a repository between multiple clients. The + clients can share the file contents (chunks), so that if client A backs + up a large file, and client B has the same file, then B does not need + to upload the large file to the repository a second time. For this to + work without confusion, the clients use a simple locking protocol that + allows only one client at a time to modify the shared data structures. + Locks do not prevent read-only access at the same time: this allows you + to restore while someone else is backing up. + + Sometimes a read-only operation happens to access a data structure at + the same time as it is being modified. This can result in a crash. It + will not result in corrupt data, or incorrect restores. However, you + may need to restart the read-only operation after a crash. + +OPTIONS + --always-restore-setuid + restore setuid/setgid bits in restored files, even if not root + or backed up file had different owner than user running restore + + --client-name=CLIENT-NAME + name of client (havelock) + + --compress-with=PROGRAM + use PROGRAM to compress repository with (one of none, deflate) + + --critical-age=AGE + for nagios-last-backup-age: maximum age (by default in hours) + for the most recent backup before statis is critical. Accepts + one char unit specifier (s,m,h,d for seconds, minutes, hours, + and days. + + --dump-repo-file-metadata + dump metadata about files? + + --generate-manpage=TEMPLATE + SUPPRESSHELP + + --generation=GENERATION + which generation to restore + + -h, --help + show this help message and exit + + --keep=KEEP + policy for what generations to keep when forgetting + + --lock-timeout=TIMEOUT + when locking in the backup repository, wait TIMEOUT seconds for + an existing lock to go away before giving up + + --no-always-restore-setuid + + + --no-dump-repo-file-metadata + + + --no-pretend, --no-dry-run, --no-no-act + + + --no-quiet + + + --no-verbose + + + --output=FILE + write output to FILE, instead of standard output + + --pretend, --dry-run, --no-act + do not actually change anything (works with backup, forget and + restore only, and may only simulate approximately real behavior) + + --quiet + be silent: show only error messages, no progress updates + + -r, --repository=REPOSITORY + name of backup repository + + --root=ROOT + what to backup + + --testing-fail-matching=REGEXP + development testing helper: simulate failures during backup for + files that match the given regular expressions + + --to=TO + where to restore + + --trace=TRACE + add to filename patters for which trace debugging logging hap- + pens + + --verbose + be verbose: tell the user more of what is going on and generally + make sure the user is aware of what is happening or at least + that something is happening and also make sure their screen is + getting updates frequently and that there is changes happening + all the time so they do not get bored and that they in fact get + frustrated by getting distracted by so many updates that they + will move into the Gobi desert to live under a rock + + --verify-randomly=N + verify N files randomly from the backup (default is zero, mean- + ing everything) + + --version + show program's version number and exit + + --warn-age=AGE + for nagios-last-backup-age: maximum age (by default in hours) + for the most recent backup before status is warning. Accepts one + char unit specifier (s,m,h,d for seconds, minutes, hours, and + days. + + Backing up + --checkpoint=SIZE + make a checkpoint after a given SIZE (1073741824) + + --deduplicate=MODE + find duplicate data in backed up data and store it only once; + three modes are available: never de-duplicate, verify that no + hash collisions happen, or (the default) fatalistically accept + the risk of collisions + + --exclude=EXCLUDE + regular expression for pathnames to exclude from backup (can be + used multiple times) + + --exclude-caches + exclude directories (and their subdirs) that contain a + CACHEDIR.TAG file (see + http://www.brynosaurus.com/cachedir/spec.html for what it needs + to contain, and http://liw.fi/cachedir/ for a helper tool) + + --exclude-from=FILE + read exclude patterns from FILE + + --leave-checkpoints + leave checkpoint generations at the end of a successful backup + run + + --no-exclude-caches + + + --no-leave-checkpoints + + + --no-one-file-system + + + --no-small-files-in-btree + + + --one-file-system + exclude directories (and their subdirs) that are in a different + filesystem + + --small-files-in-btree + put contents of small files directly into the per-client B-tree, + instead of separate chunk files; do not use this as it is quite + bad for performance + + Configuration files and settings + --config=FILE + add FILE to config files + + --dump-config + write out the entire current configuration + + --dump-setting-names + SUPPRESSHELP + + --help-all + show all options + + --list-config-files + SUPPRESSHELP + + --no-default-configs + clear list of configuration files to read + + Development of Obnam itself + --crash-limit=COUNTER + artificially crash the program after COUNTER files written to + the repository; this is useful for crash testing the applica- + tion, and should not be enabled for real use; set to 0 to dis- + able (disabled by default) + + --pretend-time=TIMESTAMP + pretend it is TIMESTAMP (YYYY-MM-DD HH:MM:SS); this is only use- + ful for testing purposes + + --sftp-delay=SFTP-DELAY + add an artificial delay (in milliseconds) to all SFTP transfers + + Encryption + --encrypt-with=ENCRYPT-WITH + PGP key with which to encrypt data in the backup repository + + --key-details + show additional user IDs for all keys + + --keyid=KEYID + PGP key id to add to/remove from the backup repository + + --no-key-details + + + --no-weak-random + + + --symmetric-key-bits=BITS + size of symmetric key, in bits + + --weak-random + use /dev/urandom instead of /dev/random to generate symmetric + keys + + Integrity checking (fsck) + --fsck-fix + should fsck try to fix problems? + + --fsck-ignore-chunks + ignore chunks when checking repository integrity (assume all + chunks exist and are correct) + + --fsck-ignore-client=NAME + do not check repository data for cient NAME + + --fsck-last-generation-only + check only the last generation for each client + + --fsck-skip-dirs + do not check anything about directories and their files + + --fsck-skip-files + do not check anything about files + + --fsck-skip-generations + do not check any generations + + --fsck-skip-per-client-b-trees + do not check per-client B-trees + + --fsck-skip-shared-b-trees + do not check shared B-trees + + --no-fsck-fix + + + --no-fsck-ignore-chunks + + + --no-fsck-last-generation-only + + + --no-fsck-skip-dirs + + + --no-fsck-skip-files + + + --no-fsck-skip-generations + + + --no-fsck-skip-per-client-b-trees + + + --no-fsck-skip-shared-b-trees + + + Logging + --log=FILE + write log entries to FILE (default is to not write log files at + all); use "syslog" to log to system log, or "none" to disable + logging + + --log-keep=N + keep last N logs (10) + + --log-level=LEVEL + log at LEVEL, one of debug, info, warning, error, critical, fa- + tal (default: info) + + --log-max=SIZE + rotate logs larger than SIZE, zero for never (default: 0) + + --log-mode=MODE + set permissions of new log files to MODE (octal; default 0600) + + Mounting with FUSE + --fuse-opt=FUSE + options to pass directly to Fuse + + Peformance + --dump-memory-profile=METHOD + make memory profiling dumps using METHOD, which is one of: none, + simple, meliae, or heapy (default: simple) + + --memory-dump-interval=SECONDS + make memory profiling dumps at least SECONDS apart + + Performance tweaking + --chunk-size=SIZE + size of chunks of file data backed up (default: 1048576) + + --chunkids-per-group=NUM + encode NUM chunk ids per group (1024) + + --idpath-bits=IDPATH-BITS + chunk id level size + + --idpath-depth=IDPATH-DEPTH + depth of chunk id mapping + + --idpath-skip=IDPATH-SKIP + chunk id mapping lowest bits skip + + --lru-size=SIZE + size of LRU cache for B-tree nodes (default: 256) + + --node-size=SIZE + size of B-tree nodes on disk; only affects new B-trees so you + may need to delete a client or repository to change this for ex- + isting repositories (default: 262144) + + --upload-queue-size=SIZE + length of upload queue for B-tree nodes (default: 128) + + SSH/SFTP + --no-pure-paramiko + + + --no-strict-ssh-host-keys + + + --pure-paramiko + do not use openssh even if available, use paramiko only instead + + --ssh-command=EXECUTABLE + alternative executable to be used instead of "ssh" (full path is + allowed, no arguments may be added) + + --ssh-host-keys-check=VALUE + If "yes", require that the ssh host key must be known and cor- + rect to be accepted. If "no", do not require that. If "ask", the + user is interactively asked to accept new hosts. The default + ("ssh-config") is to rely on the settings of the underlying SSH + client + + --ssh-key=FILENAME + use FILENAME as the ssh RSA private key for sftp access (default + is using keys known to ssh-agent) + + --ssh-known-hosts=FILENAME + filename of the user's known hosts file (default: + /home/liw/.ssh/known_hosts) + + --strict-ssh-host-keys + DEPRECATED, use --ssh-host-keys-check instead + + Option values + The SIZE value in options mentioned above specifies a size in bytes, + with optional suffixes to indicate kilobytes (k), kibibytes (Ki), + megabytes (M), mebibyts (Mi), gigabytes (G), gibibytes (Gi), terabytes + (T), tibibytes (Ti). The suffixes are case-insensitive. + +EXIT STATUS + obnam will exit with zero if everything went well, and non-zero other- + wise. + +ENVIRONMENT + obnam will pass on the environment it gets from its parent, without + modification. It does not obey any unusual environment variables, but + it does obey the usual ones when running external programs, creating + temporary files, etc. + +FILES + /etc/obnam.conf + /etc/obnam/*.conf + ~/.obnam.conf + ~/.config/obnam/*.conf + Configuration files for obnam. It is not an error for any or + all of the files to not exist. + +EXAMPLE + To back up your home directory to a server: + + obnam backup --repository sftp://your.server/~/backups $HOME + + To restore your latest backup from the server: + + obnam restore --repository sftp://your.server/~/backups \ + --to /var/tmp/my.home.dir + + To restore just one file or directory: + + obnam restore --repository sftp://your.server/~/backups \ + --to /var/tmp/my.home.dir $HOME/myfile.txt + + Alternatively, mount the backup repository using the FUSE filesystem + (note that the --to option is necessary): + + mkdir my-repo + obnam restore --repository sftp://your.server/~/backups \ + --to my-repo + cp my-repo/latest/$HOME/myfile.txt + fusermount -u my-repo + + To check that the backup worked: + + obnam verify --repository sftp://your.server/~/backups \ + /path/to/file + + To remove old backups, keeping the newest backup for each day for + ten years: + + obnam forget --repository sftp://your.server/~/backups \ + --keep 3650d + + To verify that the backup repository is OK: + + obnam fsck --repository sftp://your.server/~/backups + + To view the backed up files in the backup repository using FUSE: + + obnam mount --to my-fuse + ls -lh my-fuse + fusermount -u my-fuse + +SEE ALSO + obnam comes with a manual in HTML and PDF forms. See + /usr/share/doc/obnam if you have Obnam installed system-wide, or in the + subdirectory manual in the source tree. + + cliapp(5) + + + + OBNAM(1) diff --git a/ondisk.mdwn b/ondisk.mdwn new file mode 100644 index 0000000..f760830 --- /dev/null +++ b/ondisk.mdwn @@ -0,0 +1,405 @@ +Obnam on-disk data structures +============================= + +> **Note:** This document does not really reflect reality, right now. +> Obnam is under development, and sometimes the document and the code +> change independently. The code is authoritiative, except sometimes +> this document changes as part of planning new features. Please report or +> fix any inconsistencies. Thank you for your patience. + +[[!toc startlevel=2 levels=2]] + +Obnam is a backup tool for making **backups to hard disks,** either +local ones or over the network. It does not work with tape drives, +or optical disks. The **backup server is stupid:** it does +not run any Obnam specific code, only ssh (sftp). + +The location where +backups are stored is called the **backup store**, and may be shared between +multiple co-operating clients. The backup store resides on a backup server. The +computers that own the data being backed up are the **clients**. + +Performance is obviously a consideration. It has many aspects: + +* run-time +* RAM +* disk space, both server and client +* disk bandwidth, both server and client +* network bandwidth, between server and client + +To achieve its performance goals, +Obnam uses the B-tree variant devised by Ohad Rodeh[1], also +used by btrfs. This document is describes how it does that. + + +Characteristics of the B-tree +----------------------------- + +The **Rodeh B-trees** are designed for shadowing, i.e., +**copy-on-write updates** +of tree nodes. Nodes are not modified, instead a new copy is created +to replace it. This allows an efficient way to copy a tree and to update +the copy, while keeping the original tree intact. +The two trees will share as many nodes as possible. +Obnam calls the collection of related trees a **forest.** + +The trees use **fixed-length binary strings as keys.** +For each key, a **variable-length binary string is stored as the value.** +The tree consists of interior nodes and leaf +nodes; all values are stored in leaves only. There is a maximum size for +nodes, which in some applications would be a raw disk block, but is +a separate file for Obnam. +Value strings can be as big as a node (minus a small header), but no bigger. + +Lookups and removals from the tree can be done using **ranges:** all keys +within a range are looked up, or removed from the tree. This is an important +optimization and opens up some interesting design possibilities. + + +File data (chunk) storage +------------------------- + +File data is not stored directly in B-trees, but externally. Data +is broken up into **chunks** of suitable size (say, 64 KiB) +and assigned identifiers, +which also act as filenames. Given a chunk id, its contents can be easily +retrieved. + +**Chunks are shared** within and across clients. +This allows clients to avoid +uploading data that another client has already uploaded, +or storing the same data twice. The same mechanism allows Obnam to +efficiently back up files that have moved, or that have been modified, +or both at the same time. +If a chunk of data already exists in the store, Obnam tries hard to avoid +having to back it up again. + +To back up a chunk, Obnam chooses a random, unused 64-bit identifier, +and uploads a file of that name. The next chunk uses the next +identifier, until it hits one that has been used, at which point +it picks a new random one. + +Chunks are managed using reference counts. We will cover these later, +after discussing other details of the store. See +section "Chunks, part 2" below. + +Overview of store forests +------------------------- + +The backup store consists of several forests: + +* client list: map client names to 64-bit identifiers +* chunk list: map chunk identifiers to checksums +* chunk checksums: map chunk checksum to identifiers +* per-client data: backup generations for each client + +Locking +------- + +Locking is done only for writes. +Reads are always allowed, even while write locks exist. +This allows race conditions between readers and writers, +but thanks to copy-on-write updates +those are no different from files getting corrupted or +deleted on the server by hardware failures, +and can be treated the same. + +Each forest is locked separately. +If more than one forest needs to be locked at the same time, +they are locked in an order sorted by the name of the forest, +using the C locale. +If a client fails to lock a particular forest, +it releases all the locks it holds, +and tries again. +This avoids deadlocks, +but allows starving. +The most likely reason for starving is too many clients sharing the +same store, and that needs to be solved by increasing backup server +performance, or having more stores. + +Locks are implemented as files, which are created atomically. +Each lock file contains the name of the host that holds it +(which might not be a backup client), +and the process id on that client, and the time of creating the lock. +If the time is very old, another client may decide to break the lock. +If the backup store is encryped, then also the contents of the +lock file is encrypted. + +To reduce lock congestion, each client attempts to keep a lock for as +short a time as possible. For per-client data, this means keeping the lock +for the duration of the backup. For shared forests, updates can be +spooled: the shared forest is used read-only until the end of the backup +run, or until a checkpoint, and updated then, as quickly as possible. + +Client list forest +---------------- + +The client list forest consists of a single tree. +Its key is consists of a tuple: + +* 128 bits of MD5 checksum of the name of the client +* 64 bits of client id (for hash collision avoidance) + +The client name is typically its host name, but might be anything. +It is up to the sysadmins of the clients to co-operate so that each +client has a unique name. + +It is unlikely that there will be checksum collisions for client names, +but it's easy to not have to worry about that. The client id will be +chosen randomly. + +Chunk list forest +----------------- + +The chunk list forest consists of a single tree. +Its key consists of a 64-bit integer containing a chunk identifier. +The value is the MD5 checksum for the chunk. + +This tree is needed so that it is possible to quickly find the +checksum of a chunk, given its identifier, when removing generations +from the per-client forest. + +Chunk checksum forest +--------------------- + +The chunk checksum forest uses a tuple as a key: + +* 128-bit MD5 checksum of the data in the chunk +* 64-bit chunk id +* 64-bit client id + +The chunk id is used to handle checksum collisions. +While MD5 collisions are not likely in general use, +they are certain for people who research cryptographic hashes, +so Obnam needs to be able to handle them. +When Obnam has a chunk it wants to back up, +it does a range lookup from (md5,0) through (md5,chunkd_id_max), +and then checks if the chunk is identical to any of the chunks +already in the store. +This checking is expensive, but safety may more important than +performance here. (Obnam may provide an option to disable the +data comparison check, and rely only on the hashes.) + +The value is empty. + +Per-client forest +--------------- + +The per-client forest has one tree per backup generation or checkpoint, +and represents a snapshot of the filesystem at a specific time, +or as close to a snapshot as Obnam can make it. Obnam does not +freeze the filesystem while the backup runs, so it cannot guarantee +a true snapshot. + +The forest uses a tuple as a key: + +* 8-bit prefix +* 64-bit main key +* 8-bit subkey type +* 64-bit subkey + +The prefix is one of the following: + +* 0 for filesystem metadata +* 1 for chunk references +* 2 for generation metadata + +**For prefix 0,** the main key is the file identifier. +See below for how they are generated. + +The subkey type and subkey and value are using as follows: + +[[!table data=""" +subkey type | subkey | value +0 | file-id | full pathname to file +1 | counter 0... | up to N chunk-ids +2 | 0 | number of chunk-ids stored +3 | 0... | inode fields, user/group name, xattr, ... +"""]] + +Subkey type 0 is used to handle hash collisions for the pathnames. +They are unlikely, but it is entirely unacceptable for Obnam to fail +if they happen. Each file has an identifier that is unique within the +generation: there is a one-to-one mapping between fully qualified pathnames +and file-ids within a generation (but not across generations). + +By default, the file-id is one half of +the 128-bit MD5 of the full pathname to the file. To check for collisions, +we see if the value for key (1, default-file-id, 0, file-id) is the full +pathname for the file we are interested in. If it is not, we generate a +new file-id by some deterministic approach, and do the lookup again, +and repeat this until we find a free file-id. +Note that the default-file-id in the lookup stays the same, it will +always be the half of the MD5 of the pathname. + +Subkey type 1 is used to store a list of chunks that represent the contents +of the file. Since a file can be quite large, we store more than one chunk-id +per value, to reduce the overhead a bit. Benchmarking will determine the +suitable number of chunk-ids to put in each value, but probably the size +of the value should not be larger than about a quarter of a node. +To avoid having to do a range lookup to find the next counter, when +appending new chunk ids to an existing list, we store the number of items +at subkey type 2 (subkey 0). + +Subkey type 2 is used for file metadata, such as inode data +(`st_mtime`, `st_size`, etc), the name of the owner and group for +the file, xattr fields, etc. To reduce space use, the most common +metadata is stored in subkey value 0, encoded in a special format. +Other subkey values are used for additional metadata, which also +allows for future expansion. + +**For prefix 1,** the rest of the key is used like this: + +* main key is the chunk-id +* subkey type is 0 +* subkey is file-id for file that uses the chunk + +For prefix 1, the value is always empty. Whenever Obnam backs up a +file, it adds the key (1, chunk-id, 0, file-id). If the file existed +in the previous generation, but not in the new generation, it removes +the key. + +**For prefix 2,** the main key is fixed as the hash of +the string "generation", and the subkey type is fixed at 0. +The subkey is one of the values in the following table. + +[[!table data=""" +subkey |value +0 |generation id +1 |timestamp for the start of the generation +2 |timestamp for the end of the generation +3 |boolean (0 or 1) for whether the generation is a checkpoint +"""]] + +Timestamps are expressed in UTC as seconds since the beginning of 1970 +(i.e, Unix epoch). + +Chunks, part 2 +-------------- + +When backing up, Obnam will do roughly this: + +* clone the tree for previous generation +* traverse the filesystem tree, adding and removing files from the new tree +* when backing up a file's chunks, look up each chunk in the store, and add it + if it is missing; after this, the id of the chunk is known +* add (1, chunk-id, 0, file-id) to the per-client forest, + (chunk-id) to the chunk list forest, + and (checksum, chunk-id, client-id) to the chunk checksum forest + for any new chunks it uses; the updates to the chunk list and checksum + forests can be batched to the end of the backup run +* remove (1, chunk-id, 0, file-id) from the per-client forest, + for every chunk for any file it removes from the new generation + (no need to remove anything from the chunk checksum forest, since + previous generations will still use the chunks) + +When removing a generation, Obnam will do roughly this: + +* look up every chunk-id used in that generation +* look up each chunk-id in every other generation +* if not found in any other generation, remove + (checksum, chunk-id, client-id) from the chunk checksum forest; + look up the checksum from the chunk list forest +* if there are no keys in the range (checksum, chunk-id, 0) through + (checksum, chunk-id, client_id_max) in the chunk checksum forest, + then remove (chunk-id) from chunk list forest, and + chunk file from disk +* remove the tree for the unwanted generation + + +Repository metadata +------------------- + +There is a need to store some metadata about the repository. +This will be done in the `metadata` directory. Initially, the +only piece of information is the version of the on-disk format, +expressed as a single integer, stored in the file `metadata/format`. + +Each version of Obnam will know which on-disk formats it supports. +With the `metadata/format` file it knows if it can handle a +particular format. + +Upgrades to newer formats will happen explicitly, not implicitly. + + +Encryption +---------- + +Obnam may optionally encrypt data in the backup store. +The design for this is unclear, but will need to solve at least +the following problems: + +* each client should have access only to its own data, but still + allow sharing of data across clients +* the server admin should be able to disable access for each client, + or anyone who's compromised the client's encryption key +* the server admin should be able to do some operations on any client, + such as expire generations, remove entire clients and their corresponding + chunks, verify correctness of the whole data store ("fsck"), or + restore data after the client has been entirely lost + + +On checksum algorithms +---------------------- + +This design uses MD5 checksums for everything. +It does not require the checksum algorithm to be cryptographically +secure, and is content with having a hash that is unlikely to result +in collisions, and only cares about the probability of collisions +for performance, not correctness. +The design handles collisions, and does not assume two pieces of data +are identical just because the hashes are the same. +A shorter or faster checksum algorithm may be used instead. +Or a cryptographically stronger one. +MD5 is not recommended for applications that rely on the security of +the hash, but this design does not rely on that. + +However, it may be quite expensive to verify that data chunks with +the same checksum are the same. It may require reading back the chunk +from the server. This may not be acceptable to all users. Some users +may be willing to live with the chance of a checksum collision. +Other users might not want to pay the price for de-duplication. +Thus, it may be best for obnam to provide a tri-state option: +full verification, rely on checksums, and no de-duplication. Further, +it may be a good idea to allow the user to specify the checksum +algorithm. + + +Comments? +--------- + +Edit this page directly, or mail me at <mailto:liw@liw.fi>. Thanks. + + +Author +------ + +Obnam is being developed by [Lars Wirzenius](http://liw.fi). +Feedback and patches are most welcome. + + +References +---------- + +1. Rodeh, Ohad: "B-trees, Shadowing, and Clones". IBM Haifa Research Labs. + <http://portal.acm.org/citation.cfm?id=1326544> + <http://www.cs.tau.ac.il/~ohadrode/papers/btree_TOS.pdf> + + +Changes +------- + +* 2010-05-04: Modified the way hash collisions for filenames are + handled, based on suggestion from Richard Braakman: store secondary + hash and filename with primary hash. +* 2010-07-11: Minor tweaks. +* 2010-11-11: Change tense to current, add some notes and minor fixes. +* 2010-11-11: Add design for removal of chunks, some considerations for + encryption, more details on how keys are used by various forests in + the store, locking, and other clarifications. +* 2010-11-11: Refer to hosts as clients, for clarity. +* 2010-11-12: Typo fixes and checksum verification tri-state option + suggestion from Richard Braakman. +* 2010-11-18: Add generation metadata to per-client forest. +* 2010-11-29: Add chunk list forest. + diff --git a/signing.mdwn b/signing.mdwn new file mode 100644 index 0000000..c03f044 --- /dev/null +++ b/signing.mdwn @@ -0,0 +1,7 @@ +[[!meta title="Signing backup data in Obnam"]] + +Obnam should sign data stored into the repository to protect against +malicious fake data, and against accidental corruption. + +The signing should be safe against replay attacks. + diff --git a/status.mdwn b/status.mdwn new file mode 100644 index 0000000..89c6981 --- /dev/null +++ b/status.mdwn @@ -0,0 +1,22 @@ +Status and support +------ + +Obnam is ready for production use. See [[NEWS]] for release notes, +and [[1.0]] for the 1.0 release announcement. + +If you find a bug or other problem, please see the +[[bug-reporting]] page and report it using e-mail +to the mailing list (see below). +Debian users may also report bugs to the Debian bug tracking +system. +Using the Obnam web pages for bug reporting is not a good idea, +since they are not a good way to conduct a discussion. + +* E-mail to the + [mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net). + +See [[contact page|contact]] for more instructions on the mailing list. + +This is a free software project. Lars as the author will provide +support as well as he can (but no guarantees), and users may support +themselves, or each other, or may buy support from whomever they like. @@ -1,39 +1,53 @@ html { background: white; + color: black; font-family: serif; - margin-left: 3em; - margin-right: 2em; - margin-top: 2em; + margin: 0; + padding: 0; + border: 0; +} + +div.pageheader { + background: #A87149; + color: white; + font-family: monospace; + margin-bottom: 2em; + text-align: right; +} + +div.pageheader a { + color: white; } form#searchform { font-family: monospace; text-align: right; + margin-bottom: 2em; } div.actions { font-family: monospace; text-align: right; + margin-bottom: 2em; } div.actions ul, div.actions li { display: inline; } -div.pageheader { - font-family: monospace; - margin-bottom: 2em; -} - div.pageheader span.title { display: block; + text-align: left; font-size: 200%; font-weight: bold; font-family: sans-serif; - margin-top: 0.5em; + margin-top: 1em; } div#pagebody { + clear: right; + background: white; + color: black; } div.pagefooter { diff --git a/tutorial.mdwn b/tutorial.mdwn new file mode 100644 index 0000000..dd99e36 --- /dev/null +++ b/tutorial.mdwn @@ -0,0 +1,158 @@ +[[!meta title="Obnam tutorial"]] + +This tutorial will be migrating to the full Obnam manual, +at <http://code.liw.fi/obnam/manual/>. This version is no longer +updated, see the link for the current version. + +[[!toc ]] + +Installation +------------ + +It is easiest to install Obnam on a Debian system. If you're running +Debian `wheezy` or a later release, Obnam is included. For `squeeze` +add the following line to your `/etc/apt/sources.list` file: + + deb http://code.liw.fi/debian squeeze main + +Then run the following commands as root: + +* `apt-get update` +* `apt-get install obnam` + +The commands will complain that the PGP key used to sign the archive +is not known to apt. You can either ignore this, or add the key from +<http://code.liw.fi/apt.asc> to your key. + +For other systems, you need to install from sources. See the `README` +file for instructions. + +Configuration +------------- + +Obnam does not require a configuration file, and you can configure +everything using command line options. You can, however, use a +configuration file: save it as `~/.obnam.conf` and +make it have content like this: + + [config] + repository = sftp://your.server/home/youruser/backups/ + log = /home/liw/obnam.log + +The examples below assume you have created a configuration file, +so that options do not need to be repeated every time. + +You probably want to enable the `log` setting, so that if there is +a problem, you can find out all the information available to fix it +from the log file. + +Initial backup +-------------- + +Your first backup will be pretty big, and will take a long time. +A long backup may crash, but that is not a problem: Obnam makes +a **checkpoint** every one hundred megabytes or so. + + obnam backup $HOME + +Incremental backups +------------------- + +When you've made your initial, full backup (possibly in stages), you can +back up any changes simply by running Obnam again: + + obnam backup $HOME + +This will back up all new files, and any changed files. It will also +record which files have been deleted since the previous backup. + +You can run Obnam as often as you like. Only the changes from the +previous run are backed up. + +Multiple clients in one repository +---------------------------------- + +You can backup multiple clients to a single repository by providing the +option --client-name=<identifier> when running the program. Backup sets +will be kept separate, but data deduplication will happen across all +the sets. + +Removing old generations +------------------------ + +Eventually your backup repository will grow so big you'll want to +remove some old generations. The Obnam operation is called forget: + + obnam forget --keep=30d + +This would keep one backup from each of the last thirty calendar +days, counting from the newest backup (not current time). +If you've backed up several times during a day, only the latest +generation from that day is kept. + +Any data that is part of a generation that is to be kept will +remain in the repository. Any data that exists only in those +generations that is to be forgotten gets removed. + +Restoring data +-------------- + +You will hopefully never need this, but the whole point of having +backups is to restore data in case of a disaster. + + obnam restore --to=/var/tmp/my-recovery $HOME + +The above command will restore your entire home directory to +`/var/tmp/my-recovery`, from the latest backup generation. +If you only need some particular directory or file, you can +specify that instead: + + obnam restore --to=/var/tmp/my-recover $HOME/Archive/receipts + +If you can't remember the name of the file you need, use `obnam ls`: + + obnam ls > /var/tmp/my-recovery.list + +This will output the contents of the backup generation, in a format +similar to `ls -lAR`. Save it into a file and browse that. +(It's a fairly slow command, so it's comfortable to save to a file.) + +Using encryption +---------------- + +Obnam can use the GnuPG program to encrypt the backup. To enable +this, you need to have or create a PGP key, and then configure +Obnam to use it: + + [config] + encrypt-with = CAFEBABE + +Here, `CAFEBABE` is the **key identifier** for your key, +as reported by GnuPG. You need to have `gpg-agent` or equivalent +software configured, for now, because Obnam has no way to ask for +or configure the passphrase. + +After this, Obnam will automatically encrypt and decrypt data. + +Note that if you encrypt your backups, you'll want to back up your GPG +key in some other way. You can't restore any files from the obnam +backup without it, so you can't rely on the same obnam backup to back up +the GPG key itself. Back up your passphrase-encrypted GPG key somewhere +else, and make sure you have a passphrase strong enough to stand up to +offline brute-force attacks. Remember that if you lose access to your +GPG key, your entire backup becomes useless. + +If you enable encryption after making backups, you need to start over +with a new repository. +You can't mix encrypted and unencrypted backups in the same repository. + +(There are a bunch of Obnam commands for administering encryption. +You won't need them, unless you share the same repository with several +machines. In that case, you should read the manual page.) + +The End +------- + +Best of luck. + +See [[status]] for ways to get support, should you need anything. |
