diff options
| author | Lars Wirzenius <liw@liw.fi> | 2020-10-23 09:36:35 +0300 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2020-10-23 09:36:35 +0300 |
| commit | 3d76f0bcc3e50e01db8540592606001d7d0c6998 (patch) | |
| tree | 6ac67d8a28562484f11455e927804c820bfcd60c | |
| parent | e60f6bb52019d024346bf9cbc213e65449d9f981 (diff) | |
| download | obnam.org-3d76f0bcc3e50e01db8540592606001d7d0c6998.tar.gz | |
drop now-obsolete content
265 files changed, 0 insertions, 6880 deletions
diff --git a/bug-reporting.mdwn b/bug-reporting.mdwn deleted file mode 100644 index b2ec668..0000000 --- a/bug-reporting.mdwn +++ /dev/null @@ -1,31 +0,0 @@ -If you have a problem with Obnam, -and you want to report it (please do!), -including the following information is helpful -and makes it easier to figure out what the problem is. -Send bug reports to `obnam-support@obnam.org` please, not to Lars -directly. - -* What is the problem? -* The version of Obnam and Larch you're using, and how you installed it. -* The exact command line you used. Copy-paste it instead of - typing it again into the mail. -* If there's an error message, copy-paste that into the mail. -* The output of `obnam --dump-config`, which includes the full configuration. - Include it as an attachment. -* If you can reproduce the problem while running with `--log=obnam.log` - and `--trace=obnamlib` options, include a suitable amount from the - end of the log file. The suitable amount may depend on the situation, - but if you give the last two hundred lines, and it's - not enough, we'll ask for more. -* The output of the `env` command, in the same terminal window in which - you ran Obnam. (Again, as an attachment.) -* If your bug is about performance, please run Obnam under profiling, - and mail Lars or the [[obnam mailing list|contact]] the profiling file. - To run Obnam under profiling, install the Python profile - (`python-profiler` package in Debian/Ubuntu), and set - the `OBNAM_PROFILE` environment variable to the name of the file - with the profiling output (that's the file you should send by mail). - For example: `OBNAM_PROFILE=obnam.prof obnam backup` would run the - backup under the profiler, and write the result to `obnam.prof`. - -Thanks! diff --git a/bugs.mdwn b/bugs.mdwn deleted file mode 100644 index 01cb306..0000000 --- a/bugs.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -Open bugs in Obnam -================== - -**Note: this page has been replaced by a ticketing system.** - -* <http://distix.obnam.org/> -* <http://distix.obnam.org/obnam-support/> -* <http://distix.obnam.org/obnam-dev/> - -The above ticketing system listens in on the Obnam mailing lists, and -opens tickets for any new discussion, and adds all the mails in that -discussion to the ticket for the discussion. To open a new ticket, -just send a new mail (not as a response to an existing discussion, -however). The developers will manage the tickets via git using the -[distix](http://liw.fi/distix/) tool. The pages linked above list open -and closed tickets. - -The bugs listed below will be kept here until they're closed. New bugs -will not be added here. diff --git a/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn b/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn deleted file mode 100644 index aa4c203..0000000 --- a/bugs/Ability_to_mix_compressed__47__uncompressed__44___encrypted__47__unencrypted_data_in_repository.mdwn +++ /dev/null @@ -1,24 +0,0 @@ ->On Sun, Feb 26, 2012 at 02:12:18AM -0600, Adam Porter wrote: ->> On Sat, Feb 25, 2012 at 04:00, Lars Wirzenius wrote: ->> > Bzr now has a fix for combining compression and encryption. :) ->> ->> Thanks for fixing that. Will I need to backup from scratch when I ->> start using compression, or can compressed and uncompressed data be ->> mixed in a repo? -> ->Right now you need to re-backup everything, I think. If you open a ->bug about this, I'll make it work without having to do that. - ---- - -As requested. :) It seems like it would be a good idea for each chunk to know whether it's compressed and/or encrypted. It's possible that some backup runs might intentionally skip compression (e.g. MP3s or JPGs won't benefit much from gzip). - -I also wonder, how would Obnam handle it if a user accidentally mixed encrypted and unencrypted data in a repo? - ---- - -As of repository format 6 (Obnam 0.27?) the system should transparently decompress data even if you don't ask it to, and should be able to detect data and read uncompressed data if you ask it to compress. This is part of the 'stable filters' work. - -As such, later allowing mixing of uncompressed/compressed data should be easy. - -[[done]] diff --git a/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn b/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn deleted file mode 100644 index 7d51f39..0000000 --- a/bugs/Add_a_method_for_forgetting_checkpoint_generations.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -Currently, there seems to be no easy way to forget all (or all but the newest) checkpoint generations. Something like - - obnam --keep 1c forget - -would be nice. - --- weinzwang diff --git a/bugs/Add_more_content_browsing_possibilities.mdwn b/bugs/Add_more_content_browsing_possibilities.mdwn deleted file mode 100644 index 66d40e6..0000000 --- a/bugs/Add_more_content_browsing_possibilities.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be very useful to have a simple way to get a list of the generations a given file appears in, preferably with the relevant metadata. - -A way to walk through the backup repository would also be useful - think smbclient-like functionality; or maybe as already mentioned a fuse filesystem. - ---- - -I think the FUSE filesystem will cover all of the above. This is thus a duplicate request. --liw - -[[done]] diff --git a/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn b/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn deleted file mode 100644 index 036aae4..0000000 --- a/bugs/CLI_status_line_for_obnam_forget_isn__39__t_updated.mdwn +++ /dev/null @@ -1,25 +0,0 @@ -obnam forget doesn't update its cli status. It says - - forgetting generation 0/39 - -for a few minutes, then updates to - - forgetting generation 39/39 - -prior to finishing. The generations are removed as expected though. - ---weinzwang - -*Update:* When obnam automatically removes snapshot generations after finishing a backup, the status line disappears altogether. --weinzwang - -*Update:* I'm nitpicking here, but the status line is a bit misleading. When it says "forgetting generation 0/1" it's actually forgetting generation 1/1. Maybe something like "forgetting generations: 0/1 done" would be better. --weinzwang - - ---- - -I confirm that I can see this too. --liw - ---- - -This should be fixed in [[ttystatus]] 0.17, just released and uploaded. --liw -[[done]] diff --git a/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn b/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn deleted file mode 100644 index 5341e76..0000000 --- a/bugs/Can__39__t_set_multiple_roots_in_configfile.mdwn +++ /dev/null @@ -1,18 +0,0 @@ -While on the commandline, you can enter multiple roots to be backed up, like: - obnam --config=/etc/obnam.cnf backup sftp://user@host/etc sftp://user@host/home - -The same is not possible when setting the root in a configfile. Setting multiple root= entries simply overrides the previous ones and only picks up the last one; setting them all on one line, space-delimited, like on commandline gets interpreted as one large filename with spaces. - ---- - -This turns out to be a problem in the config file: you need to set -all the roots in one root= line, instead of having one root= per -root. - - [config] - root = /home/you, /home/them - -That'll work. I've added something to the manual page to explain that -better. Sorry about the confusion. --liw - -[[done]] diff --git a/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn b/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn deleted file mode 100644 index 2e491dd..0000000 --- a/bugs/ERROR:_Cannot_back_up___47__path__47__to__47__file:___40__61__44_____39__No_data_available__39____44_____39____47__path__47__to__47__file__39____41__.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Just started with obnam. While backing up files via sftp works fine, when I'm trying to backup local files I'm getting only errors similar to the one in the post title. - ---- - -There's not nearly enough information there to investigate the problem. Please see -the <http://liw.fi/obnam/bug-reporting/> page and then send the relevant information to the -Obnam mailing list (see <http://liw.fi/obnam/contact/> for details). Thanks. - -[[done]] diff --git a/bugs/Excessive_Memory_usage.mdwn b/bugs/Excessive_Memory_usage.mdwn deleted file mode 100644 index 2a278c6..0000000 --- a/bugs/Excessive_Memory_usage.mdwn +++ /dev/null @@ -1,45 +0,0 @@ -I'm running obnam (1st run) on a decent sized (2.8T) volume. It's been running for days and beyond being slow, it's using a good bit of memory. This does not seem reasonable. Is there something I should tweak? - -Thanks - -(unloaded switched gigE between server/client) - -<pre> -PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND -20848 root 20 0 1745m 1.4g 1620 R 70 68.9 8558:22 obnam - -# more .obnam.conf -[config] -repository: sftp://xxx/yyy/obnam/ -log-level: warning -log: /root/obnam.log -log-max: 10M -weak-random: true -lock-timeout: 10 -compress-with: deflate -</pre> - - ---- - - -I have not benchmarked Obnam's memory usage for a while, so I don't know if that is unexpectedly large. -It's certainly larger than I would like, though. You can try different sizes of the following settings: - -* `node-size` -* `upload-queue-size` -* `lru-size` - -If you change the node size, you'll need to remove the repository and start over, since it only affects new -instances of the in-repository data structures, so it's probably best to start by trying different values for the -other two settings. - ---liw - - --- - -I've reduced the default values for lru-size and upload-queue-size. Obnam should now be -using a lot less memory. --liw - -[[done]] diff --git a/bugs/Flexibilize_path_handling_when_doing_backups.mdwn b/bugs/Flexibilize_path_handling_when_doing_backups.mdwn deleted file mode 100644 index a363d40..0000000 --- a/bugs/Flexibilize_path_handling_when_doing_backups.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -[[!tag obnam-wishlist]] - -First of all, I realise that Obnam stores full paths because it is necessary for saving every file in the system, even when belonging to different users. - -However, for certain cases where just a backup of a directory is needed, this could be flexibilized, letting the backup store only the path that is given in the command line, following rsync's spirit. - -When does this show up? For example, when migrating from any other backup system, the easier way would be to dump all the generations from the older backup system, one by one, to a temporal place. For each generation, Obnam is run in order to replicate the same history. However, since Obnam stores full paths, the path to the temporary directory used for the migration is also stored. This can happen in production servers, where making the conversion into the original directory where the data belongs to is not possible. - -Rickard Nilsson suggested on the mailing list to have a "root" option that could be used for stripping the first part of the path. The stripped part would be the one not mentioned in the command line. That way, the backup will have a path computed like this: root + given_path. - -~$ obnam backup --repository=/media/backups/... --root=/ mydata - -...would be stored into /mydata instead of in /home/${USER}/mydata - -Thank you for taking this into consideration! - - ---- - -If this gets implemented, I suggest the following: - -* The `Repository` class will provide a hook for mangling the pathnames. -* The hook will get the pathname as it exists in live data and will return the pathname to store in the backup. -* The hook will be called at every point where live data pathnames are used by `Repository`. -* Someone writes a plugin that adds the suitable functionality. - ---liw - ---- -For the fun of it I added a mangle_filanem() method to Repository. What I quickly learned: If you backup /root/bar the process backs up "/", "/root", "/foo/bar". In reality you only want "bar" in the backup. - -So either the mangling hook is allowed to drop paths entirely. But this feels very crude. - -I propose not to change Repository and think of Repository just getting virtual paths from its callers. So instead the functions calling into Repository should be changed. In this case the backup command. I have stopped here. - --- Elrond - -If this ever gets implemented, the paths returned by a filename mangling -hook will have to still be absolute, I think. --liw - -However, I don't particularly want this feature. It has the promise of -hours upon hours of debugging over email, when people use it. If someone -makes a clean patch (with tests and everyting), I'll consider it, but -keeping this bug open for years hasn't resulted in that. [[done]] --liw diff --git a/bugs/FreeBSD_support.mdwn b/bugs/FreeBSD_support.mdwn deleted file mode 100644 index 801eb2b..0000000 --- a/bugs/FreeBSD_support.mdwn +++ /dev/null @@ -1,41 +0,0 @@ -## gid.diff -## no-llistxattr.diff -## o_noatime.diff -## symlink.diff -## utimensat.diff - -How I am supposed to attach a patch here? - ---- - -You need to click on the word "Attachments" below the editing area. --liw - --- - -I can't: -prohibited by allowed_attachments (user is not an admin) - -I sent you the patches by mail, sunday. - -Best regards - - ----- - -Oh, right, I haven't enabled everyone to add attachments. Sorry about that. - -I've responded by e-mail to the patches. Summary: I'm happy to accept patches for portability, but since I don't run FreeBSD (or the kFreeBSD kernel on Debian), I want the patches to apply cleanly and be otherwise acceptable: if I need to hack on the patches to make them acceptable to me, I don't know if they'll work for FreeBSD anymore. These patches achieved portability by disabling tests and features on Linux, which is not acceptable, so now I'm waiting for updated patches. - -Meanwhile, since this is best handled via e-mail (since patches can't be attached here), I'll mark this -bug as [[done]]. - ----- - -Could these patches be made public? I'd like to get obnam working on my FreeBSD server doing its backups and minimizing duplication of work would be ideal. Thanks. -- mathstuf - ---- - -I'm afraid the patches were not particularly useful as a basis for further work. Most of them -concentrated on disabling features or functionality for everyone, which is not how porting -should happen. You'd be better off grabbing the current version of Obnam and its -dependencies and running "./check" in the Obnam source tree. --liw diff --git a/bugs/Ignores_invalid_command.mdwn b/bugs/Ignores_invalid_command.mdwn deleted file mode 100644 index 5778f85..0000000 --- a/bugs/Ignores_invalid_command.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -I accidentally ran "obnam list-clients" instead of "obnam clients", and it didn't give an error, it just returned. - ---- - -I confirm this. This turns out to be an error in the [[cliapp]] framework, and is now fixed there, and the -fix will be included in the next release. --liw - -[[done]] diff --git a/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn b/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn deleted file mode 100644 index b35239b..0000000 --- a/bugs/Inaccurate_speed__47__bytes-transferred_when_resuming.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -[[!tag obnam-wishlist]] - -I'm using trickle to restrict obnam to 40 KB/sec. I just resumed a backup -a few minutes ago, and obnam currently says the rate is 145 KB/sec, and -that it's transferred 111 MB. I'm guessing it's counting the entire size -of resumed files instead of the actual bytes transferred since resuming. -Whatever the cause, it's definitely off. I've not noticed it being off -when not resuming. - ---- - -So, there were at least a couple of problems here: - -1. Obnam was counting duplicate chunks as uploaded: when obnam found a - chunk was a duplicate of an existing one, it didn't upload it, but - did count it as uploaded. This has now been fixed. -1. Obnam was showing the average upload speed over the entire backup - run. It now shows the average for the last ten seconds. - -Thus, I think this bug is fixed now. [[done]] (If not, please tell me!) - ---liw diff --git a/bugs/Local_cache_of_remote_repo_metadata.mdwn b/bugs/Local_cache_of_remote_repo_metadata.mdwn deleted file mode 100644 index e7d6e9e..0000000 --- a/bugs/Local_cache_of_remote_repo_metadata.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -[[!tag obnam-wishlist]] - -Could Obnam keep a local cache of remote repository metadata, like Duplicity does? I was looking through an Obnam log and noticed how many remote file accesses it does for each file it backs up. If it cached that data locally, it could do it much faster. The last-modified time of the repository could be stored in the remote repo, and if it matches the local cache, a lot of time could be saved. - ---- - -That is indeed a good idea. Unfortunately, the correctness of caches is often tricky, so I've been putting off implementing this until more important things work first. Also, not caching metadata forces me to do other things to make Obnam fast. But I'd like to do the caching too, some day. --liw - ---- - -[[done]] duplicate, and also an old wishlist bug. Keeping this open -isn't making this happen. --liw diff --git a/bugs/Log_performance_results_to_logfile.mdwn b/bugs/Log_performance_results_to_logfile.mdwn deleted file mode 100644 index 7bbc178..0000000 --- a/bugs/Log_performance_results_to_logfile.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -This is a wishlist item. - -If obnam is run with the --quiet option, then the performance details (time taken, speed etc) is lost. It'd be nice if that was logged to the log file. I'd suggest at a NOTICE level. - --- - -thank you for the excellent suggestion. I've now implemented it in bzr, -so this'll show up in the next Obnam release. [[done]] --liw diff --git a/bugs/Missing___34__e__34___in_commandline_output.mdwn b/bugs/Missing___34__e__34___in_commandline_output.mdwn deleted file mode 100644 index f0dc41b..0000000 --- a/bugs/Missing___34__e__34___in_commandline_output.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -During a backup, obnam regularly informs me that it's "making chckpoint". I think adding an "e" would probably solve this :) - ---weinzwang - -[[done]] diff --git a/bugs/Mutiple_bugs..mdwn b/bugs/Mutiple_bugs..mdwn deleted file mode 100644 index 2058135..0000000 --- a/bugs/Mutiple_bugs..mdwn +++ /dev/null @@ -1,94 +0,0 @@ -Obnam version: 1.0 - -Environment: backing up from multiple servers (each has its' own repo) to an NFS-shared storage. - -OS: OpenSuSE 11.3 - ---- - -0) From time to time I get strange locks, which prevents subsequent backups with lock timeout message, though, I am SURE that I'm running only one backup per-client at a time, and each client has dedicated obnam repo. How those locks happen to appear?... noone knows, but they do, for this or that obnam repo (I'm backing up 17 servers with it) - ---- - -1) When trying to fsck with fix: - - /it/bin/obnam/bin/obnam -r /data/backup/mlogin4.smware.local/obnam --fsck-fix fsck - Traceback (most recent call last): - File "/it/bin/obnam/lib/python/site-packages/cliapp/app.py", line 172, in _run - self.process_args(args) - File "/it/bin/obnam/lib/python/site-packages/obnamlib/app.py", line 158, in process_args - cliapp.Application.process_args(self, args) - File "/it/bin/obnam/lib/python/site-packages/cliapp/app.py", line 407, in process_args - method(args[1:]) - File "/it/bin/obnam/lib/python/site-packages/obnamlib/plugins/fsck_plugin.py", line 306, in fsck - for more in reversed(list(work.do() or [])): - File "/it/bin/obnam/lib/python/site-packages/obnamlib/plugins/fsck_plugin.py", line 250, in do - work.do() - File "/it/bin/obnam/lib/python/site-packages/larch/fsck.py", line 107, in do - tracing.trace('fixed it: %s' % new_node.keys()) - UnboundLocalError: local variable 'new_node' referenced before assignment - -That is strange ) - - ---- - -It would be much more helpful if you filed separate bugs as separate bugs. - - ---- -They are related. These two bugs deal with repo inconsistent state, which obnam leaves behind when something "wrong" happends with it. -On of the errors right before stalled locks are left looks like this (while backing up /local_disk/vpodrezov/ directory): - - <...> - - file_id = self.get_file_id(self.tree, filename) - File "/it/bin/obnam/lib/python/site-packages/obnamlib/clientmetadatatree.py", line 180, in get_file_id - raise KeyError('%s does not yet have a file-id' % pathname) - KeyError: '/local_disk/vpodrezov/save/0019_29052012/os/linux-2.6.36/arch/x86/boot/.svn/tmp/text-base does not yet have a file-id' - -And then if we try to backup to the repo, it ends with Lock timeout. - ------------- - - -Another error seen is like - - ERROR: Node 534798 cannot be found in the node store 3662754038122990765 - -And then the same, Lock timeout. No way to correct this (fsck --fsck-fix just doesnt' work). - ---- - -I'm confused: are you trying to fix lock timeouts with fsck instead of force-lock? - -Also, that's a nice, long traceback, but I think it would be more helpful if you made a debug log and posted that. - ---- - -What I want to say is that while backing up live filesystems (when files are being changed), from time to time obnam leaves its' repos in such a state, that no further use for backup is possble (errors, resulting in further locks, which are NOT ALWAYS may be bypassed with force lock (I see multiple lock files around the obnam repo tree, which force-lock doesn't find or know about)) - -Btw, one of the errors after which I get lock timeouts is (/net/backup/data/backup/mlogin1.smware.local/obnam/ is obnam's repo): - - ERROR: /net/backup/data/backup/mlogin1.smware.local/obnam/3662754038122990765/new/nodes/65/0/0: File exists - -// - -Sure, I'll go on then on the mailing list with debug log if it helps, but there are deffinitely problems with backing up live filesystems =(. - - ----- - -From discussion on mailing list: - -The unbound variable error has been fixed. - -The other errors seem to be caused by something in the repository having become corrupt, and -it is now not really possible to get Obnam working properly with it anymore. Ouch. And sorry. It's -no longer possible to see what went wrong either, it seems. It may have been some other error, -or it may be a problem in Obnam. Since this bug no longer seems actionable, I'm marking -it closed, but that doesn't mean eveyrthing's fine. I've made a note to my TODO that I need to -change Obnam to be more easy to debug about these problems in the future (no idea how to -achieve that yet, but we'll see). --liw - -[[done]] diff --git a/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn b/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn deleted file mode 100644 index f51a84d..0000000 --- a/bugs/Obnam_cannot_backup_with_failed_assertion_in_larch_forest.py_at_open__95__forest.py.mdwn +++ /dev/null @@ -1,136 +0,0 @@ -Just read about obnam on lwn and decided to try it! -Installed the version on the ubuntu ppa. -Unfortunately, I cannot execute the basic backup example: - - obnam backup --repository Backup/Obnam-backup Documents/BibFiles - Traceback (most recent call last): - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run - self.process_args(args) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 127, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args - method(args[1:]) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup - self.add_client(client_name) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 140, in add_client - if client_name not in self.repo.list_clients(): - File "/usr/lib/python2.7/dist-packages/obnamlib/repo.py", line 201, in list_clients - listed = set(self.clientlist.list_clients()) - File "/usr/lib/python2.7/dist-packages/obnamlib/clientlist.py", line 78, in list_clients - if self.init_forest() and self.forest.trees: - File "/usr/lib/python2.7/dist-packages/obnamlib/repo_tree.py", line 63, in init_forest - vfs=self.fs) - File "/usr/lib/python2.7/dist-packages/larch/forest.py", line 167, in open_forest - assert allow_writes is not None - AssertionError - -Here is the log: - - 2012-06-06 12:59:50 INFO obnam version 0.24.1 starts - 2012-06-06 12:59:51 DEBUG Current configuration: - [config] - output = - log = obnam.log - log-level = debug - log-max = 0 - log-keep = 10 - log-mode = 0600 - dump-memory-profile = simple - repository = /home/callegar/Backup/Obnam-backup - client-name = mandrago - node-size = 262144 - chunk-size = 1048576 - upload-queue-size = 1024 - lru-size = 500 - trace = obnamlib - idpath-depth = 3 - idpath-bits = 12 - idpath-skip = 13 - quiet = False - pretend = False - compress-with = none - encrypt-with = - keyid = - weak-random = False - symmetric-key-bits = - root = - exclude = - exclude-caches = False - one-file-system = False - checkpoint = 1073741824 - chunkids-per-group = 1024 - deduplicate = fatalist - sftp-delay = 0 - to = - generation = latest - keep = - fsck-fix = False - verify-randomly = 0 - - - 2012-06-06 12:59:51 INFO Obnam 0.24.1 starts - 2012-06-06 12:59:51 INFO Backup starts - 2012-06-06 12:59:51 INFO Checkpoints every 1073741824 bytes - 2012-06-06 12:59:51 DEBUG Exclude pattern: obnam.log - 2012-06-06 12:59:51 DEBUG opening repository (create=True) - 2012-06-06 12:59:51 DEBUG vfs_local.py:55:__init__: baseurl=/home/callegar /Backup/Obnam-backup - 2012-06-06 12:59:51 DEBUG vfs_local.py:56:__init__: create=True - 2012-06-06 12:59:51 INFO VFS: __init__: baseurl=/home/callegar/Backup/Obnam-backup - 2012-06-06 12:59:51 DEBUG vfs_local.py:64:reinit: baseurl=/home/callegar/Backup/Obnam-backup - 2012-06-06 12:59:51 DEBUG vfs_local.py:65:reinit: create=True - 2012-06-06 12:59:51 DEBUG clientlist.py:49:__init__: new ClientList - 2012-06-06 12:59:51 DEBUG chunklist.py:37:__init__: new ChunkList - 2012-06-06 12:59:51 DEBUG checksumtree.py:34:__init__: new ChecksumTree name=chunksums - 2012-06-06 12:59:51 DEBUG vfs_local.py:192:open: pathname=/home/callegar/Backup/Obnam-backup/metadata/format - 2012-06-06 12:59:51 DEBUG vfs_local.py:193:open: mode=rb - 2012-06-06 12:59:51 DEBUG repo_tree.py:53:init_forest: initializing forest dirname=clientlist - 2012-06-06 12:59:51 CRITICAL Traceback (most recent call last): - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run - self.process_args(args) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 127, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args - method(args[1:]) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup - self.add_client(client_name) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 140, in add_client - if client_name not in self.repo.list_clients(): - File "/usr/lib/python2.7/dist-packages/obnamlib/repo.py", line 201, in list_clients - listed = set(self.clientlist.list_clients()) - File "/usr/lib/python2.7/dist-packages/obnamlib/clientlist.py", line 78, in list_clients - if self.init_forest() and self.forest.trees: - File "/usr/lib/python2.7/dist-packages/obnamlib/repo_tree.py", line 63, in init_forest - vfs=self.fs) - File "/usr/lib/python2.7/dist-packages/larch/forest.py", line 167, in open_forest - assert allow_writes is not None - AssertionError - ----- - -The Python stack trace is awful for the user, but it's there as a development aid. If a user sees it, something is badly wrong in Obnam. - -Are you sure you have the right version of Obnam? How do you check that? Obnam is supposed to report the right version number in the log file or with "obnam --version", and if it isn't 1.0 then something's wrong. - -Do you have the right version of the Larch Python library installed? That stack trace looks like there's an old version installed. Are you running on Debian? Ubuntu? Somewhere else? The Ubuntu PPA is in the process of being updated to fix a few problems with wrong versions. If you're on either Ubuntu or Debian, what is the output of "dpkg -l obnam python-larch"? - ---liw - ----- - -Currently Chris's ppa (the one you're referring to in "download" page) seems broken. On ppa's page, it reports build fail for obnam package, in particular. Using that ppa and installing obnam, installs ubuntu precise packaged obnam: - - # LANGUAGE=C apt-cache policy obnam - obnam: - Installed: 0.24.1-1 - Candidate: 0.24.1-1 - Version table: - *** 0.24.1-1 0 - 500 http://it.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages - 100 /var/lib/dpkg/status ---negro - ---- - -The PPA seems to be fixed now. Can you check whether it now works for you? Thanks. --liw - -Haven't heard back, but assuming this is OK. [[done]] --liw diff --git a/bugs/Operation_not_permitted.mdwn b/bugs/Operation_not_permitted.mdwn deleted file mode 100644 index 20874d2..0000000 --- a/bugs/Operation_not_permitted.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -I try to backup my mail directory to a USB disc with - - % obnam --log=obnam.log --repository=/media/HD-PCU2/ backup ~/Mail - 00h00m00s 1 files; 0 B (0 B/s) setting upERROR: Operation not permitted - -The USB disc is mounted with the following options - - /dev/sdb1 on /media/HD-PCU2 type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000, shortname=mixed,dmask=0077,utf8=1,showexec,flush) - -The obnam.log contains - - 2012-08-16 14:18:19 INFO Backup starts - 2012-08-16 14:18:19 INFO Checkpoints every 1073741824 bytes - 2012-08-16 14:18:19 DEBUG Exclude pattern: obnam.log - 2012-08-16 14:18:19 DEBUG opening repository (create=True) - 2012-08-16 14:18:19 INFO VFS: __init__: baseurl=/media/HD-PCU2/ - 2012-08-16 14:18:19 CRITICAL Traceback (most recent call last): - File "/home/friedrich/local/lib/python2.7/site-packages/cliapp/app.py", line 169, in _run - self.process_args(args) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/app.py", line 174, in process_args - cliapp.Application.process_args(self, args) - File "/home/friedrich/local/lib/python2.7/site-packages/cliapp/app.py", line 418, in process_args - method(args[1:]) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/plugins/backup_plugin.py", line 253, in - self.add_client(client_name) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/plugins/backup_plugin.py", line 311, in - self.repo.lock_root() - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/repo.py", line 266, in lock_root - self.lockmgr.lock(['.']) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/lockmgr.py", line 81, in lock - self._lock_one(dirname) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/lockmgr.py", line 65, in _lock_one - self._fs.lock(lock_name, self.data) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/repo.py", line 70, in lock - self.fs.lock(filename, data) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/vfs_local.py", line 106, in lock - self.write_file(lockname, data) - File "/home/friedrich/local/lib/python2.7/site-packages/obnamlib/vfs_local.py", line 274, in write_file - os.link(tempname, path) - OSError: [Errno 1] Operation not permitted - -Do you have an idea what's the problem with the USB disc? A backup into my home directry works fine. - - ---- - - -Obnam currently does not work with FAT filesystems, since it needs the -filesystem to support hardlinks. --liw - ---- - -This is now fixed in bzr. --liw -[[done]] diff --git a/bugs/Other_compression_methods__63__.mdwn b/bugs/Other_compression_methods__63__.mdwn deleted file mode 100644 index f4f1732..0000000 --- a/bugs/Other_compression_methods__63__.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!tag obnam-wishlist]] - -Would Obnam benefit from being able to use other compression tools, like bzip2, p7zip, and xz? - ---- - -It would! However, implementing this is slightly tricky until I make Obnam can run external tools -in the background, rather than waiting for them to finish. The compression it uses right now is -done with the Python standard library `gzip` module, so it doesn't require executing an -external program, so the issue doesn't rise. However, we need the backgrounding feature -anyway, since encryption uses an external tool, and so there's a big performance impact -from using encryption. After backgrounding works, adding arbitrary compression filters will -be easy. --liw - -[[done]] this is an ancient wishlist bug, and keeping it open isn't helping it happen, -but it is making it harder to use the bug list. --liw diff --git a/bugs/Performance_impact_of_backup_generations__63__.mdwn b/bugs/Performance_impact_of_backup_generations__63__.mdwn deleted file mode 100644 index dda828c..0000000 --- a/bugs/Performance_impact_of_backup_generations__63__.mdwn +++ /dev/null @@ -1,55 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -I'm not sure what to do with this or if it even qualifies as a bug but here we go... - -I back up my notebook using obnam for a while now so I have about 10 clean generations, the repo is 118G big. One of the files in the backup is a 32G VM file that changes slightly between backups, but not during backups. Until a few days ago this worked really well and fast. - -Then I tried to do a partial backup via a very slow line, with snapshots every 5MB. This was interrupted (ctrl+c) after a few hours. - -The next day I continued the backup using the local network. I forgot to change the snapshot interval, so it was still 5MB. I expected the backup to finish within a few minutes as it did in the past, but after crawling through the first couple of GB of the VM file at 50MB/s the speed went down to below 1MB/s, doing a huge amount of IO on the local machine. I haven't manged to complete a backup since. - -Nearly everything I try to do with the repo now takes a lot of time. Listing the huge amount of generations takes over an hour (it lists at a rate of about 1 generation every 10 seconds or so). Listing only the genids is still very quick. - -I'm not sure what goes wrong here. Is the massive amount of generations killing the performance? - --- weinzwang - ---- - -The number of generations shouldn't impact the speed of backups. It does have an impact on -the speed of removing generations, so if you backup stalls at the end when it removes checkpoints, -then you can avoid that with the `--leave-checkpoints` option. - -Could you check logs to see where time is spent? Or even run with profiling enabled when you -list generations? `OBNAM_PROFILE=obnam.prof obnam generations` should do that. Then use -a Python script like the following to get it in cleartext: - - import pstats - import sys - - if len(sys.argv) not in [2, 3]: - sys.stderr.write('Usage: viewprof foo.prof [sort-order]\n') - sys.exit(1) - - if len(sys.argv) == 3: - order = sys.argv[2] - else: - order = 'cumulative' - - p = pstats.Stats(sys.argv[1]) - p.strip_dirs() - p.sort_stats(order) - p.print_stats() - p.print_callees() - -Then mail me the output, and I'll have a look. Thanks. --liw - ---- - -Since this problematic repository doesn't work with recent versions of obnam due to -on-disk format changes, I can't reproduce this anymore. I haven't encountered this effect -with any other repositories I have, so I think the bug can be closed. --weinzwang - --- - -Ack, closing bug. [[done]] --liw diff --git a/bugs/SSH_connections_left_open.mdwn b/bugs/SSH_connections_left_open.mdwn deleted file mode 100644 index d1c829e..0000000 --- a/bugs/SSH_connections_left_open.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -Obnam 0.26 opens 2 ssh connections at the beginning of a backup and another one for each snapshot generation it creates. All of these connections stay open until obnam terminates. For a long-running backup session that generates a lot of snapshot generations this might become a problem since for every ssh connection there are 3 processes running on the server. - -This behaviour seems to have been introduced with obnam 0.26. The previous version opened one ssh connection at the beginning of a backup and used it for everything. (Side note: This made it feasible to use password authentication, which is not possible anymore.) --weinzwang - - ---- - -I think this is related to the ssh locking changes in 0.26, and that the current bzr trunk has fixes for it. -If you could verify and report, that would be excellent. You'll need the new larch 0.19 version, though. --liw - ---- - -The problem is still the same with obnam 0.27. Every snapshot triggers a ssh authentication and the sshd processes pile up on the server. When obnam finishes, all connections are closed. --weinzwang - --- - -Found and fixed in bzr now. [[done]] --liw diff --git a/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn b/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn deleted file mode 100644 index 1353ba7..0000000 --- a/bugs/Support_SSH_host_aliases___40__and_.ssh__47__config_in_general__41__.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -I've already configured ~/.ssh/config to have a "backup" alias, with a corresponding SSH key to use, port number, and various other configuration. I'd like to just tell obnam to use sftp://backup/... and have obnam use that alias and configuration, but obnam doesn't seem to want to do that: at a minimum, it appears to ignore the port number I specified in ~/.ssh/config, so I can't easily tell what else it ignored. - ---- - -For the same reasons as discussed in -[[the other bug|option_to_not_use_paramiko__63__]], I'll mark this as [[done]]. --liw diff --git a/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn b/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn deleted file mode 100644 index 2a62a70..0000000 --- a/bugs/Support_encrypting_to_a_GPG_public_key_without_having_the_private_key.mdwn +++ /dev/null @@ -1,55 +0,0 @@ -[[!tag obnam-wishlist]] - -Normally, GPG can encrypt to a key using only the public key, without needing the private key available. I'd like to have the same model for obnam encrypted backups: perform a backup to a key using only the public key, without having the private key available. That way, I could perform backups of various systems using public keys whose corresponding private keys I keep entirely offline and secured. - ---- - -Unfortunately, Obnam requires the secret in order to do backups. It needs to download, and decrypt, -some metadata from the backup repository in order to add new backups there. See the [[obnam/ondisk]] -and [[obnam/encryption]] pages for details. --liw - ---- - -I've read those pages, but as far as I can tell, the ability to decrypt part of the repository doesn't seem like an inherent requirement to meet obnam's goals, just an implementation detail. And as far as I can tell, without some change in this area, obnam does not support letting a client perform backups without giving that client full access to previous backups. - -Use case: I want to back up to a storage system that I trust for availability but not for privacy; thus, I need encryption, and I can't do "pull" backups since I don't trust the storage server with access to the system that needs backing up. However, I also want the backups to protect me from security breaches, by giving me the ability to follow the standard advice for how to deal with a system break-in: nuke everything and restore from the last-known good backup. How can I have a "last-known good backup" if the backed-up system can access the old backups? - ---- - -I understand the desire to do what you want to do, but I haven't found a technical solution to do it. If you can -come up with one, then write it up and send me a pointer. --liw - ---- - -It'd be nice if Obnam could do this, but in the meantime, I think Duplicity can work this way. --AP - ---- - -If Obnam had a local metadata cache, could this be implemented? If Obnam had the metadata locally, could it do a quick verification that the metadata is the same as that on the remote repository, and then backup to the remote repository "blindly", using only the local cache as a guide? --AP - ---- - -A concrete existence proof: tarsnap does this. You can do tarsnap backups with only a restricted public key, and that key need not also provide access to decrypt old backups. -- Josh - ---- - -I've re-opened this bug since there's ongoing discussion. I'll have a think and respond later. --liw - ---- - -Having thought about this... - -* Obnam will probably eventually get a local cache for stuff from the repository. - However, this is not enough: we can't assume the cache is complete, since - other clients will be making changes to the repository as well. -* Obnam will thus need to read, and decrypt, files from the repository. -* Having access to the PGP private key is thus necessary for Obnam. - -I haven't looked at how duplicity and tarsnap do things. If someone comes up -with a design for Obnam that can do this, without sacrificing things such as -de-duplication across clients, please e-mail the Obnam mailing list with -suggestions. Thanks! - ---liw - -[[done]] diff --git a/bugs/Unhelpful_error_for_missing_key.mdwn b/bugs/Unhelpful_error_for_missing_key.mdwn deleted file mode 100644 index 9845930..0000000 --- a/bugs/Unhelpful_error_for_missing_key.mdwn +++ /dev/null @@ -1,24 +0,0 @@ -[[!tag obnam-wishlist]] - -I decided to try running fsck on a remote repo from a user account that doesn't have the GPG key used to encrypt with. I got this error, which isn't helpful. I think, at least, it should mention the possibility that there is a missing key. - -<pre>ERROR: Unknown filter tag encountered: '\x8c\r\x04\x03\x03\x02n\x7fX8\xa5\xd7\xf1\xbd`\xc9[\xba\xc0\x11kU\xd0\x8f\xf3\r\x97J@\x0c\xc9\x11\xaan\xefxx\x89\xe9\x9b\xac\x96\xce \x1cD{:\x02\x10I\xda\xbb;\x0e\xac\xd3d9\x18(\x0c\xe4\x8a\xf89\xed\x9a!\x85\xd0t\xd0\xaa^0\xee\x17{\xe3\x9d\xa3\x89O\xc6\x04$c\x96\xee\xf3\xbb{.|\xd5\xb0\x96Z\xb3\x11\x0f&B\xf0'</pre> - ---- - -Someone else also reported this: - -Minor improvement: using Obnam 1.0 on Debian Unstable, the error message when using an encrypted repository without specifying any key is not helpful. Please consider using a human-readable error message to warn the user that the key is missing (or wrong). - -Here is an example of error message when using "obnam fsck" without the key: - - Traceback (most recent call last): - [lots of debug info] - File "/usr/lib/python2.7/dist-packages/obnamlib/hooks.py", line 122, in run_filter_read - tag, content = data.split("\0", 1) - ValueError: need more than 1 value to unpack - - ---- - -Fixed in git now. [[done]] --liw diff --git a/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn b/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn deleted file mode 100644 index d2791ec..0000000 --- a/bugs/ValueError:_need_more_than_1_value_to_unpack.mdwn +++ /dev/null @@ -1,29 +0,0 @@ -hello, - -My backup fails with the following message at the end of Python stack trace : "ValueError: need more than 1 value to unpack" - -Steps to reach the error state : - -* did a first succesful backup of a small part of the backup source -* started a complete backup of the source -* the backup ended because of network shutdown -* restarted the backup, but could not proceed because of the previous lock remaining settled -* did obnam force-lock -* restarted backup, got this error - -overview of my configuration: - -* backup over sftp -* gpg encryption - -I will send you the log files in a separate email. - -thanks! -Damien - - ---- - -This is fixed in version 1.1. --liw - -[[done]] diff --git a/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn b/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn deleted file mode 100644 index 6e9d7a6..0000000 --- a/bugs/Warn___40__and_optionally_skip__41___backing_up_large_files.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -[[!tag obnam-wishlist]] - -One of the common (?) use cases of obnam is to backup laptops and workstations. Sometimes we grab ISOs etc and have them lying around, but they shouldn't be backed up. - -It'd be handy if obnam had a couple of extra options. One to warn about large files, the other to skip large files. They are not mutually exclusive. - -This would have saved me backing up a 700MB ISO. Ouch. - ----- -Great idea. --AP - ------------ -I second that great idea :) leto - - ---- - -[[done]] - -While I agree it is a nice idea, keeping the wishlist bug open clearly -isn't making this happen, and long lists of bugs are tedious to work -with. --liw diff --git a/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn b/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn deleted file mode 100644 index 32256d7..0000000 --- a/bugs/__34__No_such_file_or_directory__34___when_trying_to_backup.mdwn +++ /dev/null @@ -1,599 +0,0 @@ -Hi, - -I'm using obnam 1.0 and try to backup files / directories. obnam is installed on Debian Squeeze with the package frovided on http://liw.fi. No ~/.obnam.conf file is used. The obnam repository and the files to backup are stored on a partition that is mounted from an encrypted partition (cryptsetup, device-mapper). When trying to backup with - -cd /mnt/crypt/obnam - -obnam backup /mnt/crypt/myfiles/test.data - -obnam complains, that /mnt/crypt/myfiles/test.data is not available (No such file or directory). But the file is definitely available (less -f /mnt/crypt/myfiles/test.data shows data). When I comment out - -line 91: #raise OSError(err, os.strerror(err), self.cwd) - -in the file obnamlib/vfs_local.py, the backup works fine. I don't understand what the purpose of this line is (I'm not a python programmer), but for me it produces errors. Maybe you can find a solution ;-) - ---- - -Hi: there is at least one obvious bug here, which is that Obnam does not complain when you don't give an explicit backup repository (it uses the current working directory instead). I'll fix that. However, you seem to be working around that with the cd command. (Edit: --repository must now be set explicitly.) - -The line you commented out is essential. It makes sure that the repository exists. - -I'm not sure what the real error is that makes things not work for you. Could you e-mail me the log file, which has a lot extra information I could use for debugging. Thanks. - ---liw - ---- - -I am having this same problem. I had to wipe my obnam repo from March--32 GB, uploaded slowly :(--since the repo format changed. So before starting a whole, new, full backup, I decided to do a few little tests. I first backed up a small directory to a local directory and tried a few operations. Then I rsynced the repo to a remote server and ran fsck (obnam's, of course) on it. It came out fine. Listing clients and generations and keys and "ls" all worked fine. - -So then I tried to make a new backup of the same, small, local directory. It failed with: - - ERROR: sftp://me@host/~/obnam: No such file - -This was quite strange, since all the other ops worked. Finally I made a full log and here's what I see: - -<pre> -2012-06-21 03:05:51 INFO obnam version 1.0 starts -2012-06-21 03:05:51 DEBUG Current configuration: -[config] -output = -log = /tmp/obnam.log -log-level = debug -log-max = 0 -log-keep = 10 -log-mode = 0600 -dump-memory-profile = simple -repository = sftp://me@host/~/obnam -client-name = mysystem -node-size = 262144 -chunk-size = 1048576 -upload-queue-size = 1024 -lru-size = 500 -trace = -idpath-depth = 3 -idpath-bits = 12 -idpath-skip = 13 -quiet = False -pretend = False -pretend-time = -lock-timeout = 60 -crash-limit = 0 -compress-with = none -root = -exclude = -exclude-caches = False -one-file-system = False -checkpoint = 1073741824 -chunkids-per-group = 1024 -deduplicate = fatalist -leave-checkpoints = False -small-files-in-btree = False -warn-age = 27h -critical-age = 27h -sftp-delay = 0 -ssh-key = -strict-ssh-host-keys = False -ssh-known-hosts = /home/me/.ssh/known_hosts -pure-paramiko = False -to = -generation = latest -keep = -fsck-fix = False -verify-randomly = 0 -encrypt-with = mykeyid -keyid = -weak-random = False -symmetric-key-bits = - - -2012-06-21 03:05:51 INFO Backup starts -2012-06-21 03:05:51 INFO Checkpoints every 1073741824 bytes -2012-06-21 03:05:51 DEBUG Exclude pattern: /tmp/obnam.log -2012-06-21 03:05:51 DEBUG opening repository (create=True) -2012-06-21 03:05:51 INFO VFS: __init__: baseurl=sftp://me@host/~/obnam -2012-06-21 03:05:51 DEBUG executing openssh: ['ssh', '-oForwardX11=no', '-oForwardAgent=no', '-oClearAllForwardings=yes', '-oProtocol=2', '-p', '22', '-l', 'b122569', '-s', '-o', 'UserKnownHostsFile=/home/me/.ssh/known_hosts', 'host', 'sftp'] -2012-06-21 03:05:52 INFO [chan obnam SSHChannelAdapter] Opened sftp connection (server version 3) -2012-06-21 03:05:52 DEBUG [chan obnam SSHChannelAdapter] mkdir('obnam', 511) -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] stat('obnam') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] normalize('obnam') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/./lock', 'wx') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/./lock', 'wx') -> 00000000 -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata/format') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -> 00000000 -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/metadata', 511) -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', 'wx') -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', 'wx') -> 00000000 -2012-06-21 03:05:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/metadata/format') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/metadata/tmp.5e2ae5d86f54786c', '/path/to/my/user/obnam/metadata/format') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') -2012-06-21 03:05:54 DEBUG Initializing Journal for clientlist -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') -2012-06-21 03:05:54 DEBUG Automatically rolling back remaining changes -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/') -2012-06-21 03:05:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:05:55 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:05:56 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:05:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:05:58 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/metadata') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/metadata', 'r') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/metadata', 'r') -> 00000000 -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/key', 'r') -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/key', 'r') -> 00000000 -2012-06-21 03:05:59 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5', 'r') -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5', 'r') -> 00000000 -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0', 'r') -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0', 'r') -> 00000000 -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/metadata/format') -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -2012-06-21 03:06:00 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/metadata/format', 'r') -> 00000000 -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/2', 'r') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/nodes/0/0/0/2', 'r') -> 00000000 -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete') -2012-06-21 03:06:01 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes', 511) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0', 511) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0', 511) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0', 511) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', 'wx') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', 'wx') -> 00000000 -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/tmp.a6cb1927c96d46d9', '/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/refcounts', 511) -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', 'wx') -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', 'wx') -> 00000000 -2012-06-21 03:06:02 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/refcounts/tmp.f0f0e1113a89e450', '/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes', 511) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0', 511) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0', 511) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0', 511) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', 'wx') -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', 'wx') -> 00000000 -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/tmp.5e84ba749fa001e6', '/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') -2012-06-21 03:06:04 DEBUG LRUCache <larch.lru.LRUCache object at 0xb71a8e0c>: hits=0 misses=2 -2012-06-21 03:06:04 DEBUG LRUCache <larch.lru.LRUCache object at 0xb71a83ac>: hits=3 misses=2 -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/my/user/obnam/clientlist/new', 511) -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', 'wx') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', 'wx') -> 00000000 -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/new/metadata') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/tmp.46022d631cdb89f7', '/path/to/my/user/obnam/clientlist/new/metadata') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:06:04 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:06:05 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/nodes/0/0/0/5') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/delete/nodes/0/0/0/5') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:06:06 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:06:07 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/nodes/0/0/0/6') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0/6', '/path/to/my/user/obnam/clientlist/nodes/0/0/0/6') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes/0') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/nodes') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/nodes') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:06:08 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/refcounts/refcounts-0', '/path/to/my/user/obnam/clientlist/refcounts/refcounts-0') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/refcounts') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/refcounts') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/my/user/obnam/clientlist/new/metadata') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/clientlist/metadata') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/my/user/obnam/clientlist/new/metadata', '/path/to/my/user/obnam/clientlist/metadata') -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/my/user/obnam/./lock') -2012-06-21 03:06:09 DEBUG opening repository (create=False) -2012-06-21 03:06:09 DEBUG [chan obnam SSHChannelAdapter] stat('/path/to/my/user/obnam/obnam') -2012-06-21 03:06:09 CRITICAL Traceback (most recent call last): - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run - self.process_args(args) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 158, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args - method(args[1:]) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 226, in backup - self.add_client(client_name) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 293, in add_client - self.repo = self.app.open_repository(repofs=self.repo.fs.fs) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 187, in open_repository - repofs.reinit(repopath) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/sftp_plugin.py", line 58, in helper - raise OSError(e.errno, e.strerror or str(e), filename) -OSError: [Errno 2] No such file: 'sftp://me@host/~/obnam' -</pre> - -Near the end of the log, it tries to stat('/path/to/my/user/obnam/obnam'), which doesn't exist. Where did the extra "/obnam" come from? - -Also, from my cursory look at the log, it seems like the backup succeeded before it failed, yet when I run "generations" I don't see any new ones. Will I need to run "fsck" to remove orphaned nodes or files? - -Thanks for obnam! Looking forward to using it full-time. - ---AP - ---- - -AP, what was your command line? I wasn't able to reproduce this myself, so I need your help to do so. --liw - ---- - -That happened to me yesterday when setting up a my new offsite repository. -When obnam is asked to create a new repository (notice the "opening repository (create=True)" in the log above) but the repository directory already exists, then it fails with the file not found error. -I rmdir'd my empty directory on the server (and I also replaced the "~" with a fully expanded path, not sure if that's related to the error) and it started working. - ---nemoinis - ---- - -Ok, let me try this again. Hopefully this will make it clear. :) --AP - -<pre> -$ sftp me@server -Connected to server. -sftp> ls -Maildir logs obnam -sftp> ^D - -$ time obnam --log=/tmp/obnam.log --log-level=debug backup ~/Dropbox/Zim -ERROR: sftp://me@server/~/obnam/: No such file - -$ cat .obnam.conf -[config] -repository = sftp://me@server/~/obnam/ -encrypt-with = MYKEY - -$ cat /tmp/obnam.log -2012-07-07 13:47:40 INFO obnam version 1.0 starts -2012-07-07 13:47:40 DEBUG Current configuration: -[config] -output = -log = /tmp/obnam.log -log-level = debug -log-max = 0 -log-keep = 10 -log-mode = 0600 -dump-memory-profile = simple -repository = sftp://me@server/~/obnam/ -client-name = laptop -node-size = 262144 -chunk-size = 1048576 -upload-queue-size = 1024 -lru-size = 500 -trace = -idpath-depth = 3 -idpath-bits = 12 -idpath-skip = 13 -quiet = False -pretend = False -pretend-time = -lock-timeout = 60 -crash-limit = 0 -compress-with = none -root = -exclude = -exclude-caches = False -one-file-system = False -checkpoint = 1073741824 -chunkids-per-group = 1024 -deduplicate = fatalist -leave-checkpoints = False -small-files-in-btree = False -warn-age = 27h -critical-age = 27h -sftp-delay = 0 -ssh-key = -strict-ssh-host-keys = False -ssh-known-hosts = /home/me/.ssh/known_hosts -pure-paramiko = False -to = -generation = latest -keep = -fsck-fix = False -verify-randomly = 0 -encrypt-with = MYKEY -keyid = -weak-random = False -symmetric-key-bits = - - -2012-07-07 13:47:40 INFO Backup starts -2012-07-07 13:47:40 INFO Checkpoints every 1073741824 bytes -2012-07-07 13:47:40 DEBUG Exclude pattern: /tmp/obnam.log -2012-07-07 13:47:40 DEBUG opening repository (create=True) -2012-07-07 13:47:40 INFO VFS: __init__: baseurl=sftp://me@server/~/obnam/ -2012-07-07 13:47:40 DEBUG executing openssh: ['ssh', '-oForwardX11=no', '-oForwardAgent=no', '-oClearAllForwardings=yes', '-oProtocol=2', '-p', '22', '-l', 'me', '-s', '-o', 'UserKnownHostsFile=/home/me/.ssh/known_hosts', 'server', 'sftp'] -2012-07-07 13:47:42 INFO [chan obnam SSHChannelAdapter] Opened sftp connection (server version 3) -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] mkdir('obnam/', 511) -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] stat('obnam/') -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] normalize('obnam/') -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/./lock', 'wx') -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/./lock', 'wx') -> 00000000 -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata/format') -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -2012-07-07 13:47:42 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -> 00000000 -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata') -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/metadata', 511) -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', 'wx') -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', 'wx') -> 00000000 -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/metadata/format') -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/metadata/tmp.a8866c9f96aef3c3', '/path/to/me/obnam/metadata/format') -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') -2012-07-07 13:47:43 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist') -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') -2012-07-07 13:47:44 DEBUG Initializing Journal for clientlist -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') -2012-07-07 13:47:44 DEBUG Automatically rolling back remaining changes -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/') -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/') -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:44 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:47:45 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:47:46 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:47 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:48 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:49 DEBUG [chan obnam SSHChannelAdapter] rmdir('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/metadata') -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/metadata', 'r') -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/metadata', 'r') -> 00000000 -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/key', 'r') -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/key', 'r') -> 00000000 -2012-07-07 13:47:50 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/6', 'r') -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/6', 'r') -> 00000000 -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts/refcounts-0') -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/refcounts/refcounts-0', 'r') -2012-07-07 13:47:51 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/refcounts/refcounts-0', 'r') -> 00000000 -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/metadata/format') -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/metadata/format', 'r') -> 00000000 -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/2', 'r') -2012-07-07 13:47:52 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/nodes/0/0/0/2', 'r') -> 00000000 -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0/6') -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete') -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes', 511) -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0', 511) -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0/0', 511) -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0', 511) -2012-07-07 13:47:53 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', 'wx') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', 'wx') -> 00000000 -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/tmp.e1f00e0fb571930c', '/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/refcounts', 511) -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', 'wx') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', 'wx') -> 00000000 -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') -2012-07-07 13:47:54 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/refcounts/tmp.c4b90ce174b14a09', '/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes', 511) -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0', 511) -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0/0', 511) -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0', 511) -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', 'wx') -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', 'wx') -> 00000000 -2012-07-07 13:47:55 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/nodes/0/0/0/tmp.1f371a581e20dd5b', '/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') -2012-07-07 13:47:56 DEBUG LRUCache <larch.lru.LRUCache object at 0x9491e6c>: hits=0 misses=2 -2012-07-07 13:47:56 DEBUG LRUCache <larch.lru.LRUCache object at 0x9491f8c>: hits=3 misses=2 -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] mkdir('/path/to/me/obnam/clientlist/new', 511) -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', 'wx') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] open('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', 'wx') -> 00000000 -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] close(00000000) -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/new/metadata') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/tmp.334690ba416a3225', '/path/to/me/obnam/clientlist/new/metadata') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/') -2012-07-07 13:47:56 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/') -2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes') -2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0') -2012-07-07 13:47:57 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/delete/nodes/0/0/0') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/nodes/0/0/0/6') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/delete/nodes/0/0/0/6') -2012-07-07 13:47:58 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:47:59 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:48:00 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/nodes/0/0/0/7') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/nodes/0/0/0/7', '/path/to/me/obnam/clientlist/nodes/0/0/0/7') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes/0') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/nodes') -2012-07-07 13:48:01 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/nodes') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] listdir('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/refcounts/refcounts-0') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/refcounts/refcounts-0', '/path/to/me/obnam/clientlist/refcounts/refcounts-0') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/refcounts') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/refcounts') -2012-07-07 13:48:02 DEBUG [chan obnam SSHChannelAdapter] lstat('/path/to/me/obnam/clientlist/new/metadata') -2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/clientlist/metadata') -2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] rename('/path/to/me/obnam/clientlist/new/metadata', '/path/to/me/obnam/clientlist/metadata') -2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] remove('/path/to/me/obnam/./lock') -2012-07-07 13:48:03 DEBUG opening repository (create=False) -2012-07-07 13:48:03 DEBUG [chan obnam SSHChannelAdapter] stat('/path/to/me/obnam/obnam/') -2012-07-07 13:48:03 CRITICAL Traceback (most recent call last): - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 172, in _run - self.process_args(args) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 158, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 407, in process_args - method(args[1:]) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 226, in backup - self.add_client(client_name) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 293, in add_client - self.repo = self.app.open_repository(repofs=self.repo.fs.fs) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 187, in open_repository - repofs.reinit(repopath) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/sftp_plugin.py", line 58, in helper - raise OSError(e.errno, e.strerror or str(e), filename) -OSError: [Errno 2] No such file: 'sftp://me@server/~/obnam/' -</pre> - - ---- - -I've fixed this now. It turns out that the SFTP plugin did not handle -re-initialization repository paths that start with `/~/`. This will be -included in the next release. --liw - -[[done]] diff --git a/bugs/abort-if-sftp-breaks.mdwn b/bugs/abort-if-sftp-breaks.mdwn deleted file mode 100644 index f004de3..0000000 --- a/bugs/abort-if-sftp-breaks.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam does not currently seem to notice when the sftp connection -breaks. It should, and it should then abort the backup. ---liw diff --git a/bugs/acl-xattr.mdwn b/bugs/acl-xattr.mdwn deleted file mode 100644 index 0cd7130..0000000 --- a/bugs/acl-xattr.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Obnam should provide xattr and ACL support. --liw - -I only need to support extended attributes, since ACLs are implemented -in terms of those. Good, this makes things simpler. I can just read the -list of xattrs on backup, and set it on restore, and I can treat the -list as a string blob. It seems that in Linux, one needs to do a syscall -for each key/value pair for a file, but there's a library function that -operates on multiple pairs at once, and on Irix they're syscalls. Those -functions have a fairly complicated API, though. Might want to keep -things simple, at least initially. --liw - -This is now implemented for local filesystems. Paramiko does not seem -to support them for SFTP, so that will have to not work. --liw - -[[done]] diff --git a/bugs/all-filesystem-types.mdwn b/bugs/all-filesystem-types.mdwn deleted file mode 100644 index 2961ad7..0000000 --- a/bugs/all-filesystem-types.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Obnam needs tests for using every filesystem type available as live -data or repository. - -Not sure how to arrange that without root access, but there's a need -to do that. diff --git a/bugs/arch-diagram.mdwn b/bugs/arch-diagram.mdwn deleted file mode 100644 index 06aa8c3..0000000 --- a/bugs/arch-diagram.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be good to have an architecture diagram of the internals of -obnam, to make life easier for new code contributors. --liw - -[[done]] old wishlist bug, closing. --liw diff --git a/bugs/avoid-hash-collisions.mdwn b/bugs/avoid-hash-collisions.mdwn deleted file mode 100644 index eacb9d7..0000000 --- a/bugs/avoid-hash-collisions.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -It would be good for Obnam to optionally be able to verify that hash -collisions do not happen. This requires downloading the data with -the same hash, and comparing the bytes, which can be quite expensive. -However, backup programs should be reliable, and the user should be -able to choose reliability over speed. - -Three modes are needed: - -* assume hash collisions do not happen (deduplication, max speed, max risk) -* verify that hash collisions do not happen - (deduplication, no risk, performance hit) -* do not deduplicate (max speed, no risk) - ---liw - - -[[done]] diff --git a/bugs/avoid-stat-for-unchanged-directory.mdwn b/bugs/avoid-stat-for-unchanged-directory.mdwn deleted file mode 100644 index d4a95ba..0000000 --- a/bugs/avoid-stat-for-unchanged-directory.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -[[!tag obnam-wishlist]] - -This is an idea for optimizing Obnam. - -Store MD5 of string containing names + relevant metadata of all -files in a directory, then re-compute that when backing up a -directory: if checksums are same, then no file in the directory -has changed, and there's no need to check each of them separately, -saving many tree lookups - -Consider only non-dirs, since subsubdir can change without it being visible -at grandparent level. Thus, recursing is always necessary. - ---liw - - -[[done]] wishlist has been open for years, no change. --liw diff --git a/bugs/backs-parent-of-root-if-relative.mdwn b/bugs/backs-parent-of-root-if-relative.mdwn deleted file mode 100644 index b771059..0000000 --- a/bugs/backs-parent-of-root-if-relative.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!meta title="Backs up parent of live data root, if relative path used"]] - -If the live data root directory is given with a relative pathname, Obnam will back up its parent instead. --liw - -[[done]] --liw diff --git a/bugs/backup-downloads-too-much-data.mdwn b/bugs/backup-downloads-too-much-data.mdwn deleted file mode 100644 index 612b41c..0000000 --- a/bugs/backup-downloads-too-much-data.mdwn +++ /dev/null @@ -1,25 +0,0 @@ -Andrew Ruthwen reported that a small backup delta was very slow, and had these -performance stats: - - 2014-04-17 18:56:34 INFO Backup performance statistics: - 2014-04-17 18:56:34 INFO * files found: 3643 - 2014-04-17 18:56:34 INFO * files backed up: 7 - 2014-04-17 18:56:34 INFO * uploaded chunk data: 18818 bytes (18 KiB) - 2014-04-17 18:56:34 INFO * total uploaded data (incl. metadata): 3097917 - bytes (2 MiB) - 2014-04-17 18:56:34 INFO * total downloaded data (incl. metadata): - 89711629838 bytes (83 GiB) - 2014-04-17 18:56:34 INFO * transfer overhead: 89714708937 bytes (83 GiB) - 2014-04-17 18:56:34 INFO * duration: 29275.9613361 s (8h7m56s) - 2014-04-17 18:56:34 INFO * average speed: 0.642779917077 B/s - -This is way too much downloaded data. - -<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-April/002979.html> - ---liw - - -There's been some fixes to this, and Obnam is now much clearer about -the overhead. Also, most fixes are going into FORMAT GREEN ALBATROSS. -[[done]] --liw diff --git a/bugs/backup-plugin-reporting.mdwn b/bugs/backup-plugin-reporting.mdwn deleted file mode 100644 index e54df15..0000000 --- a/bugs/backup-plugin-reporting.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Fix obnam backup plugin to report when it's doing something else than -backing up a file. ---liw -[[done]] diff --git a/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn b/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn deleted file mode 100644 index 524b329..0000000 --- a/bugs/backup-pretend-should-list-filenames-to-stdout.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -make "obnam backup --pretend" print a list of would-be-backed-up files -to stdout instead of tty - - -[[done]] I have decided against this. --liw diff --git a/bugs/backup-progress-unchanged-chunks.mdwn b/bugs/backup-progress-unchanged-chunks.mdwn deleted file mode 100644 index bfa940a..0000000 --- a/bugs/backup-progress-unchanged-chunks.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -When Obnam is backing up a large file with many unchanged chunks, the -progress reporting should indicate this to the user. - -[[done]] -- this should be done now. --liw diff --git a/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn b/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn deleted file mode 100644 index 23bc3a2..0000000 --- a/bugs/bad-error-message-with-bad-larch-or-repo-version.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655229> for details. - -[[done]] diff --git a/bugs/bad-message-for-regexp-syntax-error.mdwn b/bugs/bad-message-for-regexp-syntax-error.mdwn deleted file mode 100644 index 624cd27..0000000 --- a/bugs/bad-message-for-regexp-syntax-error.mdwn +++ /dev/null @@ -1,27 +0,0 @@ -If an exclude regexp has a syntax error, the error message is awful: - - > peter@dewsbx07:/media/GreenDrive/Backups$ obnam backup $HOME - > Traceback (most recent call last): - > File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 141, in _run - > self.process_args(args) - > File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 123, in - > process_args - > cliapp.Application.process_args(self, args) - > File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 316, in - > process_args - > method(args[1:]) - > File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", - > line 95, in backup - > self.compile_exclusion_patterns() - > File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", - > line 136, in compile_exclusion_patterns - > for x in self.app.settings['exclude']] - > File "/usr/lib/python2.6/re.py", line 190, in compile - > return _compile(pattern, flags) - > File "/usr/lib/python2.6/re.py", line 245, in _compile - > raise error, v # invalid expression - > error: nothing to repeat - ---liw - -[[done]] diff --git a/bugs/baserock.mdwn b/bugs/baserock.mdwn deleted file mode 100644 index 36e5574..0000000 --- a/bugs/baserock.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -Obnam hasn't been ported to Baserock. - -[[done]] -- I'll port it when I need it, and wishlist bugs are a burden. --liw diff --git a/bugs/benchmark-fsck.mdwn b/bugs/benchmark-fsck.mdwn deleted file mode 100644 index 05b96d4..0000000 --- a/bugs/benchmark-fsck.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -`obnam-benchmark` should benchmark also the fsck operation. --liw - -next version of [[/seivot]] will do this. --liw - -[[done]] diff --git a/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn b/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn deleted file mode 100644 index 574b27a..0000000 --- a/bugs/benchmarks-make-restore-elapsed-be-zeroes.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -When running `obnam-benchmark`, the restore elapsed timers end at -00:00:00. Why? --liw - -It's not a bug: it's reporting remaining time, not elapsed time. Oops. ---liw -[[done]] diff --git a/bugs/better-way-to-upgrade-format-versions.mdwn b/bugs/better-way-to-upgrade-format-versions.mdwn deleted file mode 100644 index 35edef9..0000000 --- a/bugs/better-way-to-upgrade-format-versions.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -Instead of in-place conversions, which are error prone and clunky, -a better way would be nice. Maybe some kind of dump/undump pair, -using a streamable format? diff --git a/bugs/bgproc.mdwn b/bugs/bgproc.mdwn deleted file mode 100644 index be834fd..0000000 --- a/bugs/bgproc.mdwn +++ /dev/null @@ -1,103 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should do some processing in the background, for example uploading -of data to the backup repository. This would allow better use of the -bottleneck resource (network). Below is a journal entry with my thoughts -on how to implement that. It may be out of date by now, but we'll see. -I have a Python module to simplify the use of `multiprocessing` to do -jobs in the background (which avoids the Python global interpreter lock, -in case that matters). --liw - ---- - -Here's a design for Obnam concurrency that came to me the other -day while walking. - -The core of Obnam (and larch) is quite synchronous: read data from -file, read B-tree nodes, push chunks and B-tree nodes into repository. -Some of that can be parallelized, but not easily: it's already tricky -code, and making it even more tricky is going to require very strong -justification. - -Things like encrypting and decrypting files need to be done in parallel -with other things, for speed. These things are not really in the -core, and indeed are provided by plugins. - -So here's a way to them in parallel: - -* the core code stays synchronous, the way it is now -* whenever larch code needs to read a B-tree node, - it blocks until it gets it -* the node is read, synchronously, from wherever, and put into - a background processing queue (using Python multiprocess) -* the code that waits for the node to be processed polls the queue, - and handles any other background jobs that happen to finish while - it waits, and returns the desired node when it gets it -* when larch writes a node (after it gets pushed out - of the upload queue inside larch), it is put into a background - processing queue -* at the same time, if there were any finished background jobs, they're - handled (written to repo) -* at the end of the run, the main loop makes sure any pending background - jobs finish and are handled - -There's a complication that the B-tree code may need a node that is -not yet written to the repository, since it is still going through -a background processing queue. - -I'm going to need to restructure how hooks process files that -are written to or read from the repository. Writing should happen -asynchronously: files are put in a queue and processed in the -background, and then written to the actual repository when background -processing is finished. Reading needs to happen synchronously, since -there's a B-tree call waiting for them, but to handle the case of -needing a node that is still being processed -in the background, we need to keep track of what nodes are in the -background, and wait for them to be done before reading them. - -Reading would thus be something like this, implemented in the -`Repository` class: - - while wanted file is in write queue: - process a write queue result - - read file from repository - process file data through hooks - return file - -The write queue is more complicated (again handled somehow in the -`Repository` class): - -* a `multiprocessing.Queue` instance for holding pending jobs - - a job is a (pathname, file contents) pair -* another `Queue` instance for holding unhandled results - - (pathname, file contents) pair, where the contents may have changed -* a `set` for holding file identifiers (paths) that have been put into - the pending jobs queue, but not yet processed from the results queue - -Each plugin can provide one or more Unix commands (filters) through -which the file contents gets piped. The background processes run each -filter in turn, giving the output of the previous one as input to the -next one. - -To handle a result from a background job, the following needs to be done: - -* remove the pathname from the `set` -* write the filtered file contents into the repository - -To implement this, I'll do this: - -* All changes should be in `HookedFS` -* `write_file` and `overwrite_file` put things into the pending jobs queue, - and also call a new method `handle_background_results` -* `cat` gets changed to wait for files in the write queue, calling - `handle_background_results` -* `handle_background_results` will do what is needed - -This design isn't optimal, since writing things to the repository -isn't being done in parallel with other things, but I'll tackle that -problem later. - - -[[done]] this clearly isn't happening, so closing the old wishlist -bug --liw diff --git a/bugs/block-device-backup.mdwn b/bugs/block-device-backup.mdwn deleted file mode 100644 index d570402..0000000 --- a/bugs/block-device-backup.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam could do with a mode in which it backs up the data from a block -device, instead of the device node. If the block device contains a -filesystem, it should backup only the parts of the device that are -used by the filesystem, and skip unused parts. That could be used for -backing up disk images as well. diff --git a/bugs/c-lstat.mdwn b/bugs/c-lstat.mdwn deleted file mode 100644 index fc9399c..0000000 --- a/bugs/c-lstat.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Obnam is using the Python os.lstat method to get metadata from files. -Python returns timestamps as floats, which are awkward to store on -disk. It would be good to switch to a C extension that does the lstat -system call and returns timestamps as (pairs of) integers instead. -This would allow full timestamp precision. - ---liw - -[[done]] diff --git a/bugs/chunk-size-based-on-data-type-and-size.mdwn b/bugs/chunk-size-based-on-data-type-and-size.mdwn deleted file mode 100644 index 4fcfc33..0000000 --- a/bugs/chunk-size-based-on-data-type-and-size.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!tag obnam-wishlist]] - -Would it be beneficial to choose the size of data chunks based on the -type of data, or its size? Text data might benefit from small chunks, -whereas video data might benefit from large chunks, for example. -Possibly the size of a file would be a sufficient indicator. - -The goal would be to reduce the number of chunks, without reducing -the effectiveness of de-duplication. - -This needs to be measurements of real data to see if there are interesting -parameters. - ---liw - -[[done]] interesting ideas, but old wishlist bug. --liw diff --git a/bugs/chunks-dir-init-has-no-lock.mdwn b/bugs/chunks-dir-init-has-no-lock.mdwn deleted file mode 100644 index 36402d6..0000000 --- a/bugs/chunks-dir-init-has-no-lock.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -When Obnam initializes the `chunks` directory, for encryption, it is -done without locking. That's a race condition: two clients started -at the same time against an empty directory may result in both of -them creating the same files. - -Normal use of the `chunks` directory does not need locking: clients -are free to create new chunks files there, as long as they take care -to not overwrite existing files. - ---liw - -[[done]] in bzr diff --git a/bugs/clients-list-should-include-ids.mdwn b/bugs/clients-list-should-include-ids.mdwn deleted file mode 100644 index 00232f8..0000000 --- a/bugs/clients-list-should-include-ids.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -Obnam needs a way to show the id of each client, so one can, say, -safely remove the toplevel directory for the client. - ---- - -Obnam now has an internal interface API for accessing repositories, -and it doesn't expose the client id anymore. So this needs some -thought: should we expose the id (even though it's not otherwise -needed)? And what about future repository interface API -implementations that don't need a client id at all anymore? I suspect -it isn't going to useful enough to do this. If I get other uses -cases, I'll re-consider. [[done]], for now. --liw - diff --git a/bugs/clients-subcommand-for-non-clients.mdwn b/bugs/clients-subcommand-for-non-clients.mdwn deleted file mode 100644 index 28024d6..0000000 --- a/bugs/clients-subcommand-for-non-clients.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -The "obnam clients" command should work even for non-clients (modulo -encryption), so that you can get a list of the client names without -having to guess one. --liw - -Fixed now. [[done]] --liw diff --git a/bugs/commit-progress-reporting.mdwn b/bugs/commit-progress-reporting.mdwn deleted file mode 100644 index 5db52e9..0000000 --- a/bugs/commit-progress-reporting.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -[[!tag obnam-wishlist]] - -Improve Obnam's progress reporting during committing. - -- how much work will there be to do the commit? -- how many files to upload? -- how many files to move in journal? - - ---- - -[[done]] as this requires changes to larch and I want to not touch it anymore. -A new repository format will drop larch anyway. --liw diff --git a/bugs/committing-changes-takes-long-without-feedback.mdwn b/bugs/committing-changes-takes-long-without-feedback.mdwn deleted file mode 100644 index 3aa744c..0000000 --- a/bugs/committing-changes-takes-long-without-feedback.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -When Obnam is committing changes (at end, or during checkpoint), -there can be a long time when nothing seems to be happening. It -needs to provide some feedback of what's going on. - - --- - -[[done]] --liw diff --git a/bugs/compression-and-encryption-together-fails.mdwn b/bugs/compression-and-encryption-together-fails.mdwn deleted file mode 100644 index 3de0071..0000000 --- a/bugs/compression-and-encryption-together-fails.mdwn +++ /dev/null @@ -1,46 +0,0 @@ -Trying to create a new backup using both compression and encryption fails. The command I use is: - - obnam --compress-with=gzip --encrypt-with=ABCD1234 --repository /tmp/asd backup $HOME/stuff - -The error message printed: “error: Error -3 while decompressing data: incorrect header check” -You may view the whole stack trace at http://p.nnev.de/2368 because I couldn’t figure out how to add it properly with the formatting options. - ---xeen - --- - -The above stack trace: - - Traceback (most recent call last): - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 144, in _run - self.process_args(args) - File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 127, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 350, in process_args - method(args[1:]) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 115, in backup - self.add_client(client_name) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 143, in add_client - self.repo.add_client(client_name) - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 321, in add_client - if client_name in self.list_clients(): - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 200, in list_clients - self.check_format_version() - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 311, in check_format_version - on_disk = self.get_format_version() - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 282, in get_format_version - data = self.fs.cat('metadata/format') - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 66, in cat - repo=self.repo, toplevel=toplevel) - File "/usr/lib/python2.6/dist-packages/obnamlib/hooks.py", line 109, in call - return self.hooks[name].call_callbacks(*args, **kwargs) - File "/usr/lib/python2.6/dist-packages/obnamlib/hooks.py", line 73, in call_callbacks - data = callback(data, *args, **kwargs) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/compression_plugin.py", line 45, in toplevel_read_data - return zlib.decompress(data) - error: Error -3 while decompressing data: incorrect header check - - ---liw - -Fixed in bzr. [[done]] --liw diff --git a/bugs/compression-methods.mdwn b/bugs/compression-methods.mdwn deleted file mode 100644 index 967e391..0000000 --- a/bugs/compression-methods.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -Supporting multiple compression methods would be nice. - -Also one idea is to allow using one compression method (or no compression) while actually doing the backup, and then have a separate tool, to be run on otherwise idle time, recompress the data using some other, slower algorithm to conserve disk space without making the backup process slower. - --- SLi - -[[done]] while this would be useful, it hasn't happened, and making -the bug list longer is hampering other things. --liw diff --git a/bugs/crash_when_started_without_--log_option.mdwn b/bugs/crash_when_started_without_--log_option.mdwn deleted file mode 100644 index cbccc5d..0000000 --- a/bugs/crash_when_started_without_--log_option.mdwn +++ /dev/null @@ -1,25 +0,0 @@ -obnam 0.23 crashes when started without --log option. Here is what it spits out on the console: - - user@host:~$ obnam backup ./ - CRITICAL:root:Traceback (most recent call last): - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 132, in _run - self.setup_logging() - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 279, in setup_logging - handler = logging.NullHandler() - AttributeError: 'module' object has no attribute 'NullHandler' - - Traceback (most recent call last): - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 132, in _run - self.setup_logging() - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 279, in setup_logging - handler = logging.NullHandler() - AttributeError: 'module' object has no attribute 'NullHandler' - --- weinzwang - -This is actually a bug in cliapp 0.22, which switched to using -`logging.NullHandler`. That only exists in Python 2.7, and you're -using 2.6. Workaround: `obnam --log=/dev/null`. I'll release a new -cliapp soon, it's already fixed in bzr. --liw - -[[done]] diff --git a/bugs/crashing-early-leads-to-assertion-error.mdwn b/bugs/crashing-early-leads-to-assertion-error.mdwn deleted file mode 100644 index bda5d11..0000000 --- a/bugs/crashing-early-leads-to-assertion-error.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -> `<SLi> liw, I run obnam backup for a small while (probably not -> until even the first checkpoint) on a new repository, then ctrl-c -> and say obnam fsck, I get AssertionError on assert 'key_size' in -> ns_temp.get_metadata_keys().` - -Two things: - -* using `assert` is wrong in the code, it should raise an exception that - results in a humane error message -* crashing before the first checkpoint should not result in an error - in the first place - ---liw - -I've fixed the assertion in larch (it's now a more user-friendly -error message). The crash still occurs, but I'm not sure if it's -reasonable for Obnam to try to correct that. The same error may -happen due to other reasons, and automatically fixing things is -too likely to break things. So I'll leave the bug open, but take -it off the 1.0 blockers list. - ---liw - -I can reproduce this, by instrumenting the code so that -it crashes after having backed up the first file. The error printed -correctly identifies that the B-tree is broken. However, it is -broken in a useful way. The B-tree is going to be so broken it -does not have much useful information. There are some things that -could be done (e.g., if no key size is stored in the B-tree, add -one), but that only helps in specific cases, and I don't think -that's useful enough. Thus, I think I will just give up for this -kind of early corruption and have people start over. Sorry. -(I'd fix this if I had lots of extra time, but I don't. If someone -comes upon this bug later on, and makes a good patch, I'd be -happy to accept it, of course.) - ---liw - -[[done]] diff --git a/bugs/ctime.mdwn b/bugs/ctime.mdwn deleted file mode 100644 index 6b6433e..0000000 --- a/bugs/ctime.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should, arguably, use ctime changes to trigger backups, so that -if a file's size and mtime are the same, because whatever fool program -modified a file reset the mtime, obnam will still backup the changed -data. - -I have code for this, but it requires a repository format change, -and breaks the upgrade from format 5 to 6. --liw diff --git a/bugs/deduplication-stats.mdwn b/bugs/deduplication-stats.mdwn deleted file mode 100644 index 71a689e..0000000 --- a/bugs/deduplication-stats.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be nice if Obnam could tell you how much you've saved thanks -to the de-duplication. --liw - -From discussion on the mailing list ([archive](http://vlists.pepperfish.net/pipermail/obnam-flarn.net/2012-June/000120.html): - -* how much space will be freed if I remove this generation? -* or a set of generations? -* how much space has been saved by de-duplicated chunks in the whole repo? - -Also, how much space has been saved with compression (alone or with encryption? - -Also, store per-generation data in the generation for faster retrieval. - -* Bytes added by this generation. -* Number of added/changed/removed files in this generation. - ---liw - -[[done]] if keeping the wishlist bug open would help make this happen, -it would have happened already. --liw diff --git a/bugs/default-exclusions.mdwn b/bugs/default-exclusions.mdwn deleted file mode 100644 index c22483a..0000000 --- a/bugs/default-exclusions.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -From Enrico: It might be good to have a way for Obnam to automatically -exclude certain kinds of common stuff, such as web browser caches, -Liferea caches, etc. This should be easy to enable, and should be -off by default (safe defaults are important). diff --git a/bugs/delete-files-from-generations.mdwn b/bugs/delete-files-from-generations.mdwn deleted file mode 100644 index 5530859..0000000 --- a/bugs/delete-files-from-generations.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -If you accidentally backup some large or sensitive files, but don't want -to delete all the generations they're in, it would be handy for Obnam to -be able to delete just the specific files from the generations, and leave -the rest. diff --git a/bugs/disable-paramiko-logging.mdwn b/bugs/disable-paramiko-logging.mdwn deleted file mode 100644 index 5747efb..0000000 --- a/bugs/disable-paramiko-logging.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be good to have obnam provide an option to enable/disable -paramiko logging. It is often voluminous and might not be helpful -when reading logs for debugging. --liw - - -This doesn't seem to be a problem anymore. [[done]] --liw diff --git a/bugs/disk-full.mdwn b/bugs/disk-full.mdwn deleted file mode 100644 index ca8d892..0000000 --- a/bugs/disk-full.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -How does Obnam deal with a full disk for the repository? - -- create a disk image of suitable size and create an ext4 fs on it -- mount the disk image and use it as a repository -- backup something bigger than what will fit into the disk image -- how does obnam handle the error? - ---liw - - -Obnam now aborts properly on out-of-space errors. --liw - -[[done]] diff --git a/bugs/do-not-cross-bind-mounts.mdwn b/bugs/do-not-cross-bind-mounts.mdwn deleted file mode 100644 index 8bf7850..0000000 --- a/bugs/do-not-cross-bind-mounts.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -When --one-file-system is used, it would be nice to not cross -bind-mounts. No idea how to figure that out, but it must be -possible. --liw - -You could look at the inode numbers for . and ./foodir/.. and check they're the same? -- kinnison - -The inode check will not work if foodir is a symlink. --mathstuf diff --git a/bugs/document-black-box-testing.mdwn b/bugs/document-black-box-testing.mdwn deleted file mode 100644 index 5bcbc38..0000000 --- a/bugs/document-black-box-testing.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -Review obnam black box tests and document how you're supposed to write them. - -- hacking section in README - - ---- - -On second though, I'll just let people read yarn documentation, -these days. --liw [[done]] diff --git a/bugs/document-how-to-check-encryption-is-being-used.mdwn b/bugs/document-how-to-check-encryption-is-being-used.mdwn deleted file mode 100644 index 1c44e3d..0000000 --- a/bugs/document-how-to-check-encryption-is-being-used.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -Document how to check that Obnam is actually encrypting anything. --liw - - -This is in the manual now -(http://code.liw.fi/obnam/manual/manual.html#checking-if-a-repository-uses-encryption). -[[done]] --liw diff --git a/bugs/document-repo-upgrade.mdwn b/bugs/document-repo-upgrade.mdwn deleted file mode 100644 index 345b8b5..0000000 --- a/bugs/document-repo-upgrade.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Although there is currently no repository upgrade method, -the manpage should at least document that you need to start -over from scratch. - -Better yet, make a way to upgrade, of course. - ---liw - -[[done]] (see [[repo-upgrades-missing]] diff --git a/bugs/does-not-pass-test-suite-on-fedora17.mdwn b/bugs/does-not-pass-test-suite-on-fedora17.mdwn deleted file mode 100644 index 3f63ee0..0000000 --- a/bugs/does-not-pass-test-suite-on-fedora17.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Make Obnam pass its test suite (esp. test-locking) on Fedora 17. - -- yum install gcc python-sphinx python-devel - ---- - -I believe this is now the case, since there are Fedora packages. -[[done]] ---liw diff --git a/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn b/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn deleted file mode 100644 index d621452..0000000 --- a/bugs/does-not-unlock-when-failing-when-live-data-goes-away.mdwn +++ /dev/null @@ -1,29 +0,0 @@ - <andrewsh> 20:53:15> liw: hi, I have strange issue with obnam's locks - <andrewsh> 20:53:35> I had OSError: [Errno 2] No such file or - directory: '/media/box' - <andrewsh> 20:53:56> which is itself okay, a remote mount was - inaccessible at that moment - <andrewsh> 20:54:05> obnam said, 2012-08-29 21:10:19 INFO Unlocking - client because of error - <andrewsh> 20:54:20> however, it did not unlock the repository - really - <andrewsh> 20:54:33> and the next time it gave me LockFail: - Lock timeout - <andrewsh> 21:01:47> and this is obnam 1.0 if that matters - <andrewsh> yes - <andrewsh> liw: - <andrewsh> well, no - <andrewsh> well, both :D - <andrewsh> remote mount wasn't the repo, remote mount was what - I wanted to back up - <andrewsh> repo was on the local disk - <andrewsh> unavailability of a remote mount to back up caused - an interruption - <andrewsh> but despite telling me it's unlocking the repo, obnam - didn't do this - - ---- - -This should now be fixed in bzr. --liw -[[done]] diff --git a/bugs/done.mdwn b/bugs/done.mdwn deleted file mode 100644 index fe67f2f..0000000 --- a/bugs/done.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Closed bugs for Obnam -===================== - -[[!inline pages="page(bugs/*) and - !bugs/*/* and - !bugs/done and - link(bugs/done)" - show=0 - sort=mtime - archive=yes]] - diff --git a/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn b/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn deleted file mode 100644 index 08ce0f7..0000000 --- a/bugs/done/FTBFS:___34__FAIL:_root-is-symlink:_got_exit_code_1__44___expected_0__34__.mdwn +++ /dev/null @@ -1,298 +0,0 @@ -(I can't figure out how to simply mark a block of text as -code or monospaced, other than indenting every line manually. -The FormattingHelp page isn't all that helpful.) - -> I've reformatted. Indenting by four spaces is the markdown way, -> or you can use the pre element in HTML (markdown and Branchable -> allow a subset of HTML to be used, including pre). --liw - -When I try to build Obnam 1.0 I get this error. I noticed that the -same build error occurs in Chris's PPA, and so Obnam isn't available -on Precise or Quantal. - ----- - -<pre> -rm -rf build -cp -a test-gpghome temp.gpghome -env GNUPGHOME=temp.gpghome python setup.py check --fast -running check -run unit tests -Running test 167/387: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' -Running test 168/387: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' -Running test 387/387: test_removes_chunk (chunklist_tests.ChunkListTests - -OK -121 excluded statements -22 excluded modules -Time: 5.1 s -run black box tests -FAIL: root-is-symlink: got exit code 1, expected 0 -41/42 tests OK, 1 failures -ERROR: Command '['cmdtest', 'tests']' returned non-zero exit status 1 -</pre> - ----- - -I don't know why root-is-symlink.script is failing. There is output, -and I guess there is not supposed to be? - -(Is there really no way to use preformatted text in Markdown other -than indenting every line? This seems like a step backwards. -What's wrong with `[pre][/pre]` ?) - -<pre> -$ cat root-is-symlink.stdout-actual ---- /tmp/tmp16rwh9/data/data.summain 2012-06-09 16:39:12.000000000 -0500 -+++ /tmp/tmp16rwh9/data/restored.summain 2012-06-09 16:39:12.000000000 -0500 -@@ -11,127 +11,3 @@ - Group: me - Target: data.real - --Name: 0 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 40775 --Ino: 2 --Dev: 1 --Nlink: 3 --Uid: 1000 --Username: me --Gid: 1000 --Group: me -- --Name: 0/0 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 40775 --Ino: 3 --Dev: 1 --Nlink: 3 --Uid: 1000 --Username: me --Gid: 1000 --Group: me -- --Name: 0/0/0 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 40775 --Ino: 4 --Dev: 1 --Nlink: 2 --Uid: 1000 --Username: me --Gid: 1000 --Group: me -- --Name: 0/0/0/0 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 5 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: 7b2f423343cc5a4b5afbd8792b84aeee3b1eef68 -- --Name: 0/0/0/1 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 6 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: 99b6a6eac04a1dee31e07245c60bf6ff60c5dbb7 -- --Name: 0/0/0/2 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 7 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: ca26a73aaf98ba4634794952c29bc02bed536529 -- --Name: 0/0/0/3 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 8 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: d60b38536692c387932ffca6b59a235fa7744c22 -- --Name: 0/0/0/4 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 9 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: e5213924a2ce010a667605951a2abee56f2373da -- --Name: 0/0/0/5 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 10 --Dev: 1 --Nlink: 1 --Size: 16384 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: f1837f15230109d673bf741afc0bb8e8cd343538 -- --Name: 0/0/0/6 --Mtime: 2012-06-09 21:39:11.000000 +0000 --Mode: 100664 --Ino: 11 --Dev: 1 --Nlink: 1 --Size: 1696 --Uid: 1000 --Username: me --Gid: 1000 --Group: me --SHA1: 578a819d61ad07e996900a236cbc33b70d47cd75 -</pre> - - ---- - -The PPA now has 1.0 for Ubuntu, so I assume the problem is fixed. -I seem to recall it was a problem of a too-old dependency. Are you -still having this problem? --liw - ---- - -Thanks for reformatting my stuff. - -I still see the build failure in the PPA for Precise and Quantal: - -https://launchpad.net/~chris-bigballofwax/+archive/obnam-ppa/+packages - -It hasn't been attempted for 10 days, but I don't see any upgradable Python packages on my system, so I'm not sure it would help. - -On the other hand, I just ran debian/rules on 1.0 again, and now the output is different. Maybe I was doing it wrong before? It seems to complete, but it also says "FAILED" and that it exited non-zero. I am confused. :) - -<pre> -debian/rules -python setup.py build_ext -i -running build_ext -building 'obnamlib._obnam' extension -creating build -creating build/temp.linux-i686-2.7 -gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _obnammodule.c -o build/temp.linux-i686-2.7/_obnammodule.o -gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-i686-2.7/_obnammodule.o -o /home/me/Projects/Python/obnam/trunk/obnamlib/_obnam.so -rm -rf build -cp -a test-gpghome temp.gpghome -env GNUPGHOME=temp.gpghome python setup.py check --fast -running check -run unit tests -Running test 167/774: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' -Running test 168/774: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' -Running test 554/774: test_missing_filter_gives_tag (hooks_tests.FilterHWARNING:root:Missing tag: 'missing' -Running test 555/774: test_missing_filter_raises (hooks_tests.FilterHookWARNING:root:Missing tag: 'missing' -Running test 774/774: test_removes_chunk (chunklist_tests.ChunkListTests - -FAILED - -Statements missed by per-module tests: - Module Missed statements - obnam-1.0/obnamlib/checksumtree.py 34-39, 42, 45, 48-53, 56-63, 66-71, 75-81 - obnam-1.0/obnamlib/checksumtree.py 34-39, 42, 45, 48-53, 56-63, 66-71, 75-81 - obnam-1.0/obnamlib/chunklist.py 37-42, 45, 48-51, 54-57, 60-63 - obnam-1.0/obnamlib/chunklist.py 37-42, 45, 48-51, 54-57, 60-63 - obnam-1.0/obnamlib/clientlist.py 49-59, 62, 65, 68-69, 72, 75, 78-84, 87-93, 96-99, 102-114, 117-122, 125-132, 135-142 - obnam-1.0/obnamlib/clientlist.py 49-59, 62, 65, 68-69, 72, 75, 78-84, 87-93, 96-99, 102-114, 117-122, 125-132, 135-142 - obnam-1.0/obnamlib/clientmetadatatree.py 73-82, 85-86, 90-95, 99, 124-131, 135, 139-140, 144, 148, 152, 157-158, 163-180, 185-204, 207, 210, 213-224, 227-228, 231-232, 236-250, 253-261, 264-271, 274-277, 280-285, 288-292, 295, 298-299, 304-307, 310-311, 315, 318-323, 326-328, 331, 334-369, 372-376, 379-387, 390-432, 435-445, 448-459, 462-463, 466-469, 472-474, 477-491, 494-509, 514-517, 522-525 - obnam-1.0/obnamlib/clientmetadatatree.py 73-82, 85-86, 90-95, 99, 124-131, 135, 139-140, 144, 148, 152, 157-158, 163-180, 185-204, 207, 210, 213-224, 227-228, 231-232, 236-250, 253-261, 264-271, 274-277, 280-285, 288-292, 295, 298-299, 304-307, 310-311, 315, 318-323, 326-328, 331, 334-369, 372-376, 379-387, 390-432, 435-445, 448-459, 462-463, 466-469, 472-474, 477-491, 494-509, 514-517, 522-525 - obnam-1.0/obnamlib/encryption.py 29-36, 44, 47-49, 52-54, 57, 79-97, 102, 107, 113-129, 134, 149-151, 154-157, 160-161, 165, 168-175, 178-180, 183, 186, 189-192, 195, 198, 201-212, 222-229, 234-237, 248 - obnam-1.0/obnamlib/encryption.py 29-36, 44, 47-49, 52-54, 57, 79-97, 102, 107, 113-129, 134, 149-151, 154-157, 160-161, 165, 168-175, 178-180, 183, 186, 189-192, 195, 198, 201-212, 222-229, 234-237, 248 - obnam-1.0/obnamlib/forget_policy.py 49-73, 76-93, 111-118 - obnam-1.0/obnamlib/forget_policy.py 49-73, 76-93, 111-118 - obnam-1.0/obnamlib/hooks.py 44-45, 54-60, 64-65, 69-71, 79-81, 99-100, 103-107, 110-111, 114, 117-123, 126-137, 145-146, 155-156, 160-161, 165-168, 172-175, 179, 183, 187 - obnam-1.0/obnamlib/hooks.py 44-45, 54-60, 64-65, 69-71, 79-81, 99-100, 103-107, 110-111, 114, 117-123, 126-137, 145-146, 155-156, 160-161, 165-168, 172-175, 179, 183, 187 - obnam-1.0/obnamlib/lockmgr.py 28-34, 37-43, 52-54, 57, 61-70, 73, 77-85, 89-90 - obnam-1.0/obnamlib/lockmgr.py 28-34, 37-43, 52-54, 57, 61-70, 73, 77-85, 89-90 - obnam-1.0/obnamlib/metadata.py 80-83, 86, 89, 92, 100-109, 167-192, 207-220, 242-265, 275-316 - obnam-1.0/obnamlib/metadata.py 80-83, 86, 89, 92, 100-109, 167-192, 207-220, 242-265, 275-316 - obnam-1.0/obnamlib/pluginbase.py 25 - obnam-1.0/obnamlib/pluginbase.py 25 - obnam-1.0/obnamlib/repo.py 47-49, 52, 55-57, 62-66, 70, 73-78, 81-86, 128-156, 159, 164-167, 174-178, 187-189, 193, 197, 201, 206-211, 215-216, 220-221, 225-226, 230-231, 235-236, 240-241, 245-246, 250-251, 261-272, 276-282, 286-300, 309-322, 326-329, 339-341, 348-353, 363-367, 371, 382-396, 402-406, 410-414, 424-448, 452-462, 466-476, 480-492, 496-497, 501-502, 511-519, 530-573, 577-580, 589-590, 594-595, 600-605, 609-611, 615-616, 619, 632-655, 666-673, 677-678, 682-683, 692-693, 697-704, 717-725, 729-730, 739-740, 749-750, 765-779, 790-803 - obnam-1.0/obnamlib/repo.py 47-49, 52, 55-57, 62-66, 70, 73-78, 81-86, 128-156, 159, 164-167, 174-178, 187-189, 193, 197, 201, 206-211, 215-216, 220-221, 225-226, 230-231, 235-236, 240-241, 245-246, 250-251, 261-272, 276-282, 286-300, 309-322, 326-329, 339-341, 348-353, 363-367, 371, 382-396, 402-406, 410-414, 424-448, 452-462, 466-476, 480-492, 496-497, 501-502, 511-519, 530-573, 577-580, 589-590, 594-595, 600-605, 609-611, 615-616, 619, 632-655, 666-673, 677-678, 682-683, 692-693, 697-704, 717-725, 729-730, 739-740, 749-750, 765-779, 790-803 - obnam-1.0/obnamlib/sizeparse.py 25, 31, 37, 61, 64-66, 69-79 - obnam-1.0/obnamlib/sizeparse.py 25, 31, 37, 61, 64-66, 69-79 - obnam-1.0/obnamlib/vfs_local.py 34-39, 42-45, 55-63, 76-91, 94, 97-101, 104-109, 114-116, 119, 122-124, 127-129, 132-137, 157, 160, 185-186, 189-197, 200-203, 206, 209-212, 215-227, 230, 233, 236-238, 241-243, 246-248, 251-253, 256-267, 270-279, 282-286, 289-306, 309, 312-319 - obnam-1.0/obnamlib/vfs_local.py 34-39, 42-45, 55-63, 76-91, 94, 97-101, 104-109, 114-116, 119, 122-124, 127-129, 132-137, 157, 160, 185-186, 189-197, 200-203, 206, 209-212, 215-227, 230, 233, 236-238, 241-243, 246-248, 251-253, 256-267, 270-279, 282-286, 289-306, 309, 312-319 - -Modules missing test modules: - obnam-1.0/setup.py - obnam-1.0/test-plugins/wrongversion_plugin.py - obnam-1.0/test-plugins/oldhello_plugin.py - obnam-1.0/test-plugins/aaa_hello_plugin.py - obnam-1.0/test-plugins/hello_plugin.py - obnam-1.0/obnamlib/__init__.py - obnam-1.0/obnamlib/vfs.py - obnam-1.0/obnamlib/app.py - obnam-1.0/obnamlib/repo_tree.py - obnam-1.0/obnamlib/plugins/__init__.py - obnam-1.0/obnamlib/plugins/restore_plugin.py - obnam-1.0/obnamlib/plugins/verify_plugin.py - obnam-1.0/obnamlib/plugins/forget_plugin.py - obnam-1.0/obnamlib/plugins/show_plugin.py - obnam-1.0/obnamlib/plugins/force_lock_plugin.py - obnam-1.0/obnamlib/plugins/sftp_plugin.py - obnam-1.0/obnamlib/plugins/vfs_local_plugin.py - obnam-1.0/obnamlib/plugins/encryption_plugin.py - obnam-1.0/obnamlib/plugins/fsck_plugin.py - obnam-1.0/obnamlib/plugins/compression_plugin.py - obnam-1.0/obnamlib/plugins/backup_plugin.py - obnam-1.0/obnamlib/plugins/convert5to6_plugin.py - -0 failures, 0 errors -242 excluded statements -22 excluded modules -22 missing test modules - -Slowest tests: - 0.2 s test_latest_returns_only_generation (repo_tests.RepositoryGenspecTests - 0.3 s test_counts_files_in_first_generation (clientmetadatatree_tests.Client - 0.3 s test_latest_returns_newest_generation (repo_tests.RepositoryGenspecTes - 0.4 s test_removing_client_that_has_data_removes_the_data_as_well (repo_test - 0.4 s test_latest_returns_newest_generation (repo_tests.RepositoryGenspecTes - 0.4 s test_find_chunks_finds_what_put_chunk_puts (repo_tests.RepositoryChunk - 0.4 s test_removing_generation_works (repo_tests.RepositoryClientTests) - 0.6 s test_removing_generation_works (repo_tests.RepositoryClientTests) - 0.6 s test_removing_only_second_generation_works (repo_tests.RepositoryClien - 0.9 s test_removing_only_second_generation_works (repo_tests.RepositoryClien -Time: 15.1 s -ERROR: Command '['python', '-m', 'CoverageTestRunner', '--ignore-missing-from=without-tests']' returned non-zero exit status 1 -make: *** [override_dh_auto_test] Error 1 -</pre> - ---- - -Ah, I must have misunderstood the Launchpad page. Let's see what Chris can find out. --liw - ---- - -I don't know what or if anything was fixed, but 1.0 is in the PPA now and working. Thanks. diff --git a/bugs/encrypt-fails-gpg-batchmode.mdwn b/bugs/encrypt-fails-gpg-batchmode.mdwn deleted file mode 100644 index 838a15f..0000000 --- a/bugs/encrypt-fails-gpg-batchmode.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649769>. --liw - -The gpg agent problem is sorted, but there's a mysterious crash. -Does not seem important enough at this point to be a 1.0 blocker. --liw - -[[done]] diff --git a/bugs/encryption-improvement-suggestion.mdwn b/bugs/encryption-improvement-suggestion.mdwn deleted file mode 100644 index 1d8dc9f..0000000 --- a/bugs/encryption-improvement-suggestion.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -[[!tag obnam-wishlist]] - - <zeri> liw: I tried out obnam yesterday and noticed you guys - simply call gpg -c --batch for the symmetric encryption part - <zeri> liw: also if I am not misstaken you sort of have a master - key for the repository (64 bit hex number encrypted for all keys - in the repository that is used as passphrase) - <zeri> liw: since you do not sepcify the - s2k-algo,s2k-mode,s2k-count configuration options of gpg as - well as the compression-algo option a gpg.conf that is usually - considered good will slow down obnam to a couple of bytes/sec - <zeri> liw: since keyderivation is of no use if a sufficiently - big random secret is used you might want to consider specifying - --s2k-mode 1 to disable most of the keystrengthening in gpg and - simply hash the password once with a salt ... speeding up the - encryption of every block at least by one order of magnitude - <zeri> (default behaviour is the compute a hash chain of at least - 1024 length up to 65011712 which was in my gpg.conf) - <zeri> also I didn't check whether you do compression in obnam - but gpg can do that for you as well but it was turned off in my - gpg.conf (I used gpg primarily for large tar balls where the one - time overhead doesn't matter) - <zeri> liw: then again I might be totally wrong and overlooked - some switch to do all that without chaning the code :) - <zeri> oh and I didn't emphasise this yet ... this hash chain - i talked about before is computed for every "chunk" which were - between a couple of bytes and 16k in my test ... the effort to - compute this hashchain (to optain encryption/authentication keys) - exceeds the effort to ecrypt 16k with any blockcipher by far - I suppose - <zeri> and it's sole purpose is to prevent weak passwords for - being guesed in short time (since the computational effort to - test a password is equivalent to computing this hash chain thus - slowing bruteforce down by the factor of the length of the chain) - <zeri> "64 bit hex number encrypted" << that should have been 64 - digits :) - - -[[done]] I'm not going to override a user's gpg.conf. --liw diff --git a/bugs/errors-during-backup-should-cause-exit-1.mdwn b/bugs/errors-during-backup-should-cause-exit-1.mdwn deleted file mode 100644 index 26a6b0f..0000000 --- a/bugs/errors-during-backup-should-cause-exit-1.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -If there's an error during a backup, even if Obnam continues to back up -another file, the exit code should still be 1. - -(Thanks, weasel.) - ---liw - -[[done]] in current bzr. diff --git a/bugs/estimate-backup-time.mdwn b/bugs/estimate-backup-time.mdwn deleted file mode 100644 index d345f41..0000000 --- a/bugs/estimate-backup-time.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -It would be cool if Obnam could do a very quick estimate of the amount -of new data to be backed up, without having to perform a complete new -backup. --liw - -This is [[done]] now. --liw diff --git a/bugs/estimate-to-be-freed-space.mdwn b/bugs/estimate-to-be-freed-space.mdwn deleted file mode 100644 index b17ef44..0000000 --- a/bugs/estimate-to-be-freed-space.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!tag obnam-wishlist]] - - -It would be nice for Obnam to have a tool to answer the question -"how much space will be freed if I remove these generations?" - - - need to find list of chunks that are used only by the specified gens - - perhaps also count B-tree reduction? cound nodes that are unshared by - the relevant trees, or only shared by the trees to be deleted - - however, the B-trees are going to be a fraction (a few percent) of - the size of the chunk data, so they're not really worth it - ---liw - - -[[done]] ancient wishlist bug, closing. --liw diff --git a/bugs/exclude-after-readlink.mdwn b/bugs/exclude-after-readlink.mdwn deleted file mode 100644 index db5ea4f..0000000 --- a/bugs/exclude-after-readlink.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -Excluding files based on `readlink $file` would be nice to have. As an example, anything that symlinks to somewhere under $HOME/code/dotfiles is handled by my dotfiles system. --mathstuf - -I could see making the backup plugin have hooks to extend exclusions -so that an other plugin could provide this functionality. -However, it looks fairly specific, so I don't think I'll write a plugin myself. -Anyone who does, please send patches to the mailing list, I'll be happy -to consider them. -[[done]] - -liw diff --git a/bugs/exclude-based-on-mime-type.mdwn b/bugs/exclude-based-on-mime-type.mdwn deleted file mode 100644 index 1de5e1d..0000000 --- a/bugs/exclude-based-on-mime-type.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -Exclude based on mime type of file. - -[[done]] -- not written, but keeping a wishlist bug open clearly -isn't making it happen. --liw diff --git a/bugs/exclude-based-on-size.mdwn b/bugs/exclude-based-on-size.mdwn deleted file mode 100644 index b942350..0000000 --- a/bugs/exclude-based-on-size.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -Write Obnam plugin to exclude files based on size. - - -[[done]] -- not written, but keeping wishlist bug open isn't making -it happening. I'll write this if I really need it. --liw diff --git a/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn b/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn deleted file mode 100644 index 91a8ab4..0000000 --- a/bugs/exclude_already_existing_files_in_backup__44___does_not_remove_them.mdwn +++ /dev/null @@ -1,108 +0,0 @@ -Obnam version 1.0 - -Host: Gentoo amd64 - -When excluding a file that already exists on the backup, the file is not removed from the last backup, but it is kept, without being updated. - -The file excluded should not appear on generations created with the exclude filter. - -How to reproduce: - - ############################################# - #!/bin/sh - - rm -rf /tmp/test - mkdir -p /tmp/test/backup - touch /tmp/test/backup/file1 - touch /tmp/test/backup/file2 - - # 1st run, exclude file1 - # only file2 is present on backup - obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup - obnam -r /tmp/test/repo ls - - # 2nd run - # none excluded, both files present on backup - obnam -r /tmp/test/repo backup /tmp/test/backup - obnam -r /tmp/test/repo ls - - # 3rd run, modify files and backup with exclude=file1 - # both files present, file2 is updated on backup, file1 is neither updated, or removed - echo "000" > /tmp/test/backup/file1 - echo "000" > /tmp/test/backup/file2 - obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup - obnam -r /tmp/test/repo ls - - cd / - rm -rf /tmp/test/backup - obnam -r /tmp/test/repo restore /tmp/test/backup - ls -lha /tmp/test/backup - ############################################# - -OUTPUT: - - + rm -rf /tmp/test - + mkdir -p /tmp/test/backup - + touch /tmp/test/backup/file1 - + touch /tmp/test/backup/file2 - - # 1st run, exclude file1 - # only file2 is present on backup - + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup - Backed up 2 files, uploaded 0.0 B in 0s at 0.0 B/s average speed - + obnam -r /tmp/test/repo ls - Generation 2 (2012-06-20 09:23:41 - 2012-06-20 09:23:41) - drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / - drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp - drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test - drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup - -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2 - - # 2nd run - # none excluded, both files present on backup - + obnam -r /tmp/test/repo backup /tmp/test/backup - Backed up 3 files, uploaded 0.0 B in 0s at 0.0 B/s average speed - + obnam -r /tmp/test/repo ls - Generation 5 (2012-06-20 09:23:41 - 2012-06-20 09:23:41) - drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / - drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp - drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test - drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup - -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1 - -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2 - - # 3rd run, modify files and backup with exclude=file1 - # both files present, file2 is updated on backup, file1 is neither updated, or removed - + obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup - Backed up 2 files, uploaded 4.0 B in 0s at 13.8 B/s average speed - + obnam -r /tmp/test/repo ls - Generation 8 (2012-06-20 09:23:42 - 2012-06-20 09:23:42) - drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 / - drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp - drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test - drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup - -rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1 - -rw-rw---- 1 jordi jordi 4 2012-06-20 07:23:41 /tmp/test/backup/file2 - - # when restoring, they appear, the new file2, and the old file1. - + rm -rf /tmp/test/backup - + cd / - + obnam -r /tmp/test/repo restore /tmp/test/backup - --h--m--s 4 files 4 B (100 %) 1.4 KiB/s /tmp/test/backup - + ls -lha /tmp/test/backup - total 4,0K - drwxrwx--- 2 jordi jordi 80 jun 20 09:23 . - drwxrwx--- 4 jordi jordi 80 jun 20 09:23 .. - -rw-rw---- 1 jordi jordi 0 jun 20 09:23 file1 - -rw-rw---- 1 jordi jordi 4 jun 20 09:23 file2 - - ---- - -A workaround can be found in -<http://listmaster.pepperfish.net/pipermail/obnam-flarn.net/2013-May/001929.html>. - - ---- - -This is now fixed in git master. [[done]] --liw diff --git a/bugs/fd-leak.mdwn b/bugs/fd-leak.mdwn deleted file mode 100644 index 8808b18..0000000 --- a/bugs/fd-leak.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -Ben Kelly reported on August 31, 2012, that he's seeing crashes -due to file descriptor leaks. See list mail archive for logs and -suggested patches. I have not been able to reproduce this, however. ---liw - -There have been no new reports of this for years. [[done]] --liw diff --git a/bugs/file-id-and-client-id-collision-corruption.mdwn b/bugs/file-id-and-client-id-collision-corruption.mdwn deleted file mode 100644 index b074bf9..0000000 --- a/bugs/file-id-and-client-id-collision-corruption.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Martin Hostettler reported that Obnam doens't properly choose file-ids and -client ids: there can be collisions, resulting in badness. See the e-mail at: - -<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-May/003018.html> - ---- - -This may still happen, but it's not severe enough that I want to fix it. -Format 6 is in maintenance mode where only serious bugs get attention, and the -rest of my time goes into producing a bette format (which doesn't have this -bug at all). [[done]] --liw diff --git a/bugs/filename-hash-collisions.mdwn b/bugs/filename-hash-collisions.mdwn deleted file mode 100644 index e114f20..0000000 --- a/bugs/filename-hash-collisions.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -Obnam does not handle hash collisions in filenames. If two different -filenames have the same hash, Obnam assumes they're the same filename. - -This should be possible to test by using a very weak filename hash -function. - ---liw - -[[done]] in bzr. diff --git a/bugs/force-lock_currently_doesn__39__t_work.mdwn b/bugs/force-lock_currently_doesn__39__t_work.mdwn deleted file mode 100644 index ba30f3f..0000000 --- a/bugs/force-lock_currently_doesn__39__t_work.mdwn +++ /dev/null @@ -1,52 +0,0 @@ -[[!tag obnam-wishlist]] - -obnam force-lock currently doesn't work. As a workaround, remove the lockfiles (all files named lock inside the repository) by hand. - - find [repository path] -name lock -exec rm '{}' \; - ---weinzwang - ---- - -I confirm that I see this too. This bug exists because I changed how Obnam uses locks: it now locks -each directory properly, instead of just the per-client directory. However, I haven't fixed "force-lock" to -deal with other locks, so now it's not possible to force the locks for other directories than the per-client -one. This is awkward. - -To fix this, Obnam needs to know that it can safely remove the locks. There's two cases: - -* the lock was created by some other client; in this case, the user (not Obnam automatically) - needs to decide if it is safe to remove the lock: just running "obnam force-lock" should not - do that, instead the user should provide an option like "--really-force-locks" or something -* the lock was created by the same client, i.e., Obnam running on the same host; in this case, - if the Obnam process no longer exists, the lock can be safely removed, otherwise the locks - should not be removed (again, unless "--really-force-locks" is used) - -To implement this, we need Obnam to store the hostname and process id of the Obnam -instance that created the lock, preferably in a way that does not leak sensitive information -easily (don't store the client name in cleartext, but the md5sum of it, or something). - ---liw - ---- - -As of 0.27, force-locks unconditionally breaks locks, but the lock files will -contain sufficient information to allow us to be more intelligent about the -breaking of locks in the future. - ---kinnison - --- - -This is not good enough -- I'd like obnam to be able to break locks more kindly -- -but it's good enough for 1.0, I think, so removing the blocker tag. --liw - ---- - -Making the lock breaking more benign and intelligent is a -wishlist. Adding tag. --liw - ---- - -Turns out that this will either mean unencrypted locks or not putting -any useful info in lock files. [[done]] --liw diff --git a/bugs/forget-is-too-slow.mdwn b/bugs/forget-is-too-slow.mdwn deleted file mode 100644 index 2b37342..0000000 --- a/bugs/forget-is-too-slow.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -`obnam forget` is currently too slow to be usable. It might be much -faster if it removed all data for all generations to be removed at -once, rather than generation-by-generation. Needs experimentation and -benchmarking. - ---liw - -[[done]] diff --git a/bugs/forget-progress-reporting.mdwn b/bugs/forget-progress-reporting.mdwn deleted file mode 100644 index b063f88..0000000 --- a/bugs/forget-progress-reporting.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be nice if "obnam forget" could have more detailed progress -reporting than at the granularity of one generation. --liw - -[[done]] this is semi-duplicate, and in any case a wishlist bug that's -been open for a very long time. --liw diff --git a/bugs/forget_progress_reporting_broken.mdwn b/bugs/forget_progress_reporting_broken.mdwn deleted file mode 100644 index f3ecc26..0000000 --- a/bugs/forget_progress_reporting_broken.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -[[!tag obnam-wishlist]] - -The progress reporting for "obnam forget" seems to be broken. When -forgetting more than two generations, the progress display is stuck at - - forgetting generations: 2/64 done - -until the very end. It's updated to 64/64 right before finishing. - --- weinzwang - - -This should be fixed in git now. [[done]] --liw diff --git a/bugs/fsck-does-not-lock.mdwn b/bugs/fsck-does-not-lock.mdwn deleted file mode 100644 index 7186f2b..0000000 --- a/bugs/fsck-does-not-lock.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -`obnam fsck` does not lock anything in the repository, but it should. ---liw - -[[done]] in bzr diff --git a/bugs/fsck-rechecks-same-chunks.mdwn b/bugs/fsck-rechecks-same-chunks.mdwn deleted file mode 100644 index 208a2bb..0000000 --- a/bugs/fsck-rechecks-same-chunks.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Alexander Sitnik reported that he'd noticed that "obnam fsck" checks -all chunks in all generations, resulting in the same chunks being checked -over and over. It should speed up "obnam fsck" if each chunk was checked -only once. I note that this may need to be done with some care so that -memory use doesn't increase too much. --liw diff --git a/bugs/fsck-should-fix-refcount-errors.mdwn b/bugs/fsck-should-fix-refcount-errors.mdwn deleted file mode 100644 index 338a324..0000000 --- a/bugs/fsck-should-fix-refcount-errors.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -obnam fsck needs to be able to fix refcount errors; larch needs to be able -to fix this - --- - -[[done]] --liw diff --git a/bugs/fsck-should-not-require-user-to-be-known-client.mdwn b/bugs/fsck-should-not-require-user-to-be-known-client.mdwn deleted file mode 100644 index 22187c6..0000000 --- a/bugs/fsck-should-not-require-user-to-be-known-client.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -obnam fsck should perhaps not require user to be a known client - ---- - - liw@havelock$ ./obnam -r t.repo --client=nope fsck - Checking 57/57: extra chunks - liw@havelock$ ./obnam -r t.repo clients - havelock - liw@havelock$ - -[[done]] --liw diff --git a/bugs/fsck-should-reconstruct-chunk-btrees.mdwn b/bugs/fsck-should-reconstruct-chunk-btrees.mdwn deleted file mode 100644 index 77e68d2..0000000 --- a/bugs/fsck-should-reconstruct-chunk-btrees.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -fsck has all the necessary information it needs to reconstruct the -chunksums and chunklist B-trees in a repository. It should do so, -at least when requested. - ---liw - -[[done]] ancient wishlist bug --liw diff --git a/bugs/fsck-should-remove-extra-files.mdwn b/bugs/fsck-should-remove-extra-files.mdwn deleted file mode 100644 index 1c4e9de..0000000 --- a/bugs/fsck-should-remove-extra-files.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!tag obnam-wishlist]] - -Make obnam fsck remove extraneous files (e.g., tmp*). ---liw diff --git a/bugs/fsck-workitem-unification.mdwn b/bugs/fsck-workitem-unification.mdwn deleted file mode 100644 index 259aeda..0000000 --- a/bugs/fsck-workitem-unification.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Combine larch and obnam fsck workitems. Merge the two current WorkItem -classes, put them into larch, and adapt both code bases accordingly. ---liw - -[[done]] diff --git a/bugs/fsck_gives_warnings_for_simple_cases.mdwn b/bugs/fsck_gives_warnings_for_simple_cases.mdwn deleted file mode 100644 index 0b0889b..0000000 --- a/bugs/fsck_gives_warnings_for_simple_cases.mdwn +++ /dev/null @@ -1,21 +0,0 @@ -The fsck command gives warnings in some cases. I have one case, where it is easy to reproduce: - - host:/dev/shm% mkdir obnam 0 - host:/dev/shm% obnam -r /dev/shm/obnam backup /dev/shm/0 - Backed up 1 files, uploaded 0.0 B in 0s at 0.0 B/s average speed - host:/dev/shm% obnam -r /dev/shm/obnam fsck - warning: node 1: node must have keys - warning: node 1: node must have keys - Checking 15/14: extra chunks - - Version used: debian/stable/backports 1.0-1~bpo60+1 - -I guess this is a minor issue, but still would be nice, if fsck would be mostly calm. - ---- - -I confirm that I can see the same error. --liw - --- - -[[done]] --liw diff --git a/bugs/fsck_needs_too_much_memory.mdwn b/bugs/fsck_needs_too_much_memory.mdwn deleted file mode 100644 index cf2e147..0000000 --- a/bugs/fsck_needs_too_much_memory.mdwn +++ /dev/null @@ -1,18 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -obnam fsck's memory needs are so big I'm not able to check my repo: - - [6402316.939346] Out of memory: Kill process 18036 (obnam) score 906 or sacrifice child - [6402316.939685] Killed process 18036 (obnam) total-vm:2754908kB, anon-rss:1876312kB, file-rss:640kB - ---weinzwang - -I've pushed some optimizations in bzr, and they'll end up in the next release. -Could you check, before I release, whether that was enough? I don't have access -to a machine for a while which has enough disk space to test this properly. ---liw - -I'm happy to report that with obnam from bzr, my repo can now be checked without -problems. Props to you liw! --weinzwang - -Marking bug as [[done]]. Thanks! --liw diff --git a/bugs/fsck_progress_counts_wrong.mdwn b/bugs/fsck_progress_counts_wrong.mdwn deleted file mode 100644 index 092b08b..0000000 --- a/bugs/fsck_progress_counts_wrong.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -Reported-By: Neal Becker - - obnam fsck - Checking 5/4: client list - ---liw - ---- - -[[done]] --liw diff --git a/bugs/fstype-include-exclude-mdwn b/bugs/fstype-include-exclude-mdwn deleted file mode 100644 index 837f36e..0000000 --- a/bugs/fstype-include-exclude-mdwn +++ /dev/null @@ -1,13 +0,0 @@ -Implement --exclude-fstype and --include-fstype for Obnam. 2011-08 -whenever obnam backup is crossing a filesystem boundary (mount point), -it looks up the type of the filesystem beyond the boundary, and if it -matches an excluded filesystem type, but does not match an included -one, then it won't recurse beyond the mount point (it will back up -the actual directory, using metadata of the root of the mounted -filesystem) - -- in backup_plugin.py:can_be_backed_up, add tests after the test for - one-file-system - ---liw - diff --git a/bugs/fuse-hardlinks.mdwn b/bugs/fuse-hardlinks.mdwn deleted file mode 100644 index 8d83aed..0000000 --- a/bugs/fuse-hardlinks.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!meta title="FUSE doesn't expose hardlinks correctly"]] - -Backing up two hardlinks to the same file, then mounting the backup -repository with the FUSE plugin, results in the hardlinks being shown -as having different inode numbers. They do thus not expose the -hardlinks correctly. - -Reported by Ilya Zonov -(<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2015-April/003516.html>, -the list archive shows the message wrong, but can be decoded with -base64). diff --git a/bugs/fuse-without-generations.mdwn b/bugs/fuse-without-generations.mdwn deleted file mode 100644 index 0f95a11..0000000 --- a/bugs/fuse-without-generations.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -It seems `obnam mount` (the FUSE plugin) can't handle a client without -non-checkpoint generations. This is unfortunate, even if it is fairly -unlikely to happen. Should be easy enough to fix. ---liw diff --git a/bugs/fuse.mdwn b/bugs/fuse.mdwn deleted file mode 100644 index 62a0ee9..0000000 --- a/bugs/fuse.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be good to have a FUSE filesystem for restoring data. --liw - ---- - -I didn't get very far today, but here's a draft: <http://p.sipsolutions.net/d9720a80c9cc5e99.txt>, there's still a lot of XXX in there but maybe it helps somebody get started. I probably won't have much time to work on it. - -You start it with "obnam fuse /mountpoint", or you can do "obnam fuse -- -h" to get fuse help. Need the "--" so obnam doesn't parse the "-h" itself ... - ---Johannes - ----- - -There is a FUSE plugin in Obnam now. --liw -[[done]] diff --git a/bugs/generation-descriptions.mdwn b/bugs/generation-descriptions.mdwn deleted file mode 100644 index 09e8cc7..0000000 --- a/bugs/generation-descriptions.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -[[!tag obnam-wishlist]] - -S.B. suggests that backup generations have an optional description. - -> 2. Named generations -- There are certain generations that are more -> important than others. Some are automatically created by Obnam itself, -> some are routinely scheduled, and some were explicitly created. For -> example, I always run Obam immediately before traveling with my laptop -> in case it gets stolen or broken. The same goes for backups before -> major system upgrades. It would be nice to have something -> approximately analogous to the Windows "restore point" functionality, -> which has a description field. Sometimes they are only automatically -> created system checkpoints. But if the user explicitly creates a new -> restore point, he can add the description "before traveling to Europe" -> or "before upgrading OS" or whatever. Similarly, the automatic backup -> script could be programmed to label it as "cron backup". - diff --git a/bugs/generation-locking.mdwn b/bugs/generation-locking.mdwn deleted file mode 100644 index 50df37c..0000000 --- a/bugs/generation-locking.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -[[!tag obnam-wishlist]] - -S.B. suggests that generations could be tagged so they aren't automatically -deleted. - -> 3. Unforgettable generations -- In scenarios similar to the above, I -> would also find it useful to be able to mark certain important -> generations as "unforgettable". That way, when I run an automatic time -> based forget command, I can be sure that it will preserve certain -> milestone generations, even if they weren't the last generation of the -> month or the week or the day or whatever. - diff --git a/bugs/generation-time-stamps-wrong.mdwn b/bugs/generation-time-stamps-wrong.mdwn deleted file mode 100644 index ab9199d..0000000 --- a/bugs/generation-time-stamps-wrong.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -From Anders Wirzenius: - - Generation 8 (1970-01-01 02:00:00 - 1970-01-01 02:00:00) <------ - ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^ - -The timestamps are clearly wrong. It's as if Obnam is storing a zero -for the timestamps (zero UTC would be 02:00 Finnish time). --liw - -[[done]] diff --git a/bugs/glob-exclusion.mdwn b/bugs/glob-exclusion.mdwn deleted file mode 100644 index a3e96a2..0000000 --- a/bugs/glob-exclusion.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -[[!tag obnam-wishlist]] - -From Enrico: A way to exclude files with globs instead of regular -expressions would be nice sometimes. `--exclude-glob="*.o"` or -something like that. - ---- - -Having thought of this: I'd rather not do this myself. Globs are not -powerful enough for my own use, and I'd rather not make two kinds -of patterns. I'd be happy to accept a patch for this, but keeping the -wishlist bug open isn't going to make that happen. [[done]] --liw diff --git a/bugs/gpg-options.mdwn b/bugs/gpg-options.mdwn deleted file mode 100644 index 91cba84..0000000 --- a/bugs/gpg-options.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Obnam should have a `gnupg-home` setting to set the GnuPG home directory -(`~/.gnupg`), or possibly a `gnupg-option` string list setting to pass -any options to GnuPG. ---liw - -The home directory can be set with `$GNUPGHOME` environment variable. -Additional settings can be set in the config in that directory. An -Obnam setting seems unnecessary, therefore. I'll add it if I am told -of a specific example of why it's useful. --liw - -[[done]] diff --git a/bugs/gpg-passphrase.mdwn b/bugs/gpg-passphrase.mdwn deleted file mode 100644 index 308fa82..0000000 --- a/bugs/gpg-passphrase.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -[[!tag obnam-wishlist]] -Obnam should, optionally, ask for a gpg passphrase, for the key specified -with --encrypt-with, so that a user without a gpg agent will be able to -do encrypted backups. Obnam should read the passphrase if its -ask-passphrase setting is true, and it has access to a terminal. -It should not have a setting for the passphrase itself, just for -reading it from a terminal (just so that people who don't know -better don't put their passphrase in a config file or similar). - -Those running obnam from cron will need to have a passphraseless -key, since there's no way to give obnam a passphrase in that case, -without storing it in the crontab or a config file, and then it's -no better than not having a passphrase. - -See [Debian -bug #649769](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=649769). - ---liw - -From my understanding, having a symmetric passphrase stored in a config file is not useless at all. My purpose in encrypting the backup data is to prevent the remote server from having my data in plain-view; or if I back it up to an external drive, I wouldn't want it to be accessible to anyone who picks it up. But if someone gains access to my config file, he'll have direct access to all of my data anyway--he wouldn't need to access my backups. - -If I use a passphrase, then if my house burns down and I lose everything, I can get a new computer and download my data and decrypt it with my passphrase--which is long enough to be unfeasible to crack, yet completely memorized by me. - -If I use a key, then if my house burns down and I don't have a working copy of my key outside my house, my backups are totally useless, and I really HAVE lost everything. (Sure, I should take precautions to keep from losing my key--but things happen.) - ---Adam - -It's possible to get obnam to request a passphrase when running from cron: - -1. Ensure 'use-agent' is enabled in ~/.gnupg/gpg.conf. -2. Ensure the gpg-agent is running, and GPG_AGENT_INFO is set in your regular environment. Note that if obnam already asks for an enccryption passphrase when run normally, then 1 & 2 are already correctly set. -3. Ensure the environment obnam is called from in cron is exporting GPG_AGENT_INFO correctly. This means you must set and export the GPG_AGENT_INFO environment variable in your cron script. gpg writes this information to ~/.gnupg/gpg-agent-info-$(hostname), so in your cron script you must have: - - source "~/.gnupg/gpg-agent-info-$(hostname)" && export GPG_AGENT_INFO - -Then call obnam as normal. - -This will only work on a desktop system where there is someone to notice that a pinentry window has popped up. However it looks like there may be a way to forward the gpg-agent socket over ssh, and thus run obnam with encryption from cron on a headless remote machine (<a href="http://superuser.com/questions/161973/how-can-i-forward-a-gpg-key-via-ssh-agent">See here</a>). You'd probably have to store the private key on the remote machine though.. so not sure how useful that would be. - ---Scott - - ---- - -I continue to be of the opinion that a setting for the passphrase for -the GPG is pointless. The symmetric key is encrypted by GPG public key -only. [[done]] --liw diff --git a/bugs/help-output-groups.mdwn b/bugs/help-output-groups.mdwn deleted file mode 100644 index b5f4e2c..0000000 --- a/bugs/help-output-groups.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be nice of `obnam --help` would group the large list of options -in some sensible way. --liw - -[[done]] diff --git a/bugs/helper-script-for-bug-report-data-gathering.mdwn b/bugs/helper-script-for-bug-report-data-gathering.mdwn deleted file mode 100644 index 6f3dfa0..0000000 --- a/bugs/helper-script-for-bug-report-data-gathering.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!tag obnam-wishlist]] - -Would be nice to have a -script to gather information for reporting a bug about obnam? - -- Python version -- versions of obnam, larch, cliapp, and any other of my projects - obnam uses -- config dump -- is _obnam available? -- mount point, file system types (/proc/mounts on linux) -- linux kernel version, or other uname -a output - -[[done]] -- no script exists, but keeping the wishlist bug open isn't -helping it. I'll write one when the pain becomes too much. --liw diff --git a/bugs/inefficient-metadata.mdwn b/bugs/inefficient-metadata.mdwn deleted file mode 100644 index 9b7d0ae..0000000 --- a/bugs/inefficient-metadata.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Obnam seems to be storing metadata quite inefficiently. I did this: - -* create a directory tree with about 1.3 million empty files -* back that up - -The data is about 500 megs (directory entries); the repository is -about 82 gigabytes. - -Where is the space used? Can we store it more efficiently and if we -do, does that have an impact on runtime? - ---liw - -Workaround: use compression. -Proper fixes are going to require FORMAT GREEN ALBATROSS. [[done]] --liw diff --git a/bugs/insufficient-locking.mdwn b/bugs/insufficient-locking.mdwn deleted file mode 100644 index 4726d1f..0000000 --- a/bugs/insufficient-locking.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Obnam does not currently do locking properly. It locks the per-client -directory, but not the shared directories. --liw - - -[[done]] diff --git a/bugs/integer-too-big.mdwn b/bugs/integer-too-big.mdwn deleted file mode 100644 index 9162ad5..0000000 --- a/bugs/integer-too-big.mdwn +++ /dev/null @@ -1,36 +0,0 @@ -Fix "long too large to convert to int" problem found by Jo. - - Traceback (most recent call last): - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 135, in _run - self.process_args(args) - File "/usr/lib/python2.6/dist-packages/obnamlib/app.py", line 116, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.6/dist-packages/cliapp/app.py", line 275, in process_args - method(args[1:]) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 80, in backup - self.backup_roots(roots) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 136, in backup_roots - self.backup_metadata(pathname, metadata) - File "/usr/lib/python2.6/dist-packages/obnamlib/plugins/backup_plugin.py", line 250, in backup_metadata - self.repo.create(pathname, metadata) - File "/usr/lib/python2.6/dist-packages/obnamlib/repo.py", line 509, in create - encoded = obnamlib.encode_metadata(metadata) - File "/usr/lib/python2.6/dist-packages/obnamlib/metadata.py", line 200, in encode_metadata - len(metadata.md5 or '')) - error: long too large to convert to int - -- hypothesis: overflows int on the way to the struct, on 32-bit -- wrote script to test hypothesis, but proved it wrong: the limit where - struct gives error is 2**64-1, as is proper, so the problem is something - else -- talk to directhex to extract more information, possibly add logging to - pinpoint the value - ---liw - - -Can't seem to reproduce this, but it was hopefully related to things -now replaced by our own lstat wrapper. If you see this, re-open and -add a way to reproduce. --liw - -[[done]] diff --git a/bugs/keep-bugs-in-ikiwiki.mdwn b/bugs/keep-bugs-in-ikiwiki.mdwn deleted file mode 100644 index 5ed976f..0000000 --- a/bugs/keep-bugs-in-ikiwiki.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -I want to move my bug tracking from SD to ikiwiki. --liw - -[[done]] - diff --git a/bugs/keep-chunk-dirs-locked-less.mdwn b/bugs/keep-chunk-dirs-locked-less.mdwn deleted file mode 100644 index c0e00fd..0000000 --- a/bugs/keep-chunk-dirs-locked-less.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -For better performance, it would be good if Obnam could keep the shared -chunks B-trees locked for shorter periods of time. This can be implemented -by only updating them at the end of the backup generation (including -checkpoints), rather than updating along the way when making a backup. ---liw - -[[done]] diff --git a/bugs/keep-hours-buggy.mdwn b/bugs/keep-hours-buggy.mdwn deleted file mode 100644 index f886c4a..0000000 --- a/bugs/keep-hours-buggy.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -From mail: - -> I think there is problem in keep policy, -> see attached file, -> --keep=Nh always do same as --keep=Nd - -Seems to be true. Need cmdtest test case for reproduction. - ---- - -Actually, this seems to be mainly a documentation problem. Or an -indication that the system is too complicated, since so many people -understand it wrong. But [[done]], though I'll try to improve the -explanation in the manual. --liw diff --git a/bugs/keep-per-client-lock-across-snapshots.mdwn b/bugs/keep-per-client-lock-across-snapshots.mdwn deleted file mode 100644 index 02ffa1c..0000000 --- a/bugs/keep-per-client-lock-across-snapshots.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -Obnam should not release the per-client lock after doing a snapshot. -It should release the shared toplevel locks, though. --liw - --- - -I have forgotten why this would be useful. If I remember it again, I'll -re-open. For now, [[done]]. --liw diff --git a/bugs/keeps-dev-urandom-open.mdwn b/bugs/keeps-dev-urandom-open.mdwn deleted file mode 100644 index 1a2b43f..0000000 --- a/bugs/keeps-dev-urandom-open.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -why does obnam keep /dev/urandom open, even though nothing should be -using it? it's open whether using encryption or not - --- - -This seems to be due to the stdlib. Ignoring. [[done]] --liw diff --git a/bugs/key-managements.mdwn b/bugs/key-managements.mdwn deleted file mode 100644 index 7dd53e1..0000000 --- a/bugs/key-managements.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -There needs to be tools and documentation for key managment -with Obnam. - -How does not one replace one's key, or subkey, when it expires? diff --git a/bugs/lacks-compression-plugin.mdwn b/bugs/lacks-compression-plugin.mdwn deleted file mode 100644 index 754b031..0000000 --- a/bugs/lacks-compression-plugin.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Obnam could do with a plugin that just compresses data, without -encrypting it. Encryption might be unwanted, or unnecessary, for -example when backing to a USB drive that uses whole-disk encryption. -However, saving space by compressing data with gzip or xz is still -a reasonable thing to do. - -The compression plugin would be similar to the existing encryption -plugin, but be triggered by a different setting. - -* `compress-with`: program to compress files with; - default to gzip, but can be any program - ---liw - -[[done]] diff --git a/bugs/larch-journal-processing-progress-reporting.mdwn b/bugs/larch-journal-processing-progress-reporting.mdwn deleted file mode 100644 index dcaea07..0000000 --- a/bugs/larch-journal-processing-progress-reporting.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -When larch is processing a journal (committing or deleting at -startup, committing at end), obnam should be showing useful progress -reporting for that. - ---- - -Larch and format 6 in Obnam are in maint mode and won't get this kind -of change anymore, I'm afraid. [[done]] --liw diff --git a/bugs/live-data-backup-restore-benchmark-failure.mdwn b/bugs/live-data-backup-restore-benchmark-failure.mdwn deleted file mode 100644 index 7d12007..0000000 --- a/bugs/live-data-backup-restore-benchmark-failure.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -Tried to run obnam benchmark with real data and verification on xander. - -- failed: no restored files -- see /home/liw/tmp/benchmark-with-real-data-failed-to-restore -- investigate why - ---liw - -I no longer have access to this machine, and cannot debug the issue -anymore. Meh. --liw [[done]] diff --git a/bugs/llistxattr-fail-fatal.mdwn b/bugs/llistxattr-fail-fatal.mdwn deleted file mode 100644 index 8f7973e..0000000 --- a/bugs/llistxattr-fail-fatal.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -When obnam backs up from .gvfs or another filesystem that does not -support the llistxattr system call it crashes with this traceback: - - > File "/usr/lib/python2.7/dist-packages/obnamlib/metadata.py", line 130, in get_xattrs_as_blob - > names = fs.llistxattr(filename) - > File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 165, in llistxattr - > raise OSError((ret, os.strerror(ret), filename)) - > OSError: (95, 'Operation not supported', '/home/ben/.gvfs') - -Obnam should report the error, but continue the backup. - --- - -[[done]] --liw diff --git a/bugs/local-temp-cache.mdwn b/bugs/local-temp-cache.mdwn deleted file mode 100644 index d74349e..0000000 --- a/bugs/local-temp-cache.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655094> for details. - -[[done]] no point in keeping this open in multiple places. --liw diff --git a/bugs/lock-key-in-ram.mdwn b/bugs/lock-key-in-ram.mdwn deleted file mode 100644 index a65396a..0000000 --- a/bugs/lock-key-in-ram.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -[[!tag obnam-wishlist]] - -Joey asked: - -<joeyh> have you done anything in obnam to deal with it needing to keep -the symmetric key, decrypted, in RAM? - -<joeyh> yeah, it's tough. probably could be avoided by having gpg decrypt -the passphrase and pipe it to the encrypting gpg .. but then gpg would -constantly be using the public key - -It might be possible to have a C extension that holds the symmetric -key, locks it into RAM, and feeds it to gpg whenever necessary, -via a file descriptor. - ---liw - - -I'm going to be switching from running gpg for symmetric encryption -in the future, anyway. I'll be doing symmetric encryption in-process -using python-crypt, and that means a lot of the sensitive data is going -to be in Python strings anyway. Locking anything in memory doesn't seem -feasible. [[done]] --liw diff --git a/bugs/lock-notification.mdwn b/bugs/lock-notification.mdwn deleted file mode 100644 index 7257e33..0000000 --- a/bugs/lock-notification.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -If obnam is waiting for a lock to expire, it should say so, maybe with a hint to use force-lock to remove if obnam crashed/was killed (otherwise the user is left wondering what obnam is doing). - ---- - -This will be fixed with the next version of [[ttystatus]], which has a functioning `flush` method. --liw - --- - -[[done]] --liw diff --git a/bugs/lock-timeout-error-does-not-identify-lock.mdwn b/bugs/lock-timeout-error-does-not-identify-lock.mdwn deleted file mode 100644 index aecc43f..0000000 --- a/bugs/lock-timeout-error-does-not-identify-lock.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -When obnam gives an error for a lock timeout, it should specify what -lock file failed. --liw - -[[done]] in bzr diff --git a/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn b/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn deleted file mode 100644 index 5cd6409..0000000 --- a/bugs/log_files_that_have_been_backed_up_with_log_level_INFO.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -It would be nice to see, which files are being backed up when reducing the loglevel from DEBUG to INFO - ---- - -That seems reasonable. [[done]] --liw diff --git a/bugs/ls-output-formats.mdnw b/bugs/ls-output-formats.mdnw deleted file mode 100644 index 281d436..0000000 --- a/bugs/ls-output-formats.mdnw +++ /dev/null @@ -1,3 +0,0 @@ -[[!tag obnam-wishlist]] - -See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655229> for info. diff --git a/bugs/memory-profiling-everywhere.mdwn b/bugs/memory-profiling-everywhere.mdwn deleted file mode 100644 index 7b17e56..0000000 --- a/bugs/memory-profiling-everywhere.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -All Obnam operations should be calling the memory profiling operations, -so that all operations can be profiled when needed. The profiling functions -should go into cliapp, however. --liw -[[done]] diff --git a/bugs/missing-progress-reporting.mdwn b/bugs/missing-progress-reporting.mdwn deleted file mode 100644 index 5979b17..0000000 --- a/bugs/missing-progress-reporting.mdwn +++ /dev/null @@ -1,18 +0,0 @@ -The following operations should have progress reporting added: - - obnam [options] forget - obnam [options] fsck - obnam [options] restore - obnam [options] verify - ---liw - -restore and fsck already have progress reporting. -forget and verify need it. ---liw - -verify now has progress reporting. forget still needs it. --liw - -forget now has progress reporting too. --liw - -[[done]] diff --git a/bugs/missing-v6-restore-test.mdwn b/bugs/missing-v6-restore-test.mdwn deleted file mode 100644 index fefc0cb..0000000 --- a/bugs/missing-v6-restore-test.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -Add an Obnam cmdtest to verify v6 repos can be restored from. - -[[done]] in git master. --liw diff --git a/bugs/more-details-dedup.mdwn b/bugs/more-details-dedup.mdwn deleted file mode 100644 index c00e3b5..0000000 --- a/bugs/more-details-dedup.mdwn +++ /dev/null @@ -1,45 +0,0 @@ -The following suggestion was sent to me by private e-mail (not sure if -I can show the sender's name): - - Date: Mon, 14 Oct 2013 11:48:41 +1100 - Subject: Re: Obnam repo size with .sql dump files seem too big - - Lars, - - We implemented a strategy for identifying repeated chunks, even in - gzip-compressed files that have changes between versions. It might - work for obnam. The downside is it creates variable-sized chunks, - though you can set upper and lower limits. - - Firstly, we identify chunk boundaries by using a rolling checksum - like Adler32, rolling over say a 128 byte contiguous region. Any - other efficient FIR (box-car) checksum algorithm would work. When - the bottom N bits of the checksum are zero, declare a block - boundary. This gives blocks of average size 2**N. If you want more - certainty, you can enforce a lower limit of 2**(N-3), and upper - limit of 2**N or 2**(N+1), for example (thereby either creating, - or ignoring, the boundaries that are defined by the checksum. - - These variable-sized chunks will re-synchronise after differences - in two streams. To make it work with deflate, we flush the - compression context (the dictionary, we maintain the history, - losing 1-2% of compression) on each block boundary… but I'm not - sure that compression is necessary for obnam. - - By choosing N appropriately, you get the block size you want. We - used N=12, for internet delivery (using HTTP subrange requests) of - updated files. For each file, we publish a catalog of Adler32 and - SHA-1 block checksums. Clients download that using HTTP, then - analyse their files before making requests for blocks they lack. - - The invention of doing this in a deflate stream is due to Tim - Adam. When we discover we already have a given block, we can prime - the decompressor with that history before decompressing a received - update block. Really it should be implemented using 7zip instead - of deflate; a 32KB quotation history is too small. - ---- - -I haven't tried this yet. It will require a new repository format -version, so I've been working on adding that support instead. ---liw diff --git a/bugs/more-rsync-like.mdwn b/bugs/more-rsync-like.mdwn deleted file mode 100644 index fc2e9f1..0000000 --- a/bugs/more-rsync-like.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -Moving forward in small increments than individual chunks would allow -more rsync-like behavior, and more chance of finding duplicate data. -This might be worthwhile, for some users, some of the time. It should -be configurable, though, since it's also potentially going to be a -big performance problem. --liw - -[[done]] ancient wishlist bug --liw diff --git a/bugs/multiple-checksums.mdwn b/bugs/multiple-checksums.mdwn deleted file mode 100644 index 5585681..0000000 --- a/bugs/multiple-checksums.mdwn +++ /dev/null @@ -1,27 +0,0 @@ -[[!tag obnam-wishlist]] - -From Joey Hess: - -> My take on this is that, by choosing to use a tool that uses hashes, I -> am giving up (near-)absolute certainty for speed, or space, or whatever. -> So it's important that the hash type be good at collision resistance (for -> example, no two likely filenames should hash the same; "/etc/passwd" -> should only tend to collide with blobs that are very unlike a filename). -> It's also important that the tool be upfront about using hashes, and -> about what hash it uses. And if it's not designed to allow swapping the -> hash out when it gets broken, I will trust it less (hello git). - -Ah, the replacement of hash functions is an interesting problem. - -For pathnames, it's not at all important, I think, except perhaps for -performance, since pathnames will be compared byte-by-byte instead of -by hashes. - -For file data, replacing is easy, if one is willing to back up everything -from scratch. Supporting several hashes in the same backup store is a -little bit more work, but not a whole lot: instead of having just one -tree for mapping checksums to chunk identifiers, one would have one per -checksum algorithm. - ---liw - diff --git a/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn b/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn deleted file mode 100644 index 6201a06..0000000 --- a/bugs/multiple___34__exclude__34___in_configuration_fails.mdwn +++ /dev/null @@ -1,31 +0,0 @@ -The documentation for "exclude" says that it could be used multiple times: - ->--exclude=EXCLUDE -> regular expression for pathnames to exclude from backup (can be used multiple times) - -It works when used multiple times on the command line. However, when using a configuration with: - - [config] - exclude = /foo - exclude = /bar - exclude = /baz - -...only the last item "/baz" is actually excluded from the backup. - ---- - -The documentation doesn't actually say this anywhere (oops, sorry), so you couldn't know: the right way of doing that in the config file is to use one exclude line and separate values with commas (spaces optional). - - [config] - exclude = /foo, /bar, /baz - -That should work. - -Using multiple exclude-lines in the config would be better, but Python's ConfigParser library doesn't really deal well with that. - -I'll mark this bug closed when the obnam manpage explains how to do this with config files. - ---liw - -It's now in the manual page in bzr and fix will be included in the next -release. [[done]] --liw diff --git a/bugs/multirepository.mdwn b/bugs/multirepository.mdwn deleted file mode 100644 index 72695cf..0000000 --- a/bugs/multirepository.mdwn +++ /dev/null @@ -1,58 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should support multiple repositories, to be chosen at invocation time. - -- all repositories configured in config files -- nicknames for repositories so it's easy to choose -- --repository should accept nicknames -- choose many repositories for one run -- use all available repositories by default - ---liw - ---- -Could this also be achieved by running Obnam from a wrapper script that uses a different repository for each run? Could Obnam be run in parallel instances backing up the same data to different repos? Is that possible now? --*Adam* - ---- - -Adam, it can certainly be done by using wrapper scripts (I've been doing that), and while I haven't -actually tried it, there should be no problem with backing up to multiple repositories concurrently, -though you may need to fiddle with the configs so that they use different log files. --liw - ---- - -After some thinking, I think I don't want nicknames for repositories, I want "profiles".Here's a concrete -suggestion: - - [config] - encrypt-with = CAFEF00D - profile = all - log = /var/log/obnam/obnam-%(profile)s.log - - [profile "online"] - repository = sftp://liw@personal.backup.server.example.com/~/repo/ - use-if = ping -c1 personal.backup.server.example.com - - [profile "usb-drive"] - repository = /media/Drum/obnam-repo/ - use-if = test -d /media/Drum/obnam-repo/ - - [profile "at-work"] - repository = /mnt/backups/ - use-if = ping -c1 fs.work.example.com - pre-command = sudo mount /mnt/backups - post-command = sudo umount /mnt/backups - -- if --profile=all, then iterate automatically over all profiles -- otherwise, use only the chosen profiles -- some day: run some/all profiles in parallel in one obnam instance; initially, user may run parallel obnam instances -- log file should embed profile name somehow -- should profile be selected based on user too? that can be done with "use-if = test $USER = liw"; better support can be added later, if there's a need - ---liw - - ---- - -this is handled easily by having a set of config files and giving the -"profile" one to --config. [[done]] --liw diff --git a/bugs/needs-porting-to-rhel-sl-centos.mdwn b/bugs/needs-porting-to-rhel-sl-centos.mdwn deleted file mode 100644 index 3959236..0000000 --- a/bugs/needs-porting-to-rhel-sl-centos.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should work on RHEL, Scientific Linux, and CentOS, and other -popular distros such as Fedora, Ubuntu. Help porting these would -be welcome. --liw - - -There's a Fedora port, at least. Wishing isn't going make other ports happen, -though: users who want it on their platform, and are willing to do the work, is -what will make ports happen. Thus, keeping this bug open isn't useful. -[[done]] --liw diff --git a/bugs/no-can-two-files-as-roots.mdwn b/bugs/no-can-two-files-as-roots.mdwn deleted file mode 100644 index 456824c..0000000 --- a/bugs/no-can-two-files-as-roots.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!meta title="Can't backup two plain files as roots"]] - -Only the last file ends up in the repository. - -See -<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003103.html> -for details. diff --git a/bugs/no-chattr-support.mdwn b/bugs/no-chattr-support.mdwn deleted file mode 100644 index b0381ef..0000000 --- a/bugs/no-chattr-support.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam does not support the ext2/3/4 chattr attributes. It should back them up and -set them on restore, when possible. - -In addition, it should support the d attribute to exclude -files from being backed up. - ---liw diff --git a/bugs/non-linux-file-types.mdwn b/bugs/non-linux-file-types.mdwn deleted file mode 100644 index 55ff9fc..0000000 --- a/bugs/non-linux-file-types.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should support non-linux file types: - -* <http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/stat.h.html> -* <http://www.gnu.org/s/hello/manual/libc/Testing-File-Type.html> - ---liw - - -[[done]] There's no real way I can do this, but I'll be happy to accept good -patches. However, this wishlist has been open for years, without any action. --liw diff --git a/bugs/non-wishlist.mdwn b/bugs/non-wishlist.mdwn deleted file mode 100644 index 09c08ff..0000000 --- a/bugs/non-wishlist.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!inline pages="page(bugs/*) and - !bugs/*/* and - !bugs/done and - !bugs/wishlist-only and - !bugs/performance-only and - !tagged(obnam-wishlist) and - !tagged(obnam-performance) and - !link(bugs/done)" - show=0 - postform=no]] - diff --git a/bugs/obnam-benchmark-seivot-branch.mdwn b/bugs/obnam-benchmark-seivot-branch.mdwn deleted file mode 100644 index 65fa976..0000000 --- a/bugs/obnam-benchmark-seivot-branch.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -The `--seivot-branch` option should not be required for -`./run-benchmark`. --liw - - ---- - -[[done]] now. --liw diff --git a/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn b/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn deleted file mode 100644 index 57e9577..0000000 --- a/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -The obnam.org website should have instructions for how to clone it, -and how to send patches to it. --liw - -[[done]] --liw diff --git a/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn b/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn deleted file mode 100644 index 6d1d412..0000000 --- a/bugs/obnam_force-lock_does_not_remove_all_locks.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -force-lock in obnam 0.27-1.1 does not remove: - -./chunks/lock -or -./2275793072750507400/lock - -on a remote sftp:// repository. - ---- - -This is now fixed in bzr. --liw - -[[done]] diff --git a/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn b/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn deleted file mode 100644 index 458c5e9..0000000 --- a/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -Currently, obnam fsck reports chunks that are unused: - - chunk 16541095925909528379 not used by anyone - -but doesn't do anything about it. There should be an option to -remove those unused chunks from the repository. --weinzwang - -fsck can delete unused chunks now. [[done]] --liw diff --git a/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn b/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn deleted file mode 100644 index 2c13c25..0000000 --- a/bugs/obnam_should_handle_user_abort___40__ctrl-c__41___and_SIGTERM_gracefully.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -When I abort a backup using ctrl-c I get a stale lockfile on the sftp-Server - -Current (1.0) behavior: - - bart ~ # obnam backup --root=/etc - Enter passphrase for key '/home/mschiff/.ssh/id_dsa': - 00h00m03s 1286 files; 0 B (0 B/s) /etc/runit/runsvdir/all/getty-tty2 ^C Killed by signal 2. - (here I had hit ctrl-c) - bart ~ # obnam backup --root=/etc - Enter passphrase for key '/home/mschiff/.ssh/id_dsa': - ERROR: Lock timeout - bart ~ # - ---- - -Yeah. Obnam does not always clean up the lock files when it crashes. For some crashes it's not even possible. You can use the "obnam force-lock" command to remove the lock files, if you're sure they are not from another client using the same repository. --liw - -mschiff: You consider pressing ctrl-c to abort a backup a "crash" ? - -It's a user-induced crash, yes. It's one of the situations where Obnam could clean up, and it probably should. I believe there is a bug open about that already. --liw - - -[[done]] diff --git a/bugs/obnam_squeeze_package_dependencies.mdwn b/bugs/obnam_squeeze_package_dependencies.mdwn deleted file mode 100644 index d15e65a..0000000 --- a/bugs/obnam_squeeze_package_dependencies.mdwn +++ /dev/null @@ -1,57 +0,0 @@ -The obname package for Debian squeeze (version 0.27-1.1) has a dependency on python (>= 2.6.6-7~), but squeeze only provides python 2.6.6-3+squeeze6. - ---- - -I can't see this. From my code.liw.fi repository: - -> Depends: libc6 (>= 2.6), python2.7 | python2.6, python (>= 2.6.6-7~), python (<< 2.8), python-larch (>= 0.30~), python-ttystatus (>= 0.15), python-paramiko, python-tracing (>= 0.2), python-cliapp (>= 0.18) - -Can you point me at the .deb you found, and the URL you found it at? -I had some trouble building the 0.27 release, so I may have botched -it and put a bad version somewhere and if so, I'd like to fix that. Thanks. ---liw - ---- - -I'm referring to http://code.liw.fi/debian/pool/main/o/obnam/obnam_0.27-1.1_i386.deb, which is the package pulled in on squeeze. - -This is my /etc/apt/sources.list: - - deb http://ftp.debian.org/debian/ squeeze main - deb http://security.debian.org/ squeeze/updates main - deb http://ftp.debian.org/debian/ squeeze-updates main - deb http://code.liw.fi/debian squeeze main - -The real trouble is that there's no dedicated squeeze package for obnam 0.27. -There were dedicated squeeze packages for earlier versions of obnam (e.g. obnam_0.26-1~squeeze1_i386.deb). - -http://code.liw.fi/debian/dists/squeeze/main/binary-i386/Packages pulls in obname 0.27-1.1, which has a dependency on python (>= 2.6.6-7~), while Debian squeeze (6.0.4) only provides python 2.6.6-3+squeeze6: - - # apt-cache policy python obnam - python: - Installed: 2.6.6-3+squeeze6 - Candidate: 2.6.6-3+squeeze6 - Version table: - *** 2.6.6-3+squeeze6 0 - 990 http://ftp.de.debian.org/debian/ squeeze/main i386 Packages - 100 /var/lib/dpkg/status - obnam: - Installed: (none) - Candidate: 0.27-1.1 - Version table: - 0.27-1.1 0 - 500 http://code.liw.fi/debian/ squeeze/main i386 Packages - -Therefore, it's not possible to install obnam 0.27-1.1 on squeeze: - - The following packages have unmet dependencies: - obnam : Depends: python (>= 2.6.6-7~) but 2.6.6-3+squeeze6 is to be installed - E: Broken packages - -The fix would be to build a package for squeeze again, e.g. obnam_0.27-1.1~squeeze1_i386.deb. - ---flight - ---- - -Fixed with the 0.28 packages. [[done]] --flight diff --git a/bugs/obnammodule-malloc.mdwn b/bugs/obnammodule-malloc.mdwn deleted file mode 100644 index 731ccb2..0000000 --- a/bugs/obnammodule-malloc.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -Review `_obnammodule.c`'s use of malloc: every return value should be -checked and if there's a failure, the process should abort. - ---- - -Fixed in git master. [[done]] --liw diff --git a/bugs/option_to_not_use_paramiko__63__.mdwn b/bugs/option_to_not_use_paramiko__63__.mdwn deleted file mode 100644 index deabafe..0000000 --- a/bugs/option_to_not_use_paramiko__63__.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Could we please have an option to force obnam to not use paramiko but instead openssh or libssh2-1? I'd like to get around the whole .ssh/config debacle and other nasty surprises I had when using duplicity, and keep using known and trusted software. - ---- - -Switching away from paramiko will require a fair amount of work. For other reasons, I'm going to be looking at doing so, but it's not just a matter of "using openssh". There is, for example, no Python API for openssh. There's other Python APIs for dealing with ssh, which I'm going to investigate, but it's not a one-evening-hack kind of thing. - -Also, given that ~/.ssh/config is a configuration file for openssh, and not all programs that use the ssh protocol, I think it's unrealistic to require every program to obey it. - -If and when I switch Obnam away from paramiko, it will not be an optional thing. Having to support multiple ways of doing ssh is unnecessary complexity, so I will only support one implementation at a time. Sorry. As such, I'll mark this bug as [[done]]. - ---liw diff --git a/bugs/oserror-stino.mdwn b/bugs/oserror-stino.mdwn deleted file mode 100644 index bae2734..0000000 --- a/bugs/oserror-stino.mdwn +++ /dev/null @@ -1,37 +0,0 @@ -Robin reports: - - Traceback (most recent call last): - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 190, in _run - self.process_args(args) - File "/usr/lib/python2.7/dist-packages/obnamlib/app.py", line 173, in process_args - cliapp.Application.process_args(self, args) - File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 537, in process_args - method(args[1:]) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 286, in backup - self.backup_roots(roots) - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 396, in backup_roots - for pathname, metadata in self.find_files(absroot): - File "/usr/lib/python2.7/dist-packages/obnamlib/plugins/backup_plugin.py", line 477, in find_files - for pathname, st in self.fs.scan_tree(root, ok=self.can_be_backed_up): - File "/usr/lib/python2.7/dist-packages/obnamlib/vfs.py", line 234, in scan_tree - pairs = self.listdir2(dirname) - File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 359, in listdir2 - return sorted(result, key=lambda st: st[1].st_ino) - File "/usr/lib/python2.7/dist-packages/obnamlib/vfs_local.py", line 359, in <lambda> - return sorted(result, key=lambda st: st[1].st_ino) - AttributeError: 'exceptions.OSError' object has no attribute 'st_ino' - -Also: - - $ apt-cache policy obnam - obnam: - Geïnstalleerd: 1.3-1ubuntu1 - Kandidaat: 1.3-1ubuntu1 - Versietabel: - *** 1.3-1ubuntu1 0 - 500 http://ppa.launchpad.net/chris-bigballofwax/obnam-ppa/ubuntu/ quantal/main amd64 Packages - - ---- - -This is fixed in Obnam 1.4. [[done]] --liw diff --git a/bugs/performance-only.mdwn b/bugs/performance-only.mdwn deleted file mode 100644 index e6c4522..0000000 --- a/bugs/performance-only.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!inline pages="page(bugs/*) and - !bugs/*/* and - !bugs/done and - tagged(obnam-performance) and - !link(bugs/done)" - show=0 - postform=no]] - diff --git a/bugs/performance-with-many-hardlinks.mdwn b/bugs/performance-with-many-hardlinks.mdwn deleted file mode 100644 index d30639a..0000000 --- a/bugs/performance-with-many-hardlinks.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -[[!tag obnam-performance]] - -How does Obnam perform if the live data has massive numbers of hardlinks? -Such hardlink trees are not rare, but it would be good to at least know -how Obnam handles them. Perform a benchmark comparing three cases: - -1. a very large number (one million?) of small files, all unique -2. same, but all files identical -3. same, but all files hardlinks to the same content - -Ideally, Obnam should perform about the same for all. --liw - - -[[done]] really ancient wishlist bug --liw diff --git a/bugs/precise-checkpoints.mdwn b/bugs/precise-checkpoints.mdwn deleted file mode 100644 index be6623d..0000000 --- a/bugs/precise-checkpoints.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Obnam should do checkpoint generations at precise times, not just between -files; this will result in partial files, but that's ok, the next backup -run will continue from the partial file. --liw - -[[done]] diff --git a/bugs/pretend-does-not-work-on-empty-repo.mdwn b/bugs/pretend-does-not-work-on-empty-repo.mdwn deleted file mode 100644 index a7f58fa..0000000 --- a/bugs/pretend-does-not-work-on-empty-repo.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -`obnam backup --dry-run` doesn't work if there is no generation yet. ---liw - - -[[done]] --liw diff --git a/bugs/refactor-repository-class.mdwn b/bugs/refactor-repository-class.mdwn deleted file mode 100644 index 8363aa9..0000000 --- a/bugs/refactor-repository-class.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -The Repository class is quite big, and could perhaps be refactored to -become more manageable. ---liw - -I've merged changes to a replacement for the Repository class. -[[done]] ---liw diff --git a/bugs/regression_on_debian_squeeze_python_dependancy.mdwn b/bugs/regression_on_debian_squeeze_python_dependancy.mdwn deleted file mode 100644 index 3a3d679..0000000 --- a/bugs/regression_on_debian_squeeze_python_dependancy.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -obnam package can not be installed into Debian Squeeze due dependency error: - - $ sudo apt-get install obnam - ... - The following packages have unmet dependencies: - obnam : Depends: python (>= 2.6.6-7~) but 2.6.6-3+squeeze7 is to be installed - E: Broken packages - -This is duplicate of this (once resolved) bug: -http://liw.fi/obnam/bugs/obnam_squeeze_package_dependencies/ - -I used this repo as instructed on tutorial: - deb http://code.liw.fi/debian squeeze main - -BR, -ilkka.tengvall iki.fi - ---- - -This is fixed now. [[done]] --liw diff --git a/bugs/remove-checkpoint-generations.mdwn b/bugs/remove-checkpoint-generations.mdwn deleted file mode 100644 index 7037a1b..0000000 --- a/bugs/remove-checkpoint-generations.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -After Obnam finishes a proper generation, and that is committed, -it should remove all the checkpoint generations from the current -backup run. - -Currently, removing generations is so slow that this is impractical, -but eventually it should be done. - ---liw - - -forget is now slow, so added code to remove checkpoints made during -a backup run, if it is successful. --liw - -[[done]] diff --git a/bugs/remove-client-not-in-manpage.mdwn b/bugs/remove-client-not-in-manpage.mdwn deleted file mode 100644 index 79b8161..0000000 --- a/bugs/remove-client-not-in-manpage.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -"obnam remove-client" is not in the manpage. It also currently -doesn't work unless you use encryption. --liw - -[[done]] -- it is in the 1.6.1 manpage --liw diff --git a/bugs/remove-clients.mdwn b/bugs/remove-clients.mdwn deleted file mode 100644 index 6b0f85e..0000000 --- a/bugs/remove-clients.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam needs a way to remove clients from the repository. The current -remove-client command just deals with encryption. - -Suggested-by: Daniel Silverstone diff --git a/bugs/rename-client.mdwn b/bugs/rename-client.mdwn deleted file mode 100644 index 7a39d02..0000000 --- a/bugs/rename-client.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam needs to be able to rename a client. - -[[done]] this is a duplicate bug. --liw diff --git a/bugs/rename-clients.mdwn b/bugs/rename-clients.mdwn deleted file mode 100644 index 192c0a3..0000000 --- a/bugs/rename-clients.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam needs a way to rename clients in the client list. - -Suggested-by: Daniel Silverstone diff --git a/bugs/repo-checksum-use-digest-not-hexdigest.mdwn b/bugs/repo-checksum-use-digest-not-hexdigest.mdwn deleted file mode 100644 index 4b3d1fb..0000000 --- a/bugs/repo-checksum-use-digest-not-hexdigest.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -Currently `obnamlib.Repository.checksum` returns a hex-encoded checksum. -This wastes space, since we could just use a raw byte string of the -digest. However, fixing this requires changing the repository format, -and it's not worth for the minor benefit. When the format needs to -change anyway, then this should be done, too. --liw - -[[done]] diff --git a/bugs/repo-upgrades-missing.mdwn b/bugs/repo-upgrades-missing.mdwn deleted file mode 100644 index ea15e7f..0000000 --- a/bugs/repo-upgrades-missing.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam needs to be able to upgrade a repository to the current format -version. --liw - -There is now a convert5to6 command and I'll add something for each -new repository format in the future. [[done]] --liw diff --git a/bugs/report-actual-transferred-bytes.mdwn b/bugs/report-actual-transferred-bytes.mdwn deleted file mode 100644 index 9387f15..0000000 --- a/bugs/report-actual-transferred-bytes.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -obnam backup should report actual transferred bytes in addition to live -data bytes, for reporting of overhead, and should calculate speed with -overhead too - ---- - -Fixed in git master now. --liw -[[done]] diff --git a/bugs/restartable-restore.mdwn b/bugs/restartable-restore.mdwn deleted file mode 100644 index 0e81839..0000000 --- a/bugs/restartable-restore.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-wishlist]] - -Restore should be able to continue a restore if it gets interrupted. -Restore should, perhaps optionally, skip files that already exist, -and continue partial files. --liw - -This is better done with the FUSE plugin and, say, rsync. [[done]] --liw diff --git a/bugs/restore-exclude-option.mdwn b/bugs/restore-exclude-option.mdwn deleted file mode 100644 index 6f1903e..0000000 --- a/bugs/restore-exclude-option.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -It'd be handy if "obnam restore" could exclude parts of the -data, based on similar patterns as "obnam backup" does. --liw - ---- - -This is better done by using the FUSE plugin, now that it exists. ---liw -[[done]] diff --git a/bugs/restore-gid-when-possible.mdwn b/bugs/restore-gid-when-possible.mdwn deleted file mode 100644 index 4213b2f..0000000 --- a/bugs/restore-gid-when-possible.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -It is sometimes possible for a non-root to restore group ids (when obnam -process is in the group already). Should obnam try to do that? (Based -on suggestion from Gonéri Le Bouder.) --liw - -Seems like obnam should just try to restore whatever it has, and produce an error if it can't.-- Josh - -GNU Tar has the same issue. I should probably make Obnam have the same behavior as tar. I -expect that'll give people the fewest nasty surprises. Now I just need to figure out what tar does. --liw - -GNU Tar does not restore the group even if the user is in the group, -unless the user is root. I will keep this behavior with Obnam, since -it should be the least risky surprise: there is the least chance of -a privilege escalation from this. --liw - -[[done]] diff --git a/bugs/restore-overwrites.mdwn b/bugs/restore-overwrites.mdwn deleted file mode 100644 index d367df8..0000000 --- a/bugs/restore-overwrites.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -`obnam restore --to=/` seems to not complain that the target directory -exists. Obnam should require the target directory to not exist, so -that restores do not overwrite valuable data. --liw - - -Obnam restore now requires a new or non-empty directory to restore to. -[[done]] --liw diff --git a/bugs/restore-skip-files.mdwn b/bugs/restore-skip-files.mdwn deleted file mode 100644 index aa9cf87..0000000 --- a/bugs/restore-skip-files.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be practical if "obnam restore" could skip files based -on various criteria, at least pathname. - - ---- - -Rather than add a complicated language for excluding what to restore, -using the FUSE plugin seems like an obviously better way to do this. -[[done]] --liw diff --git a/bugs/restore-timestamp-trouble.mdwn b/bugs/restore-timestamp-trouble.mdwn deleted file mode 100644 index 2a70a16..0000000 --- a/bugs/restore-timestamp-trouble.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -Why does restore verification fail with small timestamp differences -when seivot runs summain with mtime enabled? ---liw - -[[done]] thanks to avoiding `os.lstat` and using our own lstat(2) -wrapper. diff --git a/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn b/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn deleted file mode 100644 index 50c5016..0000000 --- a/bugs/restore_from_gzip__39__d_repo_gives_confusing_error_message.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -Create repository using: -obnam --compress-with=gzip --repository --repository /tmp/backup/ backup "$HOME/" - -Try to restore: -obnam restore --repository /tmp/backup/ --to=/tmp/restore "$HOME/test" - -Error message: -ERROR: Invalid repository format version ('x\x9c3\xe5\x02\x00\x00v\x00@') -- forgot encryption? - -Either the compression used should be autodetected (maybe saved with metadata?) or the error message should be adjusted (using obnam 0.25) - ---xeen - ---- - -I confirm that I see the bug. And I agree that Obnam should automatically detect -when compression has been used and that the error message is awful. - ---- - - -Proposed solution (involving re-formatting the repository, for fun and profit) - -1. Alter the metadata store to not be filtered - * Alter Repository to not use encryption filter in the metadata files - * Alter HookedFS to allow you to ask that data filters be bypassed - * Alter Repository to ask for bypass on metadata files (format) -2. Alter the data filtering hooks to work as follows: - * When filtering for write, the output data format should always be filtername\0data. This chains for each write filter. - * When filtering for read, we read the filter name, apply its inverse on the data and then repeat, chaining until we read \0data (an empty filter name) - -Side effects: - -* Ordering of filters can change at will, because reading data is always deterministic on the stored data. -* Data is leaked about (at best) the top level filter (which might be encryption). - * Note: Encrypted repositories are obvious anyway so this should not be considered a blocker -* All files increase in size by some number of (at minimum 1) bytes. -* Repository metadata is now not affected by filters, so we can always guarantee to be able to read it and produce useful warnings/errors in case of format mismatches. - ---- - -Should be fixed in repository format 6 - -[[done]] diff --git a/bugs/ro-ops-while-rw-runs-crash.mdwn b/bugs/ro-ops-while-rw-runs-crash.mdwn deleted file mode 100644 index 50f76d4..0000000 --- a/bugs/ro-ops-while-rw-runs-crash.mdwn +++ /dev/null @@ -1,9 +0,0 @@ - <weasel> runing 'generations' while a backup or forget just is being run, - results in a backtrace - <weasel> probably should say 'repository locked', if it cannot provide the - information if work is in progress - -I wasn't able to reproduce this, at least not yet. I'll close this until -there's a traceback or recipe for reproducing. --liw - -[[done]] diff --git a/bugs/rolling-checksum-patches-need-review.mdwn b/bugs/rolling-checksum-patches-need-review.mdwn deleted file mode 100644 index 8422030..0000000 --- a/bugs/rolling-checksum-patches-need-review.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -[[!tag obnam-wishlist]] - -* Review <Thingol> 01:53:50> http://doriath.cz/obnam/rollingsum-split-2.patch - ...faster, but I don't like it much. :-) - * <Thingol> http://doriath.cz/obnam/rollingsum-split-2.patch ...but maybe - a different rolling checksum function (and/or a different window size) - would be better, it needs more testing. - - -[[done]] -- I never got around to doing this. --liw diff --git a/bugs/root-is-file-not-dir-error-message.mdwn b/bugs/root-is-file-not-dir-error-message.mdwn deleted file mode 100644 index cc20192..0000000 --- a/bugs/root-is-file-not-dir-error-message.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!tag obnam-wishlist]] - -See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654211> for info. - -[[done]] diff --git a/bugs/rsync_as_transport__63__.mdwn b/bugs/rsync_as_transport__63__.mdwn deleted file mode 100644 index abfea83..0000000 --- a/bugs/rsync_as_transport__63__.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -SFTP is very slow because of the per-command, per-file latency. rsync, on the other hand, works much more quickly for transferring large numbers of small files. I don't know how the rsync protocol works, but I assume it basically batches files together into a stream, ala tar, and lets the rsync process on the receiving end split them up. - -I am considering using a local repo and rsyncing it to a remote server rather than directly backing up with SFTP. After running a few simple tests, it seems like it should work fine. Is this a good idea, assuming I have enough disk space for a local repo? - -Even better, could Obnam somehow use rsync as a transport instead of SFTP? Maybe it could write small files locally and then rsync them to the remote repo; and for reading it could rsync small files to a local directory. It could do it in batches based on total filesize, or, perhaps more usefully, based on total number of files (backend files, not source files). Doing it this way wouldn't require much disk space--a few megs, at most, I'd think--as opposed to keeping a local copy of the entire repo. - - ---- - -Backing up to a local repository and rsyncing that to the server will work just fine. After you've rsynced you can then either continue using the local repository and rsyncing it to the server for updates, or switch completely to using the remote one only. You can't mix the two approaches, however, since that will just make Obnam confused. - -As for using rsync as a transport I don't see that as a realistic thing at this time, -but it's an innovative idea. Thanks. - -Obnam does not know beforehand which files it will need from the repository, and it needs to be able to create files in the repository concurrently with other instances of Obnam, and these defeat any benefit from rsync at this time. However, a repository-side Obnam server process might be the way to go, but there's other approaches to be used as well, such as caching files locally in a way that is safe against concurrent use of repository by multiple clients. I have those ideas written down elsewhere, so I'll mark this bug report as done. - ---liw - -[[done]] - diff --git a/bugs/salsa-tins.mdwn b/bugs/salsa-tins.mdwn deleted file mode 100644 index 4525dd4..0000000 --- a/bugs/salsa-tins.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -[[!tag obnam-performance]] - -Problem: If chunk size is reasonably large (say, a megabyte), then -most files will be smaller, and the repository ends up with a large -number of identical files. - -Idea: collect chunks into groups, called "salsa tins". - -- salsa tin = list of chunks -- salsa tin has an id -- chunk id = salsa tin id + suitable number of extra bits for - index into list -- chunk id may be 64 bits total, or 64+32, or whatever seems convenient -- no chunk gets stored alone, only in salsa tins - -This lets a client put things into the repository at will, without -synchronisation or locking beyond what the filesystem provides -(exclusive creation of files). - - ---- - -Having multiple chunks in a single file complicates the logic for -managing files in the repository, and deleting unused chunks. - -Therefore, an alternative idea: instead of shoving multiple chunks -into one file, allow files to use parts of chunks. Currently a -file's metadata lists the chunks that have its contents. Change -this to be a list of (chunk id, offset, length) triplets, where -offset and length specify a part of a chunk. This way, a client can -create one chunk that contains the data of many small files, and -they can all just use the relevant part of the chunk. Managing -removal of those files is easy: it is the current code without -modification. - ---liw - - -This is implemented in git for FORMAT GREEN ALBATROSS. [[done]] --liw diff --git a/bugs/seek-over-holes.mdwn b/bugs/seek-over-holes.mdwn deleted file mode 100644 index 9638b70..0000000 --- a/bugs/seek-over-holes.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!tag obnam-wishlist]] - -If a file is sparse, and has a large hole, it would be good to skip over -it with `SEEK_HOLE` and `SEEK_DATA`. --liw diff --git a/bugs/set-uid-git-mode-in-new-repo-files.mdwn b/bugs/set-uid-git-mode-in-new-repo-files.mdwn deleted file mode 100644 index 9cc6925..0000000 --- a/bugs/set-uid-git-mode-in-new-repo-files.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be good if Obnam could have a setting for what the uid and gid (when run -as root) and mode should be for new files it creates in the repository. - -See: - -* <http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-May/003035.html> -* <http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003038.html> diff --git a/bugs/setup-too-long-without-feedback.mdwn b/bugs/setup-too-long-without-feedback.mdwn deleted file mode 100644 index 9944f7a..0000000 --- a/bugs/setup-too-long-without-feedback.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -setting up takes too long without any feedback to show what's happening - -[[done]] --liw diff --git a/bugs/sftp-access-to-live-data-crashes.mdwn b/bugs/sftp-access-to-live-data-crashes.mdwn deleted file mode 100644 index d2806c3..0000000 --- a/bugs/sftp-access-to-live-data-crashes.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -Mysterious logging errors with sftp access to live data. - - sftp-root and sftp-repo run on xander: mysterious logging errors - Traceback (most recent call last): - File "/usr/lib/python2.6/logging/__init__.py", line 776, in emit - msg = self.format(record) - File "/usr/lib/python2.6/logging/__init__.py", line 654, in format - return fmt.format(record) - File "/usr/lib/python2.6/logging/__init__.py", line 436, in format - record.message = record.getMessage() - File "/usr/lib/python2.6/logging/__init__.py", line 306, in getMessage - msg = msg % self.args - TypeError: not enough arguments for format string - -* happened with repo on localfs, live data on sftp (sftp2) -* running with both on localfs (sftp3): worked fine -* so there's something wrong, possibly in paramiko -* tried a script to transfer 140 gigs of data with paramiko, but this failed, - the transfer speed is very slow (less than 1 MiB/s) - ---liw - ---- - -Removing 1.0 blocker tag: access to live data over sftp is limited enough -that it's not something I want to guarantee for 1.0. Fixing the limitations -may require patching paramiko or replacing it with another sftp -implementation. --liw - - - ---- - -I can't reproduce this with the current version of paramiko on Debian wheezy -(1.7.7.1-3.1) and the current obnam (1.3). [[done]] --liw diff --git a/bugs/sftp-errors.mdwn b/bugs/sftp-errors.mdwn deleted file mode 100644 index b79365e..0000000 --- a/bugs/sftp-errors.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!meta title="Sftp gives no useful error when remote disk is full"]] - -If server's disk is full, Obnam seems to give no useful error message: - - 2015-08-29 16:31:42 ERROR Can't back up - /home/liw/ick/liw.state/extrautils/builds/106/build.log: RD5FA4X: - System error: chunks/1052/1065/3340/212da18852839223: None: - Failure - - 2015-08-29 16:31:42 ERROR OSError(None, 'Failure') - -It looks like Obnam is getting too little info from paramiko, but -maybe that can be fixed. - -Reported by Bazyli Brzóska, -<http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org/2015-April/000144.html> diff --git a/bugs/sftp-get-put-for-transfer-for-speed.mdwn b/bugs/sftp-get-put-for-transfer-for-speed.mdwn deleted file mode 100644 index efa37a7..0000000 --- a/bugs/sftp-get-put-for-transfer-for-speed.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!tag obnam-performance]] - -Would it be faster to use the sftp put and get methods instead of the -current open/read/write/close code for transferring files to and from -the repository? - -[[done]] probably not worth it --liw diff --git a/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn b/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn deleted file mode 100644 index 3626452..0000000 --- a/bugs/sftp-round-trip-optimisation-results-in-scary-log-messages.mdwn +++ /dev/null @@ -1,34 +0,0 @@ -[[!tag obnam-wishlist]] - -I've done a fair bit of work to optimise away a whole bunch of round -trips over sftp. For example, renaming a file over sftp can fail, if -the target name already exists. So we try to remove it first. The -pretty way of doing that would be to check if the target exists before -trying to remove it. However, that would mean an extra round trip, -when the target does exist. - -Pretty: - - 1. does target exist? - 2. if it exists, remove - 3. rename - -Ugly: - - 1. try to remove target, and ignore a failure due to target not - existing - 2. rename - -Unfortunately, doing things the ugly way results in the log file -containing a lot of tracebacks from paramiko. They're benign, but -they look scary, and they fill the log with useless text. - -Reported-By: Jordi Marqués - - -Could obnam just catch the exception and therefore stop the scary tracebacks from appearing in the logfile? - Andrew Ruthven - - -This is fixed now. It was a problem with Obnam debugging output, -not paramiko's fault at all. --liw [[done]] - diff --git a/bugs/sftp-too-limited.mdwn b/bugs/sftp-too-limited.mdwn deleted file mode 100644 index 0b3d608..0000000 --- a/bugs/sftp-too-limited.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam is currently using paramiko as the SFTP implementation. It is a -bit more limited than the SFTP protocol is, and so some stuff that Obnam -should be doing, such as restoring hardlinks across SFTP, are not -possible. There may also be some bugs with regards to timestamp handling. - -Possible fixes: - -* patch paramiko to support more of SFTP -* switch to twisted's conch or libssh or <http://pypi.python.org/pypi/ssh/> - or python-ssh2 - ---liw - diff --git a/bugs/should-detect-encryption-automatically-for-read-operations.mdwn b/bugs/should-detect-encryption-automatically-for-read-operations.mdwn deleted file mode 100644 index 361f8a3..0000000 --- a/bugs/should-detect-encryption-automatically-for-read-operations.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should detect whether encryption is used, and whether a suitable key -is available, when reading an encrypted repository. For writing anything, -the encryption key to use should be selected with `--encrypt-with`, but -for read-only, if a key is available, it should be used, regardless of -the whether `--encrypt-with` has been set or if it has been, what key it -has been set to. - -Suggested by Damien Couroussé. - ---liw - -Should be done since a while. [[done]] --liw diff --git a/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn b/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn deleted file mode 100644 index e2af811..0000000 --- a/bugs/should-not-ignore-whole-directory-when-lstat-on-one-file-fails.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -`VFS.scan_tree` uses `listdir2`, for performance, and that does `lstat` -for each file. If the `lstat` fails, `listdir2` fails, and `scan_tree` -pretends the entire directory is empty. That's a problem: it should -skip only the items for which `lstat` failed. - -FUSE can cause the `lstat` to fail. - -Thanks, weasel, for spotting the problem and trying things until we found -the reason why his home directory was getting ignored. - ---liw - -[[done]] in current bzr diff --git a/bugs/small-chunks-in-btree.mdwn b/bugs/small-chunks-in-btree.mdwn deleted file mode 100644 index 268835c..0000000 --- a/bugs/small-chunks-in-btree.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Would it be more efficient to store small amounts of data directly in -the B-tree rather than as a separate chunk? - ---liw - -Some quick experimentation indicates that this can be useful indeed. -It reduces the amount of round trips Obnam needs to do to the repository, -which matters a lot for sftp acces. It's now implemented in bzr and will -be in the next release. --liw - -[[done]] diff --git a/bugs/small-chunks-in-separate-btree.mdwn b/bugs/small-chunks-in-separate-btree.mdwn deleted file mode 100644 index 6421963..0000000 --- a/bugs/small-chunks-in-separate-btree.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -[[!tag obnam-wishlist]] - -An idea for speed improvement: store small chunks not in the -per-client B-tree, but a separate per-client B-tree. I suspect -they are currently making the metadata too sparse in the per-client -B-tree. However, this needs to be measured. --liw - --- - -This is not going to be acceptable. Even small chunks need to be -useable for de-duplication. Need a better idea. [[done]] with this idea. ---liw diff --git a/bugs/specify-ssh-key-to-use.mdwn b/bugs/specify-ssh-key-to-use.mdwn deleted file mode 100644 index b6643e9..0000000 --- a/bugs/specify-ssh-key-to-use.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -obnam should allow specifying the ssh key for sftp to use - - ---- - -This is [[done]] now. --liw diff --git a/bugs/specify-ssh-key.mdwn b/bugs/specify-ssh-key.mdwn deleted file mode 100644 index 05932af..0000000 --- a/bugs/specify-ssh-key.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -Obnam needs a way to specify which ssh key (key file) to use. -Cf. `ssh -i` option. --liw - -At the moment, obnam only uses ssh agent for authentication. That -should be fixed. --liw - -[[done]] diff --git a/bugs/struct_calcsize_vs_len_struct_pack.mdwn b/bugs/struct_calcsize_vs_len_struct_pack.mdwn deleted file mode 100644 index 6f77c4c..0000000 --- a/bugs/struct_calcsize_vs_len_struct_pack.mdwn +++ /dev/null @@ -1,68 +0,0 @@ -A small performance improvement on startup may be possible by replacing len(key('',0,0)) with struct.calcsize(self.fmt): - - python -m timeit -s \ - 'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\ - 'struct.calcsize(fmt)' - 1000000 loops, best of 3: 0.201 usec per loop - -vs - - python -m timeit -s \ - 'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\ - 'len(struct.pack(fmt,"",0,0))' - 1000000 loops, best of 3: 0.753 usec per loop - -I can't attach files, so I can't attach the bundle I think you prefer, but here's the diff, and I'll email you the bundle as an attachment. - - === modified file 'obnamlib/checksumtree.py' - --- obnamlib/checksumtree.py 2011-07-26 13:23:55 +0000 - +++ obnamlib/checksumtree.py 2013-02-08 07:09:22 +0000 - @@ -33,7 +33,7 @@ - upload_queue_size, lru_size, hooks): - tracing.trace('new ChecksumTree name=%s' % name) - self.fmt = '!%dsQQ' % checksum_length - - key_bytes = len(self.key('', 0, 0)) - + key_bytes = struct.calcsize(self.fmt) - obnamlib.RepositoryTree.__init__(self, fs, name, key_bytes, node_size, - upload_queue_size, lru_size, hooks) - self.keep_just_one_tree = True - - === modified file 'obnamlib/chunklist.py' - --- obnamlib/chunklist.py 2011-07-26 13:23:55 +0000 - +++ obnamlib/chunklist.py 2013-02-08 07:09:22 +0000 - @@ -35,14 +35,15 @@ - - def __init__(self, fs, node_size, upload_queue_size, lru_size, hooks): - tracing.trace('new ChunkList') - - self.key_bytes = len(self.key(0)) - + self.fmt = '!Q' - + self.key_bytes = struct.calcsize(self.fmt) - obnamlib.RepositoryTree.__init__(self, fs, 'chunklist', self.key_bytes, - node_size, upload_queue_size, - lru_size, hooks) - self.keep_just_one_tree = True - - def key(self, chunk_id): - - return struct.pack('!Q', chunk_id) - + return struct.pack(self.fmt, chunk_id) - - def add(self, chunk_id, checksum): - tracing.trace('chunk_id=%s', chunk_id) - - === modified file 'obnamlib/clientlist.py' - --- obnamlib/clientlist.py 2011-07-26 13:23:55 +0000 - +++ obnamlib/clientlist.py 2013-02-08 07:09:22 +0000 - @@ -49,7 +49,7 @@ - tracing.trace('new ClientList') - self.hash_len = len(self.hashfunc('')) - self.fmt = '!%dsQB' % self.hash_len - - self.key_bytes = len(self.key('', 0, 0)) - + self.key_bytes = struct.calcsize(self.fmt) - self.minkey = self.hashkey('\x00' * self.hash_len, 0, 0) - self.maxkey = self.hashkey('\xff' * self.hash_len, obnamlib.MAX_ID, - self.SUBKEY_MAX) - - ---- - -Thank you. This has been merged now. [[done]] --liw diff --git a/bugs/symlink-as-backup-root.mdwn b/bugs/symlink-as-backup-root.mdwn deleted file mode 100644 index d5b4f29..0000000 --- a/bugs/symlink-as-backup-root.mdwn +++ /dev/null @@ -1,24 +0,0 @@ -From Lars Kruse: - -> Thus the situation seems to be that obnam stores all relevant chunks, -> but due to the symlink pointing to something outside of the repository -> I cannot restore - -From liw: This is obviously a bug. Obnam should either give an error -if the backup root is a symlink, or back it up as a symlink, not -recurse into the directory and then backup that and only then ruin -everything by backing up the symlink as a symlink. - ---- - -This should now be fixed to give an error message when you backup. -I can't currently give access to the hidden files easily, I'm afraid, -but if you change your backup root to be the directory pointed at by -the symlink, and run a backup, it should go fairly fast and give you -access to your files. - -Later, I may consider allowing the root to be a symlink to a -directory, but that'll require bigger changes than this error message, -I'm afraid. Patches to implement this would be welcome. - -[[done]] --liw diff --git a/bugs/symlink-as-parent-of-backup-root.mdwn b/bugs/symlink-as-parent-of-backup-root.mdwn deleted file mode 100644 index 0034461..0000000 --- a/bugs/symlink-as-parent-of-backup-root.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Valery Yundin reports: - -> It is a problem not only when backup root is a symbolic link, but also when -> any parent directory of backup root is a symbolic link. Unless you -> explicitly ask to restore a directory which is already below symlink. - -<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-April/002976.html> - ---liw diff --git a/bugs/symlink-as-root.mdwn b/bugs/symlink-as-root.mdwn deleted file mode 100644 index 90d052f..0000000 --- a/bugs/symlink-as-root.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Obnam follows a symlink as the backup root, when backing up, but does -not let you restore the target of the symlink. It should not follow -the symlink when it is root, for consistency. --liw - -[[done]] now --liw diff --git a/bugs/tarball-with-everything.mdwn b/bugs/tarball-with-everything.mdwn deleted file mode 100644 index e19334a..0000000 --- a/bugs/tarball-with-everything.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -[[!tag obnam-wishlist]] - -From Chris: - -> python2.6 is available in epel, so it should be simple. I've hit the -> "In order to install package foo, you need package bar". -> -> I'm sure this is a simple thing to do if I actually read the -> instructions, but for those of us who can't be bothered, a single -> tarball that just worked would be nice. -> -> Ideally one that could install in my homespace for testing purposes. - -From liw: - -I think that might be workable. It's almost all pure Python, -and can be run directly from the source directories, so a tarball -with all the source projects included, plus a little script to set -up PATH and PYTHONPATH would work. - - --- - -I've been thinking about this, and I'm afraid I don't want to put in the -effort to do this. Installation is a bit of a bother, but having packages -for the main Linux distros seems like the best way to solve this. -[[done]] ---liw diff --git a/bugs/test-against-old-format-versions.mdwn b/bugs/test-against-old-format-versions.mdwn deleted file mode 100644 index aedd086..0000000 --- a/bugs/test-against-old-format-versions.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -The test suite should verify that Obnam works in the proper way when -it encounters a repository with the old format version. Whatever that -proper way might be. --liw - -The current tests already verify that the format version is correct, -and that's all I want to support right now. --liw - -[[done]] diff --git a/bugs/test-for-full-filesystem.mdwn b/bugs/test-for-full-filesystem.mdwn deleted file mode 100644 index 740a74c..0000000 --- a/bugs/test-for-full-filesystem.mdwn +++ /dev/null @@ -1 +0,0 @@ -Obnam has no test for what happens when the filesystem fills up. diff --git a/bugs/test-suite-leave-temp-files.mdwn b/bugs/test-suite-leave-temp-files.mdwn deleted file mode 100644 index ea1831d..0000000 --- a/bugs/test-suite-leave-temp-files.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -The obnam test suite leaves temporary files lying around. They should -be cleaned up. --liw - -I can't seem to reproduce this, except by killing the test suite in -the middle, and in that case it's OK to leave temp file. [[done]] --liw diff --git a/bugs/time_based_checkpoint_generations.mdwn b/bugs/time_based_checkpoint_generations.mdwn deleted file mode 100644 index 3d00f72..0000000 --- a/bugs/time_based_checkpoint_generations.mdwn +++ /dev/null @@ -1,18 +0,0 @@ -[[!tag obnam-wishlist]] - -obnam should accept SIZE values as well as TIME values for the --checkpoint option. For instance: - - obnam backup $HOME --checkpoint=5min - -This would be very handy for connections which vary a lot in speed and quality. - --- weinzwang - - -While I agree this would be useful, I am afraid I haven't made any -move towards implementing it, and as such, keeping the bug open isn't -helping me manage the bug list. If anyone really wants this, sending -a patch (even a proof of concept one) would be effective, though. ---liw - -[[done]] diff --git a/bugs/too-slow.mdwn b/bugs/too-slow.mdwn deleted file mode 100644 index f08c022..0000000 --- a/bugs/too-slow.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -Obnam should be fast enough that it can back up my laptop's home -directory, overnight, encrypted, to an unencrypted USB hard disk. -That's 140 gigabytes of data, much of music and videos and photographs, -in 12 hours. --liw - -I can now back up 200+ gigabytes of data without encryption, to an -encrypted USB drive, in four hours. --liw - -However, removing a large number of checkpoints takes much too long, -making the total backup time much too long. --liw - -Did a full backup to a fresh repository on a USB drive. 375 gigabytes of -data, about 200 thousand files, backup took almost exactly 8 hours, -not including cleaning up checkpoints, since there was a file that -went missing during the backup. Deleted the checkpoints manually: -that took another 15 minutes. That's fast enough, marking the bug [[done]]. ---liw diff --git a/bugs/too_much_test_coverage_exclusion.mdwn b/bugs/too_much_test_coverage_exclusion.mdwn deleted file mode 100644 index 5f1b01f..0000000 --- a/bugs/too_much_test_coverage_exclusion.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Too much code is excluded from test coverage. For example, as much code as possible in plugins should be unit tested (so coverage can be meaningfully tested). --liw - ---- - -I've reviewed the code now, and most of the plugins have only code that is for that plugin, and it's usually not easily unit testable in any case. [[done]] --liw diff --git a/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn b/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn deleted file mode 100644 index ac0cf06..0000000 --- a/bugs/tput:_No_value_for___36__TERM_and_no_-T_specified.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -I have a daily backup with obnam scheduled using a cronjob. The job succeeds without problems, however the cron daemon mails the following output to me every day: - - tput: No value for $TERM and no -T specified - -I looked for the source of this message, but neither obnam nor python know about the "-T" parameter. So I tried to wrap the job in a shell script that sets the TERM environment variable if not set like this: - - if [ -z "$TERM" ]; then - export TERM=linux - fi - -But that didn't help either. Any ideas how to fix this? - ---Christian - ---- - -tput is a shell command (see "man tput"). Obnam does not use it in any way, not even -indirectly, as far as I can see. What I suspect is happening is that when you invoke commands -from your crontab, it invokes your shell, and that runs something like `.bashrc`, and you -have something that calls tput. - -As such, unless shown otherwise, I'll assume this is not a bug in Obnam, sorry. --liw -[[done]] diff --git a/bugs/transient-problems-warning.mdwn b/bugs/transient-problems-warning.mdwn deleted file mode 100644 index b22321b..0000000 --- a/bugs/transient-problems-warning.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -From Neal Becker: - -> Looks like a file disappeared during backup. No big deal, but probably -> obnam should handle this a little better (warning, rather than CRITICAL) ? - - ---- - -Obnam can't know if the problem is transient or not, I think, so it's -correct that it reports it as an error rather than warning. However, -it should not abort in that case, and I don't think it does, now. --liw - -[[done]] diff --git a/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn b/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn deleted file mode 100644 index 5759dc8..0000000 --- a/bugs/turn_--exclude-caches_into_--exclude-tagfile__61__OBNAM__95__NOBACKUP.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!tag obnam-wishlist]] - -The Idea is to change or extend the --exclude-caches feature so that one can configure which filename to look for that will make obnam skip the directory - ---- - -Changing `--exclude-caches` seems wrong to me: it has a specific purpose (to implement the cache directory tagging spec, <http://www.bford.info/cachedir/spec.html>). - -Adding a new option to ignore directories that contain a specific file (or directory) would be fine. - ---liw - ---- - -bwh points out that the owner of the directory and the tag file should -be the same. diff --git a/bugs/ugly-enoent-error-message.mdwn b/bugs/ugly-enoent-error-message.mdwn deleted file mode 100644 index 4097f6e..0000000 --- a/bugs/ugly-enoent-error-message.mdwn +++ /dev/null @@ -1,24 +0,0 @@ -[[!tag obnam-wishlist]] - -ugly error message from obnam: - - 00h28m12s 42015 files; 108.21 MiB (224.7 KiB/s) - /home/liw/.mozilla/firefox/td0gERROR: Cannot back up - /home/liw/.mozilla/firefox/td0gy7bo.default/urlclassifier3.sqlite-journal: - (2, 'No such file or directory', - '/home/liw/.mozilla/firefox/td0gy7bo.default/urlclassifier3.sqlite-journal') - -- should start error message on new line -- should not print Python tuple - ---liw - ---- - -I believe the tuple printing is now fixed. --liw - ---- - -The problem with error messages getting printed out in the middle of -progress output has now been fixed in git master. --liw -[[done]] diff --git a/bugs/ugly-sshexception-error-message.mdwn b/bugs/ugly-sshexception-error-message.mdwn deleted file mode 100644 index 893146b..0000000 --- a/bugs/ugly-sshexception-error-message.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!tag obnam-wishlist]] - -The following is ugly: - - File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 667, in _read_response - raise SSHException('Server connection dropped: %s' % (str(e),)) - SSHException: Server connection dropped: - Exception OSError: OSError(32, 'Broken pipe') in <bound method SFTPFile.__del__ of <paramiko.SFTPFile object at 0x18861cd0>> ignored - -There should not be a traceback, but a proper error message. - - ---- - -I believe this is fixed now. --liw [[done]] diff --git a/bugs/use-cliapp-setup_logging-overrides.mdwn b/bugs/use-cliapp-setup_logging-overrides.mdwn deleted file mode 100644 index c477577..0000000 --- a/bugs/use-cliapp-setup_logging-overrides.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -cliapp now allows overriding parts of the logging setup, and obnam should -use that as soon as cliapp has been released. - ---- - -Fixed in git master. [[done]] --liw diff --git a/bugs/use-cliapps-plugin-manager.mdwn b/bugs/use-cliapps-plugin-manager.mdwn deleted file mode 100644 index 4da8a86..0000000 --- a/bugs/use-cliapps-plugin-manager.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Obnam should be using cliapp's plugin manager, instead of its -own. --liw - -Fixed now. [[done]] --liw diff --git a/bugs/use-existing-node-size.mdwn b/bugs/use-existing-node-size.mdwn deleted file mode 100644 index 1c6b241..0000000 --- a/bugs/use-existing-node-size.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -When using an existing repository, Obnam should use the existing node -size even if the current Obnam `--node-size` setting is different. -It might log a warning message about it, but failing work is not -acceptable. - - -[[done]] diff --git a/bugs/use-fsync.mdwn b/bugs/use-fsync.mdwn deleted file mode 100644 index f24f86b..0000000 --- a/bugs/use-fsync.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam should, at least optionally, use fsync or other methods to ensure -that everything gets committed to disk by the kernel by the end of a -backup run. --liw - -I want this to not have a huge performance impact, though. Learning -from the lessons of dpkg, sqlite/liferea/firefox, etc, and using fsync/fdatasync -and `sync_file_range` in the right ways is going to be necessary. --liw - -Not doable over sftp, of course. --liw diff --git a/bugs/username-groupname-over-sftp-wrong.mdwn b/bugs/username-groupname-over-sftp-wrong.mdwn deleted file mode 100644 index 452d428..0000000 --- a/bugs/username-groupname-over-sftp-wrong.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -When the backup root (i.e., live data) is accessed over sftp, obnam -queries the _local_ system for the names of the user and group. This -is going to result in bad data. In this case, unless it can query -the remote server, it should just not store the username and groupname -at all. --liw - -Fixing this is going to require an obnam server mode, where the remote -side is also running obnam, and we can query that for stuff. But then -SFTP as a whole will be unnecessary. --liw - -[[done]] diff --git a/bugs/utimensat-undefined-symbol.mdwn b/bugs/utimensat-undefined-symbol.mdwn deleted file mode 100644 index 323dd5e..0000000 --- a/bugs/utimensat-undefined-symbol.mdwn +++ /dev/null @@ -1,25 +0,0 @@ -[[!tag obnam-wishlist]] - -Obnam apparently depends on "utimensat", which only appears in more recent versions of glibc. Unfortunately, CentOS5's glibc does not have this routine, and we are not upgrading to CentOS6 just yet. - - [hash@hydrogen]$ uname -a - Linux hydrogen.localdomain 2.6.18-274.18.1.el5 #1 SMP Thu Feb 9 12:45:44 EST 2012 x86_64 x86_64 x86_64 GNU/Linux - -Trying too import obnam produces this error: - - >>> import obnamlib._obnam - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - ImportError: /usr/local/lib/python2.6/site-packages/obnamlib/_obnam.so: undefined symbol: utimensat - >>> - --- hash - ---- - -`utimensat` was added to the Linux kernel in version 2.6.22 (2007), and to glibc in version 2.6 (2007). It is specified in Posix 2008. It's needed to set sub-second timestamps on files on restore. As a workaround, you can change the `_obnammodule.c` to call `utimes` instead. If you can provide a clean patch, I'll be happy to add it to Obnam proper. - ---liw - -No patch has appeared, closing bug. [[done]] --liw -(Patch will of course always be useful, please send one to the mailing list.) diff --git a/bugs/verify-pick-randomly.mdwn b/bugs/verify-pick-randomly.mdwn deleted file mode 100644 index 89085ea..0000000 --- a/bugs/verify-pick-randomly.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Currently, `obnam verify` compares all files in the backup with all files -in live data. Often it would be enough to just verify a few files, to -sample the data rather than do a full verification. Obnam should be able -to pick a desired number of files randomly from the backup and compare -those. - ---liw - -[[done]] diff --git a/bugs/verify-progress-should-be-bytes.mdwn b/bugs/verify-progress-should-be-bytes.mdwn deleted file mode 100644 index 44744a9..0000000 --- a/bugs/verify-progress-should-be-bytes.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -When "obnam verify" shows progress, it's based on number of files. -It should be based on number of bytes instead, for better progress -reporting. --liw - ---- - -[[done]] in git master now. --liw diff --git a/bugs/verify-should-continue-past-error.mdwn b/bugs/verify-should-continue-past-error.mdwn deleted file mode 100644 index 5dd063c..0000000 --- a/bugs/verify-should-continue-past-error.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -Suggested by Rafał Gwiazda: "obnam verify" should report all errors, -instead of terminating after the first problem found. - ---liw - -[[done]] diff --git a/bugs/verify-stricter.mdwn b/bugs/verify-stricter.mdwn deleted file mode 100644 index 3aa4a64..0000000 --- a/bugs/verify-stricter.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -[[!tag obnam-wishlist]] - -On Tue, Mar 11, 2014 at 04:34:13PM +0100, Thomas Schwinge wrote: - -> Alternatively, wouldn't it make sense to change to, or at least have -> available, a mode where verify works not based on the data in the backup, -> but instead of the actual user data? Wouldn't doing it in this way build -> a yet greater confidence that obnam has backed up everything alright? -> That is, instead of traversing the data in the backup and verifying -> against the corresponding user data, it'd traverse the user data (as when -> doing a backup) and verify against the data in the backup. - -I think you're right. I think we should have "obnam verify" traverse -through both the backup generation and the live data, and report as -follows: - -* if a file exists only in the backup, but not in the live data -* if a file exists only in the live data, but not in the backup - - also indicate whether the file matches (current) exclusion - rules, since that matters: a file that is in live data but is - not excluded is different from an excluded one -* if a file exists in both the backup and the live data, but is - different in any way - -I don't have time to work on this at the moment, but I'll add it to -the list of bugs in http://liw.fi/obnam/bugs/ and would be happy to -review and merge a patch for this. - ---liw - -This is best done by using diff(1) on a FUSE mounted backup-repository, -I think. Don't want to duplicate all basic command line tools in Obnam. -[[done]] --liw diff --git a/bugs/vfs-split-for-live-data-repo-access.mdwn b/bugs/vfs-split-for-live-data-repo-access.mdwn deleted file mode 100644 index 9165ea9..0000000 --- a/bugs/vfs-split-for-live-data-repo-access.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -Should obnam split VFS for repo access and live data access? repo needs -so much less, after all. This would perhaps also make it easier to add support -for things such as S3. -liw - -[[done]] I have decided against this. It's unnecessary complication in -the implementation, I think. --liw diff --git a/bugs/warn-about-common-client-names.mdwn b/bugs/warn-about-common-client-names.mdwn deleted file mode 100644 index 0dd3727..0000000 --- a/bugs/warn-about-common-client-names.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be cool if Obnam could warn about common client names, -such as `localhost` or `localhost.localdomain`. - -[[done]] closing wishlist bug since it isn't making the thing happen. --liw diff --git a/bugs/whitelist-backups.mdwn b/bugs/whitelist-backups.mdwn deleted file mode 100644 index ea61394..0000000 --- a/bugs/whitelist-backups.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!tag obnam-wishlist]] - -The ability to do whitelist-based backups would be nice. This would make it easy to separate different chunks of $HOME as different clients (e.g., just game saves as a "games" client). --mathstuf - ---- - -Whitelists tend to be dangerous: you always forget something. This makes me -reluctanct to implement them. That said, you can achieve this with the PCRE -regular expressions Obnam uses. [[done]] --liw diff --git a/bugs/whole-file-checksums.mdwn b/bugs/whole-file-checksums.mdwn deleted file mode 100644 index e72139b..0000000 --- a/bugs/whole-file-checksums.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!tag obnam-wishlist]] - -It would be good for Obnam to do the whole-file checksum with a different -checksum algorithm, or by using a suitable salt, to catch problems with -single-chunk files, e.g., when there is a hash collision. --liw - -This is essentially a duplicate of the many-checksum-algos wishlist -bug. It's coming. [[done]] --liw diff --git a/bugs/wishlist-only.mdwn b/bugs/wishlist-only.mdwn deleted file mode 100644 index 8944250..0000000 --- a/bugs/wishlist-only.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!inline pages="page(bugs/*) and - !bugs/*/* and - !bugs/done and - tagged(obnam-wishlist) and - !link(bugs/done)" - show=0 - postform=no]] - diff --git a/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn b/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn deleted file mode 100644 index 13215e8..0000000 --- a/bugs/wishlist:_how_does_memory_and_cpu_usage_of_obnam_depend_on_the_number_of_files_and_their_sizes__63__.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -[[!tag obnam-wishlist]] - -As a system administrator I'd like to see data on how the memory and cpu usage of different obnam operations depend on the number of files and their sizes before adopting obnam for my own systems. - -I imagine that in the simplest case you could use something like - - for total_size in 50M 1G 50G 400G 1T 2T 3T; do - for file_size in 1k 2k 4k 8k 16k 32k 1M 64M; do - generate_filesystem $total_size $file_size original/ - /usr/bin/time -f "%e real %P rss" backup original/ backup/ - done - done - -to get cpu and memory usage of backup operations. Repeat same for obnam fsck if you are afraid that its memory usage could depend on total_size or file_size. - -[[done]] A very old wishlist bug. Closing this, since keeping it open -clearly isn't making it happen. --liw diff --git a/bugs/xattr-changes-do-not-trigger-backup.mdwn b/bugs/xattr-changes-do-not-trigger-backup.mdwn deleted file mode 100644 index d001376..0000000 --- a/bugs/xattr-changes-do-not-trigger-backup.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -[[!tag obnam-1.0-blocker]] - -It looks as if changes to xattrs (but nothing else) will not -trigger a backup in Obnam. They should. --liw - -[[done]] diff --git a/bytemark-4aa4ed38.png b/bytemark-4aa4ed38.png Binary files differdeleted file mode 100644 index 3b9d4dc..0000000 --- a/bytemark-4aa4ed38.png +++ /dev/null diff --git a/bytemark-ff3270c4.svg b/bytemark-ff3270c4.svg deleted file mode 100644 index 2685cf8..0000000 --- a/bytemark-ff3270c4.svg +++ /dev/null @@ -1,32 +0,0 @@ -<?xml version="1.0" encoding="utf-8"?> -<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd"> -<svg version="1.0" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" - width="512.254px" height="66.05px" viewBox="0 0 512.254 66.05" enable-background="new 0 0 512.254 66.05" xml:space="preserve"> -<g id="Layer_6"> - <g> - <path fill="#004EA4" d="M51.898,50.314h11.93c4.254,0,7.958-1.193,7.958-6.281c0-3.883-2.314-6.014-7.127-6.014H51.894v12.295 - M51.898,25.715h10.73c4.25,0,6.935-1.199,6.935-5.452c0-3.326-2.78-4.529-6.935-4.529h-10.73V25.715z M31.551,0h36.162 - c17.392,0,21.09,9.815,21.09,16.56c0,6.667-3.238,10.265-8.144,12.954c5.923,2.035,11.472,6.747,11.472,16.466 - c0,13.217-11.472,20.066-23.12,20.066h-37.46V0z"/> - <polygon fill="#004EA4" points="104.822,41.719 81.601,0 104.078,0 115.086,24.331 126.557,0 148.846,0 125.171,41.719 - 125.171,66.047 104.822,66.047 "/> - <polygon fill="#004EA4" points="157.541,16.938 139.043,16.938 139.043,0 196.388,0 196.388,16.938 177.891,16.938 - 177.891,66.047 157.541,66.047 "/> - <polygon fill="#004EA4" points="194.626,0 249.293,0 249.293,16.938 214.973,16.938 214.973,25.163 246.146,25.163 - 246.146,40.887 214.973,40.887 214.973,49.121 250.307,49.121 250.307,66.047 194.626,66.047 "/> - <polygon fill="#004EA4" points="249.109,0 278.061,0 287.497,38.848 287.682,38.848 297.115,0 326.064,0 326.064,66.047 - 306.832,66.047 306.832,23.683 306.641,23.683 295.171,66.047 280.001,66.047 268.533,23.683 268.352,23.683 268.352,66.047 - 249.109,66.047 "/> - <path fill="#004EA4" d="M361.58,42.457l-5.912-20.339h-0.189l-6.384,20.339H361.58z M345.576,0h19.89l24.05,66.047H368.43 - l-2.781-9.434H344.66l-2.959,9.434h-20.444L345.576,0z"/> - <path fill="#004EA4" d="M405.146,28.865h10.634c3.792,0,8.979-0.646,8.979-6.565c0-4.163-2.314-6.566-10.088-6.566h-9.524 - L405.146,28.865 M384.794,0h38.759c11.562,0,21.55,6.389,21.55,18.88c0,6.835-3.15,14.052-9.895,16.554 - c5.544,2.125,8.965,8.229,9.708,16.463c0.278,3.229,0.372,11.096,2.221,14.15h-20.353c-1.018-3.328-1.38-6.754-1.662-10.175 - c-0.559-6.29-1.111-12.856-9.156-12.856h-10.82V66.05h-20.352V0z"/> - <polygon fill="#004EA4" points="444.636,0 464.988,0 464.988,22.755 465.176,22.755 483.292,0 508.363,0 484.41,25.812 - 512.254,66.047 486.906,66.047 470.628,40.334 464.988,46.537 464.988,66.047 444.636,66.047 "/> - <rect y="8.652" fill="#004EA4" width="20.23" height="19.252"/> - <rect y="39.874" fill="#004EA4" width="20.23" height="19.247"/> - </g> -</g> -</svg> diff --git a/contact.mdwn b/contact.mdwn deleted file mode 100644 index f95a60d..0000000 --- a/contact.mdwn +++ /dev/null @@ -1,67 +0,0 @@ -[[!meta title="Contact"]] - -E-mail -====== - -* If you have any questions about Obnam, or any problems when using - it, e-mail the [support list][]. -* If you have made improvements to Obnam, or want to discuss ways in - which you can implement them, please e-mail the - [development list][]. -* If unsure which list is appropriate, use the support list. -* The [announcement list][] is used by Obnam developers to announce - new releases, or to alert Obnam users about anything that they need - to know about. If you're using Obnam, you probably should be - subscribed to this list. It is very low volume. -* Please don't send mail directly to Lars. That prevents others from - helping out, and puts all the support burden on one person. Also, - the mailing lists get automatically imported into a ticketing - system, so there is less chance of issues being overlooked. - -The support list is available on the [Gmane]() site, -at <http://dir.gmane.org/gmane.comp.sysutils.backup.obnam>. The -official archives sometimes have messages in base64 encoding, and -their UI isn't great, so the Gmane archives are often a better way to -read the archives. (Update: Gmane is in a state of possibly being -transferred to a new owner. It might not work, for now.) - -[support list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org -[development list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org -[announcement list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-announce-obnam.org -[Gmane]: http://gmane.org/ - -IRC -=== - -The `#obnam` channel on the `irc.oftc.net` IRC network exists for -realtime discussions about Obnam. Note that the [support list][] is -strongly preferred over the IRC channel for support issues, but you're -welcome to hang out and chat if you want. - -Note that the main Obnam developer, Lars Wirzenius, rarely has time -for serious discussions on IRC, especially during working hours. - -Patches and fixes and new features -================================== - -If you want to make changes to Obnam, or this website, please clone -the git repository and send patches to the [development list][], -preferably using `git format-patch --cover-letter`. - -For learning to use git, start with these pages: - -* [Basics of version control systems][] on Yakking. -* [Git tutorial][] - -[Basics of version control systems]: http://yakking.branchable.com/posts/basics-of-cs/ -[Git tutorial]: http://www.git-scm.com/book/en/Getting-Started - - -More information -================ - -* [Lars's blog posts about Obnam](http://blog.liw.fi/tag/obnam/) - and [larch](http://blog.liw.fi/tag/larch/). -* Also, [Lars's private journal entries](http://liw.fi/obnam/journal-dump/) - about obnam and btree. -* [Sharon Kimble's blog posts about Obnam](http://www.sharons.org.uk/tag/obnam/) diff --git a/development.mdwn b/development.mdwn deleted file mode 100644 index 627f228..0000000 --- a/development.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -Development ------------ - -**Contributions are welcome.** Every level of contribution is most -appreciated: bug reports, spelling and grammar fixes, code patches, -questions on how to get started, offers to run a continuous integration -server, etc. See [[README]] for information -on how to modify the code. and ask on the mailing list or on IRC for -anything else. If anything's unclear, ask! - -Things to do, if you want to help: - -* Review manual page. It is easy for documentation to get - out of date. Pointing out problems is helpful. -* Code review, both of obnam and the larch btree library. -* Benchmarking. -* Add support for translations. -* More! Add here your own ideas. - -Other stuff: - -* [[Obnam on-disk data structures|ondisk]] -* Repository [[encryption]] and [[signing]] -* Repository [[locking]] -* [[bugs]] - - see [code.liw.fi](http://liw.fi/code/) for instructions - -Dependencies: - -* Python 2 -* paramiko -* larch: <http://liw.fi/larch/> -* ttystatus: <http://liw.fi/ttystatus/> -* CoverageTestRunner: <http://liw.fi/coverage-test-runner/> - (you only need this for running the test suite) diff --git a/donate.mdwn b/donate.mdwn deleted file mode 100644 index 6101190..0000000 --- a/donate.mdwn +++ /dev/null @@ -1,51 +0,0 @@ -[[!meta title="Donations"]] - -<pre> ------BEGIN PGP SIGNED MESSAGE----- -Hash: SHA512 - -Hi! I wrote Obnam, a backup program (http://liw.fi/obnam/). If you -like the program and would like donate a little, you can send me -gifts: - -* Amazon wishlist: - http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv - -I no longer want bitcoin, they're too painful to exchange into money. - -Obnam does not run on donations: it currently mainly runs on my free -time. Donations are not required in any way: you're more than welcome -to use it without donating, and in any case I prefer a -well-formulated, actionable bug report (see -http://obnam.org/bug-reporting/) or a patch to fix a bug or add a -feature (with appropirate updates to tests). No bug is too small to be -reported. - -I can also arrange to do Obnam development for specific features, or -support, for a fee, if you or your company needs something that isn't -happening otherwise. - -Lars Wirzenius, 2017-06-20 ------BEGIN PGP SIGNATURE----- - -iQIzBAEBCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllI7O4ACgkQbC+mFux6 -IDGuoA/6AgLfrUc4KGKgGI7NwyqqV4ap7A+Rv/O/CmSq3L68gmp+lnHKAb2kWG6o -6SNVwgAxhgVpuChZX47NBOjmB9ZjHou80743j4COGpTH+8gAamEoF58TapDUMlvQ -tPn+fFQOSBmQPyT+SVUL/A+VXJEYnFLefAkvTc8fwkUt0IrYsvh+PmlBoT8iLcQg -bBi6o1ijZugm568wAt6kt8mQjDvzr+AdrnSo+rREbqMmijnYxZkncEthZ+Qolxej -zzBxifduwGRiTX6zpUXfO14p0pytZfGLPJKQjDPlWA72XZ5qSE1QII8D1pDDfAD/ -JPk1cNsFyCiL/nN2Eb+3fhphlpaRc2DLgv454AhZSR6ppO7eQw9uWGZEjmL8+Qyj -hqnKLFiFcTmagY0z9DJnFsQfnOHZCRoQd8pHDaUDxCynDjUCRJA+2aOBtRF/rRxQ -GP1HNo4Gof9ZF//CfmBT51avc+iGEcK/yzkq5w7haEgyzRLqjqcCAjmW6cTg8ef2 -FcA+QdBGRXDKUtMO493+1s6G7ChmAiO4VWxge38EYlbyiUkv6X8ES/5AYwPoRspz -9dChIfST7N7oKN7e07jEQI0aKb61E4H7vANFXT64RTg+YGla+maJQ1vHM1HDoqfF -Dt68vc/h4TUN2XS3u+IQJu0ta0/V8xMFJOvXLiiaB6QufTNwtBw= -=uNhA ------END PGP SIGNATURE----- -</pre> - -(See [[donate.txt]] for a plain text version.) - ---- - -[[!img liw-bitcoin-qr-obnam.png]] diff --git a/donate.txt b/donate.txt deleted file mode 100644 index cac8e06..0000000 --- a/donate.txt +++ /dev/null @@ -1,41 +0,0 @@ ------BEGIN PGP SIGNED MESSAGE----- -Hash: SHA512 - -Hi! I wrote Obnam, a backup program (http://liw.fi/obnam/). If you -like the program and would like donate a little, you can send me -gifts: - -* Amazon wishlist: - http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv - -I no longer want bitcoin, they're too painful to exchange into money. - -Obnam does not run on donations: it currently mainly runs on my free -time. Donations are not required in any way: you're more than welcome -to use it without donating, and in any case I prefer a -well-formulated, actionable bug report (see -http://obnam.org/bug-reporting/) or a patch to fix a bug or add a -feature (with appropirate updates to tests). No bug is too small to be -reported. - -I can also arrange to do Obnam development for specific features, or -support, for a fee, if you or your company needs something that isn't -happening otherwise. - -Lars Wirzenius, 2017-06-20 ------BEGIN PGP SIGNATURE----- - -iQIzBAEBCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllI7O4ACgkQbC+mFux6 -IDGuoA/6AgLfrUc4KGKgGI7NwyqqV4ap7A+Rv/O/CmSq3L68gmp+lnHKAb2kWG6o -6SNVwgAxhgVpuChZX47NBOjmB9ZjHou80743j4COGpTH+8gAamEoF58TapDUMlvQ -tPn+fFQOSBmQPyT+SVUL/A+VXJEYnFLefAkvTc8fwkUt0IrYsvh+PmlBoT8iLcQg -bBi6o1ijZugm568wAt6kt8mQjDvzr+AdrnSo+rREbqMmijnYxZkncEthZ+Qolxej -zzBxifduwGRiTX6zpUXfO14p0pytZfGLPJKQjDPlWA72XZ5qSE1QII8D1pDDfAD/ -JPk1cNsFyCiL/nN2Eb+3fhphlpaRc2DLgv454AhZSR6ppO7eQw9uWGZEjmL8+Qyj -hqnKLFiFcTmagY0z9DJnFsQfnOHZCRoQd8pHDaUDxCynDjUCRJA+2aOBtRF/rRxQ -GP1HNo4Gof9ZF//CfmBT51avc+iGEcK/yzkq5w7haEgyzRLqjqcCAjmW6cTg8ef2 -FcA+QdBGRXDKUtMO493+1s6G7ChmAiO4VWxge38EYlbyiUkv6X8ES/5AYwPoRspz -9dChIfST7N7oKN7e07jEQI0aKb61E4H7vANFXT64RTg+YGla+maJQ1vHM1HDoqfF -Dt68vc/h4TUN2XS3u+IQJu0ta0/V8xMFJOvXLiiaB6QufTNwtBw= -=uNhA ------END PGP SIGNATURE----- diff --git a/download.mdwn b/download.mdwn deleted file mode 100644 index 27e4323..0000000 --- a/download.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -Download --------- - -* [direct download](http://code.liw.fi/debian/pool/main/o/obnam/); - includes source code, and Debian source and binary packages for - i386, and amd64 architectures for the releases of Debian I happen - to support -* [using apt](http://liw.fi/code/) (currently wheezy, jessie, and - unstable supported): - - `deb http://code.liw.fi/debian wheezy main` - -* For Ubuntu, see [Chris's - PPA](https://launchpad.net/~chris-bigballofwax/+archive/obnam-ppa/). -* For Gentoo, Obnam is in the official portage. - * <https://packages.gentoo.org/packages/app-backup/obnam> -* For openSUSE, see Valery Yundin's repository: - <http://download.opensuse.org/repositories/home:/Vayun/> - - spec files: - <https://build.opensuse.org/project/show?project=home%3AVayun> - -* CentOS - - yum install epel-release - - yum install obnam fuse python-fuse - -* from version control: - - `git clone git://git.liw.fi/obnam` - -The code is licensed under GNU General Public License version 3, or later. - diff --git a/encryption.mdwn b/encryption.mdwn deleted file mode 100644 index 7c8ba64..0000000 --- a/encryption.mdwn +++ /dev/null @@ -1,384 +0,0 @@ -[[!meta title="Encryption support in Obnam"]] - -Obnam needs to support encryption of the backup store. This document -describes requirements for and design of how Obnam will be using -encryption. It is currently a **DRAFT** and feedback is very much -welcome. - -[[!toc ]] - - -Goal ----- - -The goal of encryption in Obnam is to avoid unauthorized parties from -gaining access to backed up data. If the backup store is on a server -on the Internet, and someone breaks into the server, the data should -be safe from prying eyes. Likewise, if the backup store is on a USB -hard disk, and the disk gets stolen, the data should not leak. - -This document does not worry about the data as it is being transferred. -The data is encrypted on the machine running obnam, -which reads the live data and stores it in a backup store, -and there may be up to three machines involved, and the links between -them are assumed to be over sftp or another sensible protocol. -(In other words, obnam may be running on one machine, the live data -may be on another one, and the backup store on a third one.) - -For the purpose of this document, the server hosting the backup store -is considered to be unauthorized to access the data, or do anything -to the data except store it. This means that if the server needs to -things like removing unwanted backup generations, it needs to be -explicitly authorized to do so, by giving it encryption keys. By -default, it should not have that access, and will see only encrypted -data. - - -Requirements ------------- - -* All data in the backup store MUST be encrypted. - Filenames are not encrypted, but Obnam uses random filenames whenever - it can already. -* The backup store does not need to store any backup keys. It MUST be - enough for the keys to be stored in the clients only. -* For disaster recovery, it may be necessary to be able to access the - backup store only using a passphrase. This should be supported by the - design, but use of this feature MUST be optional. -* Clients can replace public keys used with the backup store at any time, - without having to re-backup anything. After keys have been replaced, the - old keys MUST NOT be able to access data in the backup store. -* The design MUST support use of multiple keys. Each client MUST be able to - use their own key, and disallow anyone else from accessing their data, - even if the backup store is shared. It should be possible for the server - to have its own key that it uses for forgetting old generations, running - fsck on the entire store, and so on. - -The encryption keys for accessing the backup store are not used for -authenticating access to the backup store, to allow flexibility in -selecting store providers. - - -Encryption methods ------------------- - -There are two relevant encryption methods for Obnam: - -* public key cryptography -* symmetric cryptography - -With public keys, there is a optional passphrase for the secret key. -With symmetric encryption, the passphrase is the key. -The user needs to manage the secret key and its passphrase in a suitable -manner. - - -Design ------- - -All data in the backup store is stored either in B-tree nodes, -or in a special directory tree for chunks of file data. Each B-tree -is stored in its own directory tree, and each node is stored in its -own file. We'll call each of these directory -trees a _toplevel_. Thus the root of a backup store consists of some -number of toplevel directories. - -The backup store is used by some set of clients, plus perhaps the -backup server. We'll call each of these a user. Each user may have -its own public/private key pair, or they may be shared between -clients. This is up to the backup admins. The design assumes they -all have unique key pairs. - -Some toplevels are private for a particular client. Others are -shared. - -Every file in a toplevel is encrypted using symmetric encryption. -The symmetric encryption key is encrypted with the public -key of every user who needs access to the toplevel. In other words, -to have access to contents of the toplevel, a user needs to be able -to decrypt the symmetric encryption key using their own key. - -To stop a user from having access to a toplevel, the symmetric encryption key -file gets re-encrypted with every other key except the unwanted one. -This does not prevent an attacker who has previously stored a -local copy of the decrypted symmetric encryption key. To stop that, every file -in the toplevel needs to be re-encrypted with a new symmetric -encryption key. - -The encrypted symmetric encryption key is stored inside -the toplevel, using a well-known name. This is the only file not -encrypted with the symmetric encryption key. - -The public keys for users who should have access to the toplevel -are also stored inside the toplevel, in a file encrypted with the -symmetric encryption key. - -The symmetric encryption keys are generated randomly, -and are of sufficient length that brute-forcing them will not be -realistic. Perhaps something like 256 bits from /dev/random. - -To give a user access to the repository, the repository admin needs -to add the user's public key to the shared toplevels: client list, -chunks, and chunk checksum data. A client may remove itself from -the repository, or an admin may do that. "Admin" in this context -is anyone with access to all the shared toplevels. - - -Example -------- - - chunks/ -- toplevel for file data chunks - key -- symmetric encryption key - userkeys -- public keys of all users of this toplevel - * -- all other files - -To set up toplevel for encryption: - -* generate a suitable amount of random data to use as symmetric encryption key -* create list of public keys to have access to toplevel -* encrypt public key list using symmetric encryption key -* upload encrypted public key list -* encrypt symmetric encryption key with every public key in list -* upload encrypted symmetric encryption key - -To add a file to the toplevel: - -* download `key` file -* decrypt `key` file using user's private key, get key text -* encrypt file using symmetric encryption key -* upload encrypted file - -To use a file in the toplevel: - -* download `key` file -* decrypt `key` using user's private key -* download desired file -* decrypt file using symmetric encryption key - - -Discussion ----------- - -The simplest approach would be to only use public key encryption, -but this makes it difficult to change the keys. Changing the keys -is necessary to handle scenarios like giving access to the shared -toplevels to a new user with a new key pair. Otherwise the -symmetric encryption key -needs to be distributed to every user, and re-distributed -if it ever changes, and this is cumbersome. It would also be possible -to re-encrypt everything in the toplevel for every new user, but -that is laughably inefficient. However, it would be acceptably -simple to support the scenario of distributing the symmetric encryption key to -every user, if the backup admin thinks storing it on the backup -server even in encrypted form is too risky. - -I have removed [[data signing|signing]] from this spec, on the suggestion of -Daniel Kahn Gilmor. Data signing will be dealt with separately. - -I am going to assume that any public keys being used are generated -by the user, not by obnam. - -I am not an encryption expert. I will not be implementing my own -encryption code, and do not even want to choose the specific -algorithms or key formats. I will be using GnuPG for all encryption -operations, because it is well-known and well-respected, and lets -me outsource all thinking. - - -Implementation outline ----------------------- - -General repository I/O operations (these correspond to the -`mkdir`, `write_file`, and `cat` operations in the Obnam VFS layer): - - def repo_mkdir(pathname): - # create a new directory - - def repo_write_file(pathname, contents): - # write a file to the repository (pathname is relative to repo) - - def repo_read_file(pathname): - # read contents of a file in the repository (name is relative to repo) - -General encryption routines: - - def generate_symmetric_key(): - # return N random bits to be used as a symmetric encryption key - - def encrypt_with_symmetric_key(data, symmetric_key): - # return data encrypted using symmetric encryption - - def decrypt_with_symmetric_key(encrypted, symmetric_key): - # return data after it has been decrypted using symmetric encryption - - def encrypt_with_pubkeys(data, pubkeys): - # return data after it has been encrypted for all of the given - # public keys - - def decrypt_with_secret_key(encrypted, secret_key): - # decrypt encrypted data using a secret key; this will fail unless - # the data was encrypted using the public key corresponding to the - # the secret key - -Keyring handling in memory: - - def create_empty_keyring(): - ... - - def add_to_keyring(keyring, key): - ... - - def keyring_contains(keyring, key): - ... - - def remove_from_keyring(keyring, key): - ... - - def encode_keyring(keyring): - # Return form of keyring that can be stored on disk. - - def decode_keyring(encoded): - # Inverse of encode_keyring. - -Create a new toplevel: - - def create_toplevel(name, pubkeys): - repo_mkdir(name) - - symmetric_key = generate_symmetric_key() - encrypted = encrypt_with_pubkeys(symmetric_key, pubkeys) - repo_write_file(name + '/key', encrypted) - - keyring = create_empty_keyring() - for pubkey in pubkeys: - add_to_keyring(keyring, pubkey) - encoded = encode_keyring(keyring) - encrypted = encrypt_symmetric(encoded, symmetric_key) - repo_write_file(name + '/userkeys', encrypted) - -Reading and writing files in a toplevel: - - def get_symmetric_key(toplevel, secret_key): - encoded = repo_read_file(toplevel + '/key') - return decrypt_with_secret_key(encoded, secret_key) - - def toplevel_read_file(toplevel, filename, secret_key): - symmetric_key = get_symmetric_key(toplevel, secret_key) - encoded = repo_read_file(toplevel + '/' + filename) - return decrypt_with_symmetric_key(encoded, symmetric_key) - - def toplevel_write_file(toplevel, filename, cleartext, secret_key): - symmetric_key = get_symmetric_key(toplevel, secret_key) - encoded = encrypt_with_symmetric_key(cleartext, symmetric_key) - repo_write_file(toplevel + '/' + filename, encoded) - -Manage keys for a toplevel: - - def read_keyring(toplevel, name, secret_key): - encoded = toplevel_read_file(toplevel, name, secret_key) - return decode_keyring(encoded) - - def write_keyring(toplevel, name, keyring, secret_key): - encoded = encode_keyring(keyring) - toplevel_write_file(toplevel, name, encoded, secret_key) - - def add_to_userkeys(toplevel, public_key, secret_key): - userkeys = read_keyring(toplevel, 'userkeys', secret_key) - if not keyring_contains(userkeys, public_key): - add_to_keyring(userkeys, public_key) - write_keyring(toplevel, 'userkeys', userkeys, secret_key) - - def remove_from_userkeys(toplevel, public_key, secret_key): - userkeys = read_keyring(toplevel, 'userkeys', secret_key) - if keyring_contains(userkeys, public_key): - remove_from_keyring(userkeys, public_key) - write_keyring(toplevel, 'userkeys', userkeys, secret_key) - -Repository client management: - - def add_client(client_public_key, admin_secret_key): - add_to_userkeys('metadata', client_public_key, admin_secret_key) - add_to_userkeys('clientlist', client_public_key, admin_secret_key) - add_to_userkeys('chunks', client_public_key, admin_secret_key) - add_to_userkeys('chunksums', client_public_key, admin_secret_key) - # client will add itself to the clientlist and create its own toplevel - - def remove_client(client_public_key, admin_secret_key): - # client may remove itself, since it has access to the symmetric keys - # we assume the client-specific toplevel has already been removed - remove_from_userkeys('chunksums', client_public_key, admin_secret_key) - remove_from_userkeys('chunks', client_public_key, admin_secret_key) - remove_from_userkeys('clientlist', client_public_key, admin_secret_key) - remove_from_userkeys('metadata', client_public_key, admin_secret_key) - - -Hooks in Obnam --------------- - -Obnam's Repository class needs to have a pair of hooks for modifying -data before it gets written to the repository, and after it has been -read. These modifications should be each other's inverse functions. -Apart from encryption, these hooks could be used for error correction -codes for data in the store, and perhaps other things. The repository -should just provide and call the hooks, and not otherwise concern -itself with encryption. - -These hooks are not needed at the VFS layer, since it is not necessary -to decrypt live data, nor encrypt data that is being restored. - -The hooks correspond to `create_toplevel`, `toplevel_read_file`, -and `toplevel_write_file` above. However, to allow chains of callbacks -for the hooks, instead of the encryption callback writing the -data out to the repository, it should return it instead. The next -callback will get the encrypted data, and add, say, error correction -codes to it. Finally, when all callbacks are done, the encrypted and -error-corrected blob gets written to the repository. - -Thus, Repository should provide the following hooks: - -* `repo-create-toplevel(name)`: called whenever the repository has created - a new toplevel directory -* `repo-write(toplevel, filename, data)`: called by the repository - prepare data to be written to the repository -* `repo-read(toplevel, filename, data)`: called by repository for data that - has been read from the repository - -The hook subsystem needs to have a way to order callbacks, and for -each callback to return a modified form of the data for the next -callback to process (instead of the next callback processing the -original data). - -Callback ordering is important so that encryption always happens -before ECC encoding: there's no point in ECC if it happens before -encryption. - -Since the universe of likely Obnam plugins is small, and it can be -assumed that plugin authors co-operate, we can achieve ordering -most simply by having an optional integer _order_ argument to -the hook registration method. - - def add_callback(self, name, callback, order=None): - -Any callbacks without an explicit order will be put at the end -of the callback chain, and all others to the beginning, sorted -by the order argument into increasing order. - -We can the arrange for the callback registrations for encryption -and ECC to use appropriate ordering: - - hooks.add_callback('repo-write', encrypt, order=1000) - hooks.add_callback('repo-write', ecc, order=2000) - -(Reverse the ordering for `repo-read`, of course.) - -Arranging for a hook to be able to modify the data is a bit -trickier. Ideally, it could just return the new data, but the -general purpose nature of the hook subsystem means that it does -not know what the arguments for a hook are. - -Thanks ------- - -Thank you to Richard Braakman, Peter Palfrader, Jaakko Niemi, -and Daniel Kahn Gillmor for feedback. Any problems that remain -are my fault. diff --git a/faq.mdwn b/faq.mdwn deleted file mode 100644 index 75dc8e0..0000000 --- a/faq.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -[[!meta title="Obnam FAQ"]] - -[[!inline pages="page(faq/*)" archive=yes template=titlepage sort=meta(title)]] diff --git a/faq/checksum-safety.mdwn b/faq/checksum-safety.mdwn deleted file mode 100644 index cab4802..0000000 --- a/faq/checksum-safety.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -[[!meta title="Checksum collisions and safety"]] - -Obnam is using the MD5 checksum algorithm for recognising duplicate -data chunks. MD5 has a reputation for being unsafe: people have -constructed files that are different, but result in the same MD5 -checksum. This is true. - -Every checksum algorithm can have collisions. Changing Obnam to, say, -SHA1, SHA2, or the as yet unreleased SHA3 would not remove the chance -of collisions. It would reduce the chance of accidental collisions, -but the chance of those is already so small with MD5 that it can be -disregarded. Or put in another way, if you care about the chance of -accidental MD5 collisions, you should be caring about accidental SHA1, -SHA2, or SHA3 collisions as well. - -Apart from accidental collisions, there are two cases where you should -worry about checksum collisions (regardless of algorithm). - -First, if you're into researching checksum collisions, you're likely -to have files that cause checksum collisions, and in that case, if you -restore after a catastrophe, you probably want to get the files back -intact, rather having Obnam confuse one with the other. - -Second, if you have an enemy who wishes to corrupt your backed up -data, they may replace some of the backed up data with other data that -has the same checksum. This way, when you restore, your data is -corrupted without Obnam noticing. - -For both of these cases, you can instruct Obnam to **verify** that -chunks of data with the same checksum actually are the same data, -instead of relying on the checksum alone. This is as safe as it can -be, but it has a big performance impact. It causes Obnam to have to -read from the repository (possibly downloading it from your backup -server) all the data you are backing up. You'll still benefit from the -de-duplication, however, so your repository size will be smaller. diff --git a/faq/compared-to.mdwn b/faq/compared-to.mdwn deleted file mode 100644 index 94df452..0000000 --- a/faq/compared-to.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!meta title="How does Obnam compare to ...?"]] - -Speaking as Lars, the main Obnam developer: If someone wants to write -(and, preferably, maintain) a comparison between various backup -programs, please do. I'll add a link to it here. However, I do not -want to do that, as it's a lot of work to do well, and it's unfair to -do it badly. diff --git a/faq/dedup.mdwn b/faq/dedup.mdwn deleted file mode 100644 index 70efc49..0000000 --- a/faq/dedup.mdwn +++ /dev/null @@ -1,68 +0,0 @@ -[[!meta title="De-duplication does not work very well for me"]] - -On some kinds of large files, Obnam's de-duplication does not work -very well, even though it should. For example, MySQL dump files -from successive days are mostly the same data, but Obnam does badly -with them. Below is an explanation of how the Obnam de-duplication -works, and why it works badly in some cases. - -Obnam does de-duplication by splitting up file data into chunks, -and storing those individually. If two files have the same data, -Obnam re-uses the already backed up chunk. So far, so good. However, -due to performance issues, Obnam currently only notices chunks when -they start at integer multiples of the chunk size. - -For example, assume a chunk size of 4 bytes, and the following two -files: - - file 1: AAAABBBBCCCC - file 2: BBBBCCCCAAAA - -In this case, Obnam will easily notice that there are three chunks -("AAAA", "BBBB", and "CCCC"), and will store them only once in the -backup repository. However, consider the following file: - - file 3: xAAAABBBBCCCC - -File 3 is identical to file 1, except that a new byte has been -inserted into the file. This makes Obnam look at file 3 as four -chunks: "xAAA", "ABBB", "BCCC", and "C". None of these chunks -match the chunks already in the backup repository. Thus, Obnam -thinks they're all new. - -There is no technical reason why Obnam could not notice that file 3 -only has one inserted byte. However, doing so would require a very -large number of lookups in the repository, and thus would be quite -slow. There may be better ways of noticing the minute difference, -and perhaps someday one of them will be implemented in Obnam. - -Note that Obnam does not do a "diff" (or "xdelta" or other such -approach) to notice differences between successive versions of -files. Doing so would make backup generations be dependent on each -other, and re-introduce "full" versus "incremental" backups in a -way that is not acceptable. - -With SQL dumps of databases, there are often small changes at -the beginning of of the file, or in the middle of the file, which -makes Obnam's de-duplication work very badly, even if the data as -such has only changed a tiny bit. - -Unfortunately, I don't know of a trick that would make the SQL -dumps work better with Obnam. In any case, you should not have -to munge your live data to suit Obnam: Obnam needs to be able -to deal with whatever data you have. Until Obnam's de-duplication -becomes better, though, perhaps someone would have a workaround? - -The best idea, untested, I have is to keep the first SQL dump, -in the live data, and then do a new dump before each backup, diff -the two dumps, delete the new dump, and then run the backup. This -way, each successive Obnam backup generation will have two files -(the original SQL dump, and the diff), and you'll need to apply -the diff to get the real dump you need to restore your database. -Does that make sense to anyone? - -See also the mailing list thread: - -* [Start](https://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2013-January/002008.html) -* [Lars's explanation](https://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2013-January/002009.html) -* [Workaround with rdiff](https://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2013-January/002014.html) diff --git a/faq/forget-missing-chunk.mdwn b/faq/forget-missing-chunk.mdwn deleted file mode 100644 index eef0510..0000000 --- a/faq/forget-missing-chunk.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!meta title="Missing chunk error with 'obnam forget'"]] - -Obnam had a bug in which "obnam forget" would remove a chunk before -removing references to the chunk. If the "obnam forget" was -interrupted, the repository would have references to chunks that had -already been removed. This would cause a later run of "obnam forget" -to try to remove a chunk that no longer exists, and that would cause a -crash. - -The crashing was fixed in Obnam version 1.13. As of that version, -"obnam forget" will ignore that a chunk is missing if it's removing -that chunk anyway. - -There remains a problem, unfixed as of Obnam 1.16, where the chunk -still exists in lookup indexes, and "obnam backup" may try to use the -chunk. diff --git a/faq/is-encrypted.mdwn b/faq/is-encrypted.mdwn deleted file mode 100644 index 4568168..0000000 --- a/faq/is-encrypted.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!meta title="How can I verify that my repository is encrypted?"]] - -If the repository contains the files `clientlist/key` and -`clientlist/userkeys`, it's encrypted. - -The files `key` and `clientlist` are in every toplevel directory in -the repository, except the one called `metadata`. diff --git a/faq/list-archive-mess.mdwn b/faq/list-archive-mess.mdwn deleted file mode 100644 index 6169191..0000000 --- a/faq/list-archive-mess.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -[[!meta title="Why are messages in the list archive encrypted?"]] - -The Obnam mailing list archives sometimes have messages that look like -they're encrypted. They aren't, it's actually just a base64 encoding, -so it's easy enough to undo. The base64 encoding in the archives -happens due to some bug in the mailing list software, which the people -hosting the lists haven't been able to fix. It will hopefully be fixed -in a future upgrade to a new version of Mailman. - -Meanwhile, [Gmane]() provides a much nicer interface to the mailing -list than Mailman's archives, and doesn't suffer from this problem. -Gmane is highly recommended. See the -<http://dir.gmane.org/gmane.comp.sysutils.backup.obnam> group there -for the Obnam support list. - -[Gmane]: http://gmane.org/ - diff --git a/faq/missing-node.mdwn b/faq/missing-node.mdwn deleted file mode 100644 index 808b4f2..0000000 --- a/faq/missing-node.mdwn +++ /dev/null @@ -1,42 +0,0 @@ -[[!meta title="Missing node or KeyError problems"]] - -Obnam prior to version 1.6.1, and [Larch][] prior to version -1.20131130, had serious, but somewhat rare bugs that could result in -an error that said "Missing node" or "KeyError". These bugs would -cause a corruption in the backup repository by either causing a file -to be deleted, when it shouldn't have been, or the reference count for -a B-tree node to be wrong. As a result, Obnam would become unable to -make a backup or, in some cases, to restore everything from a backup -(some files could be restored). - -The corruption is not necessarily noticeable on all backup runs. It -depends on the exact files that get modified during a backup -generation. Worse, `obnam fsck` isn't able to fix the corruption, -though it would find it. - -As a result, a corrupted repository may be unrecoverable. Sometimes -you can recover at least some of the repository, with some loss of -data: - -* If the corruption is only in a "per-client B-tree", recovery can - happen by removing or renaming that B-tree, but that results in all - backup generations for that client being lost. However, file content - will remain in the repository, so a fresh full backup should be - reasonably fast. The per-client B-tree is a directory that has a - large number as its name. The directory needs to be removed manully, - not using Obnam. Instead of removing it, you can also rename it. - -* If the corruption is in the `chunksums` or `chunklist` B-trees, you - can delete or rename them both. The next backup run will create new - ones. This will reduce the possibility of de-duplication, but - otherwise everything should work. - -If the corruption is any other B-tree, no recovery is possible. - -When in doubt, avoid using a repository that has ever been used with -Obnam prior to version 1.6.1, or Larch prior to version 1.20131130. -Instead, start over with a new, empty repository. - -Lars apologises for this. - -[Larch]: http://liw.fi/larch/ diff --git a/faq/name.mdwn b/faq/name.mdwn deleted file mode 100644 index c20a0b8..0000000 --- a/faq/name.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!meta title="What does the name 'Obnam' mean?"]] - -Back in 2006, when Obnam started, we wanted to have a name that was -short, reasonably pronounceable by major cultures, and did not have -too many Google hits. After much trying, we failed to come up with -anything good. However, without a name, you can't even start the -project, because what will you call its source directory? So we -chose the OBligatory NAMe. - diff --git a/faq/private-key-for-backup.mdwn b/faq/private-key-for-backup.mdwn deleted file mode 100644 index 5651f73..0000000 --- a/faq/private-key-for-backup.mdwn +++ /dev/null @@ -1,25 +0,0 @@ -[[!meta title="Why does Obnam want my PGP private key passphrase?"]] - -When doing an incremental backup, Obnam reads metadata back from the backup -repository to determine what it needs to back up. For example, names of -files and when they were last modified. The metadata is also encrypted, -and Obnam needs to decrypt it to be able to do an incremental backup. -That is why Obnam needs the passphrase. - -Depending on how your GnuPG and its related agent is configured, you -may need to type in the passphrase multiple times during a backup run. -This is because the agent may expire the passphrase: it will remember -it for, say, five minutes or an hour after you enter the passphrase, -but after that you may need to enter the passphrase again. This can be -awkward, and if you're not around to enter the passphrase, the backup -may be terminated in the middle. - -There's two ways around that: you can either configure your GnuPG -agent to remember the passphrase for a longer time, possibly -indefinitely, or you can use a private key without a passphrase. -Neither is unproblematic from a security point of view. - -In any case, it's not something that Obnam is part of. Obnam only runs -gpg, and if gpg talks to its agent, which asks for a passphrase, or -not, depending on the configuration. There's nothing Obnam can do to -affect this. diff --git a/faq/profiling.mdwn b/faq/profiling.mdwn deleted file mode 100644 index 2732708..0000000 --- a/faq/profiling.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!meta title="How can I profile obnam if things get slow?"]] - -If the `OBNAM_PROFILE` environment variable contiains a file name, the profiling data gets stored there, and can be viewed later with `obnam-viewprof`: - - OBNAM_PROFILE=obnam.prof obnam ... - obnam-viewprof obnam.prof | less - -Performance issues that are not related to a particular setup can also be observed using the [[obnam-benchmark|obnam-benchmark.1.txt]] tool. diff --git a/faq/restore-on-other-client.mdwn b/faq/restore-on-other-client.mdwn deleted file mode 100644 index 8bee78e..0000000 --- a/faq/restore-on-other-client.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -[[!meta title="How do I restore on another computer?"]] - -Use `--client-name` and set it to the name of the computer that made -the original backup. - diff --git a/faq/safe-repo-sharing.mdwn b/faq/safe-repo-sharing.mdwn deleted file mode 100644 index 9bbbef9..0000000 --- a/faq/safe-repo-sharing.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!meta title="Can Alice and Bob share repository without sharing file data?"]] - -This doesn't work particularly well. Obnam does not currently support it -at all. On a conceptual level, it could work, by turning all de-duplication -off, and having Alice and Bob encrypt their own chunks only with their -own, separate keys. - -However, it would be simpler to just have a separate repository for -each of them. That would also prevent either of them to remove the -other's files. - diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn deleted file mode 100644 index bb2b5de..0000000 --- a/faq/tuning.mdwn +++ /dev/null @@ -1,109 +0,0 @@ -[[!meta title="Performance tuning"]] - -Introduction -============ - -Obnam has a number of options for performance tuning. See the -[[manual page|obnam.1.txt]] for all the details. Below is an adapted -excerpt of e-mails written by Lionel Bouton of how to test various -values to find a good set for your situation. See the list archive for -the e-mails: [first] and [second]. - -[first]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html -[second]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003090.html - -Measurements -============ - -Tuning `lru-size` and/or `upload-queue-size` can make a significant -difference in performance. - -Here follows some test results for this situation: - -- Data to backup stored on a btrfs volume on SSD: ~155000 files, 3.66GiB. - -- Local system: 64 bit Linux, Python 2.7.5, OpenSSH 6.6p1 with hpn patches. - -- Local CPU: Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz (mostly idle). - -- Remote system: 64bit Linux, OpenSSH 6.6p1 with hpn patches, - repository data on ext4 on standard SATA 7200rpm disk, large memory - (everything should fit in memory, only writes should hit disk). - -- Very minimal changes in the data backed up during tests so - successive backups only check for differences and backup transfers - nearly no content. - -- Backup over WiFi (~1ms RTT, max speed over sftp ~3MB/s). - -I use this command line without any configuration file: - - obnam -r sftp://obnam@SERVER/~/repo --compress-with=deflate \ - --client-name=CLIENT backup DIR - -During testing I added `--lru-size <l> --upload-queue-size <q>` with -different `<l>` and `<q>` values. - -The resident memory of the Obnam process grows steadily (probably -filling caches) until it hits a pretty stable ceiling (cache full or -nothing new to put in cache) during the backup. It raises again -rapidly at the very end (during commits/unlock/...). The value -reported below is obtained either through the RES column reported by -the htop utility or the RSS column reported by "ps aux" and is the max -witnessed near the end of the backup. - -Each combination was tested at least twice unless it was considered -not interesting after the first run. Timing seems consistent enough -given the systems involved (the system hosting the repository is often -busy) and memory usage is very consistent across runs. - -Default values as fetched from `__init__.py` are: l=256, q=128. - - | Conditions | Time | Memory |Number of runs | - +--------------------+-----------------+------------+----------------+ - | default values | 22m21s - 24m51s | ~260M | 2 | - | l=10000, q=default | 13m45s - 15m03s | ~332M | 2 | - | l=default, q=250 | 08m23s - 10m29s | ~278M | 5 | - | l=default, q=350 | 02m42s - 02m49s | 272-276M | 2 | - | l=default, q=400 | 02m13s - 02m18s | 268-272M | 3 | - | l=default, q=500 | 02m10s - 02m16s | 267-272M | 3 | - | l=default, q=512 | 02m13s - 02m14s | 265-269M | 2 | - | l=512, q=512 | 01m55s - 02m06s | 322-326M | 3 | - | l=768, q=512 | 01m55s - 01m58s | 397-418M | 3 | - | l=1024, q=512 | 01m53s - 01m55s | 403-418M | 3 | - | l=2048, q=512 | 01m55s - 01m59s | 408-410M | 3 | - | l=4096, q=512 | ~01m58s | ~419M | 1 | - | l=default, q=600 | 02m14s - 02m26s | 269-272M | 4 | - | l=default, q=750 | 02m13s - 02m15s | 266-272M | 2 | - | l=default, q=1000 | 02m19s - 02m20s | ~266M | 2 | - | l=default, q=10000 | 02m23s - 02m35s | ~266M | 2 | - -So in my configuration, when nearly no data changes between backups, -`--lru-size=1024 --upload-queue-size=512` is at least 11x faster than the -default configuration. - -Discussion -========== - -`--upload-queue-size` seems to have the greatest effect without any -adverse effect (memory usage remains at the same level). -For a little extra boost with a small impact on memory usage, I can -increase `--lru-size` to 1024. - -Note that Obnam was using 100% of the CPU for most of the time in the -fastest configuration, replacing `--verbose` with `--quiet` didn't -change the running time. - -Please note that the ideal settings for my backup configuration might -differ from the ones for yours. You might get even better results after -tuning of your own. - -These parameters have a nice behaviour for tuning: `upload-queue-size` -doesn't seem to have much drawback if at all when being increased (it -begins to give signs of slowing down obnam at 10000 here but it might be -the performance variance inherent in my configuration) and increasing -`lru-size` only increases memory usage a bit without slowing things -noticeably after reaching the ideal spot. - -A good rule of thumb seems to try increasing one of these parameters by -2x or 4x and keep going at it until performance stops increasing. diff --git a/faq/website-changes.mdwn b/faq/website-changes.mdwn deleted file mode 100644 index 0f9f1ad..0000000 --- a/faq/website-changes.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -[[!meta title="How do I suggest changes to the website?"]] - -If you want to suggest changes to the website: - -* E-mail `obnam-dev@lists.pepperfish.net` with the suggestion. -* If you can, see <http://obnam.org/ikiwiki.cgi?do=branchable> and - git clone the site, then send a patch. - -Thanks! diff --git a/faq/why-nonstd-format.mdwn b/faq/why-nonstd-format.mdwn deleted file mode 100644 index ee07440..0000000 --- a/faq/why-nonstd-format.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -[[!meta title="Why is Obnam not using standard file formats, such as tar or cpio?"]] - -The standard file formats do not support the kind of advanced features -that Obnam is built for. For example, block level de-duplication, -or abolishing the difference between full and incremental generations. -It would be possible to build those features on top, say, tar, but -it would mean the tar is just a container for odd bits of data, which -still requires a custom program to restore from. - -If you're worried that you might lose your copy of Obnam, and be unable -to restore, two points: - -* You can store a copy of Obnam (in source and binary versions) in your - backup repository. -* Obnam is free software, and it's in Debian. You're pretty much guaranteed - to be able to find a copy and build a new binary. - diff --git a/faq/why-not-on-github.mdwn b/faq/why-not-on-github.mdwn deleted file mode 100644 index 6eaa707..0000000 --- a/faq/why-not-on-github.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!meta title="Why is Obnam not hosted on Github or other such service?"]] - -Github is a very nice service in many ways, but it is a proprietary -service, running (at least partly) on non-free software. Obnam is free -software, primarily developed by a free software developer, and -hosting it on a proprietary service would be unethical. - -Services such as Github often provide many things on top of mere code -hosting, such as ticketing systems. The services might be very easy to -use. It is true that Obnam, which is self-hosted by its main -developer, could benefit from some of those things. None of this takes -away the ethical dilemma of using proprietary software. - -For the difference between open source and free software, see for -example -[Software freedom: an introduction](http://yakking.branchable.com/archives/2013/08/). diff --git a/format-6.mdwn b/format-6.mdwn deleted file mode 100644 index 7acfe0f..0000000 --- a/format-6.mdwn +++ /dev/null @@ -1,403 +0,0 @@ -[[!meta title="Repository format 6"]] - -NOTE: This was originally written as the sole description of Obnam's -on-disk repository storage. At the time, there was no other repository -format, and so this text doesn't separate that which is generic from -that which is specific. This page needs to be updated to discuss the -specific parts only. For the generic parts, see [[ondisk]]. - -[[!toc startlevel=2 levels=2]] - -Obnam is a backup tool for making **backups to hard disks,** either -local ones or over the network. It does not work with tape drives, -or optical disks. The **backup server is stupid:** it does -not run any Obnam specific code, only ssh (sftp). - -The location where -backups are stored is called the **backup store**, and may be shared between -multiple co-operating clients. The backup store resides on a backup server. The -computers that own the data being backed up are the **clients**. - -Performance is obviously a consideration. It has many aspects: - -* run-time -* RAM -* disk space, both server and client -* disk bandwidth, both server and client -* network bandwidth, between server and client - -To achieve its performance goals, -Obnam uses the B-tree variant devised by Ohad Rodeh[1], also -used by btrfs. This document is describes how it does that. - - -Characteristics of the B-tree ------------------------------ - -The **Rodeh B-trees** are designed for shadowing, i.e., -**copy-on-write updates** -of tree nodes. Nodes are not modified, instead a new copy is created -to replace it. This allows an efficient way to copy a tree and to update -the copy, while keeping the original tree intact. -The two trees will share as many nodes as possible. -Obnam calls the collection of related trees a **forest.** - -The trees use **fixed-length binary strings as keys.** -For each key, a **variable-length binary string is stored as the value.** -The tree consists of interior nodes and leaf -nodes; all values are stored in leaves only. There is a maximum size for -nodes, which in some applications would be a raw disk block, but is -a separate file for Obnam. -Value strings can be as big as a node (minus a small header), but no bigger. - -Lookups and removals from the tree can be done using **ranges:** all keys -within a range are looked up, or removed from the tree. This is an important -optimization and opens up some interesting design possibilities. - - -File data (chunk) storage -------------------------- - -File data is not stored directly in B-trees, but externally. Data -is broken up into **chunks** of suitable size (say, 64 KiB) -and assigned identifiers, -which also act as filenames. Given a chunk id, its contents can be easily -retrieved. - -**Chunks are shared** within and across clients. -This allows clients to avoid -uploading data that another client has already uploaded, -or storing the same data twice. The same mechanism allows Obnam to -efficiently back up files that have moved, or that have been modified, -or both at the same time. -If a chunk of data already exists in the store, Obnam tries hard to avoid -having to back it up again. - -To back up a chunk, Obnam chooses a random, unused 64-bit identifier, -and uploads a file of that name. The next chunk uses the next -identifier, until it hits one that has been used, at which point -it picks a new random one. - -Chunks are managed using reference counts. We will cover these later, -after discussing other details of the store. See -section "Chunks, part 2" below. - -Overview of store forests -------------------------- - -The backup store consists of several forests: - -* client list: map client names to 64-bit identifiers -* chunk list: map chunk identifiers to checksums -* chunk checksums: map chunk checksum to identifiers -* per-client data: backup generations for each client - -Locking -------- - -Locking is done only for writes. -Reads are always allowed, even while write locks exist. -This allows race conditions between readers and writers, -but thanks to copy-on-write updates -those are no different from files getting corrupted or -deleted on the server by hardware failures, -and can be treated the same. - -Each forest is locked separately. -If more than one forest needs to be locked at the same time, -they are locked in an order sorted by the name of the forest, -using the C locale. -If a client fails to lock a particular forest, -it releases all the locks it holds, -and tries again. -This avoids deadlocks, -but allows starving. -The most likely reason for starving is too many clients sharing the -same store, and that needs to be solved by increasing backup server -performance, or having more stores. - -Locks are implemented as files, which are created atomically. -Each lock file contains the name of the host that holds it -(which might not be a backup client), -and the process id on that client, and the time of creating the lock. -If the time is very old, another client may decide to break the lock. -If the backup store is encryped, then also the contents of the -lock file is encrypted. - -To reduce lock congestion, each client attempts to keep a lock for as -short a time as possible. For per-client data, this means keeping the lock -for the duration of the backup. For shared forests, updates can be -spooled: the shared forest is used read-only until the end of the backup -run, or until a checkpoint, and updated then, as quickly as possible. - -Client list forest ----------------- - -The client list forest consists of a single tree. -Its key is consists of a tuple: - -* 128 bits of MD5 checksum of the name of the client -* 64 bits of client id (for hash collision avoidance) - -The client name is typically its host name, but might be anything. -It is up to the sysadmins of the clients to co-operate so that each -client has a unique name. - -It is unlikely that there will be checksum collisions for client names, -but it's easy to not have to worry about that. The client id will be -chosen randomly. - -Chunk list forest ------------------ - -The chunk list forest consists of a single tree. -Its key consists of a 64-bit integer containing a chunk identifier. -The value is the MD5 checksum for the chunk. - -This tree is needed so that it is possible to quickly find the -checksum of a chunk, given its identifier, when removing generations -from the per-client forest. - -Chunk checksum forest ---------------------- - -The chunk checksum forest uses a tuple as a key: - -* 128-bit MD5 checksum of the data in the chunk -* 64-bit chunk id -* 64-bit client id - -The chunk id is used to handle checksum collisions. -While MD5 collisions are not likely in general use, -they are certain for people who research cryptographic hashes, -so Obnam needs to be able to handle them. -When Obnam has a chunk it wants to back up, -it does a range lookup from (md5,0) through (md5,chunkd_id_max), -and then checks if the chunk is identical to any of the chunks -already in the store. -This checking is expensive, but safety may more important than -performance here. (Obnam may provide an option to disable the -data comparison check, and rely only on the hashes.) - -The value is empty. - -Per-client forest ---------------- - -The per-client forest has one tree per backup generation or checkpoint, -and represents a snapshot of the filesystem at a specific time, -or as close to a snapshot as Obnam can make it. Obnam does not -freeze the filesystem while the backup runs, so it cannot guarantee -a true snapshot. - -The forest uses a tuple as a key: - -* 8-bit prefix -* 64-bit main key -* 8-bit subkey type -* 64-bit subkey - -The prefix is one of the following: - -* 0 for filesystem metadata -* 1 for chunk references -* 2 for generation metadata - -**For prefix 0,** the main key is the file identifier. -See below for how they are generated. - -The subkey type and subkey and value are using as follows: - -[[!table data=""" -subkey type | subkey | value -0 | file-id | full pathname to file -1 | counter 0... | up to N chunk-ids -2 | 0 | number of chunk-ids stored -3 | 0... | inode fields, user/group name, xattr, ... -"""]] - -Subkey type 0 is used to handle hash collisions for the pathnames. -They are unlikely, but it is entirely unacceptable for Obnam to fail -if they happen. Each file has an identifier that is unique within the -generation: there is a one-to-one mapping between fully qualified pathnames -and file-ids within a generation (but not across generations). - -By default, the file-id is one half of -the 128-bit MD5 of the full pathname to the file. To check for collisions, -we see if the value for key (1, default-file-id, 0, file-id) is the full -pathname for the file we are interested in. If it is not, we generate a -new file-id by some deterministic approach, and do the lookup again, -and repeat this until we find a free file-id. -Note that the default-file-id in the lookup stays the same, it will -always be the half of the MD5 of the pathname. - -Subkey type 1 is used to store a list of chunks that represent the contents -of the file. Since a file can be quite large, we store more than one chunk-id -per value, to reduce the overhead a bit. Benchmarking will determine the -suitable number of chunk-ids to put in each value, but probably the size -of the value should not be larger than about a quarter of a node. -To avoid having to do a range lookup to find the next counter, when -appending new chunk ids to an existing list, we store the number of items -at subkey type 2 (subkey 0). - -Subkey type 2 is used for file metadata, such as inode data -(`st_mtime`, `st_size`, etc), the name of the owner and group for -the file, xattr fields, etc. To reduce space use, the most common -metadata is stored in subkey value 0, encoded in a special format. -Other subkey values are used for additional metadata, which also -allows for future expansion. - -**For prefix 1,** the rest of the key is used like this: - -* main key is the chunk-id -* subkey type is 0 -* subkey is file-id for file that uses the chunk - -For prefix 1, the value is always empty. Whenever Obnam backs up a -file, it adds the key (1, chunk-id, 0, file-id). If the file existed -in the previous generation, but not in the new generation, it removes -the key. - -**For prefix 2,** the main key is fixed as the hash of -the string "generation", and the subkey type is fixed at 0. -The subkey is one of the values in the following table. - -[[!table data=""" -subkey |value -0 |generation id -1 |timestamp for the start of the generation -2 |timestamp for the end of the generation -3 |boolean (0 or 1) for whether the generation is a checkpoint -"""]] - -Timestamps are expressed in UTC as seconds since the beginning of 1970 -(i.e, Unix epoch). - -Chunks, part 2 --------------- - -When backing up, Obnam will do roughly this: - -* clone the tree for previous generation -* traverse the filesystem tree, adding and removing files from the new tree -* when backing up a file's chunks, look up each chunk in the store, and add it - if it is missing; after this, the id of the chunk is known -* add (1, chunk-id, 0, file-id) to the per-client forest, - (chunk-id) to the chunk list forest, - and (checksum, chunk-id, client-id) to the chunk checksum forest - for any new chunks it uses; the updates to the chunk list and checksum - forests can be batched to the end of the backup run -* remove (1, chunk-id, 0, file-id) from the per-client forest, - for every chunk for any file it removes from the new generation - (no need to remove anything from the chunk checksum forest, since - previous generations will still use the chunks) - -When removing a generation, Obnam will do roughly this: - -* look up every chunk-id used in that generation -* look up each chunk-id in every other generation -* if not found in any other generation, remove - (checksum, chunk-id, client-id) from the chunk checksum forest; - look up the checksum from the chunk list forest -* if there are no keys in the range (checksum, chunk-id, 0) through - (checksum, chunk-id, client_id_max) in the chunk checksum forest, - then remove (chunk-id) from chunk list forest, and - chunk file from disk -* remove the tree for the unwanted generation - - -Repository metadata -------------------- - -There is a need to store some metadata about the repository. -This will be done in the `metadata` directory. Initially, the -only piece of information is the version of the on-disk format, -expressed as a single integer, stored in the file `metadata/format`. - -Each version of Obnam will know which on-disk formats it supports. -With the `metadata/format` file it knows if it can handle a -particular format. - -Upgrades to newer formats will happen explicitly, not implicitly. - - -Encryption ----------- - -Obnam may optionally encrypt data in the backup store. -The design for this is unclear, but will need to solve at least -the following problems: - -* each client should have access only to its own data, but still - allow sharing of data across clients -* the server admin should be able to disable access for each client, - or anyone who's compromised the client's encryption key -* the server admin should be able to do some operations on any client, - such as expire generations, remove entire clients and their corresponding - chunks, verify correctness of the whole data store ("fsck"), or - restore data after the client has been entirely lost - - -On checksum algorithms ----------------------- - -This design uses MD5 checksums for everything. -It does not require the checksum algorithm to be cryptographically -secure, and is content with having a hash that is unlikely to result -in collisions, and only cares about the probability of collisions -for performance, not correctness. -The design handles collisions, and does not assume two pieces of data -are identical just because the hashes are the same. -A shorter or faster checksum algorithm may be used instead. -Or a cryptographically stronger one. -MD5 is not recommended for applications that rely on the security of -the hash, but this design does not rely on that. - -However, it may be quite expensive to verify that data chunks with -the same checksum are the same. It may require reading back the chunk -from the server. This may not be acceptable to all users. Some users -may be willing to live with the chance of a checksum collision. -Other users might not want to pay the price for de-duplication. -Thus, it may be best for obnam to provide a tri-state option: -full verification, rely on checksums, and no de-duplication. Further, -it may be a good idea to allow the user to specify the checksum -algorithm. - - -Comments? ---------- - -Edit this page directly, or mail me at <mailto:liw@liw.fi>. Thanks. - - -Author ------- - -Obnam is being developed by [Lars Wirzenius](http://liw.fi). -Feedback and patches are most welcome. - - -References ----------- - -1. Rodeh, Ohad: "B-trees, Shadowing, and Clones". IBM Haifa Research Labs. - <http://portal.acm.org/citation.cfm?id=1326544> - <http://www.cs.tau.ac.il/~ohadrode/papers/btree_TOS.pdf> - - -Changes -------- - -* 2010-05-04: Modified the way hash collisions for filenames are - handled, based on suggestion from Richard Braakman: store secondary - hash and filename with primary hash. -* 2010-07-11: Minor tweaks. -* 2010-11-11: Change tense to current, add some notes and minor fixes. -* 2010-11-11: Add design for removal of chunks, some considerations for - encryption, more details on how keys are used by various forests in - the store, locking, and other clarifications. -* 2010-11-11: Refer to hosts as clients, for clarity. -* 2010-11-12: Typo fixes and checksum verification tri-state option - suggestion from Richard Braakman. -* 2010-11-18: Add generation metadata to per-client forest. -* 2010-11-29: Add chunk list forest. diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn deleted file mode 100644 index b2ef215..0000000 --- a/format-green-albatross.mdwn +++ /dev/null @@ -1,257 +0,0 @@ -[[!meta title="Repository format 'Green Albatross'"]] - -This page describes the repository format called 'Green Albatross'. It -is a preliminary format, intended to improve Obnam performance over -[[format-6]]. Current development status is PONDERING; -implementatation has started, but everything on this -page may and probably will change. Until declared stable, the on-disk -data structured WILL change without warning. Only use this format to -help with its development. - - -Introduction -============ - -For background and the big picture, please read the [[ondisk]] page. -This page only discusses details specific to this format. - -This repository format takes the following approach: - -* Don't use the [Larch](http://liw.fi/larch/) B-tree implementation. - It's intricate code and insufficiently fast. Implement any data - structures directly. -* With few, specific exceptions, repository files are never - updated. Everything is done copy-on-write. This enables caching. The - exceptions are root nodes of DAGs, so that it's easy to know where - the DAG starts. -* Data is stored in objects of various types. Objects may be small, - and to avoid having a file per object, objects are collected into - bags. This includes chunks. Each bag is stored in its own file. - Objects are identified by the bag identifier, plus an index into the - bag. -* Objects are stored in a suitable serialisation encoding. - - This is not Python pickles, since Obnam can't assume those are - stable over the lifetime of a backup repository. - - JSON is not used, since JSON is not suitable for storing binary - data, such as filenames, without adding an encoding layer on top - of JSON. - - Previously this was specified as YAML, but YAML is not fast to - parse. So it's not YAML. - - Instead, a simple, custom binary encoding will be used. This - will encode ints, booleans, binary strings, or lists or dicts - (dict keys being lists). A quick prototype shows this to be easy - (worked the first time), and fairly fast even without - optimisation. -* Encryption is done exactly like in [[format-6]]. - - -Bags -==== - -A bag is used to store a number of blobs. Bags are identified by a -random 64-bit integer. This is used as the filename of the bag. The -bags are stored in a 3-level directory structure, using the top three -octets of the bag id as the directory names. Thus, a bag whose id in -hexadecimal is 0x1234567890abcdef would be stored as -`12/34/56/1234567890abcdef.bag`. - -A bag is implemented as a Python `dict` object: - - { - 'bag-id': 'cafef00d', - 'blobs': [...], - } - -The `items` field contains the blobs. Each blob may be an arbitrary -byte string (for chunks), or an encoded Python object. - - -Object identifiers -================== - -Object identifiers are a pair consisting of the bag id and an index -into the bag. Since the identifiers are used frequently, it is -practical to store them as a unit rather than as a pair. Further, they -will be visible to the user (and, especially, the developer), so the -following syntax is used: - - BAGID.INDEX - -For example, the first and third objects stored in the bag with id -0x1234567890abcdef would be: - - 1234567890abcdef.0 - 1234567890abcdef.2 - -Note the use of hexadecimal for the bag id (so all bag identifiers are -of the same length), and indexing in decimal, starting from zero. - -We will keep bags effectively immutable so that an object id does not -need to change. This means that a bag may contain unused objects. If -it turns out that that's wasting too much data, we can "pack" bags by -replacing the unused blobs with empty values (Python's None) to save -space. This mutates a bag, but only in ways that (correct) users won't -notice. - - -Client list -=========== - -The client list is stored as `client-list/data.bag` in the repository, -and each item in the bag has the following structure: - - { - 'client-name': 'foo', - 'client-id': 123, - 'encryption-key': None, - } - - -Chunks -====== - -Chunks are stored in bags. The chunk data is just a binary blob. - - -Chunk indexes -============= - -Chunk indexes map a checksum (using the user's chosen algorithm) to a -list of chunk ids, and a chunk id to a list of client ids. The root -object of the indexes is: - - { - 'checksums': { - 'algorithm': 'SHA-1', - 'root-id': '1234567890abcdef.1', - }, - 'used-by-tree': '1234567890abcdef.0,, - } - - -Checksum to chunk ids ---------------------- - -The mapping from a checksum value to a list of chunk ids is done using -a lookup tree that is vaguely similar to a B-tree. The tree contains -index nodes and leaf nodes. Leaf nodes store the actual mappings: - - { - 'e242ed3bffccdf271b7fbaf34ed72d089537b42f': [ - '1234567890abcdef.3', - '1234567890abcdef.4', - ], - 'f1d2d2f924e986ac86fdf7b36c94bcdf32beec15': [ - '1234567890abcdef.5', - ], - } - -In other words, a leaf node is a `dict` mapping a checksum to a list -of chunk ids whose content has that checksum. Note that the list may -be longer than one item. - -An index node looks like this: - - [ - { - 'first-checksum': 'e242ed3bffccdf271b7fbaf34ed72d089537b42f', - 'last-checksum': 'f1d2d2f924e986ac86fdf7b36c94bcdf32beec15', - 'leaf-id': '1234567890abcdef.2', - 'index-id': None, - }, - ] - -In other words: - -* The index node is a list of mappings, where each mapping corresponds - to an object on the next level in the lookup tree. -* `first-checksum` is the smallest checksum in the sub-tree being - referenced. -* `last-checksum` is the largest checksum. -* `leaf-id` is the object id of the leaf node, assuming the next level - is leaf nodes. -* `index-id` is the object id of the index node, assuming the next - level is index nodes. - -The lookup tree is created in a copy-on-write manner. No node is ever -overwritten, but it may be deleted after it is no longer referenced. -The tree is not kept in balance, to keep the code maintaining as -simple as possible. - -When a new checksum is inserted into the lookup tree, it is added to -an existing leaf node by creating a new leaf node that is a copy of -the old one, and adding the new checksum to the new leaf. A big leaf -node is split in half. Any index nodes on the path to an updated leaf -node get updated. - - -Chunk id to client ids ----------------------- - -This tree is similar in structure as the checksum tree, but index nodes -look like this: - - [ - { - 'first-client-id': '1234567890abcdef.50', - 'last-client-id': '1234567890abcdef.51', - 'leaf-id': '1234567890abcdef.49', - }, - ] - -Leaf nodes look like this: - - { - '1234567890abcdef.32': [ 123, 456 ], - } - -Leaf nodes map a chunk id to a list of ids of clients that use the -chunk. - - - -Per-client data -=============== - -The data stored for each client separately starts with a CLIENT -object: - - { - 'client-name': 'foo', - 'generations': ['1234567890abcdef.77'], - } - -Each generation is a GEN object: - - { - 'started': '2015-04-06T17.35:41', - 'ended': '2015-04-06T17.35:42', - 'checkpoint': False, - 'root-dir': '1234567890abcdef.77', - } - -Each directory in the live data is stored in a DIR object. The object -stores the metadata for the directory, plus the basenames and metadata -for each file in the directory (anything thing that isn't a -subdirectory), plus the basename and DIR object id of each -subdirectory. - - { - 'metadata': { - 'st_mode': 0777, - 'st_uid': 0, - 'username': 'root', - }, - 'files': - 'README': - 'metadata': - 'st_mode': 0644, - 'st_uid': 0, - 'username': 'root' - 'subdirs': - '.git': '1234567890abcdef.127', - } - -Again, nothing is updated in-place, everything is updated -copy-on-write. When a node is updated, its parent all the way to the -root directory also get updated. diff --git a/liw-bitcoin-qr-obnam.png b/liw-bitcoin-qr-obnam.png Binary files differdeleted file mode 100644 index 528744c..0000000 --- a/liw-bitcoin-qr-obnam.png +++ /dev/null diff --git a/locking.mdwn b/locking.mdwn deleted file mode 100644 index bceafbd..0000000 --- a/locking.mdwn +++ /dev/null @@ -1,158 +0,0 @@ -[[!meta title="Locking in Obnam repositories"]] - -Obnam supports multiple clients backing up to the same repository at -the same time. It further allows the clients to share data, so that -if Alice has backed up a large file, and Bob has the same file, he -does not need to actually upload the file a second time to the backup -repository. See [[Obnam on-disk data structures|ondisk]] for more details. - -For Alice and Bob not to overwrite each others' changes to shared -data, there needs to be some locking, so that only one of them can -modify the data structures at the same time. - -The repository has six different classes of data: - -* metadata about the repository (shared, needs locking) - - currently this is the version number of the repository format -* a list of clients (shared, needs locking) -* file data chunks (shared, but does not need locking) -* encryption keys for chunks (shared, needs locking) -* shared metadata about file chunks (shared, needs locking) -* per-client file metadata (not shared, but does need locking) - -Another way to consider the data is as follows: - -* shared chunk storage -* shared and per-client B-trees, and chunk encryption keys -* repository metadata - -We can discuss the locking for each of these groups separately - -Shared chunk storage --------------------- - -The shared chunk storage consists of a directory tree, where chunks are -stored in files named after the chunk identifier. When a client is -making a backup, they only ever add chunks, and they do that by picking -a chunk identifier by random, and creating a file with that name. If -the file creation fails, the chunk id was already in use, and the client -picks a new one by random. - -This means that the filesystem takes care of atomicity, and Obnam does -not need to care about that. Adding chunks is a lock-free operation even -when multiple clients are backing up at the same time. - -Read-only access to the chunks also does not require any locking, -obviously. - -The operations that can delete chunks (forget, fsck) can also do without -locking the chunk storage. They lock the relevant B-trees, so that other -clients do not at the same time try to change the state of the repository, -and this protects the shared chunks as well. - -When encryption is used, the encryption keys need to be updated for new -clients, and this needs locking, but this is handled separately from the -actual chunks. - -Shared and per-client B-trees, and chunk encryption keys --------------------------------------------------------- - -There are several shared B-trees: the list of clients, the mapping of -chunk identifiers to chunk sums, and the mapping of chunk sums to -chunk ids. There is one B-tree per client. From a locking point of -view, they can be treated identically: if nothing else, even the -per-client tree can be considered shared, when the backup server is -doing a repository-wide fsck, for example. - -The encryption keys for chunk storage are not a B-tree, but they can be -handled as if they chunks directory is a B-tree, from a locking point of -view. - -Read-only access to the B-trees should not be locked. This is not entirely -safe: Alice may do read-only access (e.g., during a restore), while -Bob is doing a destructive operation (e.g., forget). However, the -read-only client can treat any failures due to concurrent access -the same way it treats other failures: it can try to restore as much -as it can, or it can give up. Locking does not help against bad sectors -on the repository. Not having to lock for read-only access will allow -faster access and avoids having a long-running read-only operation locking -out new backups. - -Destructive access will need locking. Thus, each B-tree will have a -lock file at its root (called 'lock'): existence of the file indicates -the tree is locked. Only one writer can have a lock at the same time. - -The client list B-tree only needs to be changed when a client is added -or removed from the tree, or its encryption details are changed. The -other shared B-trees need to be changed during every backup run of every -client. The per-client B-trees need to be locked during backup and forget -operations for that client. All B-trees will need to be locked while -an fsck runs. (Per-client B-trees need to be locked to protect against -concurrent backup, forget, and fsck operations, for example.) - -Each B-tree is locked separately, by creating a file called `lock` -inside its directory. - -Repository metadata -------------------- - -The repository metadata (the directory `metadata` at the root of -the repository, not to be confused with the similarly named files inside -B-tree directories), may need to be locked against writes. For example, -a future version of Obnam may provide a way to upgrade the repository -format to a new version, and this requires locking against any changes -to the repository. - -The repository metadata will be considered locked implicitly when the -client list B-tree is locked. - -Avoiding deadlocks and reducing lock contention on shared stuff ---------------------------------------------------------------- - -To avoid deadlocks, every client must create locks in the same order. - -1. The client list B-tree. -1. Any per-client B-trees, in the order of their directory basenames. -1. Any other shared B-trees, in the order of their directory basenames. - -(Directory basenames are ordered byte by byte, using unsigned bytes.) - -There are several scenarios in which a client may require a lock: - -* adding a new client - - lock client list, add new client to it - - create, then lock per-client B-tree - - lock the shared directories, then add the client's encryption key to them - - unlock everything -* making backups for a client - - lock per-client B-tree, add new metadata - - lock chunklist and chunksums B-trees, update with info about new chunks, - unlock (possibly do this several times, for checkpoints) - - unlock per-client B-tree -* forgetting backups for a client - - lock per-client B-tree, remove generations, gather list of chunks - that may need to be deleted, unlock - - lock chunklist and chunksums, update about chunks that aren't used - by the client anymore, possibly remove chunks from disk, unlock -* fsck for repository - - lock everything, fsck, unlock everything - -When making backups, a client will pool its updates to the shared B-trees -until it is ready to do all of them in a batch, typically at the end -of a backup, or at a checkpoint generation. This way, it can keep -the shared lock for only a short while. If the client crashes before -it can update the shared B-trees, it is unfortunate, but not -catastrophic: it will have stored some chunks in the repository that -cannot be used for de-duplication, but no actual data is lost. - -Reliability and scalability testing of locking ----------------------------------------------- - -The reliability of locking will be tested by writing a script to do -small backup generations, and running it concurrently many times -against the same repository. The script will verify that each client -can restore every generation successfully after all generations have -been backed up. Also, repository wide fsck will be run. - -The amount of data to back up in each generation shall vary for -each client, so that they are less likely to get into lockstep. diff --git a/ondisk.mdwn b/ondisk.mdwn deleted file mode 100644 index 1e1afca..0000000 --- a/ondisk.mdwn +++ /dev/null @@ -1,212 +0,0 @@ -[[!meta title="Obnam on-disk data structures"]] - -Introduction -============ - -This page gives a high abstraction level picture of what the Obnam -repository data structures are like, and the internal abstraction for -handling them. It then links to more detailed descriptions of each -different repository format. - -[[!toc levels=2]] - - -Constraints and assumptions -=========== - -The repository design is influenced by several constraints and -assumptions, some of which are described here. - -Disk-like storage ------------------ - -Storage is assumed to be disk-like, meaning any piece of data can be -accessed at about the same speed, rather than tape-like, where you'd -have to basically only append data, and where seeks are not practical. - -Shared storage --------------- - -Multiple backup clients may share storage, for lower cost or easier -administration. - -Dumb storage ------------- - -The most important of assumption is that the repository storage is -dumb: it can't do any processing of data. Basically we must assume the -repository provides only the following operations: - -* PUT some data in storage under a given filename, including a - hierarchical directory structure. This is **atomic**, meaning the - data is either completely available, or does not exist at all, under - the given name. Stored data may be replaced completely, but it can't - be updated only partly. PUT may optionally fail, if the name is - already in use. -* GET named data from storage. -* LIST all data in storage in a given directory. -* DELETE named data from storage. - -We can't, for example, request that the storage compute a checksum of -some data. We especially can't assume Obnam itself is running on the -machine providing the storage. - -Storage filesystem ------------------- - -The storage may be provided by any existing filesystem, including -VFAT, or it might not be provided by a real filesystem at all. We -can't use neat tricks like hard links to implement the repository. - -Round trip time ---------------- - -The storage may be a local filesystem, but it may also be access over -the network using some protocol such as SFTP. This means every storage -access potentially carries a large time overhead. Minimising the -number of separate accesses is necessary for good performance. - -Security and privacy --------- - -The repository design must assume an attacker has at least read-only -access to the repository. This means the design should avoid leaking -information via filenames, or other such things. Some data leak is -unavoidable: it is, for example, unavoidable that an attacker can keep -track of which files were changed when. - - -Repository structure -==================== - -The repository is divided into four kinds of areas: - -* A list of clients. -* Chunks of file content. -* Indexes to find chunks efficiently. -* Per-client data, such as what files each client has. - -These areas are mostly independent of each other. They refer to -objects using identifiers: clients and chunks have identifiers that -are random 64-bit integers, to avoid data leaks. - - -Client list ------------ - -Each backup client needs to add itself to the client list. The client -list maps the client name to the client identifier, and also lists the -client's encryption key. - - -Chunks of file content ----------------------- - -Files vary in size a lot, and thus Obnam breaks file content into -suitably small pieces, called chunks. Chunks can be re-used between -files and clients, for de-duplication. Chunks may be of variable size. -Chunks are accessed using their identifier only. - - -Chunk indexes -------------- - -For de-duplication, it is necessary to know if a given piece of file -content data is already stored in the repository. Chunk indexes -provide a mapping from the value of a checksum algorithm to a list of -chunk identifiers whose content has that checksum. - -Additionally, when removing backup generations (`obnam forget`), it is -necessary to know which clients are using a given chunk. Chunk indexes -also provide a mapping from a chunk identifier to a list of client -identifiers. - - -Per-client data ---------------- - -Each client has its own files, and manages its own backup history. For -each client the repository has its own area, where the client stores: - -* each backup generation -* all the names and metadata (`stat`(2) results, etc) for each file -* possibly some other data that is only relevant for that client - - -Feature implementation -====================== - -Some of the headline features of Obnam need to be implementable using -the repository design. This section describes how that happens. - - -De-duplication --------------- - -De-duplication is implemented by the backup process reading a file and -splitting the content up in chunks, using whatever chunking method it -chooses. It then looks up the chunk in the chunk indexes to see if the -chunk content is already in the repository. - -If there are chunks with the same checksum, the backup process can -then either decide to re-use the chunks, on the assumption that the -checksum is strong enough and that there are no collisions. -Alternatively, it can download each of the chunks from the repository -and compare the data bit by bit, to verify a match. The latter is -quite expensive in time and bandwidth, but necessary for those who -can't rely on checksums, such as those researching checksum -collisions. - -Encryption ----------- - -Storage is dumb, and so it doesn't encrypt itself, or at least Obnam -can't assume it does. Further, storage may be controlled by someone -else, such as an online storage provider, and while they may be -trusted to store data, they can't be trusted to not leak data. Thus, -Obnam encrypts data before putting it in storage (assuming the user -enables encryption). - -Encryption is done by combining symmetric and asymmetric (public key) -encryption. For each area (client list, chunks, chunk indexes, -per-client), a random symmetric encryption key is created, and all -other files in the area are encrypted using that key. The symmetric -key itself is encrypted using a list of public keys, which are -provided by the user. The public keys are stored in a file in the -area, and that file is encrypted using the symmetric key. - -This way, to access the public keys, you must be able to access the -symmetric key, and you can only do that if you have one of the public -keys. Or if you can break the encryption. - - -Internal implementation details -=============================== - -Internally, in the Obnam code base, all repository formats are -accessed using the same interface. This is so that Obnam can support -any number of repository formats without excessive code complexity. In -the code, this is in the `obnamlib/repo_interface.py` file. For -details, -[see that file](http://git.liw.fi/cgi-bin/cgit/cgit.cgi/obnam/tree/obnamlib/repo_interface.py). - -Additionally, all live data and repository storage is accessed using a -filesystem abstraction, called the VFS interface. This interface is -implemented separately for local filesystems and for SFTP servers, and -this enables Obnam to access live data as well over SFTP. In the -source tree, see `obnamlib/vfs.py` for the interface. Additional -storage providers can be added by implementing the interface for them. - - -Specific repository formats -=========================== - -Obnam implements several repository formats. Each format implements -the abstract features described above in some concrete way. Depending -on the quality of the implementation, the resulting format can work -better or worse. - -* [[Format **6**|format-6]] was introduced prior to version 1.0, and - is currently the main format. It is intended for real use. -* Format [[Green Albatross|format-green-albatross]] is an experimental - format, intended to eventually replace format 6. diff --git a/quotes.mdwn b/quotes.mdwn deleted file mode 100644 index 11f9f2c..0000000 --- a/quotes.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -[[!meta title="Quotes about Obnam"]] - -Some quotes about Obnam, from its users. - -"Even mhy can't find much bad to say about it." - -"Not only does it back up, but it restores too!" says relieved user Kinnison. -2014-01-18 IRC. diff --git a/roadmap-ga.mdwn b/roadmap-ga.mdwn deleted file mode 100644 index dfcbf60..0000000 --- a/roadmap-ga.mdwn +++ /dev/null @@ -1,83 +0,0 @@ -[[!meta title="DRAFT: Road map for FORMAT GREEN ALBATROSS"]] - -This is a DRAFT road map for the new repository format (GREEN -ALBATROSS) to become the default format in Obnam. Feedback on this -road map is welcome via the obnam-devel mailing list. - -Note that only things that affect the new repository format are -relevant for this road map. All other bugs or features are off-topic -and will not be included. - -Success criteria for being done with the new format -======================================================================== - -* All major operations need to be tolerably fast, and run in 4 GiB of - RAM. The test data set is to be the snapshot of almost 5 TiB data - from my own file server. The backup repository should use encryption - and compression. - - - backup - - forget - - restore (or possibly verify, to avoid needing another 5 TiB space) - - fsck - -* Obnam supports Attic-style chunking (lowest N bits of weak checksum - means chunk ends), and things are still tolerably fast. - -* The manual needs to be updated to cover all GA things and needs to - have a comparison between 6 and GA, and also advice for converting - to the new format ("start over" is sufficient, though a conversion - tool would be nice). - -* The obnam.org website needs to be reviewed, and any design docs - updated. - -* At least three months needs to have passed of actively asking Obnam - users to use GA, without showstopper bugs being reported. - -* No known need to change the new repository format. - - -Road map -======================================================================== - -This is the rough order of things that I know needs doing. There are -certainly things missing from the list, reality always wins over my -most careful planning. - -* Add new benchmarks for all the success criteria. All the operations - listed above should be benchmarked. Then analyze results and make - any optimizations needed. - -* Add Attic-style chunking. These chunks are not of fixed size at - fixed positions in the files. This matters to the repository format - because there may be lots more chunks, depending on settings, and - the format needs to handle that. - -* Add sparse file handling to GA. Sparse files are not used by - everyone, but those that do, really want them. Obnam currently - doesn't handle them optimally and has not way of representing them - in the repository except as long sequences of all-bits-zero bytes - -* Make sure Obnam handles the case of an unknown username or group - name of a file (only numeric uid or gid known). This is important - when it's not feasible to get the user/group name from an SFTP - server. This affects the repository format only a little, but it - needs to allow storing a value to represent "not known", and all - code needs to be to deal with that. - -* Implement in-process symmetric encryption. This is not directly - relevant for GA, but as it will not be backwards compatible with the - old way of doing encryption, I'm lumping it in with GA. Get all the - incompatibilities done at once. - -* Implement a reasonably fast fsck for GA. - -* Review obnam.org website and make any necessary updates. - -* Review Obnam manual and make any necessary updates. Co-ordinate with - translators to get non-English versions of the manual to also be - updated. - -* Make an Obnam release with a beta level version of GA, and ask - people to use it and report results. diff --git a/signing.mdwn b/signing.mdwn deleted file mode 100644 index c03f044..0000000 --- a/signing.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -[[!meta title="Signing backup data in Obnam"]] - -Obnam should sign data stored into the repository to protect against -malicious fake data, and against accidental corruption. - -The signing should be safe against replay attacks. - diff --git a/status.mdwn b/status.mdwn deleted file mode 100644 index 1bf2169..0000000 --- a/status.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Status and support ------- - -Obnam is retired. Don't use it anymore. diff --git a/sticker.png b/sticker.png Binary files differdeleted file mode 100644 index 8b343c8..0000000 --- a/sticker.png +++ /dev/null diff --git a/tag/obnam-1.0-blocker.mdwn b/tag/obnam-1.0-blocker.mdwn deleted file mode 100644 index 8113fa5..0000000 --- a/tag/obnam-1.0-blocker.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="pages tagged obnam-1.0-blocker"]] - -[[!inline pages="tagged(obnam-1.0-blocker)" actions="no" archive="yes" -feedshow=10]] diff --git a/tag/obnam-performance.mdwn b/tag/obnam-performance.mdwn deleted file mode 100644 index 22280f4..0000000 --- a/tag/obnam-performance.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="pages tagged obnam-performance"]] - -[[!inline pages="tagged(obnam-performance)" actions="no" archive="yes" -feedshow=10]] diff --git a/tag/obnam-wishlist.mdwn b/tag/obnam-wishlist.mdwn deleted file mode 100644 index 1396b20..0000000 --- a/tag/obnam-wishlist.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="pages tagged obnam-wishlist"]] - -[[!inline pages="tagged(obnam-wishlist)" actions="no" archive="yes" -feedshow=10]] diff --git a/tag/person.mdwn b/tag/person.mdwn deleted file mode 100644 index d3b7dfa..0000000 --- a/tag/person.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="pages tagged person"]] - -[[!inline pages="tagged(person)" actions="no" archive="yes" -feedshow=10]] diff --git a/tag/program.mdwn b/tag/program.mdwn deleted file mode 100644 index 49ec8e4..0000000 --- a/tag/program.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="pages tagged program"]] - -[[!inline pages="tagged(program)" actions="no" archive="yes" -feedshow=10]] diff --git a/tutorial.mdwn b/tutorial.mdwn deleted file mode 100644 index dd99e36..0000000 --- a/tutorial.mdwn +++ /dev/null @@ -1,158 +0,0 @@ -[[!meta title="Obnam tutorial"]] - -This tutorial will be migrating to the full Obnam manual, -at <http://code.liw.fi/obnam/manual/>. This version is no longer -updated, see the link for the current version. - -[[!toc ]] - -Installation ------------- - -It is easiest to install Obnam on a Debian system. If you're running -Debian `wheezy` or a later release, Obnam is included. For `squeeze` -add the following line to your `/etc/apt/sources.list` file: - - deb http://code.liw.fi/debian squeeze main - -Then run the following commands as root: - -* `apt-get update` -* `apt-get install obnam` - -The commands will complain that the PGP key used to sign the archive -is not known to apt. You can either ignore this, or add the key from -<http://code.liw.fi/apt.asc> to your key. - -For other systems, you need to install from sources. See the `README` -file for instructions. - -Configuration -------------- - -Obnam does not require a configuration file, and you can configure -everything using command line options. You can, however, use a -configuration file: save it as `~/.obnam.conf` and -make it have content like this: - - [config] - repository = sftp://your.server/home/youruser/backups/ - log = /home/liw/obnam.log - -The examples below assume you have created a configuration file, -so that options do not need to be repeated every time. - -You probably want to enable the `log` setting, so that if there is -a problem, you can find out all the information available to fix it -from the log file. - -Initial backup --------------- - -Your first backup will be pretty big, and will take a long time. -A long backup may crash, but that is not a problem: Obnam makes -a **checkpoint** every one hundred megabytes or so. - - obnam backup $HOME - -Incremental backups -------------------- - -When you've made your initial, full backup (possibly in stages), you can -back up any changes simply by running Obnam again: - - obnam backup $HOME - -This will back up all new files, and any changed files. It will also -record which files have been deleted since the previous backup. - -You can run Obnam as often as you like. Only the changes from the -previous run are backed up. - -Multiple clients in one repository ----------------------------------- - -You can backup multiple clients to a single repository by providing the -option --client-name=<identifier> when running the program. Backup sets -will be kept separate, but data deduplication will happen across all -the sets. - -Removing old generations ------------------------- - -Eventually your backup repository will grow so big you'll want to -remove some old generations. The Obnam operation is called forget: - - obnam forget --keep=30d - -This would keep one backup from each of the last thirty calendar -days, counting from the newest backup (not current time). -If you've backed up several times during a day, only the latest -generation from that day is kept. - -Any data that is part of a generation that is to be kept will -remain in the repository. Any data that exists only in those -generations that is to be forgotten gets removed. - -Restoring data --------------- - -You will hopefully never need this, but the whole point of having -backups is to restore data in case of a disaster. - - obnam restore --to=/var/tmp/my-recovery $HOME - -The above command will restore your entire home directory to -`/var/tmp/my-recovery`, from the latest backup generation. -If you only need some particular directory or file, you can -specify that instead: - - obnam restore --to=/var/tmp/my-recover $HOME/Archive/receipts - -If you can't remember the name of the file you need, use `obnam ls`: - - obnam ls > /var/tmp/my-recovery.list - -This will output the contents of the backup generation, in a format -similar to `ls -lAR`. Save it into a file and browse that. -(It's a fairly slow command, so it's comfortable to save to a file.) - -Using encryption ----------------- - -Obnam can use the GnuPG program to encrypt the backup. To enable -this, you need to have or create a PGP key, and then configure -Obnam to use it: - - [config] - encrypt-with = CAFEBABE - -Here, `CAFEBABE` is the **key identifier** for your key, -as reported by GnuPG. You need to have `gpg-agent` or equivalent -software configured, for now, because Obnam has no way to ask for -or configure the passphrase. - -After this, Obnam will automatically encrypt and decrypt data. - -Note that if you encrypt your backups, you'll want to back up your GPG -key in some other way. You can't restore any files from the obnam -backup without it, so you can't rely on the same obnam backup to back up -the GPG key itself. Back up your passphrase-encrypted GPG key somewhere -else, and make sure you have a passphrase strong enough to stand up to -offline brute-force attacks. Remember that if you lose access to your -GPG key, your entire backup becomes useless. - -If you enable encryption after making backups, you need to start over -with a new repository. -You can't mix encrypted and unencrypted backups in the same repository. - -(There are a bunch of Obnam commands for administering encryption. -You won't need them, unless you share the same repository with several -machines. In that case, you should read the manual page.) - -The End -------- - -Best of luck. - -See [[status]] for ways to get support, should you need anything. |
