diff options
| author | Lars Wirzenius <liw@liw.fi> | 2013-06-16 14:42:25 +0100 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2013-06-16 14:42:25 +0100 |
| commit | 0bd1244c1e75254407b0014add1c6cb29ef1b79c (patch) | |
| tree | 8d15eaef52a6f3dbeef67eb29376d91a29190378 | |
| parent | f5cb4c8018f3d4712d031dc9696f15afa747a99c (diff) | |
| download | obnam-0bd1244c1e75254407b0014add1c6cb29ef1b79c.tar.gz | |
Remove whitespace from ends of lines
127 files changed, 1128 insertions, 1128 deletions
@@ -127,7 +127,7 @@ Version 1.2, released 2012-10-06 Bug fixes: * Notify user of errors during backups. -* The SFTP plugin now manages to deal with repository paths starting +* The SFTP plugin now manages to deal with repository paths starting with `/~/` which already exist without crashing. * Character and block device nodes are now restored correctly. Thanks to Martin Dummer for the bug report. @@ -184,7 +184,7 @@ Version 0.29, released 2012-05-27; a RELEASE CANDIDATE * "obnam fsck" is now a bit faster. * The shared directories in the repository are now locked only during updates, allowing more efficient concurrent backups between several computers. -* Obnam now gives a better error message when a backup root is not a +* Obnam now gives a better error message when a backup root is not a directory. Thanks to Edward Allcutt for reporting the error (<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654211>). * The output format of "obnam ls" has changed. It now has one line per @@ -259,9 +259,9 @@ Version 0.25, released 2012-02-18; a BETA release Peter Palfrader for the patch. * Some clarification on how the forget policy works, prompted by questions from Peter Palfrader. -* New settings `ssh-known-hosts` (for choosing which file to check for +* New settings `ssh-known-hosts` (for choosing which file to check for known host keys), `strict-ssh-host-keys` (for disallowing unknown host - keys), and `ssh-key` (for choosing which key file to use for SSH + keys), and `ssh-key` (for choosing which key file to use for SSH connections) allow better and safer use of ssh. * Checkpoints will now happen even in the middle of files (but between chunks). @@ -299,8 +299,8 @@ USER VISIBLE CHANGES * If the `--exclude` regular expression is wrong, Obnam now gives an error message and then ignores the regexp, rather than crashing. * There is now a compression plugin, enabled with `--compress-with=gzip`. -* De-duplication mode can now be chosen by the user: the new - `--deduplicate` setting can be one of `never` (fast, but uses more space); +* De-duplication mode can now be chosen by the user: the new + `--deduplicate` setting can be one of `never` (fast, but uses more space); `verify` (slow, but handles hash collisions gracefully); and `fatalist` (fast, but lossy, if there is a hash collision). `fatalist` is the default mode. @@ -344,8 +344,8 @@ USER VISIBLE CHANGES: * The `--fsck-fix` option will now instruct `obnam fsck` to try to fix problems found. For this release, it only means fixing B-tree missing node problems, but more will follow. -* The default sizes have been changed for B-tree nodes (256 KiB) - and file contents chunks (1 MiB), based on benchmarking. +* The default sizes have been changed for B-tree nodes (256 KiB) + and file contents chunks (1 MiB), based on benchmarking. * SFTP protocol use has been optimized, which should result in some more speed. This also highlights the need to change obnam so it can do uploads in the background. @@ -504,7 +504,7 @@ Version 0.18, released 2011-07-20; a BETA release * Various other speed improvements. * Bugfix: restoring symlinks now works even if the symlink is restored before its target. Also, the permissions of the symlink (rather than its - target) are now restored correctly. Thanks to Jeff Epler for an + target) are now restored correctly. Thanks to Jeff Epler for an exemplary bug report. * New option `--one-file-system`, from Jeff Epler. * New benchmarking tool `obnam-benchmark`, which is more flexible than @@ -601,7 +601,7 @@ Improvements and other changes: * Manual page now explains making backups a little better. * Default value for `--lru-size` reduced to 500, for great improvement in memory used, without, it seems, much decrease in speed. -* `obnam verify` now reports success explicitly. Based on question +* `obnam verify` now reports success explicitly. Based on question from Joey Hess. * `obnam verify` now accepts both non-option arguments and the `--root` option. Suggested by Joey Hess. @@ -626,7 +626,7 @@ Version 0.14, released 2010-12-29; an ALPHA release This version is capable of backing up my laptop's home directory. It is, however, still an ALPHA release, and you should not rely on -it as your sole form of backup. It is also slow. But if you're +it as your sole form of backup. It is also slow. But if you're curious, now would be a good time to try it out a bit. Bug fixes: @@ -640,10 +640,10 @@ Bug fixes: the backup got dropped by the user, they are now also removed from the backup generation. Old generations obviously still have them. * Only the per-client B-tree forest should have multiple trees. Now this - actually happens, whereas previously sometimes a very large number of - new trees would be created in some forests. (What's good for rain + actually happens, whereas previously sometimes a very large number of + new trees would be created in some forests. (What's good for rain forests is not good for saving disk space.) -* When recursing through directory trees, obnam no longer follows +* When recursing through directory trees, obnam no longer follows symlinks to directories. * obnam no longer creates a missing backup store when backing up to a local disk. It never did this when backing up via sftp. (This @@ -671,7 +671,7 @@ New features and stuff: This should prevent a teeny tiny bit of information leakage. It also makes debugging things much harder. * Various refactorings and prettifications of the code has happened. - For example, several classes have been split off from the `store.py` + For example, several classes have been split off from the `store.py` module. This has also resulted in much better test coverage for those classes. * The per-client trees (formerly GenerationStore, now ClientMetadataTree) @@ -695,7 +695,7 @@ New features and stuff: which means the caller does not need to first gather a list of all ids. Such a list gets quite costly when the file is quite big (e.g., in the terabyte size). -* New `--dump-memory-profile` option was added to help do memory profiling +* New `--dump-memory-profile` option was added to help do memory profiling with meliae or heapy have been added. (Obnam's memory consumption finally got annoying enough that I did something about it.) @@ -21,7 +21,7 @@ On other systems, using the `setup.py` file should work: run "python setup.py --help" for advice. If not, please report a bug. (I've only tested `setup.py` enough for to build the Debian package.) -You need to install my Python B-tree library, and some of my other libraries +You need to install my Python B-tree library, and some of my other libraries and tools, which you can get from: * <http://liw.fi/larch/> @@ -49,14 +49,14 @@ Use To get a quick help summary of options: ./obnam --help - + To make a backup: ./obnam backup --repository /tmp/mybackup $HOME For more information, see the manual page: - man -l obnam.1 + man -l obnam.1 Hacking @@ -77,7 +77,7 @@ To build and run automatic tests: `check` is a wrapper around `python setup.py`, but since using that takes several steps, the script makes things easier. - + You need my CoverageTestRunner to run tests, see above for where to get it. A couple of scripts exist to run benchmarks and profiles: @@ -85,13 +85,13 @@ A couple of scripts exist to run benchmarks and profiles: ./obnam-benchmark --size=1m/100k --results /tmp/benchmark-results viewprof /tmp/benchmark-results/*/*backup-0.prof seivots-summary /tmp/benchmark-results/*/*.seivot | less -S - + There are two kinds of results: Python profiling output, and `.seivot` files. -For the former, `viewprof` is a little helper script I wrote, +For the former, `viewprof` is a little helper script I wrote, around the Python pstats module. -You can use your own, or get mine from extrautils +You can use your own, or get mine from extrautils (<http://liw.fi/extrautils/>). Running the benchmarks under profiling makes them a little slower (typically around 10% for me, when I've compared), but that's OK: the absolute numbers of the benchmarks are @@ -155,6 +155,6 @@ This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. - + You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/_obnammodule.c b/_obnammodule.c index f160d5af..f3e208bc 100644 --- a/_obnammodule.c +++ b/_obnammodule.c @@ -2,7 +2,7 @@ * _obnammodule.c -- Python extensions for Obna * * Copyright (C) 2008, 2009 Lars Wirzenius <liw@liw.fi> - * + * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 3 of the License, or @@ -21,7 +21,7 @@ /* * This is a Python extension module written for Obnam, the backup - * software. + * software. * * This module provides a way to call the posix_fadvise function from * Python. Obnam uses this to use set the POSIX_FADV_SEQUENTIAL and @@ -80,8 +80,8 @@ utimensat_wrapper(PyObject *self, PyObject *args) long mtime_sec, mtime_nsec; struct timespec tv[2]; - if (!PyArg_ParseTuple(args, "sllll", - &filename, + if (!PyArg_ParseTuple(args, "sllll", + &filename, &atime_sec, &atime_nsec, &mtime_sec, @@ -113,7 +113,7 @@ lstat_wrapper(PyObject *self, PyObject *args) if (ret == -1) ret = errno; - return Py_BuildValue("iKKKKKKKLLLLKLKLK", + return Py_BuildValue("iKKKKKKKLLLLKLKLK", ret, (unsigned long long) st.st_dev, (unsigned long long) st.st_ino, @@ -157,7 +157,7 @@ llistxattr_wrapper(PyObject *self, PyObject *args) o = Py_BuildValue("i", errno); free(buf); } while (o == NULL); - + return o; } @@ -186,7 +186,7 @@ lgetxattr_wrapper(PyObject *self, PyObject *args) o = Py_BuildValue("i", errno); free(buf); } while (o == NULL); - + return o; } @@ -211,7 +211,7 @@ lsetxattr_wrapper(PyObject *self, PyObject *args) static PyMethodDef methods[] = { - {"fadvise_dontneed", fadvise_dontneed, METH_VARARGS, + {"fadvise_dontneed", fadvise_dontneed, METH_VARARGS, "Call posix_fadvise(2) with POSIX_FADV_DONTNEED argument."}, {"utimensat", utimensat_wrapper, METH_VARARGS, "utimensat(2) wrapper."}, diff --git a/analyze-repository-files b/analyze-repository-files index 90a97716..d4ebaf70 100755 --- a/analyze-repository-files +++ b/analyze-repository-files @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -42,38 +42,38 @@ class Stats(object): def __init__(self): self.dirs = list() self.files = list() - + def add_dir(self, dirname, count): self.dirs.append((count, dirname)) - + def add_file(self, filename, size): self.files.append((size, filename)) - + @property def total_files(self): return len(self.files) - + @property def sum_of_sizes(self): return sum(size for size, name in self.files) - + @property def dirsizes(self): self.dirs.sort() num_dirs = len(self.dirs) - + fewest, fewest_name = self.dirs[0] most, most_name = self.dirs[-1] average = sum(count for count, name in self.dirs) / num_dirs median = self.dirs[num_dirs/2][0] return fewest, fewest_name, most, most_name, average, median - + @property def filesizes(self): self.files.sort() num_files = len(self.files) - + smallest, smallest_name = self.files[0] largest, largest_name = self.files[-1] average = sum(size for size, name in self.files) / num_files @@ -96,7 +96,7 @@ def main(): stats.add_file(pathname, stat_info.st_size) elif stat.S_ISREG(stat_info.st_mode): stats.add_file(name, stat_info.st_size) - + print "total_files:", stats.total_files print "sum of sizes:", stats.sum_of_sizes @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/check-lock-usage-from-log b/check-lock-usage-from-log index 6acc2402..44a0197d 100755 --- a/check-lock-usage-from-log +++ b/check-lock-usage-from-log @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -82,7 +82,7 @@ class LockEvent(LogEvent): def __str__(self): return 'Lock(%s)' % self.lockname - + class UnlockEvent(LockEvent): @@ -114,13 +114,13 @@ class NodeCreateEvent(LogEvent): def __str__(self): return 'NodeCreate(%s)' % self.node_id - + class NodeDestroyEvent(NodeCreateEvent): def __str__(self): return 'NodeDestroy(%s)' % self.node_id - + class NodeReadEvent(NodeCreateEvent): @@ -145,7 +145,7 @@ class CheckLocks(cliapp.Application): self.events = [] self.errors = 0 self.latest_opened_node = None - + self.patterns = [ (lock_pat, self.lock_event), (unlock_pat, self.unlock_event), @@ -155,12 +155,12 @@ class CheckLocks(cliapp.Application): (node_remove_pat, self.node_remove_event), (rename_pat, self.rename_event), ] - + self.ts = ttystatus.TerminalStatus() self.ts.format( 'Reading %ElapsedTime() %Integer(lines): %Pathname(filename)') self.ts['lines'] = 0 - + def cleanup(self): self.ts.clear() @@ -188,13 +188,13 @@ class CheckLocks(cliapp.Application): for e in self.events: self.ts['event'] = e logging.debug( - 'analysing: %s:%s: %s: %s' % + 'analysing: %s:%s: %s: %s' % (e.logfile, e.lineno, repr(e.sortkey()), str(e))) if type(e) is LockEvent: if e.lockname in current_locks: self.error( - 'Re-locking %s: %s:%s:%s' % - (e.lockname, e.logfile, e.lineno, + 'Re-locking %s: %s:%s:%s' % + (e.lockname, e.logfile, e.lineno, e.timestamp)) else: current_locks.add(e.lockname) @@ -300,7 +300,7 @@ class CheckLocks(cliapp.Application): return NodeReadEvent( filename, self.lineno, float(match.group('timestamp')), node_id) - + def node_remove_event(self, filename, line, match): return NodeDestroyEvent( filename, self.lineno, float(match.group('timestamp')), @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/create-vfat-disk-image b/create-vfat-disk-image index 00bffd2b..f412ed68 100755 --- a/create-vfat-disk-image +++ b/create-vfat-disk-image @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/debian/changelog b/debian/changelog index 8270e0b3..6033dca3 100644 --- a/debian/changelog +++ b/debian/changelog @@ -60,20 +60,20 @@ obnam (0.30-1) unstable; urgency=low obnam (0.29-1) unstable; urgency=low * New upstream release. - - "obnam backup" now writes performance statistics at the end of a - backup run. Search the log for "Backup performance statistics" + - "obnam backup" now writes performance statistics at the end of a + backup run. Search the log for "Backup performance statistics" (INFO level). - - "obnam verify" now continues past the first error. Thanks to + - "obnam verify" now continues past the first error. Thanks to Rafał Gwiazda for requesting this. - Add an `obnam-viewprof` utility to translate Python profiling output into human readable text form. - - Bug fix: If a file's extended attributes have changed in any way, + - Bug fix: If a file's extended attributes have changed in any way, the change is now backed up. - "obnam fsck" is now a bit faster. - - The shared directories in the repository are now locked only during - updates, allowing more efficient concurrent backups between several + - The shared directories in the repository are now locked only during + updates, allowing more efficient concurrent backups between several computers. - - Obnam now gives a better error message when a backup root is not a + - Obnam now gives a better error message when a backup root is not a directory. Thanks to Edward Allcutt for reporting the error. (Closes: #654211) - The output format of "obnam ls" has changed. It now has one line per @@ -124,7 +124,7 @@ obnam (0.25-1) unstable; urgency=low * New upstream version. * Depend on python-larch 0.27, for fsck WorkItem changes - * Build-depend on genbackupdata, summain, and cmdtest, which are used + * Build-depend on genbackupdata, summain, and cmdtest, which are used by tests. -- Lars Wirzenius <liw@liw.fi> Sat, 18 Feb 2012 11:45:56 +0000 diff --git a/debian/control b/debian/control index 78724bf0..6c473ecc 100644 --- a/debian/control +++ b/debian/control @@ -4,14 +4,14 @@ Uploader: Gaudenz Steinlin <gaudenz@debian.org> Section: python Priority: optional Standards-Version: 3.9.3 -Build-Depends: debhelper (>= 7.3.8), - python-all-dev (>= 2.6.3~3), - python-coverage-test-runner (>= 1.8~), +Build-Depends: debhelper (>= 7.3.8), + python-all-dev (>= 2.6.3~3), + python-coverage-test-runner (>= 1.8~), python-larch (>= 1.20130316~), python-ttystatus (>= 0.22~), - python-paramiko, - python-tracing (>= 0.6~), - fakeroot, + python-paramiko, + python-tracing (>= 0.6~), + fakeroot, python-cliapp (>= 1.20130313~), genbackupdata (>= 1.6~), summain (>= 0.18), @@ -23,26 +23,26 @@ X-Python-Version: >= 2.6 Package: obnam Provides: python-obnam Architecture: linux-any -Depends: ${shlibs:Depends}, ${python:Depends}, ${misc:Depends}, +Depends: ${shlibs:Depends}, ${python:Depends}, ${misc:Depends}, python-larch (>= 1.20130316~), python-ttystatus (>= 0.22~), - python-paramiko, - python-tracing (>= 0.6~), + python-paramiko, + python-tracing (>= 0.6~), python-cliapp (>= 1.20130313~) Description: online and disk-based backup application - Obnam makes backups. Backups can be stored on local hard disks, or online - via the SSH SFTP protocol. The backup server, if used, does not require any + Obnam makes backups. Backups can be stored on local hard disks, or online + via the SSH SFTP protocol. The backup server, if used, does not require any special software, on top of SSH. . - * Snapshot backups. Every generation looks like a complete snapshot, so you - don't need to care about full versus incremental backups, or rotate real + * Snapshot backups. Every generation looks like a complete snapshot, so you + don't need to care about full versus incremental backups, or rotate real or virtual tapes. - * Data de-duplication, across files, and backup generations. If the backup - repository already contains a particular chunk of data, it will be re-used, - even if it was in another file in an older backup generation. This way, you + * Data de-duplication, across files, and backup generations. If the backup + repository already contains a particular chunk of data, it will be re-used, + even if it was in another file in an older backup generation. This way, you don't need to worry about moving around large files, or modifying them. * Encrypted backups, using GnuPG. - * Push or pull operation, depending on what you need. You can run Obnam on - the client, and push backups to the server, or on the server, and pull + * Push or pull operation, depending on what you need. You can run Obnam on + the client, and push backups to the server, or on the server, and pull from the client over SFTP. @@ -28,7 +28,7 @@ def find_objids(fs): fs = obnamlib.LocalFS(sys.argv[1]) -repo = obnamlib.Repository(fs, obnamlib.DEFAULT_NODE_SIZE, +repo = obnamlib.Repository(fs, obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, None, obnamlib.IDPATH_DEPTH, obnamlib.IDPATH_BITS, diff --git a/find-duplicate-chunks b/find-duplicate-chunks index 8685897b..5990b3d7 100755 --- a/find-duplicate-chunks +++ b/find-duplicate-chunks @@ -26,8 +26,8 @@ def compute_checksums(f, chunk_size, offset, dirname): for checksum in compute(ff, chunk_size, offset): f.write('%s\n' % checksum) ff.close() - - + + def sort_checksums(f, checksums_name): subprocess.check_call(['sort', '-T', '.', @@ -36,12 +36,12 @@ def sort_checksums(f, checksums_name): ], stdin=file(checksums_name), stdout=f) - + def count_duplicates(f, sorted_name): subprocess.check_call(['uniq', '-c'], stdin=file(sorted_name), stdout=f) - - + + def make_report(f, counts_name, chunk_size, offset): num_diff_checksums = 0 saved = 0 @@ -64,7 +64,7 @@ def make_report(f, counts_name, chunk_size, offset): if count <= limit: counts[limit] += count break - + f.write('chunk size: %d\n' % chunk_size) f.write('offset: %d\n' % offset) f.write('#different checksums: %d\n' % num_diff_checksums) @@ -79,20 +79,20 @@ def main(): chunk_size = int(sys.argv[1]) offset = int(sys.argv[2]) dirname = sys.argv[3] - + prefix = 'data-%04d-%04d' % (chunk_size, offset) checksums_name = prefix + '.checksums' sorted_name = prefix + '.sorted' counts_name = prefix + '.counts' report_name = prefix + '.report' - + steps = ( (checksums_name, compute_checksums, (chunk_size, offset, dirname)), (sorted_name, sort_checksums, (checksums_name,)), (counts_name, count_duplicates, (sorted_name,)), (report_name, make_report, (counts_name, chunk_size, offset)), ) - + for filename, func, args in steps: if not os.path.exists(filename): print 'Step:', func.__name__ diff --git a/lock-and-increment b/lock-and-increment index 18935762..af261775 100755 --- a/lock-and-increment +++ b/lock-and-increment @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/metadata-speed b/metadata-speed index 3b6525cf..23cb1825 100755 --- a/metadata-speed +++ b/metadata-speed @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/obnam-benchmark b/obnam-benchmark index 68057a02..5da34797 100755 --- a/obnam-benchmark +++ b/obnam-benchmark @@ -1,17 +1,17 @@ #!/usr/bin/python # # Copyright 2010, 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -32,7 +32,7 @@ class ObnamBenchmark(cliapp.Application): default_sizes = ['1g/100m'] keyid = '3B1802F81B321347' opers = ('backup', 'restore', 'list_files', 'forget') - + def add_settings(self): self.settings.string(['results'], 'put results under DIR (%default)', metavar='DIR', default='../benchmarks') @@ -51,8 +51,8 @@ class ObnamBenchmark(cliapp.Application): metavar='DIR') self.settings.boolean(['with-encryption'], 'run benchmark using encryption') - - self.settings.string(['profile-name'], + + self.settings.string(['profile-name'], 'short name for benchmark scenario', default='unknown') self.settings.string_list(['size'], @@ -77,7 +77,7 @@ class ObnamBenchmark(cliapp.Application): self.settings.string(['description'], 'describe benchmark') self.settings.boolean(['drop-caches'], 'drop kernel buffer caches') self.settings.string(['seivot-log'], 'seivot log setting') - + self.settings.boolean(['verify'], 'verify restores') def process_args(self, args): @@ -87,7 +87,7 @@ class ObnamBenchmark(cliapp.Application): larch_revno = self.bzr_revno(self.settings['larch-branch']) results = self.results_dir(obnam_revno, larch_revno) - + obnam_branch = self.settings['obnam-branch'] larch_branch = self.settings['larch-branch'] if self.settings['seivot-branch']: @@ -96,13 +96,13 @@ class ObnamBenchmark(cliapp.Application): seivot = 'seivot' generations = self.settings['generations'] - + tempdir = tempfile.mkdtemp() env = self.setup_gnupghome(tempdir) - + sizes = self.settings['size'] or self.default_sizes logging.debug('sizes: %s' % repr(sizes)) - + file_size = self.settings['file-size'] profile_name = self.settings['profile-name'] @@ -144,14 +144,14 @@ class ObnamBenchmark(cliapp.Application): if self.settings['with-encryption']: argv.extend(['--encrypt-with', self.keyid]) if self.settings['description']: - argv.extend(['--description', + argv.extend(['--description', self.settings['description']]) if self.settings['verify']: argv.append('--verify') self.runcmd(argv, env=env) shutil.rmtree(tempdir) - + def require_tmpdir(self): if 'TMPDIR' not in os.environ: raise cliapp.AppException('TMPDIR is not set. ' @@ -165,7 +165,7 @@ class ObnamBenchmark(cliapp.Application): @property def hostname(self): return socket.gethostname() - + @property def obnam_branch_name(self): obnam_branch = os.path.abspath(self.settings['obnam-branch']) @@ -173,7 +173,7 @@ class ObnamBenchmark(cliapp.Application): def results_dir(self, obnam_revno, larch_revno): parent = self.settings['results'] - + counter = 0 while True: counter += 1 @@ -185,12 +185,12 @@ class ObnamBenchmark(cliapp.Application): str(counter), ] basename = '-'.join(parts) - + dirname = os.path.join(parent, basename) if not os.path.exists(dirname): os.mkdir(dirname) return dirname - + def setup_gnupghome(self, tempdir): gnupghome = os.path.join(tempdir, 'gnupghome') shutil.copytree('test-gpghome', gnupghome) @@ -208,7 +208,7 @@ class ObnamBenchmark(cliapp.Application): revno = out.strip() logging.debug('bzr branch %s has revno %s' % (branch, revno)) return revno - + def parse_size_pair(self, pair): return pair.split('/', 1) diff --git a/obnam-benchmark.1.in b/obnam-benchmark.1.in index 37e2f535..f52ee74c 100644 --- a/obnam-benchmark.1.in +++ b/obnam-benchmark.1.in @@ -1,15 +1,15 @@ .\" Copyright 2011 Lars Wirzenius <liw@liw.fi> -.\" +.\" .\" This program is free software: you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation, either version 3 of the License, or .\" (at your option) any later version. -.\" +.\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. -.\" +.\" .\" You should have received a copy of the GNU General Public License .\" along with this program. If not, see <http://www.gnu.org/licenses/>. .\" @@ -36,7 +36,7 @@ directly. Benchmarks are run using two different usage profiles: .I mailspool (all files are small), and -.I mediaserver +.I mediaserver (all files are big). For each profile, test data of the desired total size is generated, @@ -48,23 +48,23 @@ restoring, listing the contents of, and removing each generation. .PP -The result of the benchmark is a +The result of the benchmark is a .I .seivot file per profile, plus a Python profiler file for each run of .BR obnam . -These are stored in +These are stored in .IR ../benchmarks . A set of .I .seivot files can be summarized for comparison with .BR seivots-summary (1). The profiling files can be viewed with the usual Python tools: -see the +see the .B pstats module. .PP -The benchmarks are run against a version of +The benchmarks are run against a version of .B obnam checked out from version control. It is not (currently) possible to run the benchmark against an installed @@ -73,7 +73,7 @@ version of Also the .I larch Python library, -which +which .B obnam needs, needs to be checked out from version control. @@ -87,7 +87,7 @@ if the defaults are not correct. .SH ENVIRONMENT .TP .BR TMPDIR -This variable +This variable .I must be set. It controls where the temporary files (generated test data) is stored. @@ -101,7 +101,7 @@ Thus. requires that the location is set explicitly. (You can still use .I /tmp -if you want, but you have to set +if you want, but you have to set .B TMPDIR explicitly.) .SH FILES diff --git a/obnam-viewprof b/obnam-viewprof index 03c4cce0..2e88978c 100755 --- a/obnam-viewprof +++ b/obnam-viewprof @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/obnam-viewprof.1 b/obnam-viewprof.1 index b744ad11..d94807f9 100644 --- a/obnam-viewprof.1 +++ b/obnam-viewprof.1 @@ -1,15 +1,15 @@ .\" Copyright 2012 Lars Wirzenius <liw@liw.fi> -.\" +.\" .\" This program is free software: you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation, either version 3 of the License, or .\" (at your option) any later version. -.\" +.\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. -.\" +.\" .\" You should have received a copy of the GNU General Public License .\" along with this program. If not, see <http://www.gnu.org/licenses/>. .\" @@ -27,7 +27,7 @@ You can generate such output from Obnam by setting the .B OBNAM_PROFILE environment variable to a filename. The profile will be written to that filename, -and you should give it to +and you should give it to .B obnam-viewprof as an argument. .PP @@ -1,15 +1,15 @@ .\" Copyright 2010, 2011 Lars Wirzenius -.\" +.\" .\" This program is free software: you can redistribute it and/or modify .\" it under the terms of the GNU General Public License as published by .\" the Free Software Foundation, either version 3 of the License, or .\" (at your option) any later version. -.\" +.\" .\" This program is distributed in the hope that it will be useful, .\" but WITHOUT ANY WARRANTY; without even the implied warranty of .\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .\" GNU General Public License for more details. -.\" +.\" .\" You should have received a copy of the GNU General Public License .\" along with this program. If not, see <http://www.gnu.org/licenses/>. .TH OBNAM 1 @@ -35,8 +35,8 @@ A single repository may contain backups from several clients. Their data will intermingle as if they were using separate repositories, but if one client backs up a file, the others may re-use the data. .PP -.B obnam -command line syntax consists of a +.B obnam +command line syntax consists of a .I command possibly followed by arguments. The commands are list below. @@ -56,7 +56,7 @@ or just selected files. lists the clients that are backed up to the repository. .IP \(bu .B generations -lists every backup generation for a given client, +lists every backup generation for a given client, plus some metadata about the generation. .IP \(bu .B genids @@ -65,7 +65,7 @@ No other information is shown. This can be useful for scripting. .IP \(bu .B ls -lists the contents of a given generation, similar to +lists the contents of a given generation, similar to .BR "ls \-lAR" . .IP \(bu .B verify @@ -112,7 +112,7 @@ Some of the toplevel directories are shared between clients, others are specific to a client. .IP \(bu .B list\-toplevels -is like +is like .BR list\-keys , but lists toplevels and which keys can access them. .IP \(bu @@ -155,7 +155,7 @@ uploads data into the backup repository. The data is divided into chunks, and if a chunk already exists in the backup repository, it is not uploaded again. -This allows +This allows .B obnam to deal with files that have been changed or renamed since the previous backup run. @@ -163,11 +163,11 @@ It also allows several backup clients to avoid uploading the same data. If, for example, everyone in the office has a copy of the same sales brochures, only one copy needs to be stored in the backup repository. .PP -Every backup run is a +Every backup run is a .IR generation . In addition, .B obnam -will make +will make .I checkpoint generations every now and then. These are exactly like normal generations, @@ -192,7 +192,7 @@ command checks that data in the backup repository matches actual user data. It retrieves one or more files from the repository and compares them to the user data. This is essentialy the same as doing a restore, -then comparing restored files with the original files using +then comparing restored files with the original files using .BR cmp (1), but easier to use. .PP @@ -208,21 +208,21 @@ parent directories of backup root) may have changed without it being a problem. Note that you will need to specify the whole path to the files or directories to be verified, not relative to the backup root. -You still need to specify at least one of the backup roots +You still need to specify at least one of the backup roots on the command line or via the .B \-\-root option so that obnam will find the filesystem, in case it is a remote one. .SS "URL syntax" Whenever obnam accepts a URL, it can be either a local pathname, -or an +or an .B sftp URL. An sftp URL has the following form: .IP .R \fBsftp://\fR[\fIuser\fR@]\fIdomain\fR[:\fIport\fR]\fB/path .PP -where +where .I domain is a normal Internet domain name for a server, .I user @@ -246,7 +246,7 @@ You can use .B sftp URLs for the repository, or the live data (root), but note that due to limitations in the protocol, -and its implementation in the +and its implementation in the .B paramiko library, some things will not work very well for accessing live data over @@ -283,7 +283,7 @@ option. .I POLICY is comma-separated list of rules. Each rule consists of a count and a time period. -The time periods are +The time periods are .BR h , .BR d , .BR w , @@ -320,7 +320,7 @@ and does not care what the actual current time is. This means that if you stop making new backups, the existing ones won't be removed automatically. In essence, obnam pretends the current time is just after the -latest backup when +latest backup when .B forget is run. .PP @@ -384,7 +384,7 @@ option to make sure you're removing the right ones. .SS "Using encryption" .B obnam can encrypt all the data it writes to the backup repository. -It uses +It uses .BR gpg (1) to do the encryption. You need to create a key pair using @@ -403,7 +403,7 @@ All files are treated as if they were one with the contents of all files catenated. .PP The files are in INI format, -and only the +and only the .I [config] section is used (any other sections are ignored). @@ -431,7 +431,7 @@ exclude: \.wav$ .RE .fi .sp 1 -(You can use either +(You can use either .I foo=value or .I foo: value @@ -459,7 +459,7 @@ any files that contain the words or .I bar anywhere in the fully qualified pathname, -or files with names ending with a period and +or files with names ending with a period and .I mp3 (because the exclusions are regular expressions). .SS "Multiple clients and locking" diff --git a/obnamlib/app.py b/obnamlib/app.py index e4d2ae55..975ef0d5 100644 --- a/obnamlib/app.py +++ b/obnamlib/app.py @@ -31,11 +31,11 @@ import obnamlib class App(cliapp.Application): '''Main program for backup program.''' - + def add_settings(self): devel_group = obnamlib.option_group['devel'] perf_group = obnamlib.option_group['perf'] - + self.settings.string(['repository', 'r'], 'name of backup repository') self.settings.string(['client-name'], 'name of client (%default)', @@ -72,7 +72,7 @@ class App(cliapp.Application): 'add to filename patters for which trace ' 'debugging logging happens') - + self.settings.integer(['idpath-depth'], 'depth of chunk id mapping', default=obnamlib.IDPATH_DEPTH, @@ -93,7 +93,7 @@ class App(cliapp.Application): 'do not actually change anything (works with ' 'backup, forget and restore only, and may only ' 'simulate approximately real behavior)') - + self.settings.string(['pretend-time'], 'pretend it is TIMESTAMP (YYYY-MM-DD HH:MM:SS); ' 'this is only useful for testing purposes', @@ -127,7 +127,7 @@ class App(cliapp.Application): self.pm = obnamlib.PluginManager() self.pm.locations = [self.plugins_dir()] self.pm.plugin_arguments = (self,) - + self.setup_hooks() self.fsf = obnamlib.VfsFactory() @@ -135,7 +135,7 @@ class App(cliapp.Application): self.pm.load_plugins() self.pm.enable_plugins() self.hooks.call('plugins-loaded') - + self.settings['log-level'] = 'info' def deduce_client_name(self): @@ -195,7 +195,7 @@ class App(cliapp.Application): repofs.connect() else: repofs.reinit(repopath) - return obnamlib.Repository(repofs, + return obnamlib.Repository(repofs, self.settings['node-size'], self.settings['upload-queue-size'], self.settings['lru-size'], @@ -209,10 +209,10 @@ class App(cliapp.Application): def time(self): '''Return current time in seconds since epoch. - + This is a wrapper around time.time() so that it can be overridden with the --pretend-time setting. - + ''' s = self.settings['pretend-time'] diff --git a/obnamlib/checksumtree.py b/obnamlib/checksumtree.py index ecc4ab29..75d6c3a7 100644 --- a/obnamlib/checksumtree.py +++ b/obnamlib/checksumtree.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -29,12 +29,12 @@ class ChecksumTree(obnamlib.RepositoryTree): ''' - def __init__(self, fs, name, checksum_length, node_size, + def __init__(self, fs, name, checksum_length, node_size, upload_queue_size, lru_size, hooks): tracing.trace('new ChecksumTree name=%s' % name) self.fmt = '!%dsQQ' % checksum_length key_bytes = len(self.key('', 0, 0)) - obnamlib.RepositoryTree.__init__(self, fs, name, key_bytes, node_size, + obnamlib.RepositoryTree.__init__(self, fs, name, key_bytes, node_size, upload_queue_size, lru_size, hooks) self.keep_just_one_tree = True diff --git a/obnamlib/checksumtree_tests.py b/obnamlib/checksumtree_tests.py index 9fe8d3d9..01736ea7 100644 --- a/obnamlib/checksumtree_tests.py +++ b/obnamlib/checksumtree_tests.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -30,7 +30,7 @@ class ChecksumTreeTests(unittest.TestCase): self.hooks = obnamlib.HookManager() self.hooks.new('repository-toplevel-init') self.checksum = hashlib.md5('foo').digest() - self.tree = obnamlib.ChecksumTree(fs, 'x', len(self.checksum), + self.tree = obnamlib.ChecksumTree(fs, 'x', len(self.checksum), obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, obnamlib.DEFAULT_LRU_SIZE, self) diff --git a/obnamlib/chunklist.py b/obnamlib/chunklist.py index 9587ebb5..c6557d0b 100644 --- a/obnamlib/chunklist.py +++ b/obnamlib/chunklist.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -25,19 +25,19 @@ import obnamlib class ChunkList(obnamlib.RepositoryTree): '''Repository's list of chunks. - + The list maps a chunk id to its checksum. - + The list is implemented as a B-tree, with the 64-bit chunk id as the key, and the checksum as the value. - + ''' def __init__(self, fs, node_size, upload_queue_size, lru_size, hooks): tracing.trace('new ChunkList') self.key_bytes = len(self.key(0)) - obnamlib.RepositoryTree.__init__(self, fs, 'chunklist', self.key_bytes, - node_size, upload_queue_size, + obnamlib.RepositoryTree.__init__(self, fs, 'chunklist', self.key_bytes, + node_size, upload_queue_size, lru_size, hooks) self.keep_just_one_tree = True @@ -49,13 +49,13 @@ class ChunkList(obnamlib.RepositoryTree): tracing.trace('checksum=%s', repr(checksum)) self.start_changes() self.tree.insert(self.key(chunk_id), checksum) - + def get_checksum(self, chunk_id): if self.init_forest() and self.forest.trees: t = self.forest.trees[-1] return t.lookup(self.key(chunk_id)) raise KeyError(chunk_id) - + def remove(self, chunk_id): tracing.trace('chunk_id=%s', chunk_id) self.start_changes() diff --git a/obnamlib/chunklist_tests.py b/obnamlib/chunklist_tests.py index 642732a0..99cb72e7 100644 --- a/obnamlib/chunklist_tests.py +++ b/obnamlib/chunklist_tests.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -28,7 +28,7 @@ class ChunkListTests(unittest.TestCase): fs = obnamlib.LocalFS(self.tempdir) self.hooks = obnamlib.HookManager() self.hooks.new('repository-toplevel-init') - self.list = obnamlib.ChunkList(fs, + self.list = obnamlib.ChunkList(fs, obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, obnamlib.DEFAULT_LRU_SIZE, self) @@ -38,11 +38,11 @@ class ChunkListTests(unittest.TestCase): def test_raises_keyerror_for_missing_chunk(self): self.assertRaises(KeyError, self.list.get_checksum, 0) - + def test_adds_chunk(self): self.list.add(0, 'checksum') self.assertEqual(self.list.get_checksum(0), 'checksum') - + def test_adds_second_chunk(self): self.list.add(0, 'checksum') self.list.add(1, 'checksum1') diff --git a/obnamlib/clientlist.py b/obnamlib/clientlist.py index 71a1cb29..9ca3d5a2 100644 --- a/obnamlib/clientlist.py +++ b/obnamlib/clientlist.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -26,20 +26,20 @@ import obnamlib class ClientList(obnamlib.RepositoryTree): '''Repository's list of clients. - + The list maps a client name to an arbitrary (string) identifier, which is unique within the repository. - + The list is implemented as a B-tree, with a three-part key: 128-bit MD5 of client name, 64-bit unique identifier, and subkey identifier. The value depends on the subkey: it's either the client's full name, or the public key identifier the client uses to encrypt their backups. - + The client's identifier is a random, unique 64-bit integer. - + ''' - + # subkey values CLIENT_NAME = 0 KEYID = 1 @@ -51,10 +51,10 @@ class ClientList(obnamlib.RepositoryTree): self.fmt = '!%dsQB' % self.hash_len self.key_bytes = len(self.key('', 0, 0)) self.minkey = self.hashkey('\x00' * self.hash_len, 0, 0) - self.maxkey = self.hashkey('\xff' * self.hash_len, obnamlib.MAX_ID, + self.maxkey = self.hashkey('\xff' * self.hash_len, obnamlib.MAX_ID, self.SUBKEY_MAX) - obnamlib.RepositoryTree.__init__(self, fs, 'clientlist', - self.key_bytes, node_size, + obnamlib.RepositoryTree.__init__(self, fs, 'clientlist', + self.key_bytes, node_size, upload_queue_size, lru_size, hooks) self.keep_just_one_tree = True @@ -77,7 +77,7 @@ class ClientList(obnamlib.RepositoryTree): def list_clients(self): if self.init_forest() and self.forest.trees: t = self.forest.trees[-1] - return [v + return [v for k, v in t.lookup_range(self.minkey, self.maxkey) if self.unkey(k)[2] == self.CLIENT_NAME] else: @@ -112,7 +112,7 @@ class ClientList(obnamlib.RepositoryTree): key = self.key(client_name, candidate_id, self.CLIENT_NAME) self.tree.insert(key, client_name) logging.debug('Client %s has id %s' % (client_name, candidate_id)) - + def remove_client(self, client_name): logging.info('Removing client %s' % client_name) self.start_changes() @@ -130,7 +130,7 @@ class ClientList(obnamlib.RepositoryTree): for k, v in t.lookup_range(key, key): return v return None - + def set_client_keyid(self, client_name, keyid): logging.info('Setting client %s to use key %s' % (client_name, keyid)) self.start_changes() diff --git a/obnamlib/clientlist_tests.py b/obnamlib/clientlist_tests.py index b65e98ae..ff1d1e04 100644 --- a/obnamlib/clientlist_tests.py +++ b/obnamlib/clientlist_tests.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -28,7 +28,7 @@ class ClientListTests(unittest.TestCase): fs = obnamlib.LocalFS(self.tempdir) self.hooks = obnamlib.HookManager() self.hooks.new('repository-toplevel-init') - self.list = obnamlib.ClientList(fs, + self.list = obnamlib.ClientList(fs, obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, obnamlib.DEFAULT_LRU_SIZE, self) @@ -37,7 +37,7 @@ class ClientListTests(unittest.TestCase): shutil.rmtree(self.tempdir) def test_key_bytes_is_correct_length(self): - self.assertEqual(self.list.key_bytes, + self.assertEqual(self.list.key_bytes, len(self.list.key('foo', 12765, 0))) def test_unkey_unpacks_key_correctly(self): @@ -48,7 +48,7 @@ class ClientListTests(unittest.TestCase): def test_reports_none_as_id_for_nonexistent_client(self): self.assertEqual(self.list.get_client_id('foo'), None) - + def test_lists_no_clients_when_tree_does_not_exist(self): self.assertEqual(self.list.list_clients(), []) @@ -65,7 +65,7 @@ class ClientListTests(unittest.TestCase): self.list.add_client('foo') self.list.remove_client('foo') self.assertEqual(self.list.get_client_id('foo'), None) - + def test_removed_client_has_no_keys(self): self.list.add_client('foo') client_id = self.list.get_client_id('foo') diff --git a/obnamlib/clientmetadatatree.py b/obnamlib/clientmetadatatree.py index c63113fb..831b7142 100644 --- a/obnamlib/clientmetadatatree.py +++ b/obnamlib/clientmetadatatree.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -27,16 +27,16 @@ import obnamlib class ClientMetadataTree(obnamlib.RepositoryTree): '''Store per-client metadata about files. - - Actual file contents is stored elsewhere, this stores just the + + Actual file contents is stored elsewhere, this stores just the metadata about files: names, inode info, and what chunks of data they use. - + See http://braawi.org/obnam/ondisk/ for a description of how this works. - + ''' - + # Filesystem metadata. PREFIX_FS_META = 0 # prefix FILE_NAME = 0 # subkey type for storing pathnames @@ -44,14 +44,14 @@ class ClientMetadataTree(obnamlib.RepositoryTree): FILE_METADATA = 3 # subkey type for inode fields, etc DIR_CONTENTS = 4 # subkey type for list of directory contents FILE_DATA = 5 # subkey type for file data (instead of chunk) - + FILE_METADATA_ENCODED = 0 # subkey value for encoded obnamlib.Metadata(). - + # References to chunks in this generation. # Main key is the chunk id, subkey type is always 0, subkey is file id # for file that uses the chunk. PREFIX_CHUNK_REF = 1 - + # Metadata about the generation. The main key is always the hash of # 'generation', subkey type field is always 0. PREFIX_GEN_META = 2 # prefix @@ -61,20 +61,20 @@ class ClientMetadataTree(obnamlib.RepositoryTree): GEN_IS_CHECKPOINT = 3 # subkey type for whether generation is checkpoint GEN_FILE_COUNT = 4 # subkey type for count of files+dirs in generation GEN_TOTAL_DATA = 5 # subkey type for sum of all file sizes in gen - + # Maximum values for the subkey type field, and the subkey field. # Both have a minimum value of 0. TYPE_MAX = 255 SUBKEY_MAX = struct.pack('!Q', obnamlib.MAX_ID) - + def __init__(self, fs, client_dir, node_size, upload_queue_size, lru_size, repo): tracing.trace('new ClientMetadataTree, client_dir=%s' % client_dir) self.current_time = repo.current_time key_bytes = len(self.hashkey(0, self.default_file_id(''), 0, 0)) - obnamlib.RepositoryTree.__init__(self, fs, client_dir, key_bytes, - node_size, upload_queue_size, + obnamlib.RepositoryTree.__init__(self, fs, client_dir, key_bytes, + node_size, upload_queue_size, lru_size, repo) self.genhash = self.default_file_id('generation') self.chunkids_per_key = max(1, @@ -120,7 +120,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): be converted as a string, and the value must fit into 64 bits. ''' - + if type(subkey) == str: subkey = (subkey + '\0' * 8)[:8] fmt = '!B8sB8s' @@ -133,7 +133,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def fskey(self, mainhash, subtype, subkey): ''''Generate key for filesystem metadata.''' return self.hashkey(self.PREFIX_FS_META, mainhash, subtype, subkey) - + def fs_unkey(self, key): '''Inverse of fskey.''' parts = struct.unpack('!B8sB8s', key) @@ -159,13 +159,13 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def get_file_id(self, tree, pathname): '''Return id for file in a given generation.''' - + if tree in self.file_ids: if pathname in self.file_ids[tree]: return self.file_ids[tree][pathname] else: self.file_ids[tree] = {} - + default_file_id = self.default_file_id(pathname) minkey = self.fskey(default_file_id, self.FILE_NAME, 0) maxkey = self.fskey(default_file_id, self.FILE_NAME, obnamlib.MAX_ID) @@ -192,13 +192,13 @@ class ClientMetadataTree(obnamlib.RepositoryTree): if value == pathname: return file_id file_ids.add(file_id) - + while True: n = random.randint(0, obnamlib.MAX_ID) file_id = struct.pack('!Q', n) if file_id not in file_ids: break - + key = self.fskey(default_file_id, self.FILE_NAME, file_id) self.tree.insert(key, pathname) return file_id @@ -232,7 +232,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): return obnamlib.RepositoryTree.start_changes(self, *args, **kwargs) def find_generation(self, genid): - + def fill_cache(): key = self.genkey(self.GEN_ID) for t in self.forest.trees: @@ -240,7 +240,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): if t_genid == genid: self.known_generations[genid] = t return t - + if self.forest: if genid in self.known_generations: return self.known_generations[genid] @@ -371,7 +371,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def get_metadata(self, genid, filename): tree = self.find_generation(genid) file_id = self.get_file_id(tree, filename) - key = self.fskey(file_id, self.FILE_METADATA, + key = self.fskey(file_id, self.FILE_METADATA, self.FILE_METADATA_ENCODED) return tree.lookup(key) @@ -381,8 +381,8 @@ class ClientMetadataTree(obnamlib.RepositoryTree): file_id = self.set_file_id(filename) key1 = self.fskey(file_id, self.FILE_NAME, file_id) self.tree.insert(key1, filename) - - key2 = self.fskey(file_id, self.FILE_METADATA, + + key2 = self.fskey(file_id, self.FILE_METADATA, self.FILE_METADATA_ENCODED) self.tree.insert(key2, encoded_metadata) @@ -392,7 +392,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): file_id = self.get_file_id(self.tree, filename) genid = self.get_generation_id(self.tree) self.file_count -= 1 - + try: encoded_metadata = self.get_metadata(genid, filename) except KeyError: @@ -412,17 +412,17 @@ class ClientMetadataTree(obnamlib.RepositoryTree): for chunkid in self.get_file_chunks(genid, filename): key = self.chunk_key(chunkid, file_id) self.tree.remove_range(key, key) - + # Remove this file's metadata. minkey = self.fskey(file_id, 0, 0) maxkey = self.fskey(file_id, self.TYPE_MAX, self.SUBKEY_MAX) self.tree.remove_range(minkey, maxkey) - + # Remove filename. default_file_id = self.default_file_id(filename) key = self.fskey(default_file_id, self.FILE_NAME, file_id) self.tree.remove_range(key, key) - + # Also remove from parent's contents. parent = os.path.dirname(filename) if parent != filename: # root dir is its own parent @@ -430,7 +430,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): key = self.fskey(parent_id, self.DIR_CONTENTS, file_id) # The range removal will work even if the key does not exist. self.tree.remove_range(key, key) - + def listdir(self, genid, dirname): tree = self.find_generation(genid) try: @@ -476,7 +476,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def set_file_chunks(self, filename, chunkids): tracing.trace('filename=%s', filename) tracing.trace('chunkids=%s', repr(chunkids)) - + file_id = self.set_file_id(filename) minkey = self.fskey(file_id, self.FILE_CHUNKS, 0) maxkey = self.fskey(file_id, self.FILE_CHUNKS, self.SUBKEY_MAX) @@ -510,7 +510,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def chunk_in_use(self, gen_id, chunk_id): '''Is a chunk used by a generation?''' - + minkey = self.chunk_key(chunk_id, 0) maxkey = self.chunk_key(chunk_id, obnamlib.MAX_ID) t = self.find_generation(gen_id) @@ -518,7 +518,7 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def list_chunks_in_generation(self, gen_id): '''Return list of chunk ids used in a given generation.''' - + minkey = self.chunk_key(0, 0) maxkey = self.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) t = self.find_generation(gen_id) @@ -527,14 +527,14 @@ class ClientMetadataTree(obnamlib.RepositoryTree): def set_file_data(self, filename, contents): # pragma: no cover '''Store contents of file, if small, in B-tree instead of chunk. - + The length of the contents should be small enough to fit in a B-tree leaf. - + ''' tracing.trace('filename=%s' % filename) tracing.trace('contents=%s' % repr(contents)) - + file_id = self.set_file_id(filename) key = self.fskey(file_id, self.FILE_DATA, 0) self.tree.insert(key, contents) diff --git a/obnamlib/clientmetadatatree_tests.py b/obnamlib/clientmetadatatree_tests.py index 9027c580..f06f8b78 100644 --- a/obnamlib/clientmetadatatree_tests.py +++ b/obnamlib/clientmetadatatree_tests.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -42,13 +42,13 @@ class ClientMetadataTreeTests(unittest.TestCase): self.file_metadata = obnamlib.Metadata(st_mode=stat.S_IFREG | 0666, st_size=self.file_size) self.file_encoded = obnamlib.encode_metadata(self.file_metadata) - + def tearDown(self): shutil.rmtree(self.tempdir) def test_has_not_current_generation_initially(self): self.assertEqual(self.client.tree, None) - + def test_lists_no_generations_initially(self): self.assertEqual(self.client.list_generations(), []) @@ -56,7 +56,7 @@ class ClientMetadataTreeTests(unittest.TestCase): self.now = 12765 self.client.start_generation() self.assertNotEqual(self.client.tree, None) - + def lookup(x): key = self.client.genkey(x) return self.client._lookup_int(self.client.tree, key) @@ -75,7 +75,7 @@ class ClientMetadataTreeTests(unittest.TestCase): self.now = 2 self.client.start_generation() self.assertNotEqual(self.client.tree, None) - + def lookup(x): key = self.client.genkey(x) return self.client._lookup_int(self.client.tree, key) @@ -238,7 +238,7 @@ class ClientMetadataTreeTests(unittest.TestCase): self.client.create('/foo', self.file_encoded) self.client.commit() - self.assertEqual(self.client.get_generation_data(genid), + self.assertEqual(self.client.get_generation_data(genid), self.file_size) def test_finds_generation_the_first_time(self): @@ -278,7 +278,7 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.client = obnamlib.ClientMetadataTree(fs, 'clientid', obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, - obnamlib.DEFAULT_LRU_SIZE, + obnamlib.DEFAULT_LRU_SIZE, self) # Force use of filename hash collisions. self.client.default_file_id = self.client._bad_default_file_id @@ -288,7 +288,7 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.file_encoded = obnamlib.encode_metadata(self.file_metadata) self.dir_metadata = obnamlib.Metadata(st_mode=stat.S_IFDIR | 0777) self.dir_encoded = obnamlib.encode_metadata(self.dir_metadata) - + def tearDown(self): shutil.rmtree(self.tempdir) @@ -296,12 +296,12 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.assertEqual(self.client.listdir(self.clientid, '/'), []) def test_has_no_metadata_initially(self): - self.assertRaises(KeyError, self.client.get_metadata, self.clientid, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo') def test_sets_metadata(self): self.client.set_metadata('/foo', self.file_encoded) - self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), + self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), self.file_encoded) def test_creates_file_at_root(self): @@ -314,20 +314,20 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.client.create('/foo', self.file_encoded) self.client.remove('/foo') self.assertEqual(self.client.listdir(self.clientid, '/'), []) - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo') def test_creates_directory_at_root(self): self.client.create('/foo', self.dir_encoded) self.assertEqual(self.client.listdir(self.clientid, '/'), ['foo']) - self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), + self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), self.dir_encoded) def test_removes_directory_at_root(self): self.client.create('/foo', self.dir_encoded) self.client.remove('/foo') self.assertEqual(self.client.listdir(self.clientid, '/'), []) - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo') def test_creates_directory_and_files_and_subdirs(self): @@ -336,18 +336,18 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.client.create('/foo/bar', self.dir_encoded) self.client.create('/foo/bar/baz', self.file_encoded) self.assertEqual(self.client.listdir(self.clientid, '/'), ['foo']) - self.assertEqual(sorted(self.client.listdir(self.clientid, '/foo')), + self.assertEqual(sorted(self.client.listdir(self.clientid, '/foo')), ['bar', 'foobar']) - self.assertEqual(self.client.listdir(self.clientid, '/foo/bar'), + self.assertEqual(self.client.listdir(self.clientid, '/foo/bar'), ['baz']) - self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), + self.assertEqual(self.client.get_metadata(self.clientid, '/foo'), self.dir_encoded) - self.assertEqual(self.client.get_metadata(self.clientid, '/foo/bar'), + self.assertEqual(self.client.get_metadata(self.clientid, '/foo/bar'), self.dir_encoded) - self.assertEqual(self.client.get_metadata(self.clientid, '/foo/foobar'), + self.assertEqual(self.client.get_metadata(self.clientid, '/foo/foobar'), self.file_encoded) - self.assertEqual(self.client.get_metadata(self.clientid, - '/foo/bar/baz'), + self.assertEqual(self.client.get_metadata(self.clientid, + '/foo/bar/baz'), self.file_encoded) def test_removes_directory_and_files_and_subdirs(self): @@ -357,13 +357,13 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.client.create('/foo/bar/baz', self.file_encoded) self.client.remove('/foo') self.assertEqual(self.client.listdir(self.clientid, '/'), []) - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo') - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo/foobar') - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo/bar') - self.assertRaises(KeyError, self.client.get_metadata, + self.assertRaises(KeyError, self.client.get_metadata, self.clientid, '/foo/bar/baz') def test_has_no_file_chunks_initially(self): @@ -371,41 +371,41 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): def test_sets_file_chunks(self): self.client.set_file_chunks('/foo', [1, 2, 3]) - self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), + self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), [1, 2, 3]) def test_appends_file_chunks_to_empty_list(self): self.client.append_file_chunks('/foo', [1, 2, 3]) - self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), + self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), [1, 2, 3]) def test_appends_file_chunks_to_nonempty_list(self): self.client.set_file_chunks('/foo', [1, 2, 3]) self.client.append_file_chunks('/foo', [4, 5, 6]) - self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), + self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), [1, 2, 3, 4, 5, 6]) - + def test_generation_has_no_chunk_refs_initially(self): minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), []) - + def test_generation_has_no_chunk_refs_initially(self): minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), []) def test_sets_file_chunks(self): self.client.set_file_chunks('/foo', [1, 2, 3]) - self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), + self.assertEqual(self.client.get_file_chunks(self.clientid, '/foo'), [1, 2, 3]) - + def test_generation_has_no_chunk_refs_initially(self): minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), []) def test_set_file_chunks_adds_chunk_refs(self): @@ -413,7 +413,7 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): file_id = self.client.get_file_id(self.client.tree, '/foo') minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(set(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(set(self.client.tree.lookup_range(minkey, maxkey)), set([(self.client.chunk_key(1, file_id), ''), (self.client.chunk_key(2, file_id), '')])) @@ -423,7 +423,7 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): file_id = self.client.get_file_id(self.client.tree, '/foo') minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), [(self.client.chunk_key(1, file_id), '')]) def test_remove_removes_chunk_refs(self): @@ -431,13 +431,13 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): self.client.remove('/foo') minkey = self.client.chunk_key(0, 0) maxkey = self.client.chunk_key(obnamlib.MAX_ID, obnamlib.MAX_ID) - self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), + self.assertEqual(list(self.client.tree.lookup_range(minkey, maxkey)), []) - + def test_report_chunk_not_in_use_initially(self): gen_id = self.client.get_generation_id(self.client.tree) self.assertFalse(self.client.chunk_in_use(gen_id, 0)) - + def test_report_chunk_in_use_after_it_is(self): gen_id = self.client.get_generation_id(self.client.tree) self.client.set_file_chunks('/foo', [0]) @@ -451,7 +451,7 @@ class ClientMetadataTreeFileOpsTests(unittest.TestCase): gen_id = self.client.get_generation_id(self.client.tree) self.client.set_file_chunks('/foo', [0]) self.client.set_file_chunks('/bar', [1]) - self.assertEqual(set(self.client.list_chunks_in_generation(gen_id)), + self.assertEqual(set(self.client.list_chunks_in_generation(gen_id)), set([0, 1])) def test_lists_chunks_in_generation_only_once(self): diff --git a/obnamlib/encryption.py b/obnamlib/encryption.py index 61cee35b..d4a75f8d 100644 --- a/obnamlib/encryption.py +++ b/obnamlib/encryption.py @@ -1,15 +1,15 @@ # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -27,50 +27,50 @@ def generate_symmetric_key(numbits, filename='/dev/random'): '''Generate a random key of at least numbits for symmetric encryption.''' tracing.trace('numbits=%d', numbits) - + bytes = (numbits + 7) / 8 f = open(filename, 'rb') key = f.read(bytes) f.close() - + return key.encode('hex') class SymmetricKeyCache(object): '''Cache symmetric keys in memory.''' - + def __init__(self): self.clear() - + def get(self, repo, toplevel): if repo in self.repos and toplevel in self.repos[repo]: return self.repos[repo][toplevel] return None - + def put(self, repo, toplevel, key): if repo not in self.repos: self.repos[repo] = {} self.repos[repo][toplevel] = key - + def clear(self): self.repos = {} - - + + def _gpg_pipe(args, data, passphrase): '''Pipe things through gpg. - + With the right args, this can be either an encryption or a decryption operation. - + For safety, we give the passphrase to gpg via a file descriptor. The argument list is modified to include the relevant options for that. - + The data is fed to gpg via a temporary file, readable only by the owner, to avoid congested pipes. - + ''' - + # Open pipe for passphrase, and write it there. If passphrase is # very long (more than 4 KiB by default), this might block. A better # implementation would be to have a loop around select(2) to do pipe @@ -79,30 +79,30 @@ def _gpg_pipe(args, data, passphrase): keypipe = os.pipe() os.write(keypipe[1], passphrase + '\n') os.close(keypipe[1]) - + # Actually run gpg. - - argv = ['gpg', '--passphrase-fd', str(keypipe[0]), '-q', '--batch', + + argv = ['gpg', '--passphrase-fd', str(keypipe[0]), '-q', '--batch', '--no-textmode'] + args tracing.trace('argv=%s', repr(argv)) p = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, err = p.communicate(data) - + os.close(keypipe[0]) - + # Return output data, or deal with errors. if p.returncode: # pragma: no cover raise obnamlib.Error(err) - + return out - - + + def encrypt_symmetric(cleartext, key): '''Encrypt data with symmetric encryption.''' return _gpg_pipe(['-c'], cleartext, key) - - + + def decrypt_symmetric(encrypted, key): '''Decrypt encrypted data with symmetric encryption.''' return _gpg_pipe(['-d'], encrypted, key) @@ -110,23 +110,23 @@ def decrypt_symmetric(encrypted, key): def _gpg(args, stdin='', gpghome=None): '''Run gpg and return its output.''' - + env = dict() env.update(os.environ) if gpghome is not None: env['GNUPGHOME'] = gpghome tracing.trace('gpghome=%s' % gpghome) - + argv = ['gpg', '-q', '--batch', '--no-textmode'] + args tracing.trace('argv=%s', repr(argv)) p = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env) out, err = p.communicate(stdin) - + # Return output data, or deal with errors. if p.returncode: # pragma: no cover raise obnamlib.Error(err) - + return out @@ -139,32 +139,32 @@ def get_public_key(keyid, gpghome=None): class Keyring(object): '''A simplistic representation of GnuPG keyrings. - + Just enough functionality for obnam's purposes. - + ''' - + _keyring_name = 'pubring.gpg' - + def __init__(self, encoded=''): self._encoded = encoded self._gpghome = None self._keyids = None - + def _setup(self): self._gpghome = tempfile.mkdtemp() f = open(self._keyring, 'wb') f.write(self._encoded) f.close() - + def _cleanup(self): shutil.rmtree(self._gpghome) self._gpghome = None - + @property def _keyring(self): return os.path.join(self._gpghome, self._keyring_name) - + def _real_keyids(self): output = self.gpg(False, ['--list-keys', '--with-colons']) @@ -174,27 +174,27 @@ class Keyring(object): if len(fields) >= 5 and fields[0] == 'pub': keyids.append(fields[4]) return keyids - + def keyids(self): if self._keyids is None: self._keyids = self._real_keyids() return self._keyids - + def __str__(self): return self._encoded - + def __contains__(self, keyid): return keyid in self.keyids() - + def _reread_keyring(self): f = open(self._keyring, 'rb') self._encoded = f.read() f.close() self._keyids = None - + def add(self, key): self.gpg(True, ['--import'], stdin=key) - + def remove(self, keyid): self.gpg(True, ['--delete-key', '--yes', keyid]) @@ -216,7 +216,7 @@ class Keyring(object): class SecretKeyring(Keyring): '''Same as Keyring, but for secret keys.''' - + _keyring_name = 'secring.gpg' def _real_keyids(self): @@ -228,22 +228,22 @@ class SecretKeyring(Keyring): if len(fields) >= 5 and fields[0] == 'sec': keyids.append(fields[4]) return keyids - + def encrypt_with_keyring(cleartext, keyring): '''Encrypt data with all keys in a keyring.''' recipients = [] for keyid in keyring.keyids(): recipients += ['-r', keyid] - return keyring.gpg(False, - ['-e', + return keyring.gpg(False, + ['-e', '--trust-model', 'always', '--no-encrypt-to', '--no-default-recipient', ] + recipients, stdin=cleartext) - - + + def decrypt_with_secret_keys(encrypted, gpghome=None): '''Decrypt data using secret keys GnuPG finds on its own.''' return _gpg(['-d'], stdin=encrypted, gpghome=gpghome) diff --git a/obnamlib/encryption_tests.py b/obnamlib/encryption_tests.py index 657898be..72c20df0 100644 --- a/obnamlib/encryption_tests.py +++ b/obnamlib/encryption_tests.py @@ -1,15 +1,15 @@ # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -71,10 +71,10 @@ class SymmetricKeyCacheTests(unittest.TestCase): self.toplevel = 'toplevel' self.key = 'key' self.key2 = 'key2' - + def test_does_not_have_key_initially(self): self.assertEqual(self.cache.get(self.repo, self.toplevel), None) - + def test_remembers_key(self): self.cache.put(self.repo, self.toplevel, self.key) self.assertEqual(self.cache.get(self.repo, self.toplevel), self.key) @@ -82,7 +82,7 @@ class SymmetricKeyCacheTests(unittest.TestCase): def test_does_not_remember_key_for_different_repo(self): self.cache.put(self.repo, self.toplevel, self.key) self.assertEqual(self.cache.get(self.repo2, self.toplevel), None) - + def test_remembers_keys_for_both_repos(self): self.cache.put(self.repo, self.toplevel, self.key) self.cache.put(self.repo2, self.toplevel, self.key2) @@ -102,10 +102,10 @@ class GetPublicKeyTests(unittest.TestCase): self.gpghome = os.path.join(self.dirname, 'gpghome') shutil.copytree('test-gpghome', self.gpghome) self.keyid = '1B321347' - + def tearDown(self): shutil.rmtree(self.dirname) - + def test_exports_key(self): key = obnamlib.get_public_key(self.keyid, gpghome=self.gpghome) self.assert_('-----BEGIN PGP PUBLIC KEY BLOCK-----' in key) @@ -145,17 +145,17 @@ uWUO7gMi+AlnxbfXVCTEgw3xhg== def test_gets_no_keys_from_empty_encoded(self): keyring = obnamlib.Keyring(encoded='') self.assertEqual(keyring.keyids(), []) - + def test_adds_key(self): self.keyring.add(self.key) self.assertEqual(self.keyring.keyids(), [self.keyid]) self.assert_(self.keyid in self.keyring) - + def test_removes_key(self): self.keyring.add(self.key) self.keyring.remove(self.keyid) self.assertEqual(self.keyring.keyids(), []) - + def test_export_import_roundtrip_works(self): self.keyring.add(self.key) exported = str(self.keyring) @@ -179,10 +179,10 @@ class PublicKeyEncryptionTests(unittest.TestCase): passphrase = 'password1' keyring = obnamlib.Keyring(cat('test-gpghome/pubring.gpg')) seckeys = obnamlib.SecretKeyring(cat('test-gpghome/secring.gpg')) - + encrypted = obnamlib.encrypt_with_keyring(cleartext, keyring) decrypted = obnamlib.decrypt_with_secret_keys(encrypted, gpghome='test-gpghome') - + self.assertEqual(decrypted, cleartext) diff --git a/obnamlib/forget_policy.py b/obnamlib/forget_policy.py index bb776846..8b8b2c6b 100644 --- a/obnamlib/forget_policy.py +++ b/obnamlib/forget_policy.py @@ -22,11 +22,11 @@ import obnamlib class ForgetPolicy(object): '''Parse and interpret a policy for what to forget and what to keep. - + See documentation for the --keep option for details. - + ''' - + periods = { 'h': 'hourly', 'd': 'daily', @@ -34,18 +34,18 @@ class ForgetPolicy(object): 'm': 'monthly', 'y': 'yearly', } - + rule_pat = re.compile(r'(?P<count>\d+)(?P<period>(h|d|w|m|y))') - + def parse(self, optarg): '''Parse the argument of --keep. - + Return a dictionary indexed by 'hourly', 'daily', 'weekly', 'monthly', 'yearly', and giving the number of generations to keep for each time period. - + ''' - + remaining = optarg m = self.rule_pat.match(remaining) if not m: @@ -57,7 +57,7 @@ class ForgetPolicy(object): period = self.periods[m.group('period')] if result[period] is not None: raise obnamlib.Error('Forget policy may not ' - 'duplicate period (%s): %s' % + 'duplicate period (%s): %s' % (period, optarg)) result[period] = count remaining = remaining[m.end():] @@ -68,7 +68,7 @@ class ForgetPolicy(object): 'separated by commas: %s' % optarg) remaining = remaining[1:] m = self.rule_pat.match(remaining) - + result.update((x, 0) for x, y in result.iteritems() if y is None) return result @@ -80,7 +80,7 @@ class ForgetPolicy(object): 'monthly': '%Y-%m', 'yearly': '%Y', } - + matches = [] for genid, dt in genlist: formatted = dt.strftime(formats[period]) @@ -94,18 +94,18 @@ class ForgetPolicy(object): def match(self, rules, genlist): '''Match a parsed ruleset against a list of generations and times. - + The ruleset should be of the form returned by the parse method. - + genlist should be a list of generation identifiers and timestamps. Identifiers can be anything, timestamps should be an instance of datetime.datetime, with no time zone (it is ignored). - + genlist should be in ascending order by time: oldest one first. - + Return value is all those pairs from genlist that should be kept (i.e., which match the rules). - + ''' result_ids = set() diff --git a/obnamlib/hooks.py b/obnamlib/hooks.py index 34e45a06..57779bec 100644 --- a/obnamlib/hooks.py +++ b/obnamlib/hooks.py @@ -43,10 +43,10 @@ class Hook(object): def __init__(self): self.callbacks = [] self.priorities = {} - + def add_callback(self, callback, priority=DEFAULT_PRIORITY): '''Add a callback to this hook. - + Return an identifier that can be used to remove this callback. ''' @@ -54,16 +54,16 @@ class Hook(object): if callback not in self.callbacks: self.priorities[callback] = priority self.callbacks.append(callback) - self.callbacks.sort(lambda x,y: cmp(self.priorities[x], + self.callbacks.sort(lambda x,y: cmp(self.priorities[x], self.priorities[y])) - + return callback - + def call_callbacks(self, *args, **kwargs): '''Call all callbacks with the given arguments.''' for callback in self.callbacks: callback(*args, **kwargs) - + def remove_callback(self, callback_id): '''Remove a specific callback.''' if callback_id in self.callbacks: @@ -85,16 +85,16 @@ class MissingFilterError(obnamlib.Error): class FilterHook(Hook): '''A hook which filters data through callbacks. - + Every hook of this type accepts a piece of data as its first argument Each callback gets the return value of the previous one as its argument. The caller gets the value of the final callback. - + Other arguments (with or without keywords) are passed as-is to each callback. - + ''' - + def __init__(self): Hook.__init__(self) self.bytag = {} @@ -112,7 +112,7 @@ class FilterHook(Hook): def call_callbacks(self, data, *args, **kwargs): raise NotImplementedError() - + def run_filter_read(self, data, *args, **kwargs): tag, content = data.split("\0", 1) while tag != "": @@ -140,16 +140,16 @@ class FilterHook(Hook): class HookManager(object): '''Manage the set of hooks the application defines.''' - + def __init__(self): self.hooks = {} self.filters = {} - + def new(self, name): '''Create a new hook. - + If a hook with that name already exists, nothing happens. - + ''' if name not in self.hooks: @@ -166,14 +166,14 @@ class HookManager(object): return self.hooks[name].add_callback(callback, priority) else: return self.filters[name].add_callback(callback, priority) - + def remove_callback(self, name, callback_id): '''Remove a specific callback from a named hook.''' if name in self.hooks: self.hooks[name].remove_callback(callback_id) else: self.filters[name].remove_callback(callback_id) - + def call(self, name, *args, **kwargs): '''Call callbacks for a named hook, using given arguments.''' self.hooks[name].call_callbacks(*args, **kwargs) diff --git a/obnamlib/hooks_tests.py b/obnamlib/hooks_tests.py index 8c9a0a70..8ba5a594 100644 --- a/obnamlib/hooks_tests.py +++ b/obnamlib/hooks_tests.py @@ -24,37 +24,37 @@ class HookTests(unittest.TestCase): def setUp(self): self.hook = obnamlib.Hook() - + def callback(self, *args, **kwargs): self.args = args self.kwargs = kwargs - + def callback2(self, *args, **kwargs): self.args2 = args self.kwargs2 = kwargs def test_has_no_callbacks_by_default(self): self.assertEqual(self.hook.callbacks, []) - + def test_adds_callback(self): self.hook.add_callback(self.callback) self.assertEqual(self.hook.callbacks, [self.callback]) - + def test_adds_callback_only_once(self): self.hook.add_callback(self.callback) self.hook.add_callback(self.callback) self.assertEqual(self.hook.callbacks, [self.callback]) - + def test_adds_two_callbacks(self): id1 = self.hook.add_callback(self.callback) - id2 = self.hook.add_callback(self.callback2, + id2 = self.hook.add_callback(self.callback2, obnamlib.Hook.DEFAULT_PRIORITY + 1) self.assertEqual(self.hook.callbacks, [self.callback, self.callback2]) self.assertNotEqual(id1, id2) - + def test_adds_callbacks_in_reverse_order(self): id1 = self.hook.add_callback(self.callback) - id2 = self.hook.add_callback(self.callback2, + id2 = self.hook.add_callback(self.callback2, obnamlib.Hook.DEFAULT_PRIORITY - 1) self.assertEqual(self.hook.callbacks, [self.callback2, self.callback]) self.assertNotEqual(id1, id2) @@ -115,7 +115,7 @@ class FilterHookTests(unittest.TestCase): def test_never_filter_no_tags(self): self.hook.add_callback(NeverAddsFilter()) self.assertEquals(self.hook.run_filter_write("foo"), "\0foo") - + def test_never_filter_clean_revert(self): self.hook.add_callback(NeverAddsFilter()) self.assertEquals(self.hook.run_filter_read("\0foo"), "foo") @@ -152,7 +152,7 @@ class HookManagerTests(unittest.TestCase): def setUp(self): self.hooks = obnamlib.HookManager() self.hooks.new('foo') - + def callback(self, *args, **kwargs): self.args = args self.kwargs = kwargs @@ -160,14 +160,14 @@ class HookManagerTests(unittest.TestCase): def test_has_no_tests_initially(self): hooks = obnamlib.HookManager() self.assertEqual(hooks.hooks, {}) - + def test_adds_new_hook(self): self.assert_(self.hooks.hooks.has_key('foo')) - + def test_adds_new_filter_hook(self): self.hooks.new_filter('bar') self.assert_('bar' in self.hooks.filters) - + def test_adds_callback(self): self.hooks.add_callback('foo', self.callback) self.assertEqual(self.hooks.hooks['foo'].callbacks, [self.callback]) diff --git a/obnamlib/lockmgr.py b/obnamlib/lockmgr.py index 47f1f38d..87f148a7 100644 --- a/obnamlib/lockmgr.py +++ b/obnamlib/lockmgr.py @@ -1,15 +1,15 @@ # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -23,7 +23,7 @@ import obnamlib class LockManager(object): '''Lock and unlock sets of directories at once.''' - + def __init__(self, fs, timeout, client): self._fs = fs self.timeout = timeout @@ -44,10 +44,10 @@ class LockManager(object): def _time(self): # pragma: no cover return time.time() - + def _sleep(self): # pragma: no cover time.sleep(1) - + def sort(self, dirnames): def bytelist(s): return [ord(s) for s in str(s)] @@ -56,7 +56,7 @@ class LockManager(object): def _lockname(self, dirname): return os.path.join(dirname, 'lock') - + def _lock_one(self, dirname): started = self._time() while True: @@ -69,10 +69,10 @@ class LockManager(object): else: return self._sleep() - + def _unlock_one(self, dirname): self._fs.unlock(self._lockname(dirname)) - + def lock(self, dirnames): '''Lock ALL the directories.''' we_locked = [] diff --git a/obnamlib/lockmgr_tests.py b/obnamlib/lockmgr_tests.py index c1a731f9..4d69139c 100644 --- a/obnamlib/lockmgr_tests.py +++ b/obnamlib/lockmgr_tests.py @@ -1,15 +1,15 @@ # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -44,10 +44,10 @@ class LockManagerTests(unittest.TestCase): self.lm = obnamlib.LockManager(self.fs, self.timeout, '') self.lm._time = self.fake_time self.lm._sleep = lambda: None - + def tearDown(self): shutil.rmtree(self.tempdir) - + def test_has_nothing_locked_initially(self): for dirname in self.dirnames: self.assertFalse(self.locked(dirname)) diff --git a/obnamlib/metadata.py b/obnamlib/metadata.py index bba229fe..94b87a73 100644 --- a/obnamlib/metadata.py +++ b/obnamlib/metadata.py @@ -27,12 +27,12 @@ import obnamlib metadata_verify_fields = ( - 'st_mode', 'st_mtime_sec', 'st_mtime_nsec', + 'st_mode', 'st_mtime_sec', 'st_mtime_nsec', 'st_nlink', 'st_size', 'st_uid', 'groupname', 'username', 'target', 'xattr', ) metadata_fields = metadata_verify_fields + ( - 'st_blocks', 'st_dev', 'st_gid', 'st_ino', 'st_atime_sec', + 'st_blocks', 'st_dev', 'st_gid', 'st_ino', 'st_atime_sec', 'st_atime_nsec', 'md5', ) @@ -40,16 +40,16 @@ metadata_fields = metadata_verify_fields + ( class Metadata(object): '''Represent metadata for a filesystem entry. - + The metadata for a filesystem entry (file, directory, device, ...) consists of its stat(2) result, plus ACL and xattr. - + This class represents them as fields. - + We do not store all stat(2) fields. Here's a commentary on all fields: - + field? stored? why - + st_atime_sec yes mutt compares atime, mtime to see ifmsg is new st_atime_nsec yes mutt compares atime, mtime to see ifmsg is new st_blksize no no way to restore, not useful backed up @@ -67,18 +67,18 @@ class Metadata(object): st_uid yes used to restored ownership The field 'target' stores the target of a symlink. - + Additionally, the fields 'groupname' and 'username' are stored. They contain the textual names that correspond to st_gid and st_uid. When restoring, the names will be preferred by default. - + The 'md5' field optionally stores the whole-file checksum for the file. - + The 'xattr' field optionally stores extended attributes encoded as a binary blob. - + ''' - + def __init__(self, **kwargs): for field in metadata_fields: setattr(self, field, None) @@ -121,7 +121,7 @@ def _cached_getpwuid(uid): # pragma: no cover if uid not in _uid_to_username: _uid_to_username[uid] = pwd.getpwuid(uid) return _uid_to_username[uid] - + _gid_to_groupname = {} def _cached_getgrgid(gid): # pragma: no cover if gid not in _gid_to_groupname: @@ -131,7 +131,7 @@ def _cached_getgrgid(gid): # pragma: no cover def get_xattrs_as_blob(fs, filename): # pragma: no cover tracing.trace('filename=%s' % filename) - + try: names = fs.llistxattr(filename) except (OSError, IOError), e: @@ -172,7 +172,7 @@ def get_xattrs_as_blob(fs, filename): # pragma: no cover fmt = '!' + 'Q' * len(values) value_blob = struct.pack(fmt, *lengths) + ''.join(values) - return ('%s%s%s' % + return ('%s%s%s' % (struct.pack('!Q', len(name_blob)), name_blob, value_blob)) @@ -188,7 +188,7 @@ def set_xattrs_from_blob(fs, filename, blob): # pragma: no cover fmt = '!' + 'Q' * len(names) lengths_size = sizesize * len(names) lengths = struct.unpack(fmt, value_blob[:lengths_size]) - + pos = lengths_size for i, name in enumerate(names): value = value_blob[pos:pos + lengths[i]] @@ -235,7 +235,7 @@ def set_metadata(fs, filename, metadata, getuid=None): fields: we assume the caller will change st_uid, st_gid accordingly if they want to mess with things. This makes the user take care of error situations and looking up user preferences. - + ''' symlink = stat.S_ISLNK(metadata.st_mode) @@ -258,10 +258,10 @@ def set_metadata(fs, filename, metadata, getuid=None): if metadata.xattr: # pragma: no cover set_xattrs_from_blob(fs, filename, metadata.xattr) - fs.lutimes(filename, metadata.st_atime_sec, metadata.st_atime_nsec, + fs.lutimes(filename, metadata.st_atime_sec, metadata.st_atime_nsec, metadata.st_mtime_sec, metadata.st_mtime_nsec) - - + + metadata_format = struct.Struct('!Q' + # flags 'Q' + # st_mode 'qQ' + # st_mtime_sec and _nsec @@ -325,7 +325,7 @@ def encode_metadata(metadata): logging.error('ERROR: md5=%s' % repr(metadata.md5)) logging.error('ERROR: xattr=%s' % repr(metadata.xattr)) raise - return (packed + + return (packed + (metadata.groupname or '') + (metadata.username or '') + (metadata.target or '') + @@ -339,7 +339,7 @@ def decode_metadata(encoded): flags = items[0] pos = [1, metadata_format.size] metadata = obnamlib.Metadata() - + def is_present(field): i = obnamlib.metadata_fields.index(field) return (flags & (1 << i)) != 0 @@ -357,7 +357,7 @@ def decode_metadata(encoded): def decode_string(field): decode(field, 1, True, lambda i, o: encoded[o:o + items[i]]) - + decode_integer('st_mode') decode_integer('st_mtime_sec') decode_integer('st_mtime_nsec') @@ -375,6 +375,6 @@ def decode_metadata(encoded): decode_string('target') decode_string('md5') decode_string('xattr') - + return metadata diff --git a/obnamlib/metadata_tests.py b/obnamlib/metadata_tests.py index 286a86b9..f77abe1b 100644 --- a/obnamlib/metadata_tests.py +++ b/obnamlib/metadata_tests.py @@ -48,15 +48,15 @@ class FakeFS(object): return 'target' def getpwuid(self, uid): - return (self.username, None, self.st_uid, self.st_gid, + return (self.username, None, self.st_uid, self.st_gid, None, None, None) - + def getgrgid(self, gid): return (self.groupname, None, self.st_gid, None) def fail_getpwuid(self, uid): raise KeyError(uid) - + def fail_getgrgid(self, gid): raise KeyError(gid) @@ -131,11 +131,11 @@ class ReadMetadataTests(unittest.TestCase): self.fakefs = FakeFS() def test_returns_stat_fields_correctly(self): - metadata = obnamlib.read_metadata(self.fakefs, 'foo', + metadata = obnamlib.read_metadata(self.fakefs, 'foo', getpwuid=self.fakefs.getpwuid, getgrgid=self.fakefs.getgrgid) - fields = ['st_atime_sec','st_atime_nsec', 'st_blocks', 'st_dev', - 'st_gid', 'st_ino', 'st_mode', 'st_mtime_sec', + fields = ['st_atime_sec','st_atime_nsec', 'st_blocks', 'st_dev', + 'st_gid', 'st_ino', 'st_mode', 'st_mtime_sec', 'st_mtime_nsec', 'st_nlink', 'st_size', 'st_uid', 'groupname', 'username'] for field in fields: @@ -145,7 +145,7 @@ class ReadMetadataTests(unittest.TestCase): def test_returns_symlink_fields_correctly(self): self.fakefs.st_mode |= stat.S_IFLNK; - metadata = obnamlib.read_metadata(self.fakefs, 'foo', + metadata = obnamlib.read_metadata(self.fakefs, 'foo', getpwuid=self.fakefs.getpwuid, getgrgid=self.fakefs.getgrgid) fields = ['st_mode', 'target'] @@ -172,10 +172,10 @@ class SetMetadataTests(unittest.TestCase): self.metadata.st_mtime_nsec = 0 self.metadata.st_uid = 1234 self.metadata.st_gid = 5678 - + fd, self.filename = tempfile.mkstemp() os.close(fd) - + self.fs = obnamlib.LocalFS('/') self.fs.connect() @@ -184,9 +184,9 @@ class SetMetadataTests(unittest.TestCase): self.fs.lchown = self.fake_lchown obnamlib.set_metadata(self.fs, self.filename, self.metadata) - + self.st = os.stat(self.filename) - + def tearDown(self): self.fs.close() os.remove(self.filename) @@ -194,7 +194,7 @@ class SetMetadataTests(unittest.TestCase): def fake_lchown(self, filename, uid, gid): self.uid_set = uid self.gid_set = gid - + def test_sets_atime(self): self.assertEqual(self.st.st_atime, self.metadata.st_atime_sec) @@ -247,22 +247,22 @@ class MetadataCodingTests(unittest.TestCase): self.assertEqual( value1, value2, - 'attribute %s must be equal (%s vs %s)' % + 'attribute %s must be equal (%s vs %s)' % (name, value1, value2)) def test_round_trip(self): - metadata = obnamlib.metadata.Metadata(st_mode=1, - st_mtime_sec=2, + metadata = obnamlib.metadata.Metadata(st_mode=1, + st_mtime_sec=2, st_mtime_nsec=12756, st_nlink=3, - st_size=4, - st_uid=5, - st_blocks=6, + st_size=4, + st_uid=5, + st_blocks=6, st_dev=7, - st_gid=8, - st_ino=9, - st_atime_sec=10, - st_atime_nsec=123, + st_gid=8, + st_ino=9, + st_atime_sec=10, + st_atime_nsec=123, groupname='group', username='user', target='target', @@ -288,13 +288,13 @@ class MetadataCodingTests(unittest.TestCase): st_mtime_sec=signed_max, st_mtime_nsec=unsigned_max, st_nlink=unsigned_max, - st_size=signed_max, - st_uid=unsigned_max, - st_blocks=signed_max, + st_size=signed_max, + st_uid=unsigned_max, + st_blocks=signed_max, st_dev=unsigned_max, - st_gid=unsigned_max, - st_ino=unsigned_max, - st_atime_sec=signed_max, + st_gid=unsigned_max, + st_ino=unsigned_max, + st_atime_sec=signed_max, st_atime_nsec=unsigned_max) encoded = obnamlib.encode_metadata(metadata) decoded = obnamlib.decode_metadata(encoded) diff --git a/obnamlib/pluginbase.py b/obnamlib/pluginbase.py index f034c47f..3710de52 100644 --- a/obnamlib/pluginbase.py +++ b/obnamlib/pluginbase.py @@ -23,4 +23,4 @@ class ObnamPlugin(obnamlib.pluginmgr.Plugin): def __init__(self, app): self.app = app - + diff --git a/obnamlib/pluginbase_tests.py b/obnamlib/pluginbase_tests.py index 6d225ac9..10d03173 100644 --- a/obnamlib/pluginbase_tests.py +++ b/obnamlib/pluginbase_tests.py @@ -23,7 +23,7 @@ class FakeApp(object): def __init__(self): self.hooks = self - + class ObnamPluginTests(unittest.TestCase): def setUp(self): diff --git a/obnamlib/pluginmgr.py b/obnamlib/pluginmgr.py index ff0587f6..a8aec5d6 100644 --- a/obnamlib/pluginmgr.py +++ b/obnamlib/pluginmgr.py @@ -1,15 +1,15 @@ # Copyright (C) 2009 Lars Wirzenius <liw@liw.fi> -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -33,7 +33,7 @@ import os class Plugin(object): '''Base class for plugins. - + A plugin MUST NOT have any side effects when it is instantiated. This is necessary so that it can be safely loaded by unit tests, and so that a user interface can allow the user to disable it, @@ -41,73 +41,73 @@ class Plugin(object): that would normally happen should occur in the enable() method, and be undone by the disable() method. These methods must be callable any number of times. - + The subclass MAY define the following attributes: - + * name * description * version * required_application_version - + name is the user-visible identifier for the plugin. It defaults to the plugin's classname. - + description is the user-visible description of the plugin. It may be arbitrarily long, and can use pango markup language. Defaults to the empty string. - + version is the plugin version. Defaults to '0.0.0'. It MUST be a sequence of integers separated by periods. If several plugins with the same name are found, the newest version is used. Versions are - compared integer by integer, starting with the first one, and a - missing integer treated as a zero. If two plugins have the same + compared integer by integer, starting with the first one, and a + missing integer treated as a zero. If two plugins have the same version, either might be used. - - required_application_version gives the version of the minimal + + required_application_version gives the version of the minimal application version the plugin is written for. The first integer must match exactly: if the application is version 2.3.4, the plugin's required_application_version must be at least 2 and at most 2.3.4 to be loaded. Defaults to 0. - + ''' - + @property def name(self): return self.__class__.__name__ - + @property def description(self): return '' - + @property def version(self): return '0.0.0' - + @property def required_application_version(self): return '0.0.0' - + def enable_wrapper(self): '''Enable plugin. - + The plugin manager will call this method, which then calls the enable method. Plugins should implement the enable method. The wrapper method is there to allow an application to provide an extended base class that does some application specific magic when plugins are enabled or disabled. - + ''' - + self.enable() def disable_wrapper(self): '''Corresponds to enable_wrapper, but for disabling a plugin.''' self.disable() - + def enable(self): '''Enable the plugin.''' raise NotImplemented() - + def disable(self): '''Disable the plugin.''' raise NotImplemented() @@ -116,22 +116,22 @@ class Plugin(object): class PluginManager(object): '''Manage plugins. - + This class finds and loads plugins, and keeps a list of them that can be accessed in various ways. - + The locations are set via the locations attribute, which is a list. - + When a plugin is loaded, an instance of its class is created. This instance is initialized using normal and keyword arguments specified - in the plugin manager attributes plugin_arguments and + in the plugin manager attributes plugin_arguments and plugin_keyword_arguments. - + The version of the application using the plugin manager is set via the application_version attribute. This defaults to '0.0.0'. - + ''' - + suffix = '_plugin.py' def __init__(self): @@ -162,14 +162,14 @@ class PluginManager(object): def find_plugin_files(self): '''Find files that may contain plugins. - + This finds all files named *_plugin.py in all locations. The returned list is sorted. - + ''' - + pathnames = [] - + for location in self.locations: try: basenames = os.listdir(location) @@ -179,14 +179,14 @@ class PluginManager(object): s = os.path.join(location, basename) if s.endswith(self.suffix) and os.path.exists(s): pathnames.append(s) - + return sorted(pathnames) def load_plugins(self): '''Load plugins from all plugin files.''' - + plugins = dict() - + for pathname in self.plugin_files: for plugin in self.load_plugin_file(pathname): if plugin.name in plugins: @@ -207,10 +207,10 @@ class PluginManager(object): name, ext = os.path.splitext(os.path.basename(pathname)) f = file(pathname, 'r') - module = imp.load_module(name, f, pathname, + module = imp.load_module(name, f, pathname, ('.py', 'r', imp.PY_SOURCE)) f.close() - + plugins = [] for dummy, member in inspect.getmembers(module, inspect.isclass): if issubclass(member, Plugin): @@ -218,37 +218,37 @@ class PluginManager(object): **self.plugin_keyword_arguments) if self.compatible_version(p.required_application_version): plugins.append(p) - + return plugins def compatible_version(self, required_application_version): '''Check that the plugin is version-compatible with the application. - + This checks the plugin's required_application_version against the declared application version and returns True if they are compatible, and False if not. - + ''' req = self.parse_version(required_application_version) app = self.parse_version(self.application_version) - + return app[0] == req[0] and app >= req def parse_version(self, version): '''Parse a string represenation of a version into list of ints.''' - + return [int(s) for s in version.split('.')] def enable_plugins(self, plugins=None): '''Enable all or selected plugins.''' - + for plugin in plugins or self.plugins: plugin.enable_wrapper() def disable_plugins(self, plugins=None): '''Disable all or selected plugins.''' - + for plugin in plugins or self.plugins: plugin.disable_wrapper() diff --git a/obnamlib/pluginmgr_tests.py b/obnamlib/pluginmgr_tests.py index 82d8a6d1..12f9ca44 100644 --- a/obnamlib/pluginmgr_tests.py +++ b/obnamlib/pluginmgr_tests.py @@ -1,15 +1,15 @@ # Copyright (C) 2009 Lars Wirzenius <liw@liw.fi> -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -23,7 +23,7 @@ class PluginTests(unittest.TestCase): def setUp(self): self.plugin = Plugin() - + def test_name_is_class_name(self): self.assertEqual(self.plugin.name, 'Plugin') @@ -57,7 +57,7 @@ class PluginManagerInitialStateTests(unittest.TestCase): def setUp(self): self.pm = PluginManager() - + def test_locations_is_empty_list(self): self.assertEqual(self.pm.locations, []) @@ -72,7 +72,7 @@ class PluginManagerInitialStateTests(unittest.TestCase): def test_plugin_arguments_is_empty(self): self.assertEqual(self.pm.plugin_arguments, []) - + def test_plugin_keyword_arguments_is_empty(self): self.assertEqual(self.pm.plugin_keyword_arguments, {}) @@ -84,7 +84,7 @@ class PluginManagerTests(unittest.TestCase): self.pm.locations = ['test-plugins', 'not-exist'] self.pm.plugin_arguments = ('fooarg',) self.pm.plugin_keyword_arguments = { 'bar': 'bararg' } - + self.files = sorted(['test-plugins/hello_plugin.py', 'test-plugins/aaa_hello_plugin.py', 'test-plugins/oldhello_plugin.py', @@ -133,19 +133,19 @@ class PluginManagerCompatibleApplicationVersionTests(unittest.TestCase): def setUp(self): self.pm = PluginManager() self.pm.application_version = '1.2.3' - + def test_rejects_zero(self): self.assertFalse(self.pm.compatible_version('0')) - + def test_rejects_two(self): self.assertFalse(self.pm.compatible_version('2')) - + def test_rejects_one_two_four(self): self.assertFalse(self.pm.compatible_version('1.2.4')) - + def test_accepts_one(self): self.assert_(self.pm.compatible_version('1')) - + def test_accepts_one_two_three(self): self.assert_(self.pm.compatible_version('1.2.3')) diff --git a/obnamlib/plugins/backup_plugin.py b/obnamlib/plugins/backup_plugin.py index e04fea19..9aa71295 100644 --- a/obnamlib/plugins/backup_plugin.py +++ b/obnamlib/plugins/backup_plugin.py @@ -33,10 +33,10 @@ import larch class ChunkidPool(object): '''Checksum/chunkid mappings that are pending an upload to shared trees.''' - + def __init__(self): self.clear() - + def add(self, chunkid, checksum): if checksum not in self._mapping: self._mapping[checksum] = [] @@ -47,10 +47,10 @@ class ChunkidPool(object): def get(self, checksum): return self._mapping.get(checksum, []) - + def clear(self): self._mapping = {} - + def __iter__(self): for checksum in self._mapping.keys(): for chunkid in self._mapping[checksum]: @@ -62,11 +62,11 @@ class BackupPlugin(obnamlib.ObnamPlugin): def enable(self): backup_group = obnamlib.option_group['backup'] = 'Backing up' perf_group = obnamlib.option_group['perf'] - + self.app.add_subcommand('backup', self.backup, arg_synopsis='[DIRECTORY]...') self.app.settings.string_list(['root'], 'what to backup') - self.app.settings.string_list(['exclude'], + self.app.settings.string_list(['exclude'], 'regular expression for pathnames to ' 'exclude from backup (can be used multiple ' 'times)', @@ -160,7 +160,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): (1024**1, 'KiB'), (0, 'B') ] - + for size_base, size_unit in size_table: if self.uploaded_bytes >= size_base: if size_base > 0: @@ -199,7 +199,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): logging.info('Backup performance statistics:') logging.info('* files found: %s' % self.file_count) logging.info('* files backed up: %s' % self.backed_up_count) - logging.info('* uploaded data: %s bytes (%s %s)' % + logging.info('* uploaded data: %s bytes (%s %s)' % (self.uploaded_bytes, size_amount, size_unit)) logging.info('* duration: %s s' % duration) logging.info('* average speed: %s %s' % (speed_amount, speed_unit)) @@ -215,7 +215,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): logging.error(msg) if exc: logging.error(repr(exc)) - + # FIXME: ttystatus.TerminalStatus.error is quiet if --quiet is used. # That's a bug, so we work around it by writing to stderr directly. sys.stderr.write('ERROR: %s\n' % msg) @@ -224,7 +224,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): p = obnamlib.ByteSizeParser() p.set_default_unit('MiB') return p.parse(value) - + @property def pretend(self): return self.app.settings['pretend'] @@ -237,12 +237,12 @@ class BackupPlugin(obnamlib.ObnamPlugin): self.app.settings.require('repository') self.app.settings.require('client-name') - + if not self.app.settings['repository']: raise obnamlib.Error('No --repository setting. ' 'You need to specify it on the command ' 'line or a configuration file.') - + # This is ugly, but avoids having to update the dependency on # ttystatus yet again. if not hasattr(self.app.ts, 'flush'): @@ -266,7 +266,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): self.add_client(client_name) self.what('locking client') self.repo.lock_client(client_name) - + # Need to lock the shared stuff briefly, so encryption etc # gets initialized. self.what('initialising encryption for shared directories') @@ -342,10 +342,10 @@ class BackupPlugin(obnamlib.ObnamPlugin): self.repo.lock_root() if client_name not in self.repo.list_clients(): tracing.trace('adding new client %s' % client_name) - tracing.trace('client list before adding: %s' % + tracing.trace('client list before adding: %s' % self.repo.list_clients()) self.repo.add_client(client_name) - tracing.trace('client list after adding: %s' % + tracing.trace('client list after adding: %s' % self.repo.list_clients()) self.repo.commit_root() self.repo = self.app.open_repository(repofs=self.repo.fs.fs) @@ -377,7 +377,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): self.what('determining absolute path for %s' % root) self.fs.reinit(root) absroots.append(self.fs.abspath('.')) - + if not self.pretend: self.remove_old_roots(absroots) @@ -389,7 +389,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): logging.info('Backing up root %s' % root) self.what('connecting to live data %s' % root) self.fs.reinit(root) - + self.what('scanning for files in %s' % root) absroot = self.fs.abspath('.') self.root_metadata = self.fs.lstat(absroot) @@ -429,7 +429,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): def maybe_simulate_error(self, pathname): '''Raise an IOError if specified by --testing-fail-matching.''' - + for pattern in self.app.settings['testing-fail-matching']: if re.search(pattern, pathname): e = errno.ENOENT @@ -467,12 +467,12 @@ class BackupPlugin(obnamlib.ObnamPlugin): def find_files(self, root): '''Find all files and directories that need to be backed up. - + This is a generator. It yields (pathname, metadata) pairs. - + The caller should not recurse through directories, just backup the directory itself (name, metadata, file list). - + ''' for pathname, st in self.fs.scan_tree(root, ok=self.can_be_backed_up): @@ -494,7 +494,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): def can_be_backed_up(self, pathname, st): if self.app.settings['one-file-system']: - if st.st_dev != self.root_metadata.st_dev: + if st.st_dev != self.root_metadata.st_dev: logging.debug('Excluding (one-file-system): %s' % pathname) return False @@ -515,12 +515,12 @@ class BackupPlugin(obnamlib.ObnamPlugin): if data == tag_contents: logging.debug('Excluding (cache dir): %s' % pathname) return False - + return True def needs_backup(self, pathname, current): '''Does a given file need to be backed up?''' - + # Directories always require backing up so that backup_dir_contents # can remove stuff that no longer exists from them. if current.isdir(): @@ -581,7 +581,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): def backup_metadata(self, pathname, metadata): '''Back up metadata for a filesystem object''' - + tracing.trace('backup_metadata: %s', pathname) if not self.pretend: self.repo.create(pathname, metadata) @@ -604,7 +604,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): summer = self.repo.new_checksummer() max_intree = self.app.settings['node-size'] / 4 - if (metadata.st_size <= max_intree and + if (metadata.st_size <= max_intree and self.app.settings['small-files-in-btree']): contents = f.read() assert len(contents) <= max_intree # FIXME: silly error checking @@ -634,13 +634,13 @@ class BackupPlugin(obnamlib.ObnamPlugin): chunkids = [] else: self.update_progress_with_upload(len(data)) - + if not self.pretend and self.time_for_checkpoint(): logging.debug('making checkpoint in the middle of a file') self.repo.append_file_chunks(filename, chunkids) chunkids = [] self.make_checkpoint() - + tracing.trace('closing file') f.close() if chunkids: @@ -651,7 +651,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): filename) tracing.trace('done backing up file contents') return summer.digest() - + def backup_file_chunk(self, data): '''Back up a chunk of data by putting it into the repository.''' @@ -673,7 +673,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): def put(): self.update_progress_with_upload(len(data)) return self.repo.put_chunk_only(data) - + def share(chunkid): self.chunkid_pool.add(chunkid, checksum) @@ -727,14 +727,14 @@ class BackupPlugin(obnamlib.ObnamPlugin): def remove_old_roots(self, new_roots): '''Remove from started generation anything that is not a backup root. - - We recurse from filesystem root directory until getting to one of - the new backup roots, or a directory or file that is not a parent + + We recurse from filesystem root directory until getting to one of + the new backup roots, or a directory or file that is not a parent of one of the new backup roots. We remove anything that is not a new backup root, or their parent. - + ''' - + def is_parent(pathname): if not pathname.endswith(os.sep): pathname += os.sep @@ -742,7 +742,7 @@ class BackupPlugin(obnamlib.ObnamPlugin): if new_root.startswith(pathname): return True return False - + def helper(dirname): if dirname in new_roots: tracing.trace('is a new root: %s' % dirname) diff --git a/obnamlib/plugins/compression_plugin.py b/obnamlib/plugins/compression_plugin.py index 41d1eefd..20eed384 100644 --- a/obnamlib/plugins/compression_plugin.py +++ b/obnamlib/plugins/compression_plugin.py @@ -29,7 +29,7 @@ class DeflateCompressionFilter(object): def filter_read(self, data, repo, toplevel): return zlib.decompress(data) - + def filter_write(self, data, repo, toplevel): how = self.app.settings['compress-with'] if how == 'deflate': @@ -42,7 +42,7 @@ class DeflateCompressionFilter(object): data = zlib.compress(data) return data - + class CompressionPlugin(obnamlib.ObnamPlugin): @@ -52,7 +52,7 @@ class CompressionPlugin(obnamlib.ObnamPlugin): 'use PROGRAM to compress repository with ' '(one of none, deflate)', metavar='PROGRAM') - + hooks = [ ('repository-data', DeflateCompressionFilter(self.app), obnamlib.Hook.EARLY_PRIORITY), diff --git a/obnamlib/plugins/convert5to6_plugin.py b/obnamlib/plugins/convert5to6_plugin.py index 6a397785..cc5515a4 100644 --- a/obnamlib/plugins/convert5to6_plugin.py +++ b/obnamlib/plugins/convert5to6_plugin.py @@ -81,7 +81,7 @@ class Convert5to6Plugin(obnamlib.ObnamPlugin): def gunzip(self, filename, data): return zlib.decompress(data) - + def convert_format(self): self.rawfs.overwrite_file('metadata/format', '6\n') diff --git a/obnamlib/plugins/encryption_plugin.py b/obnamlib/plugins/encryption_plugin.py index 3f13cc23..891e76d5 100644 --- a/obnamlib/plugins/encryption_plugin.py +++ b/obnamlib/plugins/encryption_plugin.py @@ -40,7 +40,7 @@ class EncryptionPlugin(obnamlib.ObnamPlugin): self.app.settings.string(['symmetric-key-bits'], 'size of symmetric key, in bits', group=encryption_group) - + self.tag = "encrypt1" hooks = [ @@ -53,9 +53,9 @@ class EncryptionPlugin(obnamlib.ObnamPlugin): ] for name, callback, rev in hooks: self.app.hooks.add_callback(name, callback, rev) - + self._pubkey = None - + self.app.add_subcommand('client-keys', self.client_keys) self.app.add_subcommand('list-keys', self.list_keys) self.app.add_subcommand('list-toplevels', self.list_toplevels) @@ -65,22 +65,22 @@ class EncryptionPlugin(obnamlib.ObnamPlugin): 'remove-key', self.remove_key, arg_synopsis='[CLIENT-NAME]...') self.app.add_subcommand('remove-client', self.remove_client, arg_synopsis='[CLIENT-NAME]...') - + self._symkeys = obnamlib.SymmetricKeyCache() - + def disable(self): self._symkeys.clear() @property def keyid(self): return self.app.settings['encrypt-with'] - + @property def pubkey(self): if self._pubkey is None: self._pubkey = obnamlib.get_public_key(self.keyid) return self._pubkey - + @property def devrandom(self): if self.app.settings['weak-random']: @@ -100,10 +100,10 @@ class EncryptionPlugin(obnamlib.ObnamPlugin): def toplevel_init(self, repo, toplevel): '''Initialize a new toplevel for encryption.''' - + if not self.keyid: return - + pubkeys = obnamlib.Keyring() pubkeys.add(self.pubkey) @@ -225,7 +225,7 @@ class EncryptionPlugin(obnamlib.ObnamPlugin): print ' %s' % keyid _shared = ['chunklist', 'chunks', 'chunksums', 'clientlist'] - + def _find_clientdirs(self, repo, client_names): return [repo.client_dir(repo.clientlist.get_client_id(x)) for x in client_names] diff --git a/obnamlib/plugins/force_lock_plugin.py b/obnamlib/plugins/force_lock_plugin.py index 1e37f75d..1d5201a4 100644 --- a/obnamlib/plugins/force_lock_plugin.py +++ b/obnamlib/plugins/force_lock_plugin.py @@ -39,7 +39,7 @@ class ForceLockPlugin(obnamlib.ObnamPlugin): try: repo = self.app.open_repository() except OSError, e: - raise obnamlib.Error('Repository does not exist ' + raise obnamlib.Error('Repository does not exist ' 'or cannot be accessed.\n' + str(e)) diff --git a/obnamlib/plugins/forget_plugin.py b/obnamlib/plugins/forget_plugin.py index 51c9df61..9a54555e 100644 --- a/obnamlib/plugins/forget_plugin.py +++ b/obnamlib/plugins/forget_plugin.py @@ -22,7 +22,7 @@ import obnamlib class ForgetPlugin(obnamlib.ObnamPlugin): '''Forget generations.''' - + def enable(self): self.app.add_subcommand('forget', self.forget, arg_synopsis='[GENERATION]...') @@ -34,7 +34,7 @@ class ForgetPlugin(obnamlib.ObnamPlugin): '''Forget (remove) specified backup generations.''' self.app.settings.require('repository') self.app.settings.require('client-name') - + self.app.ts['gen'] = None self.app.ts['gens'] = [] self.app.ts.format('forgetting generations: %Index(gen,gens) done') @@ -63,8 +63,8 @@ class ForgetPlugin(obnamlib.ObnamPlugin): rules = fp.parse(self.app.settings['keep']) keeplist = fp.match(rules, genlist) keepids = set(genid for genid, dt in keeplist) - removeids = [genid - for genid, dt in genlist + removeids = [genid + for genid, dt in genlist if genid not in keepids] self.app.ts['gens'] = removeids diff --git a/obnamlib/plugins/fsck_plugin.py b/obnamlib/plugins/fsck_plugin.py index fd95e31b..89a466a3 100644 --- a/obnamlib/plugins/fsck_plugin.py +++ b/obnamlib/plugins/fsck_plugin.py @@ -26,10 +26,10 @@ import obnamlib class WorkItem(larch.fsck.WorkItem): '''A work item for fsck. - - Whoever creates a WorkItem shall set the ``repo`` to the repository + + Whoever creates a WorkItem shall set the ``repo`` to the repository being used. - + ''' @@ -70,7 +70,7 @@ class CheckFileChecksum(WorkItem): self.correct = correct self.chunkids = chunkids self.checksummer = checksummer - + def do(self): logging.debug('Checking whole-file checksum for %s' % self.filename) if self.correct != self.checksummer.digest(): @@ -107,7 +107,7 @@ class CheckDirectory(WorkItem): self.genid = genid self.dirname = dirname self.name = 'dir %s:%s:%s' % (client_name, genid, dirname) - + def do(self): logging.debug('Checking client=%s genid=%s dirname=%s' % (self.client_name, self.genid, self.dirname)) @@ -130,9 +130,9 @@ class CheckGeneration(WorkItem): self.client_name = client_name self.genid = genid self.name = 'generation %s:%s' % (client_name, genid) - + def do(self): - logging.debug('Checking client=%s genid=%s' % + logging.debug('Checking client=%s genid=%s' % (self.client_name, self.genid)) started, ended = self.repo.client.get_generation_times(self.genid) @@ -162,9 +162,9 @@ class CheckGenerationIdsAreDifferent(WorkItem): def __init__(self, client_name, genids): self.client_name = client_name self.genids = list(genids) - + def do(self): - logging.debug('Checking genid uniqueness for client=%s' % + logging.debug('Checking genid uniqueness for client=%s' % self.client_name) done = set() while self.genids: @@ -234,7 +234,7 @@ class CheckForExtraChunks(WorkItem): def __init__(self): self.name = 'extra chunks' - + def do(self): logging.debug('Checking for extra chunks') for chunkid in self.repo.list_chunks(): @@ -254,7 +254,7 @@ class CheckBTree(WorkItem): return logging.debug('Checking B-tree %s' % self.dirname) fix = self.settings['fsck-fix'] - forest = larch.open_forest(allow_writes=fix, dirname=self.dirname, + forest = larch.open_forest(allow_writes=fix, dirname=self.dirname, vfs=self.repo.fs) fsck = larch.fsck.Fsck(forest, self.warning, self.error, fix) for work in fsck.find_work(): @@ -265,7 +265,7 @@ class CheckRepository(WorkItem): def __init__(self): self.name = 'repository' - + def do(self): logging.debug('Checking repository') if not self.settings['fsck-skip-shared-b-trees']: @@ -279,11 +279,11 @@ class FsckPlugin(obnamlib.ObnamPlugin): def enable(self): self.app.add_subcommand('fsck', self.fsck) - + group = 'Integrity checking (fsck)' - + self.app.settings.boolean( - ['fsck-fix'], + ['fsck-fix'], 'should fsck try to fix problems?', group=group) @@ -335,7 +335,7 @@ class FsckPlugin(obnamlib.ObnamPlugin): self.app.ts['items'] = 0 self.app.ts.format( 'Checking %Integer(this_item)/%Integer(items): %String(item)') - + def fsck(self, args): '''Verify internal consistency of backup repository.''' self.app.settings.require('repository') @@ -344,7 +344,7 @@ class FsckPlugin(obnamlib.ObnamPlugin): self.configure_ttystatus() self.repo = self.app.open_repository() - + self.repo.lock_root() client_names = self.repo.list_clients() client_dirs = [self.repo.client_dir( @@ -382,7 +382,7 @@ class FsckPlugin(obnamlib.ObnamPlugin): self.repo.fs.close() self.app.ts.finish() - + if self.errors: sys.exit(1) diff --git a/obnamlib/plugins/restore_plugin.py b/obnamlib/plugins/restore_plugin.py index e5f4c430..ed732e55 100644 --- a/obnamlib/plugins/restore_plugin.py +++ b/obnamlib/plugins/restore_plugin.py @@ -26,23 +26,23 @@ import obnamlib class Hardlinks(object): '''Keep track of inodes with unrestored hardlinks.''' - + def __init__(self): self.inodes = dict() - + def key(self, metadata): return '%s:%s' % (metadata.st_dev, metadata.st_ino) - + def add(self, filename, metadata): self.inodes[self.key(metadata)] = (filename, metadata.st_nlink) - + def filename(self, metadata): key = self.key(metadata) if key in self.inodes: return self.inodes[key][0] else: return None - + def forget(self, metadata): key = self.key(metadata) filename, nlinks = self.inodes[key] @@ -62,7 +62,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): # just b if b is an absolute path. def enable(self): - self.app.add_subcommand('restore', self.restore, + self.app.add_subcommand('restore', self.restore, arg_synopsis='[DIRECTORY]...') self.app.settings.string(['to'], 'where to restore') self.app.settings.string_list(['generation'], @@ -78,7 +78,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): self.app.ts['total'] = 0 self.app.ts['current-bytes'] = 0 self.app.ts['total-bytes'] = 0 - + self.app.ts.format('%RemainingTime(current-bytes,total-bytes) ' '%Counter(current) files ' '%ByteSize(current-bytes) ' @@ -93,10 +93,10 @@ class RestorePlugin(obnamlib.ObnamPlugin): self.app.settings.require('generation') self.app.settings.require('to') - logging.debug('restoring generation %s' % + logging.debug('restoring generation %s' % self.app.settings['generation']) logging.debug('restoring to %s' % self.app.settings['to']) - + logging.debug('restoring what: %s' % repr(args)) if not args: logging.debug('no args given, so restoring everything') @@ -105,7 +105,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): self.downloaded_bytes = 0 self.file_count = 0 self.started = time.time() - + self.repo = self.app.open_repository() self.repo.open_client(self.app.settings['client-name']) if self.write_ok: @@ -115,9 +115,9 @@ class RestorePlugin(obnamlib.ObnamPlugin): self.fs = None # this will trigger error if we try to really write self.hardlinks = Hardlinks() - + self.errors = False - + generations = self.app.settings['generation'] if len(generations) != 1: raise obnamlib.Error( @@ -138,12 +138,12 @@ class RestorePlugin(obnamlib.ObnamPlugin): self.repo.fs.close() if self.write_ok: self.fs.close() - + self.app.ts.clear() self.report_stats() - + self.app.ts.finish() - + if self.errors: raise obnamlib.Error('There were errors when restoring') @@ -154,7 +154,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): dirname = os.path.dirname(pathname) if self.write_ok and not self.fs.exists('./' + dirname): self.fs.makedirs('./' + dirname) - + set_metadata = True if metadata.isdir(): self.restore_dir(gen, pathname, metadata) @@ -192,7 +192,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): if self.write_ok: self.fs.link('./' + link, './' + filename) self.hardlinks.forget(metadata) - + def restore_symlink(self, gen, filename, metadata): logging.debug('restoring symlink %s' % filename) @@ -206,11 +206,11 @@ class RestorePlugin(obnamlib.ObnamPlugin): elif stat.S_ISBLK(metadata.st_mode) or stat.S_ISCHR(metadata.st_mode): self.restore_device(gen, filename, metadata) else: - msg = ('Unknown file type: %s (%o)' % + msg = ('Unknown file type: %s (%o)' % (filename, metadata.st_mode)) logging.error(msg) self.app.ts.notify(msg) - + def restore_regular_file(self, gen, filename, metadata): logging.debug('restoring regular %s' % filename) if self.write_ok: @@ -297,7 +297,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): (1024**1, 'KiB'), (0, 'B') ] - + for size_base, size_unit in size_table: if self.downloaded_bytes >= size_base: if size_base > 0: @@ -336,7 +336,7 @@ class RestorePlugin(obnamlib.ObnamPlugin): logging.info('Restore performance statistics:') logging.info('* files restored: %s' % self.file_count) - logging.info('* downloaded data: %s bytes (%s %s)' % + logging.info('* downloaded data: %s bytes (%s %s)' % (self.downloaded_bytes, size_amount, size_unit)) logging.info('* duration: %s s' % duration) logging.info('* average speed: %s %s' % (speed_amount, speed_unit)) diff --git a/obnamlib/plugins/sftp_plugin.py b/obnamlib/plugins/sftp_plugin.py index 045b2c89..e24aabf2 100644 --- a/obnamlib/plugins/sftp_plugin.py +++ b/obnamlib/plugins/sftp_plugin.py @@ -43,26 +43,26 @@ import obnamlib def ioerror_to_oserror(method): '''Decorator to convert an IOError exception to OSError. - + Python's os.* raise OSError, mostly, but paramiko's corresponding methods raise IOError. This decorator fixes that. - + ''' - + def helper(self, filename, *args, **kwargs): try: return method(self, filename, *args, **kwargs) except IOError, e: logging.error(traceback.format_exc()) raise OSError(e.errno, e.strerror or str(e), filename) - + return helper class SSHChannelAdapter(object): '''Take an ssh subprocess and pretend it is a paramiko Channel.''' - + # This is inspired by the ssh.py module in bzrlib. def __init__(self, proc): @@ -87,7 +87,7 @@ class SSHChannelAdapter(object): def close(self): logging.debug('SSHChannelAdapter.close called') - for func in [self.proc.stdin.close, self.proc.stdout.close, + for func in [self.proc.stdin.close, self.proc.stdout.close, self.proc.wait]: try: func() @@ -98,11 +98,11 @@ class SSHChannelAdapter(object): class SftpFS(obnamlib.VirtualFileSystem): '''A VFS implementation for SFTP. - - - + + + ''' - + # 32 KiB is the chunk size that gives me the fastest speed # for sftp transfers. I don't know why the size matters. chunk_size = 32 * 1024 @@ -126,7 +126,7 @@ class SftpFS(obnamlib.VirtualFileSystem): obnamlib.VirtualFileSystem.log_stats(self) logging.info('VFS: baseurl=%s roundtrips=%s' % (self.baseurl, self._roundtrips)) - + def _to_string(self, str_or_unicode): if type(str_or_unicode) is unicode: return str_or_unicode.encode('utf-8') @@ -141,7 +141,7 @@ class SftpFS(obnamlib.VirtualFileSystem): # for the best pass self.create_path_if_missing = False # only create once - + def connect(self): try_openssh = not self.settings or not self.settings['pure-paramiko'] if not try_openssh or not self._connect_openssh(): @@ -167,8 +167,8 @@ class SftpFS(obnamlib.VirtualFileSystem): if self.settings and self.settings['strict-ssh-host-keys']: args += ['-o', 'StrictHostKeyChecking=yes'] if self.settings and self.settings['ssh-known-hosts']: - args += ['-o', - 'UserKnownHostsFile=%s' % + args += ['-o', + 'UserKnownHostsFile=%s' % self.settings['ssh-known-hosts']] args += [self.host, 'sftp'] @@ -226,18 +226,18 @@ class SftpFS(obnamlib.VirtualFileSystem): offered_type = offered_key.get_name() if not known_keys.has_key(offered_type): if self.settings['strict-ssh-host-keys']: - raise obnamlib.Error('No known type %s host key for %s' % + raise obnamlib.Error('No known type %s host key for %s' % (offered_type, hostname)) logging.warning('No known host key of type %s for %s; accepting ' 'offered key' % (offered_type, hostname)) - + known_key = known_keys[offered_type] if offered_key != known_key: raise obnamlib.Error('SSH server %s offered wrong public key' % hostname) - - logging.debug('Host key for %s OK' % hostname) - + + logging.debug('Host key for %s OK' % hostname) + def _authenticate(self, username): if not username: username = self._get_username() @@ -316,7 +316,7 @@ class SftpFS(obnamlib.VirtualFileSystem): self.create_path_if_missing = create self._delay() - + if self.sftp: if create: self._create_root_if_missing() @@ -373,17 +373,17 @@ class SftpFS(obnamlib.VirtualFileSystem): # fixed, this code can be removed. st.st_mtime_sec = self._force_32bit_timestamp(st.st_mtime) st.st_atime_sec = self._force_32bit_timestamp(st.st_atime) - + # Within Obnam, we pretend stat results have st_Xtime_sec and # st_Xtime_nsec, but not st_Xtime. Remove those fields. del st.st_mtime del st.st_atime - + # We only get integer timestamps, so set these explicitly to 0. st.st_mtime_nsec = 0 st.st_atime_nsec = 0 - return st + return st @ioerror_to_oserror def listdir2(self, pathname): @@ -420,7 +420,7 @@ class SftpFS(obnamlib.VirtualFileSystem): return stat.S_ISDIR(st.st_mode) def mknod(self, pathname, mode): - # SFTP does not provide an mknod, so we can't do this. We + # SFTP does not provide an mknod, so we can't do this. We # raise an exception, so upper layers can handle this (we _could_ # just fail silently, but that would be silly.) raise NotImplementedError('mknod on SFTP: %s' % pathname) @@ -429,7 +429,7 @@ class SftpFS(obnamlib.VirtualFileSystem): def mkdir(self, pathname): self._delay() self.sftp.mkdir(pathname) - + @ioerror_to_oserror def makedirs(self, pathname): parent = os.path.dirname(pathname) @@ -441,7 +441,7 @@ class SftpFS(obnamlib.VirtualFileSystem): def rmdir(self, pathname): self._delay() self.sftp.rmdir(pathname) - + @ioerror_to_oserror def remove(self, pathname): self._delay() @@ -460,7 +460,7 @@ class SftpFS(obnamlib.VirtualFileSystem): self._delay() self._remove_if_exists(new) self.sftp.rename(old, new) - + @ioerror_to_oserror def lstat(self, pathname): self._delay() @@ -475,12 +475,12 @@ class SftpFS(obnamlib.VirtualFileSystem): logging.warning('NOT changing ownership of symlink %s' % pathname) else: self.sftp.chown(pathname, uid, gid) - + @ioerror_to_oserror def chmod(self, pathname, mode): self._delay() self.sftp.chmod(pathname, mode) - + @ioerror_to_oserror def lutimes(self, pathname, atime_sec, atime_nsec, mtime_sec, mtime_nsec): # FIXME: This does not work for symlinks! @@ -541,7 +541,7 @@ class SftpFS(obnamlib.VirtualFileSystem): def _tempfile(self, dirname): '''Create a new file with a random name, return file handle and name.''' - + if dirname: try: self.makedirs(dirname) @@ -571,7 +571,7 @@ class SftpFS(obnamlib.VirtualFileSystem): self._write_helper(f, contents) f.close() self.rename(tempname, pathname) - + def _write_helper(self, f, contents): for pos in range(0, len(contents), self.chunk_size): chunk = contents[pos:pos + self.chunk_size] diff --git a/obnamlib/plugins/show_plugin.py b/obnamlib/plugins/show_plugin.py index f13480f2..75d95ae9 100644 --- a/obnamlib/plugins/show_plugin.py +++ b/obnamlib/plugins/show_plugin.py @@ -26,12 +26,12 @@ import obnamlib class ShowPlugin(obnamlib.ObnamPlugin): '''Show information about data in the backup repository. - + This implements commands for listing contents of root and client objects, or the contents of a backup generation. - + ''' - + leftists = (2, 3, 6) min_widths = (1, 1, 1, 1, 6, 20, 1) @@ -42,7 +42,7 @@ class ShowPlugin(obnamlib.ObnamPlugin): self.app.add_subcommand('ls', self.ls, arg_synopsis='[FILE]...') self.app.add_subcommand('diff', self.diff, arg_synopsis='[GENERATION1] GENERATION2') - self.app.add_subcommand('nagios-last-backup-age', + self.app.add_subcommand('nagios-last-backup-age', self.nagios_last_backup_age) self.app.settings.string(['warn-age'], @@ -51,7 +51,7 @@ class ShowPlugin(obnamlib.ObnamPlugin): 'backup before status is warning. ' 'Accepts one char unit specifier ' '(s,m,h,d for seconds, minutes, hours, ' - 'and days.', + 'and days.', metavar='AGE', default=obnamlib.DEFAULT_NAGIOS_WARN_AGE) self.app.settings.string(['critical-age'], @@ -60,7 +60,7 @@ class ShowPlugin(obnamlib.ObnamPlugin): 'recent backup before statis is critical. ' 'Accepts one char unit specifier ' '(s,m,h,d for seconds, minutes, hours, ' - 'and days.', + 'and days.', metavar='AGE', default=obnamlib.DEFAULT_NAGIOS_WARN_AGE) @@ -76,7 +76,7 @@ class ShowPlugin(obnamlib.ObnamPlugin): for client_name in self.repo.list_clients(): self.app.output.write('%s\n' % client_name) self.repo.fs.close() - + def generations(self, args): '''List backup generations for client.''' self.open_repository() @@ -88,8 +88,8 @@ class ShowPlugin(obnamlib.ObnamPlugin): else: checkpoint = '' sys.stdout.write('%s\t%s .. %s (%d files, %d bytes) %s\n' % - (gen, - self.format_time(start), + (gen, + self.format_time(start), self.format_time(end), self.repo.client.get_generation_file_count(gen), self.repo.client.get_generation_data(gen), @@ -112,16 +112,16 @@ class ShowPlugin(obnamlib.ObnamPlugin): now = self.app.time() if (now - most_recent > critical_age): self.app.output.write( - 'CRITICAL: backup is old. last backup was %s.\n' % + 'CRITICAL: backup is old. last backup was %s.\n' % (self.format_time(most_recent))) sys.exit(2) elif (now - most_recent > warn_age): self.app.output.write( - 'WARNING: backup is old. last backup was %s.\n' % + 'WARNING: backup is old. last backup was %s.\n' % self.format_time(most_recent)) sys.exit(2) self.app.output.write( - 'OK: backup is recent. last backup was %s.\n' % + 'OK: backup is recent. last backup was %s.\n' % self.format_time(most_recent)) def genids(self, args): @@ -148,14 +148,14 @@ class ShowPlugin(obnamlib.ObnamPlugin): for ls_file in args: self.show_objects(gen, ls_file) self.repo.fs.close() - + def format_time(self, timestamp): return time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(timestamp)) - + def isdir(self, gen, filename): metadata = self.repo.get_metadata(gen, filename) return metadata.isdir() - + def show_objects(self, gen, dirname): self.show_item(gen, dirname) subdirs = [] @@ -180,7 +180,7 @@ class ShowPlugin(obnamlib.ObnamPlugin): 1, # mtime -1, # name ] - + result = [] for i in range(len(fields)): if widths[i] < 0: @@ -244,10 +244,10 @@ class ShowPlugin(obnamlib.ObnamPlugin): for subdir in subdirs: self.show_diff(gen1, gen2, subdir) - + def diff(self, args): '''Show difference between two generations.''' - + if len(args) not in (1, 2): raise obnamlib.Error('Need one or two generations') @@ -293,8 +293,8 @@ class ShowPlugin(obnamlib.ObnamPlugin): if (mode & bitmap) == bitmap: perms[offset] = char perms = ''.join(perms) - - timestamp = time.strftime('%Y-%m-%d %H:%M:%S', + + timestamp = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(metadata.st_mtime_sec)) if metadata.islink(): @@ -302,12 +302,12 @@ class ShowPlugin(obnamlib.ObnamPlugin): else: name = full - return (perms, - str(metadata.st_nlink or 0), - metadata.username or '', + return (perms, + str(metadata.st_nlink or 0), + metadata.username or '', metadata.groupname or '', - str(metadata.st_size or 0), - timestamp, + str(metadata.st_size or 0), + timestamp, name) def format(self, fields): diff --git a/obnamlib/plugins/verify_plugin.py b/obnamlib/plugins/verify_plugin.py index cdd055c7..7da95d26 100644 --- a/obnamlib/plugins/verify_plugin.py +++ b/obnamlib/plugins/verify_plugin.py @@ -53,7 +53,7 @@ class VerifyPlugin(obnamlib.ObnamPlugin): raise obnamlib.Error( 'verify must be given exactly one generation') - logging.debug('verifying generation %s' % + logging.debug('verifying generation %s' % self.app.settings['generation']) if not args: self.app.settings.require('root') @@ -119,7 +119,7 @@ class VerifyPlugin(obnamlib.ObnamPlugin): metadata = self.repo.get_metadata(gen, filename) try: self.verify_metadata(gen, filename, metadata) - self.verify_regular_file(gen, filename, metadata) + self.verify_regular_file(gen, filename, metadata) except Fail, e: self.log_fail(e) self.app.ts['done'] += 1 @@ -150,7 +150,7 @@ class VerifyPlugin(obnamlib.ObnamPlugin): v1 = getattr(backed_up, field) v2 = getattr(live_data, field) if v1 != v2: - raise Fail(filename, + raise Fail(filename, 'metadata change: %s (%s vs %s)' % (field, v1, v2)) def verify_regular_file(self, gen, filename, metadata): @@ -173,11 +173,11 @@ class VerifyPlugin(obnamlib.ObnamPlugin): def walk(self, gen, args): '''Iterate over each pathname specified by arguments. - + This is a generator. - + ''' - + for arg in args: scheme, netloc, path, query, fragment = urlparse.urlsplit(arg) arg = os.path.normpath(path) diff --git a/obnamlib/plugins/vfs_local_plugin.py b/obnamlib/plugins/vfs_local_plugin.py index ff0fc048..2ebb3c40 100644 --- a/obnamlib/plugins/vfs_local_plugin.py +++ b/obnamlib/plugins/vfs_local_plugin.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/obnamlib/repo.py b/obnamlib/repo.py index e7fe5433..35fd6c5b 100644 --- a/obnamlib/repo.py +++ b/obnamlib/repo.py @@ -42,22 +42,22 @@ class BadFormat(obnamlib.Error): class HookedFS(object): '''A class to filter read/written data through hooks.''' - + def __init__(self, repo, fs, hooks): self.repo = repo self.fs = fs self.hooks = hooks - + def __getattr__(self, name): return getattr(self.fs, name) - + def _get_toplevel(self, filename): parts = filename.split(os.sep) if len(parts) > 1: return parts[0] else: # pragma: no cover raise obnamlib.Error('File at repository root: %s' % filename) - + def cat(self, filename, runfilters=True): data = self.fs.cat(filename) toplevel = self._get_toplevel(filename) @@ -76,7 +76,7 @@ class HookedFS(object): data = self.hooks.filter_write('repository-data', data, repo=self.repo, toplevel=toplevel) self.fs.write_file(filename, data) - + def overwrite_file(self, filename, data, runfilters=True): tracing.trace('overwriting hooked %s' % filename) toplevel = self._get_toplevel(filename) @@ -84,41 +84,41 @@ class HookedFS(object): data = self.hooks.filter_write('repository-data', data, repo=self.repo, toplevel=toplevel) self.fs.overwrite_file(filename, data) - + class Repository(object): '''Repository for backup data. - + Backup data is put on a virtual file system (obnamlib.VirtualFileSystem instance), in some form that the API of this class does not care about. - - The repository may contain data for several clients that share + + The repository may contain data for several clients that share encryption keys. Each client is identified by a name. - + The repository has a "root" object, which is conceptually a list of client names. - + Each client in turn is conceptually a list of generations, which correspond to snapshots of the user data that existed when the generation was created. - + Read-only access to the repository does not require locking. Write access may affect only the root object, or only a client's own data, and thus locking may affect only the root, or only the client. - + When a new generation is started, it is a copy-on-write clone of the previous generation, and the caller needs to modify the new generation to match the current state of user data. - + The file 'metadata/format' at the root of the repository contains the version of the repository format it uses. The version is specified using a single integer. ''' - + format_version = 6 def __init__(self, fs, node_size, upload_queue_size, lru_size, hooks, @@ -131,11 +131,11 @@ class Repository(object): self.node_size = node_size self.upload_queue_size = upload_queue_size self.lru_size = lru_size - + hider = hashlib.md5() hider.update(client_name) - self.lockmgr = obnamlib.LockManager(self.fs, lock_timeout, + self.lockmgr = obnamlib.LockManager(self.fs, lock_timeout, hider.hexdigest()) self.got_root_lock = False @@ -151,37 +151,37 @@ class Repository(object): self.client = None self._open_shared() self.prev_chunkid = None - self.chunk_idpath = larch.IdPath('chunks', idpath_depth, + self.chunk_idpath = larch.IdPath('chunks', idpath_depth, idpath_bits, idpath_skip) self._chunks_exists = False def _open_client_list(self): - self.clientlist = obnamlib.ClientList(self.fs, self.node_size, - self.upload_queue_size, + self.clientlist = obnamlib.ClientList(self.fs, self.node_size, + self.upload_queue_size, self.lru_size, self) def _open_shared(self): - self.chunklist = obnamlib.ChunkList(self.fs, self.node_size, - self.upload_queue_size, + self.chunklist = obnamlib.ChunkList(self.fs, self.node_size, + self.upload_queue_size, self.lru_size, self) - self.chunksums = obnamlib.ChecksumTree(self.fs, 'chunksums', + self.chunksums = obnamlib.ChecksumTree(self.fs, 'chunksums', len(self.checksum('')), - self.node_size, - self.upload_queue_size, + self.node_size, + self.upload_queue_size, self.lru_size, self) def setup_hooks(self, hooks): self.hooks = hooks - + self.hooks.new('repository-toplevel-init') self.hooks.new_filter('repository-data') self.hooks.new('repository-add-client') - + def checksum(self, data): '''Return checksum of data. - + The checksum is (currently) MD5. - + ''' checksummer = self.new_checksummer() @@ -252,10 +252,10 @@ class Repository(object): def lock_root(self): '''Lock root node. - + Raise obnamlib.LockFail if locking fails. Lock will be released by commit_root() or unlock_root(). - + ''' tracing.trace('locking root') @@ -280,14 +280,14 @@ class Repository(object): self.lockmgr.unlock(['.']) self.got_root_lock = False self._open_client_list() - + def commit_root(self): '''Commit changes to root node, and unlock it.''' tracing.trace('committing root') self.require_root_lock() for client_name in self.added_clients: self.clientlist.add_client(client_name) - self.hooks.call('repository-add-client', + self.hooks.call('repository-add-client', self.clientlist, client_name) self.added_clients = [] for client_name in self.removed_clients: @@ -298,14 +298,14 @@ class Repository(object): self.clientlist.remove_client(client_name) self.clientlist.commit() self.unlock_root() - + def get_format_version(self): '''Return (major, minor) of the on-disk format version. - + If on-disk repository does not have a version yet, return None. - + ''' - + if self.fs.exists('metadata/format'): data = self.fs.cat('metadata/format', runfilters=False) lines = data.splitlines() @@ -320,7 +320,7 @@ class Repository(object): return version else: return None - + def _write_format_version(self, version): '''Write the desired format version to the repository.''' tracing.trace('write format version') @@ -331,35 +331,35 @@ class Repository(object): def check_format_version(self): '''Verify that on-disk format version is compatbile. - + If not, raise BadFormat. - + ''' - + on_disk = self.get_format_version() if on_disk is not None and not self.acceptable_version(on_disk): raise BadFormat('On-disk repository format %s is incompatible ' 'with program format %s; you need to use a ' 'different version of Obnam' % (on_disk, self.format_version)) - + def add_client(self, client_name): '''Add a new client to the repository.''' tracing.trace('client_name=%s', client_name) self.require_root_lock() if client_name in self.list_clients(): - raise obnamlib.Error('client %s already exists in repository' % + raise obnamlib.Error('client %s already exists in repository' % client_name) self.added_clients.append(client_name) - + def remove_client(self, client_name): '''Remove a client from the repository. - + This removes all data related to the client, including all actual file data unless other clients also use it. - + ''' - + tracing.trace('client_name=%s', client_name) self.require_root_lock() if client_name not in self.list_clients(): @@ -370,10 +370,10 @@ class Repository(object): def shared_dirs(self): return [self.chunklist.dirname, self.chunksums.dirname, self.chunk_idpath.dirname] - + def lock_shared(self): '''Lock a client for exclusive write access. - + Raise obnamlib.LockFail if locking fails. Lock will be released by commit_client() or unlock_client(). @@ -387,7 +387,7 @@ class Repository(object): tracing.trace('starting changes in chunksums and chunklist') self.chunksums.start_changes() self.chunklist.start_changes() - + # Initialize the chunks directory for encryption, etc, if it just # got created. dirname = self.chunk_idpath.dirname @@ -398,7 +398,7 @@ class Repository(object): def commit_shared(self): '''Commit changes to shared B-trees.''' - + tracing.trace('committing shared') self.require_shared_lock() self.chunklist.commit() @@ -412,10 +412,10 @@ class Repository(object): self.lockmgr.unlock(self.shared_dirs) self.got_shared_lock = False self._open_shared() - + def lock_client(self, client_name): '''Lock a client for exclusive write access. - + Raise obnamlib.LockFail if locking fails. Lock will be released by commit_client() or unlock_client(). @@ -424,7 +424,7 @@ class Repository(object): tracing.trace('client_name=%s', client_name) self.require_no_client_lock() self.require_no_shared_lock() - + self.check_format_version() client_id = self.clientlist.get_client_id(client_name) if client_id is None: @@ -441,9 +441,9 @@ class Repository(object): self.current_client_id = client_id self.added_generations = [] self.removed_generations = [] - self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, + self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, self.node_size, - self.upload_queue_size, + self.upload_queue_size, self.lru_size, self) self.client.init_forest() @@ -474,7 +474,7 @@ class Repository(object): if commit_client: self.client.commit() self.unlock_client() - + def open_client(self, client_name): '''Open a client for read-only operation.''' tracing.trace('open r/o client_name=%s' % client_name) @@ -485,28 +485,28 @@ class Repository(object): self.current_client = client_name self.current_client_id = client_id client_dir = self.client_dir(client_id) - self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, - self.node_size, - self.upload_queue_size, + self.client = obnamlib.ClientMetadataTree(self.fs, client_dir, + self.node_size, + self.upload_queue_size, self.lru_size, self) self.client.init_forest() - + def list_generations(self): '''List existing generations for currently open client.''' self.require_open_client() return self.client.list_generations() - + def get_is_checkpoint(self, genid): '''Is a generation a checkpoint one?''' self.require_open_client() return self.client.get_is_checkpoint(genid) - + def start_generation(self): '''Start a new generation. - + The new generation is a copy-on-write clone of the previous one (or empty, if first generation). - + ''' tracing.trace('start new generation') self.require_client_lock() @@ -520,11 +520,11 @@ class Repository(object): def _really_remove_generations(self, remove_genids): '''Really remove a list of generations. - + This is not part of the public API. - + This does not make any safety checks. - + ''' def find_chunkids_in_gens(genids): @@ -548,7 +548,7 @@ class Repository(object): # we can remove it. self.remove_chunk(chunk_id) else: - self.chunksums.remove(checksum, chunk_id, + self.chunksums.remove(checksum, chunk_id, self.current_client_id) if not self.chunksums.chunk_is_used(checksum, chunk_id): self.remove_chunk(chunk_id) @@ -581,9 +581,9 @@ class Repository(object): def get_generation_times(self, gen): '''Return start and end times of a generation. - + An unfinished generation has no end time, so None is returned. - + ''' self.require_open_client() @@ -593,7 +593,7 @@ class Repository(object): '''Return list of basenames in a directory within generation.''' self.require_open_client() return self.client.listdir(gen, dirname) - + def get_metadata(self, gen, filename): '''Return metadata for a file in a generation.''' @@ -620,18 +620,18 @@ class Repository(object): def put_chunk_only(self, data): '''Put chunk of data into repository. - + If the same data is already in the repository, it will be put there a second time. It is the caller's responsibility to check that the data is not already in the repository. - + Return the unique identifier of the new chunk. - + ''' - + def random_chunkid(): return random.randint(0, obnamlib.MAX_ID) - + self.require_started_generation() if self.prev_chunkid is None: @@ -656,11 +656,11 @@ class Repository(object): def put_chunk_in_shared_trees(self, chunkid, checksum): '''Put the chunk into the shared trees. - + The chunk is assumed to already exist in the repository, so we just need to add it to the shared trees that map chunkids to checksums and checksums to chunkids. - + ''' tracing.trace('chunkid=%s', chunkid) @@ -671,22 +671,22 @@ class Repository(object): self.chunklist.add(chunkid, checksum) self.chunksums.add(checksum, chunkid, self.current_client_id) - + def get_chunk(self, chunkid): '''Return data of chunk with given id.''' self.require_open_client() return self.fs.cat(self._chunk_filename(chunkid)) - + def chunk_exists(self, chunkid): '''Does a chunk exist in the repository?''' self.require_open_client() return self.fs.exists(self._chunk_filename(chunkid)) - + def find_chunks(self, checksum): '''Return identifiers of chunks with given checksum. - + Because of hash collisions, the list may be longer than one. - + ''' self.require_open_client() @@ -705,13 +705,13 @@ class Repository(object): def remove_chunk(self, chunk_id): '''Remove a chunk from the repository. - + Note that this does _not_ remove the chunk from the chunk checksum forest. The caller is not supposed to call us until the chunk is not there anymore. - + However, it does remove the chunk from the chunk list forest. - + ''' tracing.trace('chunk_id=%s', chunk_id) @@ -731,21 +731,21 @@ class Repository(object): def set_file_chunks(self, filename, chunkids): '''Set ids of chunks belonging to a file. - + File must be in the started generation. - + ''' - + self.require_started_generation() self.client.set_file_chunks(filename, chunkids) def append_file_chunks(self, filename, chunkids): '''Append to list of ids of chunks belonging to a file. - + File must be in the started generation. - + ''' - + self.require_started_generation() self.client.append_file_chunks(filename, chunkids) @@ -758,7 +758,7 @@ class Repository(object): '''Returned contents of file stored in B-tree instead of chunks dir.''' self.require_open_client() return self.client.get_file_data(gen, filename) - + def genspec(self, spec): '''Interpret a generation specification.''' @@ -780,13 +780,13 @@ class Repository(object): def walk(self, gen, arg, depth_first=False): '''Iterate over each pathname specified by argument. - + This is a generator. Each return value is a tuple consisting of a pathname and its corresponding metadata. Directories are recursed into. - + ''' - + arg = os.path.normpath(arg) metadata = self.get_metadata(gen, arg) if metadata.isdir(): diff --git a/obnamlib/repo_dummy_tests.py b/obnamlib/repo_dummy_tests.py index eca24ed9..d0c4c620 100644 --- a/obnamlib/repo_dummy_tests.py +++ b/obnamlib/repo_dummy_tests.py @@ -1,15 +1,15 @@ # Copyright 2013 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. # diff --git a/obnamlib/repo_tests.py b/obnamlib/repo_tests.py index 05263555..bc92d71a 100644 --- a/obnamlib/repo_tests.py +++ b/obnamlib/repo_tests.py @@ -38,7 +38,7 @@ class RepositoryRootNodeTests(unittest.TestCase): obnamlib.IDPATH_BITS, obnamlib.IDPATH_SKIP, time.time, 0, '') - + self.otherfs = obnamlib.LocalFS(self.tempdir) self.other = obnamlib.Repository(self.fs, obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, @@ -64,13 +64,13 @@ class RepositoryRootNodeTests(unittest.TestCase): def test_does_not_accept_newer_version(self): newer_version = self.repo.format_version + 1 self.assertFalse(self.repo.acceptable_version(newer_version)) - + def test_has_none_version_for_empty_repository(self): self.assertEqual(self.repo.get_format_version(), None) - + def test_creates_repository_with_format_version(self): self.repo.lock_root() - self.assertEqual(self.repo.get_format_version(), + self.assertEqual(self.repo.get_format_version(), self.repo.format_version) def test_lists_no_clients(self): @@ -82,24 +82,24 @@ class RepositoryRootNodeTests(unittest.TestCase): def test_locks_root_node(self): self.repo.lock_root() self.assert_(self.repo.got_root_lock) - + def test_locking_root_node_twice_fails(self): self.repo.lock_root() self.assertRaises(obnamlib.Error, self.repo.lock_root) - + def test_commit_releases_lock(self): self.repo.lock_root() self.repo.commit_root() self.assertFalse(self.repo.got_root_lock) - + def test_unlock_releases_lock(self): self.repo.lock_root() self.repo.unlock_root() self.assertFalse(self.repo.got_root_lock) - + def test_commit_without_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.commit_root) - + def test_unlock_root_without_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.unlock_root) @@ -130,7 +130,7 @@ class RepositoryRootNodeTests(unittest.TestCase): def test_locks_shared(self): self.repo.lock_shared() self.assertTrue(self.repo.got_shared_lock) - + def test_locking_shared_twice_fails(self): self.repo.lock_shared() self.assertRaises(obnamlib.Error, self.repo.lock_shared) @@ -151,15 +151,15 @@ class RepositoryRootNodeTests(unittest.TestCase): def test_open_client_fails_if_format_is_incompatible(self): self.repo._write_format_version(0) self.assertRaises(obnamlib.BadFormat, self.repo.open_client, 'foo') - + def test_adding_client_without_root_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.add_client, 'foo') - + def test_adds_client(self): self.repo.lock_root() self.repo.add_client('foo') self.assertEqual(self.repo.list_clients(), ['foo']) - + def test_adds_two_clients_across_commits(self): self.repo.lock_root() self.repo.add_client('foo') @@ -168,7 +168,7 @@ class RepositoryRootNodeTests(unittest.TestCase): self.repo.add_client('bar') self.repo.commit_root() self.assertEqual(sorted(self.repo.list_clients()), ['bar', 'foo']) - + def test_adds_client_that_persists_after_commit(self): self.repo.lock_root() self.repo.add_client('foo') @@ -181,25 +181,25 @@ class RepositoryRootNodeTests(unittest.TestCase): obnamlib.IDPATH_SKIP, time.time, 0, '') self.assertEqual(s2.list_clients(), ['foo']) - + def test_adding_existing_client_fails(self): self.repo.lock_root() self.repo.add_client('foo') self.assertRaises(obnamlib.Error, self.repo.add_client, 'foo') - + def test_removing_client_without_root_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.remove_client, 'foo') - + def test_removing_nonexistent_client_fails(self): self.repo.lock_root() self.assertRaises(obnamlib.Error, self.repo.remove_client, 'foo') - + def test_removing_client_works(self): self.repo.lock_root() self.repo.add_client('foo') self.repo.remove_client('foo') self.assertEqual(self.repo.list_clients(), []) - + def test_removing_client_persists_past_commit(self): self.repo.lock_root() self.repo.add_client('foo') @@ -258,9 +258,9 @@ class RepositoryClientTests(unittest.TestCase): self.repo.lock_root() self.repo.add_client('client_name') self.repo.commit_root() - + self.otherfs = obnamlib.LocalFS(self.tempdir) - self.other = obnamlib.Repository(self.otherfs, + self.other = obnamlib.Repository(self.otherfs, obnamlib.DEFAULT_NODE_SIZE, obnamlib.DEFAULT_UPLOAD_QUEUE_SIZE, obnamlib.DEFAULT_LRU_SIZE, None, @@ -268,7 +268,7 @@ class RepositoryClientTests(unittest.TestCase): obnamlib.IDPATH_BITS, obnamlib.IDPATH_SKIP, time.time, 0, '') - + self.dir_meta = obnamlib.Metadata() self.dir_meta.st_mode = stat.S_IFDIR | 0777 @@ -284,7 +284,7 @@ class RepositoryClientTests(unittest.TestCase): def test_locking_client_twice_fails(self): self.repo.lock_client('client_name') - self.assertRaises(obnamlib.Error, self.repo.lock_client, + self.assertRaises(obnamlib.Error, self.repo.lock_client, 'client_name') def test_locking_nonexistent_client_fails(self): @@ -325,7 +325,7 @@ class RepositoryClientTests(unittest.TestCase): def test_commit_client_without_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.commit_client) - + def test_unlock_client_without_lock_fails(self): self.assertRaises(obnamlib.LockFail, self.repo.unlock_client) @@ -344,15 +344,15 @@ class RepositoryClientTests(unittest.TestCase): self.other.lock_client('client_name') self.repo.open_client('client_name') self.assert_(True) - + def test_lists_no_generations_when_readonly(self): self.repo.open_client('client_name') self.assertEqual(self.repo.list_generations(), []) - + def test_lists_no_generations_when_locked(self): self.repo.lock_client('client_name') self.assertEqual(self.repo.list_generations(), []) - + def test_listing_generations_fails_if_client_is_not_open(self): self.assertRaises(obnamlib.Error, self.repo.list_generations) @@ -523,7 +523,7 @@ class RepositoryClientTests(unittest.TestCase): pathnames = ['/%d' % i for i in range(n)] for pathname in pathnames: self.repo.create(pathname, obnamlib.Metadata()) - self.assertEqual(sorted(self.repo.listdir(gen, '/')), + self.assertEqual(sorted(self.repo.listdir(gen, '/')), sorted(os.path.basename(x) for x in pathnames)) def test_create_adds_dir(self): @@ -543,7 +543,7 @@ class RepositoryClientTests(unittest.TestCase): self.repo.lock_client('client_name') gen = self.repo.start_generation() self.repo.create('/foo', self.dir_meta) - self.assertEqual(self.repo.get_metadata(gen, '/foo').st_mode, + self.assertEqual(self.repo.get_metadata(gen, '/foo').st_mode, self.dir_meta.st_mode) def test_remove_removes_file(self): @@ -604,15 +604,15 @@ class RepositoryChunkTests(unittest.TestCase): def test_put_chunk_returns_id(self): self.repo.lock_shared() self.assertNotEqual(self.repo.put_chunk_only('data'), None) - + def test_get_chunk_retrieves_what_put_chunk_puts(self): self.repo.lock_shared() chunkid = self.repo.put_chunk_only('data') self.assertEqual(self.repo.get_chunk(chunkid), 'data') - + def test_chunk_does_not_exist(self): self.assertFalse(self.repo.chunk_exists(1234)) - + def test_chunk_exists_after_it_is_put(self): self.repo.lock_shared() chunkid = self.repo.put_chunk_only('chunk') @@ -627,17 +627,17 @@ class RepositoryChunkTests(unittest.TestCase): def test_silently_ignores_failure_when_removing_nonexistent_chunk(self): self.repo.lock_shared() self.assertEqual(self.repo.remove_chunk(0), None) - + def test_find_chunks_finds_what_put_chunk_puts(self): self.repo.lock_shared() checksum = self.repo.checksum('data') chunkid = self.repo.put_chunk_only('data') self.repo.put_chunk_in_shared_trees(chunkid, checksum) self.assertEqual(self.repo.find_chunks(checksum), [chunkid]) - + def test_find_chunks_finds_nothing_if_nothing_is_put(self): self.assertEqual(self.repo.find_chunks('checksum'), []) - + def test_handles_checksum_collision(self): self.repo.lock_shared() checksum = self.repo.checksum('data') @@ -650,7 +650,7 @@ class RepositoryChunkTests(unittest.TestCase): def test_returns_no_chunks_initially(self): self.assertEqual(self.repo.list_chunks(), []) - + def test_returns_chunks_after_they_exist(self): self.repo.lock_shared() checksum = self.repo.checksum('data') @@ -780,18 +780,18 @@ class RepositoryWalkTests(unittest.TestCase): self.dir_meta = obnamlib.Metadata() self.dir_meta.st_mode = stat.S_IFDIR | 0777 - + self.file_meta = obnamlib.Metadata() self.file_meta.st_mode = stat.S_IFREG | 0644 - + self.repo.lock_client('client_name') self.repo.lock_shared() self.gen = self.repo.start_generation() - + self.repo.create('/', self.dir_meta) self.repo.create('/foo', self.dir_meta) self.repo.create('/foo/bar', self.file_meta) - + self.repo.commit_client() self.repo.open_client('client_name') diff --git a/obnamlib/repo_tree.py b/obnamlib/repo_tree.py index 4bbcf61d..a538ee7b 100644 --- a/obnamlib/repo_tree.py +++ b/obnamlib/repo_tree.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -23,16 +23,16 @@ import obnamlib class RepositoryTree(object): '''A B-tree within an obnamlib.Repository. - + For read-only operation, call init_forest before doing anything. - + For read-write operation, call start_changes before doing anything, - and commit afterwards. In between, self.tree is the new tree to be + and commit afterwards. In between, self.tree is the new tree to be modified. Note that self.tree is NOT available after init_forest. - + After init_forest or start_changes, self.forest is the opened forest. Unlike self.tree, it will not go away after commit. - + ''' def __init__(self, fs, dirname, key_bytes, node_size, upload_queue_size, @@ -68,7 +68,7 @@ class RepositoryTree(object): def start_changes(self, create_tree=True): tracing.trace('start changes for %s', self.dirname) - + if self.forest is None or not self.forest_allows_writes: if not self.fs.exists(self.dirname): need_init = True @@ -80,7 +80,7 @@ class RepositoryTree(object): if not self.fs.exists(self.dirname): tracing.trace('create %s', self.dirname) self.fs.mkdir(self.dirname) - self.repo.hooks.call('repository-toplevel-init', self.repo, + self.repo.hooks.call('repository-toplevel-init', self.repo, self.dirname) self.forest = None self.init_forest(allow_writes=True) diff --git a/obnamlib/sizeparse.py b/obnamlib/sizeparse.py index 753caadb..479ce2e1 100644 --- a/obnamlib/sizeparse.py +++ b/obnamlib/sizeparse.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -40,10 +40,10 @@ class UnitNameError(UnitError): class ByteSizeParser(object): '''Parse sizes of data in bytes, kilobytes, kibibytes, etc.''' - + pat = re.compile(r'^(?P<size>\d+(\.\d+)?)\s*' r'(?P<unit>[kmg]?i?b?)?$', re.I) - + units = { 'b': 1, 'k': 1000, @@ -56,15 +56,15 @@ class ByteSizeParser(object): 'gb': 1000**3, 'gib': 1024**3, } - + def __init__(self): self.set_default_unit('B') - + def set_default_unit(self, unit): if unit.lower() not in self.units: raise UnitNameError(unit) self.default_unit = unit - + def parse(self, string): m = self.pat.match(string) if not m: diff --git a/obnamlib/sizeparse_tests.py b/obnamlib/sizeparse_tests.py index 8ef18b32..07ae5775 100644 --- a/obnamlib/sizeparse_tests.py +++ b/obnamlib/sizeparse_tests.py @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -23,47 +23,47 @@ class ByteSizeParserTests(unittest.TestCase): def setUp(self): self.p = obnamlib.ByteSizeParser() - + def test_parses_zero(self): self.assertEqual(self.p.parse('0'), 0) - + def test_parses_unadorned_size_as_bytes(self): self.assertEqual(self.p.parse('123'), 123) - + def test_returns_an_int(self): self.assert_(isinstance(self.p.parse('123'), int)) - + def test_parses_unadorned_size_using_default_unit(self): self.p.set_default_unit('KiB') self.assertEqual(self.p.parse('123'), 123 * 1024) - + def test_parses_size_with_byte_unit(self): self.assertEqual(self.p.parse('123 B'), 123) - + def test_parses_size_with_kilo_unit(self): self.assertEqual(self.p.parse('123 k'), 123 * 1000) - + def test_parses_size_with_kilobyte_unit(self): self.assertEqual(self.p.parse('123 kB'), 123 * 1000) - + def test_parses_size_with_kibibyte_unit(self): self.assertEqual(self.p.parse('123 KiB'), 123 * 1024) - + def test_parses_size_with_mega_unit(self): self.assertEqual(self.p.parse('123 m'), 123 * 1000**2) - + def test_parses_size_with_megabyte_unit(self): self.assertEqual(self.p.parse('123 MB'), 123 * 1000**2) - + def test_parses_size_with_mebibyte_unit(self): self.assertEqual(self.p.parse('123 MiB'), 123 * 1024**2) - + def test_parses_size_with_giga_unit(self): self.assertEqual(self.p.parse('123 g'), 123 * 1000**3) - + def test_parses_size_with_gigabyte_unit(self): self.assertEqual(self.p.parse('123 GB'), 123 * 1000**3) - + def test_parses_size_with_gibibyte_unit(self): self.assertEqual(self.p.parse('123 GiB'), 123 * 1024**3) @@ -80,7 +80,7 @@ class ByteSizeParserTests(unittest.TestCase): self.assertRaises(obnamlib.UnitNameError, self.p.parse, '1 ib') def test_raises_error_for_bad_default_unit(self): - self.assertRaises(obnamlib.UnitNameError, + self.assertRaises(obnamlib.UnitNameError, self.p.set_default_unit, 'km') def test_size_syntax_error_includes_input_string(self): diff --git a/obnamlib/vfs.py b/obnamlib/vfs.py index b1dccfc9..bf470078 100644 --- a/obnamlib/vfs.py +++ b/obnamlib/vfs.py @@ -27,7 +27,7 @@ import obnamlib class VirtualFileSystem(object): '''A virtual filesystem interface. - + The backup program needs to access both local and remote files. To make it easier to support all kinds of files both locally and remotely, we use a custom virtual filesystem interface so that @@ -44,7 +44,7 @@ class VirtualFileSystem(object): for the relative paths: directory components separated by slashes, and an initial slash indicating the root of the filesystem (in this case, the base URL). - + ''' def __init__(self, baseurl): @@ -60,18 +60,18 @@ class VirtualFileSystem(object): def connect(self): '''Connect to filesystem.''' - + def close(self): '''Close connection to filesystem.''' self.log_stats() def reinit(self, new_baseurl, create=False): '''Go back to the beginning. - + This behaves like instantiating a new instance, but possibly faster for things like SftpFS. If there is a network connection already open, it will be reused. - + ''' def abspath(self, pathname): @@ -80,7 +80,7 @@ class VirtualFileSystem(object): def getcwd(self): '''Return current working directory as absolute pathname.''' - + def chdir(self, pathname): '''Change current working directory to pathname.''' @@ -89,10 +89,10 @@ class VirtualFileSystem(object): def listdir2(self, pathname): '''Return list of basenames and stats of entities at pathname. - + The stat entity may be an exception object instead, to indicate an error. - + ''' def lock(self, lockname, data): @@ -112,11 +112,11 @@ class VirtualFileSystem(object): def mkdir(self, pathname): '''Create a directory. - + Parent directories must already exist. - + ''' - + def makedirs(self, pathname): '''Create a directory, and missing parents.''' @@ -150,10 +150,10 @@ class VirtualFileSystem(object): def llistxattr(self, pathname): '''Return list of names of extended attributes for file.''' return [] - + def lgetxattr(self, pathname, attrname): '''Return value of an extended attribute.''' - + def lsetxattr(self, pathname, attrname, attrvalue): '''Set value of an extended attribute.''' @@ -165,10 +165,10 @@ class VirtualFileSystem(object): def lutimes(self, pathname, atime_sec, atime_nsec, mtime_sec, mtime_nsec): '''Like lutimes(2). - + This isn't quite like lutimes, actually. Most importantly, it uses nanosecond timestamps rather than microsecond. This is important. - + ''' def link(self, existing_path, new_path): @@ -194,11 +194,11 @@ class VirtualFileSystem(object): def write_file(self, pathname, contents): '''Write a new file. - The file must not yet exist. The file is not necessarily written + The file must not yet exist. The file is not necessarily written atomically, meaning that if the writing fails (connection to server drops, for example), the file might exist in a partial form. The callers need to deal with this. - + Any directories in pathname will be created if necessary. ''' @@ -209,30 +209,30 @@ class VirtualFileSystem(object): def scan_tree(self, dirname, ok=None, dirst=None, log=logging.error, error_handler=None): '''Scan a tree for files. - + Return a generator that returns ``(pathname, stat_result)`` - pairs for each file and directory in the tree, in + pairs for each file and directory in the tree, in depth-first order. - + If ``ok`` is not None, it must be a function that determines if a particular file or directory should be returned. It gets the pathname and stat result as arguments, and should return True or False. If it returns False on a directory, ``scan_tree`` will not recurse into the directory. - + ``dirst`` is for internal optimization, and should not be used by the caller. ``log`` is used by unit tests and should not be used by the caller. - + Errors from calling ``listdir`` or ``lstat`` are logged, but do not stop the scanning. Such files or directories are not returned, however. If `error_handler` is defined, it is called once for every problem, giving the name and exception as arguments. - + ''' - + error_handler = error_handler or (lambda name, e: None) try: @@ -241,7 +241,7 @@ class VirtualFileSystem(object): log('listdir failed: %s: %s' % (e.filename, e.strerror)) error_handler(dirname, e) pairs = [] - + queue = [] for name, st in pairs: pathname = os.path.join(dirname, name) @@ -266,14 +266,14 @@ class VirtualFileSystem(object): yield dirname, dirst - + class VfsFactory: '''Create new instances of VirtualFileSystem.''' def __init__(self): self.implementations = {} - + def register(self, scheme, implementation, **kwargs): if scheme in self.implementations: raise obnamlib.Error('URL scheme %s already registered' % scheme) @@ -286,35 +286,35 @@ class VfsFactory: klass, kwargs = self.implementations[scheme] return klass(url, create=create, **kwargs) raise obnamlib.Error('Unknown VFS type %s' % url) - - + + class VfsTests(object): # pragma: no cover '''Re-useable tests for VirtualFileSystem implementations. - + The base class can't be usefully instantiated itself. Instead you are supposed to sub-class it and implement the API in a suitable way for yourself. - + This class implements a number of tests that the API implementation must pass. The implementation's own test class should inherit from this class, and unittest.TestCase. - + The test sub-class should define a setUp method that sets the following: - + * self.fs to an instance of the API implementation sub-class * self.basepath to the path to the base of the filesystem - + basepath must be operable as a pathname using os.path tools. If the VFS implemenation operates remotely and wants to operate on a URL like 'http://domain/path' as the baseurl, then basepath must be just the path portion of the URL. - + The directory indicated by basepath must exist, but must be empty at start. - + ''' - + non_ascii_name = u'm\u00e4kel\u00e4'.encode('utf-8') def test_abspath_returns_input_for_absolute_path(self): @@ -452,15 +452,15 @@ class VfsTests(object): # pragma: no cover def test_mkdir_raises_oserror_if_parent_does_not_exist(self): self.assertRaises(OSError, self.fs.mkdir, 'foo/bar') - + def test_makedirs_raises_oserror_when_directory_exists(self): self.fs.mkdir('foo') self.assertRaises(OSError, self.fs.makedirs, 'foo') - + def test_makedirs_creates_directory_when_parent_exists(self): self.fs.makedirs('foo') self.assert_(self.fs.isdir('foo')) - + def test_makedirs_creates_directory_when_parent_does_not_exist(self): self.fs.makedirs('foo/bar') self.assert_(self.fs.isdir('foo/bar')) @@ -537,7 +537,7 @@ class VfsTests(object): # pragma: no cover self.fs.lutimes('foo', 1, 2*1000, 3, 4*1000) self.assertEqual(self.fs.lstat('foo').st_atime_sec, 1) - # not all filesystems support sub-second timestamps; those that + # not all filesystems support sub-second timestamps; those that # do not, return 0, so we have to accept either that or the correct # value, but no other vlaues self.assert_(self.fs.lstat('foo').st_atime_nsec in [0, 2*1000]) @@ -587,10 +587,10 @@ class VfsTests(object): # pragma: no cover def test_cat_fails_for_nonexistent_file(self): self.assertRaises(IOError, self.fs.cat, 'foo') - + def test_has_read_nothing_initially(self): self.assertEqual(self.fs.bytes_read, 0) - + def test_cat_updates_bytes_read(self): self.fs.write_file('foo', 'bar') self.fs.cat('foo') @@ -620,14 +620,14 @@ class VfsTests(object): # pragma: no cover self.fs.write_file('foo', 'bar') self.fs.overwrite_file('foo', 'foobar') self.assertEqual(self.fs.cat('foo'), 'foobar') - + def test_has_written_nothing_initially(self): self.assertEqual(self.fs.bytes_written, 0) - + def test_write_updates_written(self): self.fs.write_file('foo', 'foo') self.assertEqual(self.fs.bytes_written, 3) - + def test_overwrite_updates_written(self): self.fs.overwrite_file('foo', 'foo') self.assertEqual(self.fs.bytes_written, 3) @@ -658,7 +658,7 @@ class VfsTests(object): # pragma: no cover result = list(self.fs.scan_tree(self.basepath)) pathnames = [pathname for pathname, st in result] self.assertEqual(sorted(pathnames), sorted(self.pathnames)) - + def test_scan_tree_filters_away_unwanted(self): def ok(pathname, st): return stat.S_ISDIR(st.st_mode) diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py index a55fc601..639f4451 100644 --- a/obnamlib/vfs_local.py +++ b/obnamlib/vfs_local.py @@ -49,16 +49,16 @@ class LocalFSFile(file): class LocalFS(obnamlib.VirtualFileSystem): """A VFS implementation for local filesystems.""" - + chunk_size = 1024 * 1024 - + def __init__(self, baseurl, create=False): tracing.trace('baseurl=%s', baseurl) tracing.trace('create=%s', create) obnamlib.VirtualFileSystem.__init__(self, baseurl) self.reinit(baseurl, create=create) - - # For checking that we do not unlock something we didn't lock + + # For checking that we do not unlock something we didn't lock # ourselves. self.our_locks = set() @@ -75,7 +75,7 @@ class LocalFS(obnamlib.VirtualFileSystem): self.crash_counter) def reinit(self, baseurl, create=False): - # We fake chdir so that it doesn't mess with the caller's + # We fake chdir so that it doesn't mess with the caller's # perception of current working directory. This also benefits # unit tests. To do this, we store the baseurl as the cwd. tracing.trace('baseurl=%s', baseurl) @@ -140,7 +140,7 @@ class LocalFS(obnamlib.VirtualFileSystem): def lstat(self, pathname): (ret, dev, ino, mode, nlink, uid, gid, rdev, size, blksize, blocks, - atime_sec, atime_nsec, mtime_sec, mtime_nsec, + atime_sec, atime_nsec, mtime_sec, mtime_nsec, ctime_sec, ctime_nsec) = obnamlib._obnam.lstat(self.join(pathname)) if ret != 0: raise OSError(ret, os.strerror(ret), pathname) @@ -182,7 +182,7 @@ class LocalFS(obnamlib.VirtualFileSystem): return ret def lsetxattr(self, filename, attrname, attrvalue): # pragma: no cover - ret = obnamlib._obnam.lsetxattr(self.join(filename), + ret = obnamlib._obnam.lsetxattr(self.join(filename), attrname, attrvalue) if ret != 0: raise OSError(ret, os.strerror(ret), filename) @@ -200,8 +200,8 @@ class LocalFS(obnamlib.VirtualFileSystem): assert atime_nsec is not None assert mtime_sec is not None assert mtime_nsec is not None - ret = obnamlib._obnam.utimensat(self.join(pathname), - atime_sec, atime_nsec, + ret = obnamlib._obnam.utimensat(self.join(pathname), + atime_sec, atime_nsec, mtime_sec, mtime_nsec) if ret != 0: raise OSError(ret, os.strerror(ret), pathname) @@ -324,7 +324,7 @@ class LocalFS(obnamlib.VirtualFileSystem): path = self.join(pathname) os.rename(tempname, path) self.maybe_crash() - + def _write_to_tempfile(self, pathname, contents): path = self.join(pathname) dirname = os.path.dirname(path) diff --git a/obnamlib/vfs_local_tests.py b/obnamlib/vfs_local_tests.py index 1eed56ac..f654eb81 100644 --- a/obnamlib/vfs_local_tests.py +++ b/obnamlib/vfs_local_tests.py @@ -35,15 +35,15 @@ class LocalFSTests(obnamlib.VfsTests, unittest.TestCase): shutil.rmtree(self.basepath) def test_joins_relative_path_ok(self): - self.assertEqual(self.fs.join('foo'), + self.assertEqual(self.fs.join('foo'), os.path.join(self.basepath, 'foo')) def test_join_treats_absolute_path_as_absolute(self): self.assertEqual(self.fs.join('/foo'), '/foo') - + def test_get_username_returns_root_for_zero(self): self.assertEqual(self.fs.get_username(0), 'root') - + def test_get_groupname_returns_root_for_zero(self): self.assertEqual(self.fs.get_groupname(0), 'root') diff --git a/read-live-data-with-sftp b/read-live-data-with-sftp index 7b2f0710..fc2a1c57 100755 --- a/read-live-data-with-sftp +++ b/read-live-data-with-sftp @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/run-benchmarks b/run-benchmarks index 46dead16..721a0763 100755 --- a/run-benchmarks +++ b/run-benchmarks @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -42,7 +42,7 @@ class GenerateManpage(build): print 'building manpages' for x in ['obnam', 'obnam-benchmark']: with open('%s.1' % x, 'w') as f: - runcmd(['python', x, '--generate-manpage=%s.1.in' % x, + runcmd(['python', x, '--generate-manpage=%s.1.in' % x, '--output=%s.1' % x], stdout=f) @@ -58,7 +58,7 @@ class CleanMore(clean): self.remove_pyc('test-plugins') if os.path.isdir('build'): shutil.rmtree('build') - + def remove_pyc(self, rootdir): for dirname, subdirs, basenames in os.walk(rootdir): for x in [os.path.join(dirname, base) diff --git a/test-data/repo-format-5-encrypted-gzipped.tar.gz b/test-data/repo-format-5-encrypted-gzipped.tar.gz Binary files differindex 6d97ad0b..70243365 100644 --- a/test-data/repo-format-5-encrypted-gzipped.tar.gz +++ b/test-data/repo-format-5-encrypted-gzipped.tar.gz diff --git a/test-gpghome/pubring.gpg b/test-gpghome/pubring.gpg Binary files differindex 92df0c78..aaf0dbe5 100644 --- a/test-gpghome/pubring.gpg +++ b/test-gpghome/pubring.gpg diff --git a/test-gpghome/secring.gpg b/test-gpghome/secring.gpg Binary files differindex bed16cd2..47dc4be6 100644 --- a/test-gpghome/secring.gpg +++ b/test-gpghome/secring.gpg diff --git a/test-lock-files b/test-lock-files index 7358181c..e56385d0 100755 --- a/test-lock-files +++ b/test-lock-files @@ -3,17 +3,17 @@ # Obnam test: test lock files. # # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/test-locking b/test-locking index 04221673..c5b46b69 100755 --- a/test-locking +++ b/test-locking @@ -3,17 +3,17 @@ # Obnam test: test locking with multiple clients accessing the same repo. # # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/test-many-generations b/test-many-generations index 7a6476c0..741f67c4 100755 --- a/test-many-generations +++ b/test-many-generations @@ -3,17 +3,17 @@ # Obnam test: backup and verify many generations of data. # # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/test-paramiko b/test-paramiko index cd2aab30..61a0ee3a 100644 --- a/test-paramiko +++ b/test-paramiko @@ -1,15 +1,15 @@ # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -121,7 +121,7 @@ except OSError: sys.exit(1) else: print 'good host key for %s' % hostname - + sftp = paramiko.SFTPClient.from_transport(transport) else: sftp = paramiko.SFTPClient(SSHChannelAdapter(proc)) diff --git a/test-sftpfs b/test-sftpfs index b73ce571..ee725b70 100755 --- a/test-sftpfs +++ b/test-sftpfs @@ -1,16 +1,16 @@ #!/usr/bin/python # Copyright 2010 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -49,23 +49,23 @@ class SftpTests(unittest.TestCase, obnamlib.VfsTests): 'strict-ssh-host-keys': False, 'ssh-known-hosts': os.path.expanduser('~/.ssh/known_hosts'), } - self.fs = obnamlib.plugins.sftp_plugin.SftpFS(baseurl, + self.fs = obnamlib.plugins.sftp_plugin.SftpFS(baseurl, settings=settings) self.fs.connect() - + def tearDown(self): self.fs.close() shutil.rmtree(self.basepath) - + def test_sets_path_to_absolute_path(self): self.assert_(self.fs.path.startswith('/')) - + def test_initial_cwd_is_basepath(self): self.assertEqual(self.fs.getcwd(), self.fs.path) def test_link_creates_hard_link(self): pass # sftp does not support hardlinking, so not testing it - + def test_mknod_creates_fifo(self): self.assertRaises(NotImplementedError, self.fs.mknod, 'foo', 0) @@ -78,14 +78,14 @@ class SftpTests(unittest.TestCase, obnamlib.VfsTests): self.assertEqual(self.fs.lstat('foo').st_atime_nsec, 0) self.assertEqual(self.fs.lstat('foo').st_mtime_sec, 3) self.assertEqual(self.fs.lstat('foo').st_mtime_nsec, 0) - + def test_get_username_returns_None_for_zero(self): self.assertEqual(self.fs.get_username(0), None) - + def test_get_groupname_returns_None_for_zero(self): self.assertEqual(self.fs.get_groupname(0), None) - - + + if __name__ == '__main__': logging.basicConfig(filename='/dev/null') unittest.main() diff --git a/tests/backup b/tests/backup index 5b512e3d..08bc9fd8 100755 --- a/tests/backup +++ b/tests/backup @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/backup-pretend.script b/tests/backup-pretend.script index 0cb4f362..abfc69e7 100755 --- a/tests/backup-pretend.script +++ b/tests/backup-pretend.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/compression+encryption.script b/tests/compression+encryption.script index cb387290..4cca7cce 100755 --- a/tests/compression+encryption.script +++ b/tests/compression+encryption.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/compression.script b/tests/compression.script index 7890c117..1b5b8dfa 100755 --- a/tests/compression.script +++ b/tests/compression.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/convert5to6.script b/tests/convert5to6.script index 3d2a4407..c4936043 100755 --- a/tests/convert5to6.script +++ b/tests/convert5to6.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -25,7 +25,7 @@ tar -xf "$SRCDIR/test-data/repo-format-5-encrypted-gzipped.tar.gz" -r repo --encrypt-with=1B321347 --compress-with=gzip "$SRCDIR/tests/obnam" restore \ -r repo --encrypt-with=1B321347 --compress-with=gzip --to=restored -(cd "restored/home/liw/obnam/convert-5to6/t" && +(cd "restored/home/liw/obnam/convert-5to6/t" && summain --exclude=Username --exclude=Uid --exclude=Group --exclude=Gid data) \ > restored.summain diff --git a/tests/encryption-adds-key.script b/tests/encryption-adds-key.script index 1a04921a..25968e16 100755 --- a/tests/encryption-adds-key.script +++ b/tests/encryption-adds-key.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/encryption-has-client-key-after-backup.script b/tests/encryption-has-client-key-after-backup.script index 1366959f..43369720 100755 --- a/tests/encryption-has-client-key-after-backup.script +++ b/tests/encryption-has-client-key-after-backup.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/encryption-removes-client.script b/tests/encryption-removes-client.script index 90c58da5..5d13c13c 100755 --- a/tests/encryption-removes-client.script +++ b/tests/encryption-removes-client.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/encryption-removes-key.script b/tests/encryption-removes-key.script index 30167776..06e47880 100755 --- a/tests/encryption-removes-key.script +++ b/tests/encryption-removes-key.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/encryption-replaces-key.script b/tests/encryption-replaces-key.script index d558ebb6..d2f31321 100755 --- a/tests/encryption-replaces-key.script +++ b/tests/encryption-replaces-key.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/encryption-use.script b/tests/encryption-use.script index 1edf7236..be3e0578 100755 --- a/tests/encryption-use.script +++ b/tests/encryption-use.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/exclude-cachedir.script b/tests/exclude-cachedir.script index 88ffed60..7c555867 100755 --- a/tests/exclude-cachedir.script +++ b/tests/exclude-cachedir.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/fail-on-bad-file-checksum.script b/tests/fail-on-bad-file-checksum.script index 511694ee..a4dade22 100755 --- a/tests/fail-on-bad-file-checksum.script +++ b/tests/fail-on-bad-file-checksum.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/fail-on-mangled-chunk.script b/tests/fail-on-mangled-chunk.script index 47b0d1f9..c5bf4851 100755 --- a/tests/fail-on-mangled-chunk.script +++ b/tests/fail-on-mangled-chunk.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-according-to-policy.script b/tests/forget-removes-according-to-policy.script index 1cc4eab3..12d5ab0f 100755 --- a/tests/forget-removes-according-to-policy.script +++ b/tests/forget-removes-according-to-policy.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-nothing-by-default.script b/tests/forget-removes-nothing-by-default.script index fc83d4e9..6278df15 100755 --- a/tests/forget-removes-nothing-by-default.script +++ b/tests/forget-removes-nothing-by-default.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-nothing-if-pretending.script b/tests/forget-removes-nothing-if-pretending.script index 65ef463a..1780a41f 100755 --- a/tests/forget-removes-nothing-if-pretending.script +++ b/tests/forget-removes-nothing-if-pretending.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-specified-gens.script b/tests/forget-removes-specified-gens.script index 7ce1ec67..2565590e 100755 --- a/tests/forget-removes-specified-gens.script +++ b/tests/forget-removes-specified-gens.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-unwanted-leaving-empty-generation.script b/tests/forget-removes-unwanted-leaving-empty-generation.script index 0b354b15..28b13547 100755 --- a/tests/forget-removes-unwanted-leaving-empty-generation.script +++ b/tests/forget-removes-unwanted-leaving-empty-generation.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/forget-removes-unwated-data.script b/tests/forget-removes-unwated-data.script index 73425042..7c035794 100755 --- a/tests/forget-removes-unwated-data.script +++ b/tests/forget-removes-unwated-data.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/hardlinks.script b/tests/hardlinks.script index f34048ec..3fadcd2a 100755 --- a/tests/hardlinks.script +++ b/tests/hardlinks.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/logs-for-owner-only.script b/tests/logs-for-owner-only.script index d2833f74..920620c0 100755 --- a/tests/logs-for-owner-only.script +++ b/tests/logs-for-owner-only.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/ls-generation-timestamp.script b/tests/ls-generation-timestamp.script index aa4e424b..7f05338f 100755 --- a/tests/ls-generation-timestamp.script +++ b/tests/ls-generation-timestamp.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/nagios-check.script b/tests/nagios-check.script index ecd357b8..3af550fe 100755 --- a/tests/nagios-check.script +++ b/tests/nagios-check.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/named-pipe.script b/tests/named-pipe.script index b301a27c..0f2ccd35 100755 --- a/tests/named-pipe.script +++ b/tests/named-pipe.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/named-socket.script b/tests/named-socket.script index 33641b1c..7f87a321 100755 --- a/tests/named-socket.script +++ b/tests/named-socket.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/no-roots-from-old-gens.script b/tests/no-roots-from-old-gens.script index ed3faed4..987d8b0e 100755 --- a/tests/no-roots-from-old-gens.script +++ b/tests/no-roots-from-old-gens.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/notify-error-during-backup.script b/tests/notify-error-during-backup.script index 9ce589b8..f4aae0d4 100755 --- a/tests/notify-error-during-backup.script +++ b/tests/notify-error-during-backup.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/obnam b/tests/obnam index 5be673c6..52eebc63 100755 --- a/tests/obnam +++ b/tests/obnam @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/pre-epoch-mtime.script b/tests/pre-epoch-mtime.script index 9a0bf0ef..92d85bed 100755 --- a/tests/pre-epoch-mtime.script +++ b/tests/pre-epoch-mtime.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/pretend-time.script b/tests/pretend-time.script index f9d5731d..8296b157 100755 --- a/tests/pretend-time.script +++ b/tests/pretend-time.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/pretend-time.stdout b/tests/pretend-time.stdout index 4639ae6f..0c7f0c2b 100644 --- a/tests/pretend-time.stdout +++ b/tests/pretend-time.stdout @@ -1 +1 @@ -2 2007-08-12 01:02:03 .. 2007-08-12 01:02:03 (xx files, 100000 bytes) +2 2007-08-12 01:02:03 .. 2007-08-12 01:02:03 (xx files, 100000 bytes) diff --git a/tests/remove-checkpoints.script b/tests/remove-checkpoints.script index 43f1dcf1..1a70921c 100755 --- a/tests/remove-checkpoints.script +++ b/tests/remove-checkpoints.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/restore b/tests/restore index d68a8f28..a1021be3 100755 --- a/tests/restore +++ b/tests/restore @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/restores-compressed-without-being-told-to.script b/tests/restores-compressed-without-being-told-to.script index ce7d2afc..e5abc540 100755 --- a/tests/restores-compressed-without-being-told-to.script +++ b/tests/restores-compressed-without-being-told-to.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/restores-identical-data.script b/tests/restores-identical-data.script index 9e924e3d..00aedf79 100755 --- a/tests/restores-identical-data.script +++ b/tests/restores-identical-data.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/restores-single-file.script b/tests/restores-single-file.script index 18f898f9..748d3113 100755 --- a/tests/restores-single-file.script +++ b/tests/restores-single-file.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/root-is-symlink.script b/tests/root-is-symlink.script index be56dbd8..aaedd4e0 100755 --- a/tests/root-is-symlink.script +++ b/tests/root-is-symlink.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/setup b/tests/setup index 406556f6..773cdad2 100755 --- a/tests/setup +++ b/tests/setup @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/sparse-files.script b/tests/sparse-files.script index 45055382..dde35422 100755 --- a/tests/sparse-files.script +++ b/tests/sparse-files.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/symlink.script b/tests/symlink.script index ecb215df..4276d697 100755 --- a/tests/symlink.script +++ b/tests/symlink.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/teardown b/tests/teardown index 8b06588e..ab87a5f8 100755 --- a/tests/teardown +++ b/tests/teardown @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/two-generations.script b/tests/two-generations.script index 94a4717d..3c21fc29 100755 --- a/tests/two-generations.script +++ b/tests/two-generations.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/two-roots.script b/tests/two-roots.script index 672b6c99..10417f53 100755 --- a/tests/two-roots.script +++ b/tests/two-roots.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -31,7 +31,7 @@ summain -r "$DATADIR/data" "$DATADIR/data2" > "$DATADIR/data.summain" summain -r "$DATADIR/restored/$DATADIR/data" \ "$DATADIR/restored/$DATADIR/data2" > "$DATADIR/restored.summain" -# Timestamps are whole seconds with sftp, so we need to mangle the +# Timestamps are whole seconds with sftp, so we need to mangle the # summain output to remove sub-second timestamps. if [ "$OBNAM_TEST_SFTP_ROOT" = yes ] then diff --git a/tests/unreadable-dir.script b/tests/unreadable-dir.script index 3297176b..7533b589 100755 --- a/tests/unreadable-dir.script +++ b/tests/unreadable-dir.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/unreadable-file.script b/tests/unreadable-file.script index ace97b4c..1ad4f7f8 100755 --- a/tests/unreadable-file.script +++ b/tests/unreadable-file.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/use-old-node-size.script b/tests/use-old-node-size.script index 52f64536..7c905182 100755 --- a/tests/use-old-node-size.script +++ b/tests/use-old-node-size.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/verifies-randomly.script b/tests/verifies-randomly.script index 6caf87f5..03106406 100755 --- a/tests/verifies-randomly.script +++ b/tests/verifies-randomly.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/verify b/tests/verify index 68949271..d064a98b 100755 --- a/tests/verify +++ b/tests/verify @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. @@ -21,7 +21,7 @@ set -e summain -r "$DATADIR/data" > "$DATADIR/data.summain" summain -r "$DATADIR/restored/$DATADIR/data" > "$DATADIR/restored.summain" -# Timestamps are whole seconds with sftp, so we need to mangle the +# Timestamps are whole seconds with sftp, so we need to mangle the # summain output to remove sub-second timestamps. if [ "$OBNAM_TEST_SFTP_ROOT" = yes ] then diff --git a/tests/verify-notices-changes.script b/tests/verify-notices-changes.script index 09186da9..45e01e2b 100755 --- a/tests/verify-notices-changes.script +++ b/tests/verify-notices-changes.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/xattr-change-only.script b/tests/xattr-change-only.script index 197dea5e..0c8945a3 100755 --- a/tests/xattr-change-only.script +++ b/tests/xattr-change-only.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/tests/xattr-empty.script b/tests/xattr-empty.script index 8eed8727..ee0a002d 100755 --- a/tests/xattr-empty.script +++ b/tests/xattr-empty.script @@ -1,16 +1,16 @@ #!/bin/sh # Copyright 2012 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. diff --git a/try-node-chunk-sizes-plot-results b/try-node-chunk-sizes-plot-results index 98e8030c..d114d05d 100755 --- a/try-node-chunk-sizes-plot-results +++ b/try-node-chunk-sizes-plot-results @@ -19,7 +19,7 @@ def dat_filename(op, node_size): def plotfile(op, node_size): - return ('"%s" with lines title "node size %d KiB"' % + return ('"%s" with lines title "node size %d KiB"' % (dat_filename(op, node_size), node_size)) @@ -42,7 +42,7 @@ for op in ops: with open(name, 'w') as f: for chunk_size, secs in points: f.write('%d %f\n' % (chunk_size, secs)) - + gnuplot = '''\ set terminal svg dynamic set title "%s %s" @@ -50,7 +50,7 @@ set xlabel "chunk size (KiB)" set ylabel "time (s)" ''' % (sys.argv[1], op) - gnuplot += ('plot' + + gnuplot += ('plot' + ', '.join(plotfile(op, x) for x in sorted(data.keys())) + '\n') diff --git a/verification-test b/verification-test index 701eaad5..f5f9957a 100755 --- a/verification-test +++ b/verification-test @@ -14,7 +14,7 @@ # better to use an independent tool. summain(1) is a better choice than, # say, md5sum, since it includes much of the inode metadata in the manifest, # so that restoring file permissions, etc, are also verified. -# +# # To use this script, run it one the following ways: # # ./verification-test backup REPO DIR @@ -35,17 +35,17 @@ # between Obnam backing it up and summain including it in the manifest. # # Copyright 2011 Lars Wirzenius -# +# # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. |