Merged from trunk.

10 files changed, 258 insertions, 181 deletions

diff --git a/Makefile b/Makefile
index aa73aab3..cb54d756 100644
--- a/Makefile
+++ b/Makefile
@@ -14,7 +14,7 @@ check: all
 	python blackboxtest
 	
 clean:
-	rm -f _obnam.so obnamlib/*.pyc obnamlib/plugins/*.pyc testplugins/*.pyc
+	rm -f _obnam.so obnamlib/*.pyc obnamlib/plugins/*.pyc test-plugins/*.pyc
 	rm -f blackboxtest.log blackboxtest-obnam.log obnam.prof
 	rm -f obnam-new-data.png obnam-store.png obnam-xfer.png obnam.seivot
 
diff --git a/README b/README
new file mode 100644
index 00000000..34ea3213
--- /dev/null
+++ b/README
@@ -0,0 +1,117 @@
+Obnam, a backup program
+=======================
+
+Obnam is a backup program. It is currently ALPHA quality, meaning that
+some things work, but many things don't, there are probably a lot of bugs,
+and if you rely on it, you will turn bitter and hateful, and will abandon
+all hope for civilisation, move into the rain forest, and talk to birds
+for the rest of your life.
+
+In other words, don't use it for anything real yet.
+
+
+Installation
+------------
+
+The source tree contains packaging for Debian. Run "debuild -us -uc" to
+build an installation package.
+
+On other systems, using the setup.py file might work: run
+"python setup.py --help" for advice. If not, please tell me how to fix it.
+
+You need to install my B-tree and LRU libraries, which you can get from:
+
+    http://liw.fi/btree/
+    http://liw.fi/python-lru/
+
+
+Use
+---
+
+To get a quick help summary of options:
+
+    ./obnam --help
+    
+To make a backup:
+
+    ./obnam backup --store /tmp/mybackup $HOME
+
+For more information, see the manual page:
+
+    man -l obnam.1 
+
+
+Hacking
+-------
+
+To build:
+
+    make
+    
+To run automatic tests:
+
+    make check
+    
+You need my CoverageTestRunner to run tests, get it from:
+
+    http://liw.fi/coverage-test-runner/
+
+A couple of scripts exist to run benchmarks and profiles:
+
+    ./run-benchmark
+    ./run-profile
+    ./viewprof obnam.prof cumulative | less -S
+
+The canonical version control repository for obnam itself is at:
+
+    http://code.liw.fi/obnam/bzr/
+
+The rewrite4 branch is currently the main one.
+
+If you make any changes, I welcome patches, either as plain diffs, bzr
+bundles, or public repositories I can merge from.
+
+The code layout is roughly like this:
+
+    obnamlib/               # all the real code
+    obnamlib/plugins/       # the plugin code (see pluginmgr.py)
+    obnam                   # script to invoke obnam
+    _obnammodule.c          # wrapper around some system calls
+
+In obnamlib, every code module has a corresponding test module,
+and "make check" uses CoverageTestRunner to run them pairwise. For
+each pair, test coverage must be 100% or the test will fail.
+Mark statements that should not be included in coverage test with
+"# pragma: no cover", if you really, really can't write a test.
+without-tests lists modules that have no test modules.
+
+
+Feedback
+--------
+
+I welcome bug fixes, enhancements, bug reports, suggestions, requests,
+and other feedback. I prefer e-mail (mailto:liw@liw.fi).
+
+
+Legal stuff
+-----------
+
+Most of the code is written by Lars Wirzenius. (Please provide patches
+so that can change.)
+
+The code is covered by the GNU General Public License, version 3 or later.
+
+Copyright 2010  Lars Wirzenius
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+ 
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
diff --git a/obnamlib/app.py b/obnamlib/app.py
index 0b366928..8f7401f0 100644
--- a/obnamlib/app.py
+++ b/obnamlib/app.py
@@ -31,6 +31,10 @@ class App(object):
         self.config = obnamlib.Configuration([])
         self.config.new_string(['log'], 'name of log file (%default)')
         self.config['log'] = 'obnam.log'
+        self.config.new_string(['log-level'], 
+                               'log level, one of debug, info, warning, '
+                               'error, critical (%default)')
+        self.config['log-level'] = 'info'
         self.config.new_string(['store'], 'name of backup store')
         self.config.new_string(['hostname'], 'name of host (%default)')
         self.config['hostname'] = self.deduce_hostname()
@@ -46,6 +50,7 @@ class App(object):
         self.register_command = self.interp.register
 
         self.hooks.new('plugins-loaded')
+        self.hooks.new('config-loaded')
         self.hooks.new('shutdown')
         
         self.fsf = obnamlib.VfsFactory()
@@ -61,7 +66,16 @@ class App(object):
         handler = logging.FileHandler(self.config['log'])
         handler.setFormatter(formatter)
         logger = logging.getLogger()
-        logger.setLevel(logging.DEBUG)
+        levels = {
+            'debug': logging.DEBUG,
+            'info': logging.INFO,
+            'warning': logging.WARNING,
+            'error': logging.ERROR,
+            'critical': logging.CRITICAL,
+        }
+        level_name = self.config['log-level']
+        level = levels.get(level_name.lower(), logging.DEBUG)
+        logger.setLevel(level)
         logger.addHandler(handler)
         
     def run(self):
@@ -69,6 +83,7 @@ class App(object):
         self.pm.enable_plugins()
         self.hooks.call('plugins-loaded')
         self.config.load()
+        self.hooks.call('config-loaded')
         self.setup_logging()
         if self.config.args:
             self.interp.execute(self.config.args[0], self.config.args[1:])
diff --git a/obnamlib/plugins/backup_plugin.py b/obnamlib/plugins/backup_plugin.py
index 4e270e43..9b470993 100644
--- a/obnamlib/plugins/backup_plugin.py
+++ b/obnamlib/plugins/backup_plugin.py
@@ -68,8 +68,6 @@ class BackupPlugin(obnamlib.ObnamPlugin):
             self.fs.close()
         self.store.commit_host()
 
-        self.app.hooks.call('progress-found-file', None, 0)
-
         logging.debug('backup finished')
 
     def backup_parents(self, root):
diff --git a/obnamlib/plugins/terminal_status_plugin.py b/obnamlib/plugins/terminal_status_plugin.py
index 64d1e832..6675ee93 100644
--- a/obnamlib/plugins/terminal_status_plugin.py
+++ b/obnamlib/plugins/terminal_status_plugin.py
@@ -20,161 +20,57 @@ import struct
 import sys
 import termios
 import time
+import ttystatus
 
 import obnamlib
 
 
-class TerminalStatus(object):
-
-    '''Update status to terminal.
-    
-    This is not the plugin itself, this just takes care of updating things
-    to the terminal.
-    
-    '''
-    
-    def __init__(self):
-        self.written = ''
-        self.cached = ''
-        self.when = 0
-        self.freq = 1.0
-        
-        self.width = self.get_terminal_width() - 1
-        signal.signal(signal.SIGWINCH, self.sigwinch_handler)
-
-    def get_terminal_width(self):
-        '''Return width of terminal in characters.
-
-        If this fails, assume 80.
-        
-        Borrowed and adapted from bzrlib.
-        
-        '''
-        
-        try:
-            s = struct.pack('HHHH', 0, 0, 0, 0)
-            x = fcntl.ioctl(1, termios.TIOCGWINSZ, s)
-            return struct.unpack('HHHH', x)[1]
-        except IOError:
-            return 80
-
-    def sigwinch_handler(self, signum, frame):
-        # Clear the terminal from old stuff, using the old width.
-        self.clear()
-        # Subtract one from actual terminal width to avoid wrapping
-        # by printing to the last character position in the line.
-        self.width = self.get_terminal_width() - 1
-
-    def raw_write(self, msg):
-        if sys.stdout.isatty():
-            sys.stdout.write('\b \b' * len(self.written))
-            sys.stdout.write(msg)
-            sys.stdout.flush()
-            self.written = msg
-    
-    def write(self, msg, force=False):
-        msg = msg[:self.width]
-        if force:
-            self.raw_write(msg)
-        else:
-            now = time.time()
-            if now - self.when >= self.freq:
-                self.raw_write(msg)
-                self.when = now
-        self.cached = msg
-
-    def clear(self):
-        self.write('', force=True)
-
-    def progress(self, message):
-        self.write('%s' % message)
-
-    def notify(self, message):
-        cached = self.cached
-        self.clear()
-        sys.stdout.write('%s\n' % message)
-        sys.stdout.flush()
-        self.write(cached, force=True)
-
-    def finished(self):
-        if self.cached:
-            self.write(self.cached, force=True)
-            sys.stdout.write('\n')
-        sys.stdout.flush()
-
-
 class TerminalStatusPlugin(obnamlib.ObnamPlugin):
 
-    units = [
-        (2**30, 'GiB'),
-        (2**20, 'MiB'),
-        (2**10, 'KiB'),
-        (0, 'B'),
-    ]
-
     def enable(self):
-        self.ts = TerminalStatus()
+        self.app.config.new_boolean(['quiet'], 'be silent')
+
+        self.ts = ttystatus.TerminalStatus(period=0.25)
+        self.ts['data_done'] = 0
+
         self.app.hooks.new('status')
         self.app.hooks.new('progress-found-file')
         self.app.hooks.new('progress-data-done')
         self.app.hooks.new('error-message')
+
         self.add_callback('status', self.status_cb)
         self.add_callback('progress-found-file', self.found_file_cb)
         self.add_callback('progress-data-done', self.data_done_cb)
         self.add_callback('error-message', self.error_message_cb)
-        self.add_callback('shutdown', self.ts.finished)
-        self.app.config.new_boolean(['quiet'], 'be silent')
-        self.current = ''
-        self.num_files = 0
-        self.total_data = 0
-        self.data_done = 0
+        self.add_callback('config-loaded', self.config_loaded_cb)
+        self.add_callback('shutdown', self.shutdown_cb)
         
     def disable(self):
+        self.ts.finish()
         self.ts = None
 
+    def config_loaded_cb(self):
+        if not self.app.config['quiet']:
+            self.ts.add(ttystatus.ElapsedTime())
+            self.ts.add(ttystatus.Literal(' '))
+            self.ts.add(ttystatus.Counter('current'))
+            self.ts.add(ttystatus.Literal(' files; '))
+            self.ts.add(ttystatus.ByteSize('data_done'))
+            self.ts.add(ttystatus.Literal(' '))
+            self.ts.add(ttystatus.Pathname('current'))
+
     def found_file_cb(self, filename, size):
-        self.current = filename
-        self.num_files += 1
-        self.total_data += size
-        self.update()
+        self.ts['current'] = filename
         
     def data_done_cb(self, amount):
-        self.data_done += amount
-        self.update()
+        self.ts['data_done'] += amount
 
     def status_cb(self, msg):
         if not self.app.config['quiet']:
             self.ts.notify(msg)
 
     def error_message_cb(self, msg):
-        if self.app.config['quiet']:
-            sys.stderr.write('Error: %s\n' % msg)
-        else:
-            self.ts.clear()
-            sys.stderr.write('Error: %s\n' % msg)
-            self.update()
-
-    def find_unit(self, bytes):
-        for factor, unit in self.units:
-            if bytes >= factor:
-                return factor, unit
-        
-    def scale(self, factor, bytes):
-        if factor > 0:
-            return '%.1f' % (float(bytes) / float(factor))
-        else:
-            return '0'
-
-    def update(self):
-        if self.app.config['quiet']:
-            return
-        factor, unit = self.find_unit(min(self.total_data, self.data_done))
-        total = self.scale(factor, self.total_data)
-        done = self.scale(factor, self.data_done)
-        if self.current:
-            tail = ' now: %s' % self.current
-        else:
-            tail = ''
-        self.ts.progress('%d files, %s/%s %s%s' %
-                         (self.num_files, done, total, unit, tail))
+        self.ts.notify('Error: %s' % msg)
 
+    def shutdown_cb(self):
+        self.ts.finish()
diff --git a/obnamlib/store.py b/obnamlib/store.py
index 41cf29c2..7751409d 100644
--- a/obnamlib/store.py
+++ b/obnamlib/store.py
@@ -97,12 +97,16 @@ class NodeStoreVfs(btree.NodeStoreDisk):
     def __init__(self, fs, dirname, node_size, codec):
         btree.NodeStoreDisk.__init__(self, dirname, node_size, codec)
         self.fs = fs
+        
+    def mkdir(self, dirname):
+        if not self.fs.exists(dirname):
+            self.fs.mkdir(dirname)
 
     def read_file(self, filename):
         return self.fs.cat(filename)
 
     def write_file(self, filename, contents):
-        self.fs.overwrite_file(filename, contents)
+        self.fs.overwrite_file(filename, contents, make_backup=False)
 
     def file_exists(self, filename):
         return self.fs.exists(filename)
@@ -476,48 +480,29 @@ class ChecksumTree(StoreTree):
         self.sumlen = checksum_length
         key_bytes = len(self.key('', 0))
         StoreTree.__init__(self, fs, name, key_bytes, 64*1024)
-        self.max_counter = 2**64 - 1
+        self.max_id = 2**64 - 1
 
-    def key(self, checksum, counter):
-        return struct.pack('!%dsQ' % self.sumlen, checksum, counter)
+    def key(self, checksum, number):
+        return struct.pack('!%dsQ' % self.sumlen, checksum, number)
 
     def unkey(self, key):
         return struct.unpack('!%dsQ' % self.sumlen, key)
 
-    def idstr(self, identifier):
-        return struct.pack('!Q', identifier)
-
-    def idunstr(self, idstr):
-        return struct.unpack('!Q', idstr)[0]
-
     def add(self, checksum, identifier):
         self.require_forest()
+        key = self.key(checksum, identifier)
         if self.forest.trees:
             t = self.forest.trees[-1]
-            pairs = t.lookup_range(self.key(checksum, 0),
-                                   self.key(checksum, self.max_counter))
-            idstr = self.idstr(identifier)
-            biggest = ''
-            for key, value in pairs:
-                biggest = max(biggest, key)
-                if value == idstr:
-                    break
-            else:
-                if biggest:
-                    dummy, counter = self.unkey(biggest)
-                else:
-                    counter = 0
-                t.insert(self.key(checksum, counter + 1), idstr)
         else:
             t = self.forest.new_tree()
-            t.insert(self.key(checksum, 0), self.idstr(identifier))
+        t.insert(key, '')
 
     def find(self, checksum):
         if self.init_forest() and self.forest.trees:
             t = self.forest.trees[-1]
             pairs = t.lookup_range(self.key(checksum, 0),
-                                   self.key(checksum, self.max_counter))
-            return [self.idunstr(value) for key, value in pairs]
+                                   self.key(checksum, self.max_id))
+            return [self.unkey(key)[1] for key, value in pairs]
         else:
             return []
 
@@ -525,12 +510,7 @@ class ChecksumTree(StoreTree):
         self.require_forest()
         if self.forest.trees:
             t = self.forest.new_tree(self.forest.trees[-1])
-            pairs = t.lookup_range(self.key(checksum, 0),
-                                   self.key(checksum, self.max_counter))
-            idstr = self.idstr(identifier)
-            for key, value in pairs:
-                if value == idstr:
-                    t.remove(key)
+            t.remove(self.key(checksum, identifier))
 
 
 class ChunkGroupTree(StoreTree):
diff --git a/obnamlib/vfs.py b/obnamlib/vfs.py
index 1bbce57f..b3fb62f1 100644
--- a/obnamlib/vfs.py
+++ b/obnamlib/vfs.py
@@ -156,11 +156,12 @@ class VirtualFileSystem(object):
 
         """
 
-    def overwrite_file(self, pathname, contents):
+    def overwrite_file(self, pathname, contents, make_backup=True):
         """Like write_file, but overwrites existing file.
 
         The old file isn't immediately lost, it gets renamed with
-        a backup suffix.
+        a backup suffix. The backup file is removed if make_backup is
+        set to False (default is True).
 
         """
 
diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py
index 67b5c7b5..bed45ce1 100644
--- a/obnamlib/vfs_local.py
+++ b/obnamlib/vfs_local.py
@@ -140,7 +140,7 @@ class LocalFS(obnamlib.VirtualFileSystem):
             raise
         os.remove(name)
 
-    def overwrite_file(self, pathname, contents):
+    def overwrite_file(self, pathname, contents, make_backup=True):
         path = self.join(pathname)
         dirname = os.path.dirname(path)
         fd, name = tempfile.mkstemp(dir=dirname)
@@ -163,6 +163,11 @@ class LocalFS(obnamlib.VirtualFileSystem):
         except OSError:
             pass
         os.rename(name, path)
+        if not make_backup:
+            try:
+                os.remove(bak)
+            except OSError:
+                pass
 
     def listdir(self, dirname):
         return os.listdir(self.join(dirname))
diff --git a/obnamlib/vfs_local_tests.py b/obnamlib/vfs_local_tests.py
index c0a1d098..7c0d300a 100644
--- a/obnamlib/vfs_local_tests.py
+++ b/obnamlib/vfs_local_tests.py
@@ -250,6 +250,15 @@ class LocalFSTests(unittest.TestCase):
         self.fs.overwrite_file("foo", "foobar")
         self.assertEqual(self.fs.cat("foo.bak"), "bar")
 
+    def test_overwrite_removes_bak_file(self):
+        self.fs.write_file("foo", "bar")
+        self.fs.overwrite_file("foo", "foobar", make_backup=False)
+        self.assertFalse(self.fs.exists("foo.bak"))
+
+    def test_overwrite_is_ok_without_bak(self):
+        self.fs.overwrite_file("foo", "foobar", make_backup=False)
+        self.assertFalse(self.fs.exists("foo.bak"))
+
     def test_overwrite_replaces_existing_file(self):
         self.fs.write_file("foo", "bar")
         self.fs.overwrite_file("foo", "foobar")
diff --git a/run-benchmark b/run-benchmark
index 6da8f82b..940a35c5 100755
--- a/run-benchmark
+++ b/run-benchmark
@@ -15,17 +15,73 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+set -e
+
+die()
+{
+    echo "$@" 1>&2
+    exit 1
+}
+
 SEIVOT=seivot
-REVISION=$(cd "$BRANCH" && bzr revno)
-
-$SEIVOT \
-    --output="obnam.seivot" \
-    --program=obnam \
-    --revision="$REVISION" \
-    --start-size=1m \
-    --inc-size=100k \
-    --fullcmd="$(pwd)/obnam --log /dev/null --store \$STORE backup \$DATA" \
-    --generations=10 \
-    measure
-$SEIVOT report --output="obnam.png" obnam.seivot
+OBNAM_BRANCH="."
+BTREE_BRANCH="$HOME/btree/trunk"
+
+if [ $# -ge 1 ]
+then
+    OBNAM_BRANCH="$1"
+    shift
+fi
+
+if [ $# -ge 1 ]
+then
+    BTREE_BRANCH="$1"
+    shift
+fi
 
+if [ $# -ge 1 ]
+then
+    SEIVOT="$1"
+    shift
+fi
+
+if [ $# != 0 ]
+then
+    die "Usage: $0 [obnam-branch [btree-branch [path/to/seivot]]]"
+fi
+
+SIZES="1m/1m 10m/1m 100m/10m 1000m/100m 10000m/100m"
+GENERATIONS=5
+OBNAM_REVNO=$(cd "$OBNAM_BRANCH" && bzr revno)
+BTREE_REVNO=$(cd "$BTREE_BRANCH" && bzr revno)
+
+for pair in $SIZES
+do
+    size=$(echo "$pair" | sed 's:/.*::')
+    inc=$(echo "$pair" | sed 's:.*/::')
+    echo "Benchmark run for size $size inc $inc"
+    basename="obnam-$OBNAM_REVNO-$BTREE_REVNO-$size"
+    data="$basename.seivot"
+    desc="obnam (r$OBNAM_REVNO) and btree (r$BTREE_REVNO)"
+    desc="$desc for live data size $size with $inc increments"
+    $SEIVOT \
+        --output="$data" \
+        --description="$desc" \
+        --program="obnam+btree" \
+        --revision="$OBNAM_REVNO and $BTREE_REVNO" \
+        --start-size=$size \
+        --inc-size=$inc \
+        --generations=$GENERATIONS \
+        --fullcmd="env OBNAM_PROFILE=obnam.prof \
+                   PYTHONPATH=$BTREE_BRANCH $OBNAM_BRANCH/obnam \
+                   --log /dev/null --store \$STORE backup \$DATA && \
+                   cp obnam.prof $basename-\$GEN.prof && \
+                   $(pwd)/viewprof obnam.prof cumulative \
+                    > $basename-\$GEN-cumulative.txt && \
+                   $(pwd)/viewprof obnam.prof time \
+                    > $basename-\$GEN-time.txt && \
+                   rm -f obnam.prof \
+                    " \
+    measure
+    $SEIVOT report --output="$basename.png" "$data"
+done