path: root/1.0.mdwn

[[!meta title="Obnam 1.0 (backup software); a story in many words"]]
[[!meta date="Fri,  8 Jun 2012 07:09:33 +0000"]]

Date: Fri,  8 Jun 2012 07:09:33 +0000

**tl;dr:** Version 1.0 of [Obnam](http://liw.fi/obnam/), my 
snapshotting, de-duplicating, encrypting backup program is released.
See the end of this announcement for the details.

Where we see the hero in his formative years; parental influence
----------------------------------------------------------------

From the very beginning, my computing life has involved backups.

In 1984, when I was 14, 
[my father](http://www.kolumbus.fi/arnow/) was an independent 
telecommunications consultant, which meant he needed a personal computer
for writing reports. He bought a 
[Luxor ABC-802](http://en.wikipedia.org/wiki/ABC_800#ABC_802), 
a Swedish computer with a Z80 microprocessor and two floppy drives.

My father also taught me how to use it. When I needed to
save files, he gave me not one, but two floppies, and explained
that I should store my files one one, and then copy them to the
other one every now and then.

Later on, over the years, I've made backups from a hard disk 
(30 megabytes!) to
a stack of floppies, to a tape drive installed into
a floppy interface (400 megabytes!), to a DAT drive, and various other media.
It was always a bit tedious.

The start of the quest; lengthy justification for NIH
-----------------------------------------------------

In 2004, I decided to do a full backup, by burning a copy of all my
files onto CD-R disks. It took me most of the day. Afterwards, I sat
admiring the large stack of disks, and realized that I would not ever
do that again. I'm too lazy for that. That I had done it once was an
aberration in the space-time continuum.

Switching to DVD-Rs instead CD-Rs would reduce to the number of disks to 
burn, but not enough: it would still take a stack of them. 
I needed something much better.

I had a little experience with tape drives, and that was enough to convince
me that I didn't want them. Tape drives are expensive hardware,
and the tapes also cost money. If the drive goes bad, you have to get
a compatible one, or all your backups are toast. The price per gigabyte
was coming down fast for hard drives, and it was clear that they were
about to be very competitive with tapes for price.

I looked for backup programs that I could use for disk based backups.
`rsync`, of course, was the obvious choice, but there were others.
I ended up doing what many geeks do: I wrote my own wrapper around
`rsync`. There's hundred, possibly thousands, of such wrappers around
the Internet.

I also got the idea that doing a startup to provide online backup
space would be a really cool thing. However, I didn't really do
anything about that until 2007. More on that later.

The `rsync` wrapper script I wrote used hardlinked directory trees
to provide a backup history, though not in the smart way that
[backuppc](http://backuppc.sourceforge.net/) does it. 
The hardlinks were wonderful, because they were
cheap, and provided de-duplication. They were also quite cumbersome,
when I needed to move my backups to a new disk the first time. It
turned out that a lot of tools deal very badly with directory trees
with large numbers of hardlinks.

I also decided I wanted encrypted backups. This led me to find
[duplicity](http://duplicity.nongnu.org/), which is a nice program
that does encrypted backups, but I had issues with some of its
limitations. To fix those limitations, I would have had to re-design
and possibly re-implement the entire program. The biggest limitation 
was that it treated backups as full backup, plus a sequence of 
incremental backups, which were deltas against the previous backup.

Delta based incrementals make sense for tape drives. You run a full
backup once, then incremental deltas for every day. When enough time
has passed since the full backup, you do a new full backup, and then
future incrementals are based on that. Repeat forever.

I decided that this makes no sense for disk based backups. If I already
have backed up a file, there's no point in making me backup it again,
since it's already there on the same hard disk. It makes even less
sense for online backups, since doing a new full backup would require
me to transmit all the data all over again, even though it's already
on the server.

The first battle
----------------

I could not find a program that did what I wanted to do, and like
every good [NIHolic](http://en.wikipedia.org/wiki/Not_Invented_Here), 
I started writing my own.

After various aborted attempts, I started for real in 2006. Here is
the first commit message:

    revno: 1
    committer: Lars Wirzenius <liw@iki.fi>
    branch nick: wibbr
    timestamp: Wed 2006-09-06 18:35:52 +0300
    message:
      Initial commit.

`wibbr` was the placeholder name for Obnam until we came up with
something better. We was myself and Richard Braakman, who was going
to be doing the backup startup with me. We eventually founded the
company near the end of 2006, and started doing business in 2007.

However, we did not do very much business, and ran out of money in
September 2007. We ended the backup startup experiment.
That's when I took a job with Canonical, and Obnam became a hobby
project of mine: I still wanted a good backup tool. 

In September 2007, Obnam was working, but it was not very good. 
For example, it was quite slow and wasteful of backup space.

That version of Obnam used deltas, based on the `rsync` algorithm, to
backup only changes. It did not require the user to do full and
incremental backups manually, but essentially created an endless
sequence of incrementals. It was possible to remove any generation,
and Obnam would manage the deltas as necessary, keeping the ones
needed for the remaining generations, and removing the rest. 
Obnam made it look as if each generation was independent of each other.

The wasteful part was the way in which metadata about files was
stored: each generation stored the full list of filenames and their
permissions and other inode fields. This turned out to be bigger
than my daily delta.

The lost years; getting lost in the forest
------------------------------------------

For the next two years, I did a little work on Obnam, but I did not
make progress very fast. I changed the way metadata was stored, for
example, but I picked another bad way of doing it: the new way was
essentially building a tree of directory and file nodes, and any
unchanged subtrees were shared between generations. This reduced the
space overhead per generation, but made it quite slow to look up
the metadata for any one file.

The final battle; finding cows in the forest
--------------------------------------------

In 2009 I decided to leave Canonical and after that, my Obnam hobby 
picked up in speed again. Below is a table of the number of commits
per year, from the very first commit (`bzr log -n0 | 
awk '/timestamp:/ { print $3}' | sed 's/-.*//' | uniq -c | 
awk '{ print $2, $1 }' | tac`):

    2006 466
    2007 353
    2008 402
    2009 467
    2010 616
    2011 790
    2012 282

During most of 2010 and 2011 I was unemployed, and happily hacking
Obnam, while moving to another country twice. I don't recommend that
as a way to hack on hobby projects, but it worked for me.

After Canonical, I decided to tackle the way Obnam stores data from
a new angle. Richard told me about the copy-on-write (or COW) B-trees that
btrfs uses, originally designed by Ohad Rodeh
(see [his paper](http://liw.fi/larch/ohad-btrees-shadowing-clones.pdf)
for details),
and I started reading about that. It turned out that
they're pretty ideal for backups: each B-tree stores data about
one generation. To start a new generation, you clone the previous
generation's B-tree, and make any modifications you need.

I implemented the B-tree library myself, in Python. 
I wanted something that
was flexible about how and where I stored data, which the btrfs
implementation did not seem to give me. (Also, I worship at the 
altar of NIH.)

With the B-trees, doing file deltas from the previous generation
no longer made any sense. I realized that it was, in any case, a
better idea to store file data in chunks, and re-use chunks in
different generations as needed. This makes it much easier to
manage changes to files: with deltas, you need to keep a long chain
of deltas and apply many deltas to reconstruct a particular version.
With lists of chunks, you just get the chunks you need.

The spin-off franchise; lost in a maze of dependencies, all alike
-----------------------------------------------------------------

In the process of developing Obnam, I have split off a number of
helper programs and libraries:

* [genbackupdata](http://liw.fi/genbackupdata/) 
  generates reproducible test data for backups
* [seivot](http://liw.fi/seivot/) 
  runs benchmarks on backup software (although only Obnam for now)
* [cliapp](http://liw.fi/cliapp/) 
  is a Python framework for command line applications
* [cmdtest](http://liw.fi/cmdtest/) 
  runs black box tests for Unix command line applications
* [summain](http://liw.fi/summain/) 
  makes diff-able file manifests (`md5sum` on steroids),
  useful for verifying that files are restored correctly
* [tracing](http://liw.fi/tracing/) 
  allows run-time selectable debug log messages that is really
  fast during normal production runs when messages are not printed
  
I have found it convenient to keep these split off, since I've been
able to use them in other projects as well. However, it turns out that
those installing Obnam don't like this: it would probably make sense to
have a fat release with Obnam and all dependencies, but I haven't bothered
to do that yet.

The blurb; readers advised about blatant marketing
--------------------------------------------------

The strong points of Obnam are, I think:

* **Snapshot** backups, similar to btrfs snapshot subvolumes. 
  Every generation looks like a complete snapshot,
  so you don't need to care about full versus incremental backups, or
  rotate real or virtual tapes.
  The generations share data as much as possible,
  so only changes are backed up each time.
* Data **de-duplication**, across files, and backup generations. If the
  backup repository already contains a particular chunk of data, it will
  be re-used, even if it was in another file in an older backup
  generation. This way, you don't need to worry about moving around large
  files, or modifying them.
* **Encrypted** backups, using GnuPG.

Backups may be stored on local hard disks (e.g., USB drives), any
locally mounted network file shares (NFS, SMB, almost anything with
remotely Posix-like semantics), or on any SFTP server you have access to.

What's not so strong is backing up online over SFTP, particularly with
long round trip times to the server, or many small files to back up. 
That performance is Obnam's weakest part. I hope to fix that in the future,
but I don't want to delay 1.0 for it.

The big news; readers sighing in relief
---------------------------------------

I am now ready to release version 1.0 of Obnam. Finally. It's been
a long project, much longer than I expected, and much longer than
was really sensible. However, it's ready now. It's not bug free, and
it's not as fast as I would like, but it's time to declare it ready
for general use. If nothing else, this will get more people to use
it, and they'll find the remaining problems faster than I can do on
my own.

I have packaged Obnam for Debian, and it is in `unstable`, and will
hopefully get into `wheezy` before the Debian freeze. I provide
packages built for `squeeze` on my own repository,
see the [download](http://liw.fi/obnam/download/) page.

The changes in the 1.0 release compared to the previous one:

* Fixed bug in finding duplicate files during a backup generation.
  Thanks to Saint Germain for reporting the problem.
* Changed version number to 1.0.

The future; not including winning lottery numbers
-------------------------------------------------

I expect to get a flurry of bug reports in the near future as new people
try Obnam. It will take a bit of effort dealing with that. Help is, of
course, welcome!

After that, I expect to be mainly working on Obnam performance for the
foreseeable future. There may also be a FUSE filesystem interface for
restoring from backups, and a continous backup version of Obnam. Plus
other features, too. 

I make no promises about how fast new features
and optimizations will happen: Obnam is a hobby project for me, and I
work on it only in my free time. Also, I have a bunch of things that
are on hold until I get Obnam into shape, and I may decide to do one
of those things before the next big Obnam push.

Where; the trail of an errant hacker
------------------------------------

I've developed Obnam in a number of physical locations, and I thought
it might be interesting to list them:
Espoo, Helsinki, Vantaa, Kotka, Raahe, Oulu, Tampere, Cambridge, Boston,
Plymouth, London, Los Angeles, Auckland, Wellington, Christchurch,
Portland, New York, Edinburgh, Manchester, San Giorgio di Piano.
I've also hacked on Obnam in trains, on planes, and once on a ship,
but only for a few minutes on the ship before I got seasick.

Thank you; sincerely
--------------------

* Richard Braakman, for helping me with ideas, feedback, and some
  code optimizations, and for doing the startup with me. Even though
  he has provided little code, he's Obnam's most significant contributor
  so far.
* [Chris Cormack](http://blog.bigballofwax.co.nz/), for helping to build
  Obnam for Ubuntu. I no longer use Ubuntu at all, so it's a big help to
  not have to worry about building and testing packages for it.
* [Daniel Silverstone](http://www.digital-scurf.org/), for spending a
  Saturday with me hacking Obnam, and rewriting the way repository file
  filters work (compression, encryption), thus making them not suck.
* [Tapani Tarvainen](http://tapani.tarvainen.info/) for running Obnam for
  serious amounts of real data, and for being patient while I fixed things.
* [Soile Mottisenkangas](http://docstory.fi/) for believing in me, and
  helping me overcome periods of despair.
* Everyone else who has tried Obnam and reported bugs or provided any
  other feedback. I apologize for not listing everyone.

SEE ALSO
--------

* [Obnam home page](http://liw.fi/obnam/)
  - [support](http://liw.fi/obnam/status/)
  - [NEWS](http://liw.fi/obnam/NEWS/)
  - [README](http://liw.fi/obnam/README/)
  - [manual page](http://liw.fi/obnam/obnam.1.txt)
  - [design documents](http://liw.fi/obnam/development/)
  - [Debian QA package page](http://packages.qa.debian.org/o/obnam.html)
  - [bugs](http://liw.fi/obnam/bugs/)
  - [bugs in Debian](http://bugs.debian.org/obnam)
* [Other projects of mine (many are dependencies of 
  Obnam)](http://liw.fi/tag/program/)