Date: Mon, 24 Jul 2017 13:19:42 -0400
From: Ben Boeckel <mathstuf@gmail.com>
To: "Laurence Perkins (OE)" <lperkins@openeye.net>
Message-ID: <20170724171942.GA17269@megas.kitware.com>
References: <1500484994.13826.5.camel@openeye.net>
 <20170719181232.sdqihqdqldsgzmtd@liw.fi>
 <1500571405.13826.8.camel@openeye.net>
 <20170722141756.yzxatuvogrdsh4jv@liw.fi>
 <1500916329.13826.13.camel@openeye.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <1500916329.13826.13.camel@openeye.net>
User-Agent: Mutt/1.8.3 (2017-05-23)
Cc: "obnam-support@obnam.org" <obnam-support@obnam.org>,
 "liw@liw.fi" <liw@liw.fi>
Subject: Re: Variable Chunksize
Precedence: list
Reply-To: mathstuf@gmail.com
Sender: obnam-support-bounces@obnam.org
Errors-To: obnam-support-bounces@obnam.org

On Mon, Jul 24, 2017 at 17:12:15 +0000, Laurence Perkins (OE) wrote:
> Which just made me realise what kind of space savings I could get on
> one of my archives...  I guess this just moved up the priority list a
> ways...  Any suggestions for a hash that's fast enough to crawl over
> large files without hurting performance too much while still giving
> reasonable control over the average chunk size?

What about the hashing used by casync in its chunking? From its
announcement[1]:

    The "chunking" algorithm is based on a the buzhash rolling hash
    function. SHA256 is used as strong hash function to generate digests
    of the chunks. xz is used to compress the individual chunks.

It sounds like you use a rolling hash to find the chunks and then store
them according to their SHA256 hash.

--Ben

[1]http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html

_______________________________________________
obnam-support mailing list
obnam-support@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org