From: "Laurence Perkins (OE)" <lperkins@openeye.net>
To: "liw@liw.fi" <liw@liw.fi>
Thread-Topic: Variable Chunksize
Thread-Index: AQHTALO1js8PRLWMaEGpkNs8UOGlYqJb6R8AgAGEm4CAAvDXAIADVVeA
Date: Mon, 24 Jul 2017 17:12:15 +0000
Message-ID: <1500916329.13826.13.camel@openeye.net>
References: <1500484994.13826.5.camel@openeye.net>
 <20170719181232.sdqihqdqldsgzmtd@liw.fi>
 <1500571405.13826.8.camel@openeye.net>
 <20170722141756.yzxatuvogrdsh4jv@liw.fi>
In-Reply-To: <20170722141756.yzxatuvogrdsh4jv@liw.fi>
Accept-Language: en-US
Content-Language: en-US
MIME-Version: 1.0
Cc: "obnam-support@obnam.org" <obnam-support@obnam.org>
Subject: Re: Variable Chunksize
Precedence: list
Content-Type: multipart/mixed; boundary="===============0226912654442020829=="
Mime-version: 1.0
Sender: obnam-support-bounces@obnam.org
Errors-To: obnam-support-bounces@obnam.org

--===============0226912654442020829==
Content-Language: en-US
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature"; boundary="=-mkTgCnw/w4FzwjwkBM2+"

--=-mkTgCnw/w4FzwjwkBM2+
Content-Type: text/plain; charset="UTF-7"
Content-Transfer-Encoding: quoted-printable


On Sat, 2017-07-22 at 17:17 +-0300, Lars Wirzenius wrote:
+AD4 For more precise deduplication, it's almost certainly necessary to
+AD4 use
+AD4 smaller chunks than the megabyte sized ones Obnam currently uses by
+AD4 default. Smaller chunks mean more chunks.
+AD4=20
+AD4=20
Smaller chunk size makes deduplication more precise regardless of the
type of splitting, but it should generate some pretty big savings on
similar data without reducing the chunk size because it will be better
at finding identical chunks of data since it's not relying on them
being at fixed offsets.

To illustrate, consider the following two datasets:

aaaaabbbbb

caaaaabbbbb

With a fixed chunk size of 5, the first dataset makes two chunks and
the second makes three with no chunks shared.

With a variable chunk algorithm that splits when it sees +ACI-ab+ACI, they
both make two chunks, and they share one.

As with all deduplication, the chunk size must be tuned to the type of
data you're working with, but even with the current 1MB chunk size,
files larger than that will benefit any time they get a value inserted
into the middle.  Which is a common thing for files like sparse VM
images and the like.

Which just made me realise what kind of space savings I could get on
one of my archives...  I guess this just moved up the priority list a
ways...  Any suggestions for a hash that's fast enough to crawl over
large files without hurting performance too much while still giving
reasonable control over the average chunk size?

LMP
--=-mkTgCnw/w4FzwjwkBM2+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEFbYe3ereZkZxAoz7C4CSuysVUSAFAll2KmkACgkQC4CSuysV
USCyJxAAoHCoSWxXpswfJCx2bNm4YoERGJEPYE+a1G3zbj1xnolpDZOzji6vPesB
HqIMYDzohU2fH1dqPZLDzQ8kRLrnHckrevEgdPHCRIX8ZSAS2O+tvXi9gq2CkDZ6
LrR8pbIJZ+1zWeoh6uFWmA3vN/502gyAy2aacXhJyyT7kl1u+wGpkgsyHKRSP0XJ
PU4Ecxp5nWiYsCEq7R2prowLTBGO0Of+uSIw4xlT8M4iAwq/CiBp1dLK0Tg6P1aE
97+GOxcv73fMSTERRxErfl+/Ti3KfORvp8kjYKE29TjFtsBFEw5tfedLfLk81Ae1
0aorAJM+Reg/6T8D7CYhMwcFmDG08vRQwy1OlKVaJCAn/CyOkUyaCMpEFZXZOewy
a380AvAPY6h9xYebl3Ot5MwQCLUHYVVVhKRrkOgr0cULrIuGZobUVFPHREUDwbvR
fTDFEQuoLv11NJhyA/CwGbUKLtjtiByG9EKNEUo24vj+sTNcFL87pdpQxREwgCul
Fehbaq2MRs7M6iMwDFBLPMlswMu/m66bnTQ+4e2zABiXtmlwGkWejJxY9EUgLG/Z
FOwY5qXuDF315rwHkmHUXtdVXI1zn9Ke+Rf+rB2XxpGUzv8JRHbBQbGGjVCObFxV
il40F4A6VPbiO2E7lhZ5RzytL1Smchois+43fvkahCJhiH07HsE=
=ZJgW
-----END PGP SIGNATURE-----

--=-mkTgCnw/w4FzwjwkBM2+--


--===============0226912654442020829==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
obnam-support mailing list
obnam-support@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org

--===============0226912654442020829==--