diff options
author | distix ticketing system <distix@pieni.net> | 2017-07-19 18:16:36 +0000 |
---|---|---|
committer | distix ticketing system <distix@pieni.net> | 2017-07-19 18:16:36 +0000 |
commit | f481f72c612fcfa79c8dec6f03899352b41ef8e5 (patch) | |
tree | f5ff9031814c20586b3ec611aff79f84cbc38734 | |
parent | 1f1e64259f7217d4ebf29daf42c9db3ae5b21cb1 (diff) | |
download | obnam-support-distix-f481f72c612fcfa79c8dec6f03899352b41ef8e5.tar.gz |
imported mails
-rw-r--r-- | tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom | 174 |
1 files changed, 174 insertions, 0 deletions
diff --git a/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom b/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom new file mode 100644 index 0000000..98fe8b1 --- /dev/null +++ b/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom @@ -0,0 +1,174 @@ +Return-Path: <obnam-support-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221]) + by pieni.net (Postfix) with ESMTPS id 8990D40668 + for <distix@pieni.net>; Wed, 19 Jul 2017 18:12:46 +0000 (UTC) +Received: from platypus.pepperfish.net (unknown [10.112.101.20]) + by yaffle.pepperfish.net (Postfix) with ESMTP id 716454186A; + Wed, 19 Jul 2017 19:12:46 +0100 (BST) +Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1dXtSo-0004Ki-Dm; Wed, 19 Jul 2017 19:12:46 +0100 +Received: from [10.112.101.21] (helo=mx3.pepperfish.net) + by platypus.pepperfish.net with esmtps (Exim 4.80 #2 (Debian)) + id 1dXtSn-0004KT-HZ + for <obnam-support@obnam.org>; Wed, 19 Jul 2017 19:12:45 +0100 +Received: from koom.pieni.net ([88.99.190.206] helo=pieni.net) + by mx3.pepperfish.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) + (Exim 4.89) (envelope-from <liw@liw.fi>) id 1dXtSj-0002rJ-KP + for obnam-support@obnam.org; Wed, 19 Jul 2017 19:12:45 +0100 +Received: from exolobe3.liw.fi (82-181-57-241.bb.dnainternet.fi + [82.181.57.241]) by pieni.net (Postfix) with ESMTPSA id 03E7740668; + Wed, 19 Jul 2017 18:12:35 +0000 (UTC) +Received: from liw.fi (localhost [127.0.0.1]) + by exolobe3.liw.fi (Postfix) with ESMTPS id 495D1121DE2; + Wed, 19 Jul 2017 21:12:33 +0300 (EEST) +Date: Wed, 19 Jul 2017 21:12:32 +0300 +From: Lars Wirzenius <liw@liw.fi> +To: "Laurence Perkins (OE)" <lperkins@openeye.net> +Message-ID: <20170719181232.sdqihqdqldsgzmtd@liw.fi> +References: <1500484994.13826.5.camel@openeye.net> +MIME-Version: 1.0 +In-Reply-To: <1500484994.13826.5.camel@openeye.net> +User-Agent: NeoMutt/20170113 (1.7.2) +X-Pepperfish-Transaction: 09a8-a88e-6076-e1b8 +X-Spam-Score: -3.4 +X-Spam-Score-int: -33 +X-Spam-Bar: --- +X-Scanned-By: pepperfish.net, Wed, 19 Jul 2017 19:12:45 +0100 +X-Spam-Report: Content analysis details: (-3.4 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -0.5 PPF_USER_AGENT User-Agent: exists + -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam + tool) + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +X-ACL-Warn: message may be spam +X-Scan-Signature: ee5e95cd2050c0bee0b77e76c207dc88 +Cc: "obnam-support@obnam.org" <obnam-support@obnam.org> +Subject: Re: Variable Chunksize +X-BeenThere: obnam-support@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam backup software discussion <obnam-support-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org>, + <mailto:obnam-support-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org> +List-Post: <mailto:obnam-support@obnam.org> +List-Help: <mailto:obnam-support-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org>, + <mailto:obnam-support-request@obnam.org?subject=subscribe> +Content-Type: multipart/mixed; boundary="===============0954149426135821840==" +Mime-version: 1.0 +Sender: obnam-support-bounces@obnam.org +Errors-To: obnam-support-bounces@obnam.org + + +--===============0954149426135821840== +Content-Type: multipart/signed; micalg=pgp-sha512; + protocol="application/pgp-signature"; boundary="onupudhi7tahurcf" +Content-Disposition: inline + + +--onupudhi7tahurcf +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +Content-Transfer-Encoding: quoted-printable + +This has, in fact, been discussed before, and it's my intention to +look at implementing this once FORMAT GREEN ALBATROSS is finished. The +issue is that variable-sized chunking may increase the number of +chunks by a lot, and I'm not sure Obnam's data structures will deal +with that sufficiently efficiently. + +On Wed, Jul 19, 2017 at 05:23:16PM +0000, Laurence Perkins (OE) wrote: +> Stumbled across a couple of rsync/backup type programs that use +> variable-sized chunks to improve backup performance. It might be worth +> considering adding as an option. +>=20 +> The concept is actually relatively simple. Instead of splitting chunks +> at fixed sizes, chunks are split when the data matches some heuristic.=20 +> The common way to do it is to do a byte-by-byte hash of the stream and +> put in a chunk boundary wherever the hash meets some criteria. With a +> bit of statistics you can set what the average chunksize will be just +> by tweaking the criteria slightly. (So, read say the first 1KB, hash +> it, then drop the first byte from the data to hash and add 1KB+1B to +> the end of the data to hash and repeat. You walk over the data a byte +> at a time while still keeping a big enough amount of data going into +> the hash algorithm to produce interesting output. Split the chunk when +> the first X bytes of the hash are zero or whatever's convenient.) +>=20 +> Other than that, the rest of the chunking and indexing routines work +> the same way they do now. +>=20 +> The advantage is that adding a single byte to the start of a file +> doesn't cause the entire file to be re-uploaded. It will just increase +> the size of the first chunk by one byte and re-use the rest. (Or, if +> it's exactly in the chunk boundary, it will change the first two +> chunks. But still, that's a lot less data to re-upload.) It will also +> increase the amount of deduplication between nearly-identical files. +>=20 +> The downside is the overhead involved in walking a hash algorithm +> across the data, but it's not like you need a particularly CPU +> intensive hash, or even any significant collision resistance, you just +> need one that produces enough entropy to get your chunk sizes in the +> realm you want. +>=20 +> This feature would significantly increase Obnam's deduplication +> capabilities, without affecting the complexity of the datastore (It +> already supports multiple chunk sizes in the repo.) +>=20 +> I will take a look and see if I can figure out how to add this in, but +> I'm not a particularly tidy coder on the best of days, so someone with +> a bit more Python design experience than I have might be quicker at it. +> Like... A lot quicker... +>=20 +> LMP +> _______________________________________________ +> obnam-support mailing list +> obnam-support@obnam.org +> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-o= +bnam.org +>=20 + +--=20 +I want to build worthwhile things that might last. --joeyh + +--onupudhi7tahurcf +Content-Type: application/pgp-signature; name="signature.asc" + +-----BEGIN PGP SIGNATURE----- + +iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllvoQ8ACgkQbC+mFux6 +IDEBkRAAiiFJ9Sj7W4+bWfWNRYZF9dxi82/Ur5QRYDIa45YmpTQHfMf6WKLYTqkO +aW9VMwU+75ja0SyRrF0lO0JLyMSu50GFdnDcSood2onVYGucjqM/wpBvcxT12B2H +hZPGUA6PvduPmsTVXyX2PLXJpFTETzQwXVXkFLPsp0QW88AaxSIxA54ytMEEdzE+ +x9cymKORVXIp2R5w1tIhBJSNbz9vALG7ZMPubat3yV+lhVbNVU+/i6j5n2CcAWxw +4EzTqBdy2OxmkxG51Eg+Wv+2XOh7FV5zVodmvBw32AAhXOg4zGJ8q7jFU8d+xpFw +wi3WHXyROJUaPOvqbNJONIPJHeSYfCLOKxuBGYm4m5PkNKP7wnosVkQWPi2IHf9n +dXADHlZf/3nsCZUPjXGKkJfixQee1FxpdnjmZ0hjbzFQnMADDaFceq5K8DbTg8Ie +MIatkKtIPZ4jULeBb7o6OLv0cjHamEUryenB68Gwa+24P/ea9OIK9dDbRNBpPQ9a +dOawuGhdwgAHUWJFnisFG08kKrUOdpMmSMrYUZA0IaixocEZqMXPnACXGaEN09ks +Y+ycFRscJUvqtFuEBVLHvb9ybNRfWLRE36vTwswt+1Sdl05+58TVmTzlyKE9CKia +X3vzkGtulLizenaUrJHnIbCHcAQbBGg3AzSDaMTrUpVFkXDrrS0= +=a0G8 +-----END PGP SIGNATURE----- + +--onupudhi7tahurcf-- + + +--===============0954149426135821840== +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Disposition: inline + +_______________________________________________ +obnam-support mailing list +obnam-support@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org + +--===============0954149426135821840==-- + |