summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authordistix ticketing system <distix@pieni.net>2017-07-19 18:16:36 +0000
committerdistix ticketing system <distix@pieni.net>2017-07-19 18:16:36 +0000
commitf481f72c612fcfa79c8dec6f03899352b41ef8e5 (patch)
treef5ff9031814c20586b3ec611aff79f84cbc38734
parent1f1e64259f7217d4ebf29daf42c9db3ae5b21cb1 (diff)
downloadobnam-support-distix-f481f72c612fcfa79c8dec6f03899352b41ef8e5.tar.gz
imported mails
-rw-r--r--tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom174
1 files changed, 174 insertions, 0 deletions
diff --git a/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom b/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom
new file mode 100644
index 0000000..98fe8b1
--- /dev/null
+++ b/tickets/11c32688f6ae4c039e5fef65b5007a88/Maildir/new/1500488195.M732342P27244Q1.koom
@@ -0,0 +1,174 @@
+Return-Path: <obnam-support-bounces@obnam.org>
+X-Original-To: distix@pieni.net
+Delivered-To: distix@pieni.net
+Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221])
+ by pieni.net (Postfix) with ESMTPS id 8990D40668
+ for <distix@pieni.net>; Wed, 19 Jul 2017 18:12:46 +0000 (UTC)
+Received: from platypus.pepperfish.net (unknown [10.112.101.20])
+ by yaffle.pepperfish.net (Postfix) with ESMTP id 716454186A;
+ Wed, 19 Jul 2017 19:12:46 +0100 (BST)
+Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net)
+ by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
+ id 1dXtSo-0004Ki-Dm; Wed, 19 Jul 2017 19:12:46 +0100
+Received: from [10.112.101.21] (helo=mx3.pepperfish.net)
+ by platypus.pepperfish.net with esmtps (Exim 4.80 #2 (Debian))
+ id 1dXtSn-0004KT-HZ
+ for <obnam-support@obnam.org>; Wed, 19 Jul 2017 19:12:45 +0100
+Received: from koom.pieni.net ([88.99.190.206] helo=pieni.net)
+ by mx3.pepperfish.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
+ (Exim 4.89) (envelope-from <liw@liw.fi>) id 1dXtSj-0002rJ-KP
+ for obnam-support@obnam.org; Wed, 19 Jul 2017 19:12:45 +0100
+Received: from exolobe3.liw.fi (82-181-57-241.bb.dnainternet.fi
+ [82.181.57.241]) by pieni.net (Postfix) with ESMTPSA id 03E7740668;
+ Wed, 19 Jul 2017 18:12:35 +0000 (UTC)
+Received: from liw.fi (localhost [127.0.0.1])
+ by exolobe3.liw.fi (Postfix) with ESMTPS id 495D1121DE2;
+ Wed, 19 Jul 2017 21:12:33 +0300 (EEST)
+Date: Wed, 19 Jul 2017 21:12:32 +0300
+From: Lars Wirzenius <liw@liw.fi>
+To: "Laurence Perkins (OE)" <lperkins@openeye.net>
+Message-ID: <20170719181232.sdqihqdqldsgzmtd@liw.fi>
+References: <1500484994.13826.5.camel@openeye.net>
+MIME-Version: 1.0
+In-Reply-To: <1500484994.13826.5.camel@openeye.net>
+User-Agent: NeoMutt/20170113 (1.7.2)
+X-Pepperfish-Transaction: 09a8-a88e-6076-e1b8
+X-Spam-Score: -3.4
+X-Spam-Score-int: -33
+X-Spam-Bar: ---
+X-Scanned-By: pepperfish.net, Wed, 19 Jul 2017 19:12:45 +0100
+X-Spam-Report: Content analysis details: (-3.4 points)
+ pts rule name description
+ ---- ---------------------- --------------------------------------------------
+ -0.5 PPF_USER_AGENT User-Agent: exists
+ -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam
+ tool)
+ -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
+ [score: 0.0000]
+X-ACL-Warn: message may be spam
+X-Scan-Signature: ee5e95cd2050c0bee0b77e76c207dc88
+Cc: "obnam-support@obnam.org" <obnam-support@obnam.org>
+Subject: Re: Variable Chunksize
+X-BeenThere: obnam-support@obnam.org
+X-Mailman-Version: 2.1.5
+Precedence: list
+List-Id: Obnam backup software discussion <obnam-support-obnam.org>
+List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org>,
+ <mailto:obnam-support-request@obnam.org?subject=unsubscribe>
+List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org>
+List-Post: <mailto:obnam-support@obnam.org>
+List-Help: <mailto:obnam-support-request@obnam.org?subject=help>
+List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org>,
+ <mailto:obnam-support-request@obnam.org?subject=subscribe>
+Content-Type: multipart/mixed; boundary="===============0954149426135821840=="
+Mime-version: 1.0
+Sender: obnam-support-bounces@obnam.org
+Errors-To: obnam-support-bounces@obnam.org
+
+
+--===============0954149426135821840==
+Content-Type: multipart/signed; micalg=pgp-sha512;
+ protocol="application/pgp-signature"; boundary="onupudhi7tahurcf"
+Content-Disposition: inline
+
+
+--onupudhi7tahurcf
+Content-Type: text/plain; charset=us-ascii
+Content-Disposition: inline
+Content-Transfer-Encoding: quoted-printable
+
+This has, in fact, been discussed before, and it's my intention to
+look at implementing this once FORMAT GREEN ALBATROSS is finished. The
+issue is that variable-sized chunking may increase the number of
+chunks by a lot, and I'm not sure Obnam's data structures will deal
+with that sufficiently efficiently.
+
+On Wed, Jul 19, 2017 at 05:23:16PM +0000, Laurence Perkins (OE) wrote:
+> Stumbled across a couple of rsync/backup type programs that use
+> variable-sized chunks to improve backup performance. It might be worth
+> considering adding as an option.
+>=20
+> The concept is actually relatively simple. Instead of splitting chunks
+> at fixed sizes, chunks are split when the data matches some heuristic.=20
+> The common way to do it is to do a byte-by-byte hash of the stream and
+> put in a chunk boundary wherever the hash meets some criteria. With a
+> bit of statistics you can set what the average chunksize will be just
+> by tweaking the criteria slightly. (So, read say the first 1KB, hash
+> it, then drop the first byte from the data to hash and add 1KB+1B to
+> the end of the data to hash and repeat. You walk over the data a byte
+> at a time while still keeping a big enough amount of data going into
+> the hash algorithm to produce interesting output. Split the chunk when
+> the first X bytes of the hash are zero or whatever's convenient.)
+>=20
+> Other than that, the rest of the chunking and indexing routines work
+> the same way they do now.
+>=20
+> The advantage is that adding a single byte to the start of a file
+> doesn't cause the entire file to be re-uploaded. It will just increase
+> the size of the first chunk by one byte and re-use the rest. (Or, if
+> it's exactly in the chunk boundary, it will change the first two
+> chunks. But still, that's a lot less data to re-upload.) It will also
+> increase the amount of deduplication between nearly-identical files.
+>=20
+> The downside is the overhead involved in walking a hash algorithm
+> across the data, but it's not like you need a particularly CPU
+> intensive hash, or even any significant collision resistance, you just
+> need one that produces enough entropy to get your chunk sizes in the
+> realm you want.
+>=20
+> This feature would significantly increase Obnam's deduplication
+> capabilities, without affecting the complexity of the datastore (It
+> already supports multiple chunk sizes in the repo.)
+>=20
+> I will take a look and see if I can figure out how to add this in, but
+> I'm not a particularly tidy coder on the best of days, so someone with
+> a bit more Python design experience than I have might be quicker at it.
+> Like... A lot quicker...
+>=20
+> LMP
+> _______________________________________________
+> obnam-support mailing list
+> obnam-support@obnam.org
+> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-o=
+bnam.org
+>=20
+
+--=20
+I want to build worthwhile things that might last. --joeyh
+
+--onupudhi7tahurcf
+Content-Type: application/pgp-signature; name="signature.asc"
+
+-----BEGIN PGP SIGNATURE-----
+
+iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllvoQ8ACgkQbC+mFux6
+IDEBkRAAiiFJ9Sj7W4+bWfWNRYZF9dxi82/Ur5QRYDIa45YmpTQHfMf6WKLYTqkO
+aW9VMwU+75ja0SyRrF0lO0JLyMSu50GFdnDcSood2onVYGucjqM/wpBvcxT12B2H
+hZPGUA6PvduPmsTVXyX2PLXJpFTETzQwXVXkFLPsp0QW88AaxSIxA54ytMEEdzE+
+x9cymKORVXIp2R5w1tIhBJSNbz9vALG7ZMPubat3yV+lhVbNVU+/i6j5n2CcAWxw
+4EzTqBdy2OxmkxG51Eg+Wv+2XOh7FV5zVodmvBw32AAhXOg4zGJ8q7jFU8d+xpFw
+wi3WHXyROJUaPOvqbNJONIPJHeSYfCLOKxuBGYm4m5PkNKP7wnosVkQWPi2IHf9n
+dXADHlZf/3nsCZUPjXGKkJfixQee1FxpdnjmZ0hjbzFQnMADDaFceq5K8DbTg8Ie
+MIatkKtIPZ4jULeBb7o6OLv0cjHamEUryenB68Gwa+24P/ea9OIK9dDbRNBpPQ9a
+dOawuGhdwgAHUWJFnisFG08kKrUOdpMmSMrYUZA0IaixocEZqMXPnACXGaEN09ks
+Y+ycFRscJUvqtFuEBVLHvb9ybNRfWLRE36vTwswt+1Sdl05+58TVmTzlyKE9CKia
+X3vzkGtulLizenaUrJHnIbCHcAQbBGg3AzSDaMTrUpVFkXDrrS0=
+=a0G8
+-----END PGP SIGNATURE-----
+
+--onupudhi7tahurcf--
+
+
+--===============0954149426135821840==
+Content-Type: text/plain; charset="us-ascii"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 7bit
+Content-Disposition: inline
+
+_______________________________________________
+obnam-support mailing list
+obnam-support@obnam.org
+http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org
+
+--===============0954149426135821840==--
+