From d724406b066d1b790ac584d9c8bb3a3d13aaf7a6 Mon Sep 17 00:00:00 2001 From: distix ticketing system Date: Wed, 19 Jul 2017 18:21:15 +0000 Subject: imported mails --- .../Maildir/new/1500488475.M165390P28121Q1.koom | 240 +++++++++++++++++++++ .../6e2a44fcd630472b82cf898f26a7f936/ticket.yaml | 2 + 2 files changed, 242 insertions(+) create mode 100644 tickets/6e2a44fcd630472b82cf898f26a7f936/Maildir/new/1500488475.M165390P28121Q1.koom diff --git a/tickets/6e2a44fcd630472b82cf898f26a7f936/Maildir/new/1500488475.M165390P28121Q1.koom b/tickets/6e2a44fcd630472b82cf898f26a7f936/Maildir/new/1500488475.M165390P28121Q1.koom new file mode 100644 index 0000000..7a38577 --- /dev/null +++ b/tickets/6e2a44fcd630472b82cf898f26a7f936/Maildir/new/1500488475.M165390P28121Q1.koom @@ -0,0 +1,240 @@ +Return-Path: +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221]) + by pieni.net (Postfix) with ESMTPS id 39B3F415D4 + for ; Wed, 19 Jul 2017 18:18:54 +0000 (UTC) +Received: from platypus.pepperfish.net (unknown [10.112.101.20]) + by yaffle.pepperfish.net (Postfix) with ESMTP id 6B84741879; + Wed, 19 Jul 2017 19:18:46 +0100 (BST) +Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1dXtYc-0005NF-DX; Wed, 19 Jul 2017 19:18:46 +0100 +Received: from [10.112.101.21] (helo=mx3.pepperfish.net) + by platypus.pepperfish.net with esmtps (Exim 4.80 #2 (Debian)) + id 1dXtYa-0005Mm-HX + for ; Wed, 19 Jul 2017 19:18:44 +0100 +Received: from koom.pieni.net ([88.99.190.206] helo=pieni.net) + by mx3.pepperfish.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) + (Exim 4.89) (envelope-from ) id 1dXtYY-000300-Es + for obnam-dev@obnam.org; Wed, 19 Jul 2017 19:18:44 +0100 +Received: from exolobe3.liw.fi (82-181-57-241.bb.dnainternet.fi + [82.181.57.241]) by pieni.net (Postfix) with ESMTPSA id B2D52415D4; + Wed, 19 Jul 2017 18:18:36 +0000 (UTC) +Received: from liw.fi (localhost [127.0.0.1]) + by exolobe3.liw.fi (Postfix) with ESMTPS id 5A589121DE2; + Wed, 19 Jul 2017 21:18:35 +0300 (EEST) +Date: Wed, 19 Jul 2017 21:18:34 +0300 +From: Lars Wirzenius +To: Dionisus Torimens +Message-ID: <20170719181834.pboi4gfnb3vshi3b@liw.fi> +References: +MIME-Version: 1.0 +In-Reply-To: +User-Agent: NeoMutt/20170113 (1.7.2) +X-Pepperfish-Transaction: cc71-fbaa-98a7-c677 +X-Spam-Score: -2.8 +X-Spam-Score-int: -27 +X-Spam-Bar: -- +X-Scanned-By: pepperfish.net, Wed, 19 Jul 2017 19:18:44 +0100 +X-Spam-Report: Content analysis details: (-2.8 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + 0.2 PPF_REFERENCES_NODOTS References contains no dots after the @ + 0.2 PPF_INREPLYTO_NODOTS In-Reply-To contains no dots after the @ + -0.5 PPF_USER_AGENT User-Agent: exists + -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam + tool) + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] + 0.2 PPF_NUMERIC_ENTITY RAW: Body contains numeric HTML entities +X-ACL-Warn: message may be spam +X-Scan-Signature: e056c7610997a266061b7a558bad18c3 +Cc: obnam-dev@obnam.org +Subject: Re: [rfc] [PATCH] parallelize listdir +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +Content-Type: multipart/mixed; boundary="===============9175575107155843625==" +Mime-version: 1.0 +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + + +--===============9175575107155843625== +Content-Type: multipart/signed; micalg=pgp-sha512; + protocol="application/pgp-signature"; boundary="zkikvhrkqmxob2zj" +Content-Disposition: inline + + +--zkikvhrkqmxob2zj +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +Content-Transfer-Encoding: quoted-printable + +Thanks for the suggestion. + +After FORMAT GREEN ALBATROSS is ready for production, I'm happy to +start thining about other changes, particularly architectural ones. I +would like, for example, to allow the user to tell Obnam that a +particular file may have changed and have Obnam back up that file +only, and assume no other changes to the file system. This could then +be run by a program that hooks into the kernel to notice changes to +the filesystem. You change a file, Obnam backs it up about instantly. + +But not now, the wide-winged bird is mocking me too much, I'm afraid. + +On Wed, Apr 19, 2017 at 12:44:21PM +0200, Dionisus Torimens wrote: +>
Hi,
+>=20 +>
 
+>=20 +>
an initial try to use a bit of multiprocessing. Makes obnam a b= +it faster in scannig. No change in the algorithms. I'd love to see some= + benchmarking, it feels a bit snappier. The pathos library is very helpful,= + but not in most distribution packages afaik.
+>=20 +>
 
+>=20 +>
I've been thinking and I guess a good solution in the longer ter= +m would be to have
+>=20 +>
- a scanning process to create a list with changes,
+>=20 +>
- another to prepare the uploads and
+>=20 +>
- another to actually upload files to the repository.
+>=20 +>
 
+>=20 +>
Cheers
+> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001 +> From: djtm +> Date: Wed, 19 Apr 2017 12:28:23 +0200 +> Subject: [PATCH] *** SUBJECT HERE *** +>=20 +> *** BLURB HERE *** +>=20 +> djtm (1): +> parallelize listdir +>=20 +> obnamlib/vfs_local.py | 25 +++++++++++++++---------- +> 1 file changed, 15 insertions(+), 10 deletions(-) +>=20 +> --=20 +> 2.7.4 +>=20 +> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001 +> From: djtm +> Date: Tue, 18 Apr 2017 02:54:18 +0200 +> Subject: [PATCH] parallelize listdir +>=20 +> --- +> obnamlib/vfs_local.py | 25 +++++++++++++++---------- +> 1 file changed, 15 insertions(+), 10 deletions(-) +>=20 +> diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py +> index e01617e..697ef51 100644 +> --- a/obnamlib/vfs_local.py +> +++ b/obnamlib/vfs_local.py +> @@ -24,6 +24,9 @@ import tempfile +> import time +> import tracing +> =20 +> +from pathos.multiprocessing import ProcessingPool +> +from functools import partial +> + +> import obnamlib +> =20 +> =20 +> @@ -411,17 +414,19 @@ class LocalFS(obnamlib.VirtualFileSystem): +> def listdir(self, dirname): +> return os.listdir(self.join(dirname)) +> =20 +> + def listdir2helper(self, dirname, name): +> + try: +> + st =3D self.lstat(os.path.join(dirname, name)) +> + except OSError, e: # pragma: no cover +> + st =3D e +> + ino =3D -1 +> + else: +> + ino =3D st.st_ino +> + return (ino, name, st) +> + +> def listdir2(self, dirname): +> - result =3D [] +> - for name in self.listdir(dirname): +> - try: +> - st =3D self.lstat(os.path.join(dirname, name)) +> - except OSError, e: # pragma: no cover +> - st =3D e +> - ino =3D -1 +> - else: +> - ino =3D st.st_ino +> - result.append((ino, name, st)) +> + pool =3D ProcessingPool() +> + result =3D pool.map(partial(self.listdir2helper, dirname), self.= +listdir(dirname)) +> =20 +> # We sort things in inode order, for speed when doing name looku= +ps +> # when backing up. +> --=20 +> 2.7.4 +>=20 + +> _______________________________________________ +> obnam-dev mailing list +> obnam-dev@obnam.org +> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam= +=2Eorg + + +--=20 +I want to build worthwhile things that might last. --joeyh + +--zkikvhrkqmxob2zj +Content-Type: application/pgp-signature; name="signature.asc" + +-----BEGIN PGP SIGNATURE----- + +iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllvonkACgkQbC+mFux6 +IDEqPQ/+N7ZCdOviAvqpYGR3sMGHSkd5ZO22sE4ysCOTFtO6+pDg2K1y29u8lMVQ +KOK9cNxzhglMMKKGcegq/u4DZaW2w0PG+tiZ+9xD60+PcshfsQxdKRKZVzc5Zgw/ +SpIpho7kRHJmXNvUpR417BdXhb5BZv24HEezhXA2WVtKxOl4R1U5TrqhucVXKgVi +sE5QrPL9l4Oj20cGHIaxorEPWlx/RDqLFzsnty7v6NjXg7FzR/M2RQHwJiawgeMW +J6c3e92V4qvqUy4EGt5ekpzs/ianx0dQrjim0+lcXCtL5PuyfNXDNT8wzqpfAA0r +x33E72DyGg8Jpa34AEJ0M/H9tDkhgHIv4/pENF0uukba06RdEUN8u7dARJNiTqlp +T+zIIJOkal/fTMkMbQLy0Btq3gpfnnV576JLF6/9A6L5qxM/eKa+XmEBEtmxquJB +CchKdLs50ZCIue+K/4b+CmRqrpiz4s9465hqXGtjKklSdzhgVdNy7wMYFvv2N1Tg +Kow/W8Y1y/Bh+DBGnsEbPDXh6eeHI3h962eFJsAPxCfj/Yo3JMZ01Dgl4nr7/NLZ +nYCkERmskDo9DpF9OCZgBqd1y3KMibM5kCK6j3pVKnzH5MZ/O68H09f+l1HARM03 +9cUgU2vWE1a/cBFdLaYtUXrdYF4Q+Ph/WhSVSOFEZs9vs7FaSW4= +=GZud +-----END PGP SIGNATURE----- + +--zkikvhrkqmxob2zj-- + + +--===============9175575107155843625== +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Disposition: inline + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org + +--===============9175575107155843625==-- + diff --git a/tickets/6e2a44fcd630472b82cf898f26a7f936/ticket.yaml b/tickets/6e2a44fcd630472b82cf898f26a7f936/ticket.yaml index 29c4424..9268707 100644 --- a/tickets/6e2a44fcd630472b82cf898f26a7f936/ticket.yaml +++ b/tickets/6e2a44fcd630472b82cf898f26a7f936/ticket.yaml @@ -1,3 +1,5 @@ +status: +- '' ticket-id: - 6e2a44fcd630472b82cf898f26a7f936 title: -- cgit v1.2.1