Date: Wed, 19 Jul 2017 21:18:34 +0300
From: Lars Wirzenius <liw@liw.fi>
To: Dionisus Torimens <DJtm@gmx.net>
Message-ID: <20170719181834.pboi4gfnb3vshi3b@liw.fi>
References: <trinity-fc08f006-f2ea-48b7-bbe8-5be4e764c20a-1492598661322@3capp-gmx-bs37>
MIME-Version: 1.0
In-Reply-To: <trinity-fc08f006-f2ea-48b7-bbe8-5be4e764c20a-1492598661322@3capp-gmx-bs37>
User-Agent: NeoMutt/20170113 (1.7.2)
Cc: obnam-dev@obnam.org
Subject: Re: [rfc] [PATCH] parallelize listdir
Precedence: list
Content-Type: multipart/mixed; boundary="===============9175575107155843625=="
Mime-version: 1.0
Sender: obnam-dev-bounces@obnam.org
Errors-To: obnam-dev-bounces@obnam.org


--===============9175575107155843625==
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature"; boundary="zkikvhrkqmxob2zj"
Content-Disposition: inline


--zkikvhrkqmxob2zj
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Thanks for the suggestion.

After FORMAT GREEN ALBATROSS is ready for production, I'm happy to
start thining about other changes, particularly architectural ones. I
would like, for example, to allow the user to tell Obnam that a
particular file may have changed and have Obnam back up that file
only, and assume no other changes to the file system. This could then
be run by a program that hooks into the kernel to notice changes to
the filesystem. You change a file, Obnam backs it up about instantly.

But not now, the wide-winged bird is mocking me too much, I'm afraid.

On Wed, Apr 19, 2017 at 12:44:21PM +0200, Dionisus Torimens wrote:
> <html><head></head><body><div style=3D"font-family: Verdana;font-size: 12=
=2E0px;"><div>Hi,</div>
>=20
> <div>&nbsp;</div>
>=20
> <div>an initial try to use a bit of&nbsp;multiprocessing. Makes obnam a b=
it faster in scannig. No change in the algorithms. I&#39;d love to see some=
 benchmarking, it feels a bit snappier. The pathos library is very helpful,=
 but not in most distribution packages afaik.</div>
>=20
> <div>&nbsp;</div>
>=20
> <div>I&#39;ve been thinking and I guess a good solution in the longer ter=
m would be to have</div>
>=20
> <div>- a scanning process&nbsp;to create a list with changes,</div>
>=20
> <div>- another to prepare the uploads and</div>
>=20
> <div>- another to actually upload files to the repository.</div>
>=20
> <div>&nbsp;</div>
>=20
> <div>Cheers</div></div></body></html>
> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
> From: djtm <djtm@users.noreply.github.com>
> Date: Wed, 19 Apr 2017 12:28:23 +0200
> Subject: [PATCH] *** SUBJECT HERE ***
>=20
> *** BLURB HERE ***
>=20
> djtm (1):
>   parallelize listdir
>=20
>  obnamlib/vfs_local.py | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
>=20
> --=20
> 2.7.4
>=20
> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
> From: djtm <djtm@users.noreply.github.com>
> Date: Tue, 18 Apr 2017 02:54:18 +0200
> Subject: [PATCH] parallelize listdir
>=20
> ---
>  obnamlib/vfs_local.py | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
>=20
> diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py
> index e01617e..697ef51 100644
> --- a/obnamlib/vfs_local.py
> +++ b/obnamlib/vfs_local.py
> @@ -24,6 +24,9 @@ import tempfile
>  import time
>  import tracing
> =20
> +from pathos.multiprocessing import ProcessingPool
> +from functools import partial
> +
>  import obnamlib
> =20
> =20
> @@ -411,17 +414,19 @@ class LocalFS(obnamlib.VirtualFileSystem):
>      def listdir(self, dirname):
>          return os.listdir(self.join(dirname))
> =20
> +    def listdir2helper(self, dirname, name):
> +        try:
> +            st =3D self.lstat(os.path.join(dirname, name))
> +        except OSError, e:  # pragma: no cover
> +            st =3D e
> +            ino =3D -1
> +        else:
> +            ino =3D st.st_ino
> +        return (ino, name, st)
> +
>      def listdir2(self, dirname):
> -        result =3D []
> -        for name in self.listdir(dirname):
> -            try:
> -                st =3D self.lstat(os.path.join(dirname, name))
> -            except OSError, e:  # pragma: no cover
> -                st =3D e
> -                ino =3D -1
> -            else:
> -                ino =3D st.st_ino
> -            result.append((ino, name, st))
> +        pool =3D ProcessingPool()
> +        result =3D pool.map(partial(self.listdir2helper, dirname), self.=
listdir(dirname))
> =20
>          # We sort things in inode order, for speed when doing name looku=
ps
>          # when backing up.
> --=20
> 2.7.4
>=20

> _______________________________________________
> obnam-dev mailing list
> obnam-dev@obnam.org
> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam=
=2Eorg


--=20
I want to build worthwhile things that might last. --joeyh

--zkikvhrkqmxob2zj
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllvonkACgkQbC+mFux6
IDEqPQ/+N7ZCdOviAvqpYGR3sMGHSkd5ZO22sE4ysCOTFtO6+pDg2K1y29u8lMVQ
KOK9cNxzhglMMKKGcegq/u4DZaW2w0PG+tiZ+9xD60+PcshfsQxdKRKZVzc5Zgw/
SpIpho7kRHJmXNvUpR417BdXhb5BZv24HEezhXA2WVtKxOl4R1U5TrqhucVXKgVi
sE5QrPL9l4Oj20cGHIaxorEPWlx/RDqLFzsnty7v6NjXg7FzR/M2RQHwJiawgeMW
J6c3e92V4qvqUy4EGt5ekpzs/ianx0dQrjim0+lcXCtL5PuyfNXDNT8wzqpfAA0r
x33E72DyGg8Jpa34AEJ0M/H9tDkhgHIv4/pENF0uukba06RdEUN8u7dARJNiTqlp
T+zIIJOkal/fTMkMbQLy0Btq3gpfnnV576JLF6/9A6L5qxM/eKa+XmEBEtmxquJB
CchKdLs50ZCIue+K/4b+CmRqrpiz4s9465hqXGtjKklSdzhgVdNy7wMYFvv2N1Tg
Kow/W8Y1y/Bh+DBGnsEbPDXh6eeHI3h962eFJsAPxCfj/Yo3JMZ01Dgl4nr7/NLZ
nYCkERmskDo9DpF9OCZgBqd1y3KMibM5kCK6j3pVKnzH5MZ/O68H09f+l1HARM03
9cUgU2vWE1a/cBFdLaYtUXrdYF4Q+Ph/WhSVSOFEZs9vs7FaSW4=
=GZud
-----END PGP SIGNATURE-----

--zkikvhrkqmxob2zj--


--===============9175575107155843625==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
obnam-dev mailing list
obnam-dev@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org

--===============9175575107155843625==--