1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
|
Return-Path: <obnam-dev-bounces@obnam.org>
X-Original-To: distix@pieni.net
Delivered-To: distix@pieni.net
Received: from yaffle.pepperfish.net (yaffle.pepperfish.net [88.99.213.221])
by pieni.net (Postfix) with ESMTPS id 39B3F415D4
for <distix@pieni.net>; Wed, 19 Jul 2017 18:18:54 +0000 (UTC)
Received: from platypus.pepperfish.net (unknown [10.112.101.20])
by yaffle.pepperfish.net (Postfix) with ESMTP id 6B84741879;
Wed, 19 Jul 2017 19:18:46 +0100 (BST)
Received: from ip6-localhost.nat ([::1] helo=platypus.pepperfish.net)
by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
id 1dXtYc-0005NF-DX; Wed, 19 Jul 2017 19:18:46 +0100
Received: from [10.112.101.21] (helo=mx3.pepperfish.net)
by platypus.pepperfish.net with esmtps (Exim 4.80 #2 (Debian))
id 1dXtYa-0005Mm-HX
for <obnam-dev@obnam.org>; Wed, 19 Jul 2017 19:18:44 +0100
Received: from koom.pieni.net ([88.99.190.206] helo=pieni.net)
by mx3.pepperfish.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89) (envelope-from <liw@liw.fi>) id 1dXtYY-000300-Es
for obnam-dev@obnam.org; Wed, 19 Jul 2017 19:18:44 +0100
Received: from exolobe3.liw.fi (82-181-57-241.bb.dnainternet.fi
[82.181.57.241]) by pieni.net (Postfix) with ESMTPSA id B2D52415D4;
Wed, 19 Jul 2017 18:18:36 +0000 (UTC)
Received: from liw.fi (localhost [127.0.0.1])
by exolobe3.liw.fi (Postfix) with ESMTPS id 5A589121DE2;
Wed, 19 Jul 2017 21:18:35 +0300 (EEST)
Date: Wed, 19 Jul 2017 21:18:34 +0300
From: Lars Wirzenius <liw@liw.fi>
To: Dionisus Torimens <DJtm@gmx.net>
Message-ID: <20170719181834.pboi4gfnb3vshi3b@liw.fi>
References: <trinity-fc08f006-f2ea-48b7-bbe8-5be4e764c20a-1492598661322@3capp-gmx-bs37>
MIME-Version: 1.0
In-Reply-To: <trinity-fc08f006-f2ea-48b7-bbe8-5be4e764c20a-1492598661322@3capp-gmx-bs37>
User-Agent: NeoMutt/20170113 (1.7.2)
X-Pepperfish-Transaction: cc71-fbaa-98a7-c677
X-Spam-Score: -2.8
X-Spam-Score-int: -27
X-Spam-Bar: --
X-Scanned-By: pepperfish.net, Wed, 19 Jul 2017 19:18:44 +0100
X-Spam-Report: Content analysis details: (-2.8 points)
pts rule name description
---- ---------------------- --------------------------------------------------
0.2 PPF_REFERENCES_NODOTS References contains no dots after the @
0.2 PPF_INREPLYTO_NODOTS In-Reply-To contains no dots after the @
-0.5 PPF_USER_AGENT User-Agent: exists
-1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam
tool)
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.2 PPF_NUMERIC_ENTITY RAW: Body contains numeric HTML entities
X-ACL-Warn: message may be spam
X-Scan-Signature: e056c7610997a266061b7a558bad18c3
Cc: obnam-dev@obnam.org
Subject: Re: [rfc] [PATCH] parallelize listdir
X-BeenThere: obnam-dev@obnam.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Obnam development discussions <obnam-dev-obnam.org>
List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>,
<mailto:obnam-dev-request@obnam.org?subject=unsubscribe>
List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org>
List-Post: <mailto:obnam-dev@obnam.org>
List-Help: <mailto:obnam-dev-request@obnam.org?subject=help>
List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>,
<mailto:obnam-dev-request@obnam.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============9175575107155843625=="
Mime-version: 1.0
Sender: obnam-dev-bounces@obnam.org
Errors-To: obnam-dev-bounces@obnam.org
--===============9175575107155843625==
Content-Type: multipart/signed; micalg=pgp-sha512;
protocol="application/pgp-signature"; boundary="zkikvhrkqmxob2zj"
Content-Disposition: inline
--zkikvhrkqmxob2zj
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Thanks for the suggestion.
After FORMAT GREEN ALBATROSS is ready for production, I'm happy to
start thining about other changes, particularly architectural ones. I
would like, for example, to allow the user to tell Obnam that a
particular file may have changed and have Obnam back up that file
only, and assume no other changes to the file system. This could then
be run by a program that hooks into the kernel to notice changes to
the filesystem. You change a file, Obnam backs it up about instantly.
But not now, the wide-winged bird is mocking me too much, I'm afraid.
On Wed, Apr 19, 2017 at 12:44:21PM +0200, Dionisus Torimens wrote:
> <html><head></head><body><div style=3D"font-family: Verdana;font-size: 12=
=2E0px;"><div>Hi,</div>
>=20
> <div> </div>
>=20
> <div>an initial try to use a bit of multiprocessing. Makes obnam a b=
it faster in scannig. No change in the algorithms. I'd love to see some=
benchmarking, it feels a bit snappier. The pathos library is very helpful,=
but not in most distribution packages afaik.</div>
>=20
> <div> </div>
>=20
> <div>I've been thinking and I guess a good solution in the longer ter=
m would be to have</div>
>=20
> <div>- a scanning process to create a list with changes,</div>
>=20
> <div>- another to prepare the uploads and</div>
>=20
> <div>- another to actually upload files to the repository.</div>
>=20
> <div> </div>
>=20
> <div>Cheers</div></div></body></html>
> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
> From: djtm <djtm@users.noreply.github.com>
> Date: Wed, 19 Apr 2017 12:28:23 +0200
> Subject: [PATCH] *** SUBJECT HERE ***
>=20
> *** BLURB HERE ***
>=20
> djtm (1):
> parallelize listdir
>=20
> obnamlib/vfs_local.py | 25 +++++++++++++++----------
> 1 file changed, 15 insertions(+), 10 deletions(-)
>=20
> --=20
> 2.7.4
>=20
> >From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
> From: djtm <djtm@users.noreply.github.com>
> Date: Tue, 18 Apr 2017 02:54:18 +0200
> Subject: [PATCH] parallelize listdir
>=20
> ---
> obnamlib/vfs_local.py | 25 +++++++++++++++----------
> 1 file changed, 15 insertions(+), 10 deletions(-)
>=20
> diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py
> index e01617e..697ef51 100644
> --- a/obnamlib/vfs_local.py
> +++ b/obnamlib/vfs_local.py
> @@ -24,6 +24,9 @@ import tempfile
> import time
> import tracing
> =20
> +from pathos.multiprocessing import ProcessingPool
> +from functools import partial
> +
> import obnamlib
> =20
> =20
> @@ -411,17 +414,19 @@ class LocalFS(obnamlib.VirtualFileSystem):
> def listdir(self, dirname):
> return os.listdir(self.join(dirname))
> =20
> + def listdir2helper(self, dirname, name):
> + try:
> + st =3D self.lstat(os.path.join(dirname, name))
> + except OSError, e: # pragma: no cover
> + st =3D e
> + ino =3D -1
> + else:
> + ino =3D st.st_ino
> + return (ino, name, st)
> +
> def listdir2(self, dirname):
> - result =3D []
> - for name in self.listdir(dirname):
> - try:
> - st =3D self.lstat(os.path.join(dirname, name))
> - except OSError, e: # pragma: no cover
> - st =3D e
> - ino =3D -1
> - else:
> - ino =3D st.st_ino
> - result.append((ino, name, st))
> + pool =3D ProcessingPool()
> + result =3D pool.map(partial(self.listdir2helper, dirname), self.=
listdir(dirname))
> =20
> # We sort things in inode order, for speed when doing name looku=
ps
> # when backing up.
> --=20
> 2.7.4
>=20
> _______________________________________________
> obnam-dev mailing list
> obnam-dev@obnam.org
> http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam=
=2Eorg
--=20
I want to build worthwhile things that might last. --joeyh
--zkikvhrkqmxob2zj
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEETNTnrewG6wEE1EJ3bC+mFux6IDEFAllvonkACgkQbC+mFux6
IDEqPQ/+N7ZCdOviAvqpYGR3sMGHSkd5ZO22sE4ysCOTFtO6+pDg2K1y29u8lMVQ
KOK9cNxzhglMMKKGcegq/u4DZaW2w0PG+tiZ+9xD60+PcshfsQxdKRKZVzc5Zgw/
SpIpho7kRHJmXNvUpR417BdXhb5BZv24HEezhXA2WVtKxOl4R1U5TrqhucVXKgVi
sE5QrPL9l4Oj20cGHIaxorEPWlx/RDqLFzsnty7v6NjXg7FzR/M2RQHwJiawgeMW
J6c3e92V4qvqUy4EGt5ekpzs/ianx0dQrjim0+lcXCtL5PuyfNXDNT8wzqpfAA0r
x33E72DyGg8Jpa34AEJ0M/H9tDkhgHIv4/pENF0uukba06RdEUN8u7dARJNiTqlp
T+zIIJOkal/fTMkMbQLy0Btq3gpfnnV576JLF6/9A6L5qxM/eKa+XmEBEtmxquJB
CchKdLs50ZCIue+K/4b+CmRqrpiz4s9465hqXGtjKklSdzhgVdNy7wMYFvv2N1Tg
Kow/W8Y1y/Bh+DBGnsEbPDXh6eeHI3h962eFJsAPxCfj/Yo3JMZ01Dgl4nr7/NLZ
nYCkERmskDo9DpF9OCZgBqd1y3KMibM5kCK6j3pVKnzH5MZ/O68H09f+l1HARM03
9cUgU2vWE1a/cBFdLaYtUXrdYF4Q+Ph/WhSVSOFEZs9vs7FaSW4=
=GZud
-----END PGP SIGNATURE-----
--zkikvhrkqmxob2zj--
--===============9175575107155843625==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
obnam-dev mailing list
obnam-dev@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
--===============9175575107155843625==--
|
