1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
|
Return-Path: <obnam-dev-bounces@obnam.org>
X-Original-To: distix@pieni.net
Delivered-To: distix@pieni.net
Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16])
by pieni.net (Postfix) with ESMTPS id 9B8224123D
for <distix@pieni.net>; Wed, 19 Apr 2017 11:18:49 +0000 (UTC)
Received: from platypus.pepperfish.net (unknown [10.112.100.20])
by bagpuss.pepperfish.net (Postfix) with ESMTP id 0A1F4BD1;
Wed, 19 Apr 2017 12:18:49 +0100 (BST)
Received: from ip6-localhost ([::1] helo=platypus.pepperfish.net)
by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
id 1d0ndI-0005xa-TX; Wed, 19 Apr 2017 12:18:48 +0100
Received: from inmail0 ([10.112.100.10] helo=mx0.pepperfish.net)
by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian))
id 1d0nMZ-0002SD-0O
for <obnam-dev@obnam.org>; Wed, 19 Apr 2017 12:01:31 +0100
Received: from mout.gmx.net ([212.227.17.20])
by mx0.pepperfish.net with esmtps (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128)
(Exim 4.80) (envelope-from <DJtm@gmx.net>) id 1d0n6A-0001N9-S7
for obnam-dev@obnam.org; Wed, 19 Apr 2017 11:44:36 +0100
Received: from [95.90.239.153] by 3capp-gmx-bs37.server.lan (via HTTP); Wed,
19 Apr 2017 12:44:21 +0200
MIME-Version: 1.0
Message-ID: <trinity-fc08f006-f2ea-48b7-bbe8-5be4e764c20a-1492598661322@3capp-gmx-bs37>
From: "Dionisus Torimens" <DJtm@gmx.net>
To: obnam-dev@obnam.org
Content-Type: multipart/mixed;
boundary=knika-9f351c14-7529-49fa-b1a7-f5d233f9a9fe
Date: Wed, 19 Apr 2017 12:44:21 +0200
Importance: normal
Sensitivity: Normal
X-Priority: 3
X-Provags-ID: V03:K1:ehlR11JKeHCQRT2ki6HdaAJ1aIO3+YsgbhfLrEYfWPK
L41RuxUNzzOlco9ZzfLsoh23FOdcGg/eBTXCwMMicaEGYNSPXJ
sFpRaq2dIyDFNSPVvNnorR6lzIdUQOx05IAdYJGELR2ZqxXnuc
Rubj5GO/JXtc6x/9GebDgq1upGYLn0+aY0rX5bOHOgpGfCfMtS
0NK2KOR3UrBuSlVHPuo6WLAsmj+CyUTqEpsu4ayJuRD/LeOGxt
D+Mt6x8aUFhIGhmnxafQ5J3tV/M2SlEc5XOc9Gz6QJnQ0+ojTy kH245U=
X-UI-Out-Filterresults: notjunk:1;V01:K0:5CAQ8FP+IRY=:ScZh0LWaKKUkljVWMQoRjQ
5sfoyPM9SyUkQeBUTqGRT9DVDj+8/xVKGiJ7kTir8JKf0+OO4N+7qPdKsYJyH6ABRtPJZ+Vgi
X5vfqFPBbgh4tQji+GgZmRRWdKxzxf43HKTBriBSuWnuVqAqe3i7pnkgnmpC9zHLJko2Nv4eX
qm/JtlJzpt1i12lO4S91cwu4q8AHC3rIpWWN6VU17SGokl3tl5R8g8wPIvaiaSW3HErJRIKgm
C0MYY1QV3nLwD/zNDnt79ETAzAxRFn8dctJQiCQtpiwDkE/jBVeGpbdWZ51hXZInAM2VNHioY
p8QHeLVQuZLwP9UKTEzP4eXJWFDh188jJafe2XlOVnJ8EnoNevLUsFc9Hv6lQDDu0j2hi3EuE
33grGMhNO0cdNT8uK+DaQLpIhunaqlpdF1MKFdzbZE568q89xLE9PrgTloQIYcRqbPmsLfibz
GUo2HkDf2w==
X-Pepperfish-Transaction: 9acf-e0a0-d921-5b6b
X-Spam-Score: 0.5
X-Spam-Score-int: 5
X-Spam-Bar: /
X-Scanned-By: pepperfish.net, Wed, 19 Apr 2017 11:44:36 +0100
X-Spam-Report: Content analysis details: (0.5 points)
pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust [212.227.17.20 listed in list.dnswl.org]
1.0 PPF_MSGID_DIGITS Message ID has @<digits> in it
0.4 PPF_XPRIORITY X-Priority header exists
0.5 PPF_MESSAGEID_NODOTS Message-Id contains no dots after the @
0.5 PPF_RECEIVED_HTTP Received header mentions http
1.2 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(djtm[at]gmx.net)
-0.0 SPF_PASS SPF: sender matches SPF record
-1.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.3 HTML_MESSAGE BODY: HTML included in message
0.2 PPF_NUMERIC_ENTITY RAW: Body contains numeric HTML entities
X-ACL-Warn: message may be spam
X-Scan-Signature: 805eda458b26030fb35277d5bf2b304c
Subject: [rfc] [PATCH] parallelize listdir
X-BeenThere: obnam-dev@obnam.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Obnam development discussions <obnam-dev-obnam.org>
List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>,
<mailto:obnam-dev-request@obnam.org?subject=unsubscribe>
List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org>
List-Post: <mailto:obnam-dev@obnam.org>
List-Help: <mailto:obnam-dev-request@obnam.org?subject=help>
List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>,
<mailto:obnam-dev-request@obnam.org?subject=subscribe>
Sender: obnam-dev-bounces@obnam.org
Errors-To: obnam-dev-bounces@obnam.org
--knika-9f351c14-7529-49fa-b1a7-f5d233f9a9fe
Content-Type: text/html; charset=UTF-8
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>Hi,</div>
<div> </div>
<div>an initial try to use a bit of multiprocessing. Makes obnam a bit faster in scannig. No change in the algorithms. I'd love to see some benchmarking, it feels a bit snappier. The pathos library is very helpful, but not in most distribution packages afaik.</div>
<div> </div>
<div>I've been thinking and I guess a good solution in the longer term would be to have</div>
<div>- a scanning process to create a list with changes,</div>
<div>- another to prepare the uploads and</div>
<div>- another to actually upload files to the repository.</div>
<div> </div>
<div>Cheers</div></div></body></html>
--knika-9f351c14-7529-49fa-b1a7-f5d233f9a9fe
Content-Type: text/x-patch
Content-Disposition: attachment; filename=parallel2.patch
>From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
From: djtm <djtm@users.noreply.github.com>
Date: Wed, 19 Apr 2017 12:28:23 +0200
Subject: [PATCH] *** SUBJECT HERE ***
*** BLURB HERE ***
djtm (1):
parallelize listdir
obnamlib/vfs_local.py | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
--
2.7.4
>From fe5333545cda1ac13fb10ae7b93c5f022f518f05 Mon Sep 17 00:00:00 2001
From: djtm <djtm@users.noreply.github.com>
Date: Tue, 18 Apr 2017 02:54:18 +0200
Subject: [PATCH] parallelize listdir
---
obnamlib/vfs_local.py | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/obnamlib/vfs_local.py b/obnamlib/vfs_local.py
index e01617e..697ef51 100644
--- a/obnamlib/vfs_local.py
+++ b/obnamlib/vfs_local.py
@@ -24,6 +24,9 @@ import tempfile
import time
import tracing
+from pathos.multiprocessing import ProcessingPool
+from functools import partial
+
import obnamlib
@@ -411,17 +414,19 @@ class LocalFS(obnamlib.VirtualFileSystem):
def listdir(self, dirname):
return os.listdir(self.join(dirname))
+ def listdir2helper(self, dirname, name):
+ try:
+ st = self.lstat(os.path.join(dirname, name))
+ except OSError, e: # pragma: no cover
+ st = e
+ ino = -1
+ else:
+ ino = st.st_ino
+ return (ino, name, st)
+
def listdir2(self, dirname):
- result = []
- for name in self.listdir(dirname):
- try:
- st = self.lstat(os.path.join(dirname, name))
- except OSError, e: # pragma: no cover
- st = e
- ino = -1
- else:
- ino = st.st_ino
- result.append((ino, name, st))
+ pool = ProcessingPool()
+ result = pool.map(partial(self.listdir2helper, dirname), self.listdir(dirname))
# We sort things in inode order, for speed when doing name lookups
# when backing up.
--
2.7.4
--knika-9f351c14-7529-49fa-b1a7-f5d233f9a9fe
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
obnam-dev mailing list
obnam-dev@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org
--knika-9f351c14-7529-49fa-b1a7-f5d233f9a9fe--
|
