diff options
| author | Lars Wirzenius <liw@liw.fi> | 2016-02-20 22:10:13 +0200 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2016-02-20 22:10:13 +0200 |
| commit | 8c087e41fdda5d9508b9c95ea57fbd079142633d (patch) | |
| tree | 44bbbcec3aac27d6ca26d498ac1b4f77ae15c190 /tickets/10d72738d6ce4ab7b0fc3f21c15635bc | |
| parent | ea12bbd09cf9e03d4b50f139f55925aad10dcec0 (diff) | |
| download | obnam-dev-distix-8c087e41fdda5d9508b9c95ea57fbd079142633d.tar.gz | |
imported mails
Diffstat (limited to 'tickets/10d72738d6ce4ab7b0fc3f21c15635bc')
11 files changed, 1547 insertions, 0 deletions
diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/cur/.this-dir-not-empty/.empty/empty-file b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/cur/.this-dir-not-empty/.empty/empty-file new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/cur/.this-dir-not-empty/.empty/empty-file diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/.this-dir-not-empty/.empty/empty-file b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/.this-dir-not-empty/.empty/empty-file new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/.this-dir-not-empty/.empty/empty-file diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M673089P17339Q108.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M673089P17339Q108.exolobe1 new file mode 100644 index 0000000..e020600 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M673089P17339Q108.exolobe1 @@ -0,0 +1,123 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id EDAFA2B4B0 + for <distix@pieni.net>; Sun, 7 Dec 2014 15:34:59 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id 54DE6CE2; + Sun, 7 Dec 2014 14:34:59 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxcvL-00030z-7n; Sun, 07 Dec 2014 14:34:59 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxcvJ-00030t-OW + for <obnam-dev@obnam.org>; Sun, 07 Dec 2014 14:34:57 +0000 +Received: from xvm-166-37.ghst.net + ([95.142.166.37] helo=pieni.net ident=postfix) + by mx0.pepperfish.net with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) + (Exim 4.80) (envelope-from <liw@liw.fi>) id 1XxcvG-0003cH-Sy + for obnam-dev@obnam.org; Sun, 07 Dec 2014 14:34:57 +0000 +Received: from exolobe1.liw.fi (82-181-8-107.bb.dnainternet.fi [82.181.8.107]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPSA id D5EA92B4B0; + Sun, 7 Dec 2014 15:34:51 +0100 (CET) +Received: from exolobe1.liw.fi (localhost [127.0.0.1]) + by exolobe1.liw.fi (Postfix) with ESMTPS id 2B14F45CD4; + Sun, 7 Dec 2014 16:34:51 +0200 (EET) +Date: Sun, 7 Dec 2014 16:34:49 +0200 +From: Lars Wirzenius <liw@liw.fi> +To: Matthew Dawson <matthew@mjdsystems.ca> +Message-ID: <20141207143449.GH17184@exolobe1.liw.fi> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> + <20141206192247.GK6438@exolobe1.liw.fi> + <1531092.aY1fI7Af0k@cwmtaff> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <1531092.aY1fI7Af0k@cwmtaff> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Spam-Score: -3.0 +X-Spam-Score-int: -29 +X-Spam-Bar: --- +X-Scanned-By: pepperfish.net, Sun, 07 Dec 2014 14:34:57 +0000 +X-Spam-Report: Content analysis details: (-3.0 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam + tool) -0.5 PPF_USER_AGENT User-Agent: exists + 0.2 PPF_INREPLYTO_NODOTS In-Reply-To contains no dots after the @ + 0.2 PPF_REFERENCES_NODOTS References contains no dots after the @ + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: obnam-dev@obnam.org +Subject: Re: [PATCH 0/3] RFC: Add support for Openstack Swift as backup + repository +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +On Sat, Dec 06, 2014 at 05:54:39PM -0500, Matthew Dawson wrote: +> Sounds good to me. I'll work on cleaning everything up. Regarding +> the test cases, I'm not sure what you are looking for. As the Swift +> support only plugs into the VFS layer, there isn't really anything +> sitting between the two. I do have a set of tests like the SFTP +> does, so that the regular test suite passes. + +I'd like to have: + +* Some tests that at least ensure syntactic correctness of the Python, + and getting it nitpicked by the nitpicky tests (line lengths, + copyright statements, etc) when the normal test suite ("./check") is + run, even if it doesn't test at all that the Swift plugin works. + Having the plugin in the source tree so it gets loaded when ./obnam + starts is probably sufficient. + +* Something to run separately to test the Swift plugin against an + existing Swift instance, including documentation of how to get + access to one and how to set things up so that the tests can be run. + The test-swift script is probably doing this already. + +I suspect you're pretty much there, already, actually. :) + +> If you want, I can try to make a mock instance of Swift, but I'd bet +> I'd create more bugs in the mock than the actual code. Alternately, +> if you just want a free place to try the code against, hubic.com +> provides 25G free storage using swift. + +Unless there's a general mockup of Swift, which others are using +anyway, I don't think this is worthwhile. + +> > Going forward, will you be available to maintain this code? + +> As long as I'm using Obnam/this code in my backup strategy, yes I'm +> happy to do so. Should that change in the future, I can try to +> continue to help out. For the foreseeable future, that shouldn't be +> a problem. + +Cool. That's all I can ask for, really. + +Thank you! + +-- +http://gtdfh.branchable.com/ -- GTD for hackers +http://obnam.org/ -- HAVE YOU BACKED UP TODAY? + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M699945P17339Q109.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M699945P17339Q109.exolobe1 new file mode 100644 index 0000000..e020600 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M699945P17339Q109.exolobe1 @@ -0,0 +1,123 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id EDAFA2B4B0 + for <distix@pieni.net>; Sun, 7 Dec 2014 15:34:59 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id 54DE6CE2; + Sun, 7 Dec 2014 14:34:59 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxcvL-00030z-7n; Sun, 07 Dec 2014 14:34:59 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxcvJ-00030t-OW + for <obnam-dev@obnam.org>; Sun, 07 Dec 2014 14:34:57 +0000 +Received: from xvm-166-37.ghst.net + ([95.142.166.37] helo=pieni.net ident=postfix) + by mx0.pepperfish.net with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) + (Exim 4.80) (envelope-from <liw@liw.fi>) id 1XxcvG-0003cH-Sy + for obnam-dev@obnam.org; Sun, 07 Dec 2014 14:34:57 +0000 +Received: from exolobe1.liw.fi (82-181-8-107.bb.dnainternet.fi [82.181.8.107]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPSA id D5EA92B4B0; + Sun, 7 Dec 2014 15:34:51 +0100 (CET) +Received: from exolobe1.liw.fi (localhost [127.0.0.1]) + by exolobe1.liw.fi (Postfix) with ESMTPS id 2B14F45CD4; + Sun, 7 Dec 2014 16:34:51 +0200 (EET) +Date: Sun, 7 Dec 2014 16:34:49 +0200 +From: Lars Wirzenius <liw@liw.fi> +To: Matthew Dawson <matthew@mjdsystems.ca> +Message-ID: <20141207143449.GH17184@exolobe1.liw.fi> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> + <20141206192247.GK6438@exolobe1.liw.fi> + <1531092.aY1fI7Af0k@cwmtaff> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <1531092.aY1fI7Af0k@cwmtaff> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Spam-Score: -3.0 +X-Spam-Score-int: -29 +X-Spam-Bar: --- +X-Scanned-By: pepperfish.net, Sun, 07 Dec 2014 14:34:57 +0000 +X-Spam-Report: Content analysis details: (-3.0 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam + tool) -0.5 PPF_USER_AGENT User-Agent: exists + 0.2 PPF_INREPLYTO_NODOTS In-Reply-To contains no dots after the @ + 0.2 PPF_REFERENCES_NODOTS References contains no dots after the @ + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: obnam-dev@obnam.org +Subject: Re: [PATCH 0/3] RFC: Add support for Openstack Swift as backup + repository +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +On Sat, Dec 06, 2014 at 05:54:39PM -0500, Matthew Dawson wrote: +> Sounds good to me. I'll work on cleaning everything up. Regarding +> the test cases, I'm not sure what you are looking for. As the Swift +> support only plugs into the VFS layer, there isn't really anything +> sitting between the two. I do have a set of tests like the SFTP +> does, so that the regular test suite passes. + +I'd like to have: + +* Some tests that at least ensure syntactic correctness of the Python, + and getting it nitpicked by the nitpicky tests (line lengths, + copyright statements, etc) when the normal test suite ("./check") is + run, even if it doesn't test at all that the Swift plugin works. + Having the plugin in the source tree so it gets loaded when ./obnam + starts is probably sufficient. + +* Something to run separately to test the Swift plugin against an + existing Swift instance, including documentation of how to get + access to one and how to set things up so that the tests can be run. + The test-swift script is probably doing this already. + +I suspect you're pretty much there, already, actually. :) + +> If you want, I can try to make a mock instance of Swift, but I'd bet +> I'd create more bugs in the mock than the actual code. Alternately, +> if you just want a free place to try the code against, hubic.com +> provides 25G free storage using swift. + +Unless there's a general mockup of Swift, which others are using +anyway, I don't think this is worthwhile. + +> > Going forward, will you be available to maintain this code? + +> As long as I'm using Obnam/this code in my backup strategy, yes I'm +> happy to do so. Should that change in the future, I can try to +> continue to help out. For the foreseeable future, that shouldn't be +> a problem. + +Cool. That's all I can ask for, really. + +Thank you! + +-- +http://gtdfh.branchable.com/ -- GTD for hackers +http://obnam.org/ -- HAVE YOU BACKED UP TODAY? + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M711251P17339Q110.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M711251P17339Q110.exolobe1 new file mode 100644 index 0000000..6a39e2d --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999000.M711251P17339Q110.exolobe1 @@ -0,0 +1,114 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id 543562AB46 + for <distix@pieni.net>; Sat, 6 Dec 2014 20:22:57 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id CBC5129FB; + Sat, 6 Dec 2014 19:22:56 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxKwS-0004rz-Nd; Sat, 06 Dec 2014 19:22:56 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxKwS-0004rt-4Z + for <obnam-dev@obnam.org>; Sat, 06 Dec 2014 19:22:56 +0000 +Received: from xvm-166-37.ghst.net + ([95.142.166.37] helo=pieni.net ident=postfix) + by mx0.pepperfish.net with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) + (Exim 4.80) (envelope-from <liw@liw.fi>) id 1XxKwL-000508-UX + for obnam-dev@obnam.org; Sat, 06 Dec 2014 19:22:56 +0000 +Received: from exolobe1.liw.fi (82-181-8-107.bb.dnainternet.fi [82.181.8.107]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPSA id D395E2AB46; + Sat, 6 Dec 2014 20:22:48 +0100 (CET) +Received: from exolobe1.liw.fi (localhost [127.0.0.1]) + by exolobe1.liw.fi (Postfix) with ESMTPS id 3D7A242AC7; + Sat, 6 Dec 2014 21:22:48 +0200 (EET) +Date: Sat, 6 Dec 2014 21:22:47 +0200 +From: Lars Wirzenius <liw@liw.fi> +To: Matthew Dawson <matthew@mjdsystems.ca> +Message-ID: <20141206192247.GK6438@exolobe1.liw.fi> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Spam-Score: -3.4 +X-Spam-Score-int: -33 +X-Spam-Bar: --- +X-Scanned-By: pepperfish.net, Sat, 06 Dec 2014 19:22:56 +0000 +X-Spam-Report: Content analysis details: (-3.4 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -1.0 PPF_USER_AGENT_MUTT User-Agent: contains Mutt (Mutt isn't a spam + tool) -0.5 PPF_USER_AGENT User-Agent: exists + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: obnam-dev@obnam.org +Subject: Re: [PATCH 0/3] RFC: Add support for Openstack Swift as backup + repository +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +Hi, Matthew, + +Thanks for the preliminary patches. As you said they're not ready for +merging yet, I haven't tried that, but I've had a glimpse at the code. + +On Sun, Nov 30, 2014 at 09:08:29PM -0500, Matthew Dawson wrote: +> This patch set contains an experimental work in progress Openstack Swift VFS for +> Obnam. I've only just got it working (thus it's not merge ready) but it does +> pass the VFS unit tests except for reinit support. I've done initial testing +> against hubiC's implementation, and the code has managed an initial backup + +> verify on some (real) test data. +> +> I'm posting it here for initial comments, and also to find out if such a backend +> is wanted in Obnam. If this is wanted, I'll continue improving the +> backend. + +I would welcome such a backend. While I haven't used such a thing +myself (also not an Amazon S3 instance), it's good for Obnam to +support a variety of storage options. + +Since I don't know much about Swift, I can't comment on whether the +plugin you're working on does sensible things. I'm going to trust you +on that. + +I noticed the Python code isn't formatted quite like PEP8 specifies, +and I'd prefer to see that fixed. Most importantly, source code lines +(after TAB expansion with a width of 8) should be strictly less than +80 columns. (It's something that's a bit controversion in the modern +world, but it's a rule I impose on myself and the Obnam code base.) + +I'm happy to include this in the Obnam code base, assuming clean code +and suitable tests, including some that work without access to a real +Swift instance (obviously the code doesn't then need to do much, but +it shouldn't crash). Some end-user documentation would also be nice. + +Going forward, will you be available to maintain this code? + +-- +http://gtdfh.branchable.com/ -- GTD for hackers +http://obnam.org/ -- HAVE YOU BACKED UP TODAY? + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M207056P17339Q191.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M207056P17339Q191.exolobe1 new file mode 100644 index 0000000..d531a77 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M207056P17339Q191.exolobe1 @@ -0,0 +1,205 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id 33F5921C1D + for <distix@pieni.net>; Mon, 1 Dec 2014 03:08:47 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id CD1203526; + Mon, 1 Dec 2014 02:08:46 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPu-0002tj-LY; Mon, 01 Dec 2014 02:08:46 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPs-0002rY-09 + for <obnam-dev@obnam.org>; Mon, 01 Dec 2014 02:08:44 +0000 +Received: from scadrial.mjdsystems.ca ([198.100.154.185]) + by mx0.pepperfish.net with esmtp (Exim 4.80) + (envelope-from <matthew@mjdsystems.ca>) id 1XvGPo-00057F-Pw + for obnam-dev@obnam.org; Mon, 01 Dec 2014 02:08:43 +0000 +Received: from cwmtaff.housem.mjdsystems.ca + (CPE00045a5ba0db-CM602ad073c297.cpe.net.cable.rogers.com [99.229.22.8]) + by scadrial.mjdsystems.ca (Postfix) with ESMTPSA id 1DD41185B31; + Sun, 30 Nov 2014 21:08:40 -0500 (EST) +From: Matthew Dawson <matthew@mjdsystems.ca> +To: obnam-dev@obnam.org +Date: Sun, 30 Nov 2014 21:08:31 -0500 +Message-Id: <1417399711-8672-3-git-send-email-matthew@mjdsystems.ca> +X-Mailer: git-send-email 2.0.4 +In-Reply-To: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +X-Spam-Score: -1.9 +X-Spam-Score-int: -18 +X-Spam-Bar: - +X-Scanned-By: pepperfish.net, Mon, 01 Dec 2014 02:08:43 +0000 +X-Spam-Report: Content analysis details: (-1.9 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -0.0 SPF_HELO_PASS SPF: HELO matches SPF record + -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + domain + -0.0 SPF_PASS SPF: sender matches SPF record + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: Matthew Dawson <matthew@mjdsystems.ca> +Subject: [PATCH 2/3] Properly handle missing parent directories them. +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +As Swift doesn't use directories, it allows files with paths specifying +nonexistent directories. This patch ensures files are not created in missing +directories, otherwise listdir/listdir2 fails to find them and breaks journal +commits. + +--- + obnamlib/plugins/swift_plugin.py | 58 ++++++++++++++++++++++++++++++++++++++-- + 1 file changed, 56 insertions(+), 2 deletions(-) + +diff --git a/obnamlib/plugins/swift_plugin.py b/obnamlib/plugins/swift_plugin.py +index ef51ad2..e569fd9 100644 +--- a/obnamlib/plugins/swift_plugin.py ++++ b/obnamlib/plugins/swift_plugin.py +@@ -49,6 +49,44 @@ def ioerror_to_oserror(method): + + return helper + ++def verify_parent_dir_exists(method): ++ '''Decorator to convert an IOError exception to OSError. ++ ++ Python's os.* raise OSError, mostly, but paramiko's corresponding ++ methods raise IOError. This decorator fixes that. ++ ++ ''' ++ ++ def helper(self, filename, *args, **kwargs): ++ if os.path.dirname(os.path.normpath(os.path.join(self.path, filename))) != '/': ++ try: ++ self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.dirname(os.path.normpath(os.path.join(self.path, filename))))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Parent directory %s not found!' % os.path.dirname(os.path.normpath(os.path.join(self.path, filename))), e) ++ return method(self, filename, *args, **kwargs) ++ ++ return helper ++ ++def verify_parent_dir_exists_rename(method): ++ '''Decorator to convert an IOError exception to OSError. ++ ++ Python's os.* raise OSError, mostly, but paramiko's corresponding ++ methods raise IOError. This decorator fixes that. ++ ++ ''' ++ ++ def helper(self, old, new, *args, **kwargs): ++ if os.path.dirname(os.path.normpath(os.path.join(self.path, new))) != '/': ++ try: ++ self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.dirname(os.path.normpath(os.path.join(self.path, new))))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise IOError('Parent directory %s not found!' % os.path.dirname(os.path.normpath(os.path.join(self.path, new))), e) ++ return method(self, old, new, *args, **kwargs) ++ ++ return helper ++ + class SwiftFile(object): + def __init__(self, swift, swift_container, filename, mode): + self._swift = swift +@@ -243,7 +281,7 @@ class SwiftFS(obnamlib.VirtualFileSystem): + # just fail silently, but that would be silly.) + raise NotImplementedError('mknod on swift: %s' % pathname) + +- @ioerror_to_oserror ++ @verify_parent_dir_exists + def mkdir(self, pathname, mode=obnamlib.NEW_DIR_MODE): + tracing.trace('mkdir: %s with mode %o' % (pathname, mode)) + try: +@@ -251,7 +289,10 @@ class SwiftFS(obnamlib.VirtualFileSystem): + except swiftclient.ClientException, e: + if e.http_status == 404: + raise OSError('Parent doesn\'t exist') ++ self._mkdir(pathname, mode) + ++ @ioerror_to_oserror ++ def _mkdir(self, pathname, mode=obnamlib.NEW_DIR_MODE): + try: + headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) + except swiftclient.ClientException, e: +@@ -267,7 +308,7 @@ class SwiftFS(obnamlib.VirtualFileSystem): + parent = os.path.dirname(pathname) + if parent and parent != pathname and not self.exists(parent): + self.makedirs(parent) +- self.mkdir(pathname, obnamlib.NEW_DIR_MODE) ++ self._mkdir(pathname, obnamlib.NEW_DIR_MODE) + + @ioerror_to_oserror + def rmdir(self, pathname): +@@ -302,6 +343,7 @@ class SwiftFS(obnamlib.VirtualFileSystem): + if e.errno != errno.ENOENT: + raise + ++ @verify_parent_dir_exists_rename + @ioerror_to_oserror + def rename(self, old, new): + tracing.trace('rename %s (%s) to %s (%s)' % (old, os.path.normpath(os.path.join(self.path, old)), new, os.path.normpath(os.path.join(self.path, new)))) +@@ -374,6 +416,7 @@ class SwiftFS(obnamlib.VirtualFileSystem): + raise IOError('Not a symlink!') + return contents + ++ @verify_parent_dir_exists + @ioerror_to_oserror + def symlink(self, source, destination): + tracing.trace('rename %s (%s) to %s (%s)' % (source, os.path.normpath(os.path.join(self.path, source)), destination, os.path.normpath(os.path.join(self.path, destination)))) +@@ -385,6 +428,7 @@ class SwiftFS(obnamlib.VirtualFileSystem): + return + raise IOError('Path exists!') + ++ @verify_parent_dir_exists + def open(self, pathname, mode, bufsize=-1): + tracing.trace('open %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) + if 'w' not in mode: +@@ -409,6 +453,10 @@ class SwiftFS(obnamlib.VirtualFileSystem): + def write_file(self, pathname, contents): + tracing.trace('writefile %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) + try: ++ self.makedirs(os.path.dirname(os.path.normpath(os.path.join(self.path, pathname)))) ++ except Exception: ++ pass ++ try: + self._swift.head_object(self._swift_container, os.path.normpath(os.path.join(self.path, pathname))[1:]) + except swiftclient.ClientException, e: + if e.http_status == 404: +@@ -420,6 +468,12 @@ class SwiftFS(obnamlib.VirtualFileSystem): + @ioerror_to_oserror + def overwrite_file(self, pathname, contents): + tracing.trace('overwrite_file %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ ++ try: ++ self.makedirs(os.path.dirname(os.path.normpath(os.path.join(self.path, pathname)))) ++ except: ++ pass ++ + self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], contents, headers={'x-object-meta-obnam-mode': obnamlib.NEW_FILE_MODE}) + self.bytes_written += len(contents) + +-- +2.0.4 + + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M852087P17339Q198.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M852087P17339Q198.exolobe1 new file mode 100644 index 0000000..0b7d7a9 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999008.M852087P17339Q198.exolobe1 @@ -0,0 +1,224 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id DBE642AA4E + for <distix@pieni.net>; Sat, 6 Dec 2014 23:54:53 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id 3ABFDCDF; + Sat, 6 Dec 2014 22:54:53 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxOFZ-00015x-4J; Sat, 06 Dec 2014 22:54:53 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XxOFX-00015r-Oj + for <obnam-dev@obnam.org>; Sat, 06 Dec 2014 22:54:51 +0000 +Received: from scadrial.mjdsystems.ca ([198.100.154.185]) + by mx0.pepperfish.net with esmtp (Exim 4.80) + (envelope-from <matthew@mjdsystems.ca>) id 1XxOFU-0002SQ-00 + for obnam-dev@obnam.org; Sat, 06 Dec 2014 22:54:51 +0000 +Received: from cwmtaff.localnet (d24-141-188-31.home.cgocable.net + [24.141.188.31]) + by scadrial.mjdsystems.ca (Postfix) with ESMTPSA id 73A3D1A5732; + Sat, 6 Dec 2014 17:54:45 -0500 (EST) +From: Matthew Dawson <matthew@mjdsystems.ca> +To: Lars Wirzenius <liw@liw.fi> +Date: Sat, 06 Dec 2014 17:54:39 -0500 +Message-ID: <1531092.aY1fI7Af0k@cwmtaff> +User-Agent: KMail/4.14.3 (Linux/3.17.4-gentoo; KDE/4.14.3; x86_64; ; ) +In-Reply-To: <20141206192247.GK6438@exolobe1.liw.fi> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> + <20141206192247.GK6438@exolobe1.liw.fi> +MIME-Version: 1.0 +X-Spam-Score: -1.9 +X-Spam-Score-int: -18 +X-Spam-Bar: - +X-Scanned-By: pepperfish.net, Sat, 06 Dec 2014 22:54:50 +0000 +X-Spam-Report: Content analysis details: (-1.9 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -0.5 PPF_USER_AGENT User-Agent: exists + 0.5 PPF_MESSAGEID_NODOTS Message-Id contains no dots after the @ + -0.0 SPF_HELO_PASS SPF: HELO matches SPF record + -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + domain + -0.0 SPF_PASS SPF: sender matches SPF record + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: obnam-dev@obnam.org +Subject: Re: [PATCH 0/3] RFC: Add support for Openstack Swift as backup + repository +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Content-Type: multipart/mixed; boundary="===============0725680542757677411==" +Mime-version: 1.0 +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + + +--===============0725680542757677411== +Content-Type: multipart/signed; boundary="nextPart1933280.baVF2Uklju"; + micalg="sha256"; protocol="application/pkcs7-signature" + + +--nextPart1933280.baVF2Uklju +Content-Transfer-Encoding: quoted-printable +Content-Type: text/plain; charset="us-ascii" + +On December 6, 2014 09:22:47 PM Lars Wirzenius wrote: +> Hi, Matthew, +>=20 +> Thanks for the preliminary patches. As you said they're not ready for= + +> merging yet, I haven't tried that, but I've had a glimpse at the code= +. +<snip> +>=20 +> I noticed the Python code isn't formatted quite like PEP8 specifies, +> and I'd prefer to see that fixed. Most importantly, source code lines= + +> (after TAB expansion with a width of 8) should be strictly less than +> 80 columns. (It's something that's a bit controversion in the modern +> world, but it's a rule I impose on myself and the Obnam code base.) +No problem, I'll move the code to that for the final set of patches. + +> I'm happy to include this in the Obnam code base, assuming clean code= + +> and suitable tests, including some that work without access to a real= + +> Swift instance (obviously the code doesn't then need to do much, but +> it shouldn't crash). Some end-user documentation would also be nice. +Sounds good to me. I'll work on cleaning everything up. Regarding the= + test=20 +cases, I'm not sure what you are looking for. As the Swift support onl= +y plugs=20 +into the VFS layer, there isn't really anything sitting between the two= +. I do=20 +have a set of tests like the SFTP does, so that the regular test suite = +passes. + +If you want, I can try to make a mock instance of Swift, but I'd bet I'= +d=20 +create more bugs in the mock than the actual code. Alternately, if you= + just=20 +want a free place to try the code against, hubic.com provides 25G free = +storage=20 +using swift. + +> Going forward, will you be available to maintain this code? +As long as I'm using Obnam/this code in my backup strategy, yes I'm hap= +py to=20 +do so. Should that change in the future, I can try to continue to help= + out. =20 +For the foreseeable future, that shouldn't be a problem. + +I'll work on improving the code, and I'll post a new patch set once I h= +ave=20 +something more presentable.=20 +=2D-=20 +Matthew +--nextPart1933280.baVF2Uklju +Content-Type: application/pkcs7-signature; name="smime.p7s" +Content-Disposition: attachment; filename="smime.p7s" +Content-Transfer-Encoding: base64 + +MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDJYw +ggY0MIIEHKADAgECAgEgMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1T +dGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5n +MSkwJwYDVQQDEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNzEwMjQyMTAy +NTVaFw0xNzEwMjQyMTAyNTVaMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRk +LjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMv +U3RhcnRDb20gQ2xhc3MgMiBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwggEiMA0GCSqG +SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDLKIVFnAEs+xnyq6UzjCqgDcvQVe1dIoFnRsQPCFO+y92k +8RK0Pn3MbQ2Gd+mehh9GBZ+36uUQA7Xj9AGM6wgPhEE34vKtfpAN5tJ8LcFxveDObCKrL7O5UT9W +snAZHv7OYPYSR68mdmnEnJ83M4wQgKO19b+Rt8sPDAz9ptkQsntCn4GeJzg3q2SVc4QJTg/WHo7w +F2ah5LMOeh8xJVSKGEmd6uPkSbj113yKMm8vmNptRPmM1+YgmVwcdOYJOjCgFtb2sOP79jji8uhW +R91xx7TpM1K3hv/wrBZwffrmmEpUeuXHRs07JqCCvFh9coKF4UQZvfEg+x3/69xRCzb1AgMBAAGj +ggGtMIIBqTAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUrlWDb+wx +yrn3HfqvazHzyB3jrLswHwYDVR0jBBgwFoAUTgvvGqRAW6UXaYcwyjRoQ9BBrvIwZgYIKwYBBQUH +AQEEWjBYMCcGCCsGAQUFBzABhhtodHRwOi8vb2NzcC5zdGFydHNzbC5jb20vY2EwLQYIKwYBBQUH +MAKGIWh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3Nmc2NhLmNydDBbBgNVHR8EVDBSMCegJaAjhiFo +dHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9zZnNjYS5jcmwwJ6AloCOGIWh0dHA6Ly9jcmwuc3RhcnRz +c2wuY29tL3Nmc2NhLmNybDCBgAYDVR0gBHkwdzB1BgsrBgEEAYG1NwECATBmMC4GCCsGAQUFBwIB +FiJodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kucGRmMDQGCCsGAQUFBwIBFihodHRwOi8v +d3d3LnN0YXJ0c3NsLmNvbS9pbnRlcm1lZGlhdGUucGRmMA0GCSqGSIb3DQEBBQUAA4ICAQA6qScN +yNO0FpHvaZTQacVMXH33O51KyEKSRw3IvdQxRu31YR0ZDGdSfgSoOVDVMSBSdmfQfdDInHPzV3LO +5DwUXZ+lxjv7z3PO2OkfnFkvTXPfn6dxJ5rJveDsTsCPcJ/Kp6/+qN5g+J6D/SaYcFD018B6L42r +0Z4VEBy36P4tjRtF14Ex10tl5tJFVKM16qWKQHbpjIgf73s49UB0CQ5lHT2DHKfq3oPfdNc5Mk93 +w1v4ryVb+qVrZIej8NsrWU+5r4O2IV91edDb/OtHFddZqHFFXKgS79IHE/hwQ2LW7r3sTX7cDUCg ++dfdwO8zeLxuwk2JF8crUoyrl66RGrRIhT8VoG/OJ1Y9uUlOav69V4cG8upi4ZG2l7JZFbcBFk91 +Wp+Payo5SuF61CmGFrZ386umkmpObtFacXda2O/bVoQ9xHQrzoTc/0KZTWvlZCLK3Ke/vGYT9ZdW +9lOjGsSFbXrlTA919L84iMK+48WGnvRWY28ZaVHpql43AtEGhXze6iNCbEDACy+4hkQYOytAqDgc +xAnQ937mYpeZFPyz/XK9QSt9VNFMuudWxZwDDDJKoQAoSG59Hou9lZ26UrK60nRdAQBmEPL8h2nu +WgoPh++XVQld9yuhbsWa39Pck8/lcfz5HUVGJF5mc/zk38iV7FDlF68puiryNq2KXHEpOTCCBlow +ggVCoAMCAQICAkk+MA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3Rh +cnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4 +MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMiBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0Ew +HhcNMTQwNDA4MTkzMjEyWhcNMTYwNDA5MDI1MjMwWjB0MQswCQYDVQQGEwJDQTEQMA4GA1UECBMH +T250YXJpbzEUMBIGA1UEBxMLTWlzc2lzc2F1Z2ExFzAVBgNVBAMTDk1hdHRoZXcgRGF3c29uMSQw +IgYJKoZIhvcNAQkBFhVtYXR0aGV3QG1qZHN5c3RlbXMuY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB +DwAwggEKAoIBAQCsTBWcCXBFDqO0tNTX30rmz1FCb08jhd6basnhtyWBdA3ecLoE0l5JgBX9fX8c +4N7o7AKK19VSjV7kigeT6DT2EGzsbzijkpareFq6JdZZzPuuJEP9r+uQ7YfUMKQsb64SbJMu3ewQ +pMYyuPAKkLFgeKmXkhcO6ybVyc5j0eYmt7U2J85a0RA1zJbtdEgJoR3FPTLM4aize9V8V9sBxu7U +tAm9eUd64GRzn45DlHKDUdTh5WaYrMEGmc0FKrmUp4ghuVevzRAk7+pLU/n3zTIu+p+hgi+w5nII +gU6yCIGk0ihNX1+bsKvIBfH/n0stq/nt0BHdQ4vh8S+RK8ynhxDhAgMBAAGjggLbMIIC1zAJBgNV +HRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwHQYDVR0O +BBYEFLKlaP5rHiPpOlakPhNpGcU/vEkNMB8GA1UdIwQYMBaAFK5Vg2/sMcq59x36r2sx88gd46y7 +MCAGA1UdEQQZMBeBFW1hdHRoZXdAbWpkc3lzdGVtcy5jYTCCAUwGA1UdIASCAUMwggE/MIIBOwYL +KwYBBAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xp +Y3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0 +eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNzdWVkIGFjY29yZGluZyB0byB0aGUgQ2xh +c3MgMiBWYWxpZGF0aW9uIHJlcXVpcmVtZW50cyBvZiB0aGUgU3RhcnRDb20gQ0EgcG9saWN5LCBy +ZWxpYW5jZSBvbmx5IGZvciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBjb21wbGlhbmNlIG9mIHRo +ZSByZWx5aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3Js +LnN0YXJ0c3NsLmNvbS9jcnR1Mi1jcmwuY3JsMIGOBggrBgEFBQcBAQSBgTB/MDkGCCsGAQUFBzAB +hi1odHRwOi8vb2NzcC5zdGFydHNzbC5jb20vc3ViL2NsYXNzMi9jbGllbnQvY2EwQgYIKwYBBQUH +MAKGNmh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3N1Yi5jbGFzczIuY2xpZW50LmNhLmNy +dDAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wDQYJKoZIhvcNAQELBQADggEB +AFRc8hOlsUqFsEULBPO5N1RzzlD5wHksYelS8h2eS8KsF3kqME6VA/SuF5TQ+CIGbg2hJGsoKzAU +MPqvIiw/beB9sBp2OGMSPTWZtch3d+2feMGr/qyRX6jpXQTUsYaVYryKwehLk2uMME1GYjWMxTZO +W8/5XBEVrlS3nmHgJIozPEbgIHR36PmuJTjU7isl+qJ1zqlXTN2GkicYeMK7hhWDT1tnLlEa2yv6 +LA6WYc69H9ECruWyYp3D17hhHtuW1JXhe+45l2AeO48gBiUzJf7SbmPL6CPeY4u8XP1UBtH18ZY8 +0E5MG184+a0GSqSsE5PNSiayDUGFTfBq7g+vhwYxggJVMIICUQIBATCBkzCBjDELMAkGA1UEBhMC +SUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRp +ZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDIgUHJpbWFyeSBJbnRlcm1l +ZGlhdGUgQ2xpZW50IENBAgJJPjANBglghkgBZQMEAgEFAKCBkzAYBgkqhkiG9w0BCQMxCwYJKoZI +hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDEyMDYyMjU0MzlaMCgGCSqGSIb3DQEJDzEbMBkwCwYJ +YIZIAWUDBAECMAoGCCqGSIb3DQMHMC8GCSqGSIb3DQEJBDEiBCDgqZVXKIWlrAUGYL8e+YyvHsuK +EWVp0fSKzX24XB3BrjANBgkqhkiG9w0BAQEFAASCAQCPlrZM2Pp5b07YAngCM0ydrg9aebVYLCB9 +CoK0pBIcgfrj6HG4ZFJCSk1soDisJtxVct1M6pYJATUDWjidTr2zgGEiFQ0pBzIQCTVuTEK6tH++ +ZzYjY+27v5MZ1wfjKOfEsnGQdjhSnnOlHauHuzEyGrbVkgisw0NlMUeJoSQqs8xj1FK/CHapCyYl +uT86S/uokJQ5ZscZC3+TyHK2KkKaqfu92qw4XlpMx/AJP7AVb024C6PSbDVdsPGoR4LA1fJ9pfex +IXGIbKElOzDqDBt/uF5cN/tvyr6ado/6rRB+bU+Rfsy0p1YpGOpqelc93mh297ZyAJiB0m9PcdYC +F8SWAAAAAAAA + +--nextPart1933280.baVF2Uklju-- + + + +--===============0725680542757677411== +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit +Content-Disposition: inline + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org + +--===============0725680542757677411==-- + + diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M812138P17339Q216.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M812138P17339Q216.exolobe1 new file mode 100644 index 0000000..13da0c8 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M812138P17339Q216.exolobe1 @@ -0,0 +1,648 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id 98D2121B02 + for <distix@pieni.net>; Mon, 1 Dec 2014 03:08:46 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id 4834834F2; + Mon, 1 Dec 2014 02:08:46 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPu-0002tE-5D; Mon, 01 Dec 2014 02:08:46 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPr-0002rH-Pa + for <obnam-dev@obnam.org>; Mon, 01 Dec 2014 02:08:43 +0000 +Received: from scadrial.mjdsystems.ca ([198.100.154.185]) + by mx0.pepperfish.net with esmtp (Exim 4.80) + (envelope-from <matthew@mjdsystems.ca>) id 1XvGPn-00057E-HC + for obnam-dev@obnam.org; Mon, 01 Dec 2014 02:08:43 +0000 +Received: from cwmtaff.housem.mjdsystems.ca + (CPE00045a5ba0db-CM602ad073c297.cpe.net.cable.rogers.com [99.229.22.8]) + by scadrial.mjdsystems.ca (Postfix) with ESMTPSA id AC2EC185B30; + Sun, 30 Nov 2014 21:08:38 -0500 (EST) +From: Matthew Dawson <matthew@mjdsystems.ca> +To: obnam-dev@obnam.org +Date: Sun, 30 Nov 2014 21:08:30 -0500 +Message-Id: <1417399711-8672-2-git-send-email-matthew@mjdsystems.ca> +X-Mailer: git-send-email 2.0.4 +In-Reply-To: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +References: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +X-Spam-Score: -1.8 +X-Spam-Score-int: -17 +X-Spam-Bar: - +X-Scanned-By: pepperfish.net, Mon, 01 Dec 2014 02:08:42 +0000 +X-Spam-Report: Content analysis details: (-1.8 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -0.0 SPF_HELO_PASS SPF: HELO matches SPF record + -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + domain + -0.0 SPF_PASS SPF: sender matches SPF record + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] + 0.1 PPF_SPLIT_TAG RAW: Body contains a split HTML tag +Cc: Matthew Dawson <matthew@mjdsystems.ca> +Subject: [PATCH 1/3] Initial swift support, with passing tests. +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +This adds an Openstack Swift VFS plugin, which passes most tests. + +--- + obnamlib/plugins/swift_plugin.py | 437 +++++++++++++++++++++++++++++++++++++++ + obnamlib/vfs.py | 4 +- + test-swift | 85 ++++++++ + without-tests | 1 + + 4 files changed, 525 insertions(+), 2 deletions(-) + create mode 100644 obnamlib/plugins/swift_plugin.py + create mode 100755 test-swift + +diff --git a/obnamlib/plugins/swift_plugin.py b/obnamlib/plugins/swift_plugin.py +new file mode 100644 +index 0000000..ef51ad2 +--- /dev/null ++++ b/obnamlib/plugins/swift_plugin.py +@@ -0,0 +1,437 @@ ++# Copyright (C) 2009-2014 Lars Wirzenius <liw@liw.fi> ++# ++# This program is free software; you can redistribute it and/or modify ++# it under the terms of the GNU General Public License as published by ++# the Free Software Foundation; either version 2 of the License, or ++# (at your option) any later version. ++# ++# This program is distributed in the hope that it will be useful, ++# but WITHOUT ANY WARRANTY; without even the implied warranty of ++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ++# GNU General Public License for more details. ++# ++# You should have received a copy of the GNU General Public License along ++# with this program; if not, write to the Free Software Foundation, Inc., ++# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. ++ ++ ++import errno ++import hashlib ++import logging ++import os ++import pwd ++import random ++import socket ++import stat ++import subprocess ++import time ++import traceback ++import tracing ++import urllib ++import urlparse ++import swiftclient ++ ++import obnamlib ++ ++def ioerror_to_oserror(method): ++ '''Decorator to convert an IOError exception to OSError. ++ ++ Python's os.* raise OSError, mostly, but paramiko's corresponding ++ methods raise IOError. This decorator fixes that. ++ ++ ''' ++ ++ def helper(self, filename, *args, **kwargs): ++ try: ++ return method(self, filename, *args, **kwargs) ++ except IOError, e: ++ raise OSError(e.errno, e.strerror or str(e), filename) ++ ++ return helper ++ ++class SwiftFile(object): ++ def __init__(self, swift, swift_container, filename, mode): ++ self._swift = swift ++ self._swift_container = swift_container ++ ++ self.filename = filename ++ self.mode = mode ++ if 'r' in mode: ++ self._contents = self._swift.get_object(self._swift_container, filename)[1] ++ self._lines = self._contents.splitlines() ++ self._lineindex = 0 ++ ++ def readline(self): ++ if len(self._lines) <= self._lineindex: ++ return '' ++ return self._lines[self._lineindex] + '\n' ++ self._lineindex += 1 ++ ++ def close(self): ++ pass ++ ++ #def _write_contents(self, contents): ++ #if 'w' in self.mode: ++ #self._swift.put_object(self._swift_container, swiftclient.client.quote(self.filename), contents, headers={'x-object-meta-obnam-mode': obnamlib.NEW_FILE_MODE}) ++ #else: ++ #raise IOError('Tried to write to read only file!') ++ ++class SwiftStat(object): ++ def __init__(self, pathname, headers): ++ if headers != None: ++ self.st_mode = int(headers['x-object-meta-obnam-mode']) if 'x-object-meta-obnam-mode' in headers else 0 ++ if headers['content-type'] == 'application/directory': ++ self.st_mode |= stat.S_IFDIR ++ self.st_size = 0 ++ else: ++ self.st_size = int(headers['content-length']) ++ ++ self.st_mtime_sec = int(headers['x-object-meta-obnam-mtime-sec']) if 'x-object-meta-obnam-mtime-sec' in headers else 0 ++ self.st_mtime_nsec = int(headers['x-object-meta-obnam-mtime-nsec']) if 'x-object-meta-obnam-mtime-nsec' in headers else 0 ++ self.st_atime_sec = int(headers['x-object-meta-obnam-atime-sec']) if 'x-object-meta-obnam-atime-sec' in headers else 0 ++ self.st_atime_nsec = int(headers['x-object-meta-obnam-atime-nsec']) if 'x-object-meta-obnam-atime-nsec' in headers else 0 ++ else: ++ self.st_mode = 0700 | stat.S_IFDIR ++ self.st_mtime_sec = 0 ++ self.st_mtime_nsec = 0 ++ self.st_atime_sec = 0 ++ self.st_atime_nsec = 0 ++ self.st_size = 0 ++ ++ self.st_nlink = 1 ++ self.st_uid = 0 ++ self.st_gid = 0 ++ self.st_blocks = (self.st_size / 512) + (1 if self.st_size % 512 else 0) ++ self.st_dev = 0 ++ self.st_ino = int(hashlib.md5(pathname).hexdigest()[:8], 16) ++ ++class SwiftFS(obnamlib.VirtualFileSystem): ++ ++ '''A VFS implementation for Openstack Swift. ++ ++ ++ ++ ''' ++ ++ def __init__(self, baseurl, create=False, settings=None): ++ tracing.trace('baseurl=%s', baseurl) ++ tracing.trace('create=%s', create) ++ obnamlib.VirtualFileSystem.__init__(self, baseurl) ++ self._roundtrips = 0 ++ self._swift_token = settings['swift-token'] ++ self.create_path_if_missing = create ++ self.reinit(baseurl, create=create) ++ ++ def log_stats(self): ++ obnamlib.VirtualFileSystem.log_stats(self) ++ logging.info('VFS: baseurl=%s roundtrips=%s' % ++ (self.baseurl, self._roundtrips)) ++ ++ def _create_root_if_missing(self): ++ self._swift.put_container(self._swift_container) ++ self.create_path_if_missing = False # only create once ++ ++ def connect(self): ++ tracing.trace('connect') ++ self._swift = swiftclient.client.Connection(preauthurl=self._swift_url,preauthtoken=self._swift_token) ++ if self.create_path_if_missing: ++ self._create_root_if_missing() ++ ++ def close(self): ++ tracing.trace('close') ++ logging.debug('SwiftFS.close called') ++ obnamlib.VirtualFileSystem.close(self) ++ ++ @ioerror_to_oserror ++ def reinit(self, baseurl, create=False): ++ tracing.trace('baseurl=%s', baseurl) ++ tracing.trace('create=%s', create) ++ if hasattr(self, 'path') and self._baseurl != baseurl: ++ raise NotImplementedError('Can\'t reinit yet') ++ self._baseurl = baseurl ++ scheme, netloc, path, query, fragment = urlparse.urlsplit(baseurl) ++ ++ if scheme != 'swift': ++ raise WrongURLSchemeError(url=baseurl) ++ scheme = 'https' ++ ++ (path, self._swift_container) = os.path.split(path) ++ self._swift_url = urlparse.urlunsplit((scheme, netloc, path, None, None)) ++ ++ self.path = '/' ++ ++ def getcwd(self): ++ return self.path ++ ++ @ioerror_to_oserror ++ def chdir(self, pathname): ++ tracing.trace('chdir %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ newpath = os.path.normpath(os.path.join(self.path, pathname)) ++ # Can always change to /, and its treated differently in swift. Thus just shortcut it. ++ if newpath == '/': ++ self.path = '/' ++ return ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ if headers.get('content-type') != 'application/directory': ++ raise OSError('Not a directory!') ++ self.path = newpath ++ ++ @ioerror_to_oserror ++ def listdir(self, pathname): ++ tracing.trace('listdir %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!') ++ files = self._swift.get_object(self._swift_container, '', query_string="path="+swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:])[1] ++ file_data = [] ++ for filename in files.split('\n'): ++ if filename == '': ++ continue ++ file_data.append(os.path.basename(urllib.unquote(filename))) ++ return file_data ++ ++ def listdir2(self, pathname): ++ tracing.trace('listdir2 %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!') ++ files = self._swift.get_object(self._swift_container, '', query_string="path="+swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:])[1] ++ file_data = [] ++ for filename in files.split('\n'): ++ if filename == '': ++ continue ++ header = self._swift.head_object(self._swift_container, filename) ++ file_data.append((os.path.basename(urllib.unquote(filename)), SwiftStat(os.path.normpath(os.path.join(self.path, pathname, filename))[1:], header))) ++ return file_data ++ ++ def lock(self, lockname, data): ++ try: ++ self.write_file(lockname, data) ++ except OSError, e: ++ raise obnamlib.LockFail(lock_name=lockname, reason=str(e)) ++ ++ def unlock(self, lockname): ++ self._remove_if_exists(lockname) ++ ++ def exists(self, pathname): ++ try: ++ self.lstat(pathname) ++ except OSError: ++ return False ++ else: ++ return True ++ ++ def isdir(self, pathname): ++ try: ++ st = self.lstat(pathname) ++ except OSError: ++ return False ++ else: ++ return stat.S_ISDIR(st.st_mode) ++ ++ def mknod(self, pathname, mode): ++ # SFTP does not provide an mknod, so we can't do this. We ++ # raise an exception, so upper layers can handle this (we _could_ ++ # just fail silently, but that would be silly.) ++ raise NotImplementedError('mknod on swift: %s' % pathname) ++ ++ @ioerror_to_oserror ++ def mkdir(self, pathname, mode=obnamlib.NEW_DIR_MODE): ++ tracing.trace('mkdir: %s with mode %o' % (pathname, mode)) ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.dirname(os.path.normpath(os.path.join(self.path, pathname))))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Parent doesn\'t exist') ++ ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], None, content_type='application/directory', headers={'x-object-meta-obnam-mode': mode}) ++ return ++ ++ raise OSError('Folder in a state of existance.') ++ ++ @ioerror_to_oserror ++ def makedirs(self, pathname): ++ tracing.trace('makedirs: %s' % pathname) ++ parent = os.path.dirname(pathname) ++ if parent and parent != pathname and not self.exists(parent): ++ self.makedirs(parent) ++ self.mkdir(pathname, obnamlib.NEW_DIR_MODE) ++ ++ @ioerror_to_oserror ++ def rmdir(self, pathname): ++ tracing.trace('rmdir %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ (headers, data) = self._swift.get_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException as e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ if stat.S_ISDIR(SwiftStat(os.path.normpath(os.path.join(self.path, pathname)), headers).st_mode): ++ if len(self._swift.get_object(self._swift_container, '', query_string="path="+swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:])[1]) != 0: ++ raise OSError('Directory not empty!') ++ else: ++ self._swift.delete_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ else: ++ raise OSError('Not a directory!') ++ ++ @ioerror_to_oserror ++ def remove(self, pathname): ++ tracing.trace('remove %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ data = self.lstat(pathname) ++ if stat.S_ISDIR(data.st_mode): ++ raise OSError('Tried to delete folder.') ++ else: ++ self._swift.delete_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ ++ def _remove_if_exists(self, pathname): ++ '''Like remove, but OK if file does not exist.''' ++ try: ++ self.remove(pathname) ++ except OSError, e: ++ if e.errno != errno.ENOENT: ++ raise ++ ++ @ioerror_to_oserror ++ def rename(self, old, new): ++ tracing.trace('rename %s (%s) to %s (%s)' % (old, os.path.normpath(os.path.join(self.path, old)), new, os.path.normpath(os.path.join(self.path, new)))) ++ try: ++ self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, new)))[1:], None, headers={'x-copy-from': self._swift_container + '/' + swiftclient.client.quote(os.path.normpath(os.path.join(self.path, old)))[1:]}) ++ except swiftclient.ClientException as e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ self.remove(old) ++ ++ @ioerror_to_oserror ++ def lstat(self, pathname): ++ tracing.trace('lstat %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ if os.path.normpath(os.path.join(self.path, pathname)) == '/': ++ return SwiftStat(os.path.normpath(os.path.join(self.path, pathname, pathname))[1:], None) ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ return SwiftStat(os.path.normpath(os.path.join(self.path, pathname, pathname))[1:], headers) ++ ++ @ioerror_to_oserror ++ def lchown(self, pathname, uid, gid): ++ self._delay() ++ if stat.S_ISLNK(self.lstat(pathname).st_mode): ++ logging.warning('NOT changing ownership of symlink %s' % pathname) ++ else: ++ self.sftp.chown(pathname, uid, gid) ++ ++ @ioerror_to_oserror ++ def chmod_symlink(self, pathname, mode): ++ tracing.trace('chmod_symlink %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ headers['x-object-meta-obnam-mode'] = mode ++ self._swift.post_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], headers=headers) ++ ++ @ioerror_to_oserror ++ def chmod_not_symlink(self, pathname, mode): ++ self.chmod_symlink(pathname, mode) ++ ++ @ioerror_to_oserror ++ def lutimes(self, pathname, atime_sec, atime_nsec, mtime_sec, mtime_nsec): ++ # FIXME: This does not work for symlinks! ++ # Sftp does not have a way of doing that. This means if the restore ++ # target is over sftp, symlinks and their targets will have wrong ++ # mtimes. ++ try: ++ headers = self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise OSError('Path not found!', e) ++ headers['x-object-meta-obnam-mtime-sec'] = mtime_sec ++ headers['x-object-meta-obnam-mtime-nsec'] = mtime_nsec ++ headers['x-object-meta-obnam-atime-sec'] = atime_sec ++ headers['x-object-meta-obnam-atime-nsec'] = atime_nsec ++ self._swift.post_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], headers=headers) ++ ++ def link(self, existing_path, new_path): ++ raise HardlinkError() ++ ++ def readlink(self, symlink): ++ (headers, contents) = self._swift.get_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, symlink)))[1:]) ++ info = SwiftStat(os.path.normpath(os.path.join(self.path, symlink))[1:], headers) ++ if stat.S_ISLNK(info.st_mode): ++ raise IOError('Not a symlink!') ++ return contents ++ ++ @ioerror_to_oserror ++ def symlink(self, source, destination): ++ tracing.trace('rename %s (%s) to %s (%s)' % (source, os.path.normpath(os.path.join(self.path, source)), destination, os.path.normpath(os.path.join(self.path, destination)))) ++ try: ++ self._swift.head_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, destination)))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, destination)))[1:], source, headers={'x-object-meta-obnam-is-symlink': 'true', 'x-object-meta-obnam-mode': 0777}) ++ return ++ raise IOError('Path exists!') ++ ++ def open(self, pathname, mode, bufsize=-1): ++ tracing.trace('open %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ if 'w' not in mode: ++ try: ++ self._swift.get_object(self._swift_container, os.path.normpath(os.path.join(self.path, pathname))[1:])[1] ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise IOError('Path not found!') ++ return SwiftFile(self._swift, self._swift_container, os.path.normpath(os.path.join(self.path, pathname))[1:], mode) ++ ++ def cat(self, pathname): ++ tracing.trace('cat %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ contents = self._swift.get_object(self._swift_container, os.path.normpath(os.path.join(self.path, pathname))[1:])[1] ++ self.bytes_read += len(contents) ++ return contents ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ raise IOError('Path not found!') ++ ++ @ioerror_to_oserror ++ def write_file(self, pathname, contents): ++ tracing.trace('writefile %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ try: ++ self._swift.head_object(self._swift_container, os.path.normpath(os.path.join(self.path, pathname))[1:]) ++ except swiftclient.ClientException, e: ++ if e.http_status == 404: ++ self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], contents, headers={'x-object-meta-obnam-mode': obnamlib.NEW_FILE_MODE}) ++ self.bytes_written += len(contents) ++ return ++ raise OSError('File in state of existance!') ++ ++ @ioerror_to_oserror ++ def overwrite_file(self, pathname, contents): ++ tracing.trace('overwrite_file %s (%s)' % (pathname, os.path.normpath(os.path.join(self.path, pathname)))) ++ self._swift.put_object(self._swift_container, swiftclient.client.quote(os.path.normpath(os.path.join(self.path, pathname)))[1:], contents, headers={'x-object-meta-obnam-mode': obnamlib.NEW_FILE_MODE}) ++ self.bytes_written += len(contents) ++ ++ ++class SwiftPlugin(obnamlib.ObnamPlugin): ++ ++ def enable(self): ++ swift_group = obnamlib.option_group['swift'] = 'Openstack Swift' ++ ++ self.app.settings.string( ++ ['swift-token'], ++ 'Auth token to access swift.', ++ group=swift_group) ++ ++ self.app.fsf.register('swift', SwiftFS, settings=self.app.settings) +diff --git a/obnamlib/vfs.py b/obnamlib/vfs.py +index a236ce2..129e004 100644 +--- a/obnamlib/vfs.py ++++ b/obnamlib/vfs.py +@@ -446,13 +446,13 @@ class VfsTests(object): # pragma: no cover + + def test_chdir_to_relative_works(self): + pathname = os.path.join(self.basepath, 'foo') +- os.mkdir(pathname) ++ self.fs.mkdir(pathname) + self.fs.chdir('foo') + self.assertEqual(self.fs.getcwd(), pathname) + + def test_chdir_to_dotdot_works(self): + pathname = os.path.join(self.basepath, 'foo') +- os.mkdir(pathname) ++ self.fs.mkdir(pathname) + self.fs.chdir('foo') + self.fs.chdir('..') + self.assertEqual(self.fs.getcwd(), self.basepath) +diff --git a/test-swift b/test-swift +new file mode 100755 +index 0000000..1211bef +--- /dev/null ++++ b/test-swift +@@ -0,0 +1,85 @@ ++#!/usr/bin/env python2 ++# Copyright 2010-2014 Lars Wirzenius ++# ++# This program is free software: you can redistribute it and/or modify ++# it under the terms of the GNU General Public License as published by ++# the Free Software Foundation, either version 3 of the License, or ++# (at your option) any later version. ++# ++# This program is distributed in the hope that it will be useful, ++# but WITHOUT ANY WARRANTY; without even the implied warranty of ++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ++# GNU General Public License for more details. ++# ++# You should have received a copy of the GNU General Public License ++# along with this program. If not, see <http://www.gnu.org/licenses/>. ++ ++ ++'''Test Swift. ++ ++''' ++ ++ ++import logging ++import os ++import pwd ++import shutil ++import tempfile ++import unittest ++ ++import obnamlib ++import obnamlib.plugins.swift_plugin ++ ++ ++class SwiftTests(unittest.TestCase, obnamlib.VfsTests): ++ ++ def setUp(self): ++ self.basepath = '/' ++ baseurl = 'swift://' ++ settings = { ++ 'swift-token': '', ++ } ++ self.fs = obnamlib.plugins.swift_plugin.SwiftFS(baseurl, ++ create=True, ++ settings=settings) ++ self.fs.connect() ++ objs = self.fs._swift.get_container(self.fs._swift_container)[1] ++ for obj in objs: ++ self.fs._swift.delete_object(self.fs._swift_container, obj.get('name')) ++ self.fs.close() ++ ++ def tearDown(self): ++ #objs = self.fs._swift.get_container(self.fs._swift_container)[1] ++ #for obj in objs: ++ #self.fs._swift.delete_object(self.fs._swift_container, obj.get('name')) ++ #self.fs.close() ++ pass ++ ++ def test_sets_path_to_absolute_path(self): ++ self.assert_(self.fs.path.startswith('/')) ++ ++ def test_initial_cwd_is_basepath(self): ++ self.assertEqual(self.fs.getcwd(), self.fs.path) ++ ++ def test_link_creates_hard_link(self): ++ pass # sftp does not support hardlinking, so not testing it ++ ++ def test_mknod_creates_fifo(self): ++ self.assertRaises(NotImplementedError, self.fs.mknod, 'foo', 0) ++ ++ def test_get_username_returns_None_for_zero(self): ++ self.assertEqual(self.fs.get_username(0), None) ++ ++ def test_get_groupname_returns_None_for_zero(self): ++ self.assertEqual(self.fs.get_groupname(0), None) ++ ++ def test_reinit_creates_target_if_requested(self): ++ pass # Swift only does containers, and won't work otherwise. Don't bother with this. ++ ++ def test_reinit_to_nonexistent_filename_raises_OSError(self): ++ pass # Swift only does containers, and won't work otherwise. Don't bother with this. ++ ++ ++if __name__ == '__main__': ++ logging.basicConfig(filename='/dev/null') ++ unittest.main() +diff --git a/without-tests b/without-tests +index faf164f..40adecb 100644 +--- a/without-tests ++++ b/without-tests +@@ -17,6 +17,7 @@ obnamlib/plugins/fsck_plugin.py + obnamlib/plugins/fuse_plugin.py + obnamlib/plugins/restore_plugin.py + obnamlib/plugins/sftp_plugin.py ++obnamlib/plugins/swift_plugin.py + obnamlib/plugins/show_plugin.py + obnamlib/plugins/verify_plugin.py + obnamlib/plugins/vfs_local_plugin.py +-- +2.0.4 + + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M898823P17339Q219.exolobe1 b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M898823P17339Q219.exolobe1 new file mode 100644 index 0000000..74859fc --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/new/1455999010.M898823P17339Q219.exolobe1 @@ -0,0 +1,106 @@ +Return-Path: <obnam-dev-bounces@obnam.org> +X-Original-To: distix@pieni.net +Delivered-To: distix@pieni.net +Received: from bagpuss.pepperfish.net (bagpuss.pepperfish.net [148.251.8.16]) + (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) + (No client certificate requested) + by pieni.net (Postfix) with ESMTPS id 8456421B02 + for <distix@pieni.net>; Mon, 1 Dec 2014 03:08:46 +0100 (CET) +Received: from platypus.pepperfish.net (unknown [10.112.100.20]) + by bagpuss.pepperfish.net (Postfix) with ESMTP id D1521348C; + Mon, 1 Dec 2014 02:08:45 +0000 (GMT) +Received: from localhost ([::1] helo=platypus.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPt-0002sf-OC; Mon, 01 Dec 2014 02:08:45 +0000 +Received: from inmail ([10.112.100.10] helo=mx0.pepperfish.net) + by platypus.pepperfish.net with esmtp (Exim 4.80 #2 (Debian)) + id 1XvGPr-0002rG-PN + for <obnam-dev@obnam.org>; Mon, 01 Dec 2014 02:08:43 +0000 +Received: from scadrial.mjdsystems.ca ([198.100.154.185]) + by mx0.pepperfish.net with esmtp (Exim 4.80) + (envelope-from <matthew@mjdsystems.ca>) id 1XvGPn-00057D-HA + for obnam-dev@obnam.org; Mon, 01 Dec 2014 02:08:43 +0000 +Received: from cwmtaff.housem.mjdsystems.ca + (CPE00045a5ba0db-CM602ad073c297.cpe.net.cable.rogers.com [99.229.22.8]) + by scadrial.mjdsystems.ca (Postfix) with ESMTPSA id 73D3D185B2F; + Sun, 30 Nov 2014 21:08:36 -0500 (EST) +From: Matthew Dawson <matthew@mjdsystems.ca> +To: obnam-dev@obnam.org +Date: Sun, 30 Nov 2014 21:08:29 -0500 +Message-Id: <1417399711-8672-1-git-send-email-matthew@mjdsystems.ca> +X-Mailer: git-send-email 2.0.4 +X-Spam-Score: -1.9 +X-Spam-Score-int: -18 +X-Spam-Bar: - +X-Scanned-By: pepperfish.net, Mon, 01 Dec 2014 02:08:42 +0000 +X-Spam-Report: Content analysis details: (-1.9 points) + pts rule name description + ---- ---------------------- -------------------------------------------------- + -0.0 SPF_HELO_PASS SPF: HELO matches SPF record + -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay + domain + -0.0 SPF_PASS SPF: sender matches SPF record + -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% + [score: 0.0000] +Cc: Matthew Dawson <matthew@mjdsystems.ca> +Subject: [PATCH 0/3] RFC: Add support for Openstack Swift as backup + repository +X-BeenThere: obnam-dev@obnam.org +X-Mailman-Version: 2.1.5 +Precedence: list +List-Id: Obnam development discussions <obnam-dev-obnam.org> +List-Unsubscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=unsubscribe> +List-Archive: <http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org> +List-Post: <mailto:obnam-dev@obnam.org> +List-Help: <mailto:obnam-dev-request@obnam.org?subject=help> +List-Subscribe: <http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org>, + <mailto:obnam-dev-request@obnam.org?subject=subscribe> +Sender: obnam-dev-bounces@obnam.org +Errors-To: obnam-dev-bounces@obnam.org + +Hi all, + +This patch set contains an experimental work in progress Openstack Swift VFS for +Obnam. I've only just got it working (thus it's not merge ready) but it does +pass the VFS unit tests except for reinit support. I've done initial testing +against hubiC's implementation, and the code has managed an initial backup + +verify on some (real) test data. + +I'm posting it here for initial comments, and also to find out if such a backend +is wanted in Obnam. If this is wanted, I'll continue improving the backend. + +Current issues: + - Filename quoting is inconsistent, requires research to understand what the +Openstack library requires. + - Too many round trips, mostly caused due to extra checking carried out. +Commits are especially expensive. + - Only supports token based authentication with Swift, as that is what hubiC +uses. + +For easy pulling, the code is also available at +https://gitlab.com/MJDSys/obnam.git , on the swift branch. Note that branch +has some miscellaneous fixes as well, which I submit separately once I clean +them up. + +Matthew Dawson (3): + Initial swift. Passes tests. + Handle parent directories correctly, instead of ignoring them. + Tracing for the win! + + obnamlib/plugins/swift_plugin.py | 496 +++++++++++++++++++++++++++++++++++++++ + obnamlib/vfs.py | 4 +- + test-swift | 85 +++++++ + without-tests | 1 + + 4 files changed, 584 insertions(+), 2 deletions(-) + create mode 100644 obnamlib/plugins/swift_plugin.py + create mode 100755 test-swift + +-- +2.0.4 + + +_______________________________________________ +obnam-dev mailing list +obnam-dev@obnam.org +http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/tmp/.this-dir-not-empty/.empty/empty-file b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/tmp/.this-dir-not-empty/.empty/empty-file new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/Maildir/tmp/.this-dir-not-empty/.empty/empty-file diff --git a/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/ticket.yaml b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/ticket.yaml new file mode 100644 index 0000000..f8c1849 --- /dev/null +++ b/tickets/10d72738d6ce4ab7b0fc3f21c15635bc/ticket.yaml @@ -0,0 +1,4 @@ +ticket-id: +- 10d72738d6ce4ab7b0fc3f21c15635bc +title: +- "Re: [PATCH 0/3] RFC: Add support for Openstack Swift as backup\n repository" |