To: Lars Wirzenius <liw@liw.fi>
References: <0dc53aa0-b510-8607-ba63-f527f49bdf50@univpm.it>
 <20170703053310.7y5qri5l6mduhuli@liw.fi>
 <d62610ae-0749-ec34-eb6a-625ad2107e77@univpm.it>
From: Michele Alessandrini <m.alessandrini@univpm.it>
Organization: =?UTF-8?Q?DIBET_-_Universit=c3=a0_Politecnica_delle_Marche?=
Message-ID: <6e754eb6-34a3-a1e6-9c10-dc4b053ee954@univpm.it>
Date: Mon, 3 Jul 2017 15:42:26 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <d62610ae-0749-ec34-eb6a-625ad2107e77@univpm.it>
Content-Type: multipart/mixed; boundary="------------D34B4AF69ADA0C644386FED6"
Cc: obnam-dev@obnam.org
Subject: patch N.2 for "exclude-if-present" option
Precedence: list
Sender: obnam-dev-bounces@obnam.org
Errors-To: obnam-dev-bounces@obnam.org

This is a multi-part message in MIME format.
--------------D34B4AF69ADA0C644386FED6
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi, I made some changes to the first patch:
- modified python code as suggested by Lars
- improved option description string for man page
- added a test in the yarns

I managed to add a test among the other scenarios by using only the 
existing "IMPLEMENTS" recipe. I'm confident it passes (it was not 
passing in the first tries for syntax mistakes). I confirm that I have 
an error for 2 scenarios only, I report the output here if you are 
interested.

Bye and thanks, let me know!
Michele


----------------------------------------------------------

ERROR: In scenario "use repository format 6"
step "THEN restored data in R1 matches T/manifest-1" failed,
with exit code 1:
Standard output from shell command:
     --- 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/original-manifest 
2017-07-03 15:28:21.712932607 +0200
     +++ 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/restored-manifest 
2017-07-03 15:28:21.708932607 +0200
     @@ -1,7 +1,7 @@
      Name: .
      Mtime: 2013-12-28 21:11:07 +0000
      Mode: 40775
     -Nlink: 4
     +Nlink: 1

      Name: fifo
      Mtime: 2013-12-28 21:11:07 +0000
     @@ -111,10 +111,10 @@
      Name: dir-empty
      Mtime: 2013-12-28 21:11:07 +0000
      Mode: 40775
     -Nlink: 2
     +Nlink: 1

      Name: symlink-target-dir
      Mtime: 2013-12-28 21:11:07 +0000
      Mode: 40775
     -Nlink: 2
     +Nlink: 1


Standard error from shell command:
     + GNUPGHOME=/tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg
     + mkdir -p /tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg
     + rsync -a --delete /home/michele/devel/obnam/test-gpghome/. 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/dot-gnupg/.
     + cd /tmp/tmpWC8Ic9/use_repository_format_6/datadir/R1
     + true
     + ls
     + wc -l
     + cd tmp
     + true
     + ls
     + wc -l
     + cd tmpbr5YS5
     + true
     + + wcls
      -l
     + cd data
     + true
     + ls
     + wc -l
     + break
     + summain -r --exclude=Ino --exclude=Dev --exclude=Uid 
--exclude=Username --exclude=Gid --exclude=Group --checksum=SHA1 .
     + normalise_manifest_times
     + sed /^Mtime:/s/\.[0-9]* / /
     + normalise_manifest_times 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/T/manifest-1
     + sed /^Mtime:/s/\.[0-9]* / / 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/T/manifest-1
     + diff -u 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/original-manifest 
/tmp/tmpWC8Ic9/use_repository_format_6/datadir/restored-manifest

ERROR: In scenario "use repository format 6 with in-tree data"
step "THEN restored data in R1 matches T/manifest-1" failed,
with exit code 1:
Standard output from shell command:
     --- 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/original-manifest 
2017-07-03 15:28:23.648932644 +0200
     +++ 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/restored-manifest 
2017-07-03 15:28:23.644932644 +0200
     @@ -1,7 +1,7 @@
      Name: .
      Mtime: 2014-01-07 18:36:19 +0000
      Mode: 40775
     -Nlink: 4
     +Nlink: 1

      Name: fifo
      Mtime: 2014-01-07 18:36:19 +0000
     @@ -111,10 +111,10 @@
      Name: dir-empty
      Mtime: 2014-01-07 18:36:19 +0000
      Mode: 40775
     -Nlink: 2
     +Nlink: 1

      Name: symlink-target-dir
      Mtime: 2014-01-07 18:36:19 +0000
      Mode: 40775
     -Nlink: 2
     +Nlink: 1


Standard error from shell command:
     + 
GNUPGHOME=/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg
     + mkdir -p 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg
     + rsync -a --delete /home/michele/devel/obnam/test-gpghome/. 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/dot-gnupg/.
     + cd 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/R1
     + true
     + + wc -l
     ls
     + cd tmp
     + true
     + + wc -l
     ls
     + cd tmpvR_VsS
     + true
     + ls+
     wc -l
     + cd data
     + true
     + + wc -l
     ls
     + break
     + summain -r --exclude=Ino --exclude=Dev+  --exclude=Uid 
--exclude=Username --exclude=Gid --exclude=Group --checksum=SHA1 .
     normalise_manifest_times
     + sed /^Mtime:/s/\.[0-9]* / /
     + normalise_manifest_times 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/T/manifest-1
     + sed /^Mtime:/s/\.[0-9]* / / 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/T/manifest-1
     + diff -u 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/original-manifest 
/tmp/tmpYuAW3Y/use_repository_format_6_with_in-tree_data/datadir/restored-manifest

Failed scenarios:
     - use repository format 6
     - use repository format 6 with in-tree data
ERROR: Test suite FAILED in 2 scenarios
ERROR: Command '['yarn', '-s', 'yarns/obnam.sh', '--env', 
'REPOSITORY_FORMAT=6', 'yarns/0010-introduction.yarn', 
'yarns/0020-test-environment.yarn', 'yarns/0040-generations.yarn', 
'yarns/0050-multiple-clients.yarn', 'yarns/0060-encryption.yarn', 
'yarns/0070-compression.yarn', 'yarns/0080-verify.yarn', 
'yarns/0090-lock-handling.yarn', 'yarns/0100-fuse.yarn', 
'yarns/0110-administration.yarn', 'yarns/0120-corrupt-repo.yarn', 
'yarns/0200-repo-formats.yarn', 'yarns/0300-kdirstat-integration.yarn', 
'yarns/9000-implements.yarn', 'yarns/0030-basics.yarn']' returned 
non-zero exit status 1


La seguente informativa e' inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell'ente. 
 
 
INVESTI NELLA RICERCA 
 
il 5 per mille all'Universita' Politecnica delle Marche e' un investimento per i giovani, per il loro futuro - C.F. 00382520427 
 
http://www.univpm.it/5_per_mille 

--------------D34B4AF69ADA0C644386FED6
Content-Type: text/x-patch;
 name="0000-cover-letter.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0000-cover-letter.patch"

>From 22f9c4450f331d2d057f357cd1e767c50a0ca0ae Mon Sep 17 00:00:00 2001
From: Michele Alessandrini <michelealessandrini74@gmail.com>
Date: Mon, 3 Jul 2017 15:34:19 +0200
Subject: [PATCH 0/2] *** SUBJECT HERE ***

*** BLURB HERE ***

Michele Alessandrini (2):
  Add "exclude-if-present" option to exclude directories containing give
    filename(s)
  improved python and added yarn for exclude-if-present

 manual/en/060-backing-up.mdwn                 |  4 ++
 obnamlib/plugins/exclude_if_present_plugin.py | 53 +++++++++++++++++++++++++++
 without-tests                                 |  1 +
 yarns/0030-basics.yarn                        | 26 +++++++++++++
 4 files changed, 84 insertions(+)
 create mode 100644 obnamlib/plugins/exclude_if_present_plugin.py

-- 
2.13.2


--------------D34B4AF69ADA0C644386FED6
Content-Type: text/x-patch;
 name="0001-Add-exclude-if-present-option-to-exclude.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0001-Add-exclude-if-present-option-to-exclude.patch"

>From 50fbcd08554c798456d1693d1a409badc12f6c8c Mon Sep 17 00:00:00 2001
From: Michele Alessandrini <michelealessandrini74@gmail.com>
Date: Thu, 29 Jun 2017 12:00:09 +0200
Subject: [PATCH 1/2] Add "exclude-if-present" option to exclude directories
 containing give filename(s)

---
 manual/en/060-backing-up.mdwn                 |  4 +++
 obnamlib/plugins/exclude_if_present_plugin.py | 52 +++++++++++++++++++++++++++
 without-tests                                 |  1 +
 3 files changed, 57 insertions(+)
 create mode 100644 obnamlib/plugins/exclude_if_present_plugin.py

diff --git a/manual/en/060-backing-up.mdwn b/manual/en/060-backing-up.mdwn
index ee28d6e2..f0edb38f 100644
--- a/manual/en/060-backing-up.mdwn
+++ b/manual/en/060-backing-up.mdwn
@@ -112,6 +112,10 @@ backups:
   are usually not important to back up, and tagging the directory
   can be easier than constructing a regular expression for
   `--exclude`.
+* The `--exclude-if-present` setting excludes directories that contain a
+  given file name. This option can be used more than once to specify
+  several file names, if one of them is present the directory gets
+  excluded.
 * The `--one-file-system` setting excludes any mount points and the
   contents of the mounted filesystem. This is useful for skipping,
   for example, virtual filesystems such as `/proc`, remote filesystems
diff --git a/obnamlib/plugins/exclude_if_present_plugin.py b/obnamlib/plugins/exclude_if_present_plugin.py
new file mode 100644
index 00000000..3af81711
--- /dev/null
+++ b/obnamlib/plugins/exclude_if_present_plugin.py
@@ -0,0 +1,52 @@
+# Copyright (C) 2015  Lars Wirzenius
+# Copyright (C) 2017  Michele Alessandrini
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import logging
+import os
+import stat
+
+import obnamlib
+
+
+class ExcludeIfPresentPlugin(obnamlib.ObnamPlugin):
+
+    def enable(self):
+        backup_group = obnamlib.option_group['backup'] = 'Backing up'
+
+        self.app.settings.string_list(
+            ['exclude-if-present'],
+            'exclude directories (and their subdirs) '
+            'that contain the given file (can be '
+            'used multiple times)',
+            group=backup_group)
+
+        self.app.hooks.add_callback('config-loaded', self.config_loaded)
+
+    def config_loaded(self):
+        if len(self.app.settings['exclude-if-present']) > 0:
+            self.app.hooks.add_callback('backup-exclude', self.exclude)
+
+    def exclude(self, fs=None, pathname=None, stat_result=None, exclude=None,
+                **kwargs):
+        if stat.S_ISDIR(stat_result.st_mode):
+            for filename in self.app.settings['exclude-if-present']:
+                filename_path = os.path.join(pathname, filename)
+                if fs.exists(filename_path):
+                    logging.debug('Excluding (exclude-if-present): %s',
+                                  pathname)
+                    exclude[0] = True
+                    break
diff --git a/without-tests b/without-tests
index f5979198..15d447bf 100644
--- a/without-tests
+++ b/without-tests
@@ -21,6 +21,7 @@ obnamlib/plugins/dump_repo_plugin.py
 obnamlib/plugins/encryption_plugin.py
 obnamlib/plugins/exclude_caches_plugin.py
 obnamlib/plugins/exclude_pathnames_plugin.py
+obnamlib/plugins/exclude_if_present_plugin.py
 obnamlib/plugins/force_lock_plugin.py
 obnamlib/plugins/forget_plugin.py
 obnamlib/plugins/fsck_plugin.py
-- 
2.13.2


--------------D34B4AF69ADA0C644386FED6
Content-Type: text/x-patch;
 name="0002-improved-python-and-added-yarn-for-exclude-if-presen.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-improved-python-and-added-yarn-for-exclude-if-presen.pa";
 filename*1="tch"

>From 22f9c4450f331d2d057f357cd1e767c50a0ca0ae Mon Sep 17 00:00:00 2001
From: Michele Alessandrini <michelealessandrini74@gmail.com>
Date: Mon, 3 Jul 2017 15:32:36 +0200
Subject: [PATCH 2/2] improved python and added yarn for exclude-if-present

---
 obnamlib/plugins/exclude_if_present_plugin.py |  5 +++--
 yarns/0030-basics.yarn                        | 26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/obnamlib/plugins/exclude_if_present_plugin.py b/obnamlib/plugins/exclude_if_present_plugin.py
index 3af81711..598306d1 100644
--- a/obnamlib/plugins/exclude_if_present_plugin.py
+++ b/obnamlib/plugins/exclude_if_present_plugin.py
@@ -30,14 +30,15 @@ class ExcludeIfPresentPlugin(obnamlib.ObnamPlugin):
         self.app.settings.string_list(
             ['exclude-if-present'],
             'exclude directories (and their subdirs) '
-            'that contain the given file (can be '
+            'that contain the given filename (can be '
             'used multiple times)',
+            metavar='FILENAME',
             group=backup_group)
 
         self.app.hooks.add_callback('config-loaded', self.config_loaded)
 
     def config_loaded(self):
-        if len(self.app.settings['exclude-if-present']) > 0:
+        if self.app.settings['exclude-if-present']:
             self.app.hooks.add_callback('backup-exclude', self.exclude)
 
     def exclude(self, fs=None, pathname=None, stat_result=None, exclude=None,
diff --git a/yarns/0030-basics.yarn b/yarns/0030-basics.yarn
index 226ec2a8..7590115f 100644
--- a/yarns/0030-basics.yarn
+++ b/yarns/0030-basics.yarn
@@ -273,6 +273,32 @@ Time to backup.
     AND user U restores their latest generation in repository R into X
     THEN L, restored to X, matches manifest M
 
+Exclude directories containing given filename(s)
+------------------------------------------------
+
+The `--exclude-if-present` option tells Obnam to exclude any
+directories containing the given filename.
+
+    SCENARIO exclude directories containing given filename
+    GIVEN 1k of new data in directory L/wanted
+    AND 1k of new data in directory L/unwanted
+    AND file L/unwanted/_nobackup_ with permissions 664
+
+We'll now create the manifest, but remove `L/unwanted` (and files in
+`L/unwanted`) so that it matches what we need. We do it this instead of
+creating the manifest before `L/unwanted`, because creating `L/unwanted`
+changes the timestamp of `L`.
+
+    AND a manifest of L in M
+    AND unwanted is removed from manifest M
+
+Time to backup.
+
+    AND user U sets configuration exclude-if-present to _nobackup_
+    WHEN user U backs up directory L to repository R
+    AND user U restores their latest generation in repository R into X
+    THEN L, restored to X, matches manifest M
+
 Excluded, already backed up files, are not included in next generation
 ----------------------------------------------------------------------
 
-- 
2.13.2


--------------D34B4AF69ADA0C644386FED6
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
obnam-dev mailing list
obnam-dev@obnam.org
http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org

--------------D34B4AF69ADA0C644386FED6--