Make exclude-caches scenario clearer, robust

Originally, we would create L/wanted, then do a manifest, then create
L/cache, then backup, restore, verify. This would sometimes fail, if
the full-second timestamp of L would change when L/cache was created.
So we (well, I) changed it so that we fiddled with the manifest just
before making a backup, but this was weird and confusing and would
still sometimes fail. This should now be much clearer and therefore
more robust.

3 files changed, 52 insertions, 13 deletions

diff --git a/yarns/0030-basics.yarn b/yarns/0030-basics.yarn
index ae553c14..7e4dffa2 100644
--- a/yarns/0030-basics.yarn
+++ b/yarns/0030-basics.yarn
@@ -171,23 +171,19 @@ directories tagged like that.
 
     SCENARIO exclude cache directories
     GIVEN 1k of new data in directory L/wanted
+    AND 1k of new data in directory L/cache
+    AND directory L/cache is tagged as a cache directory
 
-The directory `L` now contains what we want to restore: only the
-wanted data, no cache directory. This is a good time to create the
-manifest.
+We'll now create the manifest, but remove `L/cache` (and files in
+`L/cache`) so that it matches what we need. We do it this instead of
+creating the manifest before `L/cache`, because creating `L/cache`
+changes the timestamp of `L`.
 
     AND a manifest of L in M
+    AND cache is removed from manifest M
 
-We now create the cache directory and tag it. Then we backup,
-excluding caches, and restore, and verify. However, there's a snag:
-the manifest has a modification time for `L` that is wrong: the
-creation of the cache directory modifies it. Thus, we also modify the
-manifest to update the first `Mtime` field, i.e., the one for the
-directory `L`.
+Time to backup.
 
-    AND 1k of new data in directory L/cache
-    AND directory L/cache is tagged as a cache directory
-    AND manifest M has Mtime for the first entry set from L
     AND user U sets configuration exclude-caches to yes
     WHEN user U backs up directory L to repository R
     AND user U restores their latest generation in repository R into X
diff --git a/yarns/9000-implements.yarn b/yarns/9000-implements.yarn
index 2932373b..956c5d6d 100644
--- a/yarns/9000-implements.yarn
+++ b/yarns/9000-implements.yarn
@@ -137,6 +137,21 @@ entry.
         ' "$DATADIR/$MATCH_1" > "$DATADIR/new-manifest"
     mv "$DATADIR/new-manifest" "$DATADIR/$MATCH_1"
 
+Sometimes we create manifests with extra stuff. This allows us to
+remove them afterwards.
+
+    IMPLEMENTS GIVEN (\S+) is removed from manifest (\S+)
+    awk -v skip1="$MATCH_1" -v skip2="$MATCH_1/" '
+        $1 == "Name:" &&
+            ($2 == skip1 || substr($2, 1, length(skip2)) == skip2) {
+                paragraph = ""; ignore = 1; next }
+        NF > 0 && !ignore { paragraph = paragraph $0 "\n" }
+        NF == 0 && paragraph { printf "%s\n", paragraph; paragraph = "" }
+        NF == 0 { ignore = 0 }
+        END { if (paragraph) printf "%s", paragraph }
+    ' "$DATADIR/$MATCH_2" > "$DATADIR/$MATCH_2.new"
+    mv "$DATADIR/$MATCH_2.new" "$DATADIR/$MATCH_2"
+
 
 Obnam configuration management
 ------------------------------
diff --git a/yarns/obnam.sh b/yarns/obnam.sh
index 55fc2336..c19f532e 100644
--- a/yarns/obnam.sh
+++ b/yarns/obnam.sh
@@ -121,9 +121,37 @@ normalise_manifest_times()
 }
 
 
+# Remove "Nlink" lines for directories. It is rarely a useful thing
+# compare exactly, unlike for non-directories, since it is merely
+# extra checking that the right number of subdirectories exist. That
+# extra checking is both unnecessary (if the subdirs are in the
+# manifest, they already get checked), and harmful (if a subdirectory
+# is excluded from a backup, it won't be in the restored data
+# manifest, but the link count will be wrong).
+
+remove_nlink_for_directories()
+{
+    # This assumes tat Mode comes before Nlink, which is does in
+    # summain output.
+    awk '
+        $1 == "Mode:" && $2 ~ /^40/ { isdir = 1 }
+        $1 == "Nlink:" && isdir { skip }
+        NF > 0 { paragraph = paragraph $0 "\n" }
+        NF == 0 && paragraph {
+            printf "%s\n", paragraph
+            paragraph = ""
+            isdir = 0
+        }
+        END { if (paragraph) printf "%s", paragraph }
+    ' "$DATADIR/$MATCH_2" > "$DATADIR/$MATCH_2.new"
+}
+
+
 # Create a manifest with summain of a file or a directory.
 
 manifest()
 {
-    summain -r "$1" --exclude Ino --exclude Dev | normalise_manifest_times
+    summain -r "$1" --exclude Ino --exclude Dev |
+    normalise_manifest_times |
+    remove_nlink_for_directories
 }