| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
This is less alarming and confusing to people.
|
|
Patch-by: Antoine Brenner
|
|
|
|
|
|
|
|
This bug was exposed by a test data set provided by Rob Kendrick.
Saving reference count groups would sometimes overwrite a correct
saved group with one that had all counts set to zero. This would
happen when the "dirty set" (node ids whose refcounts had been changed
but not yet saved) contained, for example, node ids for the Nth and
N+2th group, but not the N+1th group between them. The old
save_refcounts logic would blithely save that group anyway, and if the
in-memory reference counts happened to not be in the refcount dict, it
would save zeroes instead.
To fix this, save_refcounts now only saves groups that have any dirty
refcounts, and skips saving a refcount group that is all clean.
To do this efficiently, we need to change the encode_refcounts
function signature, to get the set of keys it is to actually put into
the group. This set is now computed by save_refcounts.
This meant that all call sites for encode_refcounts need to be fixed
as well. Luckily the fix is easy: there's only one production use of
it, the rest is tests or benchmarks.
|
|
Since we no longer treat a refcount of 0 specially, these tests no
longer make any sense.
|
|
We would sometimes, rarely, have a crash that looked like this:
File "..refcountstore.py", line 76, in get_refcount
return self.refcounts[node_id]
KeyError: 32970
In other words, the object keeping track of reference counts for
B-tree nodes would think it doesn't have the reference count for a
node, when it should have it.
After much debugging, by myself and Itamar Turner-Trauring, and
greatly helped by a repeatable test case provided by Rob Kendrick,
this was tracked to an optimisation in the set_refcount method, which
would remove (to save memory) the refcount for a node (remove the
entry for a node's id in the refcount dict), when the refcount was set
to zero. Unfortunately this conflicted with an assumption in
get_refcount that a refcount for a node that was in the dirty set was
actually present in the refcount dict.
The bug fix is easy: don't remove the node from the refcount dict, and
eat the (very minor) memory consumption hit, in the name of
correctness.
|
|
|
|
|
|
|
|
What used to happen:
* Instance 1 would crash in the middle of committing a journal.
* Instance 2 would open the B-tree in read-only mode, and not
read files from the journal. It would then crash, because the
partly modified B-tree made no sense.
What happens now:
* When a B-tree is opened in read-only mode, and the journal
is partly comitted, the journal is checked in preference
for files in the B-tree proper. No confusion anymore.
|
|
|
|
|
|
|
|
The introduction of the allow_writes parameter in larch/forest.py:open_forest()
broke example.py and speed-test.
Just adding the parameter to the open_forest calls was not enough to fix the
issue. This patch fixes the issue by adding the allow_writes parameter where it
is needed beyond the open_forest calls.
Most notably, this patch changes the __init__ signature of the abstract base
class larch.NodeStore from __init__(self, node_size, codec) to
__init__(self, allow_writes, node_size, codec)
This mimicks the current signatures of NodeStoreDisk and NodeStoreMemory, which
include the use of allow_writes as first parameter, and which seem required
to actually create a forest anyway since forest.py creates node_store instances
with allow_writes as first parameter in open_forest
|
|
|
|
This makes it easier to find them on disk, since the filenames are
already in hex.
|
|
|
|
|
|
This is a preliminary step for dealing with missing children.
We're going to need to do a get_node for the children, but we
want to avoid doing that many times per node, so this change
makes it so this will be easier to do.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This makes it possible to do smoother progress reporting.
|
|
|
|
|
|
Thanks, Damien Couroussé, for reporting the bug via Debian.
|
|
|
|
|
|
|
|
|
|
No point in checking for a file to exist, if we're going to be reading
at once anyway.
|
|
|
|
|
|
|
|
Also, add a list_files method to the journal so NodeStoreDisk
can easily find a list, without having to detect externally
whether specific files exist or not, and whether they're in the
journal or not.
|
|
|
|
This should be less error prone.
|
|
If writer removes a file, we should still be able to read it in the reader
until the writer commits.
|
|
|
