Tag removal is basically a range-delete of one that doesn't change
the tree weights. This deduplicates the two methods of deleting tags and
completely gets rid of tombstoning.
Note we still need a "removed tag" encoding so that we can invalidate
tags that may be found during fetch operations. Fortunately this
encoding is basically free due to overlap with alt encoding.
- Unless there is a bug, rbyd trees should be strictly <= (2*log2(n)+1)
in height. The extra +1 from traditional red-black trees is due to the
introduced to-be-pruned alt, but since we clean those up as soon as we
can, only one will ever exist in any search path.
This also holds true with range deletion, however the definition of n
changes to the number of tree operations. This is the same for tombstoning.
- Delete-all recovery is a bit tricky because we have no tree at that
point, which is weird for an append-only data-structure.
There are two ways to represent the bounds in the search down the rbyd
tree:
1. Using lower/upper bounds and the id we are searching for:
lower bound id upper bound
| | |
v v v
<-a--b--c--d--e--f--g--h->
2. Using the lower/upper weights, which implicitly encodes the id,
saving a word:
lower weight upper weight
| |
.-'-----------. .--'-.
<-a--b--c--d--e--f--g--h->
Now that I am diving deep into the rbyd algorithm again, the lower/upper
weight based approach just isn't worth the extra mental steps required
to understand what the algorithm is doing. Besides, we likely pay for
the implicit id anyways since we need enough state to remove the
ambiguity of sparse tags.
Attempted to implement TEST_PERMUTATIONS but that went nowhere useful.
Unfortunately test defines don't have a way to represent arrays, and
adding such a representation would be unreasonably complicated.
For practical purposes it's better to just do the permutation generation
inside the tests. Maybe at some point it would be nice to integrate
permutation testing so the state gets hashed somehow as a part of the
test failure, but now is not the time.
Though this is a bit of an ugly fix...
Red edges mess everything up during deletion since we always need to
go down red edges in case we need to do a red flip. This puts our
algorithm into an awkward state.
It would be easier to do away with colors completely when we start
deleting (after all this branch of the tree _will_ be shrinking), but
this conflicts with the ultimate goal of deduplicate rbyd append.
Tree deletion is such a pain. It always seems like an easy addition to
the core algorithm but always comes with problems.
The initial plan for deletes was to iterate through all tags, tombstone,
and then adjust weights as needed. This accomplishes deletes with little
change to the rbyd algorithm, but adds a complex traversal inside the
commit logic. Doable in one commit, but complex. It also risks weird
unintuitive corner cases since the cost of deletion grows with the number
of tags being deleted (O(m log n)).
But this rbyd data structure is a tree, so in theory it's possible to
delete a whole range of tags in a single O(log n) operation.
---
This is a proof-of-concept range deletion algorithm for rbyd trees.
Note, this does not preserve rbyd's balancing properties! But it is no
worse than tombstoning. This is acceptable for littlefs as any
unbalanced trees will be rebalanced during compaction.
The idea is to follow the same underlying dhara algorithm, where we
follow a search path and save any alt pointers not taken, but we follow
both search paths that form the outside of the range, and only keep
outside edges.
For example, a tree:
.-------o-------.
| |
.---o---. .---o---.
| | | |
.-o-. .-o-. .-o-. .-o-.
| | | | | | | |
a b c d e f g h
To delete the range d-e, we would search for d, and search for e:
********o********
* *
.---***** *****---.
| * * |
.-o-. .-*** ***-. .-o-.
| | | * * | | |
a b c d e f g h
And keep the outside edges:
.--- ---.
| |
.-o-. .- -. .-o-.
| | | | | |
a b c f g h
But how do we combine the outside edges? The simpler option is to do
both searches seperately, one after the other. This would end up with a
tree like this:
.---------o
| |
.-o-. .---o
| | | |
a b c o---------.
| |
o---. .-o-.
| | | |
_ f g h
But this horribly throws off the balance of our tree! It's worse than
tombstoning, and gets worse with more tags.
An alternative strategy, which is used here, is to alternate edges as we
descend down the tree. This unfortunately is more complex, and requires
~2x the RAM, but better preserves the balance of our tree. It isn't
perfect, because we lose color information, but we can leave that up to
compaction:
.---------o
| |
.-o-. o---------.
| | | |
a b .---o .-o-.
| | | |
c o---. g h
| |
_ f
I also hope this can be merged into lfs_rbyd_append, deduplicating the
entire core rbyd append algorithm.
The idea behind the full hint is that we may stay on the current search
path, benefiting from a loaded cache. Though this should probably be
benchmarked eventually.
Considered adding --ignore-errors to watch.py, but it doesn't really
make sense with watch.py's implementation. watch.py would need to not update
in realtime, which conflicts with other use cases.
It's quite lucky a spare bit is free in the tag encoding, this means we
don't need a reserved length value as originally planned. We end up using
all of the bits that overlap the alt pointer encoding, which is nice and
unexpected.
In theory spreading rbyd trees across multiple commits shouldn't change
anything, but the whole point of testing is to catch things that are not
in the theory.
This really doesn't need to be this large. The extra 4-bits is not worth
the required extra byte of waste in every commit (CRCs tags are forced
to use a fully expanded leb128 due to a catch-22 issue).
This was a inter-abstraction-layer assumption that won't hold true with
the rbyd blocks.
The whole caching layer probably deserves a rewrite at this point.
It turns out statefulness works quite well with this algorithm (The
prototype was in Haskell, which created some artificial problems. I
think it may have just been too high-level a language for this
near-instruction-level algorithm).
This bias makes it so that tag lookups always find a tag strictly >= the
requested tag, unless we are at the end of the tree.
This makes tree traversal trivial, which is quite nice.
Need to remove ntag now, it's no longer needed.
- Moved alt encoding 0x1 => 0x4, which can lead to slightly better
lookup tables, the perturb bit takes the same place as the color bit,
which means both can be ignored in readonly operations.
- Dropped lfs_rbyd_fetchmatch, asking each lfs_rbyd_fetch to include NULL
isn't that bad.
New encoding:
tags:
iiii iiiiiii iiiiiTT TTTTTTt ttt0tpv
^--------^------^^^- 16-bit id
'------|||- 8-bit type2
'||- 5-bit type1
'|- perturb bit
'- valid bit
llll lllllll lllllll lllllll lllllll
^- n-bit length
alts:
wwww wwwwwww wwwwwww wwwwwww www1dcv
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjj jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
- primitive lfs_rbyd_fetch
- primitive lfs_rbyd_commit
- tag reading/progging and encoding machinery
The tag encoding scheme here uses pairs of leb128s, encoding either
a normal tag:
iiii iiiiiii iiiiiTT TTTTTTt ttttt0v
^--------^------^-^- 16-bit id
'------|-|- 8-bit type2
'-|- 6-bit type1
'- valid bit
llll lllllll lllllll lllllll lllllll
^- n-bit length
Or an alt pointer:
wwww wwwwwww wwwwwww wwwwwww wwwcd1v
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjj jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Note that two bits overlap the alt pointer dir/color encoding, this
is actually not a problem at all since some tags (crcs/fcrcs) don't
participate in the rbyd tree and can use these bits.
There's a number of benefits to using leb128s, which should probably
be written about, most notably is the abstraction of the device's
word-size. The "n-bits" above can be whatever word size works on the
device, trading off code-size for storage capabilities without breaking
compatibility with other devices. This will eventually be negotiated via
the superblock.
crc32c, with a polynomial of 0x11edc6f41, is generally numerically superior
to the more common crc32 standard. Catching more bit errors across
a wider range of message messages (except 2-bit errors) without any changes
to the underlying algorithm.
Philip Koopman has a large body of work exploring optimal polynomials here:
http://users.ece.cmu.edu/~koopman/crc/crc32.html
And from his experiments we know the maximum message size where we can
still detect a given number of bit errors for each polynomial:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
crc32 0x104c11db7 = ∞ 4294967263 91607 2974 268 171 91 57 34 21 12 10 10 10 - - -
crc32c 0x11edc6f41 = ∞ 2147483615 2147483615 5243 5243 177 177 47 47 20 20 8 8 6 6 1 1
So really crc32c should be prefered where possible. Koopman also has
alternative polynomials with slightly different properties, but crc32c
is already popular enough to have a decent amount of hardware support.
---
Another nice feature of crc32c is that its polynomial has even parity.
It turns out that even-parity polynomials give us the nifty property
parity(crc(m)) == parity(m).
A quick proof:
crc(m) = m(x) x^|P|-1 mod P(x)
parity(m) = m(x) x mod x+1
though note: x mod x+1 = 1, by hand
so:
parity(m) = m(x)*1 mod x+1
= m(x) mod x+1
solving for parity(crc(m)):
parity(crc(m)) = (m(x) x^|P|-1 mod P(x)) mod x+1
note: (a mod b) mod c = a mod c, if c divides b,
aka (a mod b) mod c = a mod c, if b mod c = 0
so if P(x) mod x+1 = 0,
aka if parity(P) = 0:
parity(crc(m)) = m(x) x^|P|-1 mod x+1
but, like before: x^|P|-1 mod x+1 = 1, by hand
so:
parity(crc(m)) = m(x)*1 mod x+1
= m(x) mod x+1
= parity(m)
so if parity(P) = 0:
parity(crc(m)) = parity(m)
This has the potential to replace the 1-bit counter in the metadata tags
with a more general solution that doesn't require extra state.
This of course should never happen normally, two half-orphans requires
two parents, which is disallowed in littlefs for this reason. But it can
happen if there is an outdated half-orphan later in the metadata
linked-list. The two half-orphans can cause the deorphan step to get
stuck, constantly "fixing" the first half-orphan before it has a chance
to remove the problematic, outdated half-orphan later in the list.
The solution here is to do a full check for half-orphans before
restarting the half-orphan loop. This strategy has the potential to
visit more metadata blocks unnecessarily, but avoids situations where
removing a later half-orphan will eventually cause an earlier
half-orphan to resolve itself.
Found with heuristic powerloss testing with test_relocations_reentrant_renames
after 192 nested powerlosses.
Christopher Haster