- Caching is still presenting issues with the new requirements for
rbyd trees, in this case the default bd, with 64 byte progs, revealed
and issue where rcache could become outdated when reading from disk
while ignoring what's in the pcache.
It assumes the pcache will always override the rcache, but this is not
true after pcache is flushed.
This didn't happen before as the rcache and pcache don't
interact while writing in the previous implementation. Because of
these new requirements the caching system probably deserves a
rework...
- The quick tests for sublinear space utilization don't work when
prog_size is > a byte, fortunately we should always have NOR-like
geometry under test, so we can limit these asserts to NOR-like
geometry.
- Lots of problems fitting these tests into 512-byte block_size
geometries, which is a bit concerning. This may be a larger change
from the previous implementation than expected. This may deserve more
scrutiny at small block sizes to see how things fit, since the
sublinear space utilization doesn't really kick in at this scale...
On the other hand it may just be that these tests are too aggressive
for 512-byte block sizes, since they don't yet do compaction, which
should help with padding/crc overhead...
$ ./scripts/dbgrbyd.py disk 4096 0 -t
mdir 0x0, rev 1, size 121
off tag data (truncated)
0000005e: +-+-+--> uattr 0x01 4 aa aa aa aa ....
0000000f: | | '--> uattr 0x02 4 aa aa aa aa ....
0000001d: | '----> uattr 0x03 4 aa aa aa aa ....
0000002d: | .----> uattr 0x04 4 aa aa aa aa ....
0000003d: | | .--> uattr 0x05 4 aa aa aa aa ....
0000004f: '-+-+-+> uattr 0x06 4 aa aa aa aa ....
00000004: '> uattr 0x07 4 aa aa aa aa ....
Unfortunately this tree can end up a bit confusing when alt pointers
live in unrelated search paths...
Toying around with the idea that since rbyd trees have strict height
gaurantees after compaction (2*log2(n)+1), we can proactively calculate
the maximum on-disk space required for a worst case tree+leb128
encoding.
This would _greatly_ simplify things such as metadata compaction and
splitting, and allow unstorable file metadata (too many custom
attributes) to error early.
One issue is that this calculated worst case will likely be ~4-5x worst
than the actual encoding due to leb128 compression. Though this may be an
acceptable tradeoff for the simplification and more reliable behavior.
Previously the subtype was encoded above the suptype. This was an issue
if you wanted to, say, traverse all tags in a given suptype.
I'm not sure yet if this sort of functionality is needed, it may be
useful for cleaning up/replacing classes of tags, such as file struct
tags, but not sure yet. At the very least is avoids unintuitive tag
ordering in the tree, which could potential cause problems for
create/deletes.
New encoding:
tags:
iiiiiii iiiiitt ttTTTTT TTT0trv
^----^--------^-^^^- 16-bit id
'--------|-'||- 5-bit suptype (split)
'--||- 8-bit subtype
'|- perturb/remove bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
alts:
wwwwwww wwwwwww wwwwwww www1dcv
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Also a large amount of name changes and other cleanup.
The idea here is to just use a separate variable for found tags, instead
of stealing a bit for tracking which tags were found.
I was hoping this would simplify more operations on the tags to avoid
some masking operations, but it doesn't seem like it.
Tag removal is basically a range-delete of one that doesn't change
the tree weights. This deduplicates the two methods of deleting tags and
completely gets rid of tombstoning.
Note we still need a "removed tag" encoding so that we can invalidate
tags that may be found during fetch operations. Fortunately this
encoding is basically free due to overlap with alt encoding.
- Unless there is a bug, rbyd trees should be strictly <= (2*log2(n)+1)
in height. The extra +1 from traditional red-black trees is due to the
introduced to-be-pruned alt, but since we clean those up as soon as we
can, only one will ever exist in any search path.
This also holds true with range deletion, however the definition of n
changes to the number of tree operations. This is the same for tombstoning.
- Delete-all recovery is a bit tricky because we have no tree at that
point, which is weird for an append-only data-structure.
There are two ways to represent the bounds in the search down the rbyd
tree:
1. Using lower/upper bounds and the id we are searching for:
lower bound id upper bound
| | |
v v v
<-a--b--c--d--e--f--g--h->
2. Using the lower/upper weights, which implicitly encodes the id,
saving a word:
lower weight upper weight
| |
.-'-----------. .--'-.
<-a--b--c--d--e--f--g--h->
Now that I am diving deep into the rbyd algorithm again, the lower/upper
weight based approach just isn't worth the extra mental steps required
to understand what the algorithm is doing. Besides, we likely pay for
the implicit id anyways since we need enough state to remove the
ambiguity of sparse tags.
Attempted to implement TEST_PERMUTATIONS but that went nowhere useful.
Unfortunately test defines don't have a way to represent arrays, and
adding such a representation would be unreasonably complicated.
For practical purposes it's better to just do the permutation generation
inside the tests. Maybe at some point it would be nice to integrate
permutation testing so the state gets hashed somehow as a part of the
test failure, but now is not the time.
Though this is a bit of an ugly fix...
Red edges mess everything up during deletion since we always need to
go down red edges in case we need to do a red flip. This puts our
algorithm into an awkward state.
It would be easier to do away with colors completely when we start
deleting (after all this branch of the tree _will_ be shrinking), but
this conflicts with the ultimate goal of deduplicate rbyd append.
Tree deletion is such a pain. It always seems like an easy addition to
the core algorithm but always comes with problems.
The initial plan for deletes was to iterate through all tags, tombstone,
and then adjust weights as needed. This accomplishes deletes with little
change to the rbyd algorithm, but adds a complex traversal inside the
commit logic. Doable in one commit, but complex. It also risks weird
unintuitive corner cases since the cost of deletion grows with the number
of tags being deleted (O(m log n)).
But this rbyd data structure is a tree, so in theory it's possible to
delete a whole range of tags in a single O(log n) operation.
---
This is a proof-of-concept range deletion algorithm for rbyd trees.
Note, this does not preserve rbyd's balancing properties! But it is no
worse than tombstoning. This is acceptable for littlefs as any
unbalanced trees will be rebalanced during compaction.
The idea is to follow the same underlying dhara algorithm, where we
follow a search path and save any alt pointers not taken, but we follow
both search paths that form the outside of the range, and only keep
outside edges.
For example, a tree:
.-------o-------.
| |
.---o---. .---o---.
| | | |
.-o-. .-o-. .-o-. .-o-.
| | | | | | | |
a b c d e f g h
To delete the range d-e, we would search for d, and search for e:
********o********
* *
.---***** *****---.
| * * |
.-o-. .-*** ***-. .-o-.
| | | * * | | |
a b c d e f g h
And keep the outside edges:
.--- ---.
| |
.-o-. .- -. .-o-.
| | | | | |
a b c f g h
But how do we combine the outside edges? The simpler option is to do
both searches seperately, one after the other. This would end up with a
tree like this:
.---------o
| |
.-o-. .---o
| | | |
a b c o---------.
| |
o---. .-o-.
| | | |
_ f g h
But this horribly throws off the balance of our tree! It's worse than
tombstoning, and gets worse with more tags.
An alternative strategy, which is used here, is to alternate edges as we
descend down the tree. This unfortunately is more complex, and requires
~2x the RAM, but better preserves the balance of our tree. It isn't
perfect, because we lose color information, but we can leave that up to
compaction:
.---------o
| |
.-o-. o---------.
| | | |
a b .---o .-o-.
| | | |
c o---. g h
| |
_ f
I also hope this can be merged into lfs_rbyd_append, deduplicating the
entire core rbyd append algorithm.
The idea behind the full hint is that we may stay on the current search
path, benefiting from a loaded cache. Though this should probably be
benchmarked eventually.
Considered adding --ignore-errors to watch.py, but it doesn't really
make sense with watch.py's implementation. watch.py would need to not update
in realtime, which conflicts with other use cases.
It's quite lucky a spare bit is free in the tag encoding, this means we
don't need a reserved length value as originally planned. We end up using
all of the bits that overlap the alt pointer encoding, which is nice and
unexpected.
In theory spreading rbyd trees across multiple commits shouldn't change
anything, but the whole point of testing is to catch things that are not
in the theory.
This really doesn't need to be this large. The extra 4-bits is not worth
the required extra byte of waste in every commit (CRCs tags are forced
to use a fully expanded leb128 due to a catch-22 issue).
This was a inter-abstraction-layer assumption that won't hold true with
the rbyd blocks.
The whole caching layer probably deserves a rewrite at this point.
Christopher Haster