This gives us the full 16-bit range of ids (65536) instead of the much
smaller 12-bit range (4096) when limited to truncated 4-byte leb128s.
The real motivation for the truncated 4-byte leb128s is to keep the
wasted space in crc padding down, which this doesn't matter for.
In the future this may be configurable. Or maybe not. Only if truncated
leb128 tags prove to have value, at the moment it looks more like a
premature optimization if anything...
Either way we do need to test for overflowing this.
I'm still not sure this is the best decision, since it may add some
complexity to tag parsing, but making most crcs one byte may be valuable
since these exist in every single commit.
This gives tags three high-level encodings:
in-tree tags:
iiiiiii iiiiitt ttTTTTT TTT00rv
^----^--------^--^^- 16-bit id
'--------|--||- 4-bit suptype
'--||- 8-bit subtype
'|- removed bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
out-of-tree tags:
------- -----TT TTTTTTt ttt01pv
^----^--^^- 8-bit subtype
'--||- 4-bit suptype
'|- perturb bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
alt tags:
wwwwwww wwwwwww wwwwwww www1dcv
^-^^^- 28-bit weight
'||- direction bit
'|- color bit
'- valid bit
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Having the location of the subtype flipped for crc tags vs tree tags is
unintuitive, but it makes more crc tags fit in a single byte, while
preserving expected tag ordering for tree tags.
The only case where crc tags don't fit in a single byte if is non-crc
checksums (sha256?) are added, at which point I expect the subtype to
indicate which checksum algorithm is in use.
- Caching is still presenting issues with the new requirements for
rbyd trees, in this case the default bd, with 64 byte progs, revealed
and issue where rcache could become outdated when reading from disk
while ignoring what's in the pcache.
It assumes the pcache will always override the rcache, but this is not
true after pcache is flushed.
This didn't happen before as the rcache and pcache don't
interact while writing in the previous implementation. Because of
these new requirements the caching system probably deserves a
rework...
- The quick tests for sublinear space utilization don't work when
prog_size is > a byte, fortunately we should always have NOR-like
geometry under test, so we can limit these asserts to NOR-like
geometry.
- Lots of problems fitting these tests into 512-byte block_size
geometries, which is a bit concerning. This may be a larger change
from the previous implementation than expected. This may deserve more
scrutiny at small block sizes to see how things fit, since the
sublinear space utilization doesn't really kick in at this scale...
On the other hand it may just be that these tests are too aggressive
for 512-byte block sizes, since they don't yet do compaction, which
should help with padding/crc overhead...
Toying around with the idea that since rbyd trees have strict height
gaurantees after compaction (2*log2(n)+1), we can proactively calculate
the maximum on-disk space required for a worst case tree+leb128
encoding.
This would _greatly_ simplify things such as metadata compaction and
splitting, and allow unstorable file metadata (too many custom
attributes) to error early.
One issue is that this calculated worst case will likely be ~4-5x worst
than the actual encoding due to leb128 compression. Though this may be an
acceptable tradeoff for the simplification and more reliable behavior.
Previously the subtype was encoded above the suptype. This was an issue
if you wanted to, say, traverse all tags in a given suptype.
I'm not sure yet if this sort of functionality is needed, it may be
useful for cleaning up/replacing classes of tags, such as file struct
tags, but not sure yet. At the very least is avoids unintuitive tag
ordering in the tree, which could potential cause problems for
create/deletes.
New encoding:
tags:
iiiiiii iiiiitt ttTTTTT TTT0trv
^----^--------^-^^^- 16-bit id
'--------|-'||- 5-bit suptype (split)
'--||- 8-bit subtype
'|- perturb/remove bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
alts:
wwwwwww wwwwwww wwwwwww www1dcv
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Also a large amount of name changes and other cleanup.
The idea here is to just use a separate variable for found tags, instead
of stealing a bit for tracking which tags were found.
I was hoping this would simplify more operations on the tags to avoid
some masking operations, but it doesn't seem like it.
Tag removal is basically a range-delete of one that doesn't change
the tree weights. This deduplicates the two methods of deleting tags and
completely gets rid of tombstoning.
Note we still need a "removed tag" encoding so that we can invalidate
tags that may be found during fetch operations. Fortunately this
encoding is basically free due to overlap with alt encoding.
There are two ways to represent the bounds in the search down the rbyd
tree:
1. Using lower/upper bounds and the id we are searching for:
lower bound id upper bound
| | |
v v v
<-a--b--c--d--e--f--g--h->
2. Using the lower/upper weights, which implicitly encodes the id,
saving a word:
lower weight upper weight
| |
.-'-----------. .--'-.
<-a--b--c--d--e--f--g--h->
Now that I am diving deep into the rbyd algorithm again, the lower/upper
weight based approach just isn't worth the extra mental steps required
to understand what the algorithm is doing. Besides, we likely pay for
the implicit id anyways since we need enough state to remove the
ambiguity of sparse tags.
Though this is a bit of an ugly fix...
Red edges mess everything up during deletion since we always need to
go down red edges in case we need to do a red flip. This puts our
algorithm into an awkward state.
It would be easier to do away with colors completely when we start
deleting (after all this branch of the tree _will_ be shrinking), but
this conflicts with the ultimate goal of deduplicate rbyd append.
Tree deletion is such a pain. It always seems like an easy addition to
the core algorithm but always comes with problems.
The initial plan for deletes was to iterate through all tags, tombstone,
and then adjust weights as needed. This accomplishes deletes with little
change to the rbyd algorithm, but adds a complex traversal inside the
commit logic. Doable in one commit, but complex. It also risks weird
unintuitive corner cases since the cost of deletion grows with the number
of tags being deleted (O(m log n)).
But this rbyd data structure is a tree, so in theory it's possible to
delete a whole range of tags in a single O(log n) operation.
---
This is a proof-of-concept range deletion algorithm for rbyd trees.
Note, this does not preserve rbyd's balancing properties! But it is no
worse than tombstoning. This is acceptable for littlefs as any
unbalanced trees will be rebalanced during compaction.
The idea is to follow the same underlying dhara algorithm, where we
follow a search path and save any alt pointers not taken, but we follow
both search paths that form the outside of the range, and only keep
outside edges.
For example, a tree:
.-------o-------.
| |
.---o---. .---o---.
| | | |
.-o-. .-o-. .-o-. .-o-.
| | | | | | | |
a b c d e f g h
To delete the range d-e, we would search for d, and search for e:
********o********
* *
.---***** *****---.
| * * |
.-o-. .-*** ***-. .-o-.
| | | * * | | |
a b c d e f g h
And keep the outside edges:
.--- ---.
| |
.-o-. .- -. .-o-.
| | | | | |
a b c f g h
But how do we combine the outside edges? The simpler option is to do
both searches seperately, one after the other. This would end up with a
tree like this:
.---------o
| |
.-o-. .---o
| | | |
a b c o---------.
| |
o---. .-o-.
| | | |
_ f g h
But this horribly throws off the balance of our tree! It's worse than
tombstoning, and gets worse with more tags.
An alternative strategy, which is used here, is to alternate edges as we
descend down the tree. This unfortunately is more complex, and requires
~2x the RAM, but better preserves the balance of our tree. It isn't
perfect, because we lose color information, but we can leave that up to
compaction:
.---------o
| |
.-o-. o---------.
| | | |
a b .---o .-o-.
| | | |
c o---. g h
| |
_ f
I also hope this can be merged into lfs_rbyd_append, deduplicating the
entire core rbyd append algorithm.
The idea behind the full hint is that we may stay on the current search
path, benefiting from a loaded cache. Though this should probably be
benchmarked eventually.
It's quite lucky a spare bit is free in the tag encoding, this means we
don't need a reserved length value as originally planned. We end up using
all of the bits that overlap the alt pointer encoding, which is nice and
unexpected.
This really doesn't need to be this large. The extra 4-bits is not worth
the required extra byte of waste in every commit (CRCs tags are forced
to use a fully expanded leb128 due to a catch-22 issue).
This was a inter-abstraction-layer assumption that won't hold true with
the rbyd blocks.
The whole caching layer probably deserves a rewrite at this point.
It turns out statefulness works quite well with this algorithm (The
prototype was in Haskell, which created some artificial problems. I
think it may have just been too high-level a language for this
near-instruction-level algorithm).
This bias makes it so that tag lookups always find a tag strictly >= the
requested tag, unless we are at the end of the tree.
This makes tree traversal trivial, which is quite nice.
Need to remove ntag now, it's no longer needed.
- Moved alt encoding 0x1 => 0x4, which can lead to slightly better
lookup tables, the perturb bit takes the same place as the color bit,
which means both can be ignored in readonly operations.
- Dropped lfs_rbyd_fetchmatch, asking each lfs_rbyd_fetch to include NULL
isn't that bad.
New encoding:
tags:
iiii iiiiiii iiiiiTT TTTTTTt ttt0tpv
^--------^------^^^- 16-bit id
'------|||- 8-bit type2
'||- 5-bit type1
'|- perturb bit
'- valid bit
llll lllllll lllllll lllllll lllllll
^- n-bit length
alts:
wwww wwwwwww wwwwwww wwwwwww www1dcv
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjj jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
- primitive lfs_rbyd_fetch
- primitive lfs_rbyd_commit
- tag reading/progging and encoding machinery
The tag encoding scheme here uses pairs of leb128s, encoding either
a normal tag:
iiii iiiiiii iiiiiTT TTTTTTt ttttt0v
^--------^------^-^- 16-bit id
'------|-|- 8-bit type2
'-|- 6-bit type1
'- valid bit
llll lllllll lllllll lllllll lllllll
^- n-bit length
Or an alt pointer:
wwww wwwwwww wwwwwww wwwwwww wwwcd1v
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjj jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Note that two bits overlap the alt pointer dir/color encoding, this
is actually not a problem at all since some tags (crcs/fcrcs) don't
participate in the rbyd tree and can use these bits.
There's a number of benefits to using leb128s, which should probably
be written about, most notably is the abstraction of the device's
word-size. The "n-bits" above can be whatever word size works on the
device, trading off code-size for storage capabilities without breaking
compatibility with other devices. This will eventually be negotiated via
the superblock.
Christopher Haster