This helps debug a corrupted mtree with cycles, which has been a problem
in the past.
Also fixed a small rendering issue with dbgmtree.py not connecting inner
tree edges to mdir roots correctly during rendering.
Optimizing a script? This might sound premature, but the tree rendering
was, uh, quite slow for any decently sized (>1024) btree.
The main reason is that tree generation is quite hacky in places, repeatedly
spitting out multiple copies of the inner node's rbyd trees for example.
Rather than rewrite the tree generation implementation to be smarter,
this just changes all edge representations to namedtuples (which may
reduce memory pressure a bit), and collects them into a Python set.
This has the effect of deduplicating generated edges efficiently, and
improved the rendering performance significantly.
---
I also considered memoizing rbyd tree, but dropped the idea since the
current renderer performs well enough.
In addition to plugging in the rbyd and btree renderers in dbgbtree.py,
this required wiring in rbyd trees in the mdirs and mroots.
A bit tricky, but with a more-or-less straightforward implementation thanks
to the common edge description used for the tree renderer.
For example, a relatively small mtree:
$ ./scripts/dbgmtree.py disk -B4096 -t -i
mroot 0x{0,1}.45, rev 1, weight 0
mdir ids tag ...
{0000,0001}: .---------> -1 magic 8 ...
| .-------> config 21 ...
+-+-+ btree 7 ...
0006.000a: | .-+ 0 mdir w1 2 ...
{0002,0003}: | | '-> 0.0 inlined w1 1024 ...
0006.000a: '-+-+ 1 mdir w1 2 ...
{0004,0005}: '-> 1.0 inlined w1 1024 ...
This builds on dbgrbyd.py and dbgbtree.py by allowing for quick
debugging of the littlefs mtree, which is a btree of rbyd pairs with a
few bells and whistles.
This also comes with a number of tweaks to dbgrbyd.py and dbgbtree.py,
mostly changing rbyd addresses to support some more mdir friendly
formats.
The syntax for rbyd addresses is starting to converge into a couple
common patterns, which is nice for quickly determining what type of
address you are looking at at a glance:
- 0x12 => An rbyd at block 0x12
- 0x12.34 => An rbyd at block 0x12 with trunk 0x34
- 0x{12,34} => An rbyd at either block 0x12 or block 0x34 (an mdir)
- 0x{12,34}.56 => An rbyd at either block 0x12 or block 0x34 with trunk 0x56
These scripts have also been updated to support any number of blocks in
an rbyd address, for example 0x{12,34,56,78}. This is a bit of future
proofing. >2 blocks in mdirs may be explored in the future for the
increased redundancy.
LFSR_TAG_BNAME => LFSR_TAG_BRANCH
LFSR_TAG_BRANCH => LFSR_TAG_BTREE
Maybe this will be a problem in the future if our branch structure is
not the same as a standalone btree, but I don't really see that
happening.
This became surprisingly tricky.
The main issue is knowing when to split mdirs, and how to determine
this without wasting erase cycles.
Unlike splitting btree nodes, we can't salvage failed compacts here. As
soon as the salvage commit is written to disk, the commit becomes immediately
visibile to the filesystem because it still exists in the mtree. This is
a problem if we lose power.
We're likely going to need to implement rbyd estimates. This is
something I hoped to avoid because it brings in quite a bit of
complexity and might lead to an annoying amount of storage waste since
our estimates will need to be conservative to avoid unrecoverable
situations.
---
Also changed the on-disk btree/branch struct to store a copy of the weight.
This was already required for the root of the btree, requiring the
weight to be stored in every btree pointer allows better code
deduplication at the cost of some redundancy on btree branches, where
the weight is already implied by the rbyd structure.
This weight is usually a single byte for most branches anyways.
This may be worth revisiting at some point to see if there's any other
unexpected tradeoffs.
This work already indicates we need more data-related helper
functions. We shouldn't need this many function calls to do "simple"
operations such as fetch the superconfig if it exists.
This is an absurd optimization that stems from the observation that the
branch encoding for the inner-rbyds in a B-tree is enough information to
jump directly to the trunk of the rbyd without needing an lfsr_rbyd_fetch.
This results in a pretty ridiculous performance jump from O(m log_m(n/m))
to O(log(m) log_m(n/m)).
If the complexity analysis isn't impressive enough, look at some rough
benchmarking of read operations for 4KiB-block, 1K-entry B-trees:
12KiB ^ :: :. :: .: .: :. : .: :. : : .. : : . : .: : : :
| .:: .::.::.:: ::.::::::::::::.::::::::.::::::::::::.
| : :::':: ::'::'::':: :' :':: :'::::::::': ::::::': :
before | ::: ::' :' :' :: :' '' ' ' '' : : : '' ' ' '
| ::: ''
|:
0B :'------------------------------------------------------>
.17KiB ^ ............:::::::::::::::::::::::::::::
| . .....:::::''''''''' ' ' '
| .::::::::::::
after | :':''
|.::
.:'
0B :------------------------------------------------------->
0 1K
In order for this to work, the branch encoding did need to be tweaked
slightly. Before it stored block+off, now it stores block+trunk where
"trunk" is the offset of the entry point into the rbyd tree. Both off
and trunk are enough info to know when to stop fetching, if necessary,
but trunk allows lookups to jump directly into the branches rbyd tree
without a fetch.
With the change to trunk, lfsr_rbyd_fetch has also be extended to allow
fetching of any internal trunks, not just the last trunk in the commit.
This is very useful for dbgrbyd.py, but doesn't currently have a use in
littlefs itself. But it's at least valuable to have the feature available
in case it does become useful.
Note that two cases still requires the slower O(m log_m(n/m)) lookup
with lfsr_rbyd_fetch:
1. Name lookups, since we currently use a linear-search O(m) to find names.
2. Validating B-tree rbyd's, which requires a linear fetch O(m) to
validate the checksums. We will need to do this at least once
after mount.
It's also worth mentioning this will likely have a large impact on B-tree
traversal speed. Which is huge as I am expecting B-tree traversal to be
the main bottleneck once garbage-collection (or its replacement) is
involved.
I've been wanting to make this change for a while now (tag,id => id,tag).
The id,tag order matches the common lexicographic order used for sorting
tuples. Sorting tag,id tuples by their id first is less common.
The reason for this order in the codebase is because all attrs on disk
start with their tag first, since its decoding determines the purpose of
the id field (keep in mind this includes other non-tree tags such as
crcs, alts, etc). But with the move to storing weights instead of tags
on disk, this gives us a clear point to switch from tag,w to id,tag
ordering.
I may be thinking to much about this, but it does affect a significant
amount of the codebase.
While the previous renderer was "technically correct", the attempt to
map rotated alts to their nearest neighbor just made the resulting tree
an unreadable mess.
Now the renderer prunes alts with unreachable edges (like they would be
during lfsr_rbyd_append). And aligns all alts with their destination
trunk. This results in a much more readable, if slightly less accurate,
rendering of the tree.
Example:
$ ./scripts/dbgrbyd.py -B4096 disk 0 -t
rbyd 0x0, rev 1, size 1508, weight 40
off ids tag data (truncated)
0000032a: .-+-> 0 reg w1 1 73 s
00000026: | '-> 1-5 reg w5 1 62 b
00000259: .-------+---> 6-11 reg w6 1 6f o
00000224: | .-+-+-> 12-17 reg w6 1 6e n
0000028e: | | | '-> 18 reg w1 1 70 p
00000076: | | '---> 19-20 reg w2 1 64 d
0000038f: | | .-> 21-22 reg w2 1 75 u
0000041d: | .---+---+-> 23 reg w1 1 78 x
000001f3: | | .-> 24-27 reg w4 1 6d m
00000486: | | .-----+-> 28-29 reg w2 1 7a z
000004f3: | | | .-----> 30-31 reg w2 1 62 b
000004ba: | | | | .---> 32-35 reg w4 1 61 a
0000058d: | | | | | .-> 36-37 reg w2 1 65 e
000005c6: +-+-+-+-+-+-> 38-39 reg w2 1 66 f
Sorting weights instead of ids just had a number of benefits, suggesting
this is a better design:
- Calculating the id and delta of each rbyd trunk is surprisingly
easier - id is now just lower+w-1, and no extra conditions are
needed for unr tags, which just have a weight of zero.
- Removes ambiguity around which id unr tags should be assigned to,
especially unrs that delete ids.
- No more +-1 weirdness when encoding/decoding tag ids - the weight
can be written as-is and -1 ids are infered from their weight and
position in the tree (lower+w-1 = 0+0-1 = -1).
- Weights compress better under leb128 encoding, since they are usually
quite small.
This fixed two notable bugs:
1. Using "altle 0xfff0" to terminate unreachable rbyd trunks threw off
id calculations in lfsr_rbyd_fetch searches. We derive the tag's
id+weight from the lower bound calculated as the sum of all "altle"s
and an always-followed "altle 0xfff0" throws this off.
We _could_ derive the tag's id+weight from the upper bound, inverting
this relationship, but decided to revert back to using "altgt 0" to
terminate unreachable rbyd trunks.
Using the lower bound is more intuitive, and "altgt 0" has the
benifit of supporting variable-length tags if we ever need to adopt
those.
To avoid the previous issues around 0-tag holes (which was the original
motivation for altle 0xfff0), 0-tags are now automatically adjusted
in lfsr_rbyd_lookup, and avoided in lfsr_rbyd_append.
But note! if any implemention tries to look up 0-tags, this will
eventually break! See previous commits for more info.
2. Unfortunately, we can't combine branch updates and weight updates in
lfsr_btree_commit in the general case.
If our btree contains bname tags, the weight is attached to the
bname tag, separately from the branch tag.
Branch updates in lfsr_btree_commit need two separate attrs for the
weight and branch struct for this reason, which is unfortunate.
The amount of extra conditions to make bname+branch pairs work makes
me want to redesign the inner-nodes of the btrees, but I can't think
of a better way to approach the problem.
This "mk" bit must not be written to disk, it would conflict with the
other non-tree tag encodings. But we can use this bit in the context of
lfsr_tag_append to disambiguate tags changing weight from inserting new
tags.
Note that in the context of rbyd compactions, this will make things a bit
weird, since it's no longer just a direct one-to-one copy of each tag.
To make compactions a bit easier, this implementation allows the "mk"
bit to be set on any tag and ignores it when the weight delta is zero.
It turns out that this scheme greatly simplifies the awkward
leaf-split-alt calculation that previously had several if statements to
handle different corner cases, with the caveat that "mk" tags need their
ids adjusted by +1. Added this adjustment directly into lfsr_rbyd_append
for now, so the upper-level interface can be a bit more intuitive.
Though this may need to change later if it is more confusing than
helpful.
This does not work as is due to ambiguity with grows and insertions.
Before, these were disambiguated by seperate grow and attr tags. You
effectively grew the neighboring id before claiming its weight
as yours. But now that the attr itself creates the grow/insertion,
it's ambiguous which one is intended.
Changed always-follow alts that we use to terminated grow/shrink/remove
operations to use `altle 0xfff0` instead of `altgt 0`.
`altgt 0` gets the job done as long as you make sure tag 0 never ends up
in an rbyd query. But this kept showing up as a problem, and recent
debugging revealed some erronous 0 tag lookups created vestigial alt
pointers (not necessarily a problem, but space-wasting).
Since we moved to a strict 16-bit tag, making these `altle 0xfff0`
doesn't really have a downside, and means we can expect rbyd lookups
around 0 to behave how one would normally expect.
As a (very minor) plus, the value zero usually has special encodings in
instruction sets, so being able to use it for rbyd_lookups offers a
(very minor) code size saving.
---
Sidenote: The reasons altle/altgt is how it is and asymmetric:
1. Flipping these alts is a single bit-flip, which only happens if they
are asymmetric (only one includes the equal case).
2. Our branches are biased to prefer the larger tag. This makes
traversal trivial. It might be possible to make this still work with
altlt/altge, but would require some increments/decrements, which
might cause problems with boundary conditions around the 16-bit tag
limit.
I only recently noticed there is enough information in each rbyd trunk
to infer the effective grow/shrinks. This has a number of benefits:
- Cleans up the tag encoding a bit, no longer expecting tag size to
sometimes contain a weight (though this could've been fixed other
ways).
0x6 in the lower nibble now reserved exclusively for in-device tags.
- grow/shrinks can be implicit to any tag. Will attempt to leverage this
in the future.
- The weight of an rbyd can no longer go out-of-sync with itself. While
this _shouldn't_ happen normally, if it does I imagine it'd be very
hard to debug.
Now, there is only one source of knowledge about the weight of the
rbyd: The most recent set of alt-pointers.
Note that remove/unreachable tags now behave _very_ differently when it
comes to weight calculation, remove tags require the tree to make the
tag unreachable. This is a tradeoff for the above.
The main motivation for this was issues fitting a good tag encoding into
14-bits. The extra 2-bits (though really only 1 bit was needed) from
making this not a leb encoding opens up the space from 3 suptypes to
15 suptypes, which is nothing to shake a stick at.
The main downsides:
1. We can't rely on leb encoding for effectively-infinite extensions.
2. We can't shorten small tags (crcs, grows, shrinks) to one byte.
For 1., extending the leb encoding beyond 14-bits is already
unpalatable, because it would increase RAM costs in the tag
encoder/decoder,` which must assume a worst-case tag size, and would likely
add storage cost to every alt pointer, more on this in the next section.
The current encoding is quite generous, so I think it is unlikely we
will exceed the 16-bit encoding space. But even if we do, it's possible
to use a spare bit for an "extended" set of tags in the future.
As for 2., the lack of compression is a downside, but I've realized the
only tags that really matter storage-wise are the alt pointers. In any
rbyds there will be roughly O(m log m) alt pointers, but at most O(m) of
any other tags. What this means is that the encoding of any other tag is
in the noise of the encoding of our alt pointers.
Our alt pointers are already pretty densely packed. But because the
sparse key part of alt-pointers are stored as-is, the worst-case
encoding of in-tree tags likely ends up as the encoding of our
alt-pointers. So going up to 3-byte tags adds a surprisingly large
storage cost.
As a minor plus, le16s should be slightly cheaper to encode/decode. It
should also be slightly easier to debug tags on-disk.
tag encoding:
TTTTtttt ttttTTTv
^--------^--^^- 4+3-bit suptype
'---|- 8-bit subtype
'- valid bit
iiii iiiiiii iiiiiii iiiiiii iiiiiii
^- m-bit id/weight
llll lllllll lllllll lllllll lllllll
^- m-bit length/jump
Also renamed the "mk" tags, since they no longer have special behavior
outside of providing names for entries:
- LFSR_TAG_MK => LFSR_TAG_NAME
- LFSR_TAG_MKBRANCH => LFSR_TAG_BNAME
- LFSR_TAG_MKREG => LFSR_TAG_REG
- LFSR_TAG_MKDIR => LFSR_TAG_DIR
B-trees with names are now working, though this required a number of
changes to the B-tree layout:
1. B-tree no-longer require name entries (LFSR_TAG_MK) on each branch.
This is a nice optimization to the design, since these name entries
just waste space in purely weight-based B-trees, which are probably
going to be most B-trees in the filesystem.
If a name entry is missing, the struct entry, which is required,
should have the effective weight of the entry.
The first entry in every rbyd block is expected to be have no name
entry, since this is the default path for B-tree lookups.
2. The first entry in every rbyd block _may_ have a name entry, which
is ignored. I'm calling these "vestigial names" to make them sound
cooler than they actually are.
These vestigial names show up in a couple complicated B-tree
operations:
- During B-tree split, since pending attributes are calculated before
the split, we need to play out pending attributes into the rbyd
before deciding what name becomes the name of entry in the parent.
This creates a vestigial name which we _could_ immediately remove,
but the remove adds additional size to the must-fit split operation
- During B-tree pop/merge, if we remove the leading no-name entry,
the second, named entry becomes the leading entry. This creates a
vestigial name that _looks_ easy enough to remove when making the
pending attributes for pop/merge, but turns out the be surprisingly
tricky if the parent undergoes a split/merge at the same time.
It may be possible to remove all these vestigial names proactively,
but this adds additional rbyd lookups to figure out the exact tag to
remove, complicates things in a fragile way, and doesn't actually
reduce storage costs until the rbyd is compacted.
The main downside is that these B-trees may be a bit more confusing
to debug.
These benchmarks are now more useful for seeing how these B-trees perform.
In plot.py/plotmpl.py:
- Added --legend as another alias for -l, --legend-right.
- Allowed omitting of datasets from the legend by using empty strings
in --labels.
- Do not sum multiple data points on the same x coordinate. This was a
bad idea that risks invalid results going unnoticed.
As a plus multiple data points on the same x coordinate can be abused for
a cheap representation of measurement error.
This reworks test.py/bench.py a bit to map arguments to ids as a first
step instead of defering as much as possible. This is a better design
and avoids the hackiness around -b/-B. As a plus, test_id globbing is
easy to add.
I wondered if walking in Python 2's footsteps was going to run into the
same issues and sure enough, memory backed iterators became unweildy.
The motivation for this change is that large ranges in tests, such as
iterators over seeds or permutations, became prohibitively expensive to
compile. This meant more iteration moving into tests with more steps to
reproduce failures. This sort of defeats the purpuse of the test
framework.
The solution here is to move test permutation generation out of test.py
and into the test runner itself. The allows defines to generate their
values programmatically.
This does conflict with the test frameworks support of sets of explicit
permutations, but this is fixed by also moving these "permutation sets"
down into the test runner.
I guess it turns out the closer your representation matches your
implementation the better everythign works.
Additionally the define caching layer got a bit of tweaking. We can't
precalculate the defines because of mutual recursion, but we can
precalculate which define/permutation each define id maps to. This is
necessary as otherwise figuring out each define's define-specific
permutation would be prohibitively expensive.
B-tree remove/merge is the most annoying part of B-trees.
The implementation here follows the same ideas implemented in push/split:
1. Defer splits/merges until compaction.
2. Assume our split/merge will succeed and play it out into the rbyd.
3. On the first sign of failure, revert any unnecessary changes by
appending deletes.
4. Do all of this in a single commit to avoid issues with single-prog
blocks.
Mapping this onto B-tree merge, the condition that triggers merge is
when our rbyd is <1/4 the block_size after compaction, and the condition
that aborts a merge is when our rbyd is >1/2 the block_size, since that
would trigger a split on a later compact.
Weaving this into lfsr_btree_commit is a bit subtle, but relatively
straightforward all things considered.
One downside is it's not physically possible to try merging with both
siblings, so we have to choose just one to attempt a merge. We handle
the corner case of merging the last sibling in a block explicitly, and
in theory the other sibling will eventually trigger a merge during its
own compaction.
Extra annoying are the corner cases with merges in the root rbyd that
make the root rbyd degenerate. We really should avoid a compaction in
this case, as otherwise we would erase a block that we immediately
inline at a significant cost. However determining if our root rbyd is
degenerate is tricky. We can determine a degenerate root with children
by checking if our rbyd's weight matches the B-tree's weight when we
merge. But determining a degenerate root that is a leaf requires
manually looking up both children in lfsr_btree_pop to see if they will
result in a degenerate root. Ugh.
On the bright side, this does all seem to be working now. Which
completes the last of the core B-tree algorithms.
This was a rather simple exercise. lfsr_btree_commit does most of the
work already, so all this needed was setting up the pending attributes
correctly.
Also:
- Tweaked dbgrbyd.py's tree rendering to match dbgbtree.py's.
- Added a print to each B-tree test to help find the resulting B-tree
when debugging.
Changed so there is no 1-to-1 mk-tag/id assumption, any unique ids
create a simulated lifetime to render. This fixes the issue where
grows/shrinks left-aligned ids confused dbgrbyd.py.
As a plus, now dbgrbyd.py can actually handle multi-id grow/shrinks, and
is more robust against out-of-sync grow/shrinks. This sort of lifetime issues
are when you'd want to run dgbrbyd.py, so it's a bit important this is handled
gracefully.
An example:
$ ./scripts/dbgbtree.py -B4096 disk 0xaa -t -i
btree 0xaa.1000, rev 35, weight 278
block ids name tag data
(truncated)
00aa.1000: +-+ 0-16 branch id16 3 7e d4 10 ~..
007e.0854: | |-> 0 inlined id0 1 73 s
| |-> 1 inlined id1 1 74 t
| |-> 2 inlined id2 1 75 u
| |-> 3 inlined id3 1 76 v
| |-> 4 inlined id4 1 77 w
| |-> 5 inlined id5 1 78 x
| |-> 6 inlined id6 1 79 y
| |-> 7 inlined id7 1 7a z
| |-> 8 inlined id8 1 61 a
| |-> 9 inlined id9 1 62 b
...
This added the idea of block+limit addresses such as 0xaa.1000. Added
this as an option to dbgrbyd.py along with a couple other tweaks:
- Added block+limit support (0x<block>.<limit>).
- Fixed in-device representation indentation when trees are present.
- Changed fromtag to implicitly fixup ids/weights off-by-one-ness, this
is consistent with lfs.c.
This involves many, many hacks, but is enough to test the concept
and start looking at how it interacts with different block sizes.
Note only append (lfsr_btree_push on the end) is implemented, and it
makes some assumption about how the ids can interact when splitting
rbyds.
This implements a common B-tree using rbyd's as inner nodes.
Since our rbyds actually map to sorted arrays, this fits together quite
well.
The main caveat/concern is that we can't rely on strict knowledge on the
on-disk size of these things. This first shows up with B-tree insertion,
we can't split in preparation to insert as we descend down the tree.
Normally, this means our B-tree would require recursion in order to keep
track of each parent as we descend down our tree. However, we can
avoid this by not storing our parent, but by looking it up again on each
step of the splitting operation.
This brute-force-ish approach makes our algorithm tail-recursive, so
bounded RAM, but raises our runtime from O(logB(n)) to O(logB(n)^2)
That being said, O(logB(n)^2) is still sublinear, and, thanks to
B-tree's extremely high branching factor, may be insignificant.
The way sparse ids interact with our flat id+attr tree is a bit wonky.
Normally, with weighted trees, one entry is associated with one weight.
But since our rbyd trees use id+attr pairs as keys, in theory each set of
id+attr pairs should share a single weight.
+-+-+-+-> id0,attr0 -.
| | | '-> id0,attr1 +- weight 5
| | '-+-> id0,attr2 -'
| | |
| | '-> id5,attr0 -.
| '-+-+-> id5,attr1 +- weight 5
| | '-> id5,attr2 -'
| |
| '-+-> id10,attr0 -.
| '-> id10,attr1 +- weight 5
'-------> id10,attr2 -'
To make this representable, we could give a single id+attr pair the
weight, and make the other attrs have a weight of zero. In our current
scheme, attr0 (actually LFSR_TAG_MK) is the only attr required for every
id, and it has the benefit of being the first attr found during
traversal. So it is the obvious choice for storing the id's effective weight.
But there's still some trickiness. Keep in mind our ids are derived from
the weights in the rbyd tree. So if follow intuition and implement this naively:
+-+-+-+-> id0,attr0 weight 5
| | | '-> id5,attr1 weight 0
| | '-+-> id5,attr2 weight 0
| | |
| | '-> id5,attr0 weight 5
| '-+-+-> id10,attr1 weight 0
| | '-> id10,attr2 weight 0
| |
| '-+-> id10,attr0 weight 5
| '-> id15,attr1 weight 0
'-------> id15,attr2 weight 0
Suddenly the ids in the attr sets don't match!
It may be possible to work around this with special cases for attr0, but
this would complicate the code and make the presence of attr0 a strict
requirement.
Instead, if we associate each attr set with not the smallest id in the
weight but the largest id in the weight, so id' = id+(weight-1), then
our requirements work out while still keeping each attr set on the same
low-level id:
+-+-+-+-> id4,attr0 weight 5
| | | '-> id4,attr1 weight 0
| | '-+-> id4,attr2 weight 0
| | |
| | '-> id9,attr0 weight 5
| '-+-+-> id9,attr1 weight 0
| | '-> id9,attr2 weight 0
| |
| '-+-> id14,attr0 weight 5
| '-> id14,attr1 weight 0
'-------> id14,attr2 weight 0
To be blunt, this is unintuitive, and I'm worried it may be its own
source of complexity/bugs. But this representation does solve the problem
at hand, so I'm just going to see how it works out.
- Fixed off-by-one id for unknown tags.
- Allowed block_size and block to go unspecified, assumes the block
device is one big block in that case.
- Added --buffer and --ignore-errors to watch.py, making it a bit better
for watching slow and sometimes error scripts, such as dbgrbyd.py when
watching a block device under test.
Well not really fixed, more just added an assert to make sure
lfsr_rbyd_lookup is not called with tag 0. Because our alt tags only
encode less-than-or-equal and greater-than, which can be flipped
trivially, it's not possible to encode removal of tag 0 during deletes.
Fortunately, this tag should already not exist for other pragmatic
reasons, it was just used as the initial value for traversals, where it
could cause this bug.
If we combine rbyd ids and B-tree weights, we need 32-bit ids since this
will eventually need to cover the full range of a file. This simply
doesn't fit into a single word anymore, unless littlefs uses 64-bit tags.
Generally not a great idea for a filesystem targeting even 8-bit
microcontrollers.
So here is a tag encoding that uses 3 leb128 words. This will likely
have more code cost and slightly more disk usage (we can no longer fit
tags into 2 bytes), though with most tags being alt pointers (O(m log m)
vs O(m)), this may not be that significant.
Note that we try to keep tags limited to 14-bits to avoid an extra leb128 byte,
which would likely affect all alt pointers. To pull this off we do away
with the subtype/suptype distinction, limiting in-tree tag types to
10-bits encoded on a per-suptype basis:
in-tree tags:
ttttttt ttt00rv
^--^^- 10-bit type
'|- removed bit
'- valid bit
iiii iiiiiii iiiiiii iiiiiii iiiiiii
^- n-bit id
lllllll lllllll lllllll lllllll
^- m-bit length
out-of-tree tags:
ttttttt ttt010v
^---^- 10-bit type
'- valid bit
0000000
lllllll lllllll lllllll lllllll
^- m-bit length
alt tags:
kkkkkkk kkk1dcv
^-^^^- 10-bit key
'||- direction bit
'|- color bit
'- valid bit
wwww wwwwwww wwwwwww wwwwwww wwwwwww
^- n-bit weight
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- m-bit jump
The real pain is that with separate integers for id and tag, it no
longer makes sense to combine these into one big weight field. This
requires a significant rewrite.
- Added both uattr (limited to 256) and id (limited to 65535) benchmarks
covering the main rbyd operations
- Fixed issue where --defines gets passed to the test/bench runners when
querying id-specific information. After changing the test/bench
runners to prioritize explicit defines, this causes problems for
recorded benchmark results and debug related things.
- In plot.py/plotmpl.py, made --by/-x/-y in subplots behave somewhat
reasonably, contributing to a global dataset and the figure's legend,
colors, etc, but only shown in the specified subplot. This is useful
mainly for showing different -y values on different subplots.
- In plot.py/plotmpl.py, added --labels to allow explicit configuration
of legend labels, much like --colors/--formats/--chars/etc. This
removes one of the main annoying needs for modifying benchmark results.
I'm still not sure this is the best decision, since it may add some
complexity to tag parsing, but making most crcs one byte may be valuable
since these exist in every single commit.
This gives tags three high-level encodings:
in-tree tags:
iiiiiii iiiiitt ttTTTTT TTT00rv
^----^--------^--^^- 16-bit id
'--------|--||- 4-bit suptype
'--||- 8-bit subtype
'|- removed bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
out-of-tree tags:
------- -----TT TTTTTTt ttt01pv
^----^--^^- 8-bit subtype
'--||- 4-bit suptype
'|- perturb bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
alt tags:
wwwwwww wwwwwww wwwwwww www1dcv
^-^^^- 28-bit weight
'||- direction bit
'|- color bit
'- valid bit
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Having the location of the subtype flipped for crc tags vs tree tags is
unintuitive, but it makes more crc tags fit in a single byte, while
preserving expected tag ordering for tree tags.
The only case where crc tags don't fit in a single byte if is non-crc
checksums (sha256?) are added, at which point I expect the subtype to
indicate which checksum algorithm is in use.
$ ./scripts/dbgrbyd.py disk 4096 0 -t
mdir 0x0, rev 1, size 121
off tag data (truncated)
0000005e: +-+-+--> uattr 0x01 4 aa aa aa aa ....
0000000f: | | '--> uattr 0x02 4 aa aa aa aa ....
0000001d: | '----> uattr 0x03 4 aa aa aa aa ....
0000002d: | .----> uattr 0x04 4 aa aa aa aa ....
0000003d: | | .--> uattr 0x05 4 aa aa aa aa ....
0000004f: '-+-+-+> uattr 0x06 4 aa aa aa aa ....
00000004: '> uattr 0x07 4 aa aa aa aa ....
Unfortunately this tree can end up a bit confusing when alt pointers
live in unrelated search paths...
Toying around with the idea that since rbyd trees have strict height
gaurantees after compaction (2*log2(n)+1), we can proactively calculate
the maximum on-disk space required for a worst case tree+leb128
encoding.
This would _greatly_ simplify things such as metadata compaction and
splitting, and allow unstorable file metadata (too many custom
attributes) to error early.
One issue is that this calculated worst case will likely be ~4-5x worst
than the actual encoding due to leb128 compression. Though this may be an
acceptable tradeoff for the simplification and more reliable behavior.
Previously the subtype was encoded above the suptype. This was an issue
if you wanted to, say, traverse all tags in a given suptype.
I'm not sure yet if this sort of functionality is needed, it may be
useful for cleaning up/replacing classes of tags, such as file struct
tags, but not sure yet. At the very least is avoids unintuitive tag
ordering in the tree, which could potential cause problems for
create/deletes.
New encoding:
tags:
iiiiiii iiiiitt ttTTTTT TTT0trv
^----^--------^-^^^- 16-bit id
'--------|-'||- 5-bit suptype (split)
'--||- 8-bit subtype
'|- perturb/remove bit
'- valid bit
lllllll lllllll lllllll lllllll
^- n-bit length
alts:
wwwwwww wwwwwww wwwwwww www1dcv
^^^-^- 28-bit weight
'|-|- color bit
'-|- direction bit
'- valid bit
jjjjjjj jjjjjjj jjjjjjj jjjjjjj
^- n-bit jump
Also a large amount of name changes and other cleanup.
Christopher Haster