This helps with debugging and can avoid weird issues if a file btree
ever accidentally ends up attached to id -1 (due to fs bug).
Though a separate encoding isn't strictly necessary, maybe this should
be reverted at some point.
This replaces unr with null on disk, though note both the rm bit and unr
are used in-device still, they just don't get written to disk.
This removes the need for the rm bit on disk. Since we no longer need to
figure out what's been removed during fetch, we can save this bit for both
internal and future on-disk use.
Special handling of alta allows us to avoid emitting an unr tag (now null) if
the current trunk is truly unreachable. This is minor now, but important
for a theoretical rbyd rebalance operation (planned), which brings the
rbyd overhead down from ~3x to ~2x.
These changes give us two ways to terminate trunks without a tag:
1. With an alta, if the current trunk is unreachable:
altbgt 0x403 w0 0x7b
altbgt 0x402 w0 0x29
alta w0 0x4
2. With a null, if the current trunk is reachable, either for
code convenience or because emitting an alta is impossible (an empty
rbyd for example):
altbgt 0x403 w0 0x7b
altbgt 0x402 w0 0x29
altbgt 0x401 w0 0x4
null
Yet another tag encoding, but hopefully narrowing in on a good long term
design. This change trades a subtype bit for the ability to extend
subtypes indefinitely via leb128 in the future.
The immediate benefit is ~unlimited custom attributes, though I'm not
sure how to make this configurable yet. Extended custom attributes may
have a significant impact on alt tag sizes, so it may be worth
defaulting to only 8-bit custom attributes still.
Tag encoding:
vmmmtttt 0TTTTTTT 0wwwwwww 0sssssss
^--^---^--------^--------^--------^- valid bit
'---|--------|--------|--------|- 3-bit mode
'--------|--------|--------|- 4-bit suptype
'--------|--------|- leb128 subtype
'--------|- leb128 weight
'- leb128 size/jump
This limits subtypes to 7-bits, but this seems very reasonable at the
moment.
This also seems to limit custom attributes to 7-bits, but we can use two
separate suptypes to bring this back up to 8-bits. I was planning to do
this anyways to have separate "user-attributes" and "system-attributes",
so this actually fits in really well.
This helps debug a corrupted mtree with cycles, which has been a problem
in the past.
Also fixed a small rendering issue with dbgmtree.py not connecting inner
tree edges to mdir roots correctly during rendering.
Optimizing a script? This might sound premature, but the tree rendering
was, uh, quite slow for any decently sized (>1024) btree.
The main reason is that tree generation is quite hacky in places, repeatedly
spitting out multiple copies of the inner node's rbyd trees for example.
Rather than rewrite the tree generation implementation to be smarter,
this just changes all edge representations to namedtuples (which may
reduce memory pressure a bit), and collects them into a Python set.
This has the effect of deduplicating generated edges efficiently, and
improved the rendering performance significantly.
---
I also considered memoizing rbyd tree, but dropped the idea since the
current renderer performs well enough.
In addition to plugging in the rbyd and btree renderers in dbgbtree.py,
this required wiring in rbyd trees in the mdirs and mroots.
A bit tricky, but with a more-or-less straightforward implementation thanks
to the common edge description used for the tree renderer.
For example, a relatively small mtree:
$ ./scripts/dbgmtree.py disk -B4096 -t -i
mroot 0x{0,1}.45, rev 1, weight 0
mdir ids tag ...
{0000,0001}: .---------> -1 magic 8 ...
| .-------> config 21 ...
+-+-+ btree 7 ...
0006.000a: | .-+ 0 mdir w1 2 ...
{0002,0003}: | | '-> 0.0 inlined w1 1024 ...
0006.000a: '-+-+ 1 mdir w1 2 ...
{0004,0005}: '-> 1.0 inlined w1 1024 ...
This builds on dbgrbyd.py and dbgbtree.py by allowing for quick
debugging of the littlefs mtree, which is a btree of rbyd pairs with a
few bells and whistles.
This also comes with a number of tweaks to dbgrbyd.py and dbgbtree.py,
mostly changing rbyd addresses to support some more mdir friendly
formats.
The syntax for rbyd addresses is starting to converge into a couple
common patterns, which is nice for quickly determining what type of
address you are looking at at a glance:
- 0x12 => An rbyd at block 0x12
- 0x12.34 => An rbyd at block 0x12 with trunk 0x34
- 0x{12,34} => An rbyd at either block 0x12 or block 0x34 (an mdir)
- 0x{12,34}.56 => An rbyd at either block 0x12 or block 0x34 with trunk 0x56
These scripts have also been updated to support any number of blocks in
an rbyd address, for example 0x{12,34,56,78}. This is a bit of future
proofing. >2 blocks in mdirs may be explored in the future for the
increased redundancy.
Christopher Haster