Realistically, because our btree is protected by CoW checksums, the only
place we can end up with a cycle is in our mroot chain.
This is convenient, as we don't need our btree traversal state when
traversing the mroot chain, so we can put both the tortoise state and
btree traversal state into a union, theoretically saving some RAM.
Unfortunately stack measurements show no change, even though our mtree
traversal in on the hot path. I'm not sure why this is. My best guess is
that the RAM savings is beneath the compilation noise floor, since we
currently only ever create one of these structs.
Composable parsing functions always feel a bit weird to me in C. I don't
know if this is because of something C lacks, such as multiple return
values, or if composable parsers are just inherently awkward to describe
in procedural languages because of the different levels of state.
But I think the API here is pretty ok. The main idea is that data
parsers can be added as functions in the lfsr_data_* namespace that take
lfsr_data_t as a mutable reference, updating the lfsr_data_t's internal
state as data is parsed.
In practice you only need a couple of primitives, bytes, le32s, leb128s,
that touch the internals of lfsr_data_t, and the other parsers can be
built using these.
This leverages the pointer-like abstraction of lfsr_data_t, and avoids
needing to keep track of offsets. And thanks to lfsr_data_t being
relatively cheap to make copies, this API is relatively flexible.
Some other tweaks:
- Signed leb128 overflow detection is moved up into lfs_fromleb128.
littlefs now assumes _all_ leb128s are 31-bits, which is useful for
leveraging the sign bit internally.
This also fixes the an issue in overflow detection in lfs_fromleb128
which wouldn't catch overflows in the last byte of a >32-bit leb128.
- Most lfsr_data_t functions now take a pointer. This offered a small
bit of code savings and feels more natural in C. Though most functions
that accept lfsr_data_t still take a copy. Most of these functions
would need to make a copy anyways now that the parsers are consuming,
and these copies avoid concerns about shared state.
At 3-words, lfsr_data_t is right at that boundary of questionable
reasonableness for copying, but copying is a very useful feature of
this struct.
This ends up with some decent code/stack savings:
code stack
before: 22118 2048
after: 21722 (-1.8%) 1992 (-2.7%)
Generally the more creative you get with C macros, the more
unmaintainable your codebase becomes, but in this case I think a small
bit of macro sugar for the attribute lists in littlefs goes a long way
for making the internals flexible and readable.
Attribute lists generally look like this:
LFSR_ATTRS(
LFSR_ATTR(id, TAG, delta, DATA(data)),
LFSR_ATTR(id, TAG, delta, DATA(data)),
...
LFSR_ATTR(id, TAG, delta, DATA(data)))
Which more-or-less gets expanded to this:
((const lfsr_attr_t[]){
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),
...
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),}),
attr_count
Note the use of preprocessor concatenation to put the TAG and DATA
identifiers in their respective namespaces. These can end up invoking
other macros, which allows attrs to be rather extensible.
Previously there were also LFSR_ATTR_ (note the trailing underscore)
macros to allow passing of variable tags/datas. This is replaced with
redundant macros which sort of "unwrap" themselves as a part of macro
expansion. This avoids a bunch of duplicate macro definitions.
#define LFSR_TAG_TAG(tag) (tag)
#define LFSR_DATA_DATA(data) (data)
So:
LFSR_ATTR(id, TAG(tag), delta, DATA(data))
Becomes:
((lfsr_attr_t){id, LFSR_TAG_TAG(tag), delta, LFSR_DATA_DATA(data)})
Becomes:
((lfsr_attr_t){id, tag, delta, data})
This checksum is used to keep track of if we have erased, and not yet
touched, the unused bytes trailing our current commit in the rbyd.
The working theory is that if any prog attempt is made, it will, most
likely, change the checksum of the contents, allowing littlefs to
determine if trailing erased-state is safe to use, even under powerloss.
littlefs can also perturb future data by a single bit, to force this
checksum to always be invalidated during normal operation.
The original name, "forward erased-state checksums (fcksum)", came from the
idea that the checksum "looks forward" into the next commit.
But after using them for a bit, I think the name is unnecessarily
confusing. It, uh, also looks a lot like a swear word. I think
shortening the name to just "erased-state checksums (ecksum)", even
though the previous name is already in use in a release, is reasonable.
---
It's probably hard to believe but the name change from fcrc -> ecrc
really was unrelated to the crc -> cksum change. But boy is it
convenient for avoiding an awkward name. A lot of these name changes
involved sed scripts, so I didn't notice how awkward fcksum would be to
use until writing this commit message.
For a couple reasons:
1. Organizing the overlaps this way avoid potential undefined behavior.
It turns out C does define the overlap the "initial sequence" of
union members, as long as the types are the same. But when we
overlapped the block with the size/tag fields in lfsr_btree_t, it was
probably undefined behavior.
At the very least, it would introduce a need for quite a bit of
preprocessing to make it work with different integer sizes and
redundancy levels.
2. Overlapping the blocks at the end of the rbyd struct means our block
array is natural ordered such that the first block is the "active"
block, i.e. the block with the most recent revision count that passes
checksums.
This has been useful as a debugging tool, so I would like to continue
the pattern. It is possible to mostly preserve this order with the
previous method by intentional reversing the block array when
logging or writing to disk, but it's a bit cumbersome.
2. It's unlikely we'll be able to use readonly variants of the rbyd/mdir
structs for RAM savings. Unfortunately C makes this too cumbersome.
Though if we do this should be revisited.
Here are the new overlaps. Note it's no longer possible to truncate the
types when readonly. If readonly struct are useful this will need to be
revisited again:
lfsr_rbyd_t lfsr_btree_t lfsr_mdir_t
8b 8b 8b 8b
.----+----+----+----.
8b 8b 8b 8b 8b 8b 8b 8b | mid.bid | mid.rid |
.----+----+----+----. .----+----+----+----. |----+----+----+----|
| weight |.>| weight | | weight |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| trunk | | tag | size | | trunk |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| off | | inlined data | | off |
|----+----+----+----| | | | |----+----+----+----|
| crc | | v | | crc |
|----+----+----+----| | | |----+----+----+----|
| block |..| |.>| blocks |
'----+----+----+----' '----+----+----+----' | |
| |
'----+----+----+----'
This turned out to be tricky.
At littlefs's core, we have the lfsr_rbyd_t struct. It is really
important this is as small as possible since littlefs creates many rbyd
copies in order to track state of metadata on disk.
Wrapping rbyd, we have the lfsr_btree_t struct, which can alternatively
contain a single inlined entry, accomplished by overlapping the width
field in both cases. And the lfsr_mdir_t struct, which tracks any redundant
blocks, and would be nice if the blocks lined up as neighbors so all blocks
involved in the mdir could be passed around as an array. Both of these
wrappers attempt to overlap fields of the lfsr_rbyd_t struct, which presents
a bit of a problem.
The solution here is to put the rbyd block field at the beginning of the
lfsr_rbyd_t struct, and use exactly 32-bits of padding in lfsr_btree_t
to overlap the width field even though it is not at the beginning of the
struct. To avoid inflating the lfsr_btree_t size, we sneak the inlined
size and tag into the overlapping padding. This will need special
handling if the size of these fields change, but saves a decent amount
of RAM:
lfsr_rbyd_t lfsr_btree_t lfsr_mdir_t
8b 8b 8b 8b
.----+----+----+----.
| mid.bid | mid.rid |
|----+----+----+----|
8b 8b 8b 8b 8b 8b 8b 8b | blocks |
.----+----+----+----. .----+----+----+----. | |
| block |..| tag |size|padd|.>| |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| weight |.>| weight | | weight |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| trunk | | inlined data | | trunk |
|----+----+----+----| | | | |----+----+----+----|
| off | | v | | off |
|----+----+----+----| | | |----+----+----+----|
| crc | | | | crc |
'----+----+----+----' '----+----+----+----' '----+----+----+----'
Also tried to reduce the amount of mdir usage in lfsr_mdir_commit by
better using only the arrays of relevant mdir blocks, to limited success.
The previous system of relying on test name prefixes for ordering was
simple, but organizing tests by dependencies and topologically sorting
during compilation is 1. more flexible and 2. simplifies test names,
which get typed a lot.
Note these are not "hard" dependencies, each test suite should work fine
in isolation. These "after" dependencies just hint an ordering when all
tests are ran.
As such, it's worth noting the tests should NOT error of a dependency is
missing. This unfortunately makes it a bit hard to catch typos, but
allows faster compilation of a subset of tests.
---
To make this work the way tests are linked has changed from using custom
linker section (fun linker magic!) to a weakly linked array appended to
every source file (also fun linker magic!).
At least with this method test.py has strict control over the test
ordering, and doesn't depend on 1. the order in which the linker merges
sections, and 2. the order tests are passed to test.py. I didn't realize
the previous system was so fragile.
It doesn't make sense to test more complex logic, such as t2_btree.toml,
when the logic it is built on, t1_rbyd.toml, does not past testing. The
test runner already guarantees a consistent lexicographic order, so all
we need to do is renamed these from test_* -> tn_*.
Note, if we every have more than 10 tests, we will need to bump up the
number of digits for all tests, so t1_rbyd.toml -> t01_rbyd.toml. This
is the main downside of lexicographic ordering. But we'll cross that
bridge when we get to it.
Took the opportunity to make some allocator tweaks:
- Renamed lfs.free -> lfs.lookahead, it's previous name did cause some
confusion.
- Renamed lfs.free.off -> lfs.lookahead.start
- Renamed lfs.free.i -> lfs.lookahead.next
- Renamed lfs.free.ack -> lfs.lookahead.acked
- Changed bitmap from using 32-bit words to using 8-bit bytes, dropping
the alignment requirement. One of the reasons for 32-bit alignment was
an attempt at future proofing for some sort of free-list.
This never landed, and if it did, it could have been provided without
breaking backwards compatiblity via an additional config option, at a
minor RAM cost.
We never used ffs/clz instructions for this bitmap, so I don't think
using 32-bit words offers much advantage. It just creates another
potential issue for users if their lookahead buffer is unaligned.
These changes should probably also be upstreamed to the current version.
They don't depend on anything rbyd specific.
Note, at some point lfs_alloc will need to be extended to mark block tags,
etc, as in-use during traversal.
- Renamed mpair -> mptr, may have >2 blocks in the future.
- Renamed branch -> bptr for consistency.
- Renamed other_block -> redund_rbyd.
- Changed comparison functions to use -1, 0, +1, even for unordered
types.
- Added lfs_cmp function for unioning comparisons with signed errors.
This isn't actually for performance reasons, but to reduce storage
overhead of the rbyd metadata tree, which was showing signs of being
problematic for small block sizes.
Originally, the plan for compaction was to rely on the self-balancing
rbyd append algorithm and simply append each tag to a new tree.
Unfortunately, since each append requires a rewrite of the trunk
(current search path), this introduces ~n*log(n) alts but only uses ~n alts
for the final tree. This really starts to put pressure on small blocks,
where the exponential-ness of the log doesn't kick in and overhead
limits are already tight.
Measuring lfsr_mdir_commit code size, this shows a ~556 byte cost on
thumb: 16416 -> 16972 (+3.4%). Though there are still some optimizations
on the table, this implementation needs a cleanup pass.
alt overhead code cost
rebalance: <= 28*n 16972
append: <= 24*n*log(n) 16416
Note these all assume worst case alt overhead, but we _need_ to assume
worst case for our rbyd estimations, or else the filesystem can get
stuck in unrecoverable compaction states.
Because of the code cost I'm not sure if rebalancing will stay, be
optional, or replace append-compaction completely yet.
Some implementation notes:
- Most tree balancing algorithms rely on true recursion, I suspect
recursion may be a hard requirement in general, but it's hard to find
bounded-ram algorithms.
This solution gets around the ram requirement by leveraging the fact
that our tags exist in a log to build up each layer in the tree
tail-recursively. It's interesting to note that this is a special
case of having little ram but lots of storage.
- Humorously this shouldn't result in a performance improvement. Rbyd
trees result in a worst case 2*log(n) height, and rebalancing gives us
a perfect worst case log(n) height, but, since we need an additional
alt pointer for each node in our tree, things bump back up to 2*log(n).
- Originally the plan was to terminate each node with an alt-always tag,
but during implementation I realized there was no easy way to get the
key that splits the children with awkward tree lookups. As a
workaround each node is terminated with an altle tag that contains the
key followed by an unreachable null tag. This is redundant information,
but makes the algorithm easier to implement.
Fortunately null tags use the smallest tag encoding, which isn't that
small, but that means this wastes at most 4*n bytes.
- Note this preserves the first-tag-always-ends-up-at-off=0x4 rule, which
is necessary for the littlefs magic to end up in a consistent place.
- I've dropped dropping vestigial names for now, which means vestigial
names can remain in btrees indefinitely. Need to revisit this.
- Finally figured out how to test multiple mroot extensions without an
allocator, though hopefully forcing PROG_SIZE doesn't break test
framework things at some point...
- Added tests that magic string is always in the same place. This isn't
strictly required for littlefs to work, but is a nice feature to have.
Of course, the new tests found a bug, but it was in a surprisingly
place. Accidentally allowed the revision count to be uninitialized when
compacting the mroot. At least there's a test that covers this now.
This only affects the in-device tags, not the on-disk tags.
The mk variant of tags was seeing much more use than the grow variant,
since the grow variant is really only used by the btree internals. But
since the default encoding of tags cleared the mk-bit, this led to a
bunch of extra lfsr_tag_setmk calls just to reserialize things correctly
during compact, split, etc.
Flipping the logic so the bit needs to be set to grow tags simplified
things quite a bit.
Note that mk tags do nothing when their delta is zero, so zero-delta
tags are the same in both mk/grow mode.
I intended to also add a test for cycles in the btree that backs the
mtree (and eventually other btrees), but something really curious
happened.
It turns out it's actually really hard to create a btree cycle, even
intentionally.
This is because each CoW btree pointer includes the expected CRC of
the branch's rbyd. To succesfully create a cycle that isn't trivially
detected in a validating mtree traversal, you would somehow need to
solve for a cyclic set of dependent CRCs that are still valid.
I suspect this is slightly easier than a hash-based construction, due to
the linear nature of CRCs, but still I think it's unreasonable to expect
these sort of cycles to occur in the wild. Even with filesystem bugs.
---
Note this isn't true for the mdirs, which are mutable so storing a
checksum in the pointer isn't possible. For this reason, cycle detection
is kept for mdirs during mtree traversal. This may not be strictly
necessary for the mtree, but it needed for the mroot chain.
Nonetheless, this does simplify things. Specifically it reduces the
cycle detection's tortoise state to only mdir pairs.
Validating btree nodes during lfsr_btree_lookup was useful as a
proof-of-concept, but it's not really needed if we validate btree nodes
during mtree traversal.
mtree traversal provides the first reads into the filesystem. It's how
we find the real mroot, and (in theory at the moment) it provides the core
operation for error detection in correction. With this in mind,
implementing btree node validation in mtree traversal makes a lot of
sense, with lfsr_btree_lookup leveraging an assumed successful
validation for faster/smaller btree walks.
Note that btree node validation during traversal is still optional. We
really don't want to pay this cost during block allocation for example.
---
It may look concerning that there's no related validation in btree traversal
layer itself.
It turns out that a quirk of btree traversal returning inner btree nodes on
first visit, before actually traversing the btree node, is that it's
safe for us to validte the btree node in only the mtree traversal layer.
As long as we don't continue traversing on finding a corrupted btree,
the btree traversal layer will never traverse an unvalidated btree node.
This keeps all the validation logic in the same place, mtree traversal.
I don't know if this will stay this way if/when more error correction
features are added, but it's convenient in the meantime.
Just like lfsr_btree_traversal_t, lfsr_mtree_traversal_t provides a
mechanism for traversing the mtree incrementally, including any inner
btree nodes.
This is one level more complex than btree traversal because we also need
to handle the mroot chain and traversal of rids in each mdir.
Again, mtree traversal returns temporary decoded rbyd structs for inner
nodes. Actually, mtree traversal only returns inner nodes... so maybe
using lfsr_data_t here is the wrong choice:
- tag=LFSR_TAG_BTREE => lfsr_rbyd_t
- tag=LFSR_TAG_MDIR => lfsr_mdir_t
littlefs uses an invasive linked-list in open mdirs to keep any open
files/dirs (and some special mdirs) in sync during filesystem
operations. The main benefit of this is that the filesystem doesn't need
to know the number of open files at compile time.
The implementation here introduces a new type, lfsr_openedmdir_t, for
mdirs that want to participate in the opened-mdir linked-list. This
saves a couple words of memory in the cases where the mdir does not need
to participate in the opend-mdir linked-list.
Since we are creating quite a few more mdir structs in lfsr_mdir_commit now,
the size of this struct is valuable.
The implementation of lfsr_mdir_commit knew this was coming, so aside
from the new type, adding this feature was straightforward:
1. Update opened-mdirs based on in-flight attrs.
2. Update opened-mdirs rbyd state.
3. Mark any deleted opened-mdirs with the reserved mid -2.
4. Test.
It's interesting to note the different performance characteristics of
purely CoW btrees vs our mutable mtree.
The main downside of our mtree is the need to fetch leaf mdirs. This
fetch is expensive, and can be avoided in CoW btrees by storing the
trunk in each branch's parent.
On the other hand, btrees need to propagate all changes upwards to the
root.
An interesting takeaway is that a sort of mdir-trunk cache may be a very
interesting optimization for relatively little RAM cost. This may be
something to explore in the future.
- lfsr_btree_isnull still used tag and not only weight for null trees
- relocation forgot the mid
- missed relocation when uninlining, though this fix should be cleaned up
- made revision count behavior a bit more consistent
Note that the new tests may be -Gnor exclusive, they rely quite a bit on
exactly when compaction happens...
lfsr_mdir_commit => lfsr_mdir_commit
|-> lfsr_mdir_commit_
'-> lfsr_mdir_compact_
The mess that was lfsr_mdir_commit was a growing problem. Flattening all
possible mdir operations into a single loop may have resulted in a
smaller code size, but at a significant cost to implementation
difficult, readability, bugs, etc.
This restructure splits the mdir commit logic into three components:
1. lfsr_mdir_compact_
This handles the swapping of mdir blocks, revision counts, erasing, etc.
lfsr_mdir_compact_ also accepts a range of ids, allowing it to be
called directly for mdir splitting/uninlining.
Actually, the biggest feature in lfsr_mdir_compact_, which is easy to
overlook, is that is accepts two attr lists. This seems like a weird
feature for an API, but keep in mind we have strict RAM limitations,
so we can't really concatenate attr lists easily.
There is only a single case we need two attr lists: When uninlining
an mroot we need to include 1. any pending mroot attrs, and 2. the
new mtree. But one case is enough to make attempted workarounds
excessively complicated.
Simply accepting two attr lists here resolves this.
2. lfsr_mdir_commit_
This handles the low-level mdir commit logic: It tries to do a simple
rbyd commit, and if that fails falls back to a compact/relocate loop.
Perhaps surprisingly, lfsr_mdir_commit_ does not handle mdir splits.
The exact behavior of mdir splits is context specific, so
lfsr_mdir_commit_ simple errors if lfsr_rbyd_estimate indicates
compaction will be unsuccessful.
Less surprisingly, lfsr_mdir_commit_ does not handle any
mtree/internal state updates. lfsr_mdir_commit_ is only concerned
with the specific mdir struct provided.
3. lfsr_mdir_commit
This ties together all of the mdir commit logic and provides the main
mechanism by which the rest of the filesystem interacts with mdirs.
lfsr_mdir_commit is mainly responsible for handling the side-effects
of the low-level operations:
- Propagating mtree/mroot updates caused by relocations/splits/drops
- Updating the provided mdir struct correctly if it splits/relocates
based on a rid hint
- Updating the internally tracked mroot/mtree state on success
- Updating any open mdirs on success (TODO)
This is a complicated function, but most of that complexity can be
captured in a large, but relatively simple, tree of if statements.
Not great for code cost, but this may just be a necessity of the new
mtree data-structure.
This also includes the tail-recursive mroot propagation loop, which
is an excellent example of how splitting the high/low-level logic
helps separate context-specific logic.
This still needs work, but the significantly improved readability of
lfsr_mdir_commit provides much more confidence in this design.
This already has the strong advantage that the extra mdir copies make it
clear when exactly the higher-level mdir copies are updated. This gives
us much better confidence that errors will not render the mdir state
unusable, though may be coming with a RAM cost.
Dropped the high-level "large entry" tests in exchange for these low-level
tests. The high-level tests accomplished the same thing, but worse and
less reliably.
Added some rough fixes (this whole code path needs to be rewritten).
Also made lfsr_rbyd_bisect a bit better behaved when dealing with a
small number of large entries. This was necessary for the split/drop
corner case tests since these rely on precise control of when mdirs
split.
mdirs behave a bit differently than btree nodes here. When an mdir's
weight drops to zero, we eagerly drop the mdir. Unfortunately this
introduce a large number of conditions into lfsr_mdir_commit. Maybe
there's some different way to structure to code to avoid this...
Also expanded mtree tests to cover more corner cases, these are
desperately for any confidence that mdir drops work.
This isn't the greatest coverage as we don't have a verifiable simulation.
Simulating the splitting-bucket-tree that is the mtree is tricky.
So right now this mostly just checks there's no internal assert failures and
if we have the expected number of entries afterwards.
Currently relying on lfsr_rbyd_append/appendattrs to inject extra
attributes during lfsr_mdir_commit, need to consider if this is really
the best solution. This probably results in more function calls than we
really need.
This became surprisingly tricky.
The main issue is knowing when to split mdirs, and how to determine
this without wasting erase cycles.
Unlike splitting btree nodes, we can't salvage failed compacts here. As
soon as the salvage commit is written to disk, the commit becomes immediately
visibile to the filesystem because it still exists in the mtree. This is
a problem if we lose power.
We're likely going to need to implement rbyd estimates. This is
something I hoped to avoid because it brings in quite a bit of
complexity and might lead to an annoying amount of storage waste since
our estimates will need to be conservative to avoid unrecoverable
situations.
---
Also changed the on-disk btree/branch struct to store a copy of the weight.
This was already required for the root of the btree, requiring the
weight to be stored in every btree pointer allows better code
deduplication at the cost of some redundancy on btree branches, where
the weight is already implied by the rbyd structure.
This weight is usually a single byte for most branches anyways.
This may be worth revisiting at some point to see if there's any other
unexpected tradeoffs.
Christopher Haster