We already get the leaf rbyd as a part of btree lookup, and since ids
can't be split across rbyd boundaries, we can be sure any bptr attrs
live in the same rbyd.
This can be extended to any future bptr attrs.
Aside from the small performance gain, this also means we can drop the
btree bid+tag lookups. All extra attr lookups to lookup the rbyd first.
This saves a bit of code but also avoids a set of issues with the btree
semantics where lookupnexting an extra attr can return ENOENT
prematurely when on an rbyd boundary.
As I'm typing this I realize this means we have no way to iterate over
all _tags_ in a btree, only over all _bids_. Fortunately I don't think
we will ever need the former.
code stack
before: 32136 2880
after: 31956 (-0.6%) 2880 (+0.0%)
Much like the erased-state checksums in our rbyds (ecksums), these
block-level erased-state checksums (becksums) allow us to detect failed
progs to erased parts of a block and are key to achieving efficient
incremental write performance with large blocks and frequent power
cycles/open-close cycles.
These are also key to achieving _reasonable_ write performance for
simple writes (linear, non-overwriting), since littlefs now relies
solely on becksums to efficiently append to blocks.
Though I suppose the previous block staging logic used with the CTZ
skip-list could be brought back to make becksums optional and avoid
btree lookups during simple writes (we do a _lot_ of btree
lookups)... I'll leave this open as a future optimization...
Unlike in-rbyd ecksums, becksums need to be stored out-of-band so our
data blocks only contain raw data. Since they are optional, an
additional tag in the file's btree makes sense.
Becksums are relatively simple, but they bring some challenges:
1. Adding becksums to file btrees is the first case we have for multiple
struct tags per btree id.
This isn't too complicated a problem, but requires some new internal
btree APIs.
Looking forward, which I probably shouldn't be doing this often,
multiple struct tags will also be useful for parity and content ids
as a part of data redundancy and data deduplication, though I think
it's uncontroversial to consider this both heavier-weight features...
2. Becksums only work if unfilled blocks are aligned to the prog_size.
This is the whole point of crystal_size -- to provide temporary
storage for unaligned writes -- but actually aligning the block
during writes turns out to be a bit tricky without a bunch of
unecesssary btree lookups (we already do too many btree lookups!).
The current implementation here discards the pcache to force
alignment, taking advantage of the requirement that
cache_size >= prog_size, but this is corrupting our block checksums.
Code cost:
code stack
before: 31248 2792
after: 32060 (+2.5%) 2864 (+2.5%)
Also lfsr_ftree_flush needs work. I'm usually open to gotos in C when
they improve internal logic, but even for me, the multiple goto jumps
from every left-neighbor lookup into the block writing loop is a bit
much...
Looking forward, bptr checksums provide an easy mechanism to validate
data residing in blocks. This extends the merkle-tree-like nature of the
filesystem all the way down to the data level, and is common in other
COW filesystems.
Two interesting things to note:
1. We don't actually check data-level checksums yet, but we do calculate
data-level checksums unconditionally.
Writing checksums is easy, but validating checksums is a bit more
tricky. This is made a bit harder for littlefs, since we can't hold
an entire block of data in RAM, so we have to choose between separate
bus transactions for checksum + data reads, or extremely expensive
overreads every read.
Note this already exists at the metadata-level, the separate bus
transactions for rbyd fetch + rbyd lookup means we _are_ susceptible
to a very small window where bit errors can get through.
But anyways, writing checksums is easy. And has basically no cost
since we are already processing the data for our write. So we might
as well write the data-level checksums at all times, even if we
aren't validating at the data-level.
2. To make bptr checksums work cheaply we need an additional cksize
field to indicate how much data is checksummed.
This field seems redundant when we already have the bptr's data size,
but if we didn't have this field, we would be forced to recalculate
the checksum every time a block is sliced. This would be
unreasonable.
The immutable cksize field does mean we may be checksumming more data
than we need to when validating, but we should be avoiding small
block slices anyways for storage cost reasons.
This does add some stack cost because our bptr struct is larger now:
code stack
before: 31200 2768
after: 31272 (+0.2%) 2800 (+1.1%)
Fortunately these operations are heavily tested in test_dirs. The only
difference with files is the possibility for shrubs to need to be
copied.
Bugs fixed:
- It's counterintuitive, but lfsr_rbyd_appendcompactattr _can_ error
with LFS_ERR_RANGE when we are copying a shrub. This can happen if the
underlying mdir needs compaction itself.
- It's possible to null-trunk bshrubs to appear in our filesystem
traversal. Null-trunk bshrubs don't usually appear in any stable
state, but they are created by lfsr_bshrub_alloc and lfsr_btree_commit
to represent new, yet-uncommitted shrubs.
This gets a bit tricky because we also use null-trunks to indicate if
lfsr_btree_traversal has traversed the root. We can't rely on
bid >= weight for this because zero-weight btrees are allowed.
The solution here, though maybe temporary (famous last words), is to
treat null-trunk btrees as not having a root. Which isn't really true,
but null-trunk btree roots only exist between allocator checkpoints,
so they are allowed to be unreachable.
We really need more asserts that this is the case though... At least
added an assert that we never commit/read null trunks on disk.
POSIX is notoriously full of subtle and confusing nuances. Not through
any fault of POSIX, but as a result of trying to describe a complex
system with simple and easy to use operations.
Corner cases fixed here:
- rename("dir", "file") => ENOTDIR
This is the main surprise to me, and a mistake on my part. I thought
EISDIR would be appropriate for any renames with mismatched types,
since both involve a directory. It would be simpler code-wise, and
avoid ambiguity around if "file" is not a dir, or some other file
exists in the file's path. But I guess ENOTDIR makes more sense if you
think of the destination as the target being operated on.
- remove("/") => EINVAL
- rename("/", "x") => EINVAL
- rename("x", "/") => ENOTEMPTY
- open("/") => EISDIR
It's a bit difficult to lookup what error codes around root operations
should be, since they mostly end up as EPERM on modern systems, but
this doesn't really make sense for littlefs.
The solution chosen here is to prefer directory-related errors (EISDIR,
ENOTEMPTY) when possible, and fall back to EINVAL when the only issue
is that the target is the root directory.
Also I tweaked lfsr_mtree_pathlookup a bit so mid=0 indicates the target
is the root and mid=-1 indicates the target can't be created (because of
a missing directory). I think using mid=0 for the latter is a leftover
from when mid=-1 was a bit of a mess...
It turns out permanent root bookmark creates some rather interesting
constraints on our mtree:
1. We can never delete all mids, since at least one mid needs to exist
to represent the root's bookmark.
2. We can never revert to an inlined mdir after uninlining, since our
root bookmark always exists to stop this. This is an unfortunate
downside as it would be nice to be able to reinline mdirs, but not
the end of the world.
This restricts what operations are possible, and transitively, what we
can test.
This commit drops the removal of root bookmarks in test_mtree, which was
a workaround to keep tests from early implementation running. This was
preventing some minor optimizations. This required dropping some tests,
but these tests tested operations that aren't really possible in
practice.
Dropping the removal of root bookmarks allowed for a minor optimization
in lfsr_mdir_drop, and may lead to more in the future (or maybe just
stricter asserts):
code stack
before: 31280 2648
after: 31208 (-0.2%) 2648 (+0.0%)
This ended up being much less of a simplification than I hoped it would.
It's still easier/more efficient to revert to a relocation in most cases
when dropping in an mdir split, and the small gain from simplifying how
drops/commits interact is overshadowed by the code duplication necessary
to separate lfsr_mdir_drop out from lfsr_mdir_commit:
code stack
before: 30952 2528
after: 31280 (+1.1%) 2648 (+4.7%)
Still, this does at least simplify the logical corner cases (we don't
need to abort commits when droppable anymore), and lfsr_mdir_drop is
ultimately necessary for supporting lazy file creation.
Also having a fix-orphans step during mount allows other littlefs
implementations the option to create orphanned mdirs without compat
issues. So this ends up the more flexible approach.
It _might_ be worth having both eager mdir drops and an explicit
lfsr_mdir_drop for lazy file creation in the future, but I doubt this
will end up worth the code duplication...
---
Oh right, I forgot to actually describe this change.
This trades eager mdir drops:
1. Drop mdirs from the mtree immediately as soon as their weight goes
to zero.
For lazy mdir drops:
1. Drop mdirs from the mtree in a second commit.
2. Scan and drop orphaned mdirs on the first write after mount.
This sounds very similar to the previous "deorphan" scan, which risked
an extreme performance cost during mount, but it should be noted this
orphan scan only needs to touch every mdir once. This makes it no worse
than the overhead of actually mounting the filesystem.
We can also keep an eye out for orphaned mdirs when we mount, so no
extra scan is needed unless there was an unlucky powerloss.
Eager mdir dropping sounds simpler, but thanks to deferred commits
introduces some subtle complexity around aborting commits that would
drop an mdir to zero. Remember commits are viewable on-disk as soon as a
commit completes.
In _theory_, lazy mdir drops simplify the logic around committing to
mdirs.
Though the real kicker is that lazy mdir drops are required for lazy file
creation.
The current idea for lazy file creation involves tracking mid-less
opened-but-not-yet-created files. These files can have bshrubs, so they
need space on an mdir somewhere. But they aren't actually created yet,
so they don't have an mid.
This is fine (though it's probably going to be tricky) as long as we
allocate an mid on file sync, but there is always a risk of losing power
with mdirs that contain only RAM-backed files. Fortunately, no-mids
means no orphaned files, but it does mean orphaned mdirs with no synced
contents.
Long story short, lazy mdir drops are currently a necessary evil, and
logical simplification, that unfortunately comes with some cost.
The idea here is to revert moving redund blocks into lfsr_rbyd_t, and
instead just keep a redundant copy of the rbyd blocks in the redund
blocks in lfsr_mdir_t.
Surprisingly, extra overhead in lfsr_mdir_t ended up with worse stack
usage than extra overhead in lfsr_rbyd_t. I guess we end up allocated
more mdirs than rbyds, which makes a bit of sense given how complicated
lfsr_mdir_commit is:
code stack structs
redund union: 30976 2496 1072
redund in rbyd: 30948 (-0.1%) 2528 (+1.3%) 1100 (+2.6%)
redund in mdir: 31000 (+0.1%) 2536 (+1.6%) 1092 (+1.8%)
The mdir option does seem to improve struct overhead, but this hasn't
been a reliable measurement since it doesn't take into account how many
of each struct is allocated.
Given that the mdir option is inferior in both code and stack cost, and
requires more care to keep the rbyd/redund blocks in sync, I think I'm
going to revert this for now but keep the commit in the commit history
since it's an interesting comparison.
This simplifies dependent structs with redundancy, mainly lfsr_mdir_t,
at a significant RAM cost:
code stack structs
before: 30976 2496 1072
after: 30948 (-0.1%) 2528 (+1.3%) 1100 (+2.6%)
Which, to be honest, is not as bad as I thought it would be. Though it
is still pretty bad for no new features.
The motivation for this change:
1. The organization of the previous lfsr_mdir_t struct was a bit hacky
and relied on exact padding so the redund block array and rbyd block
lined up at the right offset.
2. The previous organization prevented theoretical "read-only rbyd
structs" that could omit write-related fields, e.g. eoff and cksum.
This idea is currently unused.
3. The current mdir=level-1, btree/data=level-0 redund design makes this
RAM tradeoff pretty bad, but in theory higher btree redund levels
would need the extra redund blocks in the rbyd struct anyways.
Still, the RAM impact to the current default configuration means this
should probably be reverted...
- Renamed mdir->u.m to mdir->u.mdir.
- Prefer mdir->u.rbyd.* where possible.
- Changed file/dir mdirs to be stored directly, requiring a cast to
lfsr_openedmdir_t to enroll in the opened mdir list.
Reverted to one set of signed lfsr_mid_rid/bid functions, and tried to
make their usage more consistent.
We have two ways to compare mdirs now, lfsr_mdir_cmp (compares block
addresses) and lfsr_mdir_bid (compares mids), and it's not very clear
when to use which one. lfsr_mdir_cmp is a bit more robust in weird mid
cases (mainly inlined mdirs when mroot mid=-1), so currently preferring
that.
Also did some bit twiddling to preserve mid=-1 => bid=-1 and rid=-1,
this save a bit of code:
code stack
before: 31056 2488
after: 30972 (-0.3%) 2496 (+0.3%)
There's only one mtree in a given filesystem. With the recent
lfsr_mdir_commit restructure, it makes more sense for the mtree to be
implicit.
code stack
before: 31096 2480
after: 31016 (-0.3%) 2480 (+0.0%)
Originally, the intention of this rework was to make it possible to
shrub the mtree, i.e. allow an mshrub, i.e. inline the root rbyd of the
mtree to be inlined in the mroot.
This would allow small mtrees, 2, 3, etc mdirs, to save a block that
would be needed for the mtree's root.
But as the mshrub was progressing, minor problems kept unfolding, and
ultimately I've decided to shelve the idea of mshrubs for now. They add
quite a bit of complexity for relatively little gain:
- bshrubs are just complicated to update. They require a call to
lfsr_mdir_commit to update the inlined-root, which is a bit of a
problem when your mshrub needs to be updated inside lfsr_mdir_commit,
and your system disallows recursion...
Recursion _can_ be avoided by separate bshrub commit variants that go
through either lfsr_mdir_commit or lfsr_mdir_commit_, but this
complicates things and requires some code duplication, weakening the
value of reusing the bshrub data-structure.
- It's not always possible to compact the mshrub's backing mroot when
we need to modify the mshrub.
If an mroot becomes full and needs to split, for example, we need to
allocate the new mdirs, update the (new) mshrub, and then commit
everything into the mroot when we compact. But the "update the (new)
mshrub" step can't be done until after we compact, because the mroot
is by definition full.
This _can_ also be worked around, by building an attr list containing
all of the mshrub changes, and committing the mshrub/mroot changes in
the same transaction, but this complicates things and increases the
stack cost for the current hot-path.
- Every shrub needs a configurable shrub size, and the mshrub is no
exception. This adds another config option and complicates shared
shrub eviction code.
- The value for mshrubs is not actually that great.
Unlike file bshrubs, there's only one mshrub in the filesystem, and
I'm not sure there's a situation where a filesystem has >1 mdirs and
the exact number of allocated blocks is critical.
And this complexity is reflected in code cost and robustness, not to
mention developer time. I think for littlefs this is just not worth
doing. At least not now.
We can always introduce mshrubs in a backwards compatible manner if
needed.
---
But this rework did lead to better code organization around mdir commits
and how they update the mtree/mroot, so I'm keeping those changes.
In general lfsr_mdir_commit has been broken up into mtree/mroot specific
functions that _do_ propagate in-device changes. Any commit to the mroot
changes the on-disk state of the filesystem anyways, so the mroot commit
_must_ be the last thing lfsr_mdir_commit does.
This leads to some duplicated updates, but that's not really a problem.
Here's the new call graph inside lfsr_mdir_commit:
lfsr_mdir_commit
.---------' | | | '-----------------.
v | | '-----------------. |
lfsr_mtree_commit | '--------. | |
'---------. | | | |
v v | | |
lfsr_mroot_commit | | |
| '--------. | | |
| v v | |
| lfsr_mdir_commit_ | |
| .--------' '--------. | |
| | .-----------------|-' |
v v v v v
lfsr_mdir_commit__ lfsr_mdir_compact__
This rework didn't really impact code/stack that much. It added a bit of
code, but saved a bit of RAM. The real value is that the narrower-scoped
functions contain more focused logic:
code stack
before: 30780 2504
after: 31096 (+1.0%) 2480 (-1.0%)
This is just a useful type to have to make the code a bit more
readable.
This doesn't affect the code that much, except we are making more
on-stack copies of mptrs since the mdir doesn't technically contain
a mutable mptr. Maybe this should change?
code stack
before: 30768 2496
after: 30776 (+0.0%) 2504 (+0.3%)
As a part of the general redesign of files, all files, not just small
files, can inline some data directly in the metadata log. Originally,
this was a single piece of inlined data or an inlined tree (shrub) that
effectively acted as an overlay over the block/btree data.
This is now changed so that when we have a block/btree, the root of the
btree is inlined. In effect making a full btree a sort of extended
shrub.
I'm currently calling this a "geoxylic btree", since that seems to be a
somewhat related botanical term. Geoxylic btrees have, at least on
paper, a number of benefits:
- There is a single lookup path instead of two, this simplifies code a
bit and decreases lookup costs.
- One data structure instead of two also means lfsr_file_t requires
less RAM, since all of the on-disk variants can go into one big union.
Though I'm not sure this is very significant vs stack/buffer costs.
- The write path is much simpler and has less duplication (it was
difficult to deduplicate the shrub/btree code because of how the
shrub goes through the mdir).
In this redesign, lfsr_btree_commit_ leaves root attrs uncommitted,
allowing lfsr_bshrub_commit to finish the job via lfsr_mdir_commit.
- We don't need to maintain a shrub estimate, we just lazily evict trees
during mdir compaction. This has a side-effect of allowing shrubs to
temporarily grow larger than shrub_size before eviction.
NOTE THIS (fundamentally?) DOESN'T WORK
- There is no awkwardly high overhead for small btrees. The btree root
for two-block files should be able to comfortably fit in the shrub
portion of the btree, for example.
- It may be possible to also make the mtree geoxylic, which should
reduce storage overhead of small mtrees and make better use of the
mroot.
All of this being said, things aren't working yet. Shrub eviction during
compaction runs into a problem with a single pcache -- how do we write
the new btrees without dropping the compaction pcache? We can't evict
btrees in a separate pass becauce their number is unbounded...
We don't strictly need this for the mtree, but its impact is pretty
minimal, and it's useful for some future plans. It also makes low-level
benchmarks a bit easier to write.
The main change involves subtleties around vestigial names in leaf
rbyds (the bottom most layer of btree inner nodes). Since the mtree
terminates in mdirs, the left-most mdir in each leaf rbyd in the mtree
never actually needs a name. But in a hypothetical strict key->value
tree, every entry in the leaf rbyds need a name, and this name needs to
be respected during btree operations (mainly merges).
As a side-effect, our named btrees now require vestigial names for every
inner btree node, with the exception of the left-most inner nodes since
those can't be merged left with anything. On the bright side, being able
to assume a vestigial name on every mergable node does simplify merge
operations a bit.
It's worth noting that despite these changes, we still update vestigial
names on inner btree nodes lazily. It isn't super clear that this should
work, but it turns out that even though a leaf nodes may diverge from
the vestigial name in it's parent, it must still following the bounds of
the parent's vestigial name because of how btree lookups work. And this
property propagates up though each layer in the btree:
.---------------.
|a: |h: |-> |
'--|---|--------'
.---' '----------.
v v
.---------------. .---------------.
|a: |c: | | |i: |m: |-> |
'--|---|--------' '--|---|--------'
...--' | | '--------...
v v
.---------------. .---------------.
|d:0|e:1|f:2|-> | |j:3|k:4|l:5|-> |
'---------------' '---------------'
The exception are the left-most inner nodes, but these can never merge
left, so it doesn't really matter. The vestigial names on the left-most
inner nodes are truly vestigial:
.---------------.
|c: |e: |-> |
'--|---|--------'
.---' '--------...
v
.---------------.
|b: |d: | |
'--|---|--------'
.---' '-------...
v
.---------------.
|a:0|b:1|c:2|-> |
'---------------'
An alternative implementation may prefer to update these names eagerly,
but this would increase the amount of data written to each inner node
during btree commits. mdir updates are lazy by necessity, so even if you
adopted eager updates, the names of deleted files would still stick
around.
This did not turn out to be useful, mainly because type-agnostic
inlining requires unnecessary encoding/decoding and risks a higher RAM
allocation than is really needed. It's better to just reserve a bit in
the weight field and allow higher-level operations to use
operation-specific unions.
code stack
before: 31580 2072
after: 31160 (-1.3%) 2072 (+0.0%)
This gives the mtree a dedicated type, with direct mptrs (single mdirs)
being stored decoded, instead of encoding into leb128s. This avoids
encoding/decoding in some cases.
This change is currently a net downgrade, but only because we still have
all of the inlined btree code. Eventually this inlined btree code should
be removed:
code stack
before: 31316 2064
after: 31480 (+0.5%) 2072 (+0.4%)
Also tweaked the tests to no longer test dropping the mtree down to
zero size. Thanks to root bookmarks, we never actually do this, and it
simplifies lfsr_mdir_commit to not support this.
It's probably better to have a separate names for a tag category and any
specific name, but I can't think of a better name for this tag, and I
hadn't noticed that I was already ignoring the C prefix for CCKSUM tags
in many places.
NAME/CKSUM now mean both the specific tag and tag category, which is a
bit of a hack since both happen to be the 0th-subtype of their
categories.
Unfortunately, the tests are starting to take a painfully long time to
run. Some of this is because, in order to get interesting file
topologies, we need to move a ton of data around, but some of this is
also because our current write implementation has some problematically
expensive corner cases.
I have quite a few ideas on how to improve this, but in the meantime the
tests needed to be aggressively trimmed in order to keep development
tolerable (A happy developer is a productive developer).
This mainly meant:
- Disabled powerloss testing on file tests for now.
The reality is that naivly powerloss testing the file tests, i.e.
just truncating the file after each restart, provides very little
value and adds an extreme amount of runtime.
Removed for now. Most of the powerloss file creation concerns are
covered in the dtree tests, and we should eventually add powerloss
tests tailored to recovering files after powerloss instead of just
truncating.
- Avoided tiny fragment sizes with large file sizes.
Tiny fragments are a degenerate case and end up with excessive
overhead (1 byte fragment => 41x overhead!). But they are useful for
revealing subtle bugs. Still, it just doesn't make sense time-wise to
test with tiny fragments once the file size exceeds ~1 block.
- Limited fuzz tests to cover fewer random seeds.
We can increase these if performance improves, but even if not, we can
run these individually with a high number of seeds in CI.
Also fixed a number of bugs found by the extended testing, which is
always a good sign:
- Yet another `lfsr_data_size(&data)` vs `data.u.disk.size` typo.
This is the first time I've seen a real world argument for private
struct/class fields, but I am still against the concept.
- Fixed delta/weight miscalculation when tree-carving a left sibling.
- Fixed missing offset in hole writing during block writes.
- Worked around lfsr_file_readnext's reliance on file->size when we are
using it to write to a block. This may be more a hack than a good
long term solution though.
- Checkpointed the allocator in both lfsr_file_write and lfsr_file_sync.
Otherwise calling lfsr_file_write repeatedly can easily trigger an
incorrect ENOSPC.
- Correctly reverted both shrubs and btrees in truncate/fruncate
This gets a bit more complicated in fruncate, since either one of the
two, or both, can revert.
truncate/fruncate probably deserve a bit more work around reversions
to simpler data structures, as is.
- Added handling of shrub overflows during fruncate.
Notably not possible with truncate, shrub overflows require that we
1. flush the shrub, 2. fruncate the tree, 3. and make sure any side
effects to the buffer are handled correctly.
The original name was a bit of a mouthful.
Also dropped the default crystal_size in the test/bench runners
block_size/4 -> block_size/8. I'm already noticing large amounts of
inflation when blocks are fragmented, though I am experimenting with a
rather small fragment_size right now.
Future benchmarks/experimentation is required to figure out good values
for these.
The attempt to implement in-rbyd data slicing, being lazily coalesced
during rbyd compaction, failed pretty much completely.
Slicing is a very enticing write strategy, getting both minimal overhead
post-compaction and fast random write speeds, but the idea has some
fundamental conflicts with how we play out attrs post-compaction.
This idea might work in a more powerful filesystem, but brings back the
need to simulate rbyds in RAM, which is something I really don't want to
do (complex, bug-prone, likely adds code cost, may not even be tractable).
So, third time's the charm?
---
This new write strategy writes only datas and bptrs, and avoids dagging
by completely rewriting any regions of data larger than a configurable
crystallization threshold.
This loses most of the benefits of data crystallization, random writes
will now usually need to rewrite a full block, but as a tradeoff our
data at rest is always stored with optimal overhead.
And at least data crystallization still saves space when our data isn't
block aligned, or in sparse files. From reading up on some other
filesystem designs it seems this is a desirable optimization sometimes
referred to as "tail-packing" or "block suballocation"
Some other changes from just having more time to think about the
problem:
1. Instead of scanning to figure out our current crystal size, we can
use a simple heuristic of 1. look up left block, 2. look up right
block, 3. assume any data between these blocks contribute to our
current crystal.
This is just a heuristic, so worst case you write the first and last
byte of a block which is enough to trigger compaction into a block.
But on the plus side this avoids issues with small holes preventing
blocks from being formed.
This approach brings the number of btree lookups down from
O(crystallize_size) to 2.
2. I've gone ahead and dropped the previous scheme of coalesce_size
+ fragment_size and instead adopted a single fragment_size that
controls the size of, well, fragments, i.e. data elements stored
directly in trees.
This affects both the inlined shrub as well as fragments stored in
the inner nodes of the btree. I believe it's very similar to what is
often called "pages" in logging filesystems, though I'm going to
avoid that term for now because it's a bit overloaded.
Previously, neighboring writes that, when combined, would exceed our
coalesce_size, they just weren't combined. Now they are combined up
to our fragment size, potentially splitting the right fragment.
Before (fragment_size=8):
.---+---+---+---+---+---+---+---.
| 8 bytes |
'---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---.
| 5 bytes | 5 bytes |
'---+---+---+---+---+---+---+---+---+---'
After:
.---+---+---+---+---+---+---+---.
| 8 bytes |
'---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---.
| 8 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---'
This leads to better fragment alignment (much like our block
strategy), and minimizes tree overhead.
Any neighboring data to the right is only coalesced if it fits in the
current fragment, or would be rewritten (carved) anyways, to avoid
unnecessary data rewriting.
For example (fragment_size=8):
.---+---+---+---+---+---+---+---+---+---+---+---+---+---.
| 6 bytes | 6 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---+---+---+---+---.
| 8 bytes | 4 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---+---+---+---+---'
Other than these changes this commit is mostly a bunch of carveshrub
rewriting again, which continues to be nuanced and annoying to get
bug free.
But already there are some pretty fundamental problems.
The main issue is that, while we correctly dereference slices during
compaction, pending commits that get delayed after compaction still
point to the old block. I'm not sure there's an easy way around this
aside from aborting compaction commits or fully simulating commits,
both of which seem too costly to implement...
Also coalescing during compaction is flawed as well, since our
attributes will be outdated by the time they are committed if there is a
compaction...
Looks like it's back to the drawing board. Either our approach to
compaction needs to change, or this slice/coalescing work needs to be
reverted/redesigned...
Note this is already showing better code reuse, which is a good sign,
though maybe that's just the benefit of reimplementing similar logic
multiple times.
Now both reading and carving end up in the same lfsr_btree_readnext and
lfsr_btree_buildcarve functions for both btrees and shrubs. Both btrees
and shrubs are fundamentally rbyds, so we can share a lot of
functionality as long as we redirect to the correct commit function at
the last minute. This surprising opportunity for deduplication was
noticed while putting together the dbg scripts.
Planned logic (not actual function names):
lfsr_file_readnext -> lfsr_shrub_readnext
| |
| v
'---------> lfsr_btree_readnext
lfsr_file_flushbuffer -> lfsr_shrub_carve ------------.
.---------------------' |
v v
lfsr_file_flushshrub -> lfsr_btree_carve -> lfsr_btree_buildcarve
Though the btree part of the above statement is only a hypothetical at
the moment. Not even the shrubs can survive compaction now.
The reason is the new SLICE tag which needs low-level support in rbyd
compact. SLICE introduces indirect refernces to data located in the same
rbyd, which removes any copying cost associated with coalescing.
Previously, a large coalesce_size risked O(n^2) runtime when
incrementally append small amounts of data, but with SLICEs we can defer
coalescing to compaction time, where the copy is effectively free.
This compaction-time-coalescing is also hypothetical, which is why our
tests are failing. But the theory is promising.
I was originally against this idea because of how it crosses abstraction
layers, requiring some very low-level code that absolutely can not be
omitted in a simpler littlefs driver. But after working on the actual
file writing code for a while I've become convinced the tradeoff is
worth it.
Note coalesce_size will likely still need to be configurable. Data in
fragmenting/sparse btrees is still susceptible to coalescing, and it's
not clear the impacts of internal fragmentation when data sizes approach
the hard block_size/2 limit.
My current thinking is that these are conceptually different types, with
BTREE tags representing the entire btree, and BRANCH tags representing
only the inner btree nodes. We already have multiple btree tags anyways:
btrees attached to files, the mtree, and in the future maybe a bmaptree.
Having separate tags also makes it possible to store a btree in a btree,
though I don't think we'll ever use this functionality.
This also removes the redundant weight field from branches. The
redundant weight field is only a minor cost relative to storage, but it
also takes up a bit of RAM when encoding. Though measurements show this
isn't really significant.
New encodings:
btree encoding: branch encoding:
.---+- -+- -+- -+- -. .---+- -+- -+- -+- -.
| weight | | blocks |
+---+- -+- -+- -+- -+ ' '
| blocks | ' '
' ' +---+- -+- -+- -+- -+
' ' | trunk |
+---+- -+- -+- -+- -+ +---+- -+- -+- -+- -'
| trunk | | cksum |
+---+- -+- -+- -+- -' '---+---+---+---'
| cksum |
'---+---+---+---'
Code/RAM changes:
code stack
before: 30836 2088
after: 30944 (+0.4%) 2080 (-0.4%)
Also reordered other on-disk structs with weight/size, so such structs
always have weight/size as the first field. This may enable some
optimizations around decoding the weight/size without needing to know
the specific type in some cases.
---
This change shouldn't have affected functionality, but it revealed a bug
in a dtree test, where a did gets caught in an mdir split and the split
name makes the did unreachable.
Marking this as a TODO for now. The fix is going to be a bit involved
(fundamental changes to the opened-mdir list), and similar work is
already planned to make removed files work.
Since we need an bptr type internally, a block pointer, which is a bit
more complicated than just a single address, calling our mdir pairs
mptrs makes sense.
Oh hey, it's that piece of complexity I was worried about.
The problem was that the position calculation for new appended
right_data depended on left_overlap, which fell out of sync when
transitioning from sprout->shrub.
The fix here is to keep left_overlap/right_overlap up to date with the
model that the sprout->shrub transition is effectively doing a
shrub-wide rm first.
Hacky, but hopefully avoids bugs in the future by keeping all of these
variables in a reasonable state...
There may be a simpler way to think about how this code should function,
but I just can't see it. This may deserve a rewrite in the future.
Ended up changing the name of lfsr_mtree_traversal_t -> lfsr_traversal_t,
since this behaves more like a filesytem-wide traversal than an mtree
traversal (it returns several typed objects, not mdirs like the other
mtree functions for one).
As a part of this changeset, lfsr_btraversal_t (was lfsr_btree_traversal_t)
and lfsr_traversal_t no longer return untyped lfsr_data_ts, but instead
return specialized lfsr_{b,t}info_t structs. We weren't even using
lfsr_data_t for its original purpose in lfsr_traversal_t.
Also changed lfsr_traversal_next -> lfsr_traversal_read, you may notice
at this point the changes are intended to make lfsr_traversal_t look
more like lfsr_dir_t for consistency.
---
Internally lfsr_traversal_t now uses a full state machine with its own
enum due to the complexity of traversing the filesystem incrementally.
Because creating diagrams is fun, here's the current full state machine,
though note it will need to be extended for any
parity-trees/free-trees/etc:
mrootanchor
|
v
mrootchain
.-' |
| v
| mtree ---> openedblock
'-. | ^ | ^
v v | v |
mdirblock openedbtree
| ^
v |
mdirbtree
I'm not sure I'm happy with the current implementation, and eventually
it will need to be able to handle in-place repairs to the blocks it
sees, so this whole thing may need a rewrite.
But in the meantime, this passes the new clobber tests in test_alloc, so
it should be enough to prove the file implementation works. (which is
definitely is not fully tested yet, and some bugs had to be fixed for
the new tests in test_alloc to pass).
---
Speaking of test_alloc.
The inherent cyclic dependency between files/dirs/alloc makes it a bit
hard to know what order to test these bits of functionality in.
Originally I was testing alloc first, because it seems you need to be
confident in your block allocator before you can start testing
higher-level data structures.
But I've gone ahead and reversed this order, testing alloc after
files/dirs. This is because of an interesting observation that if alloc
is broken, you can always increase the test device's size to some absurd
number (-DDISK_SIZE=16777216, for example) to kick the can down the
road.
Testing in this order allows alloc to use more high-level APIs and
focus on corner cases where the allocator's behavior requires subtlety
to be correct (e.g. ENOSPC).
This is an exciting new function, made possible by the order-statistic
nature of our rbyds and btrees.
lfsr_file_fruncation is like truncate, but from the front. It can trim
data off of the front of files, and grow files from the front,
effectively prefixing files with zeros cheaply.
This may have some niche use cases for prefixing files with headers, but
the real killer is making logging files trivial. Up until now logging
into a file has always resulted in awkward file-swapping code when a
file gets full. Now maintaining a log is just a single fruncate call.
---
Implementation wise, lfsr_file_fruncate is very similar to
lfsr_file_truncate, except we need to always inject holes into all file
trees to adjust file contents correctly.
What do you think a file's size becomes when you:
1. seek past the end of a file
2. call write with zero data!
POSIX/etc has this case explicitly mentioned, noting that zero-sized
writes should never update the file size.
This clashes with the assumption that file writes always update the file
position, but I suppose it makes a bit of practical sense if you want
zero-sized file writes to be idempotent.
This turned out to have limited use for the tests themselves. I was
hoping to avoid the mount->format->mount fallback when powerloss
testing, but we still need it in case format was interrupted.
Still, TEST_PLS is very useful for debugging.
Previouly it was difficult to set a breakpoint at a specific location,
and after a specific powerloss event. Now all you need is this in gdb:
b <line> if test_pls == <pls>
Turns out it's hard to test file holes without seek.
It's interesting to note most of seek's buffer flush work actually
occurs lazily in lfsr_file_write, so lfsr_file_seek turns out to be a
relatively simple function.
The main purpose of this change is to introduce LFSR_DATA_CAT, a
generalized way to concatenated various data references internally.
As a side-effect lfsr_data_t has been completely restructured. Now,
lfsr_data_t can be in one of 4 modes:
If the size field's sign bit=0, the lfsr_data_t points in-device. A new,
count field, determines the encoding:
sign(size)=0, count=0 => inlined:
.---+---+---+---.
| size |
|---+---+---+---|
|c=0| inlined d | note inlined data is just enough to hold
|---+ | one encoded leb128
| ata... |
'---------------'
sign(size)=1, count=1 => direct:
.---+---+---+---. .---+---+---+---.
| size | .>| data... |
|---+---+---+---| | | . |
|c=1| | | . . .
|---+---+---+---| | . . .
| direct ptr -----' . .
'---------------'
sign(size)=1, count>=2 => indirect:
.---+---+---+---. .---+---+---+---. .---+---+---+---.
| size | .>| size | .>| data... |
|---+---+---+---| | |---+---+---+---| | | . |
|c>1| | | |c=1| | | . . .
|---+---+---+---| | |---+---+---+---| | . . .
| indirect ptr ---' | direct ptr -----' . .
'---------------' '---------------' .---+---+---+---.
| size | .>| data... |
|---+---+---+---| | | . |
|c=1| | | . . .
|---+---+---+---| | . . .
| direct ptr -----' . .
'---+---+---+---'
| . |
| . |
. . .
. .
. .
note only one indirect layer is allowed due to no recursion
If the size field's sign bit=1, the lfsr_data_t points on-disk:
sign(size)=0 => on-disk:
.---+---+---+---. .....
| size | ..'' ''..
|---+---+---+---| : : :
| block ------+->| ..:|
|---+---+---+---| | |......( )::::::|
| off -------' |:::' : |
'---------------' :' : :
''.. :.''
'''''
My goal with this commit was to test the new implementation and see how
it would impact code/RAM size before adopting it in the actual file
handling code, and the results are... not great...
code stack
before: 24668 1840
after: 25552 (+3.5%) 1920 (+4.2%)
I think most of the new cost comes from the now correct handling of
read/cmp with concatentated datas, which previously would just assert.
This change gives us LFSR_DATA_CAT, so I will be working with it for
now, but this may be worth looking at again in the future. Maybe the
correct handling of read/cmp should just be reverted to an assert...
This will stop being a problem when we actually have btrees, but for now
the fragmentation caused by byte-level syncs was easily enough to
overflow an mdir when cache size is big.
A smaller cache size is also nicer for debugging, since smaller cache
sizes results in data getting flushed to disk earlier, which is easier
to inspect than in-device buffers. And a 16-byte cache still provides
decent test coverage over cache interactions.
---
Also dropped inline_size to block_size/8. I realized while debugging
that opened shrubs take up additional space until we sync, so we need to
expect up to 2 temporary copies of shrubs when writing files.
The main improvement is moving the special inlined-file compaction logic
up into lfsr_mdir_compact__. We only need this logic for files stored in
mdirs, and thanks to its recursive nature, we weren't getting any
benefit from handling this at a lower level anyways.
This is a nice logical restructuring that probably saves a bit of code
cost in the end.
Another significant improvement is moving the staging copy of the
inlined tree's state up into the file struct itself. This solves the
problem of needed N copies of temporary inlined state when you have N
open files.
It also provides a central place to stage changes when compacting
inlined trees, which happens across several different places in the mdir
commit logic. Though some may see this as more a hack than a feature.
Also note-worthy, but minor: these changes required an additional
opened-mdir linked-list to know when the mdir is a file and may contain
an inlined tree.
Inlined files are unfortunately turning out to have more cost than
expected, mainly due to our strict no-recursion requirement.
It turns out recursively nesting (bounded) trees in a system without
recursion is a recipe for duplicating code. Though there may be other
ways to structure this.
One interesting hiccup during development is the need to have both NULL
tags and DEFERREDNULL tags in order to tell inlined trees apart from the
main tree during compaction.
And by working, I mean you can create inlined trees, just don't
compact/split/move/etc anything. But this does outline the path files
take when writing buffers into inlined trees.
"Inlined trees" in littlefs are entire small rbyd trees embedded as
secondary trees in an mdir's main rbyd tree. When fetching, we can
indicate if a given trunk belongs to the main tree or secondary tree by
setting one of the unused mode bits in the trunk's tag, now called the
"deferred" bit. This bit doesn't need to be included in the alt's "key"
field, so there's no issue with it conflicting with the alt's mode bits.
This requires a bit of tweaking lfsr_rbyd_fetch, since it needs to fall
back to the previous trunk if it discovers the most recent trunk belongs
to an inlined tree. But as a benefit we can leverage the full power of
rbyds in inlined files, including holes, partial updates, etc.
One downside is it looks like these inlined trees may involve more work
in maintining their state correctly, since they need to be sort of
"brought along" when mdirs are compacted, even if they don't actually
have a reference in the mdir yet. But the sheer amount of flexibility
this gives inlined files may make this overhead worth it.
Ran into an interesting macro-related bug. Turns out the way we are
doing implicit prefixing in TAG/ATTR macros sort of breaks how C macros
work a bit. The following does not compile:
lfsr_mdir_commit(lfs, &file->m.mdir, LFSR_ATTRS(
LFSR_ATTR(file->m.mdir.mid, DEFER, 0, DEFER(
(lfsr_rbyd_t*)&file->inlined,
LFSR_ATTR(file->buffer_pos,
DEFERRED(INLINED), +file->buffer_size, BUF(
file->buffer, file->buffer_size))))));
Or to distill it down, this does not compile:
#define LFSR_ATTR(_data) (LFSR_##_data)
#define LFSR_DEFER(_data) (LFSR_##_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(DEFER(ATTR(DATA(1))));
But this does:
#define LFSR_ATTR(_data) (_data)
#define LFSR_DEFER(_data) (_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(LFSR_DEFER(LFSR_ATTR(LFSR_DATA(1))));
Why? Well it turns out the whole way nested C macro's work is a big
hack.
A very reasonable design decision in C is to disallow recursive macro
expansions. Unlike C++, we don't want our preprocessor to suddenly stack
overflow. This rule is enforced by stopping macro expansion when a macro
contains itself. For example:
#define A() B()
#define B() A()
A()
Expands to:
A()
-> B()
-> A() (stops, probably erroring with 'A' undeclared)
But it _is_ common to want to recursively expand macro arguments. Macros
are a part of C's syntax after all, and users usually expect
expressions, such as arguments, to be context-free:
#define A(x) (x) + 1
A(A(A(A(A(0)))))
Naively this would expand to:
A(A(A(A(A(0)))))
-> (A(A(A(A(0))))) + 1 (stops)
The big hack that makes this work in C's preprocessor is the "Argument
prescan". Instead of expanding the "called" macro first, we expand any macro
inside our argument list, _then_ expand the "called" macro, and _then_
expand any new macros produced as a result of the expansion again just
for good measure.
So the above actually expands to:
A(A(A(A(A(0)))))
-> A(A(A(A((0) + 1))))
-> A(A(A(((0) + 1) + 1)))
-> A(A((((0) + 1) + 1) + 1))
-> A(((((0) + 1) + 1) + 1) + 1)
-> (((((0) + 1) + 1) + 1) + 1) + 1
This is still recursive actually! But the recursion is limited to the
actual length of the source code, so the developers likely thought this
was a reasonable tradeoff.
But what does this mean for our implicit prefixing?
#define P_A(x) P_##x
#define P_B(x) P_##x
#define P_C(x) (x)
P_A(B(A(C(0))))
None of A, B, C are in scope without prefixes, so they get expanded
after the "called" macro's expansion:
P_A(B(A(C)))
-> P_B(A(C(0)))
-> P_A(C(0)) (stops)
But this breaks when we hit the nested P_A macro.
---
For now I've gone with the temporary, and extra hacky, solution of
introducing a second LFSR_ATTR_ macro. This nesting of ATTR macros only
happens because of shrubs, and only ever goes 2 layers deep.
In the future maybe we should move away from implicit prefixing. They
have a few rough corners and may be a bit confusing for anyone new to
the code.
Christopher Haster