- Renamed mdir->u.m to mdir->u.mdir.
- Prefer mdir->u.rbyd.* where possible.
- Changed file/dir mdirs to be stored directly, requiring a cast to
lfsr_openedmdir_t to enroll in the opened mdir list.
Reverted to one set of signed lfsr_mid_rid/bid functions, and tried to
make their usage more consistent.
We have two ways to compare mdirs now, lfsr_mdir_cmp (compares block
addresses) and lfsr_mdir_bid (compares mids), and it's not very clear
when to use which one. lfsr_mdir_cmp is a bit more robust in weird mid
cases (mainly inlined mdirs when mroot mid=-1), so currently preferring
that.
Also did some bit twiddling to preserve mid=-1 => bid=-1 and rid=-1,
this save a bit of code:
code stack
before: 31056 2488
after: 30972 (-0.3%) 2496 (+0.3%)
There's only one mtree in a given filesystem. With the recent
lfsr_mdir_commit restructure, it makes more sense for the mtree to be
implicit.
code stack
before: 31096 2480
after: 31016 (-0.3%) 2480 (+0.0%)
Originally, the intention of this rework was to make it possible to
shrub the mtree, i.e. allow an mshrub, i.e. inline the root rbyd of the
mtree to be inlined in the mroot.
This would allow small mtrees, 2, 3, etc mdirs, to save a block that
would be needed for the mtree's root.
But as the mshrub was progressing, minor problems kept unfolding, and
ultimately I've decided to shelve the idea of mshrubs for now. They add
quite a bit of complexity for relatively little gain:
- bshrubs are just complicated to update. They require a call to
lfsr_mdir_commit to update the inlined-root, which is a bit of a
problem when your mshrub needs to be updated inside lfsr_mdir_commit,
and your system disallows recursion...
Recursion _can_ be avoided by separate bshrub commit variants that go
through either lfsr_mdir_commit or lfsr_mdir_commit_, but this
complicates things and requires some code duplication, weakening the
value of reusing the bshrub data-structure.
- It's not always possible to compact the mshrub's backing mroot when
we need to modify the mshrub.
If an mroot becomes full and needs to split, for example, we need to
allocate the new mdirs, update the (new) mshrub, and then commit
everything into the mroot when we compact. But the "update the (new)
mshrub" step can't be done until after we compact, because the mroot
is by definition full.
This _can_ also be worked around, by building an attr list containing
all of the mshrub changes, and committing the mshrub/mroot changes in
the same transaction, but this complicates things and increases the
stack cost for the current hot-path.
- Every shrub needs a configurable shrub size, and the mshrub is no
exception. This adds another config option and complicates shared
shrub eviction code.
- The value for mshrubs is not actually that great.
Unlike file bshrubs, there's only one mshrub in the filesystem, and
I'm not sure there's a situation where a filesystem has >1 mdirs and
the exact number of allocated blocks is critical.
And this complexity is reflected in code cost and robustness, not to
mention developer time. I think for littlefs this is just not worth
doing. At least not now.
We can always introduce mshrubs in a backwards compatible manner if
needed.
---
But this rework did lead to better code organization around mdir commits
and how they update the mtree/mroot, so I'm keeping those changes.
In general lfsr_mdir_commit has been broken up into mtree/mroot specific
functions that _do_ propagate in-device changes. Any commit to the mroot
changes the on-disk state of the filesystem anyways, so the mroot commit
_must_ be the last thing lfsr_mdir_commit does.
This leads to some duplicated updates, but that's not really a problem.
Here's the new call graph inside lfsr_mdir_commit:
lfsr_mdir_commit
.---------' | | | '-----------------.
v | | '-----------------. |
lfsr_mtree_commit | '--------. | |
'---------. | | | |
v v | | |
lfsr_mroot_commit | | |
| '--------. | | |
| v v | |
| lfsr_mdir_commit_ | |
| .--------' '--------. | |
| | .-----------------|-' |
v v v v v
lfsr_mdir_commit__ lfsr_mdir_compact__
This rework didn't really impact code/stack that much. It added a bit of
code, but saved a bit of RAM. The real value is that the narrower-scoped
functions contain more focused logic:
code stack
before: 30780 2504
after: 31096 (+1.0%) 2480 (-1.0%)
This is just a useful type to have to make the code a bit more
readable.
This doesn't affect the code that much, except we are making more
on-stack copies of mptrs since the mdir doesn't technically contain
a mutable mptr. Maybe this should change?
code stack
before: 30768 2496
after: 30776 (+0.0%) 2504 (+0.3%)
- There was a lingering strict pcache assert in lfs_bd_erase. Very
unlikely to hit, but it is possible and shouldn't be an assert now
that pcache can be left in an arbitrary state. That being said, it
was asserting on an actual bug in this case.
- Our btree traversal was not traversing the roots of zero-weight
btrees. Zero-weight btrees can happen as an intermediary step during
btree/bshrub carving. If the stars align with the block allocator and
intermediary carving states this can cause incorrect block
allocations.
- Staged updates to bsprouts/bshrubs need to be played out before
updates to opened mdirs lfsr_mdir_commit, this is just because
lfsr_file_isbsprout/isbshrub depend on mdir.block and updating the
mdirs first corrupts this.
Maybe a different organization to this code would be useful, it is
already full of TODOs.
It turned out by implicitly handling root allocation in
lfsr_btree_commit_, we were never allowing lfsr_bshrub_commit to
intercept new roots as new bshrubs. Fixing this required moving the
root allocation logic up into lfsr_btree_commit.
This resulted in quite a bit of small bug fixing because it turns out if
you can never create non-inlined bshrubs you never test non-inlined
bshrubs:
- Our previous rbyd.weight == btree.weight check for if we've reached
the root no longer works, changed to an explicit check that the blocks
match. Fortunately, now that new roots set trunk=0 new roots are no
longer a problematic case.
- We need to only evict when we calculate an accurate estimate, the
previous code had a bug where eviction occurred early based only on the
progged-since-last-estimate.
- We need to manually set bshrub.block=mdir.block on new bshrubs,
otherwise the lfsr_bshrub_isbshrub check fails in mdir commit staging.
Also updated btree/bshrub following code in the dbg scripts, which
mostly meant making them accept both BRANCH and SHRUBBRANCH tags as
btree/bshrub branches. Conveniently very little code needs to change
to extend btree read operations to support bshrubs.
Unfortunately, waiting to evict shrubs until mdir compaction does not
work because we only have a single pcache. When we evict a bshrub we
need a pcache for writing the new btree root, but if we do this during
mdir compaction, our pcache is already busy handling the mdir
compaction. We can't do a separate pass for bshrub eviction, since this
would require tracking an unbounded number of new btree roots.
In the previous shrub design, we meticulously tracked the compacted
shrub estimate in RAM, determining exactly how the estimate would change
as a part of shrub carve operations.
This worked, but was fragile. It was easy for the shrub estimate to
diverge from the actual value, and required quite a bit of extra code to
maintain. Since the use cases for bshrubs is growing a bit, I didn't
want to return to this design.
So here's a new approach based on emulating btree compacts/splits inside
the shrubs:
1. When a bshrub is fetched, scan the bshrub and calculate a compaction
estimate. Store this.
2. On every commit, find the upper bound of new data being progged, and
keep track of estimate + progged. We can at least get this relatively
easily from commit attr lists. We can't get the amount deleted, which
is the problem.
3. When estimate + progged exceeds shrub_size, scan the bshrub again and
recalculate the estimate.
4. If estimate exceeds the shrub_size/2, evict the bshrub, converting it
into a btree.
As you may note, this is very close to how our btree compacts/splits
work, but emulated. In particular, evictions/splits occur at
(shrub_size/block_size)/2 in order to avoid runaway costs when the
bshrub/btree gets close to full.
Benefits:
- This eviction heuristic is very robust. Calculating the amount progged
from the attr list is relatively cheap and easy, and any divergence
should be fixed when we recalculate the estimate.
- The runtime cost is relatively small, amortized O(log n) which is
the existing runtime to commit to rbyds.
Downsides:
- Just like btree splits, evictions force our bshrub to be ~1/2 full on
average. This combined with the 2x cost for mdir pairs, the 2x cost
for mdirs being ~1/2 full on average, and the need for both a synced
and unsynced copy of file bshrubs brings our file bshrub's overhead up
to ~16x, which is getting quite high...
Anyways, bshrubs now work, and the new file topology is passing testing.
An unfortunate surprise is the jump in stack cost. This seems to come from
moving the lfsr_btree_flush logic into the hot-path that includes bshrub
commit + mdir commit + all the mtree logic. Previously the separate of
btree/shrub commits meant that the more complex block/btree/crystal logic
was on a separate path from the mdir commit logic:
code stack lfsr_file_t
before bshrubs: 31840 2072 120
after bshrubs: 30756 (-3.5%) 2448 (+15.4%) 104 (-15.4%)
I _think_ the reality is not actually as bad as measured, most of these
flush/carve/commit functions calculate some work and then commit it in
seperate steps. In theory GCC's shrinkwrapping optimizations should
limit the stack to only what we need as we finish different
calculations, but our current stack measurement scripts just add
together the whole frames, so any per-call stack optimizations get
missed...
Note this is intentionally different from how lfsr_rbyd_fetch behaves
in lfs.c. We only call lfsr_rbyd_fetch when we need validated checksums,
otherwise we just don't fetch.
The dbg scripts, on the other hand, always go through fetch, but it is
useful to be able to inspect the state of incomplete trunks when
debugging.
This use to be how the dbg scripts behaved, but they broke because of
some recent script work.
As a part of the general redesign of files, all files, not just small
files, can inline some data directly in the metadata log. Originally,
this was a single piece of inlined data or an inlined tree (shrub) that
effectively acted as an overlay over the block/btree data.
This is now changed so that when we have a block/btree, the root of the
btree is inlined. In effect making a full btree a sort of extended
shrub.
I'm currently calling this a "geoxylic btree", since that seems to be a
somewhat related botanical term. Geoxylic btrees have, at least on
paper, a number of benefits:
- There is a single lookup path instead of two, this simplifies code a
bit and decreases lookup costs.
- One data structure instead of two also means lfsr_file_t requires
less RAM, since all of the on-disk variants can go into one big union.
Though I'm not sure this is very significant vs stack/buffer costs.
- The write path is much simpler and has less duplication (it was
difficult to deduplicate the shrub/btree code because of how the
shrub goes through the mdir).
In this redesign, lfsr_btree_commit_ leaves root attrs uncommitted,
allowing lfsr_bshrub_commit to finish the job via lfsr_mdir_commit.
- We don't need to maintain a shrub estimate, we just lazily evict trees
during mdir compaction. This has a side-effect of allowing shrubs to
temporarily grow larger than shrub_size before eviction.
NOTE THIS (fundamentally?) DOESN'T WORK
- There is no awkwardly high overhead for small btrees. The btree root
for two-block files should be able to comfortably fit in the shrub
portion of the btree, for example.
- It may be possible to also make the mtree geoxylic, which should
reduce storage overhead of small mtrees and make better use of the
mroot.
All of this being said, things aren't working yet. Shrub eviction during
compaction runs into a problem with a single pcache -- how do we write
the new btrees without dropping the compaction pcache? We can't evict
btrees in a separate pass becauce their number is unbounded...
Also limited block_size/block_count updates to only happen when the
configured value is None. This matches dbgbmap.py.
Basically just a cleanup of some bugs after the rework related to
matching dbgbmap.py. Unfortunately these scripts have too much surface
area and no tests...
This should, in theory, prevent excessive block waste when blocks gets
carved to a very small size.
At the very least, this makes crystal_size symmetrical, i.e. all blocks
smaller than crystal_size are stored as fragments, and all crystals
(sets of fragments) greater than crystal_size are stored as blocks.
Though the latter is determined heuristically.
This is similar to fragment coalescing, except we may need to merge
multiple fragments/blocks, so we check for merges in a loop (need to
determine if this is the best strategy).
This prevents runaway block allocations when writing to a file
backwards. I'm not really sure why you would write to a file backwards,
but this also has an impact on random writes.
The idea here is to:
1. Try to figure out the current "crystal" (set of fragments) we are a
part of.
2. Decide if our crystal has probably exceeded the configured crystal size
and needs to be compacted into a block.
3. Guess the local block alignment by looking at the entry immediately
left of our crystal.
Figuring out the state of our current crystal is done heuristically,
with a lookup 1 crystal-size to the left to find the start of our
crystal, followed by a lookup 1 crystal-size to the right of the crystal
start to find the end of our crystal:
-crs pos -crs pos
.-------| .-------|
-crs pos '---. | +crs '-------. +crs
.-------| |---|---. |------.
.---' | | | '---. | .---'
v v v v v v v
.---+---+---+---. +---.---+---+---. ---+---.---.---+
| crystal | blk | crystal | blk |crs| blk
'---+---+---+---' +---'---+---+---' ---+---'---'---+
This is a heuristic that doesn't catch any holes in our crystal, but
that's ok, we probably don't want small holes preventing block
compaction anyways.
Finding the block alignment then just requires looking up the entry to
the left of our crystal, if the left entry + crystal fits in a block, we
are the same block, otherwise we align to the left entry. Note this may
break our crystal during block compaction if the crystal itself is not
block aligned, but that's ok, we just recalculate the new crystal based
on the new block:
fits partial fit doesn't fit
.---+---+---+---. .---+---+---+---+---. .---+---+---+---+---.
| block |crs| | block | crs | | block |crs|
'---+---+---+---' '---+---+---+---+---' '---+---+---+---+---'
| | |
v v v
.---+---+---+---. .---+---+---+---+---. .---+---+---+---+---+- - -
| block | | block |crs| | block | block
'---+---+---+---' '---+---+---+---+---' '---+---+---+---+---+- - -
This involves at most 3 lookups, though there are some shortcuts: If
appending a file we never need to lookup the right crystal boundary,
and if we don't exceed our crystal size we don't need to figure out the
block alignment.
---
There's another variant of this scheme where we don't consider any
fragments to the right of the current fragment. This saves a lookup, but
more importantly would mean we could take advantage of ecksums when
partially rewriting part of a file.
As a tradeoff this variant does end up leaving any partially rewritten
files with >~2x storage overhead. Still, this may be interesting to
provide as an alternative write strategy in the future.
We don't strictly need this for the mtree, but its impact is pretty
minimal, and it's useful for some future plans. It also makes low-level
benchmarks a bit easier to write.
The main change involves subtleties around vestigial names in leaf
rbyds (the bottom most layer of btree inner nodes). Since the mtree
terminates in mdirs, the left-most mdir in each leaf rbyd in the mtree
never actually needs a name. But in a hypothetical strict key->value
tree, every entry in the leaf rbyds need a name, and this name needs to
be respected during btree operations (mainly merges).
As a side-effect, our named btrees now require vestigial names for every
inner btree node, with the exception of the left-most inner nodes since
those can't be merged left with anything. On the bright side, being able
to assume a vestigial name on every mergable node does simplify merge
operations a bit.
It's worth noting that despite these changes, we still update vestigial
names on inner btree nodes lazily. It isn't super clear that this should
work, but it turns out that even though a leaf nodes may diverge from
the vestigial name in it's parent, it must still following the bounds of
the parent's vestigial name because of how btree lookups work. And this
property propagates up though each layer in the btree:
.---------------.
|a: |h: |-> |
'--|---|--------'
.---' '----------.
v v
.---------------. .---------------.
|a: |c: | | |i: |m: |-> |
'--|---|--------' '--|---|--------'
...--' | | '--------...
v v
.---------------. .---------------.
|d:0|e:1|f:2|-> | |j:3|k:4|l:5|-> |
'---------------' '---------------'
The exception are the left-most inner nodes, but these can never merge
left, so it doesn't really matter. The vestigial names on the left-most
inner nodes are truly vestigial:
.---------------.
|c: |e: |-> |
'--|---|--------'
.---' '--------...
v
.---------------.
|b: |d: | |
'--|---|--------'
.---' '-------...
v
.---------------.
|a:0|b:1|c:2|-> |
'---------------'
An alternative implementation may prefer to update these names eagerly,
but this would increase the amount of data written to each inner node
during btree commits. mdir updates are lazy by necessity, so even if you
adopted eager updates, the names of deleted files would still stick
around.
This did not turn out to be useful, mainly because type-agnostic
inlining requires unnecessary encoding/decoding and risks a higher RAM
allocation than is really needed. It's better to just reserve a bit in
the weight field and allow higher-level operations to use
operation-specific unions.
code stack
before: 31580 2072
after: 31160 (-1.3%) 2072 (+0.0%)
- Not as easy to read as --ggplot, the light shades are maybe poorly
suited for plots vs other larger block elements on GitHub. I don't
know, I'm not really a graphic designer.
- GitHub may be a moving target in the future.
- GitHub is already a moving target because it has like 9 different
optional color schemes (which is good!), so most of the time the
colors won't match anyways.
- The neutral gray of --ggplot works just as well outside of GitHub.
Worst case, --github was just a preset color palette, so it could in
theory be emulated with --foreground + --background + --font-color.
The -k/--keep-going option has been more or less useless before this
since it would completely flood the screen/logs when a bug triggers
multiple test failures, which is common.
Some things to note:
- RAM management is tricky with -k/--keep-going, if we try to save logs
and filter after running everything we quickly fill up memory.
- Failing test cases are a much slower path than successes since we need
to kill and restart the underlying test_runner, its state can't be
trusted anymore. This is a-ok since hopefully you usually hope for
many more successes than failures. Unfortunately it can make
-k/--keep-going quite slow.
---
ALSO -- warning this is a tangent rant-into-the-void -- I have
discovered that Ubuntu has a "helpful" subsystem named Apport that tries
to record/log/report any process crash in the system. It is "disabled" by
default, but the way it's disabled requires LAUNCHING A PYTHON
INTERPRETER to check a flag on every segfault/assert failure.
This is what it does when it's "disabled"!
This subsystem is fundamentally incompatible with any program that
intentionally crashes subprocesses, such as our test runner. The sheer
amount of python interpreters being launched quickly eats through all
available RAM and starts OOM killing half the processes on the system.
If anyone else runs into this, a shallow bit of googling suggests the
best solution is to just disable Apport. It is not a developer friendly
subsystem:
$ sudo systemctl disable apport.service
Removing Apport brings RAM usage back down to a constant level, even
with absurd numbers of test failures. And here I thought I had memory
leak somewhere.
Previously, any labeling was _technically_ possible, but tricky to get
right and usually required repeated renderings.
It evolved out of the way colors/formats were provided: a cycled
order-significant list that gets zipped with the datasets. This works
ok for somewhat arbitrary formatting, such as colors/formats, but falls
apart for labels, where it turns out to be somewhat important what
exactly you are labeling.
The new scheme makes the label's relationship explicit, at the cost of
being a bit more verbose:
$ ./scripts/plotmpl.py bench.csv -obench.svg \
-Linorder=0,4096,avg,bench_readed \
-Lreversed=1,4096,avg,bench_readed \
-Lrandom=2,4096,avg,bench_readed
This could also be adopted in the CSV manipulation scripts (code.py,
stack.py, summary.py, etc), but I don't think it would actually see that
much use. You can always awk the output to change names and it would add
more complexity to a set of scripts that are probably already way
over-designed.
This makes more sense when using benchmarks with sparse sampling rates.
Otherwise the rate of sampling also scales the resulting measurements
incorrectly.
If the previous behavior is required (if you want to ignore buffer sizes
when amortizing read/writes for example), the -n/--size field can always
be omitted.
Note there's a bit of subtlety here, field _types_ are still infered,
but the intention of the fields, i.e. if the field contains data vs
row name/other properties, must be unambiguous in the scripts.
There is still a _tiny_ bit of inference. For most scripts only one
of --by or --fields is strictly needed, since this makes the purpose of
the other fields unambiguous.
The reason for this change is so the scripts are a bit more reliable,
but also because this simplifies the data parsing/inference a bit.
Oh, and this also changes field inference to use the csv.DictReader's
fieldnames field instead of only inspecting the returned dicts. This
should also save a bit of O(n) overhead when parsing CSV files.
1. Being able to inspect results before benchmarks complete was useful
to track their status. It also allows some analysis even if a
benchmark fails.
2. Moving these scripts out of bench.py allows them to be a bit more
flexible, at the cost of CSV parsing/structuring overhead.
3. Writing benchmark measurements immediately avoids RAM buildup as we
store intermediate measurements for each bench permutation. This may
increase the IO bottleneck, but we end up writing the same number of
lines, so not sure...
I realize avg.py has quite a bit of overlap with summary.py, but I don't
want to entangle them further. summary.py is already trying to do too
much as is...
The whitespace sensitivity of field args was starting to be a problem,
mostly for advanced plotmpl.py usage (which tbf might be appropriately
described as "super hacky" in how it uses CLI parameters):
./scripts/plotmpl.py \
-Dcase=" \
bench_rbyd_attr_append, \
bench_rbyd_attr_remove, \
bench_rbyd_attr_fetch, \
..."
This may present problems when parsing CSV files with whitespace, in
theory, maybe. But given the scope of these scripts for littlefs...
just don't do that. Thanks.
With the quantity of data being output by bench.py now, filtering ASAP
while parsing CSV files is a valuable optimization. And thanks to how
CSV files are structured, we can even avoid ever loading the full
contents into RAM.
This does end up with use filtering for defines redundantly in a few
places, but this is well worth the saved overhead from early filtering.
Also tried to clean up the plot.py/plotmpl.py's data folding path,
though that may have been wasted effort.
This is mainly to allow bench_runner to at least compile after moving
benches out of tree.
Also cleaned up lingering runner/suite munging leftover from the change
to an optional -R/--runner parameter.
This is based on how bench.py/bench_runners have actually been used in
practice. The main changes have been to make the output of bench.py more
readibly consumable by plot.py/plotmpl.py without needing a bunch of
hacky intermediary scripts.
Now instead of a single per-bench BENCH_START/BENCH_STOP, benches can
have multiple named BENCH_START/BENCH_STOP invocations to measure
multiple things in one run:
BENCH_START("fetch", i, STEP);
lfsr_rbyd_fetch(&lfs, &rbyd_, rbyd.block, CFG->block_size) => 0;
BENCH_STOP("fetch");
Benches can also now report explicit results, for non-io measurements:
BENCH_RESULT("usage", i, STEP, rbyd.eoff);
The extra iter/size parameters to BENCH_START/BENCH_RESULT also allow
some extra information to be calculated post-bench. This infomation gets
tagged with an extra bench_agg field to help organize results in
plot.py/plotmpl.py:
- bench_meas=<meas>+amor, bench_agg=raw - amortized results
- bench_meas=<meas>+div, bench_agg=raw - per-byte results
- bench_meas=<meas>+avg, bench_agg=avg - average over BENCH_SEED
- bench_meas=<meas>+min, bench_agg=min - minimum over BENCH_SEED
- bench_meas=<meas>+max, bench_agg=max - maximum over BENCH_SEED
---
Also removed all bench.tomls for now. This may seem counterproductive in
a commit to improve benchmarking, but I'm not sure there's actual value
to keeping bench cases committed in tree.
These were alway quick to fall out of date (at the time of this commit
most of the low-level bench.tomls, rbyd, btree, etc, no longer
compiled), and most benchmarks were one-off collections of scripts/data
with results too large/cumbersome to commit and keep updated in tree.
I think the better way to approach benchmarking is a seperate repo
(multiple repos?) with all related scripts/state/code and results
committed into a hopefully reproducible snapshot. Keeping the
bench.tomls in that repo makes more sense in this model.
There may be some value to having benchmarks in CI in the future, but
for that to make sense they would need to actually fail on performance
regression. How to do that isn't so clear. Anyways we can always address
this in the future rather than now.
Like many prngs, xorshift breaks down when the internal state is 0. The
common fix is to explicitly check for this and replace with a 1 when
this happens (usually when seeding, in this API we have to check every
update, this is less efficient but I don't think we really care).
As a slight tweak, this now checks for 0 but replaces it with -1. This
makes seed=0 different from seed=1, which is nice when using
seed=range(0,n) in tests/benches.
This gives the mtree a dedicated type, with direct mptrs (single mdirs)
being stored decoded, instead of encoding into leb128s. This avoids
encoding/decoding in some cases.
This change is currently a net downgrade, but only because we still have
all of the inlined btree code. Eventually this inlined btree code should
be removed:
code stack
before: 31316 2064
after: 31480 (+0.5%) 2072 (+0.4%)
Also tweaked the tests to no longer test dropping the mtree down to
zero size. Thanks to root bookmarks, we never actually do this, and it
simplifies lfsr_mdir_commit to not support this.
Before:
littlefs v2.0 0x{0,1}.232, rev 99, weight 9.256, bd 4096x256
{00a3,00a4}: 0.1 file0000 reg 32768, trunk 0xa3.a8 32768, btree 0x1a.846 32704
0.2 file0001 reg 32768, trunk 0xa3.16c 32768, btree 0xa2.be1 32704
After:
littlefs v2.0 0x{0,1}.232, rev 99, weight 9.256, bd 4096x256
{00a3,00a4}: 0.1 file0000 reg 32768, trunk 0xa3.a8, btree 0x1a.846
0.2 file0001 reg 32768, trunk 0xa3.16c, btree 0xa2.be1
Most files will have both a shrub and a btree, which makes the previous
output problematically noisy.
Unfortunately, this does lose some information: the size of the
shrub/tree, both of which may be less than the full file. But 1. this
is _technically_ redundant since you only need the block/trunk to fetch an
rbyd (though the weight is useful), and 2. The weight can still be
viewed with -s -i.
dbgbmap.py parses littlefs's mtree/btrees and displays that status of
every block in use:
$ ./scripts/dbgbmap.py disk -B4096x256 -Z -H8 -W64
bd 4096x256, 7.8% mdir, 10.2% btree, 78.1% data
mmddbbddddddmmddddmmdd--bbbbddddddddddddddbbdddd--ddddddmmdddddd
mmddddbbddbbddddddddddddddddbbddddbbddddddmmddbbdddddddddddddddd
bbdddddddddddd--ddddddddddddddddbbddddmmmmddddddddddddmmmmdddddd
ddddddddddbbdddddddddd--ddddddddddddddmmddddddddddddddddddddmmdd
ddddddbbddddddddbb--ddddddddddddddddddddbb--mmmmddbbdddddddddddd
ddddddddddddddddddddbbddbbdddddddddddddddddddddddddddddddddddddd
dddddddddd--ddddbbddddddddmmbbdd--ddddddddddddddbbmmddddbbdddddd
ddmmddddddddddmmddddddddmmddddbbbbdddddddd--ddbbddddddmmdd--ddbb
(ok, it looks a bit better with colors)
dbgbmap.py matches the layout and has the same options as tracebd.py,
allowing the combination of both to provide valuable insight into what
exactly littlefs is doing.
This required a bit of tweaking of tracebd.py to get right, mostly
around conflicting order-based arguments. This also reworks the internal
Bmap class to be more resilient to out-of-window ops, and adds an
optional informative header.
In the hack where we wait for multiple updates to fill out a full
braille/dots line we store the current pixels in a temporary array.
Unfortunately, in some cases, this is the array we modify with
updates...
A copy fixes this.
- Tried to do the rescaling a bit better with truncating divisions, so
there shouldn't be weird cross-pixel updates when things aren't well
aligned.
- Adopted optional -B<block_size>x<block_count> flag for explicitly
specifying the block-device geometry in a way that is compatible with
other scripts. Should adopt this more places.
- Adopted optional <block>.<off> argument for start of range. This
should match dbgblock.py.
- Adopted '-' for noop/zero-wear.
- Renamed a few internal things.
- Dropped subscript chars for wear, this didn't really add anything and
can be accomplished by specifying the --wear-chars explicitly.
Also changed dbgblock.py to match, this mostly affects the --off/-n/--size
flags. For example, these are all the same:
./scripts/dbgblock.py disk -B4096 --off=10 --size=5
./scripts/dbgblock.py disk -B4096 --off=10 -n5
./scripts/dbgblock.py disk -B4096 --off=10,15
./scripts/dbgblock.py disk -B4096 -n10,15
./scripts/dbgblock.py disk -B4096 0.10 -n5
Also also adopted block-device geometry argument across scripts, where
the -B flag can optionally be a full <block_size>x<block_count> geometry:
./scripts/tracebd.py disk -B4096x256
Though this is mostly unused outside of tracebd.py right now. It will be
useful for anything that formats littlefs (littlefs-fuse?) and allowing
the format everywhere is a bit of a nice convenience.
This avoids needing to return the mostly-redundant weight in the
readnext functions, and allows passing the returned data directly to
lfsr_data_read/lfsr_bd_progdata when needed.
code stack
before: 31396 2064
after: 31316 (-0.3%) 2064 (+0.0%)
- Ripped out outdated file-data representation. We don't need this.
- Changed lfsr_data_add/read/cmp to just assert when data is
concatenated data. Theoretically this is possible to implement, but
it's complicated and we never use it, so all it is is a waste of
code size...
- Added implicitly zero-filled hole representation, though this isn't
adopted in the code yet.
- Added lfsr_data_truncate/fruncate, these are really useful for
shrub/tree carving/coalescing.
---
New lfsr_data_t encoding, sign(size) indicates if the data is
on-disk/in-device, and a mode field indicates how in-device data should
be parsed:
sign(size)=1 => on-disk:
.---+---+---+---. .....
|1| size | ..'' ''..
+---+---+---+---+ : : :
| block ------+->| ..:|
+---+---+---+---+ | |......( )::::::|
| off -------' |:::' : |
'---+---+---+---' :' : :
''.. :.''
'''''
sign(size)=0, mode=0 => in-device buffer:
.---+---+---+---. .---+---+---+---.
|0| size | .>| data... |
+---+---+---+---+ | ' . '
|m=0| | | ' . '
+---+---+---+---+ | ' '
| ptr -------' ' '
'---+---+---+---' '---+---+---+---'
sign(size)=0, mode=1 => hole
.---+---+---+---.
|0| size |
+---+---+---+---+
|m=1| |
+---+ +
| |
'---+---+---+---'
sign(size)=0, mode=2 => inlined
.---+---+---+---.
|0| size |
+---+---+---+---+
|m=2| inlined d |
+---+ +
| ata... |
'---+---+---+---'
sign(size)=0, mode=3 => concatenated datas:
.---+---+---+---. .---+---+---+---.
|0| size | .>| data |
+---+---+---+---+ | + +
|m=3| c | | | | |
+---+---+---+---+ | + +
| ptr -------' | |
'---+---+---+---' +---+---+---+---+
| data |
+ +
| |
+ +
| |
+---+---+---+---+
' . '
' . '
' . '
' '
' '
'---+---+---+---'
---
Code/RAM changes:
code stack
before: 31952 2056
after: 31396 (-1.7%) 2064 (+0.4%)
I think the increased RAM cost is due to lfsr_data_add/truncate/fruncate
passing lfsr_data_t around by value, and GCC not being able to optimize
this very well since it's 3 words. I think most move optimizations stop
after 2-words...
- Added lfsr_shrub_lookupnext/lfsr_tree_lookupnext to deduplicate
the various tree lookups that need to support inlined sprouts/bptrs.
Also moved implicit bptr dereferencing here, though this may need
to be tweaked a bit to support data checksumming.
- Moved inlined sprouts/bptrs into readnext (well, lookupnext really).
This simplifies things anywhere we just need to read data from these
trees.
- Renamed lfsr_file_carveshrub/lfsr_file_carvetree ->
lfsr_shrub_carve/lfsr_tree_carve and changed parameters appropriately.
Though these aren't so clear cut. lfsr_shrub_carve still needs the
related file structure to know which mdir to commit to.
lfsr_shrub_carve will also need significant tweaking to support
recovery from failed file writes.
This also ends up losing the shrub/tree lookup reuse. There might still
be a way to deduplicate the read logic after shrub/tree lookup, but it's
probably not worth it considering this logic has become a rather small
part of the lookupnext/readnext machinery
Both shrubs and trees end up calling rbyd/btree lookupnext anyways...
In theory, it should be perfectly fine to read from a filesystem with an
invalid grm mode, so lfsr_data_readgrm has been tweaked to return
LFS_ERR_INVAL in that case.
That being said, we don't actually support read-only mounts, so the end
behavior is still the same, but this lays the groundwork for readonly
mounts in the future.
This also makes it so invalid grm modes result in LFS_ERR_INVAL when
read-only mounts aren't supported, which is what it should be anyways.
The biggest change here is the breaking up of the FLAGS config into
RFLAGS/WFLAGS/OFLAGS. This is directly inspired by, and honestly not
much more than a renaming, of the compat/ro_compat/incompat flags found
in Linux/Unix/POSIX filesystems.
I think these were first introduced in ext2? But I need to do a bit more
research on that.
RFLAGS/WFLAGS/OFLAGS provide a much more flexible, and extensible,
feature flag mechanism than the previous minor version bumps.
The (re)naming of these flags is intended to make their requirements
more clear. In order to do the relevant operation, you must understand
every flag set in the relevant flag:
- RFLAGS / incompat flags - All flags must be understood to read the
filesystem, if not understood the only possible behavior is to fail.
- WFLAGS / ro-compat flags - All flags must be understood to write to the
filesystem, if not understood the filesystem may be mounted read-only.
- OFLAGS / compat flags - Optional flags, if not understood the relevant
flag must be cleared before the filesystem can be written to, but other
than that these flags can mostly be ignored.
Some hypothetical littlefs examples:
- RFLAGS / incompat flags - Transparent compression
Is this the same as a major disk-version break? Yes kinda? An
implementation that doesn't understand compression can't read the
filesystem.
On the other hand, it's useful to have a filesystem that can read both
compressed and uncompressed variants.
- WFLAGS / ro-compat flags - Closed block-map
The idea behind a closed block-map (currently planned), is that
littlefs maintains in global space a complete mapping of all blocks in
use by the filesystem.
For such a mapping to remain consistent means that if you write to the
filesystem you must understand the closed block-map. Or in other
words, if you don't understand the closed block-map you must not write
to the filesystem.
Reading, on the other hand, can ignore many such write-related
auxiliary features, so the filesystem can still be read from.
- OFLAGS / compat flags - Global checksums
Global checksums (currently planned) are extra checksums attached to
each mdir that when combined self-validate the filesystem.
But if you don't understand global checksums, you can still read and
write the filesystem without them. The only catch is that when you write
to the filesystem, you may end up invalidating the global checksum.
Clearing the global checksum bit in the OFLAGS is a cheap way to
signal that the global checksum is no longer valid, allowing you to
still write to the filesystem without this optional feature.
Other tweaks to note:
- Renamed BLOCKLIMIT/DISKLIMIT -> BLOCKSIZE/BLOCKCOUNT
Note these are still the _actual_ block_size/block_count minus 1. The
subtle difference here was the original reason for the name change,
but after working with it for a bit, I just don't think new, otherwise
unused, names are worth it.
The minus 1 stays, however, since it avoids overflow issues at
extreme boundaries of powers of 2.
- Introduces STAGLIMIT/SATTRLIMIT, sys-attribute parallels to
UTAGLIMIT/UATTRLIMIT.
These may be useful if only uattrs are supported, or vice-versa.
- Dropped UATTRLIMIT/SATTRLIMIT to 255 bytes.
This feels extreme, but matches NAMELIMIT. These _should_ be small,
and limiting the uattr/sattr size to a single-byte leads to really
nice packing of the utag+uattrsize in a single integer.
This can always be expanded in the future if this limit proves to be a
problem.
- Renamed MLEAFLIMIT -> MDIRLIMIT and (re?)introduced MTREELIMIT.
These may be useful to limiting the mtree when needed, though it's not
clear the exact use case quite yet.
It's probably better to have a separate names for a tag category and any
specific name, but I can't think of a better name for this tag, and I
hadn't noticed that I was already ignoring the C prefix for CCKSUM tags
in many places.
NAME/CKSUM now mean both the specific tag and tag category, which is a
bit of a hack since both happen to be the 0th-subtype of their
categories.
I may be overthinking things, but I'm guessing of all the possible tag
modes we may want to add in the future, we will mostly like want to add
something that looks vaguely tag like. Like the shrub tags, for example.
It's beneficial, ordering wise, for these hypothetical future tags to
come before the cksum tags.
Current tag modes:
0x0ttt v--- tttt -ttt tttt normal tags
0x1ttt v--1 tttt -ttt tttt shrub tags
0x3tpp v-11 tttt ---- ---p cksum tags
0x4kkk v1dc kkkk -kkk kkkk alt tags
Unfortunately, the tests are starting to take a painfully long time to
run. Some of this is because, in order to get interesting file
topologies, we need to move a ton of data around, but some of this is
also because our current write implementation has some problematically
expensive corner cases.
I have quite a few ideas on how to improve this, but in the meantime the
tests needed to be aggressively trimmed in order to keep development
tolerable (A happy developer is a productive developer).
This mainly meant:
- Disabled powerloss testing on file tests for now.
The reality is that naivly powerloss testing the file tests, i.e.
just truncating the file after each restart, provides very little
value and adds an extreme amount of runtime.
Removed for now. Most of the powerloss file creation concerns are
covered in the dtree tests, and we should eventually add powerloss
tests tailored to recovering files after powerloss instead of just
truncating.
- Avoided tiny fragment sizes with large file sizes.
Tiny fragments are a degenerate case and end up with excessive
overhead (1 byte fragment => 41x overhead!). But they are useful for
revealing subtle bugs. Still, it just doesn't make sense time-wise to
test with tiny fragments once the file size exceeds ~1 block.
- Limited fuzz tests to cover fewer random seeds.
We can increase these if performance improves, but even if not, we can
run these individually with a high number of seeds in CI.
Also fixed a number of bugs found by the extended testing, which is
always a good sign:
- Yet another `lfsr_data_size(&data)` vs `data.u.disk.size` typo.
This is the first time I've seen a real world argument for private
struct/class fields, but I am still against the concept.
- Fixed delta/weight miscalculation when tree-carving a left sibling.
- Fixed missing offset in hole writing during block writes.
- Worked around lfsr_file_readnext's reliance on file->size when we are
using it to write to a block. This may be more a hack than a good
long term solution though.
- Checkpointed the allocator in both lfsr_file_write and lfsr_file_sync.
Otherwise calling lfsr_file_write repeatedly can easily trigger an
incorrect ENOSPC.
- Correctly reverted both shrubs and btrees in truncate/fruncate
This gets a bit more complicated in fruncate, since either one of the
two, or both, can revert.
truncate/fruncate probably deserve a bit more work around reversions
to simpler data structures, as is.
- Added handling of shrub overflows during fruncate.
Notably not possible with truncate, shrub overflows require that we
1. flush the shrub, 2. fruncate the tree, 3. and make sure any side
effects to the buffer are handled correctly.
Yes, erases are the more costly operation that we should highlight. But,
aside from broken code, you can never prog more than you erase.
This makes it more useful to priortize progs over erases, so erases
without an overlaying prog show up as a relatively unique blue,
indicating regions of memory that have been erased but not progged.
Too many erased-but-not-progged regions indicate a potentially wastefull
algorithm.
Christopher Haster