Now that bshrubs are limited to files, we might as well lean into it.
Instead of relying on extra traversals through our attr-list, we now
rely on bsprouts/bshrubs always existing in the opened mdir list. This
is a much more robust way to deduplicate bsprout/bshrub compaction
operations. The previous implementation already had a bug since
bsprout/bshrub calculation was forgotten from lfsr_mdir_estimate_.
This is a direct tradeoff of code and RAM. Though note the previous
impl was incomplete and would likely need a bit more code:
code stack
before: 33184 2944
after: 32980 (-0.6%) 3000 (+1.9%)
This is intended to enable lfsr_file_read to use the buffer as well.
This adds LFS_F_UNFLUSHED and internal lfsr_file_flush to manage buffer
flushing. This also results in a nice reorganization of lfsr_file_sync.
Of course, small file caching proves to be a big pain again, with
several subtle corner cases in truncate/fruncate that need to set the
LFS_F_UNFLUSHED flag so we fix small files in lfsr_file_sync.
Our crystallization threshold doesn't really describe the bounds of an
object, and I think it's a bit easier to think of it as a threshold for
block compaction.
Heck I've already been calling this the crystallization threshold all
over the code base.
An important change is this bumps the value by 1 bytes, so
crystal_thresh now describes the smallest size of a block our write
strategy will attempt to write.
Heuristically:
- data >= crystal_thresh => compacted into blocks
- data < crystal_thresh => stored as fragments
Much like the erased-state checksums in our rbyds (ecksums), these
block-level erased-state checksums (becksums) allow us to detect failed
progs to erased parts of a block and are key to achieving efficient
incremental write performance with large blocks and frequent power
cycles/open-close cycles.
These are also key to achieving _reasonable_ write performance for
simple writes (linear, non-overwriting), since littlefs now relies
solely on becksums to efficiently append to blocks.
Though I suppose the previous block staging logic used with the CTZ
skip-list could be brought back to make becksums optional and avoid
btree lookups during simple writes (we do a _lot_ of btree
lookups)... I'll leave this open as a future optimization...
Unlike in-rbyd ecksums, becksums need to be stored out-of-band so our
data blocks only contain raw data. Since they are optional, an
additional tag in the file's btree makes sense.
Becksums are relatively simple, but they bring some challenges:
1. Adding becksums to file btrees is the first case we have for multiple
struct tags per btree id.
This isn't too complicated a problem, but requires some new internal
btree APIs.
Looking forward, which I probably shouldn't be doing this often,
multiple struct tags will also be useful for parity and content ids
as a part of data redundancy and data deduplication, though I think
it's uncontroversial to consider this both heavier-weight features...
2. Becksums only work if unfilled blocks are aligned to the prog_size.
This is the whole point of crystal_size -- to provide temporary
storage for unaligned writes -- but actually aligning the block
during writes turns out to be a bit tricky without a bunch of
unecesssary btree lookups (we already do too many btree lookups!).
The current implementation here discards the pcache to force
alignment, taking advantage of the requirement that
cache_size >= prog_size, but this is corrupting our block checksums.
Code cost:
code stack
before: 31248 2792
after: 32060 (+2.5%) 2864 (+2.5%)
Also lfsr_ftree_flush needs work. I'm usually open to gotos in C when
they improve internal logic, but even for me, the multiple goto jumps
from every left-neighbor lookup into the block writing loop is a bit
much...
Looking forward, bptr checksums provide an easy mechanism to validate
data residing in blocks. This extends the merkle-tree-like nature of the
filesystem all the way down to the data level, and is common in other
COW filesystems.
Two interesting things to note:
1. We don't actually check data-level checksums yet, but we do calculate
data-level checksums unconditionally.
Writing checksums is easy, but validating checksums is a bit more
tricky. This is made a bit harder for littlefs, since we can't hold
an entire block of data in RAM, so we have to choose between separate
bus transactions for checksum + data reads, or extremely expensive
overreads every read.
Note this already exists at the metadata-level, the separate bus
transactions for rbyd fetch + rbyd lookup means we _are_ susceptible
to a very small window where bit errors can get through.
But anyways, writing checksums is easy. And has basically no cost
since we are already processing the data for our write. So we might
as well write the data-level checksums at all times, even if we
aren't validating at the data-level.
2. To make bptr checksums work cheaply we need an additional cksize
field to indicate how much data is checksummed.
This field seems redundant when we already have the bptr's data size,
but if we didn't have this field, we would be forced to recalculate
the checksum every time a block is sliced. This would be
unreasonable.
The immutable cksize field does mean we may be checksumming more data
than we need to when validating, but we should be avoiding small
block slices anyways for storage cost reasons.
This does add some stack cost because our bptr struct is larger now:
code stack
before: 31200 2768
after: 31272 (+0.2%) 2800 (+1.1%)
Also:
- Renamed GSTATE -> GDELTA for gdelta tags. GSTATE tags added as
separate in-device flags. The GSTATE tags were already serving
this dual purpose.
- Renamed BSHRUB* -> SHRUB when the tag is not necessarily operating
on a file bshrub.
- Renamed TRUNK -> BSHRUB
The tag encoding space now has a couple funky holes:
- 0x0005 - Hole for aligning config tags.
I guess this could be used for OCOMPATFLAGS in the future?
- 0x0203 - Hole so that ORPHAN can be a 1-bit difference from REG. This
could be after BOOKMARK, but having a bit to differentiate littlefs
specific file types (BOOKMARK, ORPHAN) from normal file types (REG,
DIR) is nice.
I guess this could be used for SYMLINK if we ever want symlinks in the
future?
- 0x0314-0x0318 - Hole so that the mdir related tags (MROOT, MDIR,
MTREE) are nicely aligned.
This is probably a good place for file-related tags to go in the
future (BECKSUM, CID, COMPR), but we only have two slots, so will
probably run out pretty quickly.
- 0x3028 - Hole so that all btree related tags (BTREE, BRANCH, MTREE)
share a common lower bit-pattern.
I guess this could be used for MSHRUB if we ever want mshrubs in the
future?
lfsr_ftree_t acts as a sort of proto-file type, holding enough
information for file reads/writes if the relevant mdir is known.
This lets low-level file write operations operate on a copy of the
proto-file without needing to copy the relevant mdir, file stuff, etc.
To make this work, lfsr_mdir_commit also needs to stage any bshrubs in
the attr-list, since these may not be in our opened file list, but this
is a good thing to handle implicitly anyways. We should only ever have
one untracked bshrub being operated on (multithreaded support would be a
whole other can of worms).
Unfortunately the extra machinery in lfsr_mdir_commit, and the fact that
passing two pointers around instead of one adds quite a bit of code,
means this comes with a code cost. But the tradeoff for stack cost and
no risk of stack pointers in our opened file list makes this probably
worth it:
code stack
before: 31584 2824
after: 31760 (+0.6%) 2776 (-1.7%)
This ended up being much less of a simplification than I hoped it would.
It's still easier/more efficient to revert to a relocation in most cases
when dropping in an mdir split, and the small gain from simplifying how
drops/commits interact is overshadowed by the code duplication necessary
to separate lfsr_mdir_drop out from lfsr_mdir_commit:
code stack
before: 30952 2528
after: 31280 (+1.1%) 2648 (+4.7%)
Still, this does at least simplify the logical corner cases (we don't
need to abort commits when droppable anymore), and lfsr_mdir_drop is
ultimately necessary for supporting lazy file creation.
Also having a fix-orphans step during mount allows other littlefs
implementations the option to create orphanned mdirs without compat
issues. So this ends up the more flexible approach.
It _might_ be worth having both eager mdir drops and an explicit
lfsr_mdir_drop for lazy file creation in the future, but I doubt this
will end up worth the code duplication...
---
Oh right, I forgot to actually describe this change.
This trades eager mdir drops:
1. Drop mdirs from the mtree immediately as soon as their weight goes
to zero.
For lazy mdir drops:
1. Drop mdirs from the mtree in a second commit.
2. Scan and drop orphaned mdirs on the first write after mount.
This sounds very similar to the previous "deorphan" scan, which risked
an extreme performance cost during mount, but it should be noted this
orphan scan only needs to touch every mdir once. This makes it no worse
than the overhead of actually mounting the filesystem.
We can also keep an eye out for orphaned mdirs when we mount, so no
extra scan is needed unless there was an unlucky powerloss.
Eager mdir dropping sounds simpler, but thanks to deferred commits
introduces some subtle complexity around aborting commits that would
drop an mdir to zero. Remember commits are viewable on-disk as soon as a
commit completes.
In _theory_, lazy mdir drops simplify the logic around committing to
mdirs.
Though the real kicker is that lazy mdir drops are required for lazy file
creation.
The current idea for lazy file creation involves tracking mid-less
opened-but-not-yet-created files. These files can have bshrubs, so they
need space on an mdir somewhere. But they aren't actually created yet,
so they don't have an mid.
This is fine (though it's probably going to be tricky) as long as we
allocate an mid on file sync, but there is always a risk of losing power
with mdirs that contain only RAM-backed files. Fortunately, no-mids
means no orphaned files, but it does mean orphaned mdirs with no synced
contents.
Long story short, lazy mdir drops are currently a necessary evil, and
logical simplification, that unfortunately comes with some cost.
The idea here is to revert moving redund blocks into lfsr_rbyd_t, and
instead just keep a redundant copy of the rbyd blocks in the redund
blocks in lfsr_mdir_t.
Surprisingly, extra overhead in lfsr_mdir_t ended up with worse stack
usage than extra overhead in lfsr_rbyd_t. I guess we end up allocated
more mdirs than rbyds, which makes a bit of sense given how complicated
lfsr_mdir_commit is:
code stack structs
redund union: 30976 2496 1072
redund in rbyd: 30948 (-0.1%) 2528 (+1.3%) 1100 (+2.6%)
redund in mdir: 31000 (+0.1%) 2536 (+1.6%) 1092 (+1.8%)
The mdir option does seem to improve struct overhead, but this hasn't
been a reliable measurement since it doesn't take into account how many
of each struct is allocated.
Given that the mdir option is inferior in both code and stack cost, and
requires more care to keep the rbyd/redund blocks in sync, I think I'm
going to revert this for now but keep the commit in the commit history
since it's an interesting comparison.
This simplifies dependent structs with redundancy, mainly lfsr_mdir_t,
at a significant RAM cost:
code stack structs
before: 30976 2496 1072
after: 30948 (-0.1%) 2528 (+1.3%) 1100 (+2.6%)
Which, to be honest, is not as bad as I thought it would be. Though it
is still pretty bad for no new features.
The motivation for this change:
1. The organization of the previous lfsr_mdir_t struct was a bit hacky
and relied on exact padding so the redund block array and rbyd block
lined up at the right offset.
2. The previous organization prevented theoretical "read-only rbyd
structs" that could omit write-related fields, e.g. eoff and cksum.
This idea is currently unused.
3. The current mdir=level-1, btree/data=level-0 redund design makes this
RAM tradeoff pretty bad, but in theory higher btree redund levels
would need the extra redund blocks in the rbyd struct anyways.
Still, the RAM impact to the current default configuration means this
should probably be reverted...
- Renamed mdir->u.m to mdir->u.mdir.
- Prefer mdir->u.rbyd.* where possible.
- Changed file/dir mdirs to be stored directly, requiring a cast to
lfsr_openedmdir_t to enroll in the opened mdir list.
This is just a useful type to have to make the code a bit more
readable.
This doesn't affect the code that much, except we are making more
on-stack copies of mptrs since the mdir doesn't technically contain
a mutable mptr. Maybe this should change?
code stack
before: 30768 2496
after: 30776 (+0.0%) 2504 (+0.3%)
Unfortunately, waiting to evict shrubs until mdir compaction does not
work because we only have a single pcache. When we evict a bshrub we
need a pcache for writing the new btree root, but if we do this during
mdir compaction, our pcache is already busy handling the mdir
compaction. We can't do a separate pass for bshrub eviction, since this
would require tracking an unbounded number of new btree roots.
In the previous shrub design, we meticulously tracked the compacted
shrub estimate in RAM, determining exactly how the estimate would change
as a part of shrub carve operations.
This worked, but was fragile. It was easy for the shrub estimate to
diverge from the actual value, and required quite a bit of extra code to
maintain. Since the use cases for bshrubs is growing a bit, I didn't
want to return to this design.
So here's a new approach based on emulating btree compacts/splits inside
the shrubs:
1. When a bshrub is fetched, scan the bshrub and calculate a compaction
estimate. Store this.
2. On every commit, find the upper bound of new data being progged, and
keep track of estimate + progged. We can at least get this relatively
easily from commit attr lists. We can't get the amount deleted, which
is the problem.
3. When estimate + progged exceeds shrub_size, scan the bshrub again and
recalculate the estimate.
4. If estimate exceeds the shrub_size/2, evict the bshrub, converting it
into a btree.
As you may note, this is very close to how our btree compacts/splits
work, but emulated. In particular, evictions/splits occur at
(shrub_size/block_size)/2 in order to avoid runaway costs when the
bshrub/btree gets close to full.
Benefits:
- This eviction heuristic is very robust. Calculating the amount progged
from the attr list is relatively cheap and easy, and any divergence
should be fixed when we recalculate the estimate.
- The runtime cost is relatively small, amortized O(log n) which is
the existing runtime to commit to rbyds.
Downsides:
- Just like btree splits, evictions force our bshrub to be ~1/2 full on
average. This combined with the 2x cost for mdir pairs, the 2x cost
for mdirs being ~1/2 full on average, and the need for both a synced
and unsynced copy of file bshrubs brings our file bshrub's overhead up
to ~16x, which is getting quite high...
Anyways, bshrubs now work, and the new file topology is passing testing.
An unfortunate surprise is the jump in stack cost. This seems to come from
moving the lfsr_btree_flush logic into the hot-path that includes bshrub
commit + mdir commit + all the mtree logic. Previously the separate of
btree/shrub commits meant that the more complex block/btree/crystal logic
was on a separate path from the mdir commit logic:
code stack lfsr_file_t
before bshrubs: 31840 2072 120
after bshrubs: 30756 (-3.5%) 2448 (+15.4%) 104 (-15.4%)
I _think_ the reality is not actually as bad as measured, most of these
flush/carve/commit functions calculate some work and then commit it in
seperate steps. In theory GCC's shrinkwrapping optimizations should
limit the stack to only what we need as we finish different
calculations, but our current stack measurement scripts just add
together the whole frames, so any per-call stack optimizations get
missed...
As a part of the general redesign of files, all files, not just small
files, can inline some data directly in the metadata log. Originally,
this was a single piece of inlined data or an inlined tree (shrub) that
effectively acted as an overlay over the block/btree data.
This is now changed so that when we have a block/btree, the root of the
btree is inlined. In effect making a full btree a sort of extended
shrub.
I'm currently calling this a "geoxylic btree", since that seems to be a
somewhat related botanical term. Geoxylic btrees have, at least on
paper, a number of benefits:
- There is a single lookup path instead of two, this simplifies code a
bit and decreases lookup costs.
- One data structure instead of two also means lfsr_file_t requires
less RAM, since all of the on-disk variants can go into one big union.
Though I'm not sure this is very significant vs stack/buffer costs.
- The write path is much simpler and has less duplication (it was
difficult to deduplicate the shrub/btree code because of how the
shrub goes through the mdir).
In this redesign, lfsr_btree_commit_ leaves root attrs uncommitted,
allowing lfsr_bshrub_commit to finish the job via lfsr_mdir_commit.
- We don't need to maintain a shrub estimate, we just lazily evict trees
during mdir compaction. This has a side-effect of allowing shrubs to
temporarily grow larger than shrub_size before eviction.
NOTE THIS (fundamentally?) DOESN'T WORK
- There is no awkwardly high overhead for small btrees. The btree root
for two-block files should be able to comfortably fit in the shrub
portion of the btree, for example.
- It may be possible to also make the mtree geoxylic, which should
reduce storage overhead of small mtrees and make better use of the
mroot.
All of this being said, things aren't working yet. Shrub eviction during
compaction runs into a problem with a single pcache -- how do we write
the new btrees without dropping the compaction pcache? We can't evict
btrees in a separate pass becauce their number is unbounded...
This did not turn out to be useful, mainly because type-agnostic
inlining requires unnecessary encoding/decoding and risks a higher RAM
allocation than is really needed. It's better to just reserve a bit in
the weight field and allow higher-level operations to use
operation-specific unions.
code stack
before: 31580 2072
after: 31160 (-1.3%) 2072 (+0.0%)
This is based on how bench.py/bench_runners have actually been used in
practice. The main changes have been to make the output of bench.py more
readibly consumable by plot.py/plotmpl.py without needing a bunch of
hacky intermediary scripts.
Now instead of a single per-bench BENCH_START/BENCH_STOP, benches can
have multiple named BENCH_START/BENCH_STOP invocations to measure
multiple things in one run:
BENCH_START("fetch", i, STEP);
lfsr_rbyd_fetch(&lfs, &rbyd_, rbyd.block, CFG->block_size) => 0;
BENCH_STOP("fetch");
Benches can also now report explicit results, for non-io measurements:
BENCH_RESULT("usage", i, STEP, rbyd.eoff);
The extra iter/size parameters to BENCH_START/BENCH_RESULT also allow
some extra information to be calculated post-bench. This infomation gets
tagged with an extra bench_agg field to help organize results in
plot.py/plotmpl.py:
- bench_meas=<meas>+amor, bench_agg=raw - amortized results
- bench_meas=<meas>+div, bench_agg=raw - per-byte results
- bench_meas=<meas>+avg, bench_agg=avg - average over BENCH_SEED
- bench_meas=<meas>+min, bench_agg=min - minimum over BENCH_SEED
- bench_meas=<meas>+max, bench_agg=max - maximum over BENCH_SEED
---
Also removed all bench.tomls for now. This may seem counterproductive in
a commit to improve benchmarking, but I'm not sure there's actual value
to keeping bench cases committed in tree.
These were alway quick to fall out of date (at the time of this commit
most of the low-level bench.tomls, rbyd, btree, etc, no longer
compiled), and most benchmarks were one-off collections of scripts/data
with results too large/cumbersome to commit and keep updated in tree.
I think the better way to approach benchmarking is a seperate repo
(multiple repos?) with all related scripts/state/code and results
committed into a hopefully reproducible snapshot. Keeping the
bench.tomls in that repo makes more sense in this model.
There may be some value to having benchmarks in CI in the future, but
for that to make sense they would need to actually fail on performance
regression. How to do that isn't so clear. Anyways we can always address
this in the future rather than now.
This gives the mtree a dedicated type, with direct mptrs (single mdirs)
being stored decoded, instead of encoding into leb128s. This avoids
encoding/decoding in some cases.
This change is currently a net downgrade, but only because we still have
all of the inlined btree code. Eventually this inlined btree code should
be removed:
code stack
before: 31316 2064
after: 31480 (+0.5%) 2072 (+0.4%)
Also tweaked the tests to no longer test dropping the mtree down to
zero size. Thanks to root bookmarks, we never actually do this, and it
simplifies lfsr_mdir_commit to not support this.
- Ripped out outdated file-data representation. We don't need this.
- Changed lfsr_data_add/read/cmp to just assert when data is
concatenated data. Theoretically this is possible to implement, but
it's complicated and we never use it, so all it is is a waste of
code size...
- Added implicitly zero-filled hole representation, though this isn't
adopted in the code yet.
- Added lfsr_data_truncate/fruncate, these are really useful for
shrub/tree carving/coalescing.
---
New lfsr_data_t encoding, sign(size) indicates if the data is
on-disk/in-device, and a mode field indicates how in-device data should
be parsed:
sign(size)=1 => on-disk:
.---+---+---+---. .....
|1| size | ..'' ''..
+---+---+---+---+ : : :
| block ------+->| ..:|
+---+---+---+---+ | |......( )::::::|
| off -------' |:::' : |
'---+---+---+---' :' : :
''.. :.''
'''''
sign(size)=0, mode=0 => in-device buffer:
.---+---+---+---. .---+---+---+---.
|0| size | .>| data... |
+---+---+---+---+ | ' . '
|m=0| | | ' . '
+---+---+---+---+ | ' '
| ptr -------' ' '
'---+---+---+---' '---+---+---+---'
sign(size)=0, mode=1 => hole
.---+---+---+---.
|0| size |
+---+---+---+---+
|m=1| |
+---+ +
| |
'---+---+---+---'
sign(size)=0, mode=2 => inlined
.---+---+---+---.
|0| size |
+---+---+---+---+
|m=2| inlined d |
+---+ +
| ata... |
'---+---+---+---'
sign(size)=0, mode=3 => concatenated datas:
.---+---+---+---. .---+---+---+---.
|0| size | .>| data |
+---+---+---+---+ | + +
|m=3| c | | | | |
+---+---+---+---+ | + +
| ptr -------' | |
'---+---+---+---' +---+---+---+---+
| data |
+ +
| |
+ +
| |
+---+---+---+---+
' . '
' . '
' . '
' '
' '
'---+---+---+---'
---
Code/RAM changes:
code stack
before: 31952 2056
after: 31396 (-1.7%) 2064 (+0.4%)
I think the increased RAM cost is due to lfsr_data_add/truncate/fruncate
passing lfsr_data_t around by value, and GCC not being able to optimize
this very well since it's 3 words. I think most move optimizations stop
after 2-words...
The biggest change here is the breaking up of the FLAGS config into
RFLAGS/WFLAGS/OFLAGS. This is directly inspired by, and honestly not
much more than a renaming, of the compat/ro_compat/incompat flags found
in Linux/Unix/POSIX filesystems.
I think these were first introduced in ext2? But I need to do a bit more
research on that.
RFLAGS/WFLAGS/OFLAGS provide a much more flexible, and extensible,
feature flag mechanism than the previous minor version bumps.
The (re)naming of these flags is intended to make their requirements
more clear. In order to do the relevant operation, you must understand
every flag set in the relevant flag:
- RFLAGS / incompat flags - All flags must be understood to read the
filesystem, if not understood the only possible behavior is to fail.
- WFLAGS / ro-compat flags - All flags must be understood to write to the
filesystem, if not understood the filesystem may be mounted read-only.
- OFLAGS / compat flags - Optional flags, if not understood the relevant
flag must be cleared before the filesystem can be written to, but other
than that these flags can mostly be ignored.
Some hypothetical littlefs examples:
- RFLAGS / incompat flags - Transparent compression
Is this the same as a major disk-version break? Yes kinda? An
implementation that doesn't understand compression can't read the
filesystem.
On the other hand, it's useful to have a filesystem that can read both
compressed and uncompressed variants.
- WFLAGS / ro-compat flags - Closed block-map
The idea behind a closed block-map (currently planned), is that
littlefs maintains in global space a complete mapping of all blocks in
use by the filesystem.
For such a mapping to remain consistent means that if you write to the
filesystem you must understand the closed block-map. Or in other
words, if you don't understand the closed block-map you must not write
to the filesystem.
Reading, on the other hand, can ignore many such write-related
auxiliary features, so the filesystem can still be read from.
- OFLAGS / compat flags - Global checksums
Global checksums (currently planned) are extra checksums attached to
each mdir that when combined self-validate the filesystem.
But if you don't understand global checksums, you can still read and
write the filesystem without them. The only catch is that when you write
to the filesystem, you may end up invalidating the global checksum.
Clearing the global checksum bit in the OFLAGS is a cheap way to
signal that the global checksum is no longer valid, allowing you to
still write to the filesystem without this optional feature.
Other tweaks to note:
- Renamed BLOCKLIMIT/DISKLIMIT -> BLOCKSIZE/BLOCKCOUNT
Note these are still the _actual_ block_size/block_count minus 1. The
subtle difference here was the original reason for the name change,
but after working with it for a bit, I just don't think new, otherwise
unused, names are worth it.
The minus 1 stays, however, since it avoids overflow issues at
extreme boundaries of powers of 2.
- Introduces STAGLIMIT/SATTRLIMIT, sys-attribute parallels to
UTAGLIMIT/UATTRLIMIT.
These may be useful if only uattrs are supported, or vice-versa.
- Dropped UATTRLIMIT/SATTRLIMIT to 255 bytes.
This feels extreme, but matches NAMELIMIT. These _should_ be small,
and limiting the uattr/sattr size to a single-byte leads to really
nice packing of the utag+uattrsize in a single integer.
This can always be expanded in the future if this limit proves to be a
problem.
- Renamed MLEAFLIMIT -> MDIRLIMIT and (re?)introduced MTREELIMIT.
These may be useful to limiting the mtree when needed, though it's not
clear the exact use case quite yet.
The original name was a bit of a mouthful.
Also dropped the default crystal_size in the test/bench runners
block_size/4 -> block_size/8. I'm already noticing large amounts of
inflation when blocks are fragmented, though I am experimenting with a
rather small fragment_size right now.
Future benchmarks/experimentation is required to figure out good values
for these.
The attempt to implement in-rbyd data slicing, being lazily coalesced
during rbyd compaction, failed pretty much completely.
Slicing is a very enticing write strategy, getting both minimal overhead
post-compaction and fast random write speeds, but the idea has some
fundamental conflicts with how we play out attrs post-compaction.
This idea might work in a more powerful filesystem, but brings back the
need to simulate rbyds in RAM, which is something I really don't want to
do (complex, bug-prone, likely adds code cost, may not even be tractable).
So, third time's the charm?
---
This new write strategy writes only datas and bptrs, and avoids dagging
by completely rewriting any regions of data larger than a configurable
crystallization threshold.
This loses most of the benefits of data crystallization, random writes
will now usually need to rewrite a full block, but as a tradeoff our
data at rest is always stored with optimal overhead.
And at least data crystallization still saves space when our data isn't
block aligned, or in sparse files. From reading up on some other
filesystem designs it seems this is a desirable optimization sometimes
referred to as "tail-packing" or "block suballocation"
Some other changes from just having more time to think about the
problem:
1. Instead of scanning to figure out our current crystal size, we can
use a simple heuristic of 1. look up left block, 2. look up right
block, 3. assume any data between these blocks contribute to our
current crystal.
This is just a heuristic, so worst case you write the first and last
byte of a block which is enough to trigger compaction into a block.
But on the plus side this avoids issues with small holes preventing
blocks from being formed.
This approach brings the number of btree lookups down from
O(crystallize_size) to 2.
2. I've gone ahead and dropped the previous scheme of coalesce_size
+ fragment_size and instead adopted a single fragment_size that
controls the size of, well, fragments, i.e. data elements stored
directly in trees.
This affects both the inlined shrub as well as fragments stored in
the inner nodes of the btree. I believe it's very similar to what is
often called "pages" in logging filesystems, though I'm going to
avoid that term for now because it's a bit overloaded.
Previously, neighboring writes that, when combined, would exceed our
coalesce_size, they just weren't combined. Now they are combined up
to our fragment size, potentially splitting the right fragment.
Before (fragment_size=8):
.---+---+---+---+---+---+---+---.
| 8 bytes |
'---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---.
| 5 bytes | 5 bytes |
'---+---+---+---+---+---+---+---+---+---'
After:
.---+---+---+---+---+---+---+---.
| 8 bytes |
'---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---.
| 8 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---'
This leads to better fragment alignment (much like our block
strategy), and minimizes tree overhead.
Any neighboring data to the right is only coalesced if it fits in the
current fragment, or would be rewritten (carved) anyways, to avoid
unnecessary data rewriting.
For example (fragment_size=8):
.---+---+---+---+---+---+---+---+---+---+---+---+---+---.
| 6 bytes | 6 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---+---+---+---+---'
+
.---+---+---+---+---.
| 5 bytes |
'---+---+---+---+---'
=
.---+---+---+---+---+---+---+---+---+---+---+---+---+---.
| 8 bytes | 4 bytes |2 bytes|
'---+---+---+---+---+---+---+---+---+---+---+---+---+---'
Other than these changes this commit is mostly a bunch of carveshrub
rewriting again, which continues to be nuanced and annoying to get
bug free.
- -> lfsr_shrub_t
- -> lfsr_tree_t
The idea here is to adopt "shrub" as an umbrella term for the
shrub/sprout union, and "tree" as an umbrella term for the bptr/btree
union. I think this is a bit better than calling shrub/sprout "inlined"
which is a _very_ overloaded term in this codebase (inlined in the tree?
the mdir? inlined in the C struct?).
This is a pretty big rewrite, but is necessary to avoid "dagging".
"Dagging" (I just made this term up) is when you transform a pure tree
into a directed acyclic graph (DAG). Normally DAGs are perfectly fine in
a copy-on-write system, but in littlefs's cases, it creates havoc for
future block allocator plans, and it's interaction with parity blocks
raises some uncomfortable questions.
How does dagging happen?
Consider an innocent little btree with a single block:
.-----.
|btree|
| |
'-----'
|
v
.-----.
|abcde|
| |
'-----'
Say we wanted to write a small amount of data in the middle of our
block. Since the data is so small, the previous scheme would simply
inline the data, carving the left and right sibling (in the case the
same block) to make space:
.-----.
|btree|
| |
'-----'
.' v '.
| c' |
'. .'
v v
.-----.
|ab de|
| |
'-----'
Oh no! A DAG!
With the potential for multiple pointers to reference the same block in
our btree, some invariants break down:
- Blocks no longer have a single reference
- If you remove a reference you can no longer assume the block is free
- Knowing when a block is free requires scanning the whole btree
- This split operation effectively creates two blocks, does that mean
we need to rewrite parity blocks?
---
To avoid this whole situation, this commit adopts a new crystallization
algorithm.
Instead of allowing crystallization data to be arbitrarily fragmented,
we eagerly coalesce any data under our crystallization threshold, and if
we can't coalesce, we compact everything into a block.
Much like a Knuth heap, simply checking both siblings to coalesce has
the effect that any data will always coalesce up to the maximum size
where possible. And when checking for siblings, we can easily find the
block alignment.
This also has the effect of always rewriting blocks if we are writing a
small amount of data into a block. Unfortunately I think this is just
necessary in order to avoid dagging.
At the very least crystallization is still useful for files not quite
block aligned at the edges, and sparse files. This also avoids concerns
of random writes inflating a file via sparse crystallization.
Still needs testing, though the byte-level fuzz tests were already causing
blocks to crystallize. I noticed this because of test failures which are
fixed now.
Note the block allocator currently doesn't understand file btrees. To
get the current tests passing requires -DDISK_SIZE=16777216 or greater.
It's probably also worth noting there's a lot that's not implemented
yet! Data checksums and write validation for one. Also ecksums. And we
should probably have some sort of special handling for linear writes so
linear writes (the most common) don't end up with a bunch of extra
crystallizing writes.
Also the fact that btrees can become DAGs now is an oversight and a bit
concerning. Will that work with a closed allocator? Block parity?
Added lfsr_bptr_t to represent block pointers (maybe we should rename
mblocks back to mptr), added fetching of btrees/bptrs in
lfsr_file_opencfg, added estimate tracking to our shrubs so we actually
know when to create a btree, and implemented most of the high-level
btree logic.
It's not working yet, but the biggest idea introduced here is how we
handle block alignment.
See, we really don't want awkward btree topologies to form where small
amounts of data get stuck between blocks:
.-----.--.-----.
| | | |
| | | |
'-----'--'-----'
This is wasteful, as the middle bit of data either gets represented as a
full block with its data partially covered, or as data inlined in the
btree, which comes with ~2x overhead.
The solution here is to scan for a block on either the left or right to
derive our block alignment from.
Unfortunately, since our sibling blocks could have been carved, this
requires scanning all the way from pos-2*B+1 to pos+2*B-1, a total of
4*B-2, to make sure we find a sibling if there is one.
worst case left worst case right
.-----.-----. .-----.-----.
| xxxx| | |p |xxxxx|
|xxxxx| p| | |xxxx |
'-----'-----' '-----'-----'
'----+----' '----+----'
pos-2*bs+1 pos+2*bs-1
Fortunately, at this stage, data should have had many chances to
coalesce, so hopefully the actual scan overhead should be much smaller
in practice.
Writing data to a file linearly, for example, only needs a single lookup
to find the previous block.
Turns out it's hard to test file holes without seek.
It's interesting to note most of seek's buffer flush work actually
occurs lazily in lfsr_file_write, so lfsr_file_seek turns out to be a
relatively simple function.
- coalesce_size - The amount of data allowed to coalesce into single
data entries.
- crystallize_size - How much data is allowed to be written to btree
inner nodes before needing to be compacted into a block.
Also deduplicated the test config is something I've been wanting to do
for a while. It doesn't make sense to need to modify several different
instantiations of lfs_config every time a config option is added or
removed...
The main purpose of this change is to introduce LFSR_DATA_CAT, a
generalized way to concatenated various data references internally.
As a side-effect lfsr_data_t has been completely restructured. Now,
lfsr_data_t can be in one of 4 modes:
If the size field's sign bit=0, the lfsr_data_t points in-device. A new,
count field, determines the encoding:
sign(size)=0, count=0 => inlined:
.---+---+---+---.
| size |
|---+---+---+---|
|c=0| inlined d | note inlined data is just enough to hold
|---+ | one encoded leb128
| ata... |
'---------------'
sign(size)=1, count=1 => direct:
.---+---+---+---. .---+---+---+---.
| size | .>| data... |
|---+---+---+---| | | . |
|c=1| | | . . .
|---+---+---+---| | . . .
| direct ptr -----' . .
'---------------'
sign(size)=1, count>=2 => indirect:
.---+---+---+---. .---+---+---+---. .---+---+---+---.
| size | .>| size | .>| data... |
|---+---+---+---| | |---+---+---+---| | | . |
|c>1| | | |c=1| | | . . .
|---+---+---+---| | |---+---+---+---| | . . .
| indirect ptr ---' | direct ptr -----' . .
'---------------' '---------------' .---+---+---+---.
| size | .>| data... |
|---+---+---+---| | | . |
|c=1| | | . . .
|---+---+---+---| | . . .
| direct ptr -----' . .
'---+---+---+---'
| . |
| . |
. . .
. .
. .
note only one indirect layer is allowed due to no recursion
If the size field's sign bit=1, the lfsr_data_t points on-disk:
sign(size)=0 => on-disk:
.---+---+---+---. .....
| size | ..'' ''..
|---+---+---+---| : : :
| block ------+->| ..:|
|---+---+---+---| | |......( )::::::|
| off -------' |:::' : |
'---------------' :' : :
''.. :.''
'''''
My goal with this commit was to test the new implementation and see how
it would impact code/RAM size before adopting it in the actual file
handling code, and the results are... not great...
code stack
before: 24668 1840
after: 25552 (+3.5%) 1920 (+4.2%)
I think most of the new cost comes from the now correct handling of
read/cmp with concatentated datas, which previously would just assert.
This change gives us LFSR_DATA_CAT, so I will be working with it for
now, but this may be worth looking at again in the future. Maybe the
correct handling of read/cmp should just be reverted to an assert...
The *_is* functions for testing for bit flags have proven useful, maybe
mostly due to C's bad bitwise operator precedence, but the
*_set*/*_clear* functions just add extra code without much benefit.
Relying on explicit bitwise operations also lets us use the |=/&= shortcut
operators, which are nice.
Get it? Because they're small trees!
Joking aside, having a new term for these helps structure and describe
the filesystem at a high-level without needing to say "inlined trees"
all the time.
Shrub trees are small rbyd trees inlined directly in a file's mdir.
The main improvement is moving the special inlined-file compaction logic
up into lfsr_mdir_compact__. We only need this logic for files stored in
mdirs, and thanks to its recursive nature, we weren't getting any
benefit from handling this at a lower level anyways.
This is a nice logical restructuring that probably saves a bit of code
cost in the end.
Another significant improvement is moving the staging copy of the
inlined tree's state up into the file struct itself. This solves the
problem of needed N copies of temporary inlined state when you have N
open files.
It also provides a central place to stage changes when compacting
inlined trees, which happens across several different places in the mdir
commit logic. Though some may see this as more a hack than a feature.
Also note-worthy, but minor: these changes required an additional
opened-mdir linked-list to know when the mdir is a file and may contain
an inlined tree.
Ran into an interesting macro-related bug. Turns out the way we are
doing implicit prefixing in TAG/ATTR macros sort of breaks how C macros
work a bit. The following does not compile:
lfsr_mdir_commit(lfs, &file->m.mdir, LFSR_ATTRS(
LFSR_ATTR(file->m.mdir.mid, DEFER, 0, DEFER(
(lfsr_rbyd_t*)&file->inlined,
LFSR_ATTR(file->buffer_pos,
DEFERRED(INLINED), +file->buffer_size, BUF(
file->buffer, file->buffer_size))))));
Or to distill it down, this does not compile:
#define LFSR_ATTR(_data) (LFSR_##_data)
#define LFSR_DEFER(_data) (LFSR_##_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(DEFER(ATTR(DATA(1))));
But this does:
#define LFSR_ATTR(_data) (_data)
#define LFSR_DEFER(_data) (_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(LFSR_DEFER(LFSR_ATTR(LFSR_DATA(1))));
Why? Well it turns out the whole way nested C macro's work is a big
hack.
A very reasonable design decision in C is to disallow recursive macro
expansions. Unlike C++, we don't want our preprocessor to suddenly stack
overflow. This rule is enforced by stopping macro expansion when a macro
contains itself. For example:
#define A() B()
#define B() A()
A()
Expands to:
A()
-> B()
-> A() (stops, probably erroring with 'A' undeclared)
But it _is_ common to want to recursively expand macro arguments. Macros
are a part of C's syntax after all, and users usually expect
expressions, such as arguments, to be context-free:
#define A(x) (x) + 1
A(A(A(A(A(0)))))
Naively this would expand to:
A(A(A(A(A(0)))))
-> (A(A(A(A(0))))) + 1 (stops)
The big hack that makes this work in C's preprocessor is the "Argument
prescan". Instead of expanding the "called" macro first, we expand any macro
inside our argument list, _then_ expand the "called" macro, and _then_
expand any new macros produced as a result of the expansion again just
for good measure.
So the above actually expands to:
A(A(A(A(A(0)))))
-> A(A(A(A((0) + 1))))
-> A(A(A(((0) + 1) + 1)))
-> A(A((((0) + 1) + 1) + 1))
-> A(((((0) + 1) + 1) + 1) + 1)
-> (((((0) + 1) + 1) + 1) + 1) + 1
This is still recursive actually! But the recursion is limited to the
actual length of the source code, so the developers likely thought this
was a reasonable tradeoff.
But what does this mean for our implicit prefixing?
#define P_A(x) P_##x
#define P_B(x) P_##x
#define P_C(x) (x)
P_A(B(A(C(0))))
None of A, B, C are in scope without prefixes, so they get expanded
after the "called" macro's expansion:
P_A(B(A(C)))
-> P_B(A(C(0)))
-> P_A(C(0)) (stops)
But this breaks when we hit the nested P_A macro.
---
For now I've gone with the temporary, and extra hacky, solution of
introducing a second LFSR_ATTR_ macro. This nesting of ATTR macros only
happens because of shrubs, and only ever goes 2 layers deep.
In the future maybe we should move away from implicit prefixing. They
have a few rough corners and may be a bit confusing for anyone new to
the code.
Currently limited to inlined files and only simpler truncate-writes.
But still this lets us test file creation/deletion.
This is also enough logic to make it clear that, even though we have
some powerful high-level primitives, mapping file operations onto these
is still going to be non-trivial.
Now, instead of storing a single contiguous block of config data, config
is stored as tagged metadata like any other attribute.
This allows more flexibility towards adding/removing config in the
future, without cluttering up the config with deprecated entries (see
ATA's "IDENTIFY DEVICE" response).
Most of the config entries are single leb128 limits on various integer
types, with the exception of the magic string and version (major/minor
pair).
---
Note this also includes some semantic changes to the config:
- Limits are stored as size-1. This avoid issues with integer overflow
at extreme ranges.
This was also adopted for block size (block limit) and block count
(disk limit). This deviation between on-disk config and user-facing
config risks confusion, but allows the potential for the full 2^31 range
for these values.
- The default cksum type, crc32c, has been changed to 0.
Originally this was 2 to allow the type to map to the crc width for
crc8, crc16, crc32c, crc64, etc. But dropping this idea and numbering
checksums as they are implemented simplifies things.
May come back to this.
- Storing these configs as attributes opens up of the option of on-disk
defaults when configs are missing.
I'm being a bit conservative with this one, as it's not clear to me if
we should prefer default configs (less code/storage, risk of untested
config parsing) or prefer explicit on-disk configs.
Currently the following have defaults since they seem the most obvious
to me:
- cksum type => defaults to crc32c
- redund type => defaults to parity (TODO, should this default to
no redund?)
- utag_limit => defaults to 0x7f (no special tag decoding)
- uattr_limit => defaults to block_limit (implicit)
- mbits -> mleaf_bits
- mlimit -> mleaf_limit
- mweight -> mleaf_weight
- lfsr_mridmask -> lfsr_midrmask
- lfsr_mbidmask -> lfsr_midbmask
This is a bit tricky to name, since we want to clarify it's not the
mtree limit and not the mdir's actual rbyd weight. But this also risks
confusing around the difference between mdirs/mleaves (mdirs are
mtree's leaves).
- Fixed LFSR_GRM_DSIZE upper bound, since our mids now fit in a single
leb128.
- Renamed pgrm -> ggrm. To be honest I don't have a great name for this
variable.
Adopted lfsr_rid/bid/mid/did_t where appropriate. This includes using
lfsr_rid_t for tag/rbyd weights. Although I am using lfsr_srid_t for
rbyd weights now, since it both captures the use of the sign bit and
reduces the number of casts a bit in the code.
I learned recently Zig has any-bit integers (e.g. uint31_t), and I'm
realizing how nice it would be to have those in this codebase.
Also tried to use lfs_size_t/lfs_off_t more correctly. In Linux/BSD,
only off_t is used for file-size-related operations and is usually much
larger than size_t. These were used interchangably in littlefs and their
original meaning kind of fell by the wayside. Getting their use right
will be important if littlefs ever supports different integer widths.
This reverts the big hack of treating the lfsr_dir_t as an mdir array in
lfsr_mdir_commit in an effort to deduplicate the bookmark/pos mid
updates.
It worked, but lets be honest, it was a big hack and probably not very
maintainable. It made other opened-mdir updates, such as propagation of
unerases more complex, and is made the code a bit unreadable.
We also don't really need a full mdir for the dir's bookmark, since
rewinds really aren't that common, and a single mtree lookup in that
case gets the job done. Removing the bookmark mdir (though we still need
the bookmark mid to adjust the dir pos correctly) saves 24 bytes from
every lfsr_dir_t.
It would be nice to deduplicate some of the mid logic here, but that's
been difficult because of mid-related side-effects, such as updating the
mdir's pos. There may be room for improvement here.
---
This looks pretty bad, with the additional loop over the attr-list to
update just the dir's bookmark, but it's really not that bad when
compiled, and probably worth the code readability:
code stack structs
before: 20958 1744 864
after: 21052 (+0.4%) 1744 (+0.0%) 840 (-2.9%)
This is a tricky nuance of how rbyd's erased state interacts with
possible errors during commits.
- If an rbyd passes its ecksum during rbyd-fetch, it's erased and we can
write to it.
- If an rbyd is committed to successfully and still has erased space
remaining, it's erased and we can write to it.
- But if we fail to commit to the rbyd, we can't be sure the trailing
data is still erased. It most likely isn't, and we would need to fetch
again to check the ecksum. And since errors are exceptional here, we
might as well just mark any failed commits as unerased, triggering a
compaction on the next write to the rbyd.
To make things more annoying, changing state in all error routes is
tricky to get right, and trickier to test. To keep this relatively
simple and robust, all rbyd/btree/mdir operations mark the original copy
as unerased until the commit succeeds, and then clears the unerased
state. This fits in well with how we make copies of the rbyd/btree/mdir
structs in the relevant functions.
Note this needs to affect _all_ copies of the rbyd, including any opened
mdirs, mroots, etc. This will probably still lead to some bugs in the
future...
- Added lfsr_mdir_lookupnext, for iteration through only a single mid.
This is useful for MOVE attributes.
- Renamed LFSR_MDIR_MROOTANCHOR -> LFSR_MROOTANCHOR.
- Renamed functions that operate on mdir blocks lfsr_mdir_* ->
lfsr_mblocks_*.
- Reordered arguments in lfsr_mdir_fetch.
- Renamed mrid_bits/mbid_weight -> mbits/mweight.
This adopts a previously discarded idea for compressed mids with a few
tweaks to avoiding decoding the bid/rid portions as much as possible.
The idea of compressed mids is to shove both the mid bid and mid rid
into a single integer, saving RAM and potentially helping filesystem
integration where a unique per-file integer is useful.
Unfortunately this has proven tricky. littlefs fundamentally needs two
ids, one "bid" to lookup which mdir our entry resides on, and one "rid"
to lookup the entry in the mdir. It's tempting to use two half-sized
integers (16-bit for example), but this risks surprising limitations
around the number of files when blocks are either really large or
really small.
Optimally, we'd limit the number of bits reserved for the rid to the
upper bound of number of rids that can fit in a single mdir. This would
allows for more bids when the block size is small, and more rids when
the block size is large. This should roughly approximate the limits of
a per-file integer.
With a bit of math we can estimate the upper bound to be <=block_size/16
with our current compaction strategy.
This idea was previously discarded due to the overhead of extracting the
bids/rids when we need them, but the RAM savings and file-to-integer
mapping is too useful to give up. When it became clear half-width
integers wasn't really going to work, compressed mids became the new
plan:
0bbbbbbb bbbbbbbb bbbbbbbb rrrrrrrr
^'-----------+-----------' '---+--'
'------------|-----------------|---- sign-bit, reserved for driver
| '---- nlog2(bs/16) bits for rid
| (8-bits for 4KiB blocks)
'---------------------- remaining bits for bid
(23-bits for 4KiB blocks)
To reduce the overhead of encoding/decode bids/rids a few extra features
were added to the internal mdir APIs:
1. The mtree has been changed to store mids directly. Giving each mdir
the upper bound as a weight. This allows direct lookup of mids
without any sort of bid decoding, though does bake the upper bound
estimate into the metadata of the filesystem, which isn't the
cleanest design, but if it works it works.
On the plus side, with this upper bound baked in to the filesystems,
GRMs can be encoded in a single leb128, which is nice. This may have
other savings if we ever store mids anywhere else in the filesystem.
2. rids are now mid relative in lfsr_mdir_lookup when non-negative. This
is implemented with a simple condition that is hopefully optimized
out when inlined, though there may be some room for improvement here.
3. rids are now mid relative in lfsr_mdir_commit. This was a bit tricky,
but we can leverage the existing mechanisms for bid-relative rids
used in the btree implementation.
The above changes make it so you can pass the mid around directly for
most of the mdir functions, hopefully reducing the mid decoding
overhead. This savings should only grow as more high-level filesystem
APIs are added.
Here is the resulting code/RAM changes for this entire change (from
before we adopted the mroot bit):
code stack structs
before: 20590 1784 908
after: 20890 (+1.4%) 1744 (-2.3%) 864 (-5.1%)
More on this when explaining compressed mids, but basically the idea is
instead of just storing all mdirs in our mtree as single element
entries, store each mdir in as a weighted entry, where the weight is a
known upper bound on the possible number of mid entries in a single
mdir.
With the current mid representation, this just complicates things
without much benefits. But with compressed mids it allows us to lookup
mdirs with the mid directly, and avoid decoding the bid from the mid in
some cases.
The mid-per-mdir upper bound is derived from the block size. We know:
1. Each tag needs <=2 alts+null with our current compaction strategy
2. Each tag/alt encodes to a minimum of 4 bytes
This gives us ~4*4 or ~16 bytes per mid at minimum. If we cram an mdir
with the smallest possible mids, this gives us at most ~block_size/16
mids in a single mdir before the mdir runs out of space.
Note we can't assume ~1/2 block utilization here, as an mdir may
temporarily fill with more mids before compaction occurs.
This is an intermediate commit as a part of a tangent into compressed
mids.
The idea here, is instead of using bid=-1 as a special value for mroots,
use only the top bit to indicate mroots. This allows you to compare
against the grm/other uninlined mids by masking instead of signed
comparison.
This is valuable for compressed mids since extracting bids relies on
knowledge of the block size, and becomes quite a bit more expensive.
mroot bid mroot cmp
before: 0xffffffff lfs_smax32(a, 0) == lfs_smax32(b, 0)
after: 0x80000000 (a & 0x7fffffff) == (b & 0x7fffffff)
The implementation here is a bit clumsy. I think GCC may be not that
great at optimizing out copies of structs being passed around via
inlined functions. But this is only a proof-of-concept.
Christopher Haster