There was a mismatch between the lfs3_cfg comment and the actual
crystal_thresh math where crystal_thresh=0 would break things:
- In lfs3_file_flush_, crystal_thresh=0 meant we would never resume
crystallization, leading to terrible, _terrible_, linear write
performance.
- In lfs3_file_sync and lfs3_set, it's unclear if small file commit
optimizations were working properly. I went ahead and added a
lfs3_max(lfs3->cfg->crystal_thresh, 1) just to be safe.
The other references to crystal_thresh all check for >= crystal_thresh
conditions, so shouldn't be broken (except for an unrelated bug in
lfs3_file_flushset_).
The reason for this is because crystal_thresh=1 is technically the lower
bound for this math. Allowing crystal_thresh=0 is just a convenience,
and honestly allowing it may have a been a bad idea. Maybe we should
require crystal_thresh=1 at minimum? I added a TODO.
All the new v3 config needs revisiting anyways, for defaults, etc.
---
Curiously, this actually saved code? My best guess is maybe some weird
code path in lfs3_file_flush_ was eliminated:
code stack ctx
before: 37036 2352 684
after: 37028 (-0.0%) 2352 (+0.0%) 684 (+0.0%)
We're not currently using these (at the moment it's unclear if the
original intention behind the treediff algorithms is worth pursuing),
and they are showing up in our heap benchmarks.
The good news is that means our heap benchmarks are working.
Also saves a bit of code/ctx in bmap mode:
code stack ctx
before: 37024 2352 684
after: 37024 (+0.0%) 2352 (+0.0%) 684 (+0.0%)
code stack ctx
bmap before: 38752 2456 812
bmap after: 38704 (-0.1%) 2456 (+0.0%) 800 (-1.5%)
Counterintuitively, LFS3_RDONLY + LFS3_BMAP _does_ make sense for cases
where you want to include the bmap in things like ckmeta/ckdata scans.
Though this is another argument for a LFS3_RDONLY + LFS3_NO_TRV build.
Traversals add quite a bit of code to the rdonly build that is probably
not always needed.
---
This just required another bunch of ifdefs.
Current bmap rdonly code size:
code stack ctx
rdonly: 10616 896 532
rdonly+bmap: 10892 (+2.6%) 896 (+0.0%) 636 (+19.5%)
Continued benchmarking efforts are indicating this isn't really an
optional optimization.
This brings back lazy grafting, where the file leaf is allowed to fall
out-of-date to minimize bshrub/btree updates. This is controlled by
LFS3_o_UNGRAFT, which is similar, but independent from LFS3_o_UNCRYST:
- LFS3_o_UNCRYST - File's leaf not fully crystallized
- LFS3_o_UNGRAFT - File's leaf does not match disk
Note it makes sense for files to be UNGRAFT only, in the case where the
current crystal terminates at the end-of-file but future appends are
likely. And it makes sense for files to be UNCRYST only, in cases where
we graft uncrystallized blocks so the bshrub/btree makes sense.
Which brings us to the main change from the previous lazy-grafting
implementation: lfs3_file_lookupnext no longer includes ungrafted
leaves.
Instead, functions should call lfs3_file_graft if they need
lfs3_file_lookupnext to make sense.
This significantly reduces the code cost of lazy grafting, at the risk
of needing to graft more frequently. Fortunately we don't actually need
to call lfs3_file_graft all that often:
- lfs3_file_read already flushes caches/leaves before attempting any
bshrub/btree reads for simplicity (heavy are not currently considered
a priority, if you need this consider opening two file handles).
- lfs3_file_flush_ _does_ need to call lfs3_file_graft before the
crystallization heuristic pokes, but if we can't resume
crystallization, we would probably need to graft the crystal to
satisfy the flush anyways.
---
Lazy grafting, i.e. procrastinating on bshrub/btree updates during block
appends, is an optimization previously dropped due to perceived
nicheness:
- We can only lazily graft blocks, inlined data fragments always require
bshrub/btree updates since they live in the bshrub/btree.
- Sync forces bshrub/btree updates anyways, so lazy grafting has no
benefit for most logging applications.
- This performance penalty of eagerly grafting goes away if your caches
are large enough.
Note that the last argument is a non-argument in littlefs's case. They
whole point of littlefs is that you _don't_ need RAM to fix things.
However these arguments are all moot when you consider that the "niche
use case" -- linear file writes -- is the default bottleneck for most
applications. Any file operation becomes a linear write bottleneck when
the arguments are large enough. And this becomes a noticeable issue when
benchmarking.
So... This brings back lazy grafting. But with a more limited scope
w.r.t. internal file operations (the above lfs3_file_lookupnext/
lfs3_file_graft changes).
---
Long story short, lazy grafting is back again, reverting the ~3x
performance regression for linear file writes.
But now with quite a bit less code/stack cost:
code stack ctx
before: 36820 2368 684
after: 37032 (+0.6%) 2352 (-0.7%) 684 (+0.0%)
At least at a proof-of-concept level, there's still a lot of cleanup
needed.
To make things work, lfs3_alloc_ckpoint now takes an mdir, which
provides the target for gbmap gstate updates.
When the bmap is close to empty (configurable via bmap_scan_thresh), we
opportunistically rebuild it during lfs3_alloc_ckpoints. The nice thing
about lfs3_alloc_ckpoint is we know the state of all in-flight blocks,
so rebuilding the bmap just requires traversing the filesystem + in-RAM
state.
We might still fall back to the lookahead buffer, but in theory a well
tuned bmap_scan_thresh can prevent this from becoming a bottleneck (at
the cost of more frequent bmap rebuilds).
---
This is also probably a good time to resume measuring code/ram costs,
though it's worth repeating the above note about the bmap work still
needing cleanup:
code stack ctx
before: 36840 2368 684
after: 36920 (+0.2%) 2368 (+0.0%) 684 (+0.0%)
Haha, no, the bmap isn't basically free, it's just an opt-in features.
With -DLFS3_YES_BMAP=1:
code stack ctx
no bmap: 36920 2368 684
yes bmap: 38552 (+4.4%) 2472 (+4.4%) 812 (+18.7%)
This doesn't fully replace the lookahead buffer, but at least augments
it with known bmap state when available.
To be honest, this is a minimal effort hack to try to get something
benchmarkable without dealing with all the catch-22 issues that a
self-support bmap allocator would encounter (allocating blocks for the
bmap requires a bmap, oh no).
Though now that I'm writing this, maybe this is a reasonable long-term
solution? Having the lookahead buffer to fall back on solves a lot of
problems, and, realistically, it's unlikely to be a performance
bottleneck unless the user has extreme write requests (>available
storage?).
---
Also tweaked field naming to be consistent between the bmap and
lookahead buffer.
The idea behind separate ctrled+unctrled airspaces was to try to avoid
multiple interpretations of the on-disk bmap, but I'm starting to think
this adds more complexity than it solves.
The main conflict is the meaning of "in-flight" blocks. When using the
"uncontrolled" bmap algorithm, in-flight blocks need to be
double-checked by traversing the filesystem. But in the "controlled"
bmap algorithm, blocks are only marked as "in-flight" while they are
truly in-flight (in-use in RAM, but not yet in use on disk).
Representing these both with the same "in-flight" state risks
incompatible algorithms misinterpreting the bmap across different
mounts.
In theory the separate airspaces solve this, but now all the algorithms
need to know how to convert the bmap from different modes, adding
complexity and code cost.
Well, in theory at least. I'm unsure separate airspaces actually solves
this due to subtleties between what "in-flight" means in the different
algorithms (note both in-use and free blocks are "in-flight" in the
unknown airspace!). It really depends on how the "controlled" algorithm
actually works, which isn't implemented/fully designed yet.
---
Long story short, due to a time crunch, I'm ripping this out for now and
just storing the current algorithm in the wcompat flags:
LFS3_WCOMPAT_GBMAP 0x00006000 Global block-map in use
LFS3_WCOMPAT_GBMAPNONE 0x00000000 Gbmap not in use
LFS3_WCOMPAT_GBMAPCACHE 0x00002000 Gbmap in cache mode
LFS3_WCOMPAT_GBMAPVFR 0x00004000 Gbmap in VFR mode
LFS3_WCOMPAT_GBMAPIFR 0x00006000 Gbmap in IFR mode
Note GBMAPVFR/IFR != BMAPSLOW/FAST! At least BMAPSLOW/FAST can share
bmap representations:
- GBMAPVFR => Uncontrolled airspace, i.e. in-flight blocks may or may
not be in use, need to traverse open files.
- GBMAPIFR => Controlled airspace, i.e. in-flight blocks are in use,
at least until powerloss, no traversal needed, but requires more bmap
writes.
- BMAPSLOW => Treediff by checking what blocks are in B but not in A,
and what blocks are in A but not in B, O(n^2), but minimizes bmap
updates.
Can be optimized with a bloom filter.
- BMAPFAST => Treediff by clearing all blocks in A, and then setting all
blocks in B, O(n), but also writes all blocks to the bmap twice even
on small changes.
Can be optimized with a sliding bitmap window (or a block hashtable,
though a bitmap converges to the same thing in both algorithms when
>=disk_size).
It will probably be worth unifying the bmap representation later (the
more algorithm-specific flags there are, the harder interop becomes for
users, but for now this opens a path to implementing/experimenting with
bmap algorithms without dealing with this headache.
This only works immediately after format, and only for one pass of the
disk, but it's a good way to test bmap lookups/allocation without
worrying about more complicated filesystem-wide interactions.
This abandons the data-backed cache idea due to concerns around
readability and maintainability. Mixing const/mutable buffers in
lfs3_data_t was not great.
Instead, we now just allocate an indirect lfs3_data_t on the stack in
lfs3_file_sync_ to avoid the previous undefined behavior.
This actually results in less stack usage total, due to lfs3_file_t
allocations in lfs3_set/read, and avoid the more long-term memory cost
in lfs3_file_t:
code stack ctx
before: 36832 2376 684
after: 36840 (+0.0%) 2368 (-0.3%) 684 (+0.0%)
Oh. And lfs3_file_sync_ isn't even on the stack hot-path, so this is a
net benefit over the previous cache -> data cast:
code stack ctx
before sa: 36844 2368 684
after sa: 36840 (-0.0%) 2368 (+0.0%) 684 (+0.0%)
Still less cool though.
This fixes a strict aliasing violation in lfs3_file_sync_, where we cast
the file cache -> lfs3_data_t to avoid an extra stack allocation, by
modifying the file's cache struct to use an lfs3_data_t directly.
- file.cache.pos -> file.cache.pos
- file.cache.buffer -> file.cache.d.u.buffer_
- file.cache.size -> file.cache.d.size
- (const lfs3_data_t*)&file->cache -> &file->cache.d
Note the underscore_ in file.cache.d.u.buffer_. This did not fit
together as well as I had hoped, due to different const expectation
between the file cache and lfs3_data_t.
Up until this point lfs3_data_t has only been used to refer to const
data (ignoring side-band pointer casting in lfs3_mtree_traverse*), while
the file cache very much contains mutable data. To work around this I
added data.u.buffer_ as a mutable variant, which works, but risks an
accidental const violation in the future.
---
Unfortunately this does come with a minor RAM cost, since we no longer
hide file.cache.pos in lfs3_data_t's buffer padding:
code stack ctx
before: 36844 2368 684
after: 36832 (-0.0%) 2376 (+0.3%) 684 (+0.0%)
lfs3_file_t before: 164
lfs3_file_t after: 168 (+2.4%)
I think it's pretty fair to call C's strict aliasing rules a real wet
blanket. It would be interesting to create a -fno-strict-aliasing
variant of littlefs in the future, to see how much code/RAM could be
saved if we were given free reign to abuse the available memory.
Probably not enough to justify the extra work, but it would be an
interesting experiment.
This forces our cycle detection tortoise (previously trv.u.mtortoise),
into the unused shrub leaf via pointer shenanigans.
This reclaims the remaining stack (and apparently code) we theoretically
gained from the btree traversal rework, up until the compiler got in the
way:
code stack ctx
before: 36876 2384 684
after: 36852 (-0.1%) 2368 (-0.7%) 684 (+0.0%)
And it only required some _questionably_ defined behavior.
---
It's probably not well-defined behavior, but trying to understand what
the standard actually means on this is giving me a headache. I think I
have to agree C99+strict-aliasing lost the plot on this one. Note
mtortoise is only ever written/read through the same type.
What I want:
lfs3_trv_t: lfs3_bshrub_t: lfs3_handle_t:
.---+---+---+---. .. .---+---+---+---. .. .---+---+---+---.
| handle | | handle | | handle |
| | | | | |
+---+---+---+---+ +---+---+---+---+ .. '---+---+---+---'
| root rbyd | | root rbyd |
| | | | lfs3_mtortoise_t:
+---+---+---+---+ +---+---+---+---+ .. .---+---+---+---.
| leaf rbyd | | leaf rbyd | | mtortoise |
| | | | | |
+---+---+---+---+ +---+---+---+---+ .. '---+---+---+---'
| staging rbyd | | staging rbyd |
| | | |
+---+---+---+---+ .. '---+---+---+---'
| |
: :
But I'm starting to think this is simply not possible in modern C.
At least this shows what is theoretically possible if we didn't have to
fight the compiler.
This matches other internal rbyds: btree.r, mdir.r, etc.
The intention of the single-char names is to reduce clutter around these
severely nested structs, both btrees and mdirs _are_ rbyds, so the name
doesn't really besides C-level type info.
I was hesitant on btree.leaf.rbyd, but decided consistency probably wins
here.
This comes from an observation that we never actually use the leaf cache
during traversals, and there is surprisingly little risk of a lookup
creating a conflict in the future.
Btree traversal fall into two categories:
1. Full traversals, where we traverse a full btree all at once. These
are unlikely to have lookup conflicts because everything is
usually self-contained in one chunk of logic.
2. Incremental traversals. These _are_ at risk, but in our current
design limited to lfs3_trv_t, which already creates a fully
bshrub/btree copy for tracking purposes.
This copy unintentionally, but conveniently, protects against lookup
conflicts.
So, why not reuse the btree leaf cache to hold the rbyd state during
traversals? In theory this makes lfs3_btree_traverse the same cost and
lfs3_btree_lookupnext, drops the need for lfs3_btrv_t, and simplifies
the internal API.
The only extra bit of state we need is the current target bid, which is
now expected as a caller-incremented argument similar to
lfs3_btree_lookupnext iteration.
There was a bit of futzing around with bid=-1 being necessary to
initialize traversal (to avoid conflicts with bid=-1 => 0 caused by
empty btrees). But the end result is a btree traversal that only needs
one extra word of state.
---
Unfortunately, in practice, the savings were not as great as expected:
code stack ctx
before: 36792 2400 684
after: 36876 (+0.2%) 2384 (-0.7%) 684 (+0.0%)
This does claw back some stack, but less than a full rbyd due to the
union with the mtortoise in lfs3_trv_t. The mtortoise now dominates. It
might be possible to union the mtortoise and the bshrub/btree state
better (both are not needed at the same time), but strict aliasing rules
in C make this tricky.
The new lfs3_btree_traverse is also a bit more complicated in terms of
code cost. In theory this would be offset by the simpler traversal setup
logic, but we only actually call lfs3_btree_traverse twice:
1. In lfs3_mtree_traverse
2. In lfs3_file_ck
Still, some stack savings + a simpler internal API makes this worthwhile
for now. lfs3_trv_t is also due for a revisit, and hopefully it's
possible to better union things with btree leaf caches somehow.
This is an indulgence to simplify the upcoming auxiliary btree work.
Brings back the previously-reverted per-btree leaf caches, where each
lfs3_btree_t keeps track of two rbyds: The root and the most recently
accessed leaf.
At the surface level, this optimizes repeated access to the same btree
leaf. A common pattern for a number of littlefs's operations that has
proven tricky to manually optimize:
- Btree iteration
- Pokes for our crystalization heuristic
- Checksum collision resolution for dids and (FUTURE) ddkeys
- Related rattrs attached to a single bid
But the real motivation is to drop lfs3_btree_*lookupleaf and simplify
the internal APIs. If repeated lfs3_btree_lookup*s are already
efficient, there's no reason for extra leaf-level APIs, and in theory
any logic that interacts with btrees will be simpler.
---
This comes at a cost (humorously about the same amount as the
tag-returning refactor, if you ignore the extra 28 bytes of ctx).
Unsurprisingly, increasing the size of lfs3_btree_t has the biggest
impact on stack and ctx:
code stack ctx
before: 36084 2336 656
after: 36784 (+1.9%) 2400 (+2.7%) 684 (+4.3%)
Also note from the previous commit messages: Btree leaf caching has
resulted in surprisingly little performance improvement for our current
benchmarks + implementation. It turns out if you're dominated by write
cost, optimizing btree lookups -- which already skip rbyd fetches, has
barely noticeable impact.
---
A note on reverting!
Eventually (after the auxiliary btree work) it will probably make sense
to revert this -- or at least provide a non-leaf-caching build for
code/RAM sensitive users.
I don't think this should be reverted as-is. Instead, I think we should
allow the option to just disable the leaf cache, while keeping the
simpler internal API. This would give us the best of all three worlds:
- A small code/RAM option
- Optimal btree iteration/nearby-lookup performance
- Simpler internal APIs
The only reason this isn't already implemented is because I want to
avoid fragmenting the codebase further while we're still in development
mode.
- enum lfs3_error -> enum lfs3_err
- err -> err
Really this just updates `enum lfs3_err` to match the prefixes used
everywhere else. And because enum types are kind of useless in C, this
has no effect on any other part of the codebase.
Note this includes both the lfs3_config -> lfs3_cfg structs as well as
the LFS3_CONFIG -> LFS3_CFG include define:
- LFS3_CONFIG -> LFS3_CFG
- struct lfs3_config -> struct lfs3_cfg
- struct lfs3_file_config -> struct lfs3_file_cfg
- struct lfs3_*bd_config -> struct lfs3_*bd_cfg
- cfg -> cfg
We were already using cfg as the variable name everywhere. The fact that
these names were different was an inconsistency that should be fixed
since we're committing to an API break.
LFS3_CFG is already out-of-date from upstream, and there's plans for a
config rework, but I figured I'd go ahead and change it as well to lower
the chances it gets overlooked.
---
Note this does _not_ affect LFS3_TAG_CONFIG. Having the on-disk vs
driver-level config take slightly different names is not a bad thing.
Not sure how this got overlooked. Now that graft traversals are
implemented directly in lfs3_alloc, there's no reason to store this
state globally.
Fortunately this was in a union, so it didn't actually show up in our
ctx measurements.
No code changes.
- test_traversal -> test_trvs
- lfs3_traversal_t -> lfs3_trv_t
- lfs3_btraversal_t -> lfs3_btrv_t
- t -> trv
- bt -> btrv
- lfs3_traversal_* -> lfs3_trv_*
- lfs3_btraversal_* -> lfs3_btrv_*
The traversal type is becoming one of the more fundamental types in
littlefs, and if DIR and REG both get shortened names, it makes sense
for TRV to have one as well.
This also removes the temptation to use t for traversals, which is
probably an even worse name.
---
Note that lfs3_btree_traverse, lfs3_mtree_traverse, etc, remain
unaffected. This may change in the future, but it's interesting to note
that verbs seem to need much less typing than nouns.
- lfs3_omdir_t -> lfs3_handle_t
- lfs3.omdirs -> lfs3.handles
- o -> h
- lfs3_omdir_* -> lfs3_handle_*
- lfs3_omdir_ismidopen -> lfs3_mid_isopen
From conversations with users, the term "handle" or "file handle" seems
to be the most common/easily understood term for the lfs3_file_t struct
itself. It makes sense to adopt this in our codebase.
I usually dislike inventing new names for things when prefixes can imply
a relationship (size -> ssize, cache -> rcache, shrub -> bshrub, etc),
but lfs3_omdirs_t was probably a bit much.
This replaces the `bool align` parameter that goes through all the prog
layers with an optional prog-aligned cksum stored in the lfs3_t struct.
Normally ignored, this prog-aligned cksum can be requested by setting
cksum=&lfs3->pcksum in any prog call.
Does this work? Yes. Is it a great solution? Ehhhh...
I've been tinkering with other solutions that avoid the `bool align`
parameter, but with no luck.
- `bool align`, or previously two cksum arguments, work, but create a
bit of a messy API. I'd like to find an alternative solution.
- Changing the cksum pointer to a richer lfs3_cksum_t struct with flags
also works, but would be an even messier API.
- Adding an lfs3_t side-channel, lfs3->pcache could include a pointer to
an optional prog-aligned cksum. But this would be the same/more cost
as just storing the pcksum in lfs3_t. And then we'd need to worry
about disentangling the cksum pointer on errors, etc.
- We could set a flag in lfs3->flags for alignment. This avoids the
extra 4 bytes of ctx, but still suffers from the risk of entangled
state on errors, etc.
- We could unconditionally calculate lfs3->pcksum. But then we'd be
calculating a lot of cksums we don't use (every metadata commit), and
still using the extra 4 bytes of ctx.
Lacking a good solution, using cksum=&lfs3->pcksum to indicate a
prog-aligned cksum is at least an ok solution.
I will happily change this if an alternative comes up in the future.
Another way of viewing this is that `&lfs3->pcksum` acts as a special
magic pointer value to tell the prog layers to calculate lfs3->pcksum.
A different non-NULL constant value could have worked just as well, but
those are a bit trickier to create in C.
---
Actually, there is a "better" cursed solution:
- Rely on pointer alignment to sneak a flag into the cksum pointer's
lower bits.
But, while clever, this is is outside of C's machine model and would
limit portability.
---
This trades 4 bytes of ctx for 58 bytes of code and simpler (debatable)
internal prog APIs:
code stack ctx
before: 36860 2384 652
after: 36832 (-0.1%) 2384 (+0.0%) 656 (+0.6%)
In theory this also saves stack in all the prog APIs, but none of prog
APIs end up on the stack hot-path. In our codebase the read APIs
dominate the stack thanks to block allocator traversals.
LFS3_CKDATACKSUMREADS is just too much.
The downside is it may not be clear how LFS3_CKDATACKSUMREADS interacts
with the future planned LFS3_CKREADS (LFS3_CKREADS implies
LFS3_CKDATACKSUMS + LFS3_CKMETAREDUND), but on the flip side you may
actually be able to type LFS3_CKDATACKSUMS on the first try.
Limited to nested struct fields where the names don't really matter:
- bptr.data -> bptr.d
- mdir.rbyd -> mdir.r
Ok it actually just ended up those two.
This is on the tail end of some optimization work that ended up
abandoned because of maintainability concerns. But it did highlight that
struct nesting gets a bit out-of-control when trying to both optimize
stack allocations and respect C99's strict aliasing.
Consider further fragmenting lfs3_rbyd_t for fine-grain stack
allocations:
typedef struct lfs3_rbyd {
struct lfs3_rtrunkcksum {
struct lfs3_rtrunk {
lfs3_rid_t weight;
struct lfs3_rtrunktrunk {
lfs3_block_t blocks[2];
lfs3_size_t trunk;
} rtrunktrunk;
} rtrunk;
uint32_t cksum;
} rtrunkcksum;
lfs3_size_t eoff;
} lfs3_rbyd_t;
Accessing fields just starts to get silly:
rbyd.rtrunkcksum.rtrunk.trunktrunk.trunk
At least single-char field names keeps a little bit of readability:
rbyd.ck.t.t.trunk
Or for some real examples:
- file->b.o.mdir.rbyd.weight -> file->b.o.mdir.r.weight
- bptr->data.u.disk.block -> bptr->d.u.disk.block
So we now keep blocks around until they can be replaced with a single
fragment. This is simpler, cheaper, and reduces the number of commits
needed to graft (though note arbitrary range removals still keep this
unbounded).
---
So, this is a delicate tradeoff.
On one hand, not fully fragmenting blocks risks keeping around bptrs
containing very little data, depending on fragment_size.
On the other hand:
- It's expensive, and disk utilization during random _deletes_ is not
the biggest of concerns.
Note our crystallization algorithm should still clean up partial
blocks _eventually_, so this doesn't really impact random writes.
The main concerns are lfs3_file_truncate/fruncate, and in the future
collapserange/punchhole.
- Fragmenting bptrs introduces more commits, which have their own
prog/erase cost, and it's unclear how this impacts logging operations.
There's no point in fragmenting blocks at the head of a log if we're
going to fruncate them eventually.
I figure lets err on minimizing complexity/code size for now, and if
this turns out to be a mistake, we can always revert or introduce
fragmenting >1 fragment blocks as an optional feature in the future.
---
Saves a big chunk of code, stack, and even some ctx (no more
fragment_thresh):
code stack ctx
before: 37504 2448 656
after: 37024 (-1.3%) 2416 (-1.3%) 652 (-0.6%)
This merges LFS3_o_GRAFT into LFS3_o_UNCRYST, simplifying the file write
path and avoiding the mess that is ungrafted leaves.
---
This goes for a different lazy crystallization/grafting strategy that
was overlooked before. Instead of requiring all leaves to be both
crystallized and grafted, we allow leaves to be uncrystallied, but they
_must_ be grafted (in-tree) at all times.
This gets us most of the rewrite preformance of lazy-crystallization,
without needing to worry about out-of-date file leaves.
Out-of-date file leaves were a headache for both code cost and concerns
around confusing filesystem states and related bugs.
Note LFS3_o_UNCRYST gets some extra behavior here:
- LFS3_o_UNCRYST indicates when crystallization is _necessary_, and no
longer when crystallization is _possible_.
We already keep track of when crystallization is _possible_ via bptr's
erased-state, and this lets us control recrystallization in
lfs3_file_flush_ without erased-state-clearing hacks (which probably
wouldn't work with the future ddtree).
- We opportunistically clear the UNCRYST flag if it's not possible for
future lfs3_file_crystallize_ calls to make progress:
- When we crystallize a full block
- When we hit the end of the file
- When we hit a hole
- When we hit an unaligned block
---
Note this does impact performance!
Unlike true lazy grafting, eagerly grafting means we're always
committing to the bshrub/btree more than is strictly necessary, and this
translates to more frequent btree node erases/compactions.
Current simulated benchmarks show a ~3x increase (~20us -> ~60us) in
write times for linear file writes on NOR flash.
However:
- The moment you need unaligned progs, this performance optimization
goes out the window, as we need to graft bptrs before any padding
fragments.
- This only kicks in once we start crystallizing. So any writes <
crystal_thresh (both in new files and in between blocks) are forced
to commit to the bshrub/btree every flush.
This risks a difficult to predict performance characteristic.
- If you sync frequently (logging), we're forced to crystallize/graft
anyways.
- The performance hit can be alleviated with either larger writes or
larger caches, though I realize this goes against littlefs's
"RAM-not-required" mantra.
Worst case, we can always bring back "lazy grafting" as a
high-performance option in the future.
Though note the above concerns around in-between/pre crystallization
performance. This may only make sense when cache_size >= both prog_size
and crystal_thresh.
And of course, there's a significant code tradeoff!
code stack ctx
before: 38020 2456 656
after: 37588 (-1.1%) 2472 (+0.7%) 656 (+0.0%)
Uh, ignore that stack cost. The simplified logic leads to more functions
being inlined, which makes a mess of our stack measurements because we
don't take shrinkwrapping into account.
This was quite a deep bug.
We don't track the original bshrub when grafting, so it was possible to
realloc those blocks even when we need their contents to finish the
graft operation.
This was found while experimenting with eager leaf grafting, but can
also occur when grafting data fragments.
---
In theory, the block allocator's checkpoint mechanism protects against
this.
Before we alloc, we set a checkpoint with lfs3_alloc_ckpoint. This marks
the position of the block allocator before allocation, so if we loop
around the entire block device we don't double alloc any in-flight
blocks:
ckpoint lookahead
v .---'---.
[mm---ddd-d---d-------|dd--d-ddd|--------d-----d-]
'---.---'
in-flight allocations
But this only protects _new_ blocks, _old_ blocks can be anywhere on
disk and are unprotected.
In theory again, old blocks are always tracked via copy-on-write
snapshots, but this is not the case for bshrubs while grafting!
Grafting is unfortunately a multi-commit operation (we may remove
multiple fragments that span different btree nodes), and each bshrub
commit discards the old snapshot. This creates a window where old blocks
can be double alloced _while grafting_, leading to corrupted data.
You may wonder why are we discarding the old snapshot? Why not keep
track of it until the grafting completes?
The problem there is that we need the intermediate snapshot in order for
shrubs to survive compactions. We really have 3 states:
old -> mid-graft -> new
And the only one we don't need to fallback to is the old state.
---
A couple solutions:
1. Track all three states
This would add complexity increase the cost of every lfs3_file_t.
2. Open a temporary file to track the old state
This would add complexity and a big chunk of stack to what is already
one of the critical functions on our stack hot-path.
3. Carefully make sure graft commits don't lose track of in-flight data
until an atomic commit
This doesn't work when you're trying to coalesce two data fragments
in two different btree nodes. At least not without completely
restructuring the btree commit logic.
4. Just explicitly track in-flight graft state out-of-band
This goes with option no 4., adding lfs3->graft and lfs3->graft_count to
track in-flight graft state when we're grafting. lfs3_mtree_traverse_
can include the relevant blocks during traversals, effectively masking
out graft state from the lookahead buffer.
This adds a bit of code/ctx, but is probably the cheapest option:
code stack ctx
before: 37936 2456 636
after: 38092 (+0.4%) 2456 (+0.0%) 656 (+3.1%)
Like LFS3_RDONLY and LFS3_KVONLY, LFS3_2BONLY opts-out of all of the
logic necessary for filesystems larger than 2-blocks (the mimimum size
of a mutable littlefs image).
This has potential for some pretty big savings:
- No block allocation
- No lookahead buffer
- No btrees (but yes bshrubs)
- No bptrs
- No mtree traversal
Which is I guess ~1/4 of the codebase:
code stack ctx
default: 37836 2416 636
2bonly: 27704 (-26.8%) 1872 (-22.5%) 592 (-6.9%)
This can be combined with LFS3_KVONLY for a small key-value store
compatible with the full littlefs driver:
code stack ctx
default: 37836 2416 636
kvonly: 30792 (-18.6%) 2168 (-10.3%) 636 (+0.0%)
kvonly+2bonly: 22900 (-39.5%) 1736 (-28.1%) 592 (-6.9%)
It may be possible to optimize this further, but, as is the case with
LFS3_KVONLY, balancing config-specific optimization vs maintainability
is tricky.
---
I'm not sure why, but this also reduced the default build's size a bit.
Compiler noise?
code stack ctx
before: 37860 2416 636
after: 37836 (-0.1%) 2416 (+0.0%) 636 (+0.0%)
One of the ideas behind the key-value API is that it is potentially much
cheaper than a full file API. With the key-value API, we get the
guarantee that all data must fit in RAM, and avoid headaches like
random reads/writes and needing to broadcast file state.
For an example of just how much complexity is avoided, the see the
difference between lfs3_file_flushonce_ vs the mess that is
lfs3_file_flush_ + lfs3_file_crystallize + lfs3_file_graft.
However, littlefs is designed around files, and a couple design
decisions hold back how much code saving is possible:
1. littlefs's shrubs are designed around being enrolled in the omdir
linked-list, so internally we still have most of the file open/close
code lumbering around.
2. Directories and traversals still exist, so we'd need the omdir
linked-list anyways, and we still need to broadcast _some_ changes.
3. Despite being intended for small amounts of data, lfs3_set/get can
still be used to create arbitrarily large files. So we still need all
of the bshrub/btree logic.
Which we still need for the mtree anyways, so this isn't really that
much of a downside.
It also may be possible to save more code by aggressively rewriting the
_entire_ read/write path for lfs3_set/get, to not reuse any of the
existing file logic in LFS3_KVONLY mode. But I decided against this due
to concerns around maintainability.
The duplicate lfs3_file_read + lfs3_file_readonce and lfs3_file_flush_ +
lfs3_file_flushonce_ are already enough of a concern.
Anyways, here's LFS3_KVONLY:
code stack ctx
default: 37824 2416 636
kvonly: 30936 (-18.2%) 2168 (-10.3%) 636 (+0.0%)
LFS3_RDONLY + LFS3_KVONLY is also interesting:
code stack ctx
rdonly: 10776 856 508
rdonly+kvonly: 9904 (-8.1%) 888 (+3.7%) 508 (+0.0%)
---
This also added some noise to the default build's code, mainly due to
tweaks in lfs3_file_readnext to allow better reuse in LFS3_KVONLY:
code stack ctx
before: 37824 2416 636
after: 37860 (+0.1%) 2416 (+0.0%) 636 (+0.0%)
This adds LFS3_o_WRSET as an internal-only 3rd file open mode (I knew
that missing open mode would come in handy) that has some _very_
interesting behavior:
- Do _not_ clear the configured file cache. The file cache is prefilled
with the file's data.
- If the file does _not_ exist and is small, create it immediately in
lfs3_file_open using the provided file cache.
- If the file _does_ exist or is not small, do nothing and open the file
normally. lfs3_file_close/sync can do the rest of the work in one
commit.
This makes it possible to implement one-commit lfs3_set on top of the
file APIs with minimal code impact:
- All of the metadata commit logic can be handled by lfs3_file_sync_, we
just call lfs3_file_sync_ with the found did+name in lfs3_file_opencfg
when WRSET.
- The invariant that lfs3_file_opencfg always reserves an mid remains
intact, since we go ahead and write the full file if necessary,
minimizing the impact on lfs3_file_opencfg's internals.
This claws back most of the code cost of the one-commit key-value API:
code stack ctx
before: 38232 2400 636
after: 37856 (-1.0%) 2416 (+0.7%) 636 (+0.0%)
before kv: 37352 2280 636
after kv: 37856 (+1.3%) 2416 (+6.0%) 636 (+0.0%)
---
I'm quite happy how this turned out. I was worried there for a bit the
key-value API was going to end up an ugly wart for the internals, but
with LFS3_o_WRSET this integrates quite nicely.
It also raises a really interesting question, should LFS3_o_WRSET be
exposed to users?
For now I'm going to play it safe and say no. While potentially useful,
it's still a pretty unintuitive API.
Another thing worth mentioning is that this does have a negative impact
on compile-time gc. Duplication adds code cost when viewing the system
as a whole, but tighter integration can backfire if the user never calls
half the APIs.
Oh well, compile-time opt-out is always an option in the future, and
users seem to care more about pre-linked measurements, probably because
it's an easier thing to find. Still, it's funny how measuring code can
have a negative impact on code. Something something Goodhart's law.
This adds a couple functions that treat files as simple key-value pairs:
- lfs3_get - Read a file
- lfs3_size - Get the size of a file
- lfs3_set - Write a file
- lfs3_remove - Remove a file (this one already exists!)
The idea is the only real difference between a filesystem and key-value
store in the microcontroller space is the API, and the key-value API
_is_ much easier to use.
It also opens the door to making the file API opt-out in the future to
trade code cost for feature set. littlefs will probably never be
competitive with other microcontroller-scale key-value stores, but it
may be interesting for systems already using littlefs for other storage.
And don't worry, these are still files, so they can always be opened
with the full file API when more advanced operations are needed.
These APIs also matches the custom attribute APIs, which makes sense
because they're both key-values. Any mismatch should be considered an
API bug, because the best user interface is a consistent one.
This new API is tested in tests/test_kv.toml.
---
At the moment the implementation is naive, just sitting on top of the
file API. This works remarkably well thanks to littlefs's cache
bypassing logic, but does have some downsides:
- lfs3_set always writes two commits: one for the stickynote and one for
the file sync.
Unfortunately this is a fundamental limitation of littlefs's file API.
One nice benefit of lfs3_set is in theory we can bypass this
limitation, but not if we just sit on top of the file API.
- There may be code savings from more tightly integrating the key-value
code.
This also highlighted an awkward corner case with per-file cache
configuration in which the buffer needs to be non-null even if zero. Not
the end of the world, but just a bit awkward. Maybe this deserves
revisiting in the config API rework?
---
Code changes were relatively minimal given that this is a whole new API,
unfortunately the stack took quite a hit:
code stack ctx
before: 37352 2280 636
after: 37644 (+0.8%) 2448 (+7.4%) 636 (+0.0%)
The stack surprised me, but in hindsight it makes sense. In sitting on
top of the reset of the codebase, the key-value API adds very little
code, but every stack allocation in these functions add to the stack
hot-path.
This isn't the end of the world, and it's actually probably a good thing
to have an lfs3_file_t allocated in the stack hot-path. lfs3_file_t's
size has been a bit difficult to track thanks to struct lfs3_info
dominating ctx measurements...
So:
// blablabla this is my cool function
#ifdef LFS3_COOL
int lfs3_cool(lfs3_t *lfs3);
#endif
Mainly because this reads better and moves the compilation conditions
closer to the actual declaration.
One concern is if this will interfere with future doxygen/documentation
generation, but I think we can expect future scripts to be able to parse
relevant ifdefs. For one, we want to make sure to include any required
ifdefs in generated documentation, so if a script can't even parse
ifdefs, uhhhhh...
No code changes.
rbyd.eoff has the relatively unique property of only being useful in
rdwr mode. In rdonly mode we don't care where the next erased-state
starts because we're never going to use it.
Since rbyds are used everywhere, dropping rbyd.eoff has the potential to
save a significant amount of RAM.
---
At least on paper. We were using the field in lfs3_rbyd_fetch to keep
track of the most recent valid commit perturb/eoff, which was a bit
tricky to disentangle.
Disentangling lfs3_rbyd_fetch does add a bit of code to the default
build, but saves code, stack, and ctx in the rdonly mode:
code stack ctx
rdonly before: 10640 816 524
rdonly after: 10616 (-0.2%) 808 (-1.0%) 508 (-3.1%)
default before: 37320 2280 636
default after: 37352 (+0.1%) 2280 (+0.0%) 636 (+0.0%)
In theory we could ifdef the crap out of lfs3_rbyd_fetch to claw back
this code, but 1. 32 bytes of code is really not that much code, 2. the
more rdonly and default diverge the more likely rdonly breaks, and 3. I
think the new code is a bit more readable since it avoids masking
perturb/eoff together until the last minute.
We don't need the staging shrub if we never stage shrubs!
The only hangup was reuse of the staging shrub to load bshrubs/btrees in
lfs3_file_fetch (we need to be able to fallback to the previous shrub if
we error in lfs3_file_resync), but this can be handled with a stack
allocated shrub.
If btree-leaf-caches make a return, we would need to stack allocate this
anyways due to the lopsided cost of the main/staging btrees/bshrubs
introduced to avoid wasting space on the useless
staging-shrub-leaf-cache.
This saves some code in LFS3_RDONLY, and apparently an instruction or
two in the default build (I guess stack loads/stores are cheaper?):
code stack ctx
rdonly before: 10680 840 524
rdonly after: 10640 (-0.4%) 816 (+0.0%) 524 (+0.0%)
default before: 37324 2280 636
default after: 37320 (-0.0%) 2280 (+0.0%) 636 (+0.0%)
It's not apparent in ctx because lfs3_info.name dominates (guh), but
this does save some RAM in lfs3_file_t:
rdonly ctx
lfs3_file_t before: 136
lfs3_file_t after: 112 (-17.6%)
It does add some stack cost to lfs3_file_fetch, but because this isn't
on the stack hot-path in either build, we don't really care:
default code stack ctx
lfs3_file_fetch before: 372 416 0
lfs3_file_fetch after: 368 (-1.1%) 440 (+5.8%) 0 (+0.0%)
This is the new readonly flag, to be consistent with LFS3_M_RDONLY and
friends.
Note this overlaps with LFS3_YES_RDONLY in a weird way, where
LFS3_YES_RDONLY is basically just an alias for LFS3_RDONLY. For most
flags, LFS3_THING enables the _option_ of using LFS3_M_THING, with
LFS3_YES_THING implying LFS3_M_THING in all mount calls. But
LFS3_RDONLY _disables_ the option of using LFS3_M_RDWR, so it's a bit
different...
Do we really need two flags for the same thing? Not sure. But most users
probably expect LFS3_RDONLY coming from other filesystems.
Worst case this can be revisited in the planned config API rework.
---
As for the readonly code size, this is just the first draft and limited
to mostly ifdefing out all prog/write logic paths. There's some TODOs in
the code that may save a bit more (rbyd.eoff, file.b.shrub_ for
example). But the results are looking ok:
code stack ctx
v2.11.0 rdonly: 6270 448 580
v3-alpha rdonly: 10776 (+71.9%) 840 (+87.5%) 524 (-9.7%)
It's interesting to note most of the additional code/stack cost come
from filesystem traversal. In v2, the threaded linked-list made rdonly
traversal _incredibly_ cheap. But the extra rdwr baggage of turning
littlefs into a fully connected graph made it something to be avoided
in v3.
This hits v3 with the double whammy of:
1. Filesystem traversal is more complicated since we need to keep track
of which btree and where in the btree we are
2. Everything needs to be tracked explicitly due to the new inverted
state-machine driven API (no callbacks)
Note that even if we disabled the traversal APIs, lfs3_fs_usage, cksum
checking, etc, we'd still need to traverse to rebuild gstate. Otherwise
we risk showing grmed files after a powerloss.
---
This did affect the default build a little bit, due to moving things
around for nicer ifdef groupings:
code stack ctx
default before: 37300 2280 636
default after: 37304 (+0.0%) 2280 (+0.0%) 636 (+0.0%)
Christopher Haster