While convenient, file.size is redundant info. Redundant info always
has the risk of falling out-of-sync, creating difficult to find bugs.
This was made especially apparent with dropping file-level idempotent
errors, which make possible file states quite a bit more complex (we've
given up on fully reverting errors, but we don't want errors to make the
filesystem inconsistent).
Replacing file.size with an inlinable function that derives the file
size removes this risk without too much cost. As a plus, lfsr_file_t is
one word smaller:
code stack lfsr_file_t
before: 33286 2968 112
after: 33278 (-0.0%) 2976 (+0.3%) 108 (-3.6%)
This is a big compromise in robustness in the face of errors, vs code
size and stack size.
To be clear, errors here refer to runtime errors, such as ENOSPC, EIO,
etc, not on-disk errors, though disk errors could result in a runtime
error if unrecoverable. The question is what happens to the filesystem
after it reports one of these errors to the user. Operations are
necessarily interrupted, so some in-device state may be lost.
The way I see it, there are three options, increasing in robustness, but
also increasing in code/stack cost:
1. Leave the filesystem in an inconsistent state, require an
unmount+mount cycle to continue using the filesystem.
2. Use on-stack copies to prevent corrupted state until disk commits
complete. This does not protect against intermediary states during
file operations.
3. Use on-stack copies and file snapshots to fully revert any failed
filesystem operation.
My initial thought, since this is supposed to be a robust filesystem,
was that we should try 3., fully revert any failed filesystem operation.
With some stack tradeoff, this isn't too much of a problem, until we get
to files. Files present some real problems:
1. File bshrubs need to be tracked in order to be compacted correctly.
This means our on-stack copies need to be tracked, which complicates
things a bit.
2. Bshrub estimates need to conservatively include all snapshots to
avoid compaction issues. This means if we are tracking on-stack
copies, we are effectively multiplying bshrub cost by ~3x vs ~2x.
3. We only have one file buffer. Being able to revert buffer updates
would require either unecessary disk flushes and some weird mechanism
to handle small files, or ~2x the RAM cost.
See the previous commit for more info on this.
These issues can _probably_ be worked around, with some tradeoffs, but I
think the writing is on the wall. Full reverts on errors just isn't
worth the cost for littlefs's use case.
With the snapshotting features of littlefs, it shouldn't be too hard to
still handle errors gracefully in littlefs, either by keeping two file
handles around, or reopening the file after an error.
This adds cost at the user-level, but consider that the alternative is
that all users pay roughly this cost at the filesystem-level.
Maybe in the future we should additional LFS_GLASS/LFS_TEMPERED modes to
provide all three of the above options? Let the user chose their
robustness vs code/RAM tradeoff?
---
To be clear, this change makes it so all filesystem operations are
error-idempotent, with the exception of the _contents_ of files after an
error. If an error occurs during a file operation, the contents of that
file is undefined (but also desynced, so disk is unaffected).
Code changes:
code stack
before: 33964 3080
after: 33286 (-2.0%) 2968 (-3.6%)
With these changes, O_SYNC/O_FLUSH are also much cheaper to implement.
We can see their specific costs, which, to be honest, is a bit more than
I expected since these are now just a flag check and function call:
code stack
default: 33174 2944
O_SYNC: 33224 (+0.2%) 2968 (+0.8%)
O_SYNC+O_FLUSH: 33286 (+0.3%) 2968 (+0.8%)
Now mixing in truncate/fruncate, along with desync<->sync state
transitions.
Found bugs:
- Fixed propagating LFS_F_UNSYNCED/LFS_F_UNFLUSHED state during sync
broadcasts. This is important for tracking small files correctly.
- We were not clearing the btree erased-state of other opened file
handles when we started using it, leading other file handles to have
out-of-date erased-state.
I considered moving this into lfsr_btree_commit, but file btrees are
really the only place where shared references make sense, and it feels
weird to scan file btrees every time we commit to the mtree.
- Fixed syncs not propagating to other file handles when file is synced
with disk.
It's interesting that lfsr_file_sync can actually have an effect on
the system when the disk in is-sync.
- Added O_FLUSH/O_SYNC support to lfsr_file_truncate/fruncate. This
omission was just an oversight.
Unfortunately this did add quite a bit more complexity to both
functions.
You may notice in the fix for that last bug, that lfsr_file_ftruncate
sort of drops the ball with regards to error-idempotency. This is
because, as I was trying to figure out how to recoverably move the
buffer around when fruncating small files, I realized we don't handle
small files in lfsr_file_write correctly w.r.t. error-idempotency, and
that fixing this may be intractable...
The issue is how handle overwrites for unflushed buffers.
In general, the correct thing to do when an incoming write overlaps our
file buffer, is to just write over the buffer with the new data.
Ah, but if we do this, how do we get the old data back if we run into an
error writing the data to disk? It's gone!
For normal files, this is not an issue. We can always flush to disk to
reclaim our buffer, and since a flush doesn't change the file contents,
it's fine to make this our new fallback state.
But for small files, flush is a noop, we keep these entirely in RAM.
There are some possible workarounds:
- Flush small files to disk before overwriting, sort of defeats the
purpose of caching these in RAM...
- Reread small files from disk, because that's definitely what you want
to do when you hit an error...
Also, to always have something we can read from disk implies flush
on overwrite, see above.
- Sacrificing half our buffer for staging small files. Because RAM cost
is totally not a priority...
Long story short, rethinking idempotent errors.
A recent change, motivated by user feedback, was to delay write buffer
flushes as much as possible. Before, littlefs would always flush the
buffer during lfs_file_seek, but now, buffer flushes can be delayed all
the way to lfsr_file_read, or even skipped entirely thanks to bypassing
reads.
This is all fine and dandy, except it's easy to imagine a use case where
a user might really not want a _write_ error to pop out of a _read_
call.
With this new behavior, avoiding this situation is impossible.
So enters a function common to other filesystems: lfsr_file_flush.
However it's value is quite a bit different here. Unlike flush in other
filesystems, this flush does not necessarily make data accessible on
disk. It only writes to the pending file snapshot, which is not
accessible until lfsr_file_sync.
This makes flush a function with a rather narrow scope in littlefs
(pretty much just preventing write errors in read), but since we had
already implemented this function for internal plumbing, it adds _very_
little cost.
I'm more concerned about potential user confusion around sync vs flush.
Curiously, exposing lfsr_file_flush actually _saved_ code size for some
reason. Not sure what would make that happen:
code stack
before: 33544 3072
flush: 33536 (-0.0%) 3072 (+0.0%)
flush+O_FLUSH: 33548 (+0.0%) 3072 (+0.0%)
The motivation for this comes from the observation that many users call
sync on every file write. Much more than I expected. I think one reason
is in embedded systems it's common to just write structs to disk, either
the whole file or to a log.
O_SYNC exists in POSIX/Lunix/etc, so it makes sense to provide in
littlefs. In theory it's just one extra function call, and may even save
in total application cost (though we don't measure this) by reducing the
number of function calls at the application-level.
---
Unfortunately in-practice turned out to be quite a bit different than
in-theory... The main culprit being the improved guarantees around error
atomicity...
The ideal guarantee is that if there is an error during a write, the
entire write operation is reverted. Combining this with O_SYNC means we
need to hold a copy of the origin file state all thwe way through our
sync call. This got a bit messy...
The annoying part isn't even the functionality! Our system of tracking
btree/bshrub snapshots is quite robust! The problems were entirely with:
1. Figuring out how the heck to avoid clobbering the old file buffer
state.
2. Figuring out how the internal APIs should work while passing around a
bunch of staging state.
For 1., fortunately, thanks to bypassing writes, and some careful
pointer manipulation, we can void buffer clobbing. And for 2. just some
internal API work was needed. Internally all syncs end up in
lfsr_ftree_sync, though this feels a bit clumsy since the functionality
is not really ftree related...
Unfortunately, all of this added up to quite a bit more code cost than
I had hoped. In theory, adding some sort of LFS_CERAMIC/LFS_GLASS modes
that relax error atomicity for code size could help with most of this?
But it needs some thought:
code stack
before: 33324 3072
after: 33544 (+0.7%) 3072 (+0.0%)
We end up passing intmax_t pointers around, but without a cast. This
results in a warning. Adding a cast fixes the warning. This is in the
printing logic, not the actual comparison, so hiding warnings with this
cast is not a concern here.
I also flipped the type we compare with to use the right-hand side. The
pretty-assert code already treats the right-hand as the "expected" value
(I wonder if this is an english language quirk), so I think it makes
sense to use the right-hand side as the "expected" type.
While I'm happy to have figured out the previous range update logic, it
is quite complicated, and that translates to code cost. Updating the
buffer with the write tail is 1. simpler, and 2. keeps the buffer
updated with the most recent relevant data.
Relying on the previous bypassing/buffer strategy would have been error
prone anyways. The optimization could fall apart the moment you
_increase_ the buffer size, and it doesn't extend well to multiple open
file handles.
The problem with multiple file handles is we don't know what range of
bytes have been affected when we broadcast a sync. So we just overwrite
all other open file buffers.
Fortunately, LFS_O_DESYNC provides an interesting way to avoid this, and
optimize in-file readd->writes.
Code changes, not really that much in the grand scheme of things. For
comparison I included the code cost for unconditionally clearing the
buffer during bypassing writes:
code stack
update overlap: 33356 3072
clear buffer: 33292 (-0.2%) 3072 (+0.0%)
overwrite buffer: 33316 (-0.1%) 3072 (+0.0%)
None of the available options sit well with me.
Worst case writes states after an error:
1. Maintain on-stack snapshots for entire write operation:
on-disk: abcdefghijklmnopqrstuvwxyz
write: JKLMN
error!
on-disk: abcdefghiJKlmnopqrstuvwxyz
2. Maintain on-stack snapshots for lfsr_ftree_carve:
on-disk: abcdefghijklmnopqrstuvwxyz
write: JKLMN
error!
on-disk: abcdefghijklmnopqrstuvwxyz
3. Don't maintain on-stack snapshots, rely on btree/bshrub atomicity:
on-disk: abcdefghijklmnopqrstuvwxyz
write: JKLMN
error!
on-disk: abcdstuvwxyz
Something else to consider, the on-stack snapshots increase pressure on
the available shrub_size, which must include all tracked bshrubs in the
mdir, and currently doesn't deduplicate more than checking for identical
trunks. In effect, shrubs are limited to ~shrub_size/3, which isn't
great...
Since we can't get rid of the extra shrub cost when atomic carve
operations, I'm going to revert this, since we might as well just track
all file operations and provide a fully atomic API... Element of least
surprise and all thath...
But this revert may itself be reverted in the future.
Maybe we should provide some sort of LFS_LESSATOMIC flag to allow opt-in
to non-atomic file writes for code/stack savings?
This is an attempt to reduce the overhead of on-stack snapshots during
writes, by relaxing the gaurantees provided by lfsr_file_write during
errors.
Before, thanks to the on-stack snapshot, file writes could revert to the
previous state if an error occurred. Now, on-stack snapshots are limited
to lfsr_ftree_carve, so only the state change in lfsr_ftree_carve is
reverted.
This should behave relatively predictably, since lfsr_ftree_flush calls
lfsr_ftree_carve in a normal order. If an error occurs, some, none, or
all of the data is actually written. For truncate/fruncate, there is
only one call to carve, so these remain atomic.
It's tempting to want to push this lower. If you push the atomic
operations down to the btree/bshrub level, individual btree/bshrub commits
are already atomic, so tracking on-stack bshrubs could be dropped
completely. But failures at the sub-carve level get weird! Thanks to
block crystallization and the use of order-statistic operations, the
resulting file can be quite unpredictable. For example:
on-disk: abcdefghijklmnopqrstuvwxyz
write: JKLMN
error!
on-disk: abcdstuvwxyz
With atomic carves, worst case is something like this:
on-disk: abcdefghijklmnopqrstuvwxyz
write: JKLMN
error!
on-disk: abcdefghiJKlmnopqrstuvwxyz
Thanks to file-level snapshotting, the original file can still be
recovered (and is on-disk until sync is called), but the
unpredictability is concerning.
Alternatively, if we had btree range removals, lfsr_ftree_carve could be
entirely atomic (and more efficient).
Unfortunately, btree range removals still seem like they would be
difficult to implement, and will probably be out of scope for some time.
The added code cost will also likely outweigh any savings from dropping
on-stack bshrub tracking. Still, this is probably worth looking into in
the future.
Code changes:
code stack
before: 33356 3072
after: 33006 (-1.0%) 2992 (-2.6%)
The condition for checking if the left fragment could be coalesced was
wrong, preventing a common coalescing chance in linear rewrites, and
leaving a weird 1-flush-sized alignment issue in the fragments:
Was: lfsr_data_size(&bptr.data) < lfs->cfg->fragment_size
Should be: fragment_end - (bid-(weight-1)) <= lfs->cfg->fragment_size
This was correct for the right fragment coalescing, not sure how the
left ended up messed up.
(Actually I do know why, the math here is dense, complex, and subtle,
a nasty combo.)
Desynchronized files are a new concept intended to capture some useful
quirks of the previous multiple-open-file behavior.
This adds:
- LFS_O_DESYNC - Mark a file as desync during open
- lfsr_file_desync - Mark a file as desync whenever
- lfsr_file_sync - Mark a file as NOT desync, and sync the file
Desynced files:
1. Don't recieve updates from writes to other file handles. This makes
desynced files act as a sort of snapshot of the file at the time it
was marked desync.
2. Don't call lfsr_file_sync on close. Unless lfsr_file_sync is
explicitly called, changes to desynced files are not reflected on
disk and not broadcasted to other file handles.
A side-effect of 2., is that this gives you a quick way to abort a file
write. Marking a file as desync and then closing the file will never
error.
Additionally, if an error occurs during a write operation, the file is
implicitly marked as desync. This provides graceful write aborting in
unlikely error cases. This has actually always been a feature in
littlefs, it was just named differently and didn't have an optional
recovery mode.
Since littlefs actually has to do more work to keep files in sync, the
desync feature is quite cheap:
code stack
before: 33324 3072
after: 33360 (+0.1%) 3072 (+0.0%)
- Uninitialized warning in lfsr_mtree_namelookup because we adjust
mdir.mid unconditionally even if there is an error.
Putting the mid adjustment behind a check for ENOENT _should_ fix
this, but try as I might, GCC just can't figure out what's going on.
Adding a blanket rid_=0 in lfsr_mdir_namelookup is the best workaround
I can think of right now.
- Uninitialized warnings caused by not knowing if err is <0 in functions
where err gets merged with positive results.
I think this stems all the way from bd functions, since those poke
outside of littlefs. But it's a good idea to assert on this in
functions where err needs to be <0 anyways.
The __builtin_unreachable in lfs_util.h makes this double as a
compiler hint even in release mode, which is nice.
- GCC thinks rid is uninitialized in lfsr_btree_commit when checking
if left sibling can be merged.
As far as I can tell this actually can't happen because of the above
parent.trunk=0 check. It's not to hard to look at all code paths that
set parent.trunk=!0, they also set rid to some valid value.
I think the cause is GCC giving up because gotos are involved.
These uninitialized warnings are especially annoying since they are tied
to optimization passes. Which means they only show up when compiling -Os
and show up inconsistently (I don't know if there's more behind -O3 for
example).
This combined with frequent false positives, exacerbated by our heavy
use of out-pointers, makes this warning quite frustrating to deal with.
I would be tempted to turn this warning off if is wasn't so valuable.
Now setting estimate=-1 will force a recalculation on the next bshrub
commit. This centralizing the annoying bshrub estimate calculation, so
code that allocs/fetches bshrubs can be simplifies to just struct
assignment.
As a plus, you don't pay the estimate calculation cost when only reading
a file.
Also dropped lfsr_bshrub_alloc/fetch, these weren't really useful
functions and their naming could be misleading.
I wonder if bshrubs will ever stop feeling like a big hack.
Also tweaked bshrub estimates a bit so they only include
bsprouts/bshrubs. I think we can get away with this by including
max(bptr, trunk, btree) + shrub_size when we get around to calculating
file limits.
We are now including an extra tag when estimating bsprout size, but
that's not the end of the world. At least all calls to
lfsr_bsprout_estimate__ are consistent now.
These changes were more for code organization/runtime improvements, the
benefit to code cost is minimal:
code stack
before: 33364 3072
after: 33336 (-0.1%) 3072 (+0.0%)
We weren't updating other opened files with bshrub estimates, which
meant there was a risk bshrub estimates fall out of date, and mdir
commits lock up with ERANGE.
We need more testing over these estimate conditions. Unfortunately
they're a big headache in the new design...
The bshrub estimate now expects to include all synced struct contents,
so we need to set it to something greater than zero when converting from
a bsprout/bptr/btree.
But honestly, all of this bshrub estimate stuff needs work...
1. We used a temporary bshrub outside of the tracked ftree to commit
without clobbering the ftree. This breaks the invariant that all
bshrubs are tracked during mdir commit.
By luck, the bshrubs were still working. They normally don't get
staged properly without this invariant, but:
1. These are always new bshrubs, so no state for mdir compact to be
aware of.
2. These are always immediately SHRUBCOMMITed, so they happen to get
staged.
But we can avoid breaking this invariant by encoding the commit
first. We may clobber the ftree, but this is what our on-stack ftree
copies are for.
2. We weren't including becksums when converting bptrs to bshrubs.
Maybe we shouldn't care about this, since we don't create direct
bptrs in normal operation. But I guess we'll be compat friendly for
now...
Now, when files are synced, they broadcast their disk changes to any other
opened file handles. In effect, all open files match disk after a sync
call to any opened file handle pointing to that file.
This was a much requested feature, as the previous behavior (multiple
opened file handles maintain independent snapshots) is pretty different
from other filesystems. It's also quite difficult to implement outside
of the filesystem, since you need to track all opened files, requiring
either unbounded RAM or a known upper limit.
---
A bit unrelated, but this commit also changes bshrub estimate
calculation to include all opened file handles. This adds some annoying
complexity, but is necessary to prevent sporadic ERANGE errors when
the same file is opened multiple times.
The current implementation just refetches on-disk metadata. This adds
some maybe unnecessary metadata lookups, but simplifies things by
avoiding the tracking of on-disk sprout/shrub size, which risks falling
out of date. Keep in mind we only recalculate the estimate every
~inline_size/2 bytes written.
Just like lfsr_mdir_estimate, this scales O(n^2) with the number of
opened files (this are basically the same function... hmmm... can they
be deduplicated?). This is unlikely to be a problem for littlefs's use
case, but just something to be aware of.
Code changes:
code stack
before: 32920 3032
after: 33192 (+0.8%) 3048 (+0.5%)
- Reworked *_estimate functions to use swapping bounds/variables much
like lfsr_rbyd_appendattr.
- Merged *_estimate_ into *_estimate. Mainly for (perhaps misdirected)
code cleanliness reasons. The compiler is already going to be inlining
these single-calls since inlining is always worthwhile.
- Renamed internal mdir-commit-related functions to have the __ suffix
even if there's not a direct naming conflict.
This is a bit of a weird naming scheme, but it's useful to hinting
that all the *__ functions are at the same logic level. It also hints
that you probably shouldn't call these unless you're dealing with mdir
internals. Functions like lfsr_mdir_compact__ will probably just break
if called outside of lfsr_mdir_commit.
- Moved *_estimate functions to be closer to *_compact functions, since
these are closely related (*_estimate is basically a soft *_compact).
These changes saved a bit of code, but added a surprising amount stack
cost. I'm guessing this is related using swap functions. Maybe ptr
aliasingis causes problem, or swapping breaks compiler invariants about
variable locations:
code stack
before: 32948 2984
after: 32920 (-0.1%) 3032 (+1.6%)
Maybe we should consider an alternative impl for both the *_estimate
and *_appendattr variable swapping...
This adds more internal functions to deduplicate things:
- lfsr_bsprout_isbsprout
- lfsr_bsprout_isbleaf
- lfsr_bsprout_size
- lfsr_bsprout_cmp
- lfsr_bshrub_commit__
- lfsr_bsprout_estimate__
- lfsr_bshrub_estimate__
- lfsr_bsprout_compact__
- lfsr_bshrub_compact__
This doesn't actually save that much code, I guess our attr-list
iterations are quite cheap, but it does make mdir_commit__/compact__
a bit easier to read:
code stack
before: 32964 2984
after: 32948 (+0.0%) 2984 (+0.0%)
Since we don't need this for bshrub/bsprout negotiation, we might as
well drop it to keep concerns separate.
code stack
before: 32980 3008
after: 32964 (+0.0%) 2984 (-0.5%)
Now that bshrubs are limited to files, we might as well lean into it.
Instead of relying on extra traversals through our attr-list, we now
rely on bsprouts/bshrubs always existing in the opened mdir list. This
is a much more robust way to deduplicate bsprout/bshrub compaction
operations. The previous implementation already had a bug since
bsprout/bshrub calculation was forgotten from lfsr_mdir_estimate_.
This is a direct tradeoff of code and RAM. Though note the previous
impl was incomplete and would likely need a bit more code:
code stack
before: 33184 2944
after: 32980 (-0.6%) 3000 (+1.9%)
Now instead of relying on lfsr_f_isunsynced, which will probably have
issues with multiple opened files, we do O(n^2) scan sover the opened
file list to figure out what hasn't already been compacted.
What's interesting is the slight difference in lfsr_mdir_estimate_ and
lfsr_mdir_compact__. In lfsr_mdir_compact__, by updating all opened
bshrubs/bsprouts, we trivially prevent duplicate compactions. But for
lfsr_mdir_estimate_, we need to prevent duplicates without modification.
So instead, we only include the _last_ reference in the opened file
list. This avoids duplicates while also avoiding multiple mdir lookups.
The remaining puzzle is including in-flight in-attr bshrubs. Traversing
over two data-structures is already complicated/expensive. Traversing
over three might be a bit much...
Before, rdonly buffers were dropped, requiring a re-read for files open
RDWR. Now the buffer is updated with whatever data overlaps.
This doesn't add _that_ much code cost, and may be beneficial for
certain rd/wr patterns in different parts of a file (we don't drop
buffers at all in bypassing writes). Though it may be better to open two
separate file handles in this use case...
We will need this logic anyways for updating multiple in-sync opened
file handles. And maybe this logic can be deduplicated then.
code stack
before: 32848 2944
after 32908 (+0.2%) 2944 (+0.0%)
This can happend if we read some data into our buffer (or write+flush+seek
in some weird pattern) and then do a bypassing write that overlaps the
buffer.
The solution here is to make sure the buffer is cleared when doing
bypassing writes.
In theory, we could be a bit smarter by checking and only clearing when
overlap occurs, or even updating the buffer with the new contents like
we do in the bd cache layer, but the value would probably be minimal.
File writes are already expected to clobber buffered reads.
This is more a cludge than a real fix, we should probably be handled in
the bd-wrapper layer, but that's all a mess right now and slated for
future work.
The problem is that our "rewinding" of block-level progs for alignment
leaves our bd caches with stale data. This is normally fine, but then we
go read the stale data to calculate our becksum.
Dropping our caches before calculating the becksums fixes this, at least
temporarily.
This will always fail due to prog alignment, but we won't notice because
the align flag gets set in the flush loop. So it's not a _hard_ error,
but results in an unnecessary btree commit and a bunch of extra reading.
Some of these checks aren't necessary if crystal_thresh >= prog_size,
which should always be the case, but wanting crystal_size=0 to be a
shorthand for "no crystallization" makes it's possible. Maybe we should
set crystal_thresh = lfs_max32(crystal_thresh, prog_size) in the
future...
Unfortunately we don't really have a way to prove that block-level
erased-state checksums are working short of benchmarking, so sure
enough block-level erased-state checksums were not working.
Some of the issues were a bit silly:
- Forgot so zero-init the checksum.
- Typo meant we were calculating the becksum of the wrong block.
- Flushing when buffer is non-empty triggered redundant flushes if the
buffer was filled for reading.
The last one is a bit more fundamental.
Humorously, it wasn't caught earlier because the buffer is always
in-sync with disk. So redundant flushes aren't an _error_, but they sure
hurt performance.
The fix is a little bit tricky. We want to flush only when
LFS_F_UNFLUSHED is set, but this confuses lfsr_file_write into thinking
it can clobber small file buffers.
The solution here is to only flush when LFS_F_UNFLUSHED is set, always
set LFS_F_UNFLUSHED on small files, which makes a bit of sense if you
think about it.
This means LFS_F_UNFLUSHED can be set on read-only and in-sync files,
but that should hopefully not be an issue.
This is useful for debugging checksum mismatches on disk.
And since dbgblock.py has some relatively flexible options for slicing
the disk, this can be used to find the checksum of any on-disk data
pretty easily.
I think ctags's defaults may have changed recently or something, but
it's struggling to find function definitions when there's a
predeclaration in the same file.
Adding --fields=+n (line numbers) fixes this.
This lets us cache more of each btree entry during reads, since the
buffer is no longer filled with unlikely-to-be-relevant, earlier btree
entries.
Also added lfsr_ftree_readnext for reading one btree entry, which is a
bit more flexible than the hint mechanism.
Code changes:
code stack
before: 32776 2944
after: 32804 (+0.1%) 2944 (+0.0%)
Now all functions maintain the invariant that "small files" (inlinable,
bufferable, single-fragment files, these can be stored in a single data
attr) always reside entirely in the file's buffer.
This invariant wasn't strictly maintained before, unfortunately it's
tricky to maintain for truncate/fruncate while also providing error
recovery without unnecessary data flushes. But after dealing with a
number of problematic corner cases related to small files, I decided to
just do the unnecessary data flushes.
At the very least, by also checking for truncate/fruncate calls where
the data is already available, we can avoid pathological cases such as
small file logging. With this tweak, unnecessary data flushes should be
somewhat uncommon.
In addition to providing a strong invariant, handling small file caching
in truncate/fruncate is nice in that the complicated logic is entirely
contained in truncate/fruncate, meaning you don't pay the code cost if
you don't use these functions.
Code changes:
code stack
before: 32816 2936
after 32776 (-0.1%) 2944 (+0.3%)
This trades off fewer bus transactions when performing small linear
reads for the possibility of reading more data than is strictly
needed. It also introduces potential buffer thrashing when opening a
file RDWR, but this can be avoided by using two different file handles.
The current implementation only reads at most one fragment/block past
the requested read into the buffer. Reading any more doesn't really
save any reads, since we need to do an additional btree lookup for each
fragment/block.
This change was motivated by the observation that not using the buffer
in lfsr_file_read absolutely destroys performance when doing byte-level
linear reads.
In theory, buffering could be left up to the users, but buffering in the
filesystem allows us to be a bit smarter since we know the exact tree
layout.
---
This effectively reverts the file buffer changes to be much closer to
the previous version, though instead of flushing in lfsr_file_seek, we
flush in lfsr_file_read. This may be surprising to users, but we _must_
flush in one of these two functions. Flushing in lfsr_file_read moves
flushing to the last possible moment, increasing the chances we can
avoid flushing entirely.
This also makes lfsr_file_seek not touch disk at all, which is nice.
At the very least it may be useful to expose lfsr_file_flush to the
users so flushing in lfsr_file_read can be manually avoided...
This is intended to enable lfsr_file_read to use the buffer as well.
This adds LFS_F_UNFLUSHED and internal lfsr_file_flush to manage buffer
flushing. This also results in a nice reorganization of lfsr_file_sync.
Of course, small file caching proves to be a big pain again, with
several subtle corner cases in truncate/fruncate that need to set the
LFS_F_UNFLUSHED flag so we fix small files in lfsr_file_sync.
Note sure why we weren't hitting this earlier, but I've been hitting
this race condition a bunch recently and it's annoying.
Now every failed process kills the other test processes unconditionally.
It's not clear if this actually _fixes_ the race condition or just makes
it less likely, but it's good enough to keep the test script user
friendly.
This restores the previous handling of inlined files, which was a strict
requirement in earlier versions, now more of an optimization.
littlefs now tries to keep small inlinable files in RAM. This doesn't
always work because inlined files can be quite large, so littlefs only
tries if it knows the inlineable file fits in the file's buffer.
I had a bit of a hard time working truncate/fruncate into the scheme. It
seems simple on paper: if truncate/frunate make a file "small", move it
into our buffer. But if we already have data in our buffer, we can't make
the file small without potentially clobbering the buffer. This conflicts
with error recovery and leaves us in a bit of a bind.
Two options:
1. Flush the file on truncate/fruncate so we can use the buffer for
caching small files.
This risks unnecessary disk writes. Consider fruncating a small
log you know fits in the file buffer.
2. Allow small files to sometimes not be cached in the buffer. Fix this
state on sync.
This risks bugs caused by relying on the sometimes-incorrect small
file invariant.
This can still also cause unnecessary disk writes if you, say, write
to the file between truncate and sync.
This goes with the option 2. I think option 1 has more failure cases
that can be problematic and option 2 allows us to avoid disk writes in
more cases. But it isn't clear what the best option is.
While definitely winning cool points, uncrc32cs have a number of
problems:
1. uncrc32c is relatively unflexible, being limited to only CRC-related
checksums, and probably violating some properties of cryptographic
hashes if possible there.
2. Code savings are minimal, a reversed crc32c implementation is a only
a little less costly than the logic to save aligned CRCs, and since
it's not on the hot-path, the stack cost is ~zero.
3. uncrc32c may come with a high computation cost.
We aren't measuring this, but uncrc32c either operates at the
bit-level, or requires a second set of tables which is unreasonable
for littlefs's use case.
With uncrc32c you need to update the checksum based on every bit in
the extra unaligned data, up to prog_size. With flcksums it's just a
copy of a word, and prog_size has no impact.
So for now dropping uncrc32c, though this can always be reverted in the
future.
Checksumming unaligned data during block compaction is surprisingly
tricky. We don't know if our data will be aligned until after
a potentially unbounded number lookups, we need to write data into our
pcache as we go to avoid unnecessary lookups, but if we end up unaligned
we need to revert our checksum to the checksum of the aligned data.
The way I see it there are 4 options:
1. Calculate the checksum after writing data into the block.
This is the most expensive option, requiring a full second read of
the data to calculate the checksum. It is simple though.
2. Do a pass over the btree to figure out alignment before writing.
This at least only reads metadata twice, so is more efficient than
the 1st option.
3. Keep track of the aligned checksum on each flush, falling back to the
last flushed checksum if we need to correct alignment.
This solution is flexible though requires some extra state to track
multiple checksums.
4. Leverage the math behind CRCs to run the CRC backwards when we
truncate for alignment.
This works, though a bit inefficiently, but is strictly tied to
CRC-related checksums.
By inefficient I mean that we would likely be limited to a bit-level
"uncrc32c". It's possible to create nibble/byte tables for uncrc32c,
but this adds significant code cost for a relatively uncritical
function.
I was hopeful that we could leverage the existing tables in both
functions, but unfortunately it doesn't work out like that. You could
scan the crc32c table to find the constant to reverse, but this
requires ~16*2 or ~256 operations vs "naive" ~8 operations per byte.
This commit implements both 3 and 4, defaulting to 4 unless
LFS_NO_UNCRC32C is defined.
The current lfs_uncrc32c implementation is a simple bit-level
implementation, but does allow for crc32c truncation without any extra
state.
code stack
before: 32044 2880
uncrc32c: 32108 (+0.2%) 2880 (+0.0%)
flcksum: 32132 (+0.3%) 2880 (+0.0%)
Our crystallization threshold doesn't really describe the bounds of an
object, and I think it's a bit easier to think of it as a threshold for
block compaction.
Heck I've already been calling this the crystallization threshold all
over the code base.
An important change is this bumps the value by 1 bytes, so
crystal_thresh now describes the smallest size of a block our write
strategy will attempt to write.
Heuristically:
- data >= crystal_thresh => compacted into blocks
- data < crystal_thresh => stored as fragments
This trades a runtime check for a different function call. Enforcing
some minor semantics in the function's type/asserts.
This also makes it so there are no special tag bits used during rbyds
lookup, only rbyd commits.
In theory this saves a bit of code, we don't have a runtime check, but
in practice the extra function apparently outweighs the cost of the
runtime check:
code stack
before: 31956 2880
after: 32024 (+0.2%) 2880 (+0.0%)
We already get the leaf rbyd as a part of btree lookup, and since ids
can't be split across rbyd boundaries, we can be sure any bptr attrs
live in the same rbyd.
This can be extended to any future bptr attrs.
Aside from the small performance gain, this also means we can drop the
btree bid+tag lookups. All extra attr lookups to lookup the rbyd first.
This saves a bit of code but also avoids a set of issues with the btree
semantics where lookupnexting an extra attr can return ENOENT
prematurely when on an rbyd boundary.
As I'm typing this I realize this means we have no way to iterate over
all _tags_ in a btree, only over all _bids_. Fortunately I don't think
we will ever need the former.
code stack
before: 32136 2880
after: 31956 (-0.6%) 2880 (+0.0%)
This avoids a hacky shoehorning of our attr-list while also making
lfsr_btree_carve a nice reflection of lfsr_ftree_lookupnext.
I'm kind of surprised how little this impacts code size, maybe const
propagation already pushed the attr bounds through?
code stack
before: 32124 2888
after: 32128 (+0.0%) 2880 (-0.3%)
- Becksums in lfsr_ftree_carve can be preserved when slicing a right
sibling block.
The best way I can think to do this right now is to simply copy the
becksum tag as an extra attr. This unfortunately means
lfsr_ftree_carve doesn't exactly scale well, each new block-related
tag will increase the stack cost by at least one attr.
Fortunately we really should have that many block-related tags. The
only other planned one currently being content/parity ids.
- Fixed incorrect becksum check that meant we were only leveraging
becksums where bptr.off = 0.
Humorously, since lfsr_ftree_carve didn't support becksums, bptr.off
was always 0 when becksums were involved.
- Fixed issues where bptr.off != 0 in lfsr_ftree_flush (only caused by
becksums) was not accounted for, leading to progs beyond a block
during block compaction.
Code changes:
code stack
before: 32100 2832
after: 32124 (+0.1%) 2888 (+1.9%)
Block/fragment relevant variables are at least now localized to their
respective loops, though block writing still has some ugly gotos to
skip a few lookups when erased-state is found.
Though finding erased-state does sort of just break through the rest of
the block-writing heuristics, so maybe it's good that the code matches
the underlying logic...
Also made crystal_size lookups a bit more aggressive. The previous logic
assumed worst-case crystal size when near the beginning of a file, and
best best-case crystal size near the end. At the very least, this is
wildly inconsistent with crystals in the middle of sparse files.
Much like the erased-state checksums in our rbyds (ecksums), these
block-level erased-state checksums (becksums) allow us to detect failed
progs to erased parts of a block and are key to achieving efficient
incremental write performance with large blocks and frequent power
cycles/open-close cycles.
These are also key to achieving _reasonable_ write performance for
simple writes (linear, non-overwriting), since littlefs now relies
solely on becksums to efficiently append to blocks.
Though I suppose the previous block staging logic used with the CTZ
skip-list could be brought back to make becksums optional and avoid
btree lookups during simple writes (we do a _lot_ of btree
lookups)... I'll leave this open as a future optimization...
Unlike in-rbyd ecksums, becksums need to be stored out-of-band so our
data blocks only contain raw data. Since they are optional, an
additional tag in the file's btree makes sense.
Becksums are relatively simple, but they bring some challenges:
1. Adding becksums to file btrees is the first case we have for multiple
struct tags per btree id.
This isn't too complicated a problem, but requires some new internal
btree APIs.
Looking forward, which I probably shouldn't be doing this often,
multiple struct tags will also be useful for parity and content ids
as a part of data redundancy and data deduplication, though I think
it's uncontroversial to consider this both heavier-weight features...
2. Becksums only work if unfilled blocks are aligned to the prog_size.
This is the whole point of crystal_size -- to provide temporary
storage for unaligned writes -- but actually aligning the block
during writes turns out to be a bit tricky without a bunch of
unecesssary btree lookups (we already do too many btree lookups!).
The current implementation here discards the pcache to force
alignment, taking advantage of the requirement that
cache_size >= prog_size, but this is corrupting our block checksums.
Code cost:
code stack
before: 31248 2792
after: 32060 (+2.5%) 2864 (+2.5%)
Also lfsr_ftree_flush needs work. I'm usually open to gotos in C when
they improve internal logic, but even for me, the multiple goto jumps
from every left-neighbor lookup into the block writing loop is a bit
much...
Mainly by not looking up left neighbors after the first of many
block/fragment/crystal writes.
The right neighbors should already be avoiding redundant lookups since
redundant lookups imply a full fragment/block on the not-last write.
Additionally, once we fail our crystallization check, we can assume all
future fragment writes will fail, so we only need to do that lookup
once.
There's probably still more to optimize, but the way these heuristics
interact are tricky...
Christopher Haster