Now mixing in truncate/fruncate, along with desync<->sync state
transitions.
Found bugs:
- Fixed propagating LFS_F_UNSYNCED/LFS_F_UNFLUSHED state during sync
broadcasts. This is important for tracking small files correctly.
- We were not clearing the btree erased-state of other opened file
handles when we started using it, leading other file handles to have
out-of-date erased-state.
I considered moving this into lfsr_btree_commit, but file btrees are
really the only place where shared references make sense, and it feels
weird to scan file btrees every time we commit to the mtree.
- Fixed syncs not propagating to other file handles when file is synced
with disk.
It's interesting that lfsr_file_sync can actually have an effect on
the system when the disk in is-sync.
- Added O_FLUSH/O_SYNC support to lfsr_file_truncate/fruncate. This
omission was just an oversight.
Unfortunately this did add quite a bit more complexity to both
functions.
You may notice in the fix for that last bug, that lfsr_file_ftruncate
sort of drops the ball with regards to error-idempotency. This is
because, as I was trying to figure out how to recoverably move the
buffer around when fruncating small files, I realized we don't handle
small files in lfsr_file_write correctly w.r.t. error-idempotency, and
that fixing this may be intractable...
The issue is how handle overwrites for unflushed buffers.
In general, the correct thing to do when an incoming write overlaps our
file buffer, is to just write over the buffer with the new data.
Ah, but if we do this, how do we get the old data back if we run into an
error writing the data to disk? It's gone!
For normal files, this is not an issue. We can always flush to disk to
reclaim our buffer, and since a flush doesn't change the file contents,
it's fine to make this our new fallback state.
But for small files, flush is a noop, we keep these entirely in RAM.
There are some possible workarounds:
- Flush small files to disk before overwriting, sort of defeats the
purpose of caching these in RAM...
- Reread small files from disk, because that's definitely what you want
to do when you hit an error...
Also, to always have something we can read from disk implies flush
on overwrite, see above.
- Sacrificing half our buffer for staging small files. Because RAM cost
is totally not a priority...
Long story short, rethinking idempotent errors.
A recent change, motivated by user feedback, was to delay write buffer
flushes as much as possible. Before, littlefs would always flush the
buffer during lfs_file_seek, but now, buffer flushes can be delayed all
the way to lfsr_file_read, or even skipped entirely thanks to bypassing
reads.
This is all fine and dandy, except it's easy to imagine a use case where
a user might really not want a _write_ error to pop out of a _read_
call.
With this new behavior, avoiding this situation is impossible.
So enters a function common to other filesystems: lfsr_file_flush.
However it's value is quite a bit different here. Unlike flush in other
filesystems, this flush does not necessarily make data accessible on
disk. It only writes to the pending file snapshot, which is not
accessible until lfsr_file_sync.
This makes flush a function with a rather narrow scope in littlefs
(pretty much just preventing write errors in read), but since we had
already implemented this function for internal plumbing, it adds _very_
little cost.
I'm more concerned about potential user confusion around sync vs flush.
Curiously, exposing lfsr_file_flush actually _saved_ code size for some
reason. Not sure what would make that happen:
code stack
before: 33544 3072
flush: 33536 (-0.0%) 3072 (+0.0%)
flush+O_FLUSH: 33548 (+0.0%) 3072 (+0.0%)
The motivation for this comes from the observation that many users call
sync on every file write. Much more than I expected. I think one reason
is in embedded systems it's common to just write structs to disk, either
the whole file or to a log.
O_SYNC exists in POSIX/Lunix/etc, so it makes sense to provide in
littlefs. In theory it's just one extra function call, and may even save
in total application cost (though we don't measure this) by reducing the
number of function calls at the application-level.
---
Unfortunately in-practice turned out to be quite a bit different than
in-theory... The main culprit being the improved guarantees around error
atomicity...
The ideal guarantee is that if there is an error during a write, the
entire write operation is reverted. Combining this with O_SYNC means we
need to hold a copy of the origin file state all thwe way through our
sync call. This got a bit messy...
The annoying part isn't even the functionality! Our system of tracking
btree/bshrub snapshots is quite robust! The problems were entirely with:
1. Figuring out how the heck to avoid clobbering the old file buffer
state.
2. Figuring out how the internal APIs should work while passing around a
bunch of staging state.
For 1., fortunately, thanks to bypassing writes, and some careful
pointer manipulation, we can void buffer clobbing. And for 2. just some
internal API work was needed. Internally all syncs end up in
lfsr_ftree_sync, though this feels a bit clumsy since the functionality
is not really ftree related...
Unfortunately, all of this added up to quite a bit more code cost than
I had hoped. In theory, adding some sort of LFS_CERAMIC/LFS_GLASS modes
that relax error atomicity for code size could help with most of this?
But it needs some thought:
code stack
before: 33324 3072
after: 33544 (+0.7%) 3072 (+0.0%)
Desynchronized files are a new concept intended to capture some useful
quirks of the previous multiple-open-file behavior.
This adds:
- LFS_O_DESYNC - Mark a file as desync during open
- lfsr_file_desync - Mark a file as desync whenever
- lfsr_file_sync - Mark a file as NOT desync, and sync the file
Desynced files:
1. Don't recieve updates from writes to other file handles. This makes
desynced files act as a sort of snapshot of the file at the time it
was marked desync.
2. Don't call lfsr_file_sync on close. Unless lfsr_file_sync is
explicitly called, changes to desynced files are not reflected on
disk and not broadcasted to other file handles.
A side-effect of 2., is that this gives you a quick way to abort a file
write. Marking a file as desync and then closing the file will never
error.
Additionally, if an error occurs during a write operation, the file is
implicitly marked as desync. This provides graceful write aborting in
unlikely error cases. This has actually always been a feature in
littlefs, it was just named differently and didn't have an optional
recovery mode.
Since littlefs actually has to do more work to keep files in sync, the
desync feature is quite cheap:
code stack
before: 33324 3072
after: 33360 (+0.1%) 3072 (+0.0%)
Now, when files are synced, they broadcast their disk changes to any other
opened file handles. In effect, all open files match disk after a sync
call to any opened file handle pointing to that file.
This was a much requested feature, as the previous behavior (multiple
opened file handles maintain independent snapshots) is pretty different
from other filesystems. It's also quite difficult to implement outside
of the filesystem, since you need to track all opened files, requiring
either unbounded RAM or a known upper limit.
---
A bit unrelated, but this commit also changes bshrub estimate
calculation to include all opened file handles. This adds some annoying
complexity, but is necessary to prevent sporadic ERANGE errors when
the same file is opened multiple times.
The current implementation just refetches on-disk metadata. This adds
some maybe unnecessary metadata lookups, but simplifies things by
avoiding the tracking of on-disk sprout/shrub size, which risks falling
out of date. Keep in mind we only recalculate the estimate every
~inline_size/2 bytes written.
Just like lfsr_mdir_estimate, this scales O(n^2) with the number of
opened files (this are basically the same function... hmmm... can they
be deduplicated?). This is unlikely to be a problem for littlefs's use
case, but just something to be aware of.
Code changes:
code stack
before: 32920 3032
after: 33192 (+0.8%) 3048 (+0.5%)
Christopher Haster